ADventures in Law

Home | ADventures in Law

In a recent case involving claims made by <a href="https://www.monarch.com/" target="_blank" rel="noreferrer noopener">Monarch Media</a> that its budgeting app helped users save money and manage finances, <a href="https://bbbprograms.org/media/newsroom/decisions/monarch-money" target="_blank" rel="noreferrer noopener">the National Advertising Division (NAD) took issue with the consumer survey data</a> the advertiser shared that was behind the claims. It is a good read for companies considering basing claims on surveys of their customers. What were some of the survey design issues that caused NAD concern?

core/paragraph

Questions were designed to have yes/no answers and forced participants to answer one way or the other. NAD was concerned that this framework may have introduced bias as there is a known tendency to “yea say,” or agree with, the questions. It is important to offer “I don’t know” and “no preference” options. In many other surveys, NAD has found fault with surveys asking participants whether they agree or disagree with a statement, even with these additional options. A best practice is to frame questions in as neutral a manner as possible to try to overcome the yea-saying phenomenon.

The questions didn’t match the claims. This was a flaw the advertiser tried to fix up front by rephrasing the claims, but even the redesigned claims suffered from a poor fit between the questions and the answers. For example, one of the claims was that “7 out of 10 couples feel more aligned in their finances after joining. The company changed the claim to 7 out of 10 couples say Monarch improved their money conversations with their partner.” The question Monarch asked, however, was whether as a result of joining Monarch a user has better financial conversations with their partner. NAD noted that although the survey asked individual user questions, the claim references responses by couples. So, there was no basis for reporting how couples reacted to joining Monarch. Similarly, Monarch offered a yes/no answer to the question of whether after joining Monarch users better understood where their money was going to support a claim that “80% of members say Monarch gave them a better picture of their money than before.” NAD felt the claim was much broader than the framing of the question. This may seem picky to some, but it illustrates how seriously NAD takes the issue of fitting the claim to the survey question.

The survey required participants to guess. Monarch asked participants how much they saved or reduced their spending to support a numeric claim about average savings customers reported per month or year. NAD has issues with conflating savings and reducing expenses, but it also was concerned that the short survey did not seem to allow or make it easier for participants to check on their progress, essentially inviting them to answer from memory. NAD had concerns that such specific quantified savings claims could not be based on unaided recall as it encouraged estimating or perhaps even guessing.

There were other issues that NAD noted but did not pass judgment on, including the fact that Monarch invited a large number of current customers to take the survey but cut off participation after a few hundred responses. Making sure the sample size is robust and representative will be a topic for a blog another day! But from this case some good takeaways are to make sure you frame your claims to match quite precisely the framing of the questions. Don’t ask complicated questions that may encourage fudging or guessing. Draft the questions in neutral ways and always include “no preference” and “I don’t know” options.

NAD Has Clear Preference for ‘No Preference’ Options in Consumer Surveys, and Other Best Practices for Survey Designs for Claim Support