Conjoint Surveys Can Lead to Inflated Values of Minor Product Features

Q&A with Professor David Reibstein and Principal Robert Vigil

In this Q&A, David Reibstein, the William Stewart Woodside Professor of Marketing at The Wharton School of the University of Pennsylvania, and Analysis Group Principal Robert Vigil discuss Professor Reibstein’s research, which shows that choice-based conjoint (CBC) surveys can lead to inflated values of product features when CBC is used incorrectly.

What are conjoint surveys?

Reibstein: Conjoint surveys are widely used in the field of marketing science to help researchers determine how consumers value product features. Rather than directly asking respondents how much they value certain features, conjoint surveys show respondents a series of product profiles that vary based on selected features, and ask them to choose which product they would purchase.

Respondents’ choices are used to determine, among other things, how important the features are to the customers sampled. Because price is often one of the included features, researchers are able to determine how much consumers are willing to pay for the features in the survey by examining the relative importance of the other features compared to price.

What kind of product features can be tested in a conjoint survey?

Reibstein: They can include any tangible characteristic of the product, such as its color or price, and any intangible claims or statements made about the product, such as “100% natural” or “toxin-free.”

How are conjoint surveys used in litigation cases?

Vigil: They can serve a variety of purposes, depending on the nature of the case. In patent cases, for example, conjoint surveys have been used to estimate how much consumers are willing to pay for product features that are accused of infringing a patent. In false advertising cases, they have been used to estimate the additional amount consumers are willing to pay for a product because of the presence of one or more false claims. In product liability cases, they have been used to estimate how much plaintiffs would have been willing to pay for a product had they known about certain undisclosed defects.

In each of these cases, the information from the survey is used to calculate the damages that must be paid.

When conducting conjoint surveys in litigation cases, how do researchers choose which features to include in their surveys?

Reibstein: To avoid cognitive difficulty for respondents, conjoint design limits the number of features that can be included in the survey, often to between five and seven features. As a result, researchers choose a subset of features to present to respondents, and then ask them to assume that all other features not presented are the same across choices. By necessity, the features or product claims at issue in the litigation, which are often not the primary drivers of customer demand, are included in the survey. This often requires the survey designer to omit more important features, which can introduce certain challenges to using conjoint.

Can you give us an example of the challenges you are talking about?

Reibstein: We have conducted research that shows that omitting major features can inflate the value of minor features that are included in the survey. While we believe that the degree of inflation depends on the survey design, as well as the product and the features or claims being examined, we find that omitting major features can cause the value, or willingness to pay, of minor features to be inflated by up to 200%.

That seems like a substantial difference. Before we get into the reasons for the inflation, how do you distinguish between major and minor features?

Reibstein: The distinction is rooted in whether the feature is a principal driver of the purchasing decision. Major features are such drivers, though they may not be the only features consumers care about, while minor features are not, even though consumers often have a preference about them.

For example, consumers probably have some preference for which side of their car the gas cap is on. But that feature will likely not determine whether they would buy a particular car, or which car they would select.   

So why does omitting major features cause the value of minor features to be inflated in a conjoint survey?

Reibstein: For at least two reasons. First, the CBC design can lead to a situation where the products presented to the respondent have the same major features and differ only as to the minor features. In these situations, respondents are forced to make a choice based solely on the minor features when, in reality, these features may not be purchase-decision drivers at all.

Second, when presented with minor features that they otherwise would not consider when buying a product, respondents may place more weight on those attributes because their attention will be drawn to them in the survey. This is often referred to as “focalism bias.”

Can you give us a recent example of a conjoint survey used in litigation that may have suffered from this problem?

Vigil: In one class action involving an alleged defect with the touchscreen on certain Apple iPhones (Thomas Davidson, et al. v. Apple, Inc.), the expert asked respondents to choose between phones differing in storage capacity, screen size, talk time, price, and the touchscreen defect. Other features, such as camera capability, processor, and brand were omitted. The results of this conjoint led to damages estimates associated with the touchscreen defect of between $323 and $432, which equates to between 51.7% and 69.2% of the price of the phone. This is almost certainly overstated given that the defect did not make the screen inoperable and manifested in less than 6% of phones.

Are you saying that conjoint analysis should never be used in situations where all major features cannot be included?

Reibstein: Not at all. Under certain circumstances, the relative preferences of features can still be validly estimated when major features are omitted. If a CBC is only being used to determine whether one feature is more important than another, its application is appropriate. It’s only when researchers seek to determine how much more important a feature is in terms of dollars that biases of the sort identified here begin to creep in.

Do you have any suggestions for how conjoint analysis can be used when you want to estimate the dollar value of minor features?

Reibstein: I would recommend that all major features of the relevant products be included, even if this means that you run a different survey for each relevant minor feature of interest, so that you can include more of the major features in the survey. While no CBC can ever perfectly mimic reality, the more major attributes that are included, the more valid the willingness-to-pay estimates that result from the conjoint will be.

Another possibility would be to utilize a different form of conjoint, such as a ratings-based conjoint. While a CBC only tells a researcher that a consumer prefers one product over another, a ratings-based conjoint informs the researcher exactly how much a consumer prefers one product over another. This increase in information may allow for a more accurate willingness-to-pay estimate.

Vigil: I also think that anytime you are trying to do this type of analysis, it’s important to check whether the analysis generates predictions consistent with outcomes that are observable in the marketplace. In other words, it’s important to assess the external validity of the results, if at all possible.