Introducing Surveys into Evidence: How to Avoid Bias

Consumer surveys have been gaining prominence in an increasingly broad range of civil litigation.

Used in trademark infringement matters for decades, surveys are increasingly employed in cases involving mergers (to predict future consumer behavior), patents (to quantify damages), false advertising (to evaluate consumer harm), collusive behavior (to assess the impact on consumer demand), and employment-related class actions (to fill evidentiary gaps). When properly vetted, survey evidence can be a crucial component of a litigation strategy – especially in situations where other sources of data are unavailable.

The relevance and usefulness of surveys submitted by experts in any legal context, however, are depen­d­ent on how they are designed and implemented. The avoidance of bias, either in fact or appearance, is central not only to a survey’s admissibility, but also to the probative weight accorded to the survey expert’s testimony. Three types of bias are especially relevant in this context. 

Selection Bias. A key element of a reliable surv­ey involves identifying the appropriate “universe” of respondents. If the sample is overly broad, the survey runs the risk of including results not relevant to the question at hand. If it is overly narrow, it may not give the trier of fact the full picture. Either misstep may lead to the exclusion of survey results from evidence.

Information-related Bias. An appropriate and admissible survey must ask the right questions in the right way. The phrasing of questions, methodology, experimental design, and survey administration all can be subject to scrutiny in a courtroom. Recent litigation outcomes also suggest that the survey expert’s decision process in determining how questions are asked should be made as transparent as possible to the trier of fact.

Analytical Bias. Different survey and experimental designs require different methods to analyze the data. Analytical bias can enter if the analytical method doesn’t match the design, or if the results are interpreted in ways that favor the researcher’s own point of view. To help counter a jurist’s natural skepticism about the subjective nature of surveys, survey experts and their teams should also be able to support the survey results with other evidence and analyses.

Survey evidence is likely to remain a crucial component of many litigation strategies. Following best practices to eliminate bias in survey design and implementation can significantly aid the case for the survey’s admission and impact. ■

Rebecca Kirk Fair, Managing Principal
Laura O'Laughlin, Principal

Adapted from "Ensuring Validity and Admissibility of Consumer Surveys," by Rebecca Kirk Fair and Laura O'Laughlin, published in Consumer Litigation, Winter 2017