One of These Things Is Not Like the Others: Proper and Improper Uses of Logit Models in Expert Analysis

In a recent antitrust case, the judge excluded the results from a commonly used economic model. What does this mean for its future use?

What does an app that can tell you whether a plant is poison ivy or not have in common with an app you use to learn Italian? Not much, you might think – and the judge who oversaw In re: Google Play Store Antitrust Litigation would agree with you.

The distinction between a plant identifier app and a language learning app – or, rather, the failure to recognize the distinction – was at the center of US District Judge James Donato’s ruling to toss out an economic expert’s damages calculations.

For each app in the Google Play Store, the plaintiffs’ expert used a statistical model known as a logit model to calculate the pass-through rate – that is, the portion of Google’s charges to developers that the developers allegedly passed on to consumers.

According to the plaintiffs’ expert, the formula he developed for pass-through rates was based on the share of demand each app held in the category in which it was found in the Play Store. However, the method the expert used to estimate the share of each app was called into question by the defendant and subsequently scrutinized by Judge Donato.

A closer look at Judge Donato’s decision reveals that his criticism was aimed not principally at logit models in general, but rather at the misuse of a specific underlying characteristic: the independence from irrelevant alternatives (IIA) property.

Logit models and the IIA property

Since their development by Nobel Prize-winning economist Daniel McFadden in the 1970s, logit models have been used regularly to analyze consumer preferences for specific product attributes based on past choices. These preferences are then used to calculate demand curves – which, in turn, can be used to analyze how much consumer demand in a given market will shift to a substitute product when the price of a different product increases.

The IIA property of logit models is one of the features that makes them a popular choice because it reduces the computational need associated with the model. In plain terms, the IIA property means that introducing a product into the defined market, or removing a product from the market, will not change the relative odds of a consumer choosing between any other alternatives in that market.

Here’s a textbook example of how the IIA property works, adapted from one used by Professor McFadden in one of his papers on the topic.¹

What’s the problem?

For the IIA proportionality feature to be valid, however, all choices must be considered substitutes for all others, and the degree of their substitutability must be approximately proportional to their market shares. Indeed, the plaintiffs’ expert acknowledged that, when calculating app market shares, the logit model he used treated all apps in their respective Play Store categories as substitutes for one another.

That’s where PictureThis became a problem.

Google Play Store apps are categorized using broad rules that allow self-selection by app developers. PictureThis was in the store’s Education category, along with language-learning apps such as Rosetta Stone and Duolingo, as well as a multitude of other apps.

According to Judge Donato’s reasoning, while the two language apps named would be logical substitutes for one another, a plant identifier app would not:

Interestingly, at the class certification stage of Google Play Store, the defendant’s expert did not challenge the admissibility of the plaintiffs’ logit model. Consequently, the question at trial went to the weight of the evidence, not its admissibility.

In the end, Judge Donato excluded the results of the logit model because the plaintiffs failed to “provide usable guidance on what to do with the myriad differences and distinctions between apps within the Google Play Store categories.” The judge held that such evidence would be more likely to confuse the issue than clarify it for the jurors.

What does this mean for economic experts?

Logit-based models relying on the IIA property can continue to play a role going forward, particularly when the data available are limited. For example, data can sometimes be a limiting factor when evaluating whether a merger may lessen competition. In such cases, a logit-based model may offer convenient options for antitrust practitioners to obtain a preliminary sense of the competitive harm a merger or acquisition might engender.

However, the Google Play Store ruling underscores the need for practitioners to verify that the logit model they’re using – and its IIA property – reflects real-world choices, and that the underlying demand model and choice set are appropriately designed.

One way to do this is by employing statistical tests to determine whether IIA is a reasonable assumption for a market that’s being studied. Or an expert might consider using models that ease the IIA property, such as a nested logit model, or a more flexible random coefficients model, pioneered by Berry, Levinsohn, and Pakes in 1995.²

With a nested model, for example, imagine that a commuter can either take public transportation to work – the first nest, consisting of either a subway or bus – or drive there – the second nest, either driving alone or carpooling. If the IIA property is not an appropriate assumption when deciding between public transportation and driving, but it is an appropriate assumption when selecting options among either public transportation or driving options, then a nested logit structure may be a useful alternative to consider.

However, as Judge Donato’s ruling clearly brings into focus, regardless of the test or model employed, practitioners should use real-world data to check whether their predictions align with observed behavior. ■

Kristof Zetenyi, Vice President
Big Banternghansa, Vice President

Endnotes

1. McFadden, D., “Conditional Logit Analysis of Qualitative Choice Behavior,” in Frontiers in Econometrics, ed. P. Zaremba (1974).
2. Berry S., J. Levinsohn, and A. Pakes, “Automobile Prices in Market Equilibrium,” Econometrica: Journal of the Econometric Society, (1995).