United States v. Google/Findings of Fact/Section 3C
- C. Internal Quality Studies
134. In 2020, Google assessed the impact of degrading aspects of its search quality for about three months, specifically its large ranking components (e.g., Navboost, Synonyms). See UPX1082 at 294. The experiment tested a quality decline of 1 IS point, a measure of search quality equivalent to the loss of two times the information contained on all of Wikipedia. See id.; Tr. at 6323:12-17 (Nayak) (“If we took Wikipedia out of our index, completely out of our index, then that would lead to an IS loss of roughly about a half point.”); id. at 4771:4–4773:9 (Whinston) (describing this experiment). This quality-reduction experiment correlated with only a 0.66–0.99% decline in global search revenue. UPX1082 at 294. In short, this study demonstrates that a significant quality depreciation by Google would not result in a significant loss of revenues. See id. But see id. at 6329:22-25 (Nayak) (“[I]f you made much larger IS changes, the relationship might not stay linear. It might become nonlinear. There might be inflection points where if you make search much worse, for example, you might actually lose a lot more traffic[.]”).
135. Google has at times tracked its competitors’ market shares or standing by identifying other GSEs and comparing Google to those rivals. See UPX399 at 965–66 (2014 document referring to Google, Bing, and Yahoo); UPX475 at 744 (2018 email chain and attachment calculating market share against other GSEs); UPX268 at 182 (2020 slide deck comparing Google, Bing, DDG, Qwant, and Ecosia).
136. When Google evaluates its own quality, it does so by conducting side-by-side experiments with other search engines. See Tr. at 6466:4-18 (Nayak) (discussing UPX2033) (describing side-by-side Google-Bing analysis with respect to queries relating to COVID-19). These studies involve IS4 rating systems that use human raters to compare results. Id. at 8099:425 (Gomes). In the past, Google has compared its latency and search results quality (using IS differences) to Bing’s. See id.; id. at 6457:13-21 (Nayak) (discussing UPX2022, a 2017 document comparing Google and Bing’s relative latency); id. at 7771:12-25 (Pichai). Google engages in an ongoing evaluation of Bing as part of its work. Id. at 8099:23-25 (Gomes).
137. Latency measures the speed with which a GSE returns search results and is an important quality metric. Id. at 1345:15-17 (Dischler). In 2017, Google analyzed its latency relative to Bing and determined that, for certain popular queries on Google, 25% of the time, the SERP took more than three seconds to load. UPX2026 at 122. Bing was “dramatically faster[.]” Id. at 123. Its first result arrived sooner 98% of the time. Id. This translated to about 300 milliseconds faster than Google. UPX2022 at 590. In response, Google launched Project Folly, “an attempt at instituting a set of projects and policies and processes to decrease latency.” Tr. at 6458:12-19 (Nayak). The project was a success. Id.
138. Google has also evaluated its privacy protections and IS metrics compared to those of DDG. Id. at 8099:17-19 (Gomes) (Google “use[s] IS4 and human raters to compare against competitors like” DDG).
139. Google does not compare latency or IS scores with social media platforms like TikTok “because they’re very different experiences.” Id. at 6467:8-14 (Nayak); id. at 8100:4-8 (Gomes) (IS ratings comparison with Facebook is “not something that [Google] could do easily”). The same is true with respect to specialized vertical providers like Amazon. See id. at 8100:1-3 (Gomes).
140. That said, Google has assessed the competitive threat posed by specialized vertical providers and social media platforms. For instance, in 2021, Google sought to understand whether younger users relied on social media instead of Google for search; the study concluded that youth have different behaviors that drive their desired search experience, one of which is increased importance on receiving recommendations from individuals. Id. at 8206:24–8208:11, 8249:23–8250:25 (Reid). Among “Generation Z” participants (defined as participants between the ages of 18–24 who use TikTok daily), 63% reported that they use TikTok as a search engine. DX241 at .032. And a 2015 Google User Experience Research study concluded that Google users frequently used specialized vertical providers’ mobile applications. See DX62A at .027–.028.