Google’s exclusive arrangement granting it access to Reddit’s vast repository of content for $60 million is now live. Google initially announced this partnership in February, and it seems Reddit is now acting on the deal. This aims to bolster Google AI capabilities while limiting other search engines’ ability to access Reddit’s data. However, Reddit currently denies that the restrictions are because of its deal with Google.
Reddit’s Robots.txt Update
On July 1, 2024, Reddit implemented a robots.txt update, blocking automated agents from scraping its site, except for Google. The file specifies, “User-agent: * Disallow: /,” effectively cutting off all bots unless explicitly allowed. The decision is aimed at curbing the commercial misuse of Reddit’s publicly available data.
Platforms like DuckDuckGo, Microsoft Bing, and Mojeek can no longer serve recent Reddit content. Users on these search engines encounter outdated or fragmented information. DuckDuckGo now shows only limited links without descriptive content, and Mojeek’s CEO, Colin Hayhurst, confirms their crawler has been blocked since June. Microsoft also stated that Bing had stopped retrieving new Reddit data following the robots.txt change.
Google’s Strategic Advantage
Google’s exclusive access to Reddit’s data enhances its position in the search engine landscape. This deal enables Google to harvest Reddit content for AI training, improving its AI products while restricting competitors’ ability to provide comprehensive search results. The introduction of Google-Extended, a bot for Google’s generative AI tools, in 2023 further strengthens Google’s strategic standing.
Tim Rathschmidt, a spokesperson for Reddit, clarified that the restrictions on other search engines are unrelated to their collaboration with Google. Discussions with various search engines failed because some could not commit to enforceable agreements regarding the usage of Reddit’s data, particularly for AI applications. Reddit highlights that academic and non-commercial entities will retain access under certain conditions.
Last Updated on November 7, 2024 3:31 pm CET
