Amid the demand for dats used on AI projects, the Wikipedia-style service Baidu Baike recently banned the search engine crawlers of Google and Bing from indexing its online content, according to the South China Morning Post.
A new update of the robots.txt filw of Baidu Baike’s, which is a file that communicates and tells search engine crawlers which web addresses can accessed from a site, has blocked the abilities of both Googlebot and Bingbot crawlers to index content from the Chinese platform.
From all indications, it appears to have occurred around August 8th, per the report, which references the Wayback Machine, and internet archiving service.
Neither Google or Microsoft have commented publicly on the development.
