Tag

robots.txt

Close-up of HTML code on a computer screen in bright colours

SEO & AI search

Google and Microsoft back one robots.txt rule to block AI training

Google and Microsoft will read one shared robots.txt rule that tells AI firms not to train on a site. Bing adds a separate setting for AI answers.

31 Oct 20263 sourcesGoogle Search Central Blog, Microsoft Bing Blog, Search Engine Roundtable

Laptop on a wooden desk showing website code

Meta puts its commercial terms on hold. Google and OpenAI tie ad bots to ads

Meta paused its new terms on October 28 and set no new date. Google and OpenAI limit their ad bots to pages that carry ads.

26 Oct 202612 sourcesReuters via The Spokesman-Review, Meta Commercial Terms (preview), Meta for Developers: Meta Web Crawlers, Google Search Central: special-case crawlers, Google Search Central: common crawlers, OpenAI: Overview of OpenAI crawlers, EUR-Lex: Directive (EU) 2019/790, Polish Chamber of Publishers (PIW), Reuters via BusinessDay, PPC Land, Nieman Lab, The Register

SEO & AI search

Meta Commercial Terms let it crawl business websites for AI from October 30

Meta's updated Commercial Terms, live from October 30, allow it to crawl the websites of businesses on Facebook, Instagram or WhatsApp and use the content to train its products.

18 Oct 20262 sourcesNieman Lab, Search Engine Journal

Code editor on a monitor with a file tree open on the left

SEO & AI search

Google says blocking Mediapartners-Google now touches Ad Manager, not just AdSense

Google widened the description of its Mediapartners-Google crawler on 17 September 2026. Rules aimed at it now affect Ad Manager, not only AdSense.

18 Sep 20261 sourcesGoogle crawler documentation