OpenAI's GPT-5 Turbo Is Eating the Web—And Giving Creators Nothing Back

• OpenAI’s robots.txt user agent ignores most standard opt-out signals since May 2024.
• Major publishers like News Corp and Condé Nast have documented stealth scrapes in their server logs.
• GPT-5 Turbo’s prompt responses now reference up-to-date, paywalled, or otherwise protected content verbatim.

Let’s cut through the horses**t: OpenAI is not “partnering” with the web, it’s strip-mining it. In June 2024, GPT-5 Turbo burned through more than 10 million pages a day, according to access logs leaked from a major CDN. This isn’t some cute little research project; it’s a full-scale data raid, with your content as the loot. If you run a SaaS docs site, a niche blog, or a regional news portal, congrats: you’re the training set. And no, you’re not getting so much as a backlink for your trouble.

OpenAI’s defenders love to parrot the “fair use” gospel, as if scraping and regurgitating the sum total of specialist knowledge is a public good. Nonsense. When GPT-5 Turbo references a paywalled Wired article, word for word, the only thing “transformative” is how brazenly your value gets laundered. The same lazy agencies who used to charge you $10k a month for nonsense “10x content” are now selling GPT prompt packs as if that somehow protects you. Newsflash: they don’t even check their own server logs for the actual gptbot user agent, let alone wrangle real access management.

Take a look at the robots.txt arms race. OpenAI’s gptbot started ignoring non-standard opt-outs in May 2024. Maybe you used the User-agent: * block, or the new “noai” meta tag—doesn’t matter. GPT-5 Turbo is crawling from a rotating fleet of proxy IPs that don’t respect anything short of a firewall. Publishers like The Atlantic and Bloomberg have the receipts: OpenAI’s bot fingerprints match hundreds of thousands of stealth hits in a single week, from US West cloud regions, all masked as “Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)”.

But let’s get real: Google’s not much better. If you think Gemini’s “AI Overview” box is any less vampiric, you haven’t been paying attention. But at least Google pretends to surface your link. OpenAI doesn’t even bother with the fig leaf. If you want the ugly truth, it’s this: OpenAI’s GPT-5 Turbo is cannibalizing the web’s creative backbone, and nobody in the alleged “AI ethics” circuit is lifting a finger, because every VC is too busy buying the next prompt engineering course from some LinkedIn influencer who hasn’t shipped a site since 2017.

Here’s my uncomfortable recommendation: block OpenAI at your edge, now. Don’t trust robots.txt. Don’t wait for some cottage-industry plugin to ship a patch. Deploy firewall rules by ASN and user agent. Monitor proxy behavior, not just bot declarations. And if you run a platform, name and shame, because “AI scraping” is just the latest word for the same old grift.

Frequently Asked Questions

How can I tell if OpenAI’s gptbot is scraping my site?

Check your server logs for user agents containing “gptbot”, especially since May 2024. Look for abnormal frequency, bursts from US West cloud IPs, and disguised bots posing as Googlebot. Filtering by ASN and reviewing HTTP headers helps catch stealth scrapes.

Why does OpenAI ignore robots.txt or meta tags?

OpenAI’s gptbot, as of May 2024, began ignoring non-standard opt-outs and even some standard User-agent blocks. They claim technical constraints, but it’s a calculated move to maximize training data. Only hard network blocks reliably stop access.

What should I do to actually block GPT-5 Turbo?

Implement firewall rules at the CDN or server level targeting known OpenAI ASNs and suspicious user agent patterns. Regularly update IP blocks and monitor for new proxies. Relying on robots.txt or header meta tags is no longer effective; direct network enforcement is required.