Caught Between Crawlers and Clicks: The Publisher’s AI Jail Dilemma
So, you create a fresh and juicy piece of content for your website, hit publish, but your first reader is… not even a human, but a bot (or two, of a whole party of bots). Some are your good old buddies (hello, Googlebot), others are something new – AI crawlers. And, of course, just like any innovation, these guys raise many questions and debate: say, some publishers dream of squeezing into ChatGPT’s recommendations, while others believe AI crawlers are here to eat your content for breakfast and never send a single click back.
Either way, it feels dramatic, like you can’t win, but is that really so? Let’s figure it out.
Contents
From One Bot to a Whole Zoo of Crawlers
Just a couple of years ago, life was so much simpler! You mainly cared about Googlebot and maybe Bingbot, depending on the markets you work in. They crawled your pages, indexed them, and sent you traffic. Now you’re dealing with a whole zoo: GPTBot (OpenAI), Google-Extended, ClaudeBot, PerplexityBot, CCBot, and more.
Several reports show that AI crawlers are no longer an exotic, small part of global traffic. They are so deeply rooted in our lives already that some analyses suggest that a big slice of all hits to many sites now comes from bots related to LLMs (Large Language Models), not humans.
Also, a recent scandal with one open-source documentation site, Read the Docs, came to the surface not so long ago. The owners measured its bandwidth and found that after it blocked AI crawlers, traffic dropped from around 800GB a day to 200GB! And that is actually a 75% cut in bandwidth and about $1,500/month saved.
So, on one side, AI crawlers may give you some visibility in AI tools – that’s true. What is more, today, chatbots are commonly perceived as being less motivated than other digital spaces, meaning they are more objective, less selling, and therefore can be trusted. This sounds like a good-to-have traffic source.
However, on the other side, they cost money, burn server resources, and don’t actually show ads or buy anything. What is more, they can deliver content from your website right to a user directly, which means they will have fewer reasons to visit your site. And this already makes some publishers reach for that big red “block” button.
Why It Feels Like AI Jail: Zero-Click Search and Lost CTR
As we already know, the real pain for publishers is not just that AI crawlers read your site, but rather what happens next in search results. The problem is that Google’s AI Overviews and other AI summaries answer user questions directly on the SERP, often using publisher content as raw material.
When we asked our SEO expert, for a blunt summary, he said:
AI crawlers are not malicious, but they are existentially disruptive.
They are built with one main goal in mind: to keep users inside their own platforms (and it looks like they are doing pretty well). Now, Google, OpenAI, Perplexity, and similar players all want to be the final answer box, not just the gateway that sends people to your site. To do that, they extract and summarize your content, then serve it back to the user – often without giving you a meaningful click in return.
This shifts SEO from the old game of rankings to a new game of feeding large language models. Those models don’t always credit you properly, and they definitely don’t always send traffic. Before, Google’s classic cannibalization didn’t go beyond snippets, but now it’s deeper than that. The question is now asked and answered inside a closed system, and in many cases, your site is simply background fuel.
That said, the picture isn’t all doom, because AI doesn’t completely flatten everyone. Models still prefer content that is authoritative, structured, and well-maintained. They still struggle to fully replace things like real experts, proprietary datasets and databases, first-hand experience, failures, and scars.
So yes, AI crawlers are disruptive. But if you build something more than just another how-to article, you’re not powerless in this new landscape.
Where AI Really Hurts
Our SEO expert states that AI-driven results can break visibility and CTR, but they do it selectively, not across the board. The most vulnerable group is classic informational, “zero-click” intent searches – all those queries where users really just want a short, clear answer:
- “How to…”
- “What is…”
- “Symptoms of…”
- “How many X per day…”
For this type of query, the same pattern keeps showing up. Impressions often stay stable or even grow, so on paper, your visibility looks fine; however, once AI Overviews or similar blocks appear, clicks start to collapse. Users see a ready-made answer at the top of the SERP and simply have no reason to visit the site that actually owns the content.
This is very similar to what we already saw with Featured Snippets and “People Also Ask,” just with the volume turned up. You can still have 100,000 impressions for a page, but your CTR evaporates because the answer is already on the results page.
Recent third-party research backs that up:
- Semrush found that AI Overviews now appear in over 13% of queries, and they hit informational content the hardest.
- Data, reported by the Guardian, shows some news publishers losing up to 80% of click-through rate when their links sit under an AI summary instead of at the top of a “normal” result set.
- And as Reuters notes, independent publishers in Europe have already filed antitrust complaints against Google, arguing that AI Overviews use their content without a fair opt-out while still cutting heavily into traffic and revenue.
From an SEO perspective, as our SEO expert highlights:
The pattern is now pretty clear: when a SERP includes AI modules, CTR tends to drop. When it doesn’t, the click curve looks much closer to the classic pre-AI world. For many sites, top-3 organic positions are suddenly behaving more like positions 6-10 used to.
In practice, that often looks like this:
Branded queries are still relatively stable, because users are actively looking for a specific brand or site.
Non-branded informational queries fall sharply once AI blocks appear above the traditional results.
On real projects, as our SEO specialist says, it’s no longer unusual to see a number one ranking slip from something like 28-40% CTR before AI features to roughly 10-22% CTR afterwards, depending on the niche and query type. It doesn’t happen for every keyword, but it happens often enough that you can’t pretend it’s a fluke.
Content Types That Suffer the Most in the AI Era
Now, it looks like this question needs additional attention, so we asked our SEO specialist which kinds of content are getting hit the hardest, and which still have a real shot at winning clicks. He said:
1. Simple informational content, like “How to reset your iPhone”
At the very top of the casualty list sits simple informational content. This includes definition posts, very basic step-by-step tutorials, and short guides.
If the answer comfortably fits into a short paragraph or a neat little list – “How to reset your iPhone,” “What is CPC,” or “Symptoms of food poisoning” – AI and Google’s own modules can:
- Grab the core information
- Display it directly in the SERP
- Leave users with almost no reason to click through
The verdict from our SEO expert is tough but honest: this kind of content is unsustainable on its own. It only makes sense if it is clearly connected to:
- A strong brand or authority in the niche
- A unique workflow or methodology that others don’t have
- A useful tool sitting behind it (a calculator, checker, quiz, etc.)
2. Commodity news
Next comes “commodity” news – general tech updates, trending gossip, crypto price blurbs, and basic “Apple announced new feature” pieces.
Here, AI summaries stacked on top of Google Top Stories create a particularly bad combo. Users get the gist of the story right there, and unless your coverage offers something unique, they rarely dig deeper, unless you can deliver:
- Original reporting
- Exclusive, first-party data
- Recognition as a trusted, first-party news source
And if you cannot, you’re basically competing against hundreds of AI-written rewrites. The verdict: this segment is essentially dead unless you’re already a recognized news site or brand.
3. Generic and affiliate “Top-X” reviews
Classic SEO/affiliate pages are another big casualty. You know the type:
- “Best VPN for Netflix”
- “Top-10 laptops for students”
- And pages built mostly from Amazon reviews, specs, and stock photos
AI can now generate comparison overviews for these products in seconds. Users get a solid high-level summary without needing to click through ten nearly identical listicles that most likely include bias and ads.
The only review pages that still reliably rank and earn are those that show:
- Hands-on testing
- Original scoring frameworks with expert opinion
- Real benchmarks, photos, videos, and proprietary data
In other words, they survive only if you inject serious E-E-A-T steroids (Experience, Expertise, Authority, Trust).
4. Simple recipes
Simple, mainstream recipe content is in a similar boat. For queries like “how to cook chicken breast,” Google now often shows:
- Instant step lists
- Ingredient breakdowns
- Image carousels
Users can basically cook the dish without ever visiting your site. There are still exceptions, though:
- Unique culinary storytelling or personality-driven cooking blogs
- Niche or ethnic cuisine with a strong community
- Highly opinionated or experimental food content
Overall, the verdict here is low viability unless you’re clearly niche, opinionated, or building a brand rather than just chasing keywords.
Where AI Doesn’t Fully Replace Clicks
The story changes when user intent shifts from “give me the answer” to “help me decide.” As our SEO expert claims, AI crawlers and AI-generated SERP modules do not break visibility equally for all query types, and this is where things get more encouraging for publishers.
Our SEO expert told us that queries with transactional or commercial intent are much less affected:
1. Deep analytical content (original, not an AI remix)
Long, thoughtful analysis is one of your best weapons. This includes:
- Long-form research pieces
- Benchmarking studies
- Real-world case studies and campaign breakdowns
- Experiments (“we tested this feature, here’s what really happened”)
- Industry data with charts and commentary
AI can summarize existing information, but it cannot magically generate:
- Your proprietary datasets
- Your real campaign results
- Your insider access and trade secrets (not sure you want to share them either, but just in case)
Think of articles like:
- “We ran $50k in TikTok ads and here are the 9 real ROI learnings”
- “100 Telegram mini apps analyzed: median retention and monetization models”
- “Real-world CPMs by GEO (2026) – TikTok vs Meta”
Search results are an increasingly rewarding experience, evidence, and authority. The direction is basically: “Tell me something nobody else knows.”
2. Interactive tools
Interactive tools are another powerful category, and they are extremely future-proof. We’re talking about:
- Calculators and calendars
- ROI estimators and forecast tools
- Keyword clustering visualizers
- Product selectors and decision trees
These assets actually solve the problem for the user and lock the user into your interface, not Google’s or a chatbot’s. What is more, this type of content tends to generate shares, bookmarks, and backlinks.
Think of mortgage or loan calculators, pregnancy due date calculators, keyword clustering tools like Thruuu or Keyword Cupid, and YMYL financial health tools. If you want future-proof SEO, don’t just publish another “10 tips” article – build something users will come back for.
3. Proprietary databases and indexes
Any content that revolves around a structured, proprietary database is also hard for AI to fully replace. That includes pages where you collect and normalize data, and users can browse, filter, and search through it.
Examples might include:
- SaaS pricing trackers
- App download indices
- CPM tables by GEO for different ad networks
- Scam-reporting databases
- Catalogs of Telegram bots, mini apps, or offers
These kinds of pages often become natural link magnets, strong brand assets, and massive topical authority boosters in their niche.
4. Expert commentary and first-person POV
AI can mimic tone, but it cannot fake your actual experience. It doesn’t have your personal failures, your €10k campaign scars, or your accurate industry intuition. Formats that tend to win here include:
- Founder or creator AMAs (Ask Me Anything) turned into structured breakdowns
- “What we tried that failed (and why)” articles
- “Lessons from $10M ad spend” style posts
- “How we built agency SEO from 0 to 200 clients”
This is E-E-A-T in its most weaponized form: instead of recycling already-existing knowledge, you’re sharing lived experience.
5. Multimedia-first content
AI Overviews currently struggle with video-first knowledge and rich, proprietary visuals. Real footage, authentic screenshots, and complex charts are still hard to compress into a neat AI answer. Things that tend to survive and thrive:
- YouTube videos and embedded clips
- TikTok content integrated into your pages
- Real dashboards, graphs, and screenshots from your tools or ad accounts
If you show real data and real screens, you’re automatically harder to replace.
6. Opinionated and “spiky” content
Finally, there’s an opinion! What can be more human than a critical (and maybe just a bit provoking) opinion?
We know that AI is mostly trained to be safe, generic, and neutral, but Google, on the other hand, increasingly rewards content with a clear, strong point of view, like contrarian takes, deep, niche expertise and bold, opinionated frameworks.
Think of headlines like:
- “Why this popular strategy is trash”
- “Stop doing X, do this instead, because”
- “Everyone’s wrong about X – here’s the data”
Opinions require identity, and identity requires authority – that’s something AI fundamentally doesn’t have.
Blocking AI Bots: Is It Safe for SEO?
Now, should you be radical towards those AI crawlers? You’ve probably seen the advice: “If you don’t want AI to steal your content, just block GPTBot and its friends.” That naturally leads to a nervous follow-up: if we block AI bots like GPTBot or CCBot but keep Googlebot, are we doing something dangerous to our SEO strategy? Our in-house expert responds:
Blocking AI bots ≠ blocking search crawlers
The first thing to understand is that robots.txt is granular. You can allow Googlebot while blocking AI scrapers. A simple example:
- User-agent: GPTBot
Disallow: / - User-agent: CCBot
Disallow: / - User-agent: Googlebot
Allow: / - User-agent: Bingbot
Allow: /
In practice, it works like this:
Googlebot handles organic search, rankings, and classic SEO.
AI bots (GPTBot, ClaudeBot, CCBot, PerplexityBot) handle model training, retrieval, and AI-style answers.
These are related but separate ecosystems, so breathe out. However, to make things even clearer, from here, it’s helpful to split the answer into two: direct SEO impact and indirect ecosystem effects.
Direct SEO impact, or else – your rankings
Our SEO expert’s verdict is clear: there is no direct SEO penalty just for blocking AI bots. Google’s ranking systems primarily care about:
- Googlebot
- Googlebot-Mobile
- Googlebot-Image
- AdsBot-Google (for paid)
As long as these crawlers can access your pages as intended, your indexing and rankings are safe. Blocking GPTBot or other non-Google AI crawlers does not automatically tank your SEO.
Indirect ecosystem risks
The crucial nuance comes from the indirect side. If your content is not available to AI models or AI-powered aggregators, you may miss out on:
- Spots inside AI search snippets
- Links and citations in AI answers that could have pointed to your site
- Traffic from new, AI-driven discovery channels such as Bing + ChatGPT integrations, Google’s experimental SGE/AI Overviews variants, and tools like Perplexity that sometimes display links
According to our SEO expert, this can be especially important if you rely heavily on news content, product comparison sites, and review hubs, tutorial/how-to content, and long-tail informational blogs.
In that sense, blocking AI bots can mean fewer brand mentions inside AI tools and fewer referrals from AI-powered experiences. So, from an SEO viewpoint, you can frame it like this:
Direct rankings are safe, as long as Googlebot and friends are allowed.
Indirect AI visibility can be at potential risk – your site might be invisible inside AI answers and AI search experiences.
What If You Don’t Want Any Crawlers?
Publishers don’t block AI crawlers just out of principle. There are very concrete reasons, but they are not about SEO:
- Content usage and/or hallucination: LLMs can summarize or remix your content without clear attribution and sometimes with mistakes.
- Competitive scraping: your content gets mined at scale by competitors or aggregators, simpler and faster.
- Legal, licensing, and IP concerns: you may simply not want your work training commercial AI without a deal.
- Costs and performance: bandwidth, servers, and CDNs are expensive, and bots don’t pay you for that.
Blocking GPTBot, ClaudeBot, CCBot, and similar agents can reduce that exposure and help protect both your content footprint and your infrastructure.
Not Just Robot.txt: How to Block AI Crawlers Smartly
So if you decide to get rid of those crawlers, we also have a piece of advice for you. Instead of copying someone’s robots.txt snippet from Twitter, you’d better think in terms of a real crawl policy.
Robots.txt is still your first line of defence, but many AI scrapers completely ignore it. That’s why more and more publishers implement layered protection, such as:
- Server-level rules (.htaccess, Nginx configs)
- Web Application Firewalls (WAF rules)
- Services like Cloudflare’s AI bot blocking and advanced bot management
- Rate-limiting or blocking known bad IP ranges and data centers
Some large platforms now even experiment with “pay per crawl” models, where heavy access by big AI companies is monetized rather than handed out for free. Reddit is a well-known example: first restricting many crawlers, then turning its data into licensing deals with Google and OpenAI.
Don’t rely on CAPTCHAs alone
During research for this piece, an affLIFT thread emerged where a member described a ChatGPT-based agent successfully navigating Cloudflare’s “I’m not a robot” Turnstile check on its own. The agent mimicked human behavior – mouse movement, timing, browser signals – well enough to pass.
One member joked that agents controlling a computer like a human “will have no problem clicking that,” and another pointed out that security systems need to evolve fast or CAPTCHAs will become useless.
So yes, CAPTCHAs still help against basic scripts – but they are no longer a serious shield against modern AI agents. If you care about who copies your pages, you’ll need:
- Clear rules about which bots you allow
- Edge-level blocking for the worst offenders
- Ongoing monitoring of your traffic patterns
Watch logs and analytics like a hawk! You will have to check your server logs for known AI User-Agents regularly, monitor bandwidth, response times, and suspicious IP ranges, as well as constantly compare analytics with monetization:
Many “sessions” but low engagement and earnings? > likely bots.
Huge bandwidth spikes without any revenue bump? > also suspicious.
To Sum Up
In short, AI crawlers aren’t pure evil, but they are rewriting the rules: they hit simple informational content the hardest, leave some commercial and expert formats still standing, and force you to think in terms of a real crawl policy instead of blind trust.
Your best move now is to protect your site from pointless bot load, invest in content and tools that AI can’t easily replace, and squeeze as much value as possible from every human who still lands on your pages. And if you want those human visits to turn into actual money, make sure your Monetag Smartlink or other preferred ad format is set. Track the efficiency of Monetag ads and see if the profit stays as high as always – this is also a great sign of your website getting out of the AI “jail”!
