We Analyzed Millions of AI Bot Scrapes. Here Are 6 Crazy Things We Found

We all know that AI bots are scraping our sites, using what they find to deliver answers in chatbots and AI search. But just how frequently are the AI platforms hitting our sites? What pages are they most interested in? Are scrapes resulting in traffic? Or are AI bots just scraping our content to summarize in their results and train their bots without any real benefit to us?

These are the questions our team has been asking for months. So we’ve been using an AI analytics tool to track all the activity originating from AI bots on our sites to understand what they’re doing behind the scenes.

And let me tell you…it was quite interesting. In some cases, it was interesting in a good way; we could see the pages that AI is scraping the most and glean insightful takeaways from that. In other cases, it was not so good; we could see for sure that while AI bots are scraping our site millions of times, they’re certainly not sending millions of visits our way (we’re not surprised, but a girl can dream).

So if, like us, you’re curious what AI bots are doing on your site, what types of content they’re interested in, and what ratio of traffic they’re potentially sending your way, I’ve got you covered.

Contents

• AI bots scraped our site millions of times but referred a much lower volume of traffic
• Our most resource-intensive data studies are the pages most frequently scraped
• AI bots and real people have different reading habits
• Some pages have a higher scrape-to-referral ratio (AKA they’re easier to plagiarize)
• OpenAI was by far the top scraper
• The AI scrape-to-referral ratio is getting worse over time

6 crazy things we found from tracking our AI analytics data

Here are the most interesting, kind of scary, or just plain maddening takeaways from tracking our AI scraped data.

1. AI bots scraped our site millions of times but referred a much lower volume of traffic

Here’s the headline: AI scraped our site over 4.45 million times in just six months. What do we have to show for it? A scrape-to-referral ratio of 201:1. Which, to spell it out, means that we received one visit for every 201 times AI bots scraped our site.

That means that AI systems consumed our content at a rate more than 200X higher than they referred users back to our site.

I’m being a little redundant, but it’s worth driving the point home.

Especially when you consider that AI bots scraping your site does have an impact on your site. With bots now making up over half of web traffic, their frequent visits to your site can actually increase your hosting bandwidth and any costs associated with that.

It’s also getting harder to track and mitigate any potential issues that arise from their frequent visits.

“It used to be easier to classify the bots visiting our sites and to know if they were malicious, or search crawlers, or RAG bots,” said Justin Al-Qudah, director of web strategy and growth for LocaliQ.

“We’ve seen a massive increase in the number of bots hitting our sites, and with that increase, the majority have an unknown classification. This means more time reading up on the purpose of the bots and determining whether we block their crawls.”

“Not only that, but some have found fun new ways to sneak into our analytics, which means troubleshooting just to get a clear view into your analytics,” Justin said.

👀 Looking for more ways to drive people to your site? Free guide >> 25 Ways to Increase Traffic to Your Website

2. Our most resource-intensive data studies are the pages most frequently scraped

When we looked at the most-scraped pages on our site, we noticed almost all of them had something in common: they were data-focused.

Top scraped pages:

1. Facebook Ads Benchmarks (2025)
2. Our homepage
3. ChatGPT Plus vs. Free
4. Google Ads Benchmarks (2025)
5. Best Time to Post on Social Media
6. Conversion Rate Benchmarks
7. Average Cost Per Click
8. Facebook Ads Benchmarks (2017)
9. 2026 Video Marketing Trends
10. Facebook Ads Benchmarks (2024)

Our benchmarks (Google Ads, Facebook ads, and conversion rate), a post around the average cost per click, and data on the best time to post on social media hit the top 10. These are some of our most resource-intensive posts—requiring hours of data analysis, time identifying and reaching out to experts to contextualize the data, and design resources to create our data visualization pieces. To know that these pages are getting scraped most frequently without, in many cases, driving traffic or even credit is pretty frustrating.

Funnily enough, the main post that doesn’t really fit the mix—ChatGPT vs. Free—is one that doesn’t get a ton of traffic, but it was one of our top-scraped pages. Which begs the question: Does AI love reading about itself?

But I digress. Aside from that post, this list shows that AI bots heavily favor (at least on our site) data, statistics, and definitions. Which isn’t a surprise—GEO experts have been preaching the importance of first-party data if you want to get sourced in AI search—but it’s definitely validated with this list.

⬇️ Download some of our best reports: 2026 Google Ads Benchmarks + our Facebook Ads Benchmarks >>

3. AI bots and real people have different reading habits

While there is some overlap in our most-scraped pages and our most-visited pages on our website (for instance, our advertising benchmarks are consistently top performers), there’s a lot of variation. Strange, almost like bots and real people have different goals. People are looking for ideas, tools, and data to help inform their strategies. And bots are looking for first-party data and research they can take and use as their own.

Our most-visited pages overall—not from AI referrals—include two of our free tools (our Free Keyword Research Tool and our Free Google Ads Performance Grader if you want to check them out 👀) and some more evergreen content or ideas-related posts.

As we’ve talked about in our blog before, free tools are a bit of an AI-search buster—AI search bots can’t replicate these tools in search results, and there’s not much for them to crawl, aside from recommending them when someone asks for a set of tools.

4. Some pages have a higher scrape-to-referral ratio (AKA they’re easier to plagiarize)

When we look at the pages that are generating the lowest scrape-to-referral ratio, we see data-related posts winning again, along with in-depth learning resources. This is likely because people are seeing the data sourced in AI search results and then clicking to get more details, validation, or (in our dreams) expert insights to explain the data.

When we look at the pages with the highest scrape-to-referral ratio, we see that they are probably instances where someone is getting the full answer directly from an AI search result.

If you ask for social media image sizes, an AI platform like ChatGPT is probably just replicating (or simply stealing) a chart from our post (or from a smattering of posts) and calling it a day. There aren’t many opportunities to follow up. (AI search platforms, you’re welcome!)

5. OpenAI was by far the top scraper

And…we’ve got a stalker. It’s OpenAI with by far the largest share of AI crawler traffic to our site. I’m talking nearly 3X the number of scrapes compared to Meta. This isn’t surprising, as OpenAI, which powers ChatGPT, is one of the largest AI chatbots around, with an over 77% market share.

Speaking of Meta, we were surprised to see them and ByteDance not only on the list, but so high on the list. Are people really using their AI search functions frequently enough to warrant this level of scrapes? Curious. Especially considering Google is so low on the list, and we actually see a good number of referrals from AI Mode. However, our SEO analyst did inform me that he’s seeing Meta AI start to send some referrals our way.

6. The AI scrape-to-referral ratio is getting worse over time

Annoyingly, AI is sending us less traffic over time. The scrape-to-referral ratio worsened by 19.2% from 207:1 to 248:1 over a three-month period.

Could this be because as more people rely on AI search, they’re clicking into results less frequently? Perhaps. Google searches now end without a click nearly 70% of the time, up from just over 60% two years ago.

Searchers seem to be getting more comfortable getting their answers directly from AI chatbots without validating the information directly from the source.

AI chatbots have also adjusted the way they present and source information, oftentimes removing or hiding the original data or source.

This search result from ChatGPT answers my question without a single source listed.

This shift has caused what Daniel Stanica is calling “the great blogging collapse.” As blogs have seen traffic decline due to AI search (in his study, the average blog had lost 85% of organic traffic over four years), many businesses have stopped publishing altogether. And even news publishers are feeling the heat—with many beginning to consider blocking AI bots from crawling their websites at all.

As bots continue crawling at such high rates without sending meaningful traffic back to websites, this problem is only going to compound. Which means less relevant, insightful, and useful content for humans to consume (and less good content for bots to learn—or steal—from).

What this all means

Why share this data? Well, first of all, our team found it particularly interesting as we’ve been hearing how important it is to optimize your content to get found in AI search results. To do that, obviously your site is going to need to be scraped.

As evidenced by some of the most-scraped pages on our site, AI bots are simply taking our first-party data, research, and insights and replicating it directly within their results. This isn’t surprising; we understand what the modern search experience looks like. But between this and our data on whether or not you should try to optimize for AI Overviews, it does beg the question:

Is the rate at which AI bots are scraping our site really worth the measly amount of traffic it’s sending our way?

Many other website owners have the same concerns, as evidenced by this note on Reddit:

Secondly, it is interesting to see the pages that AI bots are scraping and sending traffic to at a higher rate. Those are areas we might want to focus on to play the GEO game.

Overall, we know that AI search isn’t going anywhere, and the modern search experience has made AI a big piece of the puzzle. So, we’ll take this data and use it to continue adjusting and optimizing our own content plan for the channels, topics, and strategies that work best for us (not necessarily the bots).