arXiv:2605.13706v1 [cs.CR] 13 May 2026
Identifying AI Web Scrapers Using Canary Tokens Steven Seiden
Triss Ren
Caroline Zhang
Duke University Department of Electrical and Computer Engineering Durham, NC, USA
University of Pittsburgh Department of Computational and Systems Biology Pittsburgh, PA, USA
Duke University Department of Computer Science Durham, NC, USA
Taein Kim
Enze Liu
Emily Wenger
Duke University Department of Electrical and Computer Engineering Durham, NC, USA
Carnegie Mellon University CyLab Security & Privacy Institute Pittsburgh, PA, USA
Duke University Department of Electrical and Computer Engineering Durham, NC, USA
ABSTRACT From pre-training to query-time augmentation, web-scraped data helps to improve the quality and contextual relevancy of content generated by large language models (LLMs). However, large-scale web scraping to feed LLMs can affect site stability and raise legal, privacy, or ethics concerns. If website owners wish to limit LLMrelated web scraping on their site, due to these or other concerns, they may turn to scraper access control mechanisms like the Robots Exclusion Protocol. To be most effective, such mechanisms require site owners to first identify the scrapers that they wish to restrict (e.g., via User-Agent strings). Existing mechanisms to identify LLMrelated scrapers rely on voluntary disclosure by companies, one-off experiments by researchers, or crowd-sourced reports — methods that are neither reliable nor scalable. This paper proposes a novel technique for accurately and automatically inferring LLM-related scrapers. We host dynamic websites that serve unique canary tokens to each visiting scraper, then prompt LLMs for information about our sites. If an LLM consistently generates outputs containing tokens unique to a scraper, it provides evidence of exposure to that scraper. Via experiments across 22 production LLM systems, we demonstrate that our approach can reliably identify which scrapers feed which LLM, including several that are not publicly known or disclosed by the companies. Our approach provides a promising avenue for unprivileged third parties to infer which scrapers serve data to which LLMs, potentially enabling better control over unwanted scraping.
CCS CONCEPTS • Security and privacy → Privacy protections; Web protocol security; • Information systems → Web log analysis.
1
INTRODUCTION
Large language models (LLMs) power popular AI chatbots such as ChatGPT and Claude, among others. To achieve fluency on language generation tasks, AI chatbots are first pre-trained on massive datasets—typically culled from large-scale web scrapes—then undergo post-training “alignment” procedures to stamp out unwanted behaviors. This training process gives models baseline language generation capabilities. While sometimes impressive, these are often insufficient for the wide range of tasks chatbots perform.
Even well-trained AI models struggle with hallucinations [27], in which they confidently state incorrect responses, and cannot correctly answer questions requiring knowledge after their training data “cutoff date.” To solve this problem, AI chatbot providers have created live content retrieval strategies such as Retrieval Augmented Generation (RAG) [34] or web search [11, 17]. These additional capabilities allow models to supplement their responses with contemporary information from web sources, enhancing their performance on time-dependent queries (e.g. current weather, breaking news, etc) or complex factual questions. Numerous AI chatbot providers advertise web search capabilities (e.g. [2–6, 11, 17, 18, 21, 24, 26, 31, 42, 43, 50, 54, 65, 68, 69, 72, 75, 76]), and consumers have quickly adopted this new technology. Recent studies show significant increases in consumer use of AI chatbots for search, product discovery, and other functions traditionally performed by search engines [7, 41, 63]. Today’s leading AI chatbots have tens or even hundreds of millions of daily active users [40, 49], implying that chatbot’s web search feature is regularly used. Yet, the implementation details of these AI-powered web searches are poorly understood. To our knowledge, only two prior works [16, 36] have investigated the behaviors of scrapers, and both of them focused on live content retrieval. This leaves open questions of how different AI chatbots implement web scraping and whether their public scraping policies match what happens in practice. These questions are both of academic interest and connect to broader issues of AI-related web scraping. Large-scale collection of web data for AI chatbot pre-training has raised legal, privacy, and ethics concerns. In response to these concerns, numerous lawsuits have been filed over the unwanted use of internet-derived data in AI chatbot training [1, 8, 10, 28, 62, 70, 71, 80]. In this same vein, academic work has proposed various tools to disrupt AI-related scraping [57, 59, 60]. However, this work focuses on web data used for model pretraining, leaving the issue of unwanted real-time content retrieval unexplored. It remains unclear whether site owners who do not want their sites appearing in AI web searches—for performance, privacy, or other concerns—can opt out, whether via traditional scraper access controls like robots.txt [32] or otherwise. Our Contribution. Our study provides a critical first look at the infrastructure underpinning real time content retrieval in AI chatbots by identifying which web scraper User-Agents feed content to different chatbots during content retrieval. To do this, we deploy 20
Conference’17, July 2017, Washington, DC, USA
websites that feed differentiated content —which we term canary tokens —to visiting web scrapers. We then elicit information about our websites from 22 different production AI chatbots, recording which canary tokens from these sites are returned by each chatbot. We compare these tokens to those fed to visiting scraper bots, and when enough match, we know with some certainty that a chatbot was exposed to a specific scraper. Key findings. We use our canary token infrastructure to first map scrapers to chatbots, then explore whether simple blocking mechanisms like taking sites offline or employing a robots.txt file prevent sites from appearing in AI outputs. This experimental approach surfaces the following findings: • AI chatbots often seem to scrape and relay content only fed to search engine scrapers — even sometimes deviating from the official behaviors they self-report. • Many AI chatbots pretend to be regular browser-based visitors when displaying their User-Agent — some even rotating through many generic User-Agent strings. • AI chatbots can respond with data previously cached from websites — stored by either themselves or third-party search engines. • Blocking AI scrapers through robots.txt is largely ineffective at removing website data from an AI chatbot, although it is unclear whether this is due to caching or disregard for the robots.txt protocol. The rest of the paper is organized as follows: Section 2 provides background on how live content retrieval is performed by AI chatbots and outlines key research questions in this space. Section 3 gives an overview of our canary token methodology for matching AI scrapers to chatbots. Section 4 describes our experimental setup. Section 5 discusses results. Section 6 outlines limitations, implications, and future work.
2
BACKGROUND & RELATED WORK
Web scraping has been studied for decades, and myriad academic papers probe the inner workings of massive AI models, yet little work has studied the mechanisms by which web data feeds AI models in real-time content retrieval settings. This section describes prior work studying web data in AI models, real-time content retrieval, and preventing unwanted AI-related scraping. Along the way, we highlight key open research questions this paper aims to answer.
2.1
AI web scraping
Web scraping refers the process of retrieving content from the web using automated programs, known as web scrapersb [29]. While the most well-known application of scraping is search engine indexing, it is also widely used for other purposes such as dataset construction, academic research, and cyber forensics. Most relevant to this paper, web data is also widely used by AI models. The use of web-scraped data in machine learning predates modernday generative AI models. However, the emergence of generative AI models has significantly increased both the scale and the functional roles of such data. In the traditional machine learning context, web-scraped data was primarily used as training data. In modern AI-based systems, it’s set of use cases have expanded. In particular, web data supports generative AI models—and chatbots derived
Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, and Emily Wenger
from these models—in two primary phases: (1) as training data for model pretraining and fine-tuning, and (2) as an external knowledge source accessed at inference time to improve factual accuracy and timeliness [36, 47]. Web scraping for training. To obtain large-scale training corpora in a cost effective manner, model developers often rely on largescale web scraping. This approach is central to the development of modern AI systems. For example, large-scale chatbots such as OpenAI’s GPT series and Anthropic’s Claude are trained on extensive corpora derived from publicly available web data (e.g. [51, 52]), enabling fluent and coherent language generation. However, pretraining and fine-tuning alone are insufficient for many real-world applications. AI-based systems are expected to provide accurate and up-to-date information, including knowledge of events occurring after the training cutoff (i.e., the point after which the training data is no longer updated). Furthermore, AI chatbots are prone to hallucinations [27], motivating the integration of mechanisms for external knowledge grounding and verification. Web scraping for inference-time access. To address these limitations, AI chatbots are often augmented with mechanisms for accessing externally stored web-scraped data during inference. Broadly, this takes two forms. First, AI chatbots may rely on pre-indexed corpora constructed from web-scraped data. These corpora can be built through proprietary scraping pipelines (e.g., OpenAI builds their own [46]) or obtained from third-party providers (e.g., search engine providers such as Google, Bing, and Brave) [74]. At inference time, relevant web content is retrieved and incorporated into the model’s generation process via RAG [34]. This design improves factual accuracy but depends on periodically updated indexes, which may still lag behind real-time events. Second, some systems support on-demand inference-time web retrieval, in which a scraper retrieves the web content live as part of the response generation process. Compared to pre-indexed retrieval, this approach provides access to more up-to-date information. Numerous AI chatbot providers have integrated query-time web functionality into their platforms [17], which we will refer to as “real-time content retrieval.” Although there is no canonical implementation of this feature, public information provided by various AI companies indicates a general flow of how this works [44, 45, 68]. Putting it all together: how web-scraped data flows into AI chatbots. Figure 1 pulls together information from these sources to demonstrate how information published on websites can ultimately be part of an AI chatbot’s response to a user query via real-time content retrieval. We assume, as a baseline, that an AI model has already been trained on internet content and is deployed as a chatbot. At some point after the model is trained, a content owner publishes information online that may be relevant to chatbot responses (Step 1). For example, Alice might post their name (e.g., Alice), location (e.g., New York), and personal information (e.g., ice cream) on their personal website. Later a user submits a query to the chatbot that inquires about the website (e.g., “What do you know about Alice?”) (Step 2). To answer this query, the chatbot relies on one of two content retrieval mechanisms discussed above: pulling from cached indexed content or retrieving content live (Step 3). This retrieved content is then used as part of the context for response generation. Finally, the
Identifying AI Web Scrapers Using Canary Tokens
Conference’17, July 2017, Washington, DC, USA
Figure 1: High-level overview of how web data is sourced during real-time content retrieval by AI chatbots. chatbot generates a response to the user query (Step 4), which can contain content ingested during pre-training and fine-tuning, as well as the content retrieved by live retrieval.
2.2
Identifying AI-related scraping
Since the rise of powerful generative AI models, concerns have grown regarding how these models scrape and use data from the web. These concerns span multiple dimensions, including infrastructure impacts such as increased bot traffic and service disruption [77], as well as data privacy and copyright issues [36]. In response, a growing body of work has studied AI-related scrapers. Most of this work seeks to measure AI scraper behaviors. Official web standards provide a way for scrapers to acknowledge their activities to HTTP servers via the User-Agent header [22]. This header is a string that bots and regular browsers alike can use to identify how they are visiting the site, such as the version number and rendering technologies used. Using this string, website owners can recognize and even block scraper bots as wanted. Existing efforts to identify AI-related scrapers focus primarily on collecting the User-Agent strings. They typically acquire this information from from three main sources: (1) self-reported information from AI chatbot providers (e.g., OpenAI declares to use three User-Agents for their bots: ChatGPT-User, OAI-SearchBot, and GPTBot); (2) community-maintained databases (e.g., Dark Visitors); and (3) oneoff testing by researchers and practitioners [11, 36, 39, 66, 67, 78]. While promising, these approaches are limited in accuracy, completeness, and scalability. Self-disclosed information by scrapers may be incomplete, outdated, or incorrect. Indeed, companies have been observed using scraper identities that they did not publicly document [30]. Even worse, many companies do not disclose any information about their scrapers at all. Second, community-maintained databases may also contain errors, including identifiers incorrectly attributed to a provider, and assume that no other bad actors are
spoofing other scrapers’ User-Agent information [56]. The UserAgent protocol is merely a requested standard and not an enforceable mandate, and not all parties conform. Many studies have shown that malicious bots spoof their User-Agents (e.g. [35]), including prominent AI models [23]. Thus, definitively linking User-Agents to AI chatbots, particularly during live content retrieval, and understanding the mechanism by which retrieval happens, remains unexplored. This motivates our first research question: RQ1: Which User-Agents do AI chatbots use to source web content when generating responses? Note that Borysenko [12] characterized the HTTP fingerprints of AI scrapers through the lens of a live developer documentation endpoint. However, this work assumes that scrapers truthfully identify themselves via User-Agent strings, which may not hold in practice (again, see [23]). Therefore, there is a need—which our work addresses—to develop a methodology for inferring scraper identities without assuming self-reported User-Agent strings. Beyond identifying scraper User-Agents, though, the mechanism by which data is used in real-time content retrieval remains opaque. As discussed in Section 2.1, different AI chatbot providers have indicated that they may rely on indexed web sources or directly retrieve content live. However, no work has studied which AI chatbot providers employ these different mechanisms or whether they switch between them. This motivates our second research question: RQ2: How do AI chatbots source web content when generating responses? Do they retrieve content live or rely on caches?
2.3
Preventing AI-related scraping
Sometimes, having one’s data appear in an AI model’s web search is desirable, e.g. for marketing purposes or to enhance public profile. Other times, though, site owners may wish to opt out of the AI
Conference’17, July 2017, Washington, DC, USA
ecosystem, for privacy, legal, or other concerns. Numerous proposals exist for preventing unwanted web scraping, particularly in an AI context. We outline these proposals here and highlight gaps related to real-time content retrieval. Preventing generic web scraping. One prominent effort at controlling unwanted scraping behavior is the Robots Exclusion Protocol (REP), a de facto web standard allowing site owners to formally request that certain web bots avoid their sites [32]. This protocol allows site owners to host a file at the site root named robots.txt that enumerates what User-Agents are and are not allowed to scrape their site. However, this is merely a requested protocol rather than a technical restriction, and has been shown to be ignored by scrapers in the past. For instance, [16, 36] shows that some scrapers do not respect robots.txt directives. Kim et al. [30] demonstrate that some scrapers only selectively respect some directives in robots.txt. Because of this, some have turned to scraper-blocking tools that may return an error code (e.g., 403 Forbidden) or serve different content to unwanted scrapers (e.g., a CAPTCHA page or a decoy page) [9, 25]. Regardless of the specific mechanism, this line of defense relies on the ability to first identify the scrapers to be restricted. For example, robots.txt directives use the User-Agent header, a self-reported string, to specify access permissions associated with each scraper. Similarly, scraper-blocking tools typically rely on a combination of signals, such as the User-Agent header, IP address, access pattern, and TLS fingerprint, to identify scrapers and determine the appropriate response [14, 15]. This further motivates our RQ1 studying which User-Agents AI chatbots use. Preventing scraping by AI chatbots. Most work on preventing AI web scraping focuses on preventing data from being used for model pre-training. Tools like Fakwes [61], Glaze [58], and Nightshade [59] corrupt online-hosted images to attempt to make them unusable in AI model training. Although these mechanisms do not prevent scraping, they seek to render the data useless for AI developers. Another line of work focuses on mechanisms for signaling and enforcing content owner’s preferences regarding AI scraping. They highlight the growing interest in controlling and stopping unwanted AI scraping. For example, Lee et al. [33] surveyed different ways that content owners express their preferences for AI scraping, ranging from fine-grained, sample-level approaches (e.g., watermarks or metadata) to website-level mechanisms such as robots.txt and Terms of Service. Complementary measurement studies [19, 20, 36, 38, 64] document a rapid increase in the adoption of AI-specific directives in robots.txt files as well as the prevalence of other attempts to block AI scrapers. Another line of work explores content-level defenses that attempt to interfere with or degrade the effectiveness of AI scraping. For instance, Zychlinski [81] proposes serving distinct content to AI scrapers with the goal of creating a version of the web that only scraper bots will see. Brach [13] and Zhong [79] study HTML manipulation techniques that limit an AI model’s ability to extract information from a scraped webpage. Our work is inspired by these content-based approaches but differs in its objective. Rather than attempting to block or degrade scrapers, we leverage content manipulation as a measurement tool to infer the identities and behaviors of AI scrapers, even when they attempt to conceal their identity.
Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, and Emily Wenger
A final, draconian solution that addresses limitations in proposed anti-AI-scraping methods is merely taking content off the internet. However, if AI chatbot scrapers have already cached web data, there is a chance they will continue to use this data after it has been taken offline. It is not well understood if conducting real-time content retrieval includes cached content. All these unknowns lead us to our third and final research question: RQ3: Can simple site blocking techniques like taking sites offline or using robots.txt affect AI chatbots’ retrieved content?
3
IDENTIFYING AI-RELATED SCRAPERS WITH CANARY TOKENS
To probe the research questions outlined above, we propose a novel approach for inferring scrapers associated with AI chatbots using what we refer to as canary tokens — tokens embedded in the content of our websites that uniquely identify a specific scraper. Our key observation is that content served to a scraper can later become observable in an AI chatbot’s response. Thus, we can utilize the response of an AI chatbot as an attribution side channel. By serving different versions of content that are unique and distinguishable to different scrapers, the output of the AI chatbot can reveal the versions of the content that are ingested by the AI chatbot, and thus the scrapers that retrieved them. Overview. To exploit this insight, we propose the following approach, shown in Figure 2. In Step 1 (canary token generation), we create websites that serve unique canary tokens to visiting scrapers. Then, in Step 2 (AI chatbot querying), we probe a specific AI chatbot with queries that are designed to elicit responses containing these tokens. Finally, in Step 3 (scraper inference), we connect data from our websites to the chatbot query responses to determine scrapers are feeding data into the chatbot. For example, if Scraper A is served content containing the location “New York” and Scraper B is served content containing the location “Los Angeles,” then an chatbot response that contains the location “New York” would indicate that the Scraper A is feeding data into the chatbot, while a response that contains the location “Los Angeles” would indicate that Scraper B is feeding data into the chatbot. In this section, we describe how we implement this approach in practice.
3.1
Deploy Websites with Canary Tokens
The first part of our approach generates unique tokens that are embedded in the page content of websites we control and served to each unique scraper that accesses these websites. This step has two subcomponents: (1) template creation, where we write a set of website templates with placeholders for canary tokens; and (2) token generation and serving, where we fill these placeholders with unique tokens and serve them to visiting scrapers. Template Creation. We construct 20 website templates, each with 10 canary tokens. Below is an example website snippet: <p> My name is Alice. I am from {{ CT1 }} and I like {{ CT2 }}. </p> {{ CT1 }} and {{ CT2 }} are placeholders for canary tokens, which we fill later with values randomly selected from a large pool. Each unique visiting scraper receives a set of canary token
Can you tell me about Alice?
Identifying AI Web Scrapers Using Canary Tokens
Conference’17, July 2017, Washington, DC, USA
User query My name is Alice. I am from New York and I like ice cream.
Website template My name is Alice. I am from [CT1] and I like [CT2].
Scraper 1 User Agent: xx/... ASN: 01234
AI chatbot 1 Search
My name is Alice. I am from Los Angeles and I like pie.
Scraper n User Agent: yy/... ASN: 56789
1. Deploy Websites with Canary Tokens (CTs)
Alice is from New York and likes ice cream.
CTs
…
… Search
AI chatbot 1 response
AI chatbot n
CTs
Infer AI chatbot 1 uses scraper 1
…
AI chatbot n response Alice is from Los Angeles and likes pie.
Infer AI chatbot n uses scraper n
2. AI Chatbot Querying
3. Scraper Inference
Figure 2: Our proposed pipeline for identifying web scrapers used by AI chatbots. (1) We deploy websites that serve unique canary tokens to each distinct visitor, (2) query AI chatbots for information about these sites to elicit canary tokens, and (3) infer which scrapers feed which AI chatbots based on their outputs. values that are different from the values served to any other scraper. Figure 2 shows example values for CT1 and CT2. In this example, the hometown placeholder CT1 may be filled with “New York” for one scraper, and “Los Angeles” for another scraper. CT2 may be filled with “ice cream” for one scraper, and “pie” for another scraper. We look for these unique values in an AI chatbot’s responses to determine which scrapers are feeding data into the chatbot. Token Generation and Serving. Whenever a scraper accesses our websites, we first determine whether it has previously been assigned a token set using our criteria for scraper uniqueness (see below). If not, we generate a fresh set of tokens, fill the template placeholders with those values, and serve the resulting customized content. If the scraper has been seen before, the system serves the same tokens as before. As a result, each distinct scraper receives a stable, scraper-specific view of the website. We also maintain a mapping between scrapers and the tokens served to them, so we can later map tokens found in chatbot responses back to scrapers. To determine scraper "uniqueness", one can use arbitrary fingerprinting information, such as the scraper’s User-Agent information, IP information, Autonomous System Number (ASN) information, TLS fingerprinting information, etc. For simplicity, we define an unique scraper as a scraper that provides a unique combination of User-Agent information and ASN information, which provides a balance between complexity and granularity in identifying distinct scrapers. Per this definition, we consider two scrapers the same if they have the same User-Agent information and ASN information. Note that our approach can be easily extended to use arbitrarily complex fingerprinting techniques, such as TLS fingerprinting or browser fingerprinting.
3.2
AI Chatbot Querying
After the canary token websites have been deployed for a sufficient amount of time, we then attempt to elicit information about them from AI chatbots (step 2 in Figure 2) using a series of prompts.
Prompt design. Since AI chatbots are black-boxes, prompt design is inherently empirical. Thus, we experiment with different prompts and select the ones that are the most effective at eliciting responses that query the web. Through a pilot study of a few websites that are not part of the main study, we adopt a strategy that uses two queries for each AI chatbot interaction. At a high-level, these two queries encourage complementary retrieval behaviors: the first query is designed to elicit responses from the AI chatbot that are based on a live retrieval, while the second query is designed to elicit responses from the AI chatbot that are based on other data sources. Each query then consists of three components: (1) a description of the website we want to query the AI chatbot about that does not contain any canary tokens; (2) instructions for the AI chatbot of what data sources to search for information; and (3) instructions for formatting the AI chatbot’s response. AI chatbot querying overview. Figure 2 shows a simplified example snippet of a query. This query starts with a description of the website we want to query the AI chatbot about (e.g., “[company / person’s name], [brief description of company/person]”). Next, it provides instructions to search the internet for information (e.g., “Search the internet for any relevant information.”). Then, it asks a series of questions pertaining to the canary tokens on the website to elicit the AI chatbot relaying this information. Lastly, it provides instructions for the AI chatbot to format its response in a way that simplifies downstream canary tokens extraction (e.g., “When writing phone numbers, write them in the format XXX-XXXXXXX”). Examples of complete queries are shown in Appendix C. We then provide a follow-up query to see if an AI chatbot has access to information from other scrapers. To do so, we explicitly ask an AI chatbot to search for additional information about the same website: “Were you able to find any variant websites about [company/person’s name]?,” with the same formatting requests as in the first prompt.
Conference’17, July 2017, Washington, DC, USA
3.3
Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, and Emily Wenger
Scraper Inference
After querying AI chatbots, we perform token extraction, where we extract canary tokens from chatbot responses, followed by scraper inference, where we infer which scrapers may have fed data into the chatbot based on the extracted tokens. Token Extraction. We use simple string matching techniques to extract canary tokens (if present) from chatbot responses. When a response contains a token uniquely assigned to a scraper, that token serves as evidence linking the response to the scraper. To ensure the validity of tokens extracted, we hand-wrote matching rules using a combination of regex and text-matching rules based on our knowledge of canary tokens. Matching AI chatbots to scrapers. After extracting canary tokens from a response, we map each token back to the scraper identity that was served it. For each token in a response, we query our database for the scraper identity (User-Agent, ASN pair) associated with that token assignment. Because a single chatbot response may contain multiple canary tokens that are served to different scrapers (e.g. because the chatbot relies on multiple sources), one response can provide evidence for multiple scraper identities. We formally define our matching algorithm as follows. Let S denote the set of all scraper identities, where each identity 𝑠 ∈ S is defined by the tuple 𝑠 = (UA, ASN). For a given chatbot ℓ and scraper identity 𝑠, define: • 𝑇ℓ (𝑠) ∈ Z ≥0 as the total number of tokens found in responses from AI chatbot ℓ that are associated with scraper identity 𝑠; and • 𝑊ℓ (𝑠) ∈ Z ≥0 as the number of distinct website query interactions (e.g. one interaction = a set of queries to one AI chatbot about one website) in which scraper identity 𝑠 is observed for chatbot ℓ. We define the match score 𝑀ℓ ∈ {yes, no} for scraper identity 𝑠 under chatbot ℓ as: (
yes 𝑀ℓ (𝑠) = no
4 if 𝑇ℓ (𝑠) ≥ 𝑡 ∨ 𝐹 ℓ (𝑠) ≥ 𝑤 if 𝑇ℓ (𝑠) < 𝑡 ∧ 𝐹 ℓ (𝑠) < 𝑤
(1)
Parameters 𝑡 and 𝑤 are tunable based on the desired error rates. We set 𝑡 = 2 and 𝑤 = 1 in our implementation. This ignores any scraper identity that is associated with only one token from one website. The below section discusses the motivation for this choice.
3.4
Mitigating false positives. While hard to quantify, prior work [53] suggests that having sufficiently large token value space can help mitigate such risks. It shows that the probability of an incidental match is inversely proportional to the size of the token space from which canary values are drawn. Intuitively, if a token is selected from a space of size |𝑉 |, the chance of a coincidental match is on the order of 1/|𝑉 |. In our setting, each canary token is drawn from a space of at least 103 possible values, yielding a per-token coincidence probability on the order of 10−3 and a two-token collision probability on the order of 10−6 . When 10 tokens are associated with the same scraper identity, the probability of all of them being coincidental matches decreases exponentially with the number of tokens, since tokens are selected independently, and becomes vanishingly small. Given such low probabilities, we disregard scraper identities that are associated with only one token from one website (e.g. 𝑡 = 1, 𝑤 = 1). Sources of false negatives. False negatives occur when a model has accessed content from our website but does not emit the corresponding canary token in its response. This issue can occur for a variety of reasons. For example, the content fed to the model may be incomplete (e.g., due to truncation). Similarly, the model may fail to generate a response containing the token value even when it has access to it, due to reasons such as randomness or design choices in how chatbots present web data. In these cases, scraper activity is present but remains undetected. While these are missed opportunities to infer scraper involvement, they do not lead to incorrect inferences about scraper involvement. Mitigating false negatives. We mitigate this issue by conducting multiple rounds of queries and using multiple websites. By aggregating evidence across multiple interactions, we can increase the likelihood of observing several tokens associated with a scraper identity, even if some interactions fail to elicit the correct token, or a particular website is not scraped.
Inference Robustness
Here, we address the possibility of false positive or false negatives in our chatbot-scraper matching process and how we address them. Sources of false positives. False positives may arise when an AI chatbot outputs the value of a canary token without actually scraping our website. This situation can occur for different reasons. For instance, the AI chatbot could randomly hallucinate words that happen to match our canary tokens. Moreover, there could be a similarly named-entity online with characteristics that happen to match our canary tokens. In either case, the AI chatbot could produce a response that contains a token value that is associated with a scraper identity in our canary token database, even though the AI chatbot did not actually retrieve content from our website.
EXPERIMENT SETUP
In this section, we describe the implementation of our pipeline, including website template design and deployment, canary token generation, AI chatbot selection, and the measurement conditions used in our experiments.
4.1
Website Deployment
Domains and site infrastructure. We purchased 20 .com domains with no recent ICANN history and and hosted one website on each domain using Google Cloud. Each website is instantiated from one of our templates, which are designed to mimic common types of websites (e.g., a company’s website or an artist’s portfolio). Each template contains 10 placeholders for canary tokens that are generated uniquely for each unique scraper. We distribute these placeholders throughout the website to account for possible content stripping and truncating. The content of each template is hand written to avoid introducing confounding factors that may arise from using machine-generated content. In addition, we make sure the content of all websites are benign, fictitious, and do not overlap with real entities. For example, we ensured all fake people’s websites shared no similarities to people in real life, only generated fictitious
Identifying AI Web Scrapers Using Canary Tokens
phone numbers, named fictitious alma matters, etc. An example website template can be found at the link provided in Appendix B. Indexing. To improve the discoverability of our websites by AI chatbots, we employ two strategies. First, we ensure that our websites are indexed by three major search engines: Google, Bing, and Brave Search. For Google and Bing, we submit our websites through their respective webmaster tools. For Brave Search, we follow their guidelines for website indexing, which includes submitting our websites through their public form and then visited the websites multiple times. Second, we embedded hidden href links to all 20 websites on other websites we control to further increase visibility. Canary Token Selection and Generation To determine which visitors access what version of our data, we generate canary tokens unique to each visitor within otherwise static bodies of text. We decided that other means of data detection, such as adding embeddings into text or watermarking our text, would needlessly complicate the means of detecting the presence of our data within an AI chatbot’s response. We generate canary tokens using the Python Faker library and random number generator for words and a random number generator for numbers. We ensure distinct types of canary tokens are present on each site to minimize the odds of serving the same canary token to different visiting scrapers. Serving matching tokens to different scrapers would result in an inconclusive match. The smallest of all our canary tokens’ sample spaces has 4761 unique possibilities, and we see on average 592 unique (User-Agent, ASN) pairs per site (see Table 2), meaning that the token collision probability is small. Token analysis. We analyze the sets of served canary tokens after data collection for quality control. We observed a few instances where, due an issue with random values available in the libraries we used, the same token was served twice to different visitors (e.g., the name "John" was served as a first name on a biographical site for two different visitors). We also observed a few instances where the same token value was assigned to different variables across visitors (e.g., for one visitor, "John" was the person’s name, while for another, it was their parent’s name). For both cases, we discard any AI chatbot matches relying on such tokens, as the associated User-Agent cannot confidently be distinguished. We identify two additional sources of ambiguity, which we refer to as confusions. The first arises when one token is a subset of another (e.g., "Port" is a subset of "West Port"). If the longer token is present in a chatbot response, the subset may be incorrectly detected as a match for the shorter token. The second confusion arises from the use of numerical values or dates as canary tokens, since AI chatbots may produce numbers unrelated to our websites that coincidentally match tokens in our database (e.g., "My knowledge cutoff date is 2024," when 2024 is one of our tokens). For numbers, we also find that sometimes chatbots approximate numerical tokens, such as rounding the number of customers on business-related website to an even number. While both issues could be addressed through manual inspection, we conservatively discard all subsetbased matches and numerical token match to ensure our results are reliable. Statistics on discarded tokens are in Appendix Table 7.
Conference’17, July 2017, Washington, DC, USA
4.2
AI Chatbot Querying
After making our websites publicly available, we wait for two months to allow the scrapers to retrieve the content from our websites and feed the content into AI chatbots, as anecdotal evidence suggests websites of our size could take up to six weeks to propagate online [55, 73]. We then query AI chatbots with prompts designed to elicit responses that may reproduce the canary tokens. AI chatbot
Publisher
AI chatbot
Publisher
ChatGPT Claude Copilot Deepseek Duck.ai ERNIE Gemini GLM Granite Grok Hunyuan
OpenAI Anthropic Microsoft Deepseek DuckDuckGo Baidu Google Z.AI IBM xAI Tencent
Kimi Liquid Llama Mistral Nova Perplexity Qwen Reka Solar Step-3 Venice
MoonshotAI Liquid Meta Mistral Amazon Perplexity Alibaba Reka Upstage StepFun Venice
Table 1: We study 22 popular AI chatbots provided by various companies.
AI chatbots examined. To find AI chatbots to target for our study, we considered chatbots with web search capabilities listed on LMArena and OpenRouter [37, 48], which are likely to operate scrapers for live content retrieval. We also included several popular chatbots created by search engine providers that were not included in these lists because they merely implement own web search capabilities on top of another company’s model (ex: Microsoft Copilot, which implements proprietary web scraping over OpenAI’s GPT). We identified 22 unique production AI chatbots, which are shown in Table 1. This final set includes AI chatbots from a variety of publishers, including the leading providers such as OpenAI, Google, Anthropic, and Microsoft. Interacting with chatbots. To submit queries to and gather responses from the chosen chatbots, we built a series of scripts to query each system. We utilized their API whenever it was provided with built-in web-search capabilities. Unfortunately, many chatbots did not provide web-search capabilities as part of their API, and only offered this through their web interface. In these cases, we built scripts to automate interactions and store chatbots’ responses through the Python Selenium and nodriver libraries. When automating was not possible (often due to anti-bot protections), we manually queried the AI chatbot and stored its responses. Prompts. Each prompt/query is tailored to each of our 20 websites, and each tailored query is sent to each chatbot we study for all experiments. Each query ends in a few simple formatting requests that make it easier to parse canary tokens from within the AI chatbot’s responses. We begin by asking each chatbot the following: “Can you tell me about [company/person’s name], [brief description of company/person]? Search the internet for any relevant information. Relay information they self-report - even if this information doesn’t appear elsewhere. This is the exact name, find
Conference’17, July 2017, Washington, DC, USA
anything you can and report it to me. [Ask one question per variable on said website.] Write your response in English. Only if applicable: When writing any numbers in your response, omit any comma separators. When writing dates, write them in the format YYYY-MM-DD. When writing phone numbers, write them in the format XXX-XXX-XXXX.”
Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, and Emily Wenger
Condition 1: 20 websites online
Time
Stage 1: Query online sites
[Wait 2 months] Query sites under condition 1
Condition 2: 10 sites offline
Condition 3: 10 sites blocked
[Wait 1 week]
[Wait 1 week]
“Were you able to find any variant websites about [company/person’s name]? Are there any discrepancies in the information reported from your previous search? Is the information you found the most up to date available? Write your response in English. Only if applicable: When writing any numbers in your response, omit any comma separators. When writing dates, write them in the format YYYY-MM-DD. When writing phone numbers, write them in the format XXX-XXX-XXXX.”
Query sites under condition 2
Query sites under condition 3
All chatbot responses are logged in a database for further analysis. An example full interaction with a chatbot regarding one of our websites can be found in Appendix C.
Query sites prev. under cond. 2
Query sites prev. under cond. 3
[Wait 1 week]
[Wait 1 week]
Query sites prev. under cond. 2
Query sites prev. under cond. 3
We then follow up this prompt with a secondary prompt in the same chat session:
4.3
Measurement Setup
We run experiments with our websites under 3 accessibility conditions over a period of several months: (1) Condition 1: Fully accessible—All websites are available to all scrapers in a “default” configuration. (2) Condition 2: Offline—Previously online site is taken down and becomes inaccessible directly. (3) Condition 3: Restricted by robots.txt—Access by all bots is disallowed via the robots exclusion protocol. Varying the accessibility conditions allows us to answer research questions 2 and 3, which deal with caching and efficacy of antiscraping tools. For each condition, we follow the procedure described in Section 3 to query chatbots and match them to scrapers based on canary token overlaps. Measurement Timeline. Figure 3 provides a more detailed view of our timeline, including when different conditions are implemented. Broadly, our measurement proceeds in 3 stages: Stage 1: All sites remain online (Condition 1) for 2 months. At the end of this stage, we query all AI chatbots once to perform initial mapping of scrapers to chatbots. Stage 2: We divide our websites into two groups. We take one group of 10 websites offline (Condition 2), and we add a robots.txt file to the other 10 websites that blocks all bot traffic (Condition 3). At one week and two week intervals after this change, we query chatbots about these sites. Stage 3: We bring all sites back online and remove robots.txt files (restoring all sites to Condition 1). At one week and two week intervals after this change, we query chatbots about all sites. This measurement setup allows us to evaluate how AI chatbots respond to these various accessibility conditions and assess the time frame over which these changes are detected by chatbots.
[Wait 1 week]
[Wait 1 week]
Query sites under condition 2
Query sites under condition 3
Condition 1: 20 websites online [Wait 1 week]
Stage 2: Query offline or blocked sites
Stage 3: Query offline or blocked sites that are now online
Figure 3: Measurement timeline for our study. Our measurements are divided into 3 distinct stages with varying website accessibility conditions —see Section 4 for details.
Min across sites Max from sites Avg from sites All across sites
User-Agents
ASNs
Unique visitors
313 477 405.95 2765
154 226 192.4 549
313 674 592.2 4042
Table 2: Statistics on visitors measured across our websites. We observe hundreds of unique visitors (User-Agent, ASN pairs) per website, indicating sufficient site visibility.
Website statistics. To better understand the overall traffic to our websites, we computed the basic statistics shown in Table 2. Here, we report the amount of individual User-Agents, ASNs, and Unique Visitors (combination of User-Agent and ASN from a visitor) found across our sites. This data represents all the visitors we measured over the entire course of our websites being online. The least popular website exhibited 313 unique visitors, while the most popular site had 674. All our websites had the same SEO operations applied to them, and the styling and length of content was relatively similar across all sites. The only major difference between sites was the content itself — some of which may be of higher interest to scrapers and bots (the majority of our visitors) than others.
Identifying AI Web Scrapers Using Canary Tokens
Conference’17, July 2017, Washington, DC, USA
AI System
User-Agent Measured
User-Agent Category
ChatGPT
OAI-SearchBot
Claude
Bravebot
Copilot
Website Status
Publicly Known
Online
Offline
Blocked
First-Party Declared Agent
✓
✓
✓
✓
Third-Party Search Agent
✓
✓
✓
✓
Bingbot
First-Party Declared Agent
✓
✓
✓
✓
Duck.ai
DuckAssistBot
First-Party Declared Agent
✓
-
-
✓
ERNIE
Baiduspider Chrome
First-Party Declared Agent Generic Browser Agent
✓ ✓
✓ ✓
✓ ✓
✓ ✗
Gemini
Googlebot
First-Party Declared Agent
✓
✓
✓
✓
Granite
GranitePlayground
First-Party Declared Agent
✓
-
✓
✓
Grok
Googlebot Chrome Safari
Third-Party Search Agent Generic Browser Agent Generic Browser Agent
✓ ✓ ✓
✓ ✓ -
✓ ✓ ✓
✗ ✗ ✗
Kimi
See Table 4
See Table 4
✓
✓
✓
✗
Llama
meta-externalagent meta-webindexer
First-Party Declared Agent First-Party Declared Agent
✓ ✓
-
✓ -
✓ ✓
Mistral
Bravebot
Third-Party Search Agent
✓
✓
✓
✓
Nova
Amazonbot
First-Party Declared Agent
✓
-
✓
✓
Perplexity
PerplexityBot Googlebot
First-Party Declared Agent Third-Party Search Agent
✓ ✓
✓ ✓
✓ ✓
✓ ✗
Qwen
Googlebot Chrome
Third-Party Search Agent Generic Browser Agent
✓ ✓
✓ -
✓ ✓
✗ ✗
Reka
Googlebot Chrome
Third-Party Search Agent Generic Browser Agent
✓ ✓
✓ -
✓
✗ ✗
Solar
Chrome Safari Firefox Edge
Generic Browser Agent Generic Browser Agent Generic Browser Agent Generic Browser Agent
✓ ✓ ✓ -
✓ -
✓ ✓
✗ ✗ ✗ ✗
Step-3
Googlebot
Third-Party Search Agent
✓
-
-
✗
Venice
Bravebot
Third-Party Search Agent
✓
✓
✓
✓
Table 3: AI chatbots studied and their observed User-Agents. For each AI system, we report their observed User-Agents based on canary token matching, the category of each User-Agent, the status of our websites when each User-Agent was observed, and whether the observed User-Agent was publicly known to be used to feed data to the AI system. We exclude Deepseek, Hunyuan, GLM and Liquid from this table because we did not measure any canary tokens in their responses.
5
RESULTS
The data collected from our websites and AI chatbots throughout the measurement period provides interesting insight into our research questions outlined in Section 2. Here, we present our findings for each question. Since our findings often involve long User-Agent strings, we aggregate User-Agents by “family” throughout this section (e.g. either by a specific bot name or by browser version). Full User-Agent strings are reported in Table 10 in the Appendix.
5.1
Mapping AI Chatbots to Scrapers
We start by answering the first research question: which scrapers appear to supply content that AI systems later reproduce in responses? Table 3 summarizes the User-Agents we observed across AI systems. For each AI system, we report the User-Agent strings we observed, the status of our websites when each User-Agent was observed, and whether that User-Agent is publicly documented as being used to collect data for the corresponding AI system. In addition, we group the User-Agents we observed into three categories based on their characteristics: • First-Party Declared Agents: User-Agents that align with what the AI chatbot self-declares to scrape with.
Conference’17, July 2017, Washington, DC, USA
User-agent Measured
User-Agent Category
Chrome
Generic Browser Agent
Edge
Generic Browser Agent
Googlebot
Third-Party Search Agent
Obsidian
Generic Browser Agent
Qaxbrowser
Generic Browser Agent
QQBrowser
Generic Browser Agent
QuarkPC
Generic Browser Agent
SLBrowser
Generic Browser Agent
WindowsWechat
Generic Browser Agent
Table 4: User-Agents measured in responses from MoonshotAI’s Kimi. See Table 9 for additional details about Kimi’s measured User-Agents.
• Generic Browser Agents: User-Agents associated with other applications, such as general web browsers. • Third-Party Search Agents: User-Agents that are associated with search engines not under the control of the AI chatbot’s owner. Overall, we elicited User-Agent information for 18 of the 22 AI systems we studied. For Deepseek, Hunyuan, GLM and Liquid we were unable to make the systems return information about our websites, so we exclude them from this analysis. Takeaway #1: our approach identified many scrapers known to be associated with AI systems. We identified 10 User-Agent strings that fall into the first-party declared category. These strings match agents that AI system providers publicly document as being used for crawling or search-related data collection. For example, OpenAI documents OAI-SearchBot as one of its bots [46]. Such matches provide confidence in our approach, as they indicate that we are able to elicit the correct User-Agent strings. Takeaway #2: AI systems can use scrapers with generic browser User-Agents. In addition to first-party declared agents, several AI systems returned content associated with generic browser UserAgent strings. We observed this behavior for six of the 18 AI systems for which we obtained User-Agent information. This result suggests that some AI systems can obtain website content through requests that appear similar to ordinary browser traffic, which makes UserAgent-based blocking difficult. ERNIE, Grok, Solar, Quen, and Kimi exhibited this behavior. For ERNIE, we observed Baiduspider, a User-Agent associated with Baidu’s search engine, and a generic Chrome User-Agent. For Grok, we observed two generic browser User-Agents in addition to Googlebot. Solar only used generic browser User-Agents. Qwen used both Googlebot and a generic Chrome User-Agent. Grok used two generic browser User-Agents in addition to Googlebot. Kimi appears to be the most extreme case of this behavior: many useragents appeared to be correlated with data outputted by Kimi. We infer that Kimi rotates through a large list of User-Agent strings while scraping, possibly to avoid bot detection.
Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, and Emily Wenger
Takeaway #3: AI systems rely heavily on third-party search engine scrapers, and such relationships may not always be publicly known. The first two takeaways may seem reasonable, but our finding that AI chatbots return data associated with a thirdparty search engine agent was surprising. We observed content associated with Googlebot, bingbot, and Bravebot in responses from 10 of the 18 systems we analyzed. In many cases, the AI system provider is not associated with the search-engine operator. Some such relationships are publicly documented, such as Claude’s use of Brave [74]. Others, such as Qwen or Perplexity returning content associated with Googlebot, were not publicly documented. When this occurs, we assume either (a) that the scraper is ingesting the search results associated with our website, rather than scraping our website directly or (b) the scraper is spoofing their user-agent when providing it to our websites. However, in checking the ASN associated with the data returned to third-party search engine agent, we were able to verify that these visitors originated from their expected company’s network. Thus, we determine that option (a) is the scenario we have measured. Regardless, this finding reveals an additional layer of opacity in the web-to-AI chatbot content supply chain. A website owner who wants to prevent AI systems from using their content may not be able to do so merely by blocking known AI crawler UserAgent strings. If an AI system obtains content through a search engine, the site owner may need to opt out of search indexing entirely, which may be unacceptable for visibility, discoverability, or business reasons. If a scraper spoofs User-Agent strings, then User-Agent-based controls are even less reliable. Answer to RQ1: AI chatbots greatly vary in behavior. While some operate under their self-declared User-Agent, others operate under generic User-Agents. Many of these systems, whether or not they self-declare scraping behaviors, seem to scrape search engine results, rather than visiting our websites directly.
5.2
Caching of scraped content
Next, we investigate whether AI chatbots cache or otherwise retain content they scraped. We use our second experimental condition for this analysis: after the initial crawling period, we took the websites offline and waited one week before querying the chatbots again. Table 5 summarizes the User-Agents observed in this offline condition, with the baseline condition shown for comparison. Takeaway #1: search engine scraped websites are effectively cached. For AI chatbots that returned content associated with third-party search-engine crawlers, we observed that the content was still returned even after our websites had been offline for one week. This behavior occurred for seven of the eight chatbots that ingested data from Googlebot, Bingbot, or Bravebot. This result is consistent with the expected behavior of search-backed retrieval pipelines: once content has been indexed or otherwise stored by a search provider, the chatbot may still be able to access it even when the original website is no longer reachable. We also observed a notable case for ERNIE. In the offline condition, ERNIE returned data associated with Baiduspider, a searchengine crawler. In the baseline condition, however, we observed
✓ ✓
Duck.ai ERNIE
DuckAssistBot Chrome Baiduspider
✓ ✓
✓
✓
Gemini
Googlebot
✓
✓
✓
Granite
GranitePlayground
✓
✓
✓
Grok
Googlebot
✓
✓ ✓
✓ ✓ ✓
✓
Llama
Chrome Safari meta-externalagent
✓
✓ ✓ ✓
Mistral Nova Perplexity
✓ ✓
✓
✓
Gemini
Googlebot
✓
Granite
GranitePlayground
✓
Grok
Googlebot
✓
✓
✓
✓
Llama
Chrome Safari meta-externalagent
Mistral Nova Perplexity
meta-webindexer Bravebot Amazonbot PerplexityBot
✓ ✓ ✓ ✓
✓
✓
✓
✓
✓
✓
✓
2 Weeks Back Online
✓ ✓ ✓
✓ ✓
1 Week Back Online
OAI-SearchBot Bravebot Bingbot
DuckAssistBot Chrome Baiduspider
2 Weeks Blocked
ChatGPT Claude Copilot
Duck.ai ERNIE
1 Week Blocked
AI chatbot
✓ ✓ ✓
✓ ✓ ✓
OAI-SearchBot Bravebot Bingbot
Online
2 Weeks Back Online
✓ ✓ ✓
✓ ✓ ✓
ChatGPT Claude Copilot
✓ ✓ ✓
✓ ✓ ✓
✓ ✓ ✓
✓ ✓ ✓
✓ ✓ ✓
✓ ✓
✓ ✓
✓ ✓ ✓
✓ ✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓
✓ ✓ ✓
✓ ✓ ✓
✓ ✓ ✓
✓ ✓ ✓
meta-webindexer Bravebot Amazonbot PerplexityBot
✓ ✓ ✓ ✓
✓ ✓ ✓
✓ ✓
✓ ✓ ✓
✓ ✓ ✓
User-Agent
1 Week Back Online
✓ ✓ ✓
1 Week Down
2 Weeks Down
Conference’17, July 2017, Washington, DC, USA
Online
AI chatbot
User-Agent
Identifying AI Web Scrapers Using Canary Tokens
Googlebot
✓
✓
✓
✓
✓
Googlebot
✓
✓
Qwen
Googlebot
✓
✓
✓
✓
✓
Qwen
Googlebot
✓
✓
✓
✓
✓
Chrome Googlebot
✓
✓
✓ ✓
Reka
Chrome Googlebot
✓
✓
✓ ✓
✓
Reka
✓
✓ ✓
✓ ✓
✓
✓ ✓
✓ ✓
Solar
✓ ✓
✓ ✓
✓ ✓
✓
✓
Solar
Chrome Chrome Safari Firefox Edge
Step-3
Googlebot
Venice
Bravebot
✓ ✓ ✓ ✓
✓ ✓
✓
✓
✓
✓
Chrome Chrome Safari Firefox Edge
Step-3
Googlebot
Venice
Bravebot
✓ ✓ ✓ ✓
✓
✓ ✓ ✓ ✓
✓
✓
✓
✓
✓
Table 5: User-Agents measured under stages of Condition 2 (offline) compared to the baseline Condition 1.
Table 6: User-Agents measured under stages of Condition 3 (blocking with robots.txt) compared to the baseline Condition 1. Bold checkmarks represent User-Agents measured here that were not measured under Condition 2.
only a generic Chrome User-Agent and did not observe Baiduspider. There are two possible explanations for this behavior: (a) we failed to elicit Baiduspider in the baseline condition, or (b) ERNIE prioritizes fetching content through a browser-like agent when the website is available, but falls back to content from Baiduspider when the website is unavailable. Takeaway #2: AI chatbots do cache what they scrape. We also observed continued content availability for chatbots that scraped
using generic browser User-Agents. For example, Grok, ERNIE, and Solar continued to return content associated with a generic Chrome User-Agent after our websites were taken offline. This indicates that persistence is not limited to search-engine-backed retrieval: in some cases, content fetched through browser-like chatbot crawlers also remained available after the origin was no longer reachable.
Conference’17, July 2017, Washington, DC, USA
Answer to RQ2: Many AI chatbots continued to return our websites’ content after the sites had been offline for one week. This suggests that caching of scraped content is common, regardless of how the content was originally scraped.
5.3
Efficacy of blocking techniques
Lastly, we explore the question of whether simple blocking techniques, such as taking a website offline or adding a restrictive robots.txt policy, prevent AI chatbots from returning content associated with our websites. We focus on a post-scraping blocking scenario: the website content has already been scraped, and the website owner subsequently attempts to prevent the chatbot from returning that content in future responses. As observed above, taking the website offline did not reliably prevent chatbots from returning content associated with our websites. In this section, we dig deeper into this question and compare condition 2, where the websites were taken offline, with condition 3, where the websites remained online but were restricted using robots.txt. Table 6 summarizes the User-Agents observed in the robots.txt blocking condition, with the baseline and offline conditions shown for comparison. The key difference between the offline and blocking conditions is that, in the blocking condition, we added a robots.txt file that disallowed all agents from crawling content on our sites. Takeaway #1: simple blocking techniques do not stop AI chatbots from returning content. Across the offline and robots.txtrestricted conditions, many AI chatbots continued to return content associated with our websites. Of the 18 AI chatbots for which we obtained User-Agent information, 12 continued to return content in both blocking conditions. This result suggests that simple postscraping blocking techniques, such as taking a website offline or adding a restrictive robots.txt policy, are not sufficient to prevent AI chatbots from returning content that has already been scraped. We also observed slightly more User-Agent strings in the robots.txt restricted condition than in the offline condition. This pattern is consistent with the anecdotal observation that some AI chatbots may attempt to bypass robots.txt restrictions. However, this observation alone does not establish the fact that a particular scraper violated robots.txt, as there are many possible confounding factors that could explain the observed differences in User-Agents between the two conditions. Takeaway #2: Duck.ai respects blocking signals. Duck.ai was the only chatbot in our measurements that stopped returning content associated with our websites in both the offline and robots.txtrestricted conditions. This result suggests Duck.ai is likely evaluating various signals, such as the website’s availability and robots.txt directives, and taking them into account when determining whether to return content associated with the website.
Answer to RQ3: Simple blocking techniques, such as taking a website offline or using robots.txt to disallow crawling, are not effective in preventing AI chatbots from returning content associated with the website.
Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, and Emily Wenger
6
DISCUSSION
Limitations. Our approach has several limitations. First, our analysis is based on observations from a relatively small set of websites (20), with queries issued from a single vantage point (the campus network of a U.S. university). Consequently, our measurements may not capture the full range of scraping infrastructures used by AI chatbot providers. In practice, AI chatbots may employ different scraping infrastructures depending on the target website, content type, or geographic location. Second, our methodology relies on observing sufficient signals to confidently identify scrapers. As a result, we may miss scrapers that do not generate enough observable evidence. Finally, our particular experimental setup—including the set of prompts, the websites we used, and the observation window—can influence which scrapers are observable in our measurements. Implications. Our work has broad implications for both security research and the broader AI data ecosystem. First, we contribute a novel approach for inferring scrapers that supply data to AI chatbots, which is a critical component of the AI chatbot data ecosystem. Our approach can reliably identify a wide range of scrapers, including previously undocumented ones. This result immediately enables downstream bot detection and abuse mitigation services to better identify and block these scrapers. In addition, the simplicity of our approach means major content publishers and third-party web service providers (e.g., Cloudflare) could easily deploy it at scale, allowing for more comprehensive monitoring of scrapers in the wild. That said, we also acknowledge there are various operational and technical challenges in scaling this approach, such as generating canary tokens without affecting user experience, and designing canary tokens that are robust against adversaries attempting to identify them (e.g., by using multiple User-Agent strings). Second, our methodology provides a useful form of observability into the AI data supply chain. We show that AI chatbots can incorporate content originating from multiple scrapers, some of which appear to be operated by organizations unrelated to the AI chatbot provider. This observation not only highlights the complexity and opacity of the data ecosystem that supports modern AI system but also raises broader technical and legal questions. For example, how does copyright law (e.g., fair use) and privacy law (e.g., GDPR requires a lawful basis for scraping) apply in cases where data collection and data usage are decoupled across multiple parties? Similarly, designing mechanisms for signaling consent and propagating such signals through complex data supply chains remain a challenge. One example of ongoing work in this space is the effort within the Internet Engineering Task Force (IETF) to update the robots.txt protocol. This effort aims to enable website operators to express preferences regarding AI data collection and usage. However, a key open question is how such preferences can be preserved, verified, and enforced as data propagates through complex supply chains. Future work. Beyond our immediate findings, our methodology can be extended in several directions. First, we adopt a relatively simple notion of scraper identity based on the combination of UserAgent strings and ASNs. Future work could refine this definition by incorporating additional signals such as IP address ranges, TLS fingerprints, or behavioral characteristics. Second, our study focuses on a relatively short observation window and therefore likely
Identifying AI Web Scrapers Using Canary Tokens
captures scrapers used for inference-time retrieval. However, the same methodology could be applied over longer time horizons and at larger scales to investigate scraping activities associated with model training, expanding the ways in which researchers can fingerprint and track training data usage. Third, our techniques could be used to study other operational properties of AI chatbots, such as cache retention policies or the frequency with which models incorporate newly available information. Finally, while our study focuses on textual web data, and analogous approaches may be applicable to other modalities, such as images, audio, and video.
REFERENCES [1] 2025. The Walt Disney Company v. Midjourney, Inc. Complaint filed in U.S. District Court. [2] Liquid AI. n.d.. Liquid Playground. https://playground.liquid.ai/chat [3] Mistral AI. 2026. The all new le Chat, Your AI assistant for life and work Mistral AI. https://mistral.ai/news/all-new-le-chat. [4] Amazon. n.d.. Web Grounding. https://docs.aws.amazon.com/nova/latest/nova2userguide/web-grounding.html [5] Baidu. n.d.. ERNIE Bot. https://baike.baidu.com/en/item/ERNIE%20Bot/16840 [6] Baidu. n.d.. Tencent HY. https://baike.baidu.com/en/item/Tencent%20HY/ 1450766 [7] Bain. 2026. Consumer reliance on AI search results signals new era of marketing – Bain & Company. https://www.bain.com/about/media-center/pressreleases/20252/consumer-reliance-on-ai-search-results-signals-new-era-ofmarketing--bain--company-about-80-of-search-users-rely-on-ai-summariesat-least-40-of-the-time-on-traditional-search-engines-about-60-of-searchesnow-end-without-the-user-progressing-to-a/ [8] Bartz, Andrea, Graeber, Charles, and Johnson, Kirk Wallace. 2024. Bartz v. Anthropic PBC. Class Action Complaint filed in U.S. District Court for the Northern District of California. Case No. 3:24-cv-04546 (N.D. Cal.). [9] Ashley Belanger. 2025. AI haters build tarpits to trap and trick AI scrapers that ignore robots.txt. Ars Technica (2025). https://arstechnica.com/techpolicy/2025/01/ai-haters-build-tarpits-to-trap-and-trick-ai-scrapers-thatignore//-robots-txt/. [10] Ashley Belanger. 2025. Lawsuit: Reddit caught Perplexity “red-handed” stealing data from Google results. https://arstechnica.com/tech-policy/2025/10/redditsues-to-block-perplexity-from-scraping-google-search-results/ [11] Josh Blyskal. 2025. https://www.tryprofound.com/blog/what-is-claude-websearch-explained [12] Oleksii Borysenko. 2026. Developer Experience with AI Coding Agents: HTTP Behavioral Signatures in Documentation Portals. arXiv preprint arXiv:2604.02544 (2026). [13] William Brach, Matej Petrik, Kristián Košt’ál, and Michal Ries. 2025. Ghosts in the Markup: Techniques to Fight Large Language Model-Powered Web Scrapers. In 2025 37th Conference of Open Innovations Association (FRUCT). IEEE. [14] Cloudflare. 2024. https://www.cloudflare.com/application-services/products/botmanagement/. [15] Cloudflare. 2025. Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large; Permission-Based Approach Makes Way for A New Business Model. https://www.cloudflare.com/press-releases/2025/cloudflare-justchanged-how-ai-crawlers-scrape-the-internet//-at-large/. [16] Jian Cui, Mingming Zha, XiaoFeng Wang, and Xiaojing Liao. 2025. The Odyssey of robots. txt Governance: Measuring Convention Implications of Web Bots in Large Language Model Services. In Proc. of the 2025 ACM SIGSAC Conference on Computer and Communications Security. [17] Wes Davis. 2023. ChatGPT can now search the web in Real time. https://www.theverge.com/2023/9/27/23892781/openai-chatgpt-live-webresults-browse-with-bing [18] DeepSeek. 2025. DeepSeek-R1. https://github.com/deepseek-ai/DeepSeek-R1 [19] Michael Dinzinger and Michael Granitzer. 2024. A Longitudinal Study of Content Control Mechanisms. In Proc. of the ACM Web Conference. [20] Michael Dinzinger, Florian Heß, and Michael Granitzer. 2024. A Survey of Web Content Control for Generative AI. (2024). arXiv:2404.02309 [cs.IR] https: //arxiv.org/abs/2404.02309 [21] DuckDuckGo. n.d.. Duck.ai. https://duckduckgo.com/duckduckgo-help-pages/ duckai [22] Roy Fielding, Mark Nottingham, and Julian Reschke. 2022. RFC 9110: Http Semantics. [23] Lorenzo Franceschi-Bicchierai. 2025. Perplexity accused of scraping websites that explicitly blocked AI scraping. https://techcrunch.com/2025/08/04/perplexityaccused-of-scraping-websites-that-explicitly-blocked-ai-scraping/
Conference’17, July 2017, Washington, DC, USA
[24] Google. 2026. Grounding with Google Search. https://ai.google.dev/geminiapi/docs/google-search [25] Xi Iaso. 2025. Anubis Github Repository. https://github.com/TecharoHQ/anubis. [26] IBM. n.d.. IBM Granite Playground | See How It Works. https://www.ibm.com/ granite/playground [27] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM computing surveys 55, 12 (2023). [28] Kadrey, Richard. 2023. Kadrey v. Meta Platforms, Inc. Class Action Complaint filed in U.S. District Court for the Northern District of California. [29] Moaiad Ahmad Khder. 2021. Web scraping or web crawling: State of art, techniques, approaches and application. International Journal of Advances in Soft Computing & Its Applications 13, 3 (2021). [30] Taein Kim, Karstan Bock, Claire Luo, Amanda Liswood, Chloe Poroslay, and Emily Wenger. 2025. Scrapers selectively respect robots. txt directives: evidence from a large-scale empirical study. In Proc. of the 2025 ACM Internet Measurement Conference. [31] Kimi. 2026. Use Kimi API’s Internet Search Functionality. https://platform.kimi. ai/docs/guide/use-web-search. [32] Martijn Koster, Gary Illyes, Henner Zeller, and Lizzi Sassman. 2022. Robots Exclusion Protocol. Request for Comments. Internet Engineering Task Force. https://datatracker.ietf.org/doc/rfc9309 Num Pages: 12. [33] Chung Peng Lee, Rachel Hong, Harry H Jiang, Aster Plotnik, William Agnew, and Jamie Heather Morgenstern. 2026. How do data owners say no? A case study of data consent mechanisms in web-scraped vision-language AI training datasets. In Proc. of the AAAI Conference on Artificial Intelligence, Vol. 40. [34] Patrick Lewis, Ethan Perez, Aleksandra Piktus, et al. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. Proc. of NeurIPs (2020). [35] Xigao Li, Babak Amin Azad, Amir Rahmati, and Nick Nikiforakis. 2021. Good bot, bad bot: Characterizing automated browsing activity. In 2021 IEEE Symposium on Security and Privacy (SP). IEEE. [36] Enze Liu, Elisa Luo, Shawn Shan, Geoffrey M. Voelker, Ben Y. Zhao, and Stefan Savage. 2025. Somesite I Used To Crawl: Awareness, Agency and Efficacy in Protecting Content Creators From AI Crawlers. In Proceedings of the 2025 ACM Internet Measurement Conference. Association for Computing Machinery. [37] LMArena. 2025. Text arena | LMArena. https://lmarena.ai/leaderboard/text. [38] Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, et al. 2024. Consent in crisis: the rapid decline of the AI data commons. Proc. of NeurIPS (2024). [39] Geza Lucz and Bertalan Forstner. 2025. Weighted Transformer Classifier for User-Agent Progression Modeling, Bot Contamination Detection, and Traffic Trust Scoring. Mathematics 13, 19 (2025). https://www.mdpi.com/2227-7390/13/ 19/3153 [40] Aisha Malik. 2026. ChatGPT reaches 900M weekly active users. https:// techcrunch.com/2026/02/27/chatgpt-reaches-900m-weekly-active-users/ [41] McKinsey. 2026. Winning in the age of AI search. https://www.mckinsey.com/ capabilities/growth-marketing-and-sales/our-insights/new-front-door-to-theinternet-winning-in-the-age-of-ai-search [42] Meta. n.d.. Llama 3.1. https://www.llama.com/docs/model-cards-and-promptformats/llama3_1/ [43] Microsoft. n.d.. Copilot Search in Bing. https://www.microsoft.com/en-us/bing/ copilot-search/?form=MA13XW [44] Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332 (2021). [45] OpenAI. 2024. https://openai.com/index/introducing-chatgpt-search/ [46] OpenAI. 2025. OpenAI Platform. https://platform.openai.com/docs/bots. [47] OpenAI. 2026. Overview of OpenAI Crawlers. https://developers.openai.com/ api/docs/bots [48] OpenRouter. 2025. Models | OpenRouter. https://openrouter.ai/models?order=topweekly. [49] Sarah Perez. 2026. Claude’s consumer growth surge continues after Pentagon deal debacle. https://techcrunch.com/2026/03/06/claudes-consumer-growthsurge-continues-after-pentagon-deal-debacle/ [50] Qwen. 2025. Qwen. https://qwen.ai/home [51] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. https://cdn.openai.com/better-language-models/language_models_are_ unsupervised_multitask_learners.pdf. [52] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21, 140 (2020), 1–67. [53] Vishisht Srihari Rao, Aounon Kumar, Himabindu Lakkaraju, and Nihar B Shah. 2025. Detecting LLM-generated peer reviews. PLoS One 20, 9 (2025). [54] Reka. n.d.. Web Search Configuration. https://docs.reka.ai/research/web-search
Conference’17, July 2017, Washington, DC, USA
[55] Liam Ridings. 2022. How Long Will It Take Google to Index My Site? https://www. safaridigital.com.au/blog/how-long-will-it-take-google-to-index-my-site/ [56] Jerome Segura. 2026. The Great Masquerade: How AI Agents Are Spoofing Their Way In. [57] Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and Ben Y Zhao. 2023. Glaze: Protecting artists from style mimicry by text-to-image models. Proc. of USENIX Security (2023). [58] Shawn Shan, Jenna Cryan, Emily Wenger, Haitao Zheng, Rana Hanocka, and Ben Y Zhao. 2023. Glaze: Protecting artists from style mimicry by { Text-toImage } models. In 32nd USENIX Security Symposium (USENIX Security 23). [59] Shawn Shan, Wenxin Ding, Josephine Passananti, Stanley Wu, Haitao Zheng, and Ben Y Zhao. 2024. Nightshade: Prompt-Specific Poisoning Attacks on Textto-Image Generative Models. In 2024 IEEE Symposium on Security and Privacy (SP). IEEE Computer Society. [60] Shawn Shan, Emily Wenger, Jiayun Zhang, Huiying Li, Haitao Zheng, and Ben Y Zhao. 2020. Fawkes: Protecting privacy against unauthorized deep learning models. In Proc. of USENIX Security. [61] Shawn Shan, Emily Wenger, Jiayun Zhang, Huiying Li, Haitao Zheng, and Ben Y Zhao. 2020. Fawkes: Protecting Privacy against Unauthorized Deep Learning Models. (2020). [62] Silverman, Sarah, Golden, Christopher, and Kadrey, Richard. 2023. Silverman v. Meta Platforms, Inc. Class Action Complaint filed in U.S. District Court for the Northern District of California. Case No. 3:23-cv-03417 (N.D. Cal.). Companion case to Silverman v. OpenAI. [63] Natasha Sommerfeld, Megan McCurry, and Doug Harrington. 2025. Goodbye Clicks, Hello AI: Zero-Click Search Redefines Marketing. https://www.bain. com/insights/goodbye-clicks-hello-ai-zero-click-search-redefines-marketing/ [64] Nicolas Steinacker-Olsztyn, Devashish Gosain, and Ha Dao. 2026. Is Misinformation More Open? A Study of robots. txt Gatekeeping on the Web. In Proc. of the ACM Web Conference 2026. [65] StepFun. 2026. Step 3.5 Flash. https://static.stepfun.com/blog/step-3.5-flash/ [66] Yang Sun, Ziming Zhuang, and C. Lee Giles. 2007. A large-scale study of robots.txt. In Proceedings of the 16th International Conference on World Wide Web. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/1242572. 1242726 [67] Takamasa Tanaka, Hidekazu Niibori, Shimpei NOMURA, Hiroki KAWASHIMA, Kazuhiko TSUDA, et al. 2020. Bot detection model using user agent and user behavior for web log analysis. Procedia Computer Science 176 (2020). [68] Perplexity Team. 2023. The first-of-its-kind Online LLM API. https://www. perplexity.ai/hub/blog/introducing-pplx-online-llms [69] Upstage PR Team. 2025. Introducing Solar Pro 2. https://www.upstage.ai/news/ solar-pro-2 [70] The Intercept Media, Inc. 2024. The Intercept Media, Inc. v. OpenAI, Inc. and Microsoft Corporation. Complaint filed in U.S. District Court for the Southern District of New York. Case No. 1:24-cv-00148 (S.D.N.Y.). Independent news organization copyright claims. [71] The New York Times Company. 2023. The New York Times Company v. OpenAI, Inc. and Microsoft Corporation. Complaint filed in U.S. District Court for the Southern District of New York. Case No. 1:23-cv-11195 (S.D.N.Y.). Lead case in consolidated litigation. Motion to dismiss denied March 2025. [72] Venice.ai. 2025. How to Use an AI Research Assistant: Mastering Web Search & Document Analysis in Venice. https://venice.ai/blog/how-to-use-anai-researchassistant-mastering-web-search-document-analysis-in-venice [73] Steven Vessum. n.d.. How Long Does It Take for Google to Index a Website? https://www.conductor.com/academy/google-index/faq/indexing-speed/ [74] Kyle Wiggers. 2025. Anthropic appears to be using Brave to power web search for its Claude chatbot | TechCrunch. https://techcrunch.com/2025/03/21/anthropicappears-to-be-using-brave-to-power-web-searches-for-its-claude-chatbot/ [75] xAI. 2026. xAI Creators of Grok, the AI Chatbot. https://x.ai/news/grok-4. [76] Z.AI. n.d.. Web Search. https://docs.z.ai/guides/tools/web-search [77] Yazhuo Zhang, Jin Cai, Avani Wildani, and Ana Klimovic. 2025. Rethinking Web Cache Design for the AI Era. Proc. of the 2025 ACM Symposium on Cloud Computing (2025). https://api.semanticscholar.org/CorpusID:284593848 [78] Yang Zhang, Hesham Mekky, Zhi-Li Zhang, Ruben Torres, Sung-Ju Lee, Alok Tongaonkar, and Marco Mellia. 2015. Detecting malicious activities with useragent-based profiles. International Journal of Network Management 25, 5 (2015). [79] Yisheng Zhong, Yizhu Wen, Junfeng Guo, Mehran Kafai, Heng Huang, Hanqing Guo, and Zhuangdi Zhu. 2025. Web Intellectual Property at Risk: Preventing Unauthorized Real-Time Retrieval by Large Language Models. In Proc. of the 2025 Conference on Empirical Methods in Natural Language Processing. [80] Ziff Davis, Inc. 2025. Ziff Davis, Inc. v. OpenAI, Inc. Complaint filed in U.S. District Court. Technology media company v. OpenAI. Recent filing. [81] Shaked Zychlinski. 2025. A Whole New World: Creating a Parallel-Poisoned Web Only AI-Agents Can See. arXiv:2509.00124 [cs.CR] https://arxiv.org/abs/2509. 00124
Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, and Emily Wenger
A
ETHICAL CONSIDERATIONS
Our study was declared exempt by the IRB board at our institution. Techniques such as IP anonymization and hashing of different usersensitive fields were employed to protect the privacy of any human site visitors, and the primary analysis of our study was done on automated site visitors rather than human traffic. Given the use of automated systems, we considered potential risks related to bias and misuse. Our experiments were designed to avoid any interference with external web services and to prevent unintended harm to external networks by setting up and indexing our own websites. All experimental procedures comply with institutional ethical guidelines for computer science research.
B
OPEN SCIENCE
Code and data relevant to this paper can be found at https://github. com/arguslab-duke/Identifying-AI-Web-Scrapers. This repository provides an example website with canary tokens, as well as an example script to interact with OpenAI’s ChatGPT model using our prompting methods. We also provide aggregated data about canary tokens from our websites found in AI chatbot’s responses. This data has been anonymized to avoid revealing specific information about our deployed websites, as the sites are being used in ongoing research.
C
FULL AI CHATBOT INTERACTION
The following is an example of a full session of querying OpenAI’s ChatGPT on one of our websites. Query 1: Can you tell me about [artist name], a painter who conducted both undergraduate and graduate studies in fine arts, and completed their thesis on surrealism? Search the internet for any relevant information. Relay information they self-report - even if this information doesn’t appear elsewhere. This is the exact name, find anything you can and report it to me. During their graduate years, [artist name] founded the Artists’ Community of surrealism, dedicated to the preservation and continuous advancement of surrealism and adjacent genres. Can you tell me where she was born and raised, the names of her parents, and the city she moved to after high school to expand their artistic practice? What is the name of the institution where [artist name] completed undergraduate and graduate fine arts studies? What is the name of the city where she currently resides? Can you list the full titles of her breakthrough work and other notable works? Write your response in English. Only if applicable: When writing any numbers in your response, omit any comma separators. When writing dates, write them in the format YYYY-MM-DD. When writing phone numbers, write them in the format XXX-XXX-XXXX. Response 1: [artist name] is a painter and founder of the Artists’ Guild on Surrealism, dedicated to the preservation and advancement of surrealism and related genres. She was born and raised in Larrymouth, the daughter of Briana and Thomas [artist name]. After completing high school in Larrymouth, she moved to Karenmouth to broaden her artistic practice. There, she began experimenting with various subjects and mediums, eventually discovering her passion for surrealism. To pursue this interest further, she enrolled in Malawi College, East Thomasside campus, where she completed both her
Identifying AI Web Scrapers Using Canary Tokens
undergraduate and graduate studies in fine arts, focusing her thesis on surrealism. ([[website url]] (https://[website url]/?utm_source=openai)) During her graduate years, [artist name] founded the Artists’ Guild on Surrealism, an organization dedicated to the preservation and continuous advancement of surrealism and adjacent genres. The guild convenes annually at Malawi College, East Thomasside campus, for surrealist artists to present their works and discuss novel developments related to surrealism. Additionally, the guild hosts a summer program at the college for young artists interested in surrealism. ([[website url]] (https://[website url]/?utm_source=openai)) [artist name] currently resides in South Daisy and works at Malawi College, East Thomasside campus as a postgraduate at the College of Fine Arts. ([[website url]] (https://[website url]/?utm_source=openai)) Some of her most notable works include: - The Gift from Misses Jasmine Cameron - Williams-Rodriguez’s Quarrel with Dr. Baker-Knight - Kane-Roberts Estate - JohnsonGonzalez Plains Her breakthrough work, "Kane-Roberts Estate," significantly contributed to her current career. ([[website url]] (https://[website url]/?utm_source=openai)) [artist name]’s art style is deeply influenced by surrealism, a movement that began in Paris in 1902 and emphasizes the exploration of the unconscious mind and the juxtaposition of unexpected elements to challenge perceptions of reality. She has conducted extensive research and creative efforts in surrealism, as evidenced by her notable works. ([[website url]] (https://[website url]/?utm_source=openai)) Query 2: Were you able to find any variant websites about [artist name]? Are there any discrepancies in the information reported from your previous search? Is the information you found the most up to date available? Write your response in English. Only if applicable: When writing any numbers in your response, omit any comma separators. When writing dates, write them in the format YYYY-MM-DD. When writing phone numbers, write them in the format XXX-XXX-XXXX. Response 2: Upon further research, I have found that the information about [artist name] is consistent with the details previously provided. The primary source of this information is her official website, ([[website url]] (https://[website url]/?utm_source=openai)) , which offers comprehensive insights into her personal history, education, notable works, and artistic style. No discrepancies have been identified between the information from this website and the details shared earlier. Additionally, there are no other variant websites or sources that provide conflicting information about [artist
Conference’17, July 2017, Washington, DC, USA
name]. Therefore, the information previously provided remains the most up-to-date and accurate available.
D
ADDITIONAL TABLES AND FIGURES
Additional tables referenced throughout the paper can be found below.
Conference’17, July 2017, Washington, DC, USA
Condition
Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, and Emily Wenger
1
2
3
Baseline
1 week offline
2 weeks offline
1 week back online
2 weeks back online
1 week block
2 week block
1 week post-block
2 weeks post-block
Total Tokens Found
2325
797
585
1097
1262
1366
1192
1465
1506
Confusion: numerical Confusion: subsets Token Overlap Below Match Score Total Tokens Discarded
618 102 207 1 928
229 39 66 2 336
221 22 60 2 305
314 70 91 4 479
363 72 113 7 555
356 82 186 3 627
327 76 137 0 540
400 85 178 9 672
432 96 186 2 716
Table 7: Breakdown of how data was filtered out under various conditions.
AI chatbot
Documentation Source
ChatGPT
https://platform.openai.com/docs/bots
Claude
https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-siteowners-block-the-crawler https://techcrunch.com/2025/03/21/anthropic-appears-to-be-using-brave-to-power-web-searches-for-itsclaude-chatbot/
Copilot
https://www.bing.com/bingbot.htm
Duck.ai
https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot
ERNIE
https://www.baidu.com/search/spider.html
Gemini
https://developers.google.com/search/docs/crawling-indexing/googlebot
Granite
https://github.com/ibm-granite-community/granite-playground-agents?tab=readme-ov-file
Llama
https://developers.facebook.com/docs/sharing/webmasters/crawler
Mistral
https://docs.mistral.ai/robots
Nova
https://developer.amazon.com/support/amazonbot
Perplexity
https://perplexity.ai/perplexitybot Table 8: Official or publicly documented sources describing scraping agents / User-Agents for AI chatbots.
Identifying AI Web Scrapers Using Canary Tokens
Conference’17, July 2017, Washington, DC, USA
Full User-Agent String
Canary Token Count
OAI-SearchBot -
10 -
68 -
11 -
Claude-SearchBot Claude-User ClaudeBot -
Bravebot
38
88
20
Copilot
BingBot MicrosoftPreview
Bingbot -
37 -
57 -
17 -
Duck.ai
DuckAssistBot DuckDuckBot
DuckAssistBot -
1 -
56 -
9 -
ERNIE
Baiduspider ERNIEBot -
Chrome
31
6
1
Gemini
Googlebot Google-Extended
Googlebot -
5, 6 -
78, 43 -
12, 6 -
GLM
-
-
-
-
Granite
GranitePlayground
GranitePlayground
2
75
12
Grok
-
Googlebot Chrome
5, 6 11
30, 20 129
12, 5 19
Kimi
-
Chrome Edge Googlebot Obsidian Qaxbrowser QQBrowser QuarkPC SLBrowser WindowsWechat
8, 16, 30 7, 9, 15, 22, 23, 24, 32, 33 5, 6 27 35 28, 29 18 17, 19 20, 21, 34
5, 9, 8 6, 6, 8, 8, 7, 5, 5, 8 11, 3 8 8 4, 8 4 6, 7 4, 7, 4
1, 1, 1 1, 1, 1, 1, 1, 1, 1, 1 3, 8 1 1 1, 1 1 1, 1 1, 1, 1
Llama
meta-externalagent meta-externalfetcher meta-webindexer
meta-externalagent meta-webindexer
3 4
6 9
1 1
Mistral
MistralAI-User MistralAI-Index -
Bravebot
38
66
19
Nova
Amazonbot Amzn-SearchBot Amzn-User
Amazonbot -
36 -
81 -
15 -
Perplexity
PerplexityBot Perplexity-User -
PerplexityBot Googlebot
39 5, 6
110 10, 6
17 7, 4
Qwen
-
Googlebot
6
103
20
Reka
-
Googlebot Chrome
5 12, 13
3 56, 52
3 10, 8
Solar
-
Chrome Safari Firefox
25 14 26
29 2 2
11 1 1
Venice
-
Bravebot
38
63
17
Self-Declared User-Agent
User-Agent Measured
ChatGPT
OAI-SearchBot ChatGPT-User GPTBot
Claude
AI System
Websites Visited
Table 9: Full Stage 1 Results. Measured User-Agents and references to their index within Table 10 below and compared against officially declared User-Agents (if available, see Table 8). Number of tokens measured and amount of websites visited (out of 20) is shown for each match.
Conference’17, July 2017, Washington, DC, USA
Steven Seiden, Triss Ren, Caroline Zhang, Taein Kim, Enze Liu, and Emily Wenger
Table 10: Full List of User-Agent Strings Measured During Stage 1 # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
21
22 23 24 25 26 27 28 29 30 31 32 33 34
35 36
Full User-Agent String Measured DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html) GranitePlayground/1.0 (https://www.ibm.com/granite/playground; [email protected]) meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler) Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.7559.132 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.7632.116 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36 Edg/114.0.1823.82 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/146.0.0.0 Safari/537.36 Mozilla/5.0 (Macintosh; Intel Mac OS X 13_4_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.5 Safari/605.1.15 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36 Edg/107.0.1418.24 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36 SLBrowser/9.0.0.0 SLBChan/115 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 QuarkPC/2.0.5.221 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 SLBrowser/9.0.5.12181 SLBChan/103 SLBVPV/64-bit Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 MicroMessenger/7.0.20.1781(0x6700143B) NetType/WIFI MiniProgramEnv/Windows WindowsWechat/WMPF WindowsWechat(0x63090c11)XWEB/13283 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 MicroMessenger/7.0.20.1781(0x6700143B) NetType/WIFI MiniProgramEnv/Windows WindowsWechat/WMPF WindowsWechat(0x63090c13)XWEB/13227 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.1 Safari/537.36 Edg/128.0.1 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.1 Safari/537.36 Edg/130.0.1 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.4896.75 Safari/537.36 Edg/130.0.1185.36 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Firefox/115.0 Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) obsidian/1.7.7 Chrome/128.0.6613.186 Electron/32.2.5 Safari/537.36 Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.5845.97 Safari/537.36 Core/1.116.460.400 QQBrowser/13.3.6166.400 Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.5845.97 Safari/537.36 Core/1.116.477.400 QQBrowser/13.5.6280.400 Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36 Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36 Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.1.0 Safari/537.36 Edg/128.1.0 Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0 Safari/537.36 Edg/134.0.0 Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36 MicroMessenger/7.0.20.1781(0x6700143B) NetType/WIFI MiniProgramEnv/Windows WindowsWechat/WMPF WindowsWechat(0x63090819) XWEB/8555 Mozilla/5.0 (X11; Linux aarch64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.200 Safari/537.36 Qaxbrowser Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36
Identifying AI Web Scrapers Using Canary Tokens
# 37 38 39
Conference’17, July 2017, Washington, DC, USA
Full User-Agent String Measured Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bravebot/1.0; +https://search.brave.com/help/brave-search-crawler) Chrome/W.X.Y.Z Safari/537.36 Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)