Cloudflare has changed the way website owners can manage AI crawlers, replacing a broad “block AI bots” approach with more precise controls for Search, Training and Agent activity.
That change matters because not all AI crawlers are doing the same thing.
Some crawlers index website content so it can appear in search results or AI-powered answers. Others collect content for model training. Some act in real time on behalf of users, visiting websites to retrieve information or complete tasks.
Until recently, website owners often faced a blunt choice: allow AI crawlers or block them.
Cloudflare’s new controls make that decision more granular.
Businesses can now distinguish between crawlers that support search visibility, crawlers that may use content for AI training, and agents interacting with websites on behalf of users. Cloudflare has also introduced a “Disallow AI Training” option designed to let major search crawlers continue indexing content while signalling that the same content should not be used for AI training.
For website owners, this creates an important strategic question.
Blocking AI crawlers may help protect valuable content from being used for model training, but blocking the wrong crawler can also reduce search visibility and limit how easily content can be discovered, understood or surfaced by search and AI systems.
The issue is therefore no longer simply whether AI bots should be allowed.
It is deciding which AI behaviours support the business, which create risk, and how much visibility a website owner is prepared to trade for greater control over their content.
For businesses investing in SEO, AEO and GEO, those decisions are becoming part of the broader visibility strategy rather than just another security setting inside Cloudflare.
What Has Cloudflare Changed With AI Crawlers?
Cloudflare has replaced its broad “Block AI Bots” approach with more granular controls that separate AI crawler activity into three categories: Search, Training and Agent.
This gives website owners more control over how different crawlers interact with their content rather than forcing them to make one decision that applies to all AI-related activity.
The three categories reflect different crawler behaviours.
Search crawlers are used to discover and index content for search results and answer systems.
Training crawlers access content for use in training or fine-tuning AI models.
Agent crawlers access websites in real time on behalf of users, for example when an AI assistant retrieves information or performs a task.
Cloudflare is also changing its recommended setting for AI Training crawlers from Block to Disallow AI Training.
That distinction is important.
Rather than technically blocking all relevant crawlers from accessing the site, the Disallow AI Training setting is designed to allow major mixed-use search crawlers to continue indexing content for search while signalling that the content should not be used for AI training.
Cloudflare is automatically migrating existing configurations into the new system.
If a website previously had Block AI Bots or Managed Robots.txt enabled, those settings are being converted into separate Search, Training and Agent controls. According to Cloudflare, manual changes made during the migration period will be preserved.
Website owners who have never changed their AI crawler settings will generally remain on an Allow configuration.
For most businesses, the immediate action is therefore not to start blocking crawlers.
It is to review the new settings and make sure they still align with the website’s search visibility, AI visibility and content protection objectives.
What Is the Difference Between Search, Training and Agent Crawlers?
Cloudflare’s new AI crawler controls separate activity into three categories: Search, Training and Agent. The difference between them comes down to what the crawler is trying to do with your website content.
| Crawler type | What it does | Typical business impact |
|---|---|---|
| Search | Crawls and indexes content for search results and answer systems | Supports discoverability |
| Training | Accesses content for use in training or fine-tuning AI models | May use content without generating direct referral traffic |
| Agent | Accesses a website in real time on behalf of a user | Can retrieve information or interact with the site |
Search Crawlers
Search crawlers discover and index content so it can be surfaced when users search for information.
For most commercial websites, this activity is valuable because it supports visibility in search engines and other discovery environments.
Blocking search crawlers can reduce how easily content is indexed and found, which is why businesses focused on SEO and organic visibility need to be particularly careful with these settings.
Training Crawlers
Training crawlers access website content so it can be used to train or fine-tune AI models.
This is where many website owners have concerns.
The content may contribute to an AI system’s future capabilities without necessarily sending traffic back to the original website or providing a direct commercial benefit to the publisher.
Cloudflare’s new controls are designed to give website owners a way to treat this activity differently from normal search crawling.
Agent Crawlers
Agent crawlers operate differently again.
Rather than indexing content for future search or using it to train a model, an AI agent may access a website in real time on behalf of a user.
For example, an AI assistant might visit a website to retrieve information, check availability, compare options or complete part of a task.
This type of activity is likely to become increasingly important as AI systems move beyond simply answering questions and begin performing actions for users.
The key point is that AI crawler activity is no longer one single category.
A business may want search crawlers to access its website, restrict training activity and take a more cautious approach to agents.
That is why Cloudflare’s move away from a single Block AI Bots switch is significant. It allows website owners to make different decisions based on what each type of crawler is actually doing.
Why Mixed-Use Crawlers Created a Problem
One of the biggest challenges with AI crawler management is that some crawlers can serve more than one purpose.
A crawler may be used to index content for search while also supporting other AI-related functions. That creates a problem for website owners who want to remain discoverable in search but do not necessarily want their content used for model training.
Under a broad blocking approach, the distinction could be difficult to manage.
Blocking a crawler completely might stop unwanted training activity, but it could also stop that same crawler from accessing content for search indexing or other legitimate discovery purposes.
For businesses that depend on organic visibility, that is a significant trade-off.
Googlebot, Bingbot and Applebot are examples of major crawlers that Cloudflare identifies as relevant to this mixed-use issue. Website owners may want these crawlers to continue accessing content for search and discovery while placing restrictions on how that content is used for AI training.
This is why Cloudflare’s move towards more granular controls matters.
Instead of treating every crawler as either completely trusted or completely blocked, website owners can now separate the intent behind the crawl.
The goal is to make it possible to say:
Yes, you can crawl this content for search.
But also:
No, we do not want this content used for AI training.
That distinction is particularly important for businesses investing in SEO, AEO and broader AI visibility.
If search crawlers are blocked unintentionally, the result can be reduced indexing and discoverability. If everything is allowed by default, valuable content may be used in ways the website owner did not intend.
Mixed-use crawlers therefore turn AI crawler management into a strategic visibility decision rather than a simple security setting.
What Is Cloudflare’s “Disallow AI Training” Setting?
Cloudflare’s Disallow AI Training setting is designed to let website owners restrict the use of their content for AI model training without automatically blocking major search crawlers from accessing the site for search and discovery.
This is a significant change from a simple block-or-allow approach.
When Disallow AI Training is enabled, Cloudflare uses robots.txt-based preferences to signal that the website owner does not want content used for AI training. At the same time, accountable mixed-use crawlers can continue accessing the site for permitted search-related purposes.
In practical terms, the setting is intended to separate two very different activities:
Crawling content so it can appear in search or answer experiences
and
Crawling content so it can be used to train or fine-tune an AI model
That distinction is important for businesses that want to protect valuable website content without unnecessarily reducing search visibility.
It is also important to understand that Disallow AI Training is not the same as technically blocking a crawler.
A disallow instruction communicates the website owner’s preference to compliant crawlers. A block, on the other hand, prevents the crawler from accessing the site at all.
That means the effectiveness of Disallow AI Training depends on crawlers respecting the instruction.
Cloudflare has positioned the setting as the recommended approach for many website owners because it offers a more balanced option than completely blocking mixed-use crawlers.
For businesses focused on SEO and organic visibility, that balance is particularly important.
A company may want Google, Bing and other search systems to continue discovering and indexing its content while still making it clear that the same content should not be used for AI training.
The key takeaway is simple:
Disallow AI Training is intended to protect content usage rights without automatically sacrificing search access.
What Is the Difference Between “Disallow AI Training” and “Block”?
The difference is straightforward but important.
Disallow AI Training tells compliant crawlers not to use your content for AI model training while still allowing permitted search-related crawling.
Block prevents the crawler from accessing the website altogether.
That means the two settings can have very different consequences for visibility.
If a website owner chooses Disallow AI Training, major mixed-use crawlers may still be able to access and index the site for search purposes while respecting the website owner’s preference not to use the content for training.
If the crawler is blocked instead, it cannot access the content at all.
For businesses that rely on organic search, this distinction matters because some crawlers are used across multiple functions.
A complete block can therefore solve one problem while creating another.
For example, a business may want to stop content from being used for AI training but still want that content indexed by major search engines. Using a full block against a mixed-use crawler could interfere with that visibility.
This is why Cloudflare has shifted its recommended approach for Training crawlers towards Disallow AI Training rather than simply blocking them.
The practical difference can be summarised like this:
| Setting | What it does | Potential visibility impact |
|---|---|---|
| Disallow AI Training | Signals that content should not be used for AI training while permitting supported search crawling | Lower risk to search visibility |
| Block | Prevents the crawler from accessing the site | Can reduce indexing and discoverability if the crawler also supports search |
The important point is that blocking is a technical access decision, while disallowing training is a usage preference.
For most business websites, those should not be treated as the same thing.
Before selecting Block, website owners should understand exactly which crawler is being blocked, what other functions it performs and whether restricting access could unintentionally affect search or AI-driven discovery.
Will Blocking AI Training Hurt SEO?
Using Cloudflare’s Disallow AI Training setting should not, by itself, prevent major search crawlers from indexing a website for search where those crawlers support the distinction.
The greater SEO risk comes from using a full Block setting against crawlers that also perform search-related functions.
If Googlebot, Bingbot or another search crawler cannot access your content, that can affect crawling, indexing and ultimately search visibility. The extent of the impact will depend on which crawler is blocked and what role it plays.
This is why the difference between disallowing training and blocking access is so important.
A business may want to prevent its content from being used to train AI models while still allowing search engines to discover and index the same pages. Cloudflare’s newer controls are designed to make that separation easier.
For most businesses investing in SEO, the objective should be to avoid unnecessary restrictions on legitimate search crawling.
Before changing crawler settings, website owners should therefore ask:
Does this crawler help search engines discover or index my content?
If the answer is yes, a complete block may create an unintended visibility problem.
Businesses should also be cautious about assuming that all AI-related crawlers should be treated the same way. Some may be used only for training, while others support search, answers or other discovery functions.
The safest approach is to manage crawler behaviour based on purpose rather than applying a blanket block across all AI-related activity.
In practical terms, that means businesses can potentially protect content from training use without sacrificing the search visibility they have spent years building.
Can Blocking AI Crawlers Hurt AEO or GEO Visibility?
Potentially, yes.
If an AI or search system cannot access your website content, it may have fewer opportunities to understand, index, cite or surface that content in search results or AI-generated answers.
That does not mean crawler access is the only factor that determines AEO or GEO visibility.
AI systems can also rely on search indexes, third-party websites, structured data, citations, licensed datasets and other sources when generating answers. A website could therefore still be mentioned or referenced even if a particular crawler is restricted.
However, blocking access can reduce one of the pathways through which your content becomes discoverable.
This is particularly important for businesses investing in AEO and GEO.
AEO focuses on making content clear, structured and useful enough to answer specific customer questions.
GEO focuses more broadly on building the authority, depth and credibility needed for a business or source to be cited or referenced within AI-generated responses.
Both strategies depend on discoverability.
If a business creates detailed, authoritative content but then prevents relevant search or AI systems from accessing it, some of the value of that content may be reduced.
This is why blanket blocking can create an unintended trade-off.
A business may successfully restrict unwanted training activity while also reducing the likelihood that its content can be found and used in legitimate search or answer experiences.
The better approach is to separate content protection from content discoverability.
For many businesses, that could mean allowing search-related crawling, disallowing AI training where appropriate and reviewing agent access separately.
The objective is not to make a website accessible to every crawler.
It is to make sure crawler restrictions align with the business’s visibility strategy rather than accidentally working against it.
Should Businesses Allow Search Crawlers?
For most commercial websites, allowing legitimate search crawlers will usually support the business’s visibility objectives.
Search crawlers help search engines discover, understand and index website content. If those crawlers cannot access the site, pages may not be crawled properly, indexed as expected or surfaced when potential customers search for relevant products, services or information.
That makes search crawler access particularly important for businesses investing in SEO.
It can also matter for AEO and GEO because search indexes and other discovery systems may contribute to how content is found, interpreted and surfaced in AI-generated answers.
This does not mean every crawler identifying itself as a search bot should automatically be trusted.
Website owners still need to consider crawler legitimacy, security, server load and whether the crawler is actually providing a useful discovery function.
But broadly blocking major legitimate search crawlers simply because they are associated with AI can create unnecessary visibility risk.
For most lead-generation and ecommerce websites, the more sensible approach is to separate search access from training access.
That means allowing legitimate search crawlers where discoverability is important, while managing AI training preferences separately through controls such as Cloudflare’s Disallow AI Training setting.
There may be exceptions.
A private portal, membership website, proprietary research platform or business with commercially sensitive content may have legitimate reasons to restrict access more heavily.
For a public-facing business website, however, search visibility normally depends on allowing the systems responsible for discovering and indexing that content to access it.
The key question is therefore not simply:
Should this crawler be allowed?
It is:
What does this crawler do, and does that activity support or conflict with the business’s objectives?
Should Businesses Allow AI Training Crawlers?
There is no single answer that applies to every website.
Whether a business should allow AI training crawlers depends on the value of its content, how much control it wants over that content, and whether there is a strategic benefit in allowing broader AI access.
For some businesses, restricting training may make sense.
A publisher, research organisation, specialist consultancy or company producing valuable proprietary content may not want that material used to train AI models without permission or compensation.
In those cases, Cloudflare’s Disallow AI Training setting provides a way to express that preference without necessarily blocking supported search crawling.
Other businesses may take a more open approach.
A company publishing educational content primarily to increase brand awareness and market reach may place greater value on broad accessibility, particularly if it believes wider AI exposure could support future discoverability.
However, it is important not to confuse AI training with search visibility.
Allowing a crawler to use content for model training does not guarantee that the business will receive referral traffic, citations, recommendations or stronger visibility in AI-generated answers.
Likewise, disallowing training does not automatically mean the business disappears from search or AI-powered discovery.
Those are different functions.
For most businesses, the decision should therefore come down to a simple strategic question:
Do we want our content used to help train AI models, and if not, can we restrict that use without unnecessarily reducing legitimate discovery?
That is why Cloudflare’s newer controls are more useful than a single Block AI Bots switch.
They allow website owners to protect content from one type of use without automatically treating every AI-related crawler as a threat.
The right setting should reflect the role the website plays in the business.
A public lead-generation site may prioritise visibility.
A content publisher may prioritise intellectual property protection.
A business with commercially sensitive information may choose tighter controls again.
The important point is that AI training access should now be treated as a deliberate content strategy decision rather than something left on or off by default.
What About AI Agents?
AI agents represent a different type of crawler activity from search indexing or model training.
Instead of visiting a website to build a search index or collect data for future AI training, an agent can access a site in real time on behalf of a user.
That could include retrieving information, checking availability, comparing options or interacting with parts of a website as part of a broader task.
As AI systems become more capable, this type of activity is likely to become more common.
For businesses, that creates both opportunities and risks.
An AI agent could make it easier for potential customers to access information about a product or service without navigating the website manually. In that sense, agent access could become another form of customer discovery and interaction.
However, websites also need to consider what those agents are allowed to do.
A site may be comfortable allowing an agent to read public product information but less comfortable allowing automated interaction with forms, account areas, checkout processes or other transactional elements.
That is why Cloudflare separates Agent activity from Search and Training.
There is not yet the same established framework for expressing a simple “do not use this for agents” preference in the way website owners can signal restrictions on AI training.
For many businesses, Agent access should therefore be reviewed based on how the website functions.
A simple lead-generation website may have relatively limited risk.
An ecommerce store, booking platform, membership site or website with transactional functionality may need to think more carefully about what automated agents should be able to access and interact with.
The key point is that AI agents are not simply another version of a search crawler.
They can act on behalf of users in real time, which means website owners increasingly need to consider not only who can read their content, but who or what can interact with their digital systems.
What Happens to Existing Cloudflare Settings?
Cloudflare is automatically migrating customers from its older AI crawler controls to the new Search, Training and Agent framework.
If a website previously had Block AI Bots or Managed Robots.txt enabled, those settings will be converted into the newer, more granular controls.
For example, where Block AI Bots was previously set to Block, Cloudflare says the migrated configuration will generally become:
Search: Allow
Training: Disallow AI Training
Agent: Block on pages with ads
This reflects the broader shift away from treating all AI crawler activity as one category.
Website owners who have never changed their crawler settings will not be automatically moved into a more restrictive configuration. According to Cloudflare, those sites will remain on Allow.
Cloudflare is also rolling the interface changes out gradually. Website owners may therefore see the new controls appear at different times over the rollout period.
If manual changes are made before the migration is completed, Cloudflare says those changes will be preserved.
The migration is considered complete when the old Block AI Bots switch is removed from the Cloudflare dashboard.
For most businesses, this means no urgent action is required.
However, websites that previously used the older blocking controls should still review the new configuration once the migration is complete.
The important question is whether the automatically migrated settings still match the business’s objectives for search visibility, AI training and agent access.
A setting that made sense under a simple block-or-allow system may not be the best choice now that those different crawler behaviours can be managed separately.
What Settings Should Most Business Websites Consider?
For many standard business websites focused on visibility, a sensible starting point is to allow legitimate search crawling, restrict AI training where appropriate, and review agent access separately.
A practical configuration might look like this:
| Setting | Possible starting point |
|---|---|
| Search | Allow |
| Training | Disallow AI Training |
| Agent | Review based on business need |
This is not a universal recommendation.
The right configuration depends on what the website is designed to achieve, how valuable or sensitive the content is, and how important search and AI visibility are to the business.
For most lead-generation and ecommerce websites, Search: Allow will usually make sense because legitimate search crawlers need access to discover and index public content.
For Training, many businesses may prefer Disallow AI Training rather than a full block. This allows the business to signal that its content should not be used for model training without unnecessarily restricting supported search crawling.
Agent access requires a more individual decision.
A simple informational website may have little reason to restrict agents heavily if they are only retrieving public content.
An ecommerce store, booking platform, membership site or website with forms and transactional functionality may need to take a more cautious approach because agents could potentially interact with more than just public information.
Publishers and businesses with valuable proprietary content may also choose stricter settings than a typical commercial website.
The most important point is that these settings should reflect business objectives rather than being left on defaults without review.
For a website investing heavily in SEO, AEO and GEO, blocking legitimate discovery systems can work against visibility goals.
For a business whose content has significant intellectual property value, allowing unrestricted training may create a different type of risk.
The best approach is therefore to treat Search, Training and Agent as three separate decisions rather than assuming one setting should apply to all AI crawler activity.
Why This Matters for SEO, AEO and GEO Strategy
AI crawler management is no longer just a technical or security decision.
It is becoming part of visibility strategy.
For years, one of the core technical SEO questions was simple: Can search engines crawl and index the website?
That question still matters, but businesses now need to think more broadly.
Can search engines access the content?
Can AI systems understand and use it?
Can the content appear in direct answers?
Can the business be cited or referenced in AI-generated responses?
And does the business want that same content used for model training?
Those are different questions and they can require different settings.
For SEO, blocking legitimate search crawlers can reduce indexing and organic visibility.
For AEO, restricting access to useful, well-structured content may reduce the opportunities for that content to be surfaced when users ask direct questions.
For GEO, accessibility can also matter because AI systems need reliable sources and signals to understand which businesses, brands and content are credible enough to reference or cite.
At the same time, allowing every crawler unrestricted access is not automatically the right strategy either.
A business may want its content discoverable in search and AI-generated answers while still restricting the use of that content for model training.
That is what makes Cloudflare’s more granular approach useful.
Instead of treating AI access as an all-or-nothing decision, website owners can begin aligning crawler permissions with different visibility objectives.
For businesses investing in SEO, AEO and GEO, the key principle is simple:
Do not make crawler decisions in isolation.
A setting that looks sensible from a security or content-protection perspective can still create unintended consequences for discoverability if it blocks systems the business relies on for visibility.
Crawler management should therefore be reviewed alongside technical SEO, robots.txt, indexing, structured content and broader AI visibility strategy.
The objective is to protect valuable content without making the business harder to find.
AI Crawlers Are Changing the Economics of Web Content
The rise of AI crawling is changing the relationship between website owners and the platforms that consume their content.
Traditional search engines generally created a relatively clear value exchange.
A search engine crawled and indexed a website, then sent users back to that website through search results. The website owner received visibility and referral traffic in return for allowing the search engine to access its content.
AI systems can operate differently.
Content may be crawled, processed and incorporated into AI systems without generating the same level of direct referral traffic back to the original source.
That creates a new commercial question for publishers and businesses:
What value does the website receive in return for allowing its content to be accessed?
For some businesses, broad accessibility may still be worthwhile because appearing in search results, AI answers or recommendation environments can increase brand awareness and customer discovery.
For others, particularly publishers, research organisations and businesses producing valuable proprietary content, unrestricted AI training may offer less obvious commercial benefit.
This is one reason crawler controls are becoming more sophisticated.
Website owners increasingly need to distinguish between activity that supports discoverability and activity that primarily benefits the AI system consuming the content.
Search crawling may generate traffic.
AI answer visibility may generate brand exposure or citations.
Training activity may contribute to the development of an AI model without necessarily providing a measurable return to the content creator.
Those different outcomes should not automatically be treated as equivalent.
The growth of AI also means website content is becoming valuable in more ways than simply attracting human visitors.
It can be indexed, summarised, cited, used to answer questions, incorporated into datasets or accessed by automated agents.
That makes crawler policy a commercial issue as much as a technical one.
The question for businesses is no longer simply whether a crawler is legitimate.
It is whether the way that crawler uses the content aligns with the value the business expects to receive in return.
Cloudflare’s new controls reflect that shift by giving website owners more ability to make those decisions separately rather than applying one rule to every automated system.
What Should Website Owners Do Now?
The first step is to review the new Cloudflare crawler settings rather than assuming the default or migrated configuration is automatically right for your website.
Start by checking how Search, Training and Agent activity is currently configured.
For most public-facing business websites, search access will usually be important because legitimate search crawlers need to discover and index the site.
Training access is a separate decision.
If the business does not want its content used for AI model training, Disallow AI Training may provide a more balanced option than applying a full block to mixed-use crawlers.
Agent access should then be reviewed based on how the website functions and what level of automated interaction the business is comfortable allowing.
Website owners should also review their robots.txt configuration and make sure there are no conflicting directives that could unintentionally restrict important crawlers.
After changing crawler settings, monitor the website rather than assuming the configuration is working as intended.
That should include checking:
- Google Search Console for crawling and indexing issues
- Bing Webmaster Tools where relevant
- organic traffic and search visibility
- Cloudflare crawler activity
- any unexpected changes in indexing or AI referral traffic
Businesses investing in AEO and GEO should also monitor whether important pages remain accessible to the systems they want discovering and using their content.
The key is to avoid making crawler decisions based purely on labels such as “AI bot”.
A crawler may support search, training, agent activity or multiple functions.
Before blocking anything, understand what the crawler does and whether restricting it could affect the business’s visibility.
For most website owners, the objective should be straightforward:
Allow the discovery activity that supports the business, restrict the uses that do not, and monitor the impact of any changes.
How Overt Digital Marketing Approaches AI Crawler Management
At Overt Digital Marketing, we would not recommend allowing or blocking AI crawlers simply because they are associated with AI.
The right decision depends on what the crawler does, what the website is trying to achieve and how important the content is to the business.
For most commercial websites, the priority is to protect valuable content without reducing discoverability.
That means looking at crawler settings alongside SEO, AEO, GEO, robots.txt, indexing and the wider technical setup of the website rather than treating Cloudflare’s AI controls as an isolated security setting.
For example, a lead-generation website may benefit from allowing legitimate search crawling while restricting AI training.
A publisher or research-based business may want stronger controls around training because its content carries greater commercial or intellectual property value.
An ecommerce or booking website may need to review AI agent access more carefully because automated systems could potentially interact with transactional elements of the site.
The important part is understanding the trade-off before making changes.
Blocking too broadly can reduce visibility.
Allowing everything without review can give crawlers access to content in ways the business may not want.
Our approach is therefore to assess each type of crawler against the role it plays in the customer discovery journey and the commercial value of the content being accessed.
The objective is not to maximise crawler access.
It is to create the right balance between visibility, control and commercial value.
As AI-powered search and automated agents become a larger part of how people discover and interact with businesses online, crawler management will increasingly become part of broader digital marketing strategy rather than just a technical configuration.
Frequently Asked Questions
What is an AI crawler?
An AI crawler is an automated bot that accesses website content for purposes such as search indexing, AI model training or real-time agent activity. Different crawlers can perform different functions, which is why they should not all be treated the same way.
Should I block AI crawlers from my website?
Not necessarily. Blocking every AI-related crawler can reduce search and AI discoverability. A better approach is to separate Search, Training and Agent activity and decide which behaviours support your business objectives.
Will blocking AI training affect Google rankings?
Using Cloudflare’s Disallow AI Training setting is designed to restrict training use without automatically blocking supported search crawling. However, fully blocking a crawler that also supports search can affect crawling and indexing, which may reduce organic visibility.
What is Cloudflare’s Disallow AI Training setting?
Disallow AI Training tells compliant crawlers that website content should not be used for AI model training while allowing supported search-related crawling to continue. It is different from a technical block, which prevents the crawler from accessing the site altogether.
What is the difference between Search, Training and Agent crawlers?
Search crawlers discover and index content for search and answer experiences. Training crawlers access content for AI model training or fine-tuning. Agent crawlers visit websites in real time on behalf of users to retrieve information or perform tasks.
Can I block AI training but still appear in Google?
Yes, that is one of the main purposes of Cloudflare’s more granular controls. A business can allow legitimate search crawling while signalling that its content should not be used for AI training, provided the crawler supports and respects that distinction.
Can blocking AI bots hurt ChatGPT or AI visibility?
Potentially. If relevant search or AI systems cannot access your content, they may have fewer opportunities to understand, index, cite or surface it. However, AI visibility can also be influenced by search indexes, third-party sources, citations and other data.
What is the difference between robots.txt and blocking a crawler?
Robots.txt communicates crawling preferences to bots that choose to respect those instructions. A technical block prevents the crawler from accessing the website regardless of its preferences. The two approaches therefore provide different levels of control.
Should I allow Googlebot, Bingbot and Applebot?
For most public-facing business websites focused on search visibility, allowing legitimate major search crawlers will usually support discoverability. Website owners should still review the purpose of each crawler before applying broader AI-related restrictions.
Where do I manage AI crawler settings in Cloudflare?
Cloudflare’s AI crawler controls can be reviewed within its security and crawler management settings. Website owners should look for the newer Search, Training and Agent controls as Cloudflare completes the migration away from the older Block AI Bots setting.
