Malaysian Websites Blocking AI Crawlers: 2026 Survey

What did the 100-site check find?

On 20 October 2026, 13 of the 100 websites checked had a robots.txt rule that blocked at least one of 17 AI crawlers. None blocked all 17. The most common situation wasn't a clear "yes/no" for AI - it was simply a robots.txt file that didn't mention AI crawlers at all.

These numbers cover three different scenarios. A site can publish rules that allow or block named bots. It can have no robots.txt file at all (which the robots.txt standard treats as no crawler restrictions). Or it can prevent an outside script from reading the file. That last case isn't evidence of blocking - it's just unknown.

Only 18 of the 78 readable files named an AI crawler. The other 60 didn't say anything about the bots in this check. In plain terms: silence is common, and under the robots.txt convention it generally leaves crawlers without a site-specific instruction to avoid the site.

The most frequently blocked crawler was OpenAI's GPTBot, blocked by nine sites. Common Crawl's CCBot and ByteDance's Bytespider were each blocked by eight, followed by Anthropic's ClaudeBot at seven. Google-Extended (used to signal whether Google may use pages for its AI models) was blocked by four.

Zyte, a web data company, also reported in a separate 2026 survey that 13.4% of 1,106 Malaysian websites it examined blocked AI crawlers, while 78.2% had a robots.txt file. That overall result is directionally similar, but it's not a direct like-for-like comparison: sample selection and measurement can differ. This 100-site check is a named-site snapshot, not a national census.

Which Malaysian sectors block AI crawlers?

Blocking was most common in news. Five of the 15 news sites checked blocked at least one AI crawler. In other sectors, blocks were present but not widespread. In this sample, I found no checked bank, telco, travel site, or university (with a readable robots.txt file) that blocked any of the 17 bots.

The 13 sites with confirmed blocks were New Straits Times, Harian Metro, KWSP, EdgeProp, Bernama, Maukerja, Lotus's Malaysia, Free Malaysia Today, Ricebowl, The Edge Malaysia, Bank Negara Malaysia, Zalora Malaysia and foodpanda Malaysia.

New Straits Times and Harian Metro were the strictest, each blocking 12 of the 17 checked bots. KWSP and EdgeProp each blocked eight. Their robots.txt files were identical, including a section labelled as Cloudflare-managed content. That matters operationally: the policy may be applied through shared infrastructure rather than being written from scratch by each publisher.

The news result also varied by language. Among the eight English-language news sites with readable files, four blocked an AI crawler: New Straits Times, Free Malaysia Today, The Edge Malaysia and Bernama. Harian Metro was the only confirmed block among four Malay-language sites; Berita Harian's file couldn't be read. None of the three Chinese-language newspapers in the sample blocked any of the 17 crawlers.

Outside news, the confirmed blocks included KWSP and Bank Negara Malaysia in government and agencies; Zalora and Lotus's in e-commerce; Maukerja and Ricebowl in jobs; EdgeProp in property; and foodpanda. Those examples are too few to claim a sector-wide rule. What they do suggest is that crawler decisions can come from different concerns - content licensing, server load, control over company data, or simply inherited configuration.

For business readers, the main point is simple: sector norms don't replace a decision. A publisher may have different commercial reasons than a bank, retailer, or recruitment site. Copying another company's robots.txt without understanding what each line does can lead to unintended effects - especially if a rule affects both training and discovery.

Which AI crawlers are blocked, and why does the distinction matter?

"AI crawler" isn't a single type of bot. Some crawlers collect pages that may be used for model training. Others build an AI search index. A third group fetches a page because a user asked an assistant to inspect it. Blocking one doesn't automatically block the others.

Across the 13 blocking sites, the pattern looks more like limiting training than rejecting all AI-powered discovery. Ten blocked at least one training-oriented crawler while leaving OpenAI's OAI-SearchBot and PerplexityBot available. Only four blocked a bot that fetches pages for AI search or a user's question.

Foodpanda Malaysia is a clear example. Its robots.txt blocks GPTBot, but it explicitly allows OAI-SearchBot and ChatGPT-User. In effect: training collection is blocked, but crawlers tied to search and user-requested access are allowed. Whether that trade-off makes sense depends on the business. The key point is that these controls aren't interchangeable.

Google's naming can also confuse people. Google-Extended isn't the same as Googlebot. Blocking Google-Extended is meant to restrict certain uses of content in Google's AI systems, while ordinary Google Search crawling is treated separately. In the sample, four sites blocked Google-Extended - fewer than the nine that blocked GPTBot. New Straits Times and Harian Metro, for example, blocked GPTBot and ClaudeBot but allowed Google-Extended.

Bot names can change too. Lotus's Malaysia listed older Anthropic identifiers (including "anthropic-ai" and "Claude-Web"), but not ClaudeBot. A rule for a retired or different user-agent string may not match what an operator expects. The practical lesson: check current crawler documentation and test the exact user-agent rules, rather than assuming that blocking one label covers every product or access mode.

What can robots.txt tell you - and what can't it tell you?

robots.txt is public instructions for crawlers, not a technical access-control system. A compliant crawler is expected to follow the rules. A non-compliant crawler can still request the page unless something else - like a firewall, CDN, or other controls - stops it.

For this check, I made one request to each selected site's robots.txt over HTTPS on 20 October 2026. The script followed up to three redirects, waited at most 10 seconds, and made no other page requests. Then a second, separate script re-read the 78 readable files and recalculated the 1,560 crawler verdicts. It found no differences. The files were evaluated according to the robots.txt standard, including the rule that the longest matching path takes precedence.

The sample covered 100 major Malaysian websites across ten sectors. Thirty-nine came from Similarweb's public Malaysia rankings for September 2026; the rest were selected as recognizable services in their sectors. This list wasn't presented as the 100 most visited Malaysian sites. If a bare domain didn't serve content, the check used the working address.

A few limits are worth keeping in mind:

• This is a snapshot. robots.txt files change. A policy observed on 20 October may not be the policy a crawler sees today.
• A blocked robots.txt file doesn't tell you the policy inside it. Thirteen sites returned a 403, failed to respond, or had another connection problem. A 403 might reflect bot screening, but it doesn't show what the file says or how the real AI crawler is treated.
• The check focused on the home page path. A site can block bots for specific sections while leaving the root path available. JobStreet Malaysia, for example, restricted some company pages and job listings while still allowing the home page - so it counted as allowing under this method.
• robots.txt can't show firewall or CDN behavior. A CDN or security service can block a crawler before it even reads robots.txt. And a site may publish a disallow rule that an uncooperative bot ignores.
• The test script isn't the actual crawler. Websites can treat different user-agents, networks, and request patterns differently.

These limits don't make the data useless. They define what the data measures: publicly available robots.txt instructions for a set of bot names and the home page path at one point in time. It doesn't prove what an AI company actually collected, whether a site appears in an AI answer, or whether a firewall blocks a particular crawler.

How should a Malaysian business set its AI crawler policy?

Start with the business decision, not with bot names. A company should decide separately whether it's comfortable with: (1) model training, (2) AI search indexing, and (3) assistants fetching pages for users. Those uses raise different issues for licensing, traffic, discoverability, and user experience.

A practical review can follow five steps:

1. Separate training from discovery

   Ask two questions: may AI companies use public pages for model training, and may AI search tools read pages to retrieve, quote, or link to them? You might say yes to both, no to both, or allow discovery while restricting training. Make the distinction explicit in your policy, and confirm that the crawler groups you block or allow actually match those goals.

   Blocking training bots can make sense for licensed editorial content, proprietary datasets, or pages where reuse carries real commercial risk. Allowing search crawlers may still be useful if customers use AI assistants to find products, support information, or expertise. Either way, there's no guarantee of a specific business outcome - you're setting crawler instructions, not controlling results.

2. Map each bot to a job

   Don't treat every user-agent that includes an AI company's name as equivalent. Separate training-oriented agents (like GPTBot or ClaudeBot) from search agents (like OAI-SearchBot and PerplexityBot) and from user-triggered agents (like ChatGPT-User). Also, verify current user-agent names before you change the file - older identifiers may no longer match the crawler you think you're controlling.

3. Decide by page type, not only by domain

   A one-size rule can be simpler, but it can also block useful access to public documentation, product pages, or support articles. And robots.txt isn't a privacy or authentication tool - so don't rely on it to keep sensitive customer data safe. Use access controls for confidential material, and use robots.txt rules to express preferences about public content.

   When it fits your needs, use narrower path rules. JobStreet Malaysia's approach - restricting some company pages and job listings while still allowing the home page - shows how granular rules can work. The trade-off is more operational complexity: the more detailed the rules, the more careful you need to be when you test them after site updates.

4. Check the layer in front of your site

   A crawler can be blocked by a CDN or firewall before it ever reaches robots.txt. Ask the team (or provider) managing your infrastructure whether known AI bots are blocked, challenged, or rate-limited. If you want a bot to access your content, test the actual response for that bot's user-agent where possible. A public allow rule doesn't guarantee success if the edge layer returns a 403.

5. Monitor results, not just the file

   Keep a dated copy of your policy. Recheck it when crawler names or business priorities change, and test representative paths after you deploy updates. Look at server logs for crawl volume, errors, and requested paths. That gives you a more grounded basis for decisions than treating "AI traffic" as a single metric. The real question is whether access supports your business goals without creating unacceptable cost, exposure, or reuse risk.

A good policy isn't automatically the most permissive or the most restrictive. It's the one that clearly states the trade-off you intend, works across robots.txt and infrastructure layers, and can be maintained as products and crawlers evolve.