Agent readiness

Does your website block AI agents?

Probably, if it is large enough to run bot protection. Bot protection exists to stop scrapers, and an AI agent fetching a page looks exactly like one — there is no setting that says “this automated reader is a customer”. On 27 October 2026 we tested eight major hotel websites with an automated reader. Five of them refused us outright.

That matters because the protection cannot tell the difference either. It is not distinguishing a hostile scraper from an assistant acting for a traveller who wants to book a room. It is answering one question — is this a real browser with a real person behind it — and for an agent, the honest answer is no.

What we actually measured

We fetch a page the way an agent or a crawler does: no JavaScript, no browser cookies, asking for the HTML. Eight hotel brands, one homepage each, measured 27 October 2026. Every row below was re-checked against a live fetch before this page was published.

Five of the eight refused an automated reader, and no two refused in the same way: a blunt block page, an empty response, a browser check, and two failures that are plainly the site’s own error handling rather than a deliberate wall. The last kind is the one worth noticing, because it means the site is not defending itself against anything — it is simply unable to serve that request, and it cannot tell an agent that.

This is not a hotel problem. It is a size problem. The larger a site gets, the more likely it runs something in front of it, and the more likely an agent sees a wall instead of a business.

Why this is worse than it looks

A block is not a dead end for every visitor — a person with a browser gets through it without noticing. But an agent that meets one has no way past it. It cannot solve the verification, it cannot run the JavaScript, and it will not try twice.

So the traffic does not fail loudly. It simply never arrives, and you cannot see it in your analytics, because a visitor who was turned away at the edge was never a session. The owner of a blocked site has no signal at all that anything is wrong.

How to check yours in a minute

Fetch your homepage without running JavaScript and without cookies, and look at what comes back. Three things tell you that your site is refusing readers:

• The page is very short — a few hundred bytes, when a real homepage is usually tens of kilobytes.
• It contains phrases like “Access Denied”, “Just a moment”, “Checking your browser”, “Verifying your browser” or “Please enable JS”.
• It loads a script from a domain containing captcha or a bot-protection vendor you do not recognise.

You can run the same check at mcphubz.com/scan. If your site turns the reader away, it will tell you that plainly and explain what to do about it, rather than showing you a score computed from a page it never saw.

What to do instead of opening the wall

You do not have to weaken your protection to serve agents. The better answer is to stop requiring an agent to read your pages at all, and publish what you offer in a form a machine can read directly. Two established options:

• Structured data — schema.org markup embedded in your HTML, describing your services, prices, availability and booking options. Marked up properly, it tells an agent what you offer without any parsing. The vocabulary is public at schema.org.
• llms.txt — a plain text file at the root of your site describing what the business is and linking to the pages that matter. The convention is documented at llmstxt.org. MCPHubz publishes one of its own at mcphubz.com/llms.txt.

Both are plain files, so they are served from the same edge that is currently blocking readers. A page that does not need to be read cannot be blocked.

Common questions