Engineering for AI Search & Assistants
A growing share of "search" now happens inside AI assistants — ChatGPT, Claude, Perplexity, Copilot, Gemini — and inside Google's own AI Overviews and AI Mode. When someone asks an assistant "best detailer near me" or "who can rebuild my restaurant's website," the answer is assembled from web pages the assistant's crawlers could reach, parse, and trust. Making a website legible to those systems is now part of the job. Very little of it is new; most of it is web fundamentals finally being enforced.
Who's crawling, and how
The major AI crawlers identify themselves and publish their user agents: OpenAI's
GPTBot and OAI-SearchBot, Anthropic's ClaudeBot, Perplexity's
PerplexityBot, Google's Google-Extended token (controlling AI training use, separate
from Googlebot), and Microsoft's Bingbot feeding Copilot. Two engineering facts matter:
- Most AI crawlers execute little or no JavaScript. They fetch HTML. A site whose content only exists after a framework boots is, to these systems, an empty page. Server-rendered or static HTML is the single highest-leverage decision for AI visibility.
- They respect robots.txt — which cuts both ways. A blanket block-everything rule (or a WAF bot rule) written years ago to stop scrapers now silently removes a business from AI answers. Crawl policy has become a business decision and deserves a deliberate one.
Semantic HTML is the whole ballgame
Assistants extract facts and quote passages. Extraction quality tracks markup quality almost linearly:
- One
<h1>that says what the page is;<h2>/<h3>forming a real outline — heading hierarchy is the skeleton these systems chunk content by. <main>,<nav>,<article>,<footer>landmarks, so boilerplate is separable from substance.- Facts stated in prose, near the entity they describe: hours, address, service area, and prices written as sentences a system can lift verbatim — not locked inside images or icon grids.
- Structured data reinforcing the same facts (see our structured data research) — entity clarity for machines that resolve businesses against knowledge graphs.
The best AI search optimization is indistinguishable from honest technical SEO plus clear writing. There is no secret channel.
What llms.txt actually is
llms.txt is a proposed convention (from Answer.AI's Jeremy Howard, 2024): a Markdown
file at the site root offering language models a curated map of a site's most useful content —
a table of contents written for machines with small context windows. Status report, honestly
given: no major AI provider has committed to fetching it, and Google has said publicly
it doesn't use it. We publish one on this site anyway — it costs a few hundred bytes, several
smaller tools do read it, and this lab exists precisely to hold positions we can revise with
evidence. We treat it as a low-cost hedge, and we do not sell it to anyone as an AI-visibility
guarantee.
Measuring AI visibility
This is the least mature part of the field, but it is measurable:
- Server logs / edge analytics: count AI-crawler hits by user agent — we log these per-site on every BYHS property, which tells us which assistants can see which pages.
- Referral traffic: ChatGPT, Perplexity, and Gemini send identifiable referrers when a user clicks through a citation.
- Direct interrogation: periodically asking the major assistants the questions a customer would ask, and recording whether and how the business is cited.
The BYHS position
We build every site server-rendered or static, semantically structured, fact-dense, with deliberate crawler policy and entity-anchored structured data — and we let AI crawlers in by default on production sites, because being quotable is the point. Demo and staging properties are the opposite: explicitly blocked, so unfinished work never leaks into an assistant's answer. Both policies are standards precisely because we tested them here first.