
Clean Code, Clear Answers: Technical AI Visibility
Ido Navarro and Ben Cohen explore the technical infrastructure of AI visibility, breaking down how models crawl raw HTML, the role of clean markdown and llms.txt, and why sites must be built for both search engines and autonomous agents.
Chapter 1
Why AI Can't See Your Website
Ido
Ahrefs looked at 66.6 million ChatGPT responses and about 15 million fan-out queries. Only 14.4% of those responses triggered fan-out searches at all, and they averaged roughly 1.6 sub-queries. So your page isn't competing in some giant arena. It has to match one or two searches. Miss them, and the model never sees you.
Ben cohen
That connects to our earlier episode on GEO versus SEO, where we talked about fan-outs and third-party mentions. This time I want the technical layer. Because here's what scares me. A brand can win the strategy and still lose, because its prices sit inside JavaScript the crawler never runs.
Ido
Yes. Googlebot crawls the HTML, then does a second pass to render JavaScript. The reporting we've seen says the AI tools tested, including Perplexity Sonar Pro, Gemini 2.5 Flash, Claude 4.0 Sonnet and OpenAI models, load only the initial HTML. So pricing in a client-side widget or behind an accordion that needs a click is effectively invisible.
Ben cohen
And that matters more because of what ChatGPT does when it searches. In the Ahrefs data, about 6.8% of fan-outs used the site: operator, and 75.9% of the most common ones pointed at a brand's own site. The word official showed up in 13.1% of English fan-outs in September. Price fan-outs rose 47% in a month.
Ido
So the model is literally going to your front door and asking for the price. Then it finds an empty shell.
Ben cohen
But does it always trust the brand? Comparisons look different, right? For factual questions, 63.4% of citations went to official sites. For A versus B questions, that fell to 46.4%.
Ido
So on facts you must be the source. On comparisons, others tell your story. Own the facts, earn the mentions. One caveat: Ahrefs also reported Reddit citations down 88% in six months, so those source patterns move fast. Don't build a strategy on one snapshot.
Ben cohen
Okay, so what does clean delivery look like? You've got an analogy.
Ido
Your website is a beautiful restaurant. Lighting, furniture, music. Great for guests. A clean markdown version is the printed menu. The AI doesn't need to inspect the decor. It needs the dishes and prices, without spending its limited context on navigation, scripts and repeated interface pieces.
Ben cohen
Which is where llms.txt comes in. Jeremy Howard proposed it. A markdown file at your site root, and llms-full.txt goes further with the flattened full content.
Ido
Think of llms.txt as a concise index and llms-full.txt as the whole menu with descriptions. But be careful. It's voluntary. It isn't robots.txt, it doesn't enforce anything, and nobody can promise it moves rankings. Support is real in documentation sites, like Anthropic, Perplexity and Zapier have published them, but broad adoption by AI search is unproven.
Ben cohen
So the honest answer to will this make me visible is no. It's one piece of readiness.
Ido
Same with schema. JSON-LD for organization, product, FAQ gives machines explicit context. But it must match what's visible on the page. Schema that says one price while the page says another is worse than none.
Ben cohen
And tools? LLMagnet is one example in this category. It tracks how you appear across ChatGPT, Claude and Gemini, runs prompt audits, and can manage llms.txt. Useful, but you can do the core of it by hand, and no tool fixes weak content.
Ido
Then there's the third audience. Humans, machines that interpret, and agents that act. An agent that understands your booking page but can't use the form leaves without buying. Clear labels, predictable flows, maybe APIs. That's likely the next layer after mobile and accessibility, though still early.
Chapter 2
What To Do This Week
Ben cohen
So what can someone do this week?
Ido
First, ask Claude, ChatGPT and Gemini your main buyer question. Check whether the description, pricing and specs match reality. Second, view the page source. If your core facts aren't in plain HTML, move them there.
Ben cohen
Then review crawler access in robots.txt, because blocking without understanding the consequences can make you invisible. Add accurate schema, consider llms.txt, and publish original material worth referencing. And don't stuff keywords for AI or mass produce thin pages.
Ido
And don't make the human experience worse for machines. Also, citations alone aren't success. Check accuracy and relevance too.
Ben cohen
My prediction, and it's only an opinion: within two or three years, serving a clean version of a page for machines will feel as normal as a mobile layout.
Ido
Mine is that agents completing tasks will become part of how we judge whether a site works. The goal isn't only visits. It's being discovered, understood, trusted, represented and eventually used correctly.
Ben cohen
Your small action: ask several AI systems the same customer question and compare. If you enjoyed this, follow Be Visible and share it with someone who owns a website.