Optimizing Websites for AI Search and Autonomous Agents

Ben cohen

A business buyer can shortlist vendors, compare them, and nearly decide without ever opening your website. Forrester found that 94% of business buyers now use AI search when buying, and they rate it above vendor websites, product experts, and sales reps. So how does a brand stay visible when the interface to the web is an assistant?

Ido

That's the question for this episode of Be Visible. I'm Ido Navarro.

Ben cohen

I'm Ben Cohen. And the useful part is that the data shows two different behaviors. AI trusts your site for some questions and mostly ignores it for others.

Ido

Start with Ahrefs' study of ChatGPT. They looked at 66.6 million responses and 15 million fan-out queries, which are the extra searches the model runs behind the scenes. Only 14.4% of responses contained any fan-out, about 1.6 queries each. So your page has to match one or two searches, or the model never sees it.

Ben cohen

And when it does search a single site with the site: operator, 75.9% of the most common targets were brand sites. Apple, IKEA, Nike. Price fan-outs rose 47% in one month, from 6% to about 8.9% of queries. For facts, your own domain is the source.

Ido

Which means a price hidden in an image or a script is a price the model can't quote. Now the flip side. In a separate Ahrefs test of 20,000 US prompts, 63.4% of citations for factual questions went to official sites. For comparisons like A versus B, that fell to 46.4%.

Ben cohen

So the moment the buyer asks which is better, you lose home advantage. Third party reviews and publishers take over. Ahrefs also noted that partner publications with OpenAI licensing deals doubled their citation share. It's their data, so treat it as a snapshot.

Ido

And then there's Google. AI Overviews appeared on 82.9% of branded desktop searches in Ahrefs' data, and in their sample of 100 big brands, 76.2% of those sat in position one, above the brand's own site. External citations leaned on Wikipedia at 71.9%, YouTube at 38.6%, LinkedIn at 22.8%.

Ben cohen

So even someone typing your name might read a summary built from LinkedIn and a YouTube video you didn't make. But is that really different from SEO? Isn't this just good content and good links?

Ido

Much of it is. Crawlability, clear pages, authority still matter. What's new is that being found isn't enough. You need to be understood, selected, and quoted. And your site now has three audiences: the human, the machine reading it, and the agent that wants to act.

Ben cohen

Take the machine. What does it actually see?

Ido

Think of your site as a beautifully designed restaurant. Humans love the lighting and the furniture. A model wants the menu. Clean markdown is that menu. No navigation chrome, no scripts, no repeated widgets burning tokens. Some WordPress tools, LLMagnet is one example, serve a markdown version at the page's permalink plus .md.

Ben cohen

I build a free tool that strips a page down to what AI sees, and people are often stunned. The hero section they paid for is empty text. Does adding llms.txt fix that?

Ido

No, and let's be clear. It's a plain markdown index: a title, a short summary, links with brief descriptions. Mintlify truncates descriptions at 300 characters. llms-full.txt bundles the full content. Useful for documentation sites, but it's an emerging convention, not a ranking switch. If your facts are thin or you block crawlers, a tidy map to nothing helps nobody.

Ben cohen

Same for schema, I'd guess.

Ido

Right, schema for product, organization, author, FAQ gives explicit context. But if the JSON-LD says one price and the page says another, you've damaged trust. Generic or auto generated schema drifts from the copy. Schema supports clear content. It can't replace it.

Ben cohen

Now the agent. A research agent reads. A buying agent has to click.

Ido

So a form that fails silently, or only works through client side scripts, breaks it. A site an agent understands but can't use is incomplete. There are early discovery files, like a well known mcp.json, but adoption is uneven. That's experimental. Clear labels and predictable actions are safe bets today.

Ben cohen

And measurement. LLMagnet, for instance, logs visits from GPTBot, ClaudeBot, PerplexityBot and audits agent readiness, alongside Rank Math or Yoast. Useful, but a log tells you who visited, not whether the answer was accurate. You can check that manually.

Ido

Which is the action for this week. Ask several AI systems your top customer questions, including a comparison with a competitor. Do they state your facts correctly? Who do they cite?

Ben cohen

Visibility isn't just getting visits anymore. It's making sure machines can find, understand, trust, and eventually use your brand correctly. If this helped, follow Be Visible.

Ido

And go check what the models say about you.