Building Websites for Humans, AI, and Agents

Ben cohen

So if you go to a website today, who is it actually built for? Most people say, well, the customer, right? The human sitting at the screen.

Ido

Yeah, obvious answer. A human visual visitor.

Ben cohen

Right. But really, right now, we are designing for three totally different audiences at the exact same time. You have the human, who cares about pictures and styling. You have the machine interpreter, like ChatGPT or Claude or Gemini, trying to parse what the page even means. And now you have autonomous agents, software that actually takes actions like booking a demo or buying a product.

Ido

Wait, so, so three separate audiences on one single web page?

Ben cohen

Exactly. And the problem is, the way we have been building websites for humans actively breaks how machines read them.

Ido

Um, okay, walk me through that. Why does human design mess up the machine side?

Ben cohen

Think about a fancy restaurant. A human walks in and sees soft lighting, velvet chairs, beautiful wallpaper. That is your JavaScript, your CSS, your fancy video background. But an AI crawler is not there to enjoy the ambiance. It just wants the menu. It wants to know what do you serve, how much does it cost, and are you open on Tuesdays?

Ido

Right, right. So if it has to crawl through layers of velvet wallpaper just to find out if you serve pasta, it, it just gets lost or gives up.

Ben cohen

It literally runs out of token budget! If an LLM crawler has to parse five hundred lines of code bloat, like nesting divs inside divs from heavy page builders just to find your pricing table, it might drop or misinterpret your core value proposition before it even gets to the features.

Ido

Oh wow. Yeah, I see this all the time with WordPress setups. People stack up these drag and drop page builders like Elementor or Divi, and visually it looks slick, but under the hood it is just mountains of token heavy code wrappers. So when ClaudeBot or GPTBot comes by, it is like digging through a landfill to find a business card.

Ben cohen

That is such a good way to put it. A landfill to find a business card. And this is why people keep saying, oh, SEO is dead, SEO is dead. But it is not dead at all. It just split in two.

Ido

How do you mean split in two?

Ben cohen

Traditional SEO gets crawlers to your front door. It handles indexation. But Generative Engine Optimization, or GEO, determines whether an AI model actually understands your content enough to cite you, recommend your brand, and answer a buyer query with your name in it.

Ido

So traditional gets you found, but GEO gets you quoted in the answer.

Ben cohen

Precisely.

Ido

Okay, so if clean markdown and pure structured data is the clean menu for these models, how do we actually build that infrastructure on a normal website without stripping away all the visual stuff for human readers?

Ben cohen

It starts with schema markup. Think of schema as the official ID badge your content wears for AI. When you put schema for articles, or products, or Organization and FAQ schema on a page, you are explicitly labeling the data. You are telling the model, this question is answered by this three sentence paragraph, published by this verified author.

Ido

And FAQ schema especially is huge for this, right? Because it gives question and answer pairs on a silver platter.

Ben cohen

It really is the gold mine for generative engines. But here is the critical rule: your schema data must strictly match what is visible on the page. If you try to feed the AI hidden text or misleading schema to game the answers, models will pick up on the discrepancy and hit you with trust penalties.

Ido

Right, makes sense. What about these files people keep talking about, like llms dot text?

Ben cohen

Ah, llms dot text. So there is a bit of a myth going around that just dropping an llms dot text file on your root folder is some magic ranking bullet. It is not.

Ido

It is not?

Ben cohen

No. What it actually is, is a curated website directory for machines. Instead of forcing a crawler to wander through thousands of blog tags and visual pages, your llms dot text file points the AI straight to high yield documentation, pricing summaries, and technical specifications. It is like giving the waiter a short single page cheat sheet.

Ido

Ah, got it. So it guides them directly to the good stuff. But how do you even know if any of this is working? Like, how do you see if ChatGPT or Gemini is actually reading your site or citing you?

Ben cohen

Well, that is where tools like LLMagnet come in. LLMagnet helps you understand how large language models read, rank, and connect with your content. It tracks real AI bot traffic, like actual visits from ChatGPT, Gemini, and Claude, and gives you a single AI Visibility Score.

Ido

Wait, so you can actually see when a bot from Perplexity or OpenAI hits your specific pages?

Ben cohen

Yeah, you get visual traffic breakdowns, prompt tracking to see which user prompts mention your brand, and automated reporting. Plus, it checks if your robots dot text file is accidentally blocking crawlers like GPTBot or ClaudeBot. Because you would be shocked how many site owners block AI crawlers by default in their security settings and have no idea why they are invisible in generative search.

Ido

That is a massive blind spot. You think you are secure, but you just locked the door on the exact engines recommending products to buyers!

Ben cohen

Exactly. And as we move into autonomous buying agents, software agents that complete forms, make reservations, or execute purchases on behalf of users, this gets even more urgent. If your form fields are unlabelled or your buttons rely on messy scripts, an agent gives up and goes to your competitor who has agent ready code.

Ido

So clean code, explicit schema, curated machine text, and tracking your bot visibility. That is the baseline for surviving the machine web.

Ben cohen

That is the playbook. Make it easy for machines to read, and they will make it easy for customers to find you.

Ido

Love it. Time to go check my robots dot text file right now.