How AI Crawlers Read Your Site: Clean Code, Schema, and Trust

Ben cohen

So 9 out of 10 B2B tech buyers are using tools like ChatGPT and Claude to do their purchase research now. 9 out of 10. And yet, I keep seeing these, these brand websites that go to check their robots dot txt file, and right there, plain as day, they are blocking GPTBot or ClaudeBot or PerplexityBot. They are just shutting the door on the exact AI engines their customers are using.

Ido

It, it is unbelievable, right? You check robots dot txt and you see disallow GPTBot. I mean, do NOT block GPTBot, Google Extended, ClaudeBot, or other AI crawlers. If you block them, you make yourself completely invisible to AI search. You're just, you're erased from the answer.

Ben cohen

Right, erased! And even, even when they do let the bots in, there is this whole other issue with how the pages are actually built. Like, most modern websites are built with all these nested div tags, heavy JavaScript frameworks, tracking pixels, header menus, endless footer links. When an AI crawler hits that, it has to parse all of that junk inside its context window. It's paying a massive token tax just to find your actual content.

Ido

Yeah, exactly. People forget that LLMs process tokens, not visual layouts. Every extra line of boilerplate HTML is work the model has to do. That's why tools like Salespeak have popped up with these LLM optimizers. They literally strip out the headers, the sidebars, the template markup, and serve a clean, streamlined text version to the AI bot. It drops the computational cost for the engine, and boom, your retrievability goes through the roof.

Ben cohen

It, it reminds me of an overworked manuscript editor, you know? Imagine handing an editor a book draft, but every single page is covered in fifty brightly colored sticky notes saying click here, check out our nav bar, look at our copyright info. The editor is going to lose their mind trying to read the actual story. The clean text version just gives them the manuscript straight up.

Ido

That is such a good analogy! And I actually saw this in a site audit just last month. We took a blog post that was buried under a heavy page builder, stripped out all the unnecessary layout bloat for the AI user agent, and within two weeks, brand citations in Perplexity jumped noticeably. When you stop wasting the model's token budget on your footer links, it actually reads what you wrote.

Ben cohen

So if clean code gets the crawler through the door, how do we make sure it actually understands what it's reading? Because LLMs are stochastic, right? They're working on probability, not static keyword indexes.

Ido

Right, and that's where schema markup comes in. Schema is basically the ID badge your content wears for AI. Without it, the model is guessing whether a string of text is a product name, an author, or a review. With structured data like Article schema, Person schema, or FAQPage schema, you are giving the engine explicit semantic labels. It doesn't have to guess or risk hallucinating.

Ben cohen

It's like putting a detailed nutritional facts label right on the front of a box of cereal. Instead of forcing the AI to analyze every ingredient in the recipe, you just put a clear label that says, hey, here is thirty grams of protein, here is who made it, here is when it was packaged.

Ido

Yes! That's a great way to put it. And when it comes to formatting, tactical Q and A structure is huge. You put a clear question in an H2 tag, and right below it, you write a concise 2 to 3 sentence direct summary, paired with an FAQ schema block. That gives the LLM a pre structured answer on a silver platter, ready to copy and paste right into a generated overview.

Ben cohen

2 to 3 sentences. That's so specific. But what about trust? Like, how does Perplexity or a Google AI Overview decide your answer is better than someone else's?

Ido

Freshness and authority signals. They look at the dateModified attribute in your Article schema to see if the content is maintained. And they check for Person schema linked to real author credentials, like a verified LinkedIn profile. Fresh dates plus verified humans equals high trust.

Ben cohen

Hmm. It makes me wonder, though. Are we moving toward a web where design doesn't even matter for humans anymore? Like, are we just building websites for machines now?

Ido

I mean, we are definitely marketing to machines first in order to reach humans second. If the AI doesn't pick you as a source, the human user never sees your link in the answer anyway. It's a shift, for sure.

Ben cohen

A huge shift. Clean up the robots file, add your schema ID badges, and keep your code tight. Good stuff today, Ido.

Ido

Yeah, fun talk, Ben. Catch you next time.