Why AI Search Ignores Your Site

Ben cohen

You check your site analytics, you are sitting at number two on Google for your main keyword, everything looks amazing. But then you open Perplexity or ChatGPT, you ask for the top recommended software in your exact space, and your brand is completely gone.

Ido

Yeah, total crickets. It is like your whole company just disappeared from the internet the second someone asks an AI instead of typing into a search bar.

Ben cohen

It, it actually happened to a B2B SaaS client I was working with recently. They were doing everything right for traditional SEO. Great backlinks, high search rankings. But on Perplexity? Nothing. Zero citations. And when we went under the hood to see why AI models were ignoring them, we found two massive roadblocks. First, their security plugin was aggressively blocking crawlers in their robots dot text file. Bots like GPTBot, ClaudeBot, and PerplexityBot were literally walled off from reading the site.

Ido

Wait, so the security plugin was trying to protect against scraper bots, but in doing that, it was basically locking the door on ChatGPT and Claude?

Ben cohen

It was telling the AI crawlers, go away, you are not allowed here. So if you go to your site dot com slash robots dot text right now, you have to verify that AI crawlers are not blocked. Blocking them makes you completely invisible to AI engines.

Ido

That is such a simple fix, but if you do not know to look for it, you are just left wondering why nobody is finding you through AI search.

Ben cohen

Right. But even after we opened the door for the crawlers, there was a second problem. Their content had no freshness signals. They were missing the dateModified schema attribute. Rank Math and Yoast can generate this, but if you do not explicitly pass that last modified date, the model assumes the information might be stale or out of date, so it skips over it when picking sources to cite.

Ido

It is like handing someone a menu, but there is no date on it, and the code around it is so cluttered they cannot even read the specials.

Ben cohen

That is a great way to put it. Think of heavy page builders like Elementor or Divi. They render beautiful pages for human visual browsers, but underneath, they create thousands of lines of unnecessary code bloat, like endless nested divs and template HTML. For an AI model reading tokens, that extra noise makes it much harder to extract core facts.

Ido

Right, because AI engines do not browse your site like a human visitor visually scrolling down a page. They scan for structured, extractable signals. If they have to wade through a mountain of visual styling code just to find the price or the key feature, they are just going to grab the answer from a site that gives it to them clean.

Ben cohen

Exactly. Schema markup is the ID badge your content wears for AI. Without schema, the model has to guess what your page is about. With it, the model knows instantly, this is an article, written by this specific author, updated on this date, answering this exact question.

Ido

So if classic rank tracking like keyword position is becoming obsolete because LLMs are generative and stochastic, how are marketers actually measuring their presence inside these models now?

Ben cohen

We have to shift to dedicated AI visibility platforms. I actually built a free tool called AI Content Reader at GetAISO to help people see what their pages look like to models, but for full tracking, platforms like LLMagnet help you understand how large language models read, rank, and connect with your content. LLMagnet tracks actual AI bot traffic, shows you which prompts are mentioning your brand, and gives you a unified visibility score across systems like ChatGPT, Gemini, Claude, and Perplexity.

Ido

So instead of tracking if you are rank three for a single keyword, you are tracking your actual retrievability and share of voice inside the AI model's answers.

Ben cohen

Precisely. And if someone wants a quick win right now, like a two minute change they can make on WordPress, what is the best entry point?

Ido

The absolute fastest win is structuring your posts with direct answer blocks right at the top. You put a clear question as your H2 heading, and then in the first two to three sentences immediately following it, you give a direct, self contained answer. Then you wrap that Q and A pair in FAQPage schema using a plugin like Rank Math.

Ben cohen

That format gives the AI model a quote on a silver platter. Pages with FAQ schema are significantly more likely to be cited in generative answers because the extraction work is already done for them.

Ido

Now, what about this whole trend around generating text files like llms dot txt and llms full dot txt? People talk about them like they are a magic silver bullet that guarantees you get cited.

Ben cohen

Yeah, there is a lot of hype there. Generating an llms dot txt file is great for guiding crawlers to your most important pages, and LLMagnet can automate that for you. But putting up a text file alone will not magically fix your visibility if your underlying content lacks authority and structure.

Ido

Meaning, if your site is published under a generic admin or webmaster username with no real author credentials, an llms dot txt file is not going to save you.

Ben cohen

Not at all. AI models evaluate who wrote the content, not just what was written. You need credentialed Person schema, real biographical details on your author profile, consistent links to external profiles like LinkedIn, and clear E E A T trust signals. The text file is just a roadmap; the actual content still needs to prove its credibility.

Ido

So the big takeaway is clear. Check your robots dot text, add clean schema markup with modification dates, format your content for easy extraction, and build real author authority. Do that, and you stop being invisible to the engines people are using to discover products today.

Ben cohen

Spot on. Good stuff today.

Ido

Yeah, talk soon.