Short answer · 2 min read

SEO

8 Common Robots.txt Issues That Hide a Site from Google

In this article 4
  1. The eight issues and their fixes
  2. How to check yours in ten minutes
  3. When not to hire us
  4. Sources

One wrong line in a robots.txt file can take a whole site out of Google. The most expensive of the common robots.txt issues is the oldest: a site launches with the staging rule Disallow: / still in place, and nobody notices until the enquiries stop. Here are eight issues and the fix for each.

A robots.txt file with two lines, User-agent: * and Disallow: /, and the note: one line, and the whole site is closed to search engines.

The eight issues and their fixes

IssueFix
The staging rule Disallow: / survived the launchRemove it on launch day, then open your own /robots.txt and read it
The file is not at the root of the hostIt works only at /robots.txt, and each subdomain needs its own (Google, 2026)
Folders with CSS and JavaScript are blockedAllow them: Google loads those files to see the page as a visitor does
A page is “hidden” with DisallowBlocking is not removal. Use noindex or a password (Google Search Central, 2026)
Rules Google does not read, such as crawl-delayGoogle supports four fields: user-agent, allow, disallow, sitemap (Google, 2026)
Wrong letter case in a pathPaths are case-sensitive: /Blog/ and /blog/ are two different rules
No sitemap lineAdd Sitemap: with the full address of the sitemap
AI crawlers blocked without a decisionA site that opts out of OAI-SearchBot is not shown in ChatGPT search answers (OpenAI, 2026). Decide first, then write the rule

How to check yours in ten minutes

Type your address followed by /robots.txt and read what is there. If you see Disallow: / under User-agent: *, call your developer now. Then open the robots.txt report in Google Search Console: it shows the file Google last fetched.

Two statements side by side. Disallow stops the visit. Noindex stops the listing. A page you want out of search needs the second.

Sources

  1. Google Search Central, Introduction to robots.txt 2026
  2. Google, How Google interprets the robots.txt specification 2026
  3. OpenAI, Overview of OpenAI Crawlers 2026
Related page SEO and AI search visibility See the service page

More on SEO

More on
SEO

One question per article, with a date and an author.

All articlesEvery article on SEO, and the other topics.

All on SEO →

Contact

Ask the question the article did not answer.

Or send a note.

We reply within one business day. No newsletters, no spam.