Blogger Website Gets "Indexed, though blocked by robots.txt blogger" Message

So has your blogger website get a message from Google saying "Indexed, though blocked by robots.txt blogger" message? I did, and thought I would share my experience resolving it.

-

The other day I got a message from Google saying that there is a coverage issue with my site, with the following message:

"Indexed, though blocked by robots.txt blogger"

Yet I don't have a robots.txt file present that I installed.

I did some searching around and this seems to be a new message that started showing up in late 2018, in conjunction with their new search console.  According to some, it's only a warning and not a problem, but I hate getting error messages about my site.

So I went to my Google Search Console, hit up 'Crawl,' then robots.txt Tester.

I fired it up and it came back saying this is my latest robots.txt file... despite my NOT having set one. This is what it saw:

"User-agent: Mediapartners-Google
Disallow:

User-agent: *
Disallow: /search
Allow: /
"

If I didn't know any better, I'd swear Google slipped in its own robots file, but I came away with one question, and that was what the heck is Mediapartners-Google?  Turns out that Mediapartners-Google is from Adsense

According to Adsense Help, to allow Adsense crawler, you need to remove the following two lines of text from your robots.txt file:

User-agent: Mediapartners-Google
Disallow: /

-

But as you can see, my file, the one that does not exist when I go to look for it, says I am not disallowing it.

But the next set of lines,

User-agent: *
Disallow: /search
Allow: /

Indicates that I am only blocking paths that start with /search, like

    http://example.com/search
    http://example.com/search.html
    http://example.com/searchengine
    http://example.com/search/
    http://example.com/search/index.html

-

In my case, when I look at my site's issue, there is an entire realm of posts under my site,

http://www..com/search/label/

So the question is, do I leave this block of text in robots.txt or remove it so that my labels (Blogger's version of tags) are also searched by search engines? On normal sites, tags sometimes show up in search engines. So I'm going to remove the Disallow comment in the robots.txt test page, and then set up a custom file via the Blogger control page, Settings, Search Preferences, Custom robots.txt.

{ support.google.com }

-

So in the end, it's a warning message, not an issue. So far.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -