Following this article by Sigt, it occurred to me to talk about what a robots.txt should be like for a Wordpress blog, analyzing a little what those of some of the most successful blogs that I read regularly are like.
By creating a robots.txt file, we are telling the different search engine bots which files or paths they should follow and index and which ones they should not. For example, if we have a folder with private information about us, and we are not interested in it being accessible through search engines, we must disable it (“Disallow”) in the robots.txt file, since by default the search engines search the entire site. In Wordpress, for example, the wp-admin or wp-includes folders only contain information about the administration panel and the internal workings of the CMS, so they could be perfectly “disabled” for indexing, as they have done in AlexSeo as follows:
Disallow: /wp-admin/
Disallow: /wp-includes/In Sigt they go one step further, and come to the conclusion that all files or folders that begin with “wp-”, the problem with this is that by doing so you are also vetoing the entry to wp-content, which could contain elements that we want to be indexed, such as images. To do this we have to allow (“Allow”) the indexing of files of a certain type from certain folders (In this case uploads):
Allow: /wp-content/uploads/*.gif
Allow: /wp-content/uploads/*.png
Allow: /wp-content/uploads/*.jpgSigt’s method is much more restrictive than AlexSeo’s, although the choice of one or the other depends on what the blogger considers most comfortable.
You may want the Google search engine to index your images (In LaMatePorUnYogur they have indexing disabled, while in other sites they have it activated, so it depends on each one), so it would be recommended that you add a line to allow entry to the Google Images bot:
User-agent: Googlebot-Image
Allow: /I don’t know to what extent it is necessary or efficient to add a couple of lines for Google Images if you already have “User-agent: *” set (which works for all search engine bots), so if someone can explain this point to me even better.
WordPress has a fairly significant problem with duplicate content, since by default all feeds and trackbacks are indexed. To avoid this, you have to add a few lines to robots.txt, which practically all blogs (Inkilino, LaMatePorUnYogur or Sigt) have in common:
Disallow: /?s=
Disallow: /?ref=
Disallow: /feed
Disallow: /comments/feed
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/*/feed/$
Disallow: /*/*/feed/rss/$
Disallow: /*/*/trackback/$
Disallow: /*/*/*/feed/$
Disallow: /*/*/*/feed/rss/$
Disallow: /*/*/*/trackback/$Lastly, but not less important than the above, is to facilitate access to the Sitemap of our blog (We should have one, yes), indicating the file path as follows:
Sitemap: https://bocabit.com/sitemap.xml
In this way, putting all of the above together, the final result that should work for everyone could be like the following:
User-agent: *
Disallow: /wp-
Allow: /wp-content/uploads/
Disallow: /?s=
Disallow: /?ref=
Disallow: /feed
Disallow: /comments/feed
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/*/feed/$
Disallow: /*/*/feed/rss/$
Disallow: /*/*/trackback/$
Disallow: /*/*/*/feed/$
Disallow: /*/*/*/feed/rss/$
Disallow: /*/*/*/trackback/$
User-agent: Googlebot-Image
Allow: /Sitemap: https://tuSitio.com/sitemap.xml
Something that has caught my attention is that some sites like Inkilino have added a couple of lines to allow the Adsense bot to enter the entire site:
User-agent: Mediapartners-Google Allow: /
And that’s it for today’s lesson, although I still have some doubts, such as whether the Allow tags are more important than the Disallow tags in the sense that they override some values of the latter, and I would appreciate it if I am wrong about something, to be corrected ;)
PSTA: I’m surprised that Helektron doesn’t have a robots.txt ;)
EDITED: If we want our blog to appear in Google Blog Search, we must add the following line, but in this case we must take into account that Google could consider it as duplicate content, so it would be advisable to add it a couple of months after the blog already has a certain “prestige” so that it has a higher PageRank than the entry feed:
Allow: /feed/$