Block directories using robots.txt

Restrict search engines from crawling to the root directory by adding the code snippet below to robots.txt file.

User-agent: *

Disallow: /cgi-bin/

Disallow: /wp-admin/

Disallow: /wp-includes/

Disallow: /xmlrpc.php

Disallow: /wp-content/plugins/

Disallow: /wp-content/cache/

Disallow: /wp-content/themes/

Disallow: /trackback/

Disallow: /feed/

Disallow: /comments/

Disallow: /category/

Disallow: /trackback/

Disallow: /feed/

Disallow: /comments/

Disallow: /*?

Allow: /wp-content/uploads/

Leave a comment

Your email address will not be published. Required fields are marked *

%d bloggers like this: