Better Robots.txt & .htaccess Rules for WordPress

My wife and I run several WordPress powered websites and try to stay on top of things on all things WordPress. But, one place we heavily rely on another person’s expertise is with robots.txt and .htaccess rules.

Sure, we can handle the simple tasks with those files. But, when someone like Jeff Starr, author of Digging into WordPress (aff link) is putting his time and energy into creating some comprehensive rules, why not use it.


Plus, he is kind enough to share it with the community.

Robots.txt for WordPress

You can direct search engines to index directories and files you want and ask it to ignore sections you don’t want it to. It is a good practice to do this regularly so you avoid some of the pitfalls in not doing so.

Check out the following post for a nice writeup on this topic and you can use that as a starting point in creating your own.

Better Robots.txt Rules for WordPress

.htaccess for WordPress

If you use pretty permalinks in WordPress, you already have a .htaccess file. Besides that you can also use .htacess to blacklist spiders and spam bots from hitting your website.

Jeff has put together a nice .htaccess blacklist that he is running on his site at the moment.

5G Firewall beta

I am working on using these files as a startup list and crafting my own robots.txt and .htaccess files. You can probably tell I have in my RSS feeds. 🙂

Explore Tags: ,

One Response to “Better Robots.txt & .htaccess Rules for WordPress”

  1. dissertation uk July 26, 2011 at 11:09 am #

    Thanks for info