My wife and I run several WordPress powered websites and try to stay on top of things on all things WordPress. But, one place we heavily rely on another person’s expertise is with robots.txt and .htaccess rules.
Sure, we can handle the simple tasks with those files. But, when someone like Jeff Starr, author of Digging into WordPress (aff link) is putting his time and energy into creating some comprehensive rules, why not use it.
Plus, he is kind enough to share it with the community.
Robots.txt for WordPress
You can direct search engines to index directories and files you want and ask it to ignore sections you don’t want it to. It is a good practice to do this regularly so you avoid some of the pitfalls in not doing so.
Check out the following post for a nice writeup on this topic and you can use that as a starting point in creating your own.
.htaccess for WordPress
If you use pretty permalinks in WordPress, you already have a .htaccess file. Besides that you can also use .htacess to blacklist spiders and spam bots from hitting your website.
Jeff has put together a nice .htaccess blacklist that he is running on his site at the moment.
I am working on using these files as a startup list and crafting my own robots.txt and .htaccess files. You can probably tell I have digwp.com in my RSS feeds.