Robots.txt for WordPress MU
From LinuxReviews
This is a Robots Exclusion Standard (robots.txt) file which is ideal for WordPress MU.
| File: robots.txt |
# This rule means it applies to all user-agents
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: error_log
Disallow: index-install.php
Disallow: wp-activate.php
Disallow: wp-atom.php
Disallow: wp-blog-header.php
Disallow: wp-comments-post.php
Disallow: wp-commentsrss2.php
Disallow: wp-config.php
Disallow: wp-cron.php
Disallow: wp-feed.php
Disallow: wp-forum.phps
Disallow: wp-links-opml.php
Disallow: wp-login.php
Disallow: wp-mail.php
Disallow: wp-pass.php
Disallow: wp-rdf.php
Disallow: wp-rss.php
Disallow: wp-rss2.php
Disallow: wp-settings.php
Disallow: wp-trackback.php
Disallow: wpmu-cleanup.php
Disallow: wpmu-settings.php
Disallow: xmlrpc.php
== Alternatively ==
{{Config file|robots.txt|
<pre>
# Disallow all files ending with these extensions
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.tar$
Disallow: /*.tgz$
Disallow: /*.cgi$
Disallow: /*.xhtml$
|
[edit] Special bots
Add this if you site uses Google Adsense:
| File: robots.txt |
# This is the ad bot for google User-agent: Mediapartners-Google* # Allow Everything Disallow: |
[edit] See also
- Robots.txt for WordPress MU