How to add items to robots.txt with WordPress

One thing that I love about my job is that most days I learn something new. Today I learned that WordPress auto-generates its own robots.txt file, and that there is a filter that you can hook into to allow you to add your own rules to the file.

Working with one of our partners (we often provide business-to-business support (b2b) at Make Do) I asked server access to add a quick rule to the robots.txt file to prevent PDFs from being indexed on a site we were working on. The client informed me that WordPress generates its own robots.txt, and that there is a hook to let you add your own rules.

I quickly took to Google, and found out about the robots_txt filter. Using this new found information I hand crafted this handy little snippet:

/**
 * Filter function used to disallow pdfs from robots.txt
 *
 * @param    string  $output     The output of the robots.txt file
 * @return   string  $public     If the site is public facing
 */
function disallow_pdf_index( $output, $public ) {

    if ( '1' === $public ) {
        $output .= "Disallow: *.pdf\n";
    }
    return $output;
}
add_filter( 'robots_txt', 'disallow_pdf_index', 0, 2 );

So there you go, a short, but handy little blog post detailing how you too can add entries to your robots.txt file using WordPress.

Posted by Matt Watson

Matt Watson is co-founder of the WordPress agency Make Do. Matt loves learning about personal, professional and web development. Learn more about Matt.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.