How to edit robots.txt in WordPress

By default, WordPress does not have a robots.txt file. When requested, WordPress dynamically generates a response:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

This can be overridden by creating a real robots.txt file in the root WordPress folder. If the file exists, it will be used instead of the dynamically generated response.

How to edit robots.txt

If you have an SEO plugin, use it to create the file. SEO plugins tend to support robots.txt because the file can have a dramatic impact on search engine visibility. There are also other plugins that you can explore in the plugin directory. It is generally safer to use a plugin than to directly connect to the file system. But if you don’t have a suitable plugin and do not wish to install one, you can create and edit robots.txt through the file system.

Here are some plugin and non-plugin ways to edit robots.txt:

1️⃣ AIOSEO plugin

The AIOSEO plugin has a rule builder for editing robots.txt:

1. From the side menu, click All in One SEO > Tools.
2. You should already be on the editor. If not, click Robots.txt Editor near the top.
3. Ensure the Enable Custom Robots.txt toggle is on (blue).
4. Use the rule builder to make the desired changes.

For more information, see Using the Robots.txt Tool in All in One SEO – AIOSEO. If you are a multisite, see Robots.txt Editor for Multisite Networks – AIOSEO.

2️⃣ Yoast SEO plugin

The Yoast SEO plugin has a robots.txt editor:

1. From the side menu, click Yoast SEO > Tools
2. Click File editor
3. If you haven’t yet created the file, click Create robots.txt file
4. Make any changes in the text box
5. Click Save changes to robots.txt

For more information, see The robots.txt file in Yoast SEO • Yoast.

3️⃣ WP File Manager

WP File Manager is a popular file management plugin. You can use it to create the file.

1. From the side menu, select WP File Manager.
2. Go to the root WordPress folder if not already there.
3. If you see an existing robots.txt file, right-click and select Editor.
4. If not, click the New File button on the toolbar. It is the 5th icon from the left. You can also upload instead if you prefer to create on your computer first (6th icon from the left).
5. Save changes.
6. Confirm you can read the file from the Internet: double-click the file in WP File Manager to open in your browser. You can also go directly to yourdomain.com/robots.txt.

4️⃣ SFTP

You can use SFTP with a client like FileZilla to edit the file.

1. Connect to your site over SFTP with a client such as FileZilla. You will need to get SFTP credentials from your hosting provider.
2. Navigate to the root directory of your WordPress installation. For confirmation, your root folder also contains wp-config.php as well as sub-folders wp-content, wp-includes, etc. Your WordPress root folder may be deeper down than the root folder of your SFTP connection.
3. If you have robots.txt, your client should let you edit. In FileZilla, for example, you can right-click and select View/Edit.
4. Once saved, double check permissions. The file must be readable by the public but not writeable. A typical permission setting is 644, which means:
   – Owner has Read & Write
   – Group has Read
   – Public (Others) has Read.
5. Confirm the public can view the file by visiting yourdomain.com/robots.txt. If you get an error, make sure permissions are set properly and you are creating the file in the correct folder.

Reference

• The Web Robots Pages – technical information about the robots.txt file.

See also

• How to edit wp‑config.php with FileZilla – if you are using FileZilla, this guide provides detailed instructions for editing a file with FileZilla. You can use this guide but replace wp-config.php with robots.txt.