Let\u2019s get practical and build a solid file you\u2019ll feel confident deploying.<\/p>\n
<\/p>\n
![\"ar](\"https:\/\/webzilla.global\/nz\/wp-content\/uploads\/sites\/2\/2025\/09\/output-26.png\")
<\/p>\n
<\/p>\n
What robots.txt actually controls<\/h2>\n
Robots.txt implements the Robots Exclusion Protocol. Polite crawlers fetch this file first, then decide where they\u2019re allowed to crawl. The file does not enforce security. It signals intent. Good bots follow it. Bad bots might ignore it.<\/p>\n
Key ideas to keep in mind:<\/p>\n
- \n
- It controls crawling, not indexing. A disallowed URL can still appear in search if other sites link to it. Search engines might list the URL without a snippet because they never crawled the content.<\/li>\n
- It\u2019s public. Anyone can view https:\/\/yoursite.com\/robots.txt<\/a>, so do not list sensitive paths you would rather keep private.<\/li>\n
- It\u2019s a request, not a rulebook. For anything truly private, use authentication or restrict access at the server or application level.<\/li>\n<\/ul>\n
<\/p>\n
<\/p>\n
When does a robots.txt file help?<\/h2>\n
- \n
- You want to reduce crawl load from low\u2011value areas like internal search, filters, cart, login, or paginated duplicates<\/li>\n
- You\u2019re wrangling faceted navigation and parameter chaos on a large ecommerce site<\/li>\n
- You run documentation or blog archives and want bots to focus on fresh, canonical pages<\/li>\n
- You want to communicate your sitemap location to help discovery<\/li>\n<\/ul>\n
<\/p>\n
<\/p>\n
Crawling vs indexing: which tool to use and when<\/h2>\n
Use robots.txt for crawl control. Use meta robots tags or X\u2011Robots\u2011Tag for index control. They\u2019re complementary.<\/p>\n
\n\n
\n Aspect<\/th>\n robots.txt<\/th>\n Meta robots tag<\/th>\n X\u2011Robots\u2011Tag (HTTP header)<\/th>\n<\/tr>\n \n Where it lives<\/td>\n \/robots.txt at the site root<\/td>\n In the of an HTML page<\/td>\n In the HTTP response headers<\/td>\n<\/tr>\n \n What it controls<\/td>\n Crawling access to paths<\/td>\n Indexing and link following on that page<\/td>\n Indexing and link following for any resource type<\/td>\n<\/tr>\n \n When it applies<\/td>\n Before a bot fetches a URL<\/td>\n After the bot fetches the page<\/td>\n When the URL is fetched<\/td>\n<\/tr>\n \n Scope<\/td>\n Global or directory patterns<\/td>\n Page level<\/td>\n Per URL, and easy to apply to whole file types<\/td>\n<\/tr>\n \n Good for<\/td>\n Blocking low\u2011value sections from crawl<\/td>\n Keeping individual pages out of search results<\/td>\n No\u2011indexing PDFs, images, or other non\u2011HTML files<\/td>\n<\/tr>\n \n Limitation<\/td>\n Advisory only; blocked pages might still appear by URL<\/td>\n Bot must crawl the page to see the tag<\/td>\n Bot must fetch the resource to see the header<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Two practical rules:<\/p>\n
- \n
- If you need a page kept out of search results, use noindex. That page must be crawlable for the tag or header to be seen, so do not disallow it in robots.txt.<\/li>\n
- If you want to reduce crawl on an entire section that offers little search value, disallow it in robots.txt.<\/li>\n<\/ul>\n
<\/p>\n
<\/p>\n
Before you write the file<\/h2>\n
Start with a quick inventory.<\/p>\n
- \n
- List content that must remain crawlable: key pages, CSS, JavaScript, images used in templates, APIs that render content, sitemap files<\/li>\n
- List content that wastes crawl capacity or clutters reports: cart, login, checkout, thank\u2011you pages, internal search, filtered URLs, staging endpoints<\/li>\n<\/ul>\n
Avoid blocking required resources. Google needs CSS and JS to render and evaluate your pages. If crawlers can\u2019t fetch them, mobile and structured data assessments can go sideways. When in doubt, test with the URL Inspection tool in Search Console after you deploy your file.<\/p>\n
<\/p>\n
<\/p>\n
Step\u2011by\u2011step: create a robots.txt that works<\/h2>\n
1. Decide what to allow and what to block<\/h3>\n
A simple starting set for many sites:<\/p>\n
- \n
- Allow everything by default<\/li>\n
- Disallow low\u2011value paths: \/cart\/, \/checkout\/, \/login\/, \/search\/, \/thank\u2011you\/<\/li>\n
- Keep assets open: \/css\/, \/js\/, \/assets\/, \/api\/<\/li>\n<\/ul>\n
2. Target all crawlers or specific ones<\/h3>\n
You can write a group that applies to all bots, or tailor a group to a named user\u2011agent. For example, you might restrict a less important crawler more aggressively than Googlebot or Bingbot.<\/p>\n
3. Write the directives in plain text<\/h3>\n
Use a basic editor like Notepad on Windows or TextEdit in plain text mode on Mac. Save the file as robots.txt. Do not use a word processor.<\/p>\n
Structure:<\/p>\n
- \n
- Start a group with User-agent<\/li>\n
- Add Disallow and Allow rules for that group<\/li>\n
- Use blank lines to separate groups<\/li>\n
- Add your sitemap URL at the end<\/li>\n<\/ul>\n
Example:<\/p>\n
Tips that prevent common mistakes:<\/p>\n
- \n
- A blank Disallow line means everything is allowed for that group<\/li>\n
- Specific rules win over broader ones<\/li>\n
- Comments begin with # and are ignored by crawlers, which makes them perfect for documenting intent<\/li>\n<\/ul>\n
4. Upload the file to your root directory<\/h3>\n
Place robots.txt at the top level of the domain. The final URL must be exactly https:\/\/www.example.co.nz\/robots.txt<\/a>. Use your hosting file manager, an FTP client, or your CMS.<\/p>\n
WordPress users can manage robots.txt with Yoast SEO, Rank Math, or by placing a physical file on the server. If you use a plugin interface, ensure it writes to the actual root.<\/p>\n
5. Validate and test immediately<\/h3>\n
- It\u2019s a request, not a rulebook. For anything truly private, use authentication or restrict access at the server or application level.<\/li>\n<\/ul>\n
![\"Example](\"https:\/\/webzilla.global\/nz\/wp-content\/uploads\/sites\/2\/2025\/09\/1cc9c0cf159382dcba25e0023734d806.png\")
<\/p>\n![\"Robots.txt](\"https:\/\/webzilla.global\/nz\/wp-content\/uploads\/sites\/2\/2025\/09\/2f837cc6d09a109e32694b2402ba2671.png\")
<\/p>\n![\"Robots.txt](\"https:\/\/webzilla.global\/nz\/wp-content\/uploads\/sites\/2\/2025\/09\/ba1bf7d10fd1715886265ce99468071b.png\")
<\/p>\n