Robots.txt is a plain text file that tells search engine crawlers which parts of your website they can and cannot crawl. It sits at the root of your domain. Use it to guide crawlers, not to hide private pages, because it doesn't guarantee anything stays out of Google.
What robots.txt actually does
Robots.txt is one of those boring technical files that only gets attention after it breaks something. One wrong line can stop Google crawling your website. That means your lovely new service pages, product pages or blog posts might as well be locked in a cupboard.
When a search engine bot visits your site, it normally checks robots.txt first. The file gives crawl instructions, such as which folders to avoid or where your sitemap lives. Good bots, like Googlebot and Bingbot, usually follow those rules. Dodgy bots can ignore them completely.
That last bit matters. Robots.txt is not a security tool. It is a polite sign on a door, not a lock. If you put sensitive URLs in there, you may actually make them easier for nosey people to find because the file is public.
Google explains robots.txt in its Search Central documentation, and the Robots Exclusion Protocol is also covered in RFC 9309. You don't need to read both unless you enjoy punishment. You just need to use the file carefully.
Where robots.txt lives and what it looks like
Your robots.txt file should live at the root of your website. For example, if your site is www.yourdomain.co.uk, the file should be available at www.yourdomain.co.uk/robots.txt.
Not in a random folder. Not uploaded to your media library. Not buried inside your WordPress theme. The root.
A very simple robots.txt file might look like this:
User-agent: *
Disallow:
Sitemap: https://www.yourdomain.co.uk/sitemap.xml
That says all crawlers can crawl the site, and it gives them the sitemap location.
A more typical WordPress version might look like this:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.yourdomain.co.uk/sitemap_index.xml
That blocks the WordPress admin area while still allowing the admin-ajax.php file that some themes and plugins need. Boring? Yes. Useful? Also yes.
Robots.txt is case-sensitive in URL paths. /Private/ and /private/ are not always the same thing. This is exactly the kind of tiny detail that causes big, stupid SEO problems.
The rules that actually matter
Most small business websites only need a handful of robots.txt rules. You do not need a 400-line monster copied from a forum in 2016. That is how websites end up accidentally telling Google to bugger off.
Here are the main bits you need to understand:
| Directive | What it does | Example |
|---|---|---|
User-agent |
Names the crawler the rule applies to | User-agent: Googlebot |
* |
Applies the rule to all crawlers | User-agent: * |
Disallow |
Tells crawlers not to crawl a path | Disallow: /checkout/ |
Allow |
Lets crawlers crawl a path inside a blocked area | Allow: /wp-admin/admin-ajax.php |
Sitemap |
Points crawlers to your XML sitemap | Sitemap: https://www.yourdomain.co.uk/sitemap.xml |
The most common setup is User-agent: *, which means the following rules apply to all bots.
A blank Disallow: means nothing is blocked. Disallow: / means everything is blocked. That tiny slash is the difference between business as usual and Google being told to ignore your whole website.
Some crawlers support extra rules like Crawl-delay, but Google does not use it. If your server is struggling with crawling, fix the server or crawl settings properly. Don't chuck random directives into robots.txt and hope the internet behaves.
Robots.txt controls crawling, not indexing
This is the bit that catches people out.
Robots.txt tells search engines whether they can crawl a URL. It does not directly tell them whether they can index it. If Google discovers a blocked URL from another website, an internal link or an old sitemap, it may still show the URL in search results without crawling the page content.
That means robots.txt is the wrong tool for removing a page from Google.
If you want a page out of the index, use a proper noindex tag or an X-Robots-Tag header. But here's the annoying part: Google needs to be allowed to crawl the page to see that noindex instruction. If you block it in robots.txt, Google may never see the removal instruction.
For private pages, use password protection, logins or server-level restrictions. Don't rely on robots.txt for anything confidential. If you wouldn't print the URL on a flyer and leave it in a café, don't put it in a public robots.txt file and pretend it's secure.
When a small business should use robots.txt
Most small business websites need robots.txt for tidying up crawler behaviour, not for trying to outsmart Google. If your site has 12 pages, there is no need to start acting like you are running Amazon.
Useful reasons to use robots.txt include:
- Blocking admin areas that search engines do not need to crawl
- Stopping internal search result pages being crawled
- Keeping staging or test areas away from crawlers, although password protection is better
- Pointing search engines to your XML sitemap
- Reducing wasted crawling on duplicate filter pages in larger sites
For local businesses, the priority is usually simple: make sure Google can crawl your location pages, service pages, reviews, contact details and anything else that helps people find and trust you. If you are trying to rank across Cheshire or nearby towns, robots.txt should not be getting in the way of your local SEO work.
If leads have dropped after a redesign, robots.txt is one of the first files worth checking. It takes seconds to inspect and can save weeks of guessing.
What not to block if you care about rankings
Do not block important pages, CSS files, JavaScript files, images used on key pages or anything Google needs to understand how your site works.
Years ago, some people blocked /wp-content/, /wp-includes/, theme folders, plugin folders and scripts because they thought it made the site cleaner. It usually made Google see a broken version of the page. If Google cannot render your layout, menus, images or interactive elements properly, it may struggle to understand the page.
For WordPress sites, be careful with broad folder blocks. Blocking /wp-admin/ is normal. Blocking half your website's resources because some SEO plugin spat out a scary recommendation is not.
Also avoid blocking pages just because they are weak. If a page is thin, duplicate or useless, fix it, merge it, redirect it or noindex it properly. Robots.txt is not a skip bin for poor content decisions.
This is where technical SEO earns its keep. A robots.txt file is only one part of the technical setup. It sits alongside indexing rules, redirects, canonicals, sitemaps, page speed, mobile rendering and all the other stuff that quietly decides whether Google can work with your site.
Robots.txt and sitemaps should work together
Robots.txt and your XML sitemap are different things, but they often sit next to each other in the crawl process.
Your sitemap tells search engines which URLs you want them to find and consider. Robots.txt tells crawlers where they are allowed to go. If your sitemap lists a page but robots.txt blocks that page, you are giving Google mixed signals. That is not clever SEO. It is just messy.
A good sitemap should include the pages you actually want indexed: service pages, product pages, key category pages, useful guides and main business information. It should not be stuffed with junk URLs, old redirects, test pages, tags, internal search results or admin areas.
The robots.txt file can include one or more Sitemap: lines. Use the full sitemap URL. If you use WordPress with an SEO plugin, the sitemap might be called sitemap_index.xml rather than sitemap.xml.
If you recently launched a new website and nothing is happening in Google, check the sitemap and robots.txt together. A new design means very little if crawlers are being sent mixed instructions behind the scenes.
How to check your robots.txt without breaking things
You do not need to be a developer to do a basic robots.txt check. You do need to be calm and not start editing things while panicking because your competitor is above you on Google.
Use this simple process:
- Open your browser and visit
yourdomain.co.uk/robots.txt. - Check whether the file loads or returns a 404.
- Look for
Disallow: /, especially if your whole site has vanished from search. - Check whether important folders, services or product URLs are blocked.
- Make sure your sitemap URL is present and loads correctly.
- Use Google Search Console's URL Inspection tool to test any page you are worried about.
If Search Console says a URL is blocked by robots.txt, do not ignore it. Find the matching rule and work out whether the block is deliberate.
Also check after a website migration or redesign. Developers sometimes block staging sites with Disallow: /, which is fine while building. The problem starts when that same file goes live on the real domain. I have seen it happen. More than once. It is never a fun phone call.
Before changing anything, copy the current robots.txt file and save it somewhere. That way, if your edit makes things worse, you can put it back quickly instead of trying to remember what you broke.
Common robots.txt mistakes I keep seeing
The biggest robots.txt mistake is blocking the whole site by accident. It usually looks like this:
User-agent: *
Disallow: /
That line has its uses on a staging site, but on your live business website it is a disaster. Google is being told not to crawl anything.
Another common mistake is trying to use robots.txt as a privacy shield. If you list /private-client-documents/ in robots.txt, you have just told every crawler and every curious human where that folder might be. Use proper access control instead.
People also block important resources by being too aggressive. A line like Disallow: /wp-content/ might block images, theme files and other assets Google needs to render your pages. The page may still exist, but Google gets a half-baked view of it.
Then there is the old Noindex: directive in robots.txt. Google no longer supports it as a robots.txt rule. If you want noindex, put it on the page or send it as a header.
And finally, watch for old rules left behind after redesigns. A folder that used to be useless might now contain your best landing pages. Robots.txt does not know that. It just follows the file.
Copy and adapt these robots.txt examples carefully
These examples are starting points, not magic spells. Your website may need something different depending on how it is built.
For a normal small business WordPress site:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://www.yourdomain.co.uk/sitemap_index.xml
For a site where everything should be crawlable:
User-agent: *
Disallow:
Sitemap: https://www.yourdomain.co.uk/sitemap.xml
For a staging site that should not be crawled:
User-agent: *
Disallow: /
But do not rely on that last one alone. A staging site should also be password protected. Robots.txt only speaks to crawlers that choose to listen.
For blocking internal search results:
User-agent: *
Disallow: /?s=
Disallow: /search/
Sitemap: https://www.yourdomain.co.uk/sitemap.xml
Be careful with parameter blocking. Some ecommerce sites use URL parameters for filters, sorting, tracking or product variations. Block the wrong thing and you can make important product pages harder to crawl.
If you are not sure, don't guess. Guessing at technical SEO is like rewiring your shop because you watched a bloke on YouTube do it in his garage. Sometimes it works. Sometimes smoke comes out.
Robots.txt and AI crawlers
Robots.txt is not just about Google anymore. AI crawlers may also check it, including crawlers used to collect information for AI tools and answer engines. Some obey robots.txt. Some may not. The file is still a useful place to state your crawling preferences.
You can target specific bots if you know their user-agent name. For example:
User-agent: GPTBot
Disallow: /
That tells GPTBot not to crawl the site, if it respects the rule. But blocking AI crawlers is a business decision, not a default setting. Some businesses want fewer crawlers using their content. Others want to be found and cited more often by AI search systems.
Robots.txt also does not tell AI tools which pages best explain your business. It only says what may be crawled. If you are thinking about AI search visibility, that is a different conversation involving content structure, entities, clear answers and crawlable authority signals. SEO Bridge covers this through AI, AEO and GEO services.
The short version: robots.txt can control access for some AI crawlers, but it does not magically make your business appear in AI answers.
When robots.txt needs professional attention
If you run a small brochure site, robots.txt is usually simple. If you run ecommerce, booking systems, member areas, faceted navigation, multiple locations or a site that has been through several rebuilds, it deserves proper attention.
Get help if you see sudden drops in impressions, crawling errors in Google Search Console, pages marked as blocked unexpectedly or a messy file full of rules nobody understands. Also get help before a migration, not three weeks after traffic has fallen off a cliff.
A good robots.txt setup will not win rankings by itself. It removes stupid obstacles. That is still valuable. SEO is often less about secret tricks and more about not making it hard for Google to read the bloody website.
If your file is clean, your sitemap is accurate and your important pages are crawlable, you have done the job properly. Then you can focus on the stuff that actually brings leads: useful pages, strong local signals, proper optimisation, credible links and content that answers real customer questions.
Frequently Asked Questions
What is robots.txt used for? Robots.txt is used to tell search engine crawlers which areas of a website they should or should not crawl. It is mainly used to block admin areas, reduce crawling of duplicate or low-value pages and point crawlers to the XML sitemap. It should not be used to hide private information.
Can robots.txt remove a page from Google? No. Robots.txt controls crawling, not indexing. A blocked URL can still appear in Google if it is discovered through links or other signals. To remove a page from the index, use a proper noindex tag, an X-Robots-Tag header or Google Search Console removal tools where appropriate.
Where should I put my robots.txt file? Your robots.txt file should sit at the root of your domain, such as yourdomain.co.uk/robots.txt. If it is placed in a subfolder, crawlers will not treat it as the site's main robots.txt file. Most CMS platforms and SEO plugins can create or edit it for you.
Is it bad if my website does not have robots.txt? Not always. If your site has no robots.txt file, search engines will usually assume they can crawl everything they can access. For very small websites, that may be fine. A robots.txt file becomes more useful when you need to block admin areas, manage duplicate URLs or list your sitemap.
What is the most dangerous robots.txt mistake? The most dangerous mistake is adding Disallow: / to a live website. That tells compliant crawlers not to crawl any pages on the site. It is commonly used on staging sites, then accidentally pushed live during a redesign. If rankings or traffic vanish after launch, check this first.
