Robots.txt is a text file that sets out crawling rules for search engine crawlers. It specifies which pages and sections they may crawl and which they should skip. The file is available at an address such as https://example.com/robots.txt and may include a link to an XML sitemap.
For example, an online store has categories, products, internal search results, and sorting options. Configuring robots.txt helps limit unnecessary crawling while keeping useful pages accessible.
However, blocking crawling is not the same as removing a page from Google. To configure the file correctly, you need to understand when to use Disallow and when to use noindex or server-side access restrictions.
Below, we will look at where to find robots.txt, how to read it, how to write rules, and how to check their effect on specific pages.
What is robots.txt used for?
A search engine crawler visits pages, downloads their content, and passes the data to the search engine for processing. This is crawling. Indexing is a separate process that follows: the search engine analyzes the document and decides whether to add it to its database. Allowing crawling does not, by itself, guarantee that a page will appear in search results.
Robots.txt controls crawler access to URLs. For example, you can restrict crawling of a technical section that contains no useful content for search. Google describes the file as a tool for managing crawling, including reducing server load. Read more about the purpose of robots.txt.
Crawling rules are particularly useful when the same products are available with different sorting options, display modes, and tracking parameters in their URLs. First, determine which variations serve no purpose in search and which are standalone landing pages.
For a small service website, the task is usually simpler: check that the homepage, service pages, and articles are accessible. Configuring the file does not replace work on site structure and content. If the site is not attracting visitors, other causes need checking too. These are covered in the article on why a website may not be visible in Google.
Where is robots.txt located, and how can you view it?
To find robots.txt, add /robots.txt after the domain in your browser: https://example.com/robots.txt. If the file is accessible, you will see its rules as plain text. You can view another website’s robots.txt in the same way without logging in to its admin panel.
The file belongs at the root of the site. The address https://example.com/blog/robots.txt does not replace the main robots.txt file. If the blog is in the /blog/ directory, its crawling rules are controlled by the file at https://example.com/robots.txt.
Rules are checked separately for subdomains: shop.example.com and example.com are different hosts. The protocol matters too: a file served over HTTPS does not automatically set rules for HTTP. Location requirements are explained in Google’s guide to creating robots.txt.
There may be no separate file in your hosting account: some content management systems, such as WordPress, generate the response automatically. Before editing, determine whether the current rules come from a physical file, the CMS, or an SEO plugin.
You can view robots.txt by adding /robots.txt after the site’s domain
How robots.txt works: directives and special characters
A robots.txt file is organized into groups. Each group first identifies the crawler on a User-agent line, followed by restrictions and permissions. Each instruction goes on a separate line. The table shows the main fields and example lines used to build the file.
Directive
Purpose
Example
User-agent
Identifies the crawler the group applies to
User-agent: Googlebot
Disallow
Blocks crawling of matching URLs
Disallow: /search/
Allow
Allows crawling of matching URLs, including those within a blocked section
Allow: /search/help/
Sitemap
Specifies the full URL of an XML sitemap or sitemap index
Sitemap: https://example.com/sitemap.xml
The value User-agent: * defines the general group. If there is a separate group for Googlebot, it selects that group; the general rules are not added to it automatically. A separate Google group is therefore only needed when its settings differ. How Google selects rule groups.
Paths are case-sensitive, and individual characters matter: /Catalog/ and /catalog/ are different. The rule Disallow: /catalog also covers /catalog-new/, while Disallow: /catalog/ only covers the path with the trailing slash and URLs beneath it.
Character
Meaning
Example
/
Separates parts of a path; in Disallow: /, the restriction covers the whole site
Disallow: /search/
*
Matches any sequence of characters in a path pattern, including an empty sequence
Disallow: /reports/*.pdf$
$
Marks the end of the URL being matched
Disallow: /search/$
#
Starts a comment that the crawler ignores
# Internal search
For example, Disallow: /search/$ blocks crawling of a URL ending in /search/, but not /search/help/ or /search/?q=phone. When rules conflict, the most specific matching rule applies; if Allow and Disallow are equally specific, Google allows crawling. A line’s position in the file does not determine its priority. How rule matching works.
Older templates may include Host, Crawl-delay, and Clean-param. Google does not support these fields.
A more specific Allow rule lets crawlers access the required subsection within a blocked section
Which pages can you block in robots.txt?
First, define the goal: restrict crawling of a URL, remove a document from search results, or protect access to information. Applying the same rule to every utility page and similar URL can block content that is useful for search.
For example, an online store may have a “running shoes” filter. If it has a useful landing page and there is search demand for the topic, it should not be blocked along with every other filter. However, numerous combinations of sorting options may need crawl restrictions. For large websites, Google recommends managing URLs that should not consume crawling resources. Guidance on managing crawling.
Page or resource type
What to check before restricting crawling
Internal search results
Whether they should appear in search results and whether any URLs are already indexed
Sorting options and technical parameters
Whether the content changes and whether useful pages match the pattern
Catalog filters
Whether there is search demand and a separate SEO strategy for these URLs
Cart, comparison, and checkout pages
Whether pages already known to the search engine should first be excluded using noindex
Account pages and confidential files
Whether authentication or another server-side access restriction is in place
CSS, JavaScript, and images
Whether the resources are needed to render and understand pages correctly
To exclude an accessible HTML page from Google, you can use <meta name="robots" content="noindex">. However, the crawler must be able to read this instruction. If you also block the page in robots.txt, it cannot download the HTML and discover noindex. Google also does not support a Noindex: entry within robots.txt itself. How to block indexing.
Passwords, backups, and personal data need to be protected through server controls or authentication. Listing their paths in robots.txt is not enough: visitors can read the file, and not all bots follow its rules. This also applies to a staging site containing non-public data.
How to create and configure robots.txt: examples
Save a copy of the current robots.txt and collect example URLs that should remain accessible or, conversely, should be blocked. You will need them to check the rules after making changes.
Create the file in a plain text editor and save it with UTF-8 encoding under the name robots.txt. A Word document or a file named robots.txt.txt will not work. If the rules are generated by a CMS or plugin, use its designated editing feature.
Example 1. A file with no crawl restrictions. Here, the general group imposes no restrictions, and the Sitemap line specifies the sitemap location:
An empty Disallow: value does not block crawling. By contrast, Disallow: / blocks all URLs for crawlers that apply this group unless a matching rule allows access. A single character fundamentally changes the meaning.
Example 2. Blocking crawling of a specific section. Suppose that, after reviewing the site structure, you decide to restrict crawling of all URLs beginning with /search/:
The rule covers /search/results/ and /search/?q=phone, but not /?s=phone, which has a different path. This is why templates found online need to be checked against your own site’s URLs.
Example 3. An exception within a blocked section. If /search/help/ contains public help content that should remain accessible to crawlers, add a more specific permission:
The rule now allows crawling of /search/help/ and URLs beginning with that path, while the rest of /search/ remains blocked. More ways to use groups and restrictions are shown in Google’s examples.
In each example, replace example.com with your domain and enter the actual sitemap URL, which may differ from /sitemap.xml. After saving, publish the file at the site root or apply the CMS settings, then open /robots.txt in your browser. Check the published result, not just the contents of the editor.
How to check robots.txt
Open /robots.txt in a private browser window without logging in. Compare the published rules with your saved version: the CMS, a plugin, or caching may cause differences.
Next, check the HTTP response status and how the rules apply. A successful response serving the file directly should return 200 OK. A 404 response means the file is absent and does not, by itself, prevent Google from crawling the site. Server errors in the 5xx range are handled differently and may pause crawling. How Google handles robots.txt responses.
To check specific pages, prepare a small set of test URLs in advance:
The homepage and a priority service or category page.
A product or article that should appear in search results.
A URL from the section you have blocked.
A URL that should remain accessible because of an Allow rule.
Similar URLs with parameters, different capitalization, or a different path ending.
In Google Search Console, use the robots.txt report. It shows whether the file was fetched, when it was last requested, any issues found, and the contents of the fetched version. For an individual page, use the URL Inspection tool and run a live URL test to find out whether crawling is allowed. Help with the robots.txt report.
The Search Console report shows when Google fetched robots.txt and whether it found any issues
Check separately whether Google is allowed to crawl an important page
After a change, the search engine needs time to fetch the new version. In an emergency, the report lets you request a recrawl of the file. This does not guarantee that unblocked pages will be crawled and indexed immediately.
Common robots.txt configuration mistakes
Syntax validation identifies errors in how rules are written, but it does not determine which pages should be blocked. A technically valid file can still prevent important sections from being crawled.
For example, a site-wide crawl restriction may remain after a staging site goes live. An overly broad filter pattern can also block existing landing pages. Check both the rule itself and its effect.
Mistake
Possible consequence
How to fix it
Leaving the site-wide Disallow: / restriction in place
Crawlers that apply this group cannot crawl important URLs without matching exceptions
Remove the unintended restriction and check key pages
Putting several paths on one line
For example, /cart/ /search/ does not create two separate restrictions
Write a separate Disallow line for each path
Using the wrong path or capitalization
The restriction applies to the wrong URLs
Compare the rule with the actual URLs
Blocking required CSS or JavaScript
The crawler has difficulty processing the page correctly
Restore access to the required resources
Expecting an HTML page to be removed after adding Disallow
A URL known to the search engine may remain in search results
Choose a method of excluding the page from search that matches the goal
Combining a crawl restriction with noindex
The crawler cannot read the page’s indexing restriction
Allow access so the crawler can process noindex
Copying another site’s entire file
Some rules are unnecessary, while others may block useful content
Review every path and group
Specifying a sitemap that does not exist
The Sitemap line points to an unavailable document
Check the full sitemap URL and its contents
Before publishing, record which rule was added, why it is needed, and which URLs were checked. This helps you trace the reasoning behind changes, especially when several people edit the file.
If it is unclear why pages are missing from search, also check noindex, canonical URLs, server responses, and internal links. This analysis is part of a website SEO audit: I review restrictions and prepare recommendations. Learn more in the article on what an SEO audit includes.
Frequently asked questions about robots.txt
What happens if a site has no robots.txt file?
The absence of the file does not prevent Google from crawling the site. If /robots.txt returns 404, there are no restrictions from that file. Whether pages are indexed still depends on their accessibility, content, and other settings.
Does a small website or landing page need robots.txt?
If all public pages can be crawled and no special restrictions are needed, a separate set of blocking rules may be unnecessary. Google explicitly states that small sites generally do not need robots.txt. First, make sure the CMS is not publishing unexpected restrictions. Google’s help page on robots.txt.
Can I download a universal robots.txt file?
You can download an example, but the right file depends on the site’s structure. Even sites using the same CMS may have different URLs for search, filters, and utility sections. Use a template as a starting point and check every line against your own URLs.
Why does a page blocked in robots.txt still appear in Google?
Blocking crawling does not guarantee that a URL will disappear from search results. Google may know about the URL through links. To exclude an accessible page from search, use noindex and allow the crawler to read that instruction.
What is the difference between robots.txt and sitemap.xml?
Robots.txt sets crawling rules, while an XML sitemap lists the URLs you want the search engine to know about. Including a page in a sitemap does not override a crawl restriction or guarantee indexing. The two files serve different purposes and should be consistent with each other. What sitemaps are used for.
Dmytro is an excellent specialist who fully supported the development of a new turnkey project. We will continue working on SEO and paid search advertising.
SEO Analytics for a Website at the Development Stage
★★★★★5/5
Dmytro is a specialist in his field. Responsive, does not work by template and always meets halfway to get the job done more effectively. We are satisfied with his work.
