Googlebot

May 3, 2026

Robots.txt

Robots.txt is a text file placed in a website’s root directory that instructs search engine crawlers which pages or sections to access or avoid. 123

How Robots.txt Controls Crawler Access

Robots.txt operates through directives that specify permissions for web crawlers, such as Googlebot, Bingbot, and other automated agents. The file contains 2 primary directive types: “User-agent” identifies the specific crawler, and “Disallow” blocks access to designated URLs or directories. Website administrators place robots.txt at the domain root (example.com/robots.txt) to ensure discoverability. Crawlers read this file before accessing other site content. The protocol follows the Robots Exclusion Standard, established in 1994. Syntax errors in robots.txt can accidentally block critical pages from indexing.

Robots.txt Directive Comparison

DirectiveFunctionExample
User-agentSpecifies target crawlerUser-agent: Googlebot
DisallowBlocks URL accessDisallow: /admin/
AllowPermits URL accessAllow: /public/
SitemapDeclares sitemap locationSitemap: https://example.com/sitemap.xml

Robots.txt Frequently Asked Questions:

Can robots.txt completely prevent page indexing? Robots.txt prevents crawling but does not guarantee removal from search results if external links reference blocked URLs.

Does robots.txt affect website security? Robots.txt provides no security protection and publicly exposes paths that administrators want hidden from crawlers.

How do search engines respond to robots.txt violations? Reputable search engines honor robots.txt directives, though malicious bots frequently ignore these instructions.