What Is Crawling? Definition for SEO
The process in which search engine bots systematically browse websites and record their content.
What is crawling?
Crawling is the process in which search engine bots (also called spiders or crawlers) systematically browse the web, follow links and record the content of web pages.
Example: a bakery adds a new page for its catering offer and links to it from the homepage. The next time the bot visits the homepage, it follows that link, finds the new page and fetches its content.
Crawling is only the first step. Whether a fetched page is stored in the search index and can appear in results is decided afterwards, during indexing.
How does Googlebot work?
Googlebot starts with a list of known URLs, follows the links on those pages and discovers new content this way. How often it crawls a site depends on the site's authority and on how frequently it is updated.
- Crawl budget – every website has a limited budget that Google spends on crawling it
- Robots.txt – controls which areas may be crawled
- Sitemap – helps Google find all important pages
For SEO, this means a page that bots cannot reach will not show up in search results. AI search systems also rely on content they can fetch, as our article on how AI search selects its sources explains. Crawlability is part of technical SEO.
Frequently asked questions
What is crawling?
Crawling is the process in which search engine bots browse the web, follow links and record the content of web pages.
Why does crawling matter for SEO?
A page that bots cannot crawl cannot be indexed and will not appear in search results. Crawl budget, robots.txt and the sitemap influence which pages Google reaches.
What is an example of crawling?
A bakery links a new catering page from its homepage. On its next visit, the bot follows that link, finds the new page and fetches its content.
How does crawling differ from indexing?
Crawling means fetching a page. Indexing means Google processes it and stores it in the search index, and Google can crawl a page and still decide not to index it.