Crawling

What is crawling?

Crawling is the process in which a search engine, such as Googlebot, scours the web by following links and fetching pages. In doing so, the search engine reads the content and code of each page. Crawling is the first step before indexing: a page that cannot be reached usually does not appear in search results.

Why is crawling important?

• Getting discovered: without a crawl, Google does not know your page exists.
• Fresh results: the more often Google visits your pages, the sooner changes are picked up.
• Foundation of SEO: technical obstacles can make good content invisible.

Examples of crawling

• Googlebot follows a link in your menu to a new service page.
• A crawler reads your sitemap and visits every URL listed.
• A page blocked in robots.txt is not fetched.

How do you support crawling?

You cannot force crawling, but you can make it easier. Use clear internal links, an up-to-date sitemap and a robots.txt that does not block important pages. Check in Google Search Console that pages are fetched without errors. We can help with technical SEO.