Skip to main content

Can I Choose Which Page to Crawl?

Mike Kamo
  • Updated

In this article, you'll learn whether it's possible to select specific pages for crawling in Ubersuggest, how the crawling process works, and what you can do to influence which pages are audited. If you're running a website audit and want control over which URLs get analyzed, here's what you need to know.

Can You Select Specific Pages to Crawl?

You can’t manually select individual pages for crawling in Ubersuggest. The tool is built to perform a comprehensive crawl of your entire website, rather than allowing URL-by-URL selection.

That means when you start a site audit, Ubersuggest will scan a broad range of pages from your domain based on how your site is structured, rather than a list you provide.

How Does Ubersuggest Decide Which Pages to Crawl?

Ubersuggest doesn’t follow a fixed crawl order, so the specific pages it scans may vary each time you run an audit. However, there is a general process:

1. The crawl typically begins with your homepage.

2. From there, the tool discovers additional URLs through internal links and your sitemap.

3. It looks for your sitemap at the location defined in your robots.txt file.

So while you can’t choose exact pages, you can influence which ones are prioritized by ensuring your sitemap is complete and your internal linking structure is strong.

How to Start Crawling Your Website

To begin a crawl in Ubersuggest, follow these simple steps:

1. Go to the Site Audit page.

2. Enter your domain URL to initiate the crawl.

Site Audit: Run an audit for your website to discover SEO Issues and know how to fix them

Even though you can't target specific pages, entering a valid domain ensures the audit starts from the right place and expands from there.

Need more help? Check out the article Running a Site Audit with Ubersuggest

How Many Pages Can Be Crawled?

The number of pages Ubersuggest can crawl depends on:

  • Your website’s size

  • The limits of your Ubersuggest plan

For example:

  • Free and lower-tier plans may crawl a few hundred pages

  • Higher-tier plans can audit up to 10,000 pages

This makes it important to choose a plan that fits your site's size and your SEO needs.

Can You Influence What Gets Crawled?

Yes—indirectly. While you can’t handpick URLs, here’s how you can guide the crawler:

  • Update your sitemap.xml with the pages you want to prioritize.

  • Ensure all significant pages are linked internally from the homepage or other key areas.

  • Use your robots.txt file to allow or disallow specific sections of your site.

These steps won’t guarantee exact control, but they will help shape what Ubersuggest crawls during an audit.

If you have specific needs or run into issues, feel free to reach out to our support team by clicking on the ¨Help¨ button.