Skip to main content

Troubleshooting Ubersuggest Site Audit Crawl Issues

Mike Kamo
  • Updated

Although search engines may index a website, certain crawlers, including Ubersuggest's Site Audit, might encounter difficulties accessing its content. Various factors can contribute to this problem, including restrictions set by the Content Management System (CMS), server configuration, or other hosting and security tools.

This article brings together everything you need to diagnose and fix a blocked or stalled Site Audit crawl: the warning signs, the most common causes, how to troubleshoot each one, and how to whitelist Ubersuggest's IPs across the most popular platforms.

Impact on SEO

These limitations can prevent search engines and other crawlers from effectively crawling and understanding your website's content, potentially impacting its visibility in search results. If Ubersuggest can't crawl your site, search engines might also struggle, which can hurt your SEO since they won't be able to index your content effectively.

Signs Your Site Audit Is Being Blocked

Site Audit Only Scans One Page (Or No Pages) Of My Website

Typically, if the Site Audit scans only one page or no pages, it is blocked from accessing your website.

Site Audit dashboard showing Pages Discovered and SEO Issues Discovered

The Audit Shows "Site Crawl Has Stopped"

If a Site Audit scans only a single page or displays the message “Site crawl has stopped” on the audit page, it indicates that access to the website is being blocked. This error means Ubersuggest couldn't complete the site audit for your website.

Common Causes of Crawler Blockages

Robots.txt Directives: Your site's robots.txt file might contain directives that instruct crawlers to exclude certain pages or directories from indexing.

Server-Side Configurations: Security measures such as firewalls might inadvertently block crawler access. These configurations can be overly restrictive, preventing legitimate crawlers from accessing your site.

CMS and Hosting Tools: Some CMS or hosting tools may have settings that restrict crawler access to certain parts of your site. This can include authentication requirements or other access controls.

Security Measures: Firewalls, DDoS protection, and other security tools can sometimes block crawlers if they are not configured correctly. This can include IP blocking or rate limiting that prevents crawlers from indexing your site efficiently.

Site Accessibility: Ensure your website is accessible. If it's down or slow, Ubersuggest can't crawl it.

Homepage Set to Noindex: Check if your homepage is set to ‘noindex.’ If so, change it to allow indexing.

Improper Redirects: Verify your website uses proper 301 redirects, including a clean redirect from http:// to https://. Canonical loops or redirect chains can also stop a crawl.

Plugins: Some CMS plugins (security, caching, or firewall plugins in particular) can block our crawler without you realizing it.

Authentication: Password protection, staging environments, or login walls will block our crawler from accessing your pages.

Rate Limiting: Some servers block or throttle an IP after it makes too many requests in a short period, which can interrupt a crawl partway through.

Unfortunately, if any of these settings can be observed on your website, our robots won't be able to properly analyze your website.

How to Troubleshoot Each Cause

Work through the following checklist. Most crawl issues are resolved by one of these steps:

  1. Whitelist our IP addresses. This is the most common fix. See the full list and platform-specific instructions below.
  2. Disable plugins temporarily. Security, firewall, or caching plugins are common culprits. Disable them one at a time to see if the crawl resumes.
  3. Check your robots.txt file. Make sure it isn't disallowing our crawler. To view it, enter example.com/robots.txt in your browser (replacing “example.com” with your website address).
  4. Check for noindex meta directives. Use your browser's “Inspect Element” tool to check your homepage's meta tags: open your website, right-click the page, and select Inspect (or a similar option depending on your browser) to view the page's HTML and confirm it isn't set to noindex.
  5. Verify your redirects. Manually enter your website URL in an incognito window to check that it redirects properly from http:// to https:// and doesn't fall into a redirect loop.
  6. Remove authentication barriers. Make sure no login page, password protection, or staging environment is blocking our crawl.
  7. Review your .htaccess file. Check for any access restrictions that might be blocking our IPs or user agent.
  8. Optimize site performance. Improve page load speed to reduce timeouts during the crawl.
  9. Adjust rate limits. Make sure your server doesn't block our IPs after a certain number of requests.

If you've checked these steps and are still having issues, your website's security system is most likely blocking our robots at the server or firewall level, which brings us to whitelisting.

Whitelisting Ubersuggest's IPs

Our engineers recommend always handling our IPs as exceptions to your server or firewall rules. This eliminates the possibility of our bots being blocked from crawling your domain.

Below is the list of IP addresses you should whitelist to prevent any access issues:

  • 94.130.93.30
  • 168.119.141.170
  • 168.119.99.190
  • 168.119.99.191
  • 168.119.99.192
  • 168.119.99.193
  • 168.119.99.194
  • 134.209.42.109
  • 68.183.60.34
  • 68.183.60.80
  • 68.183.54.131
  • 68.183.49.222
  • 68.183.149.30
  • 68.183.157.22
  • 68.183.149.129

It's also relevant to know the ports we connect on:

  • For HTTP, we connect to port 80.
  • For HTTPS, we connect to port 443 by default.

If you need the name of our user agent, please use:

User-Agent: RSiteAuditor

Whitelisting by Platform

Here's how to whitelist our IPs on the most common servers and platforms:

Apache

On Apache servers, IP whitelisting is typically done by updating the .htaccess file or the virtual host configuration.

  1. Locate and open either the .htaccess file in your website's root directory or the virtual host configuration file.
  2. Use the Require IP directive to whitelist an IP address, for example:
Require ip 94.130.93.30

Repeat for each IP address in the list above, then confirm that Ubersuggest can now successfully crawl and access your website.

Nginx

On Nginx servers, IP whitelisting is managed through the server or location block configuration. Add the following, using allow for each of our IPs and deny all at the end:

allow 94.130.93.30;
allow 168.119.141.170;
allow 168.119.99.190;
allow 168.119.99.191;
allow 168.119.99.192;
allow 168.119.99.193;
allow 168.119.99.194;
allow 134.209.42.109;
allow 68.183.60.34;
allow 68.183.60.80;
allow 68.183.54.131;
allow 68.183.49.222;
allow 68.183.149.30;
allow 68.183.157.22;
allow 68.183.149.129;
deny all;

Confirm that Ubersuggest can successfully access your website after saving your changes.

Cloudflare

Cloudflare makes it simple to whitelist IPs using the IP Access Rules feature. This allows trusted IPs, like Ubersuggest's, to bypass firewall restrictions and access your website without interruption.

  1. Log in to your Cloudflare account.
  2. Choose the Firewall icon and click the Tools tab.
  3. Under IP Access Rules, enter each Ubersuggest IP address as the Value.
  4. Set the Action to Allow.
  5. Choose whether it applies to this domain or all domains in your account.
  6. Click Add to save each IP.
  7. Confirm that the new rules appear in the list. Changes take effect immediately.

WordPress (Wordfence)

On WordPress sites, IP whitelisting is best handled through the Wordfence Security plugin. This ensures trusted IPs, such as Ubersuggest's, can access your site without being blocked by the firewall.

  1. In your WordPress dashboard, go to the Wordfence plugin.
  2. Click All Options (or “Settings” and then “All Options,” depending on your version) to open the advanced firewall settings.
  3. Under Advanced Firewall Options, find the field labeled “Whitelisted IP addresses that bypass all rules” (sometimes called “IP Whitelist”).
  4. Enter each Ubersuggest IP address, one per line.
  5. Save your changes.

After saving, Wordfence will immediately allow these IPs to access your website without restrictions.

By completing these steps, Ubersuggest will be able to crawl your website smoothly on your chosen platform, whether it's Apache, Nginx, Cloudflare, or WordPress. Be sure to test access afterward and confirm that your site's security settings are still working as expected.

Why Should I Whitelist Ubersuggest's IPs?

SEO tools need to crawl and analyze websites to gather data. If their IPs are blocked by the site's firewall, the tools won't function correctly. Whitelisting these IPs ensures they can access the site without any issues, allowing the tools to perform their tasks, such as crawling and indexing, without interference.

Still Need Help?

If you have any questions, click the Help button at the bottom of your screen. Our support team is ready to assist you anytime.