Although search engines may index a website, certain crawlers might encounter difficulties accessing its content. Various factors can contribute to this problem, including restrictions set by the Content Management System (CMS) or other hosting tools.
Impact on SEO
These limitations can prevent search engines and other crawlers from effectively crawling and understanding your website's content, potentially impacting its visibility in search results.
Site Audit-Only Scans One Page (Or No Pages) Of My Website
Typically, if the Site Audit scans only one page or no pages, it is blocked from accessing your website.
Common Causes of Crawler Blockages
Robots.txt Directives: Your site's robots.txt file might contain directives that instruct crawlers to exclude certain pages or directories from indexing.
Server-Side Configurations: Security measures such as firewalls might inadvertently block crawler access. These configurations can be overly restrictive, preventing legitimate crawlers from accessing your site.
CMS and Hosting Tools: Some CMS or hosting tools may have settings that restrict crawler access to certain parts of your site. This can include authentication requirements or other access controls.
Security Measures: Firewalls, DDoS protection, and other security tools can sometimes block crawlers if they are not configured correctly. This can include IP blocking or rate limiting that prevents crawlers from indexing your site efficiently.
Unfortunately, if any of these settings can be observed on your website, our robots won’t be able to properly analyze your website.
Our IP Addresses for Whitelisting
Our engineers state that it is always recommended that our IPs be handled as exceptions to your server/firewall rules.
Below is the list of our IP addresses that you should whitelist to prevent any access issues:
94.130.93.30
168.119.141.170
168.119.99.190
168.119.99.191
168.119.99.192
168.119.99.193
168.119.99.194
134.209.42.109
68.183.60.34
68.183.60.80
68.183.54.131
68.183.49.222
68.183.149.30
68.183.157.22
68.183.149.129
It is also relevant to know the ports we connect:
For HTTP, we connect to port 80.
For HTTPS to 443 by default.
If you need the name of our user agent, please use:
User-Agent: RSiteAuditor
For additional instructions, please check the article How to Whitelist Ubersuggest Ips Inside My Domain
The idea behind whitelisting IPs is to eliminate the possibility of our bots being blocked from crawling your domain.