Knowledgebase: Documentation
Current ShrinkTheWeb datacenter IP addresses
Posted by Brandon Elliott on 29 August 2014 12:39 PM

In some cases, our crawler may be tagged as a site scraper or other spammy type of attack. In that case, our IP(s) may get blocked, resulting in the inability to capture certain web pages. This happens most frequently when monitored sites are refreshed too often in our system. We allow refreshes as often as once every 180 seconds, for maximum flexibility, but doing so may look like an attack to monitored websites, which may then block the IP and lock us out.

We used to publish a list of IPs for our web page capture generators, but our platform is so distributed now, including use of various cloud services, that it is virtually impossible to maintain an up-to-date list of IPs. This is especially true with some of the cloud providers, as they may reassign IPs as they see fit and not notify us.

As a recommendation, we suggest to refresh sites only when needed and no more often than once per week or daily with very large sites that change often (i.e. cnn.com, yahoo.com, etc).

Please note that our IPs are subject to change and this list may be out-of-date. Last Updated: 12/31/2015

(0 vote(s))
Helpful
Not helpful

Contact Us | PagePix Benefits | Learn More | STW Forums | Our Partners | Privacy Policy | Terms of Use

©2016 ShrinkTheWeb. All rights reserved. ShrinkTheWeb is a registered trademark of ShrinkTheWeb.