Over the weekend we implemented a fix to our Crawler which addressed the central issue and greatly improved the crawl throughput. The crawlers have been crawling happily along since then and we're now almost through the queue which means crawls are no longer delayed. We thank you for your patience and once again apologize for any inconvenience.
Posted Nov 02, 2020 - 12:49 UTC
We're still working on catching up with the crawl queue. We have generally identified what the issues are, and are implementing multiple strategies for improving the throughput of the crawler.
Posted Oct 30, 2020 - 11:10 UTC
We are currently experiencing a larger than usual crawl queue, which means some crawls will be delayed. We're looking into the situation with priority.