A bug was introduced in our Crawler infrastructure on Friday 14 Sept., which caused a queue of job items to build up over the weekend. The situation was further compounded by a misconfiguration in an underlying messaging component. The bug was fixed Monday morning, and the queue had been completely processed by Tuesday morning (both CET).
Only a small part of our customers will have been affected by this bug, and those affected will experience crawl delays of no more than three days.
We have improved our alerting to make sure we notice bugs faster in the future, and will also be reviewing the configuration of our messaging components.