AI caused our systems to crash – and what we did about it
A behind-the-scenes look at how AI crawlers contributed to intermittent downtime – and the steps we took to resolve it.
Over the last few weeks, some of our websites began experiencing short, sharp bursts of 502 Bad Gateway errors. Users would click a link and instantly hit an error – no loading delay, no pattern – just brief outages that typically resolved themselves within 60–90 seconds.
We have been investigating the issue with our partners at WP Engine and what we found was unexpected.
What We Found
- The issue affected users across multiple networks, but disappeared when switching to a mobile hotspot or another network connection.
- When cross-referencing server access logs, we noticed a high frequency of requests from AI crawlers – specifically user agents like:
GPTBot(OpenAI)Meta External Agent
While these bots are typically benign, the volume and timing of their visits lined up with the symptoms we observed: high load, server response delays, and 502s triggered at the network level.
What We Did
After confirming the pattern, we worked with WP Engine on a multi-pronged solution:
- Server Migration: We moved to a more resilient server variant better suited to our traffic and resource profile.
- Bot Management: We began blocking specific AI user agents from resource-heavy paths using web rules.
- Monitoring Enhancements: We’re testing tools like Dark Visitor to gain deeper visibility into bot activity.
- Content Strategy Review: We’re reviewing which parts of our sites should be open to crawlers and which need to be protected.
What You Should Know
AI crawlers are becoming more active across the web – and they may already be crawling your site. While not malicious, their intensity can trigger performance issues if left unmanaged.
We believe this is something more businesses will need to consider going forward.
Watch this space – In this series, we focus on the emergency of AI (Artificial Intelligence) and its impact on your systems and your sites, We expect share practical steps any site owner can take to manage AI crawlers without sacrificing visibility.
