The Cost of Being Crawled: LLM Bots and Vercel Image API Pricing

Summary

On Friday, Feb 7, 2025 we had an incident with our Next.js web app hosted on Vercel that could've cost us $7,000 if we didn't notice it in time. We had a spike in LLM bot traffic coming from Amazonbot, Claudebot, Meta and an unknown bot. Together they sent 66.5k requests to our site within a single day. Bots scraped thousands of images that used Vercel's Image Optimization API, which cost us $5 per 1k images. The misconfiguration on our side combined with the aggressive bot traffic created an economically risky situation for our tiny bootstrapped startup. Metacast is a podcast tech startup. Our main product is a podcast app for iOS and Android. For every podcast episode on the platform, our web app has a web page. Our platform has ~1.4M episodes, which means we have 1.4M web pages that are discoverable by crawlers. These pages are generated server-side at request time, then cached. First, we received a cost alert from Vercel saying that we've hit 50% of the budget for resources metered by usage. We looked into it and saw that it's driven by the Image Optimization API, which peaked on Feb 7. Every page in the podcast directory has an image of a podcast cover (source image dimensions are 3000x3000px). With Image Optimization, podcast covers were reduced to 1/10th of the size, then cached. Image Optimization made the web app really snappy. It worked like a charm, except it turned out to be very expensive. Vercel charges $5 for every 1,000 images optimized. With thousands of requests coming our way, we were accumulating cost at the rate of $5 per each 1k image requests. In the worst case scenario, if all 1.4M images were crawled we'd hypothetically be looking at a $7k bill from Vercel. We looked at the user agents of requests in the Firewall in Vercel and saw Amazonbot, ClaudeBot, meta_externalagent and an unknown bot disguising itself as a browser. We can't say definitively which bots were downloading images, because we are on the Pro plan on Vercel and no longer have ...