I vibe coded my own Scrapingbee alternative and replaced my $249/mo plan with a $1/mo scraper

✨ Okay with lots of help from @javilopen and his Spanish scraping friends I've managed to vibe code my own @Scrapingbee alternative and replace my $249/mo Scrapingbee plan with my own $1/mo scraper running on my VPS!

First I tried OpenSERP, an open source project that seems freshly maintained but it didn't work on my VPS IP or laptop IP and got blocked fast

Then I tried to use residential IP proxies from IPRoyal but they were already quite tainted IPs meaning Google already knew they were scrapers and just blocked about 60% of them already, not so great, also didn't work!

Then I discovered OpenSERP uses a vanilla headless Chrome, Google instantly detects that and blocks that, so literally the entire repo is useless cause it won't even work? I don't get why they don't say that in the docs? OpenSERP is even worse than just basic CURL, as my AI said: "plain curl got 7% blocked but OpenSERP's headless browser got 100% blocked"

So then I just built my own with Playwright and a real Google Chrome on my VPS, headful and persistent context, and slow pacing like every 30 seconds or so

And I got some better residential IP recommendations (which I can't tell you because thenhey t stop working!)

Now I'm at about 90% success rate, which is similar or better than Scrapingbee

And it costs me ~$1/mo (measured by GBs, obviously for text scraping I barely use a lot) instead of the $249/mo I paid Scrapingbee!

My only remaining SaaS now are:

• Cloudflare Domains
• Cloudflare Email Sending
• Cloudflare R2 Storage
• Backblaze B2 for backups
• Hetzner for all my servers for all my sites (VPS)
• xAI for all LLMs for all my sites (so many things like moderation, writing emails, anything)

On levels.io

I've tried for weeks to build my own ScrapingBee but I'll keep paying $99/mo

Originally posted on X · 💬 Reply to this on X