We build. You grow.
Get best community software hereStart a social network, a fan-site, an education project with oxwall - free opensource community software
Need to scrape a website without getting blocked | Forum
Zurcke
Dec 10 '24
I’m trying to scrape a site with a ton of pages in a short amount of time, but I keep running into a block after a while. If I slow down the requests, I can’t finish in time. What’s the workaround here?
Yauppari
Dec 10 '24
Bro, you gotta use proxy servers for this, no question about it. When you’re hitting a site hard, they’re gonna notice all those requests coming from the same IP and shut you down. Proxies let you spread out the requests across different IPs, so it looks like they’re coming from a bunch of legit users instead of just you. I’ve had the same problem before, and proxies https://www.proxymadata.com/ saved my life. Look for something solid, like residential or rotating proxies. They’re harder to block and keep your scraping game smooth. Give it a shot; it’ll make a world of difference.
