Rendered at 09:29:50 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
faangguyindia 1 days ago [-]
If you want to access data from websites which prevent it, you gotta use a headless browser with Residential Proxy Network Like Bright Data (formerly Luminati).
nicbou 1 days ago [-]
Our industry's understanding of consent is terrifying
jeong_jeong 22 hours ago [-]
It’s called hacker news, bro
ccgreg 3 hours ago [-]
I'm a life-long hacker, and my crawler crawls with consent.
4lx87 2 days ago [-]
I'm curious, how do you deal with Cloudflare and similar anti-bot systems? Just keep shopping the job around to different proxies?
faangguyindia 17 hours ago [-]
it's fairly simple, you use browser profiles and you visit multiple website like a normal guy using residential proxyy network
and cloudflare cannot detect you this way.
the older your browser profile is, the less often cloudflare bans.
fragmede 22 hours ago [-]
Cloudflare reads this forum. By answering your question here, they burn that workaround. Why would someone do that? (No one bring up Warframe)
2 days ago [-]
fragmede 22 hours ago [-]
have you already incorporated common crawl into your index?
ccgreg 3 hours ago [-]
Common Crawl is a sample of the web, so it's not that directly helpful for someone wanting to make a product price dataset.
and cloudflare cannot detect you this way.
the older your browser profile is, the less often cloudflare bans.