one more step. just a moment. attention required! enable javascript and cookies to continue. verifying the security of your connection. your browsing activity has been paused. content unavailable in your region. 429 too many requests. 403 forbidden. connection has timed out.
The future is here. But instead of flying cars or holodeck, what you get is tracking cookies and captcha everywhere.
Maybe someday you will need to have your rectum scanned by Google or Meta nee Facebook before your browser does a HTTP request.
@elana๐งprofilefollowI have a doctorate in ancient stuff but these days I'm a full-time plague avoider. Born and raised in Toronto, Canada, now living in Kyoto, Japan.wrote:
Built a couple lambdas today to send account verification and onboarding drip emails for folks who sign up for my app but donโt make it all the way through the onboarding flow. Pretty cool to be able to quickly bang out self-contained infra for this instead of having to pay some SaaS vendor.
I hate crawlers that donโt advertise theyโre crawlers. โn badly written. They donโt respect the canonical URLs so they keep getting themselves into a loop instead of actually crawling the site properly. They make the logs really hard to read. Canโt block them theyโre distributed.
I am curious what an actual solution to this problem could be. Everything I've read about it feels like a bandaid โ from robots.txt to cloudflare turnstile โ there has to be a better way!
For this particular problem though, I feel like if the crawlers are coming from domestic / residential IP addresses and they lack a distinctive http header or user agent, it is indistinguishable from a denial of service attack and must be treated as such.
It would require permaban on IPs which would require permalogging of IPs in order to track them, and we are currently obfuscating them before hitting the DB with daily rotating salt so an anonymous user canโt be tracked. So weโd have to give up privacy/security :/
But as long as people can still use the site it's OK, right? Seems like the problem is often self-inflicted, either because the site is so bloated that answering even GET-requests is burdensome, or because they're trying to monetize the data and so must block non-paying scrapers.