I hate crawlers that donโt advertise theyโre crawlers. โn badly written. They donโt respect the canonical URLs so they keep getting themselves into a loop instead of actually crawling the site properly. They make the logs really hard to read. Canโt block them theyโre distributed.
I am curious what an actual solution to this problem could be. Everything I've read about it feels like a bandaid โ from robots.txt to cloudflare turnstile โ there has to be a better way!
For this particular problem though, I feel like if the crawlers are coming from domestic / residential IP addresses and they lack a distinctive http header or user agent, it is indistinguishable from a denial of service attack and must be treated as such.
It would require permaban on IPs which would require permalogging of IPs in order to track them, and we are currently obfuscating them before hitting the DB with daily rotating salt so an anonymous user canโt be tracked. So weโd have to give up privacy/security :/
But as long as people can still use the site it's OK, right? Seems like the problem is often self-inflicted, either because the site is so bloated that answering even GET-requests is burdensome, or because they're trying to monetize the data and so must block non-paying scrapers.
We almost had one, but stopped after the main setup but before signing on. Wasn't convinced of the value. Seemed like they were just doing portfolio reallocation for a pretty penny.
What are you experiencing?
Had one for ~8 years. Talked to them maybe 10 times, and they only ever did basic indexing w some tax loss harvesting. We should have just self managed.
This may become a thing. Have you seen the results of PISA? School performance is going down in the OCDE and richer students are going down faster (they have more tech available). Maybe the future si for those who ditch the computers.
@erikprofilefollowI'm Nobody! Who are you?
Are you - Nobody - too?
Then there's a pair of us!
Dont tell! they'd banish us - you know!wrote:
I'm Nobody! Who are you?
Are you - Nobody - too?
Then there's a pair of us!
Dont tell! they'd banish us - you know!
How dreary - to be - Somebody!
How public - like a Frog -
To tell your name - the livelong June -
To an admiring Bog!
@amyprofilefollowa work in progress; i love cats ๐ฑreplied to@erikprofilefollowI'm Nobody! Who are you?
Are you - Nobody - too?
Then there's a pair of us!
Dont tell! they'd banish us - you know!:
Nobody is perfect. Iโm nobody.
@procyonprofilefollowlate is the hour in which this conjuror chooses to appearreplied to@erikprofilefollowI'm Nobody! Who are you?
Are you - Nobody - too?
Then there's a pair of us!
Dont tell! they'd banish us - you know!:
Yesterday upon the stair
I saw a man who wasn't there
He wasn't there again today
How I wish he'd go away!