I hate crawlers that donโt advertise theyโre crawlers. โn badly written. They donโt respect the canonical URLs so they keep getting themselves into a loop instead of actually crawling the site properly. They make the logs really hard to read. Canโt block them theyโre distributed.
I am curious what an actual solution to this problem could be. Everything I've read about it feels like a bandaid โ from robots.txt to cloudflare turnstile โ there has to be a better way!
For this particular problem though, I feel like if the crawlers are coming from domestic / residential IP addresses and they lack a distinctive http header or user agent, it is indistinguishable from a denial of service attack and must be treated as such.
It would require permaban on IPs which would require permalogging of IPs in order to track them, and we are currently obfuscating them before hitting the DB with daily rotating salt so an anonymous user canโt be tracked. So weโd have to give up privacy/security :/
But as long as people can still use the site it's OK, right? Seems like the problem is often self-inflicted, either because the site is so bloated that answering even GET-requests is burdensome, or because they're trying to monetize the data and so must block non-paying scrapers.
Bad night, first the boy woke up at midnight and went with him, then at 4 am heavy rain started so went to bring the bike to the terrace, then thunderstorm and kids were scary, then the cat meowed at 6am. Iโm okay.
I still remember Max getting sandwiched between the two Ferraris in the first ever wet night race (Singapore 2017). Today he finally managed to win a race there. That's quite a long wait.
Iโm an insane person who built their own coding harness, in Rust. Iโve called it Rho. Please forgive this landing page: https://rho-code.dev I just wanted an easy central spot.
Not insane at all. People should build their own instead of using other people's agents. It's easy to do and the results can be great for you. I've built several; one that is effectively "stuck" in Kakoune: youtube.com/โฆ/playlistYouTubeLLMs stuck in Kakoune
Yeah. It's a very large project with lots of dependencies. <The system prompt is long and wastes tokens. If you have ideas for tools (or automations like the agents.sh "loop" I wrote) it's easier if you understand the harness.
Unfortunately no. Just starting it. Got the proof of concept loading a dir with its sub-dirs then rendering on the GPU in textures silky smooth. 1.7mb executable, but will need to level up to SDL and that will double the size I think.
@fastidiousprofilefollow"Beware those that would deny you access to information, for in their heart they dream themselves your master." โ Sid Meyersreplied to@avioliprofilefollowProgrammer by day and night. Lover of art. Mixing art and programming is the dream.:
@avioliprofilefollowProgrammer by day and night. Lover of art. Mixing art and programming is the dream.replied to@fastidiousprofilefollow"Beware those that would deny you access to information, for in their heart they dream themselves your master." โ Sid Meyers:
I agree. It is #zilchnotesfollow for now. Will do better next time, but in my defence the text above the text box (when writing a new message, not a reply) says โWhatโs on your mindโ. :-P
I hate it when I donโt remember if I took my midday pill or not. Not that it matters a lot, but still, annoying. I have notifications set up but I discard them by habit because they are exact times and my sleep pattern fluctuates.
I once saw someone flip their pill bottle upside down after taking a pill. Not sure it'd help me, because I suspect that somehow I'd still only pay half-attention when taking pills and so still get forgetful and confused :shrug:
I think I may be use this for now to spill out my guts a bit. Set out boundaries guys. I was basically ignoring how important that is, whatever is at work, family, relations. Be loved being you, you don't have to meet any expectations beyond yours.