Any conversation

thx ^^ its in computational chemistry, knock yourself out: doi.org/10.1021/acs.jpca.6c02038

moved to berlin for my bachelors and it took some time to find footing. Im not the best student but I ended up getting lucky to snag a student assistant position by applying/presenting at a summer school (targeted more toward master/phd). then that became my thesis and it dragged quite a long time, bureacracy and supervision and such. But the results were interesting and i learned a ton along the way. Im really happy I yolo applied to that program

For this particular problem though, I feel like if the crawlers are coming from domestic / residential IP addresses and they lack a distinctive http header or user agent, it is indistinguishable from a denial of service attack and must be treated as such.

It would require permaban on IPs which would require permalogging of IPs in order to track them, and we are currently obfuscating them before hitting the DB with daily rotating salt so an anonymous user can’t be tracked. So we’d have to give up privacy/security :/

But as long as people can still use the site it's OK, right? Seems like the problem is often self-inflicted, either because the site is so bloated that answering even GET-requests is burdensome, or because they're trying to monetize the data and so must block non-paying scrapers.