This type of large-scale crawling should be considered a DDoS and the people behind it should be charged with cyber crimes and sent to prison.
If it’s disrupting their site, it is a crime already. The problem is finding the people behind it. This won’t be some guy on his dorm PC and they’ll likely be in places interpol can’t reach.
they’ll likely be in places interpol can’t reach
Like some Microsoft data center
Huawei
Applying the Computer Fraud and Abuse Act to corporations? Sign me up! Hey, they’re also people, aren’t they?
Put the entire datacenter buildings into prison
I think they call that a “job” already
good luck with that! not only is a company doing it, which means no individual person will go to prison, but it’s from a chinese company with no regard for any laws that might get passed
The people determining US legislation have said, “how can we achieve skynet if our tech trillionaire company sponsors can’t evade copyright or content licensing?” But they also say if “we don’t spend every penny you have on achieving US controlled Skynet, then China wins.”
Speculating on “Huawei network can solve this”, doesn’t mean that all the bots are Chinese, but does confirm that China has a lot of AI research, and Huawei GPUs/NPUs are getting used, and successfully solving this particular “I am not a robot challenge”.
It’s really hard to call “amateur coding challenge” competition web site a national security threat, but if you hype Huawei enough, then surely the US will give up on AI like it gave up on solar, and maybe EVs. “If we don’t adopt Luddite politics and all become Amish, then China wins” is a “promising” new loser perspective on media manipulation.
When you realize that you live in a cyberpunk novel. The AI is cracking the ICE. https://cyberpunk.fandom.com/wiki/Black_ICE
I love seeing how much influence William Gibson had on cyberpunk.
It’s not intentional but the chap ended up writing works that defined both the Cyberpunk (Neuromancer) and Steampunk (The Difference Engine) genres.
Can’t deny that influence.
most the ICE I’ve read about are white.
haven’t tried it, it’s in the closed apples store… but it’s a start…
Do we all want the fucking Blackwall from Cyberpunk 2077?
Fucking NetWatch?
Because this is how we end up with them.
…excuse me, I need to go buy a digital pack of cigarettes for the angry voice in my head.
Consider nicotine+
What was that?
I was sucking on my nicotine nipple, err, I mean my vape.
(Hey, its a more affordable stimulant addiction than coffee now!)
No, not the drug; the app.
Oh, well shit, I had not heard of this lol.
I am partial to I2P as … potentially, an entirely new, full internet paradigm, not just filesharing, but I will look into this too!
It’s a soulseek client, basically. You can share files, chat, put your interests in your profile, etc. It’s basically like social media, minus the posts. The only algorithm that exists is the one that shows people with similar interests. You can also view the most common interests. You can also add disinterests, which are the exact opposite.
That does sound very intetesting!
I blocked almost all big players in hosting, China, Ruasia, Vietnam and now they’re now bombarding my site with residential IP address from all over the world. They must be using compromised smart home devices or phones with malware.
Soon everything on the internet will be behind a wall.
This isn’t sustainable for the ai companies, when the bubble pops it will stop.
In the mean time, sites are getting DDOS-ed by scrapers. One way to stop your site from getting scraped is having it be inaccessible… which is what the scalpers are causing.
Normally I would assume DDOS-ing is performed in order to take a site offline. But ai-scalpers require the opposite. They need their targets online and willing. One would think they’d be a bit more careful about the damage they cause.
But they aren’t, because capitalism.
If they had the slightest bit of survival instinct they’d share a archive.org / Google-ish scraper and web cache infrastructure, and pull from those caches, and everything would just be scraped once, repeated only occasionally.
Instead they’re building maximally dumb (as in literally counterproductive and self harming) scrapers who don’t know what they’re interacting with.
At what point will people start to track down and sabotage AI datacenters IRL?
Not necessarily compromised, I saw a VPN provider (don’t remember the name) that offered a free tier where the client accepts being used for this.
And I suspect that in the future some VPN companies will be exposed doing the same but with their paid customers.
There are many commercial VPNs offering residential IPs. I doubt they use malware.
Business idea: AWS, but hosted entirely within the computing power of AI web crawlers.
Just reading the Ender’s game series, very on-point.
Tap for spoiler
We’ll wake up one day only to realise an entity living ‘in the wires’ is the only thing keeping the internet alive.
As long as NetWatch keeps them behind the Blackwall, we’re all good.
Reminds me of the “store data inside slow network requests for the in-transit duration”. It was a fun article to read.
Link, please?
http://tom7.org/harder/ (has links to the paper and also to the video)
Thanks!
I believe they are talking about Harder Drive: https://youtu.be/JcJSW7Rprio
Thanks!
I like the idea but couldn’t you just go the more direct route and mine crypto?
Like a public service CAPTCHA / BOINC hybrid
I really feel like scrapers should have been outlawed or actioned at some point.
But they bring profits to tech billionaires. No action will be taken.
No, the reason no action will be taken is because Huawei is a Chinese company. I work for a major US company that’s dealing with the same problem, and the problematic scrapers are usually from China. US companies like OpenAI rarely cause serious problems because they know we can sue them if they do. There’s nothing we can do legally about Chinese scrapers.
I thought Anthropic was also very abusive with their scraping?
Maybe to others, but not to us. Or if they are, they’re very good at masking their traffic.
Can you not just block China?
We do, somewhat. We haven’t gone as far as a blanket ban of Chinese CIDR ranges because there’s a lot of risks and bureaucracy associated with a move like that. But it probably makes sense for a small company like Codeberg, since they have higher risk tolerance and can move faster.
I use a tool that downloads a website to check for new chapters of series every day, then creates an RSS feed with the contents. Would this be considered a harmful scraper?
The problem with AI scrapers and bots is their scale, thousands of requests to webpages that the internal server cannot handle, resulting in slow traffic.
Does your tool respect the site’s robots.txt?
Unfortunately, robots.txt cannot express rate limits, so it would be an overly blunt instrument for things like GP describes. HTTP 429 would be a better fit.
Crawl-delayis just that, a simple directive to add to robots.txt to set the maximum crawl frequency. It used to be widely followed by all but the worst crawlers …Crawl-delay
It’s a nonstandard extension without consistent semantics or wide support, but I suppose it’s good to know about anyway. Thanks for mentioning it.
I was responding to their question if scraping the site is considered harmful. I would say as long as they are not ignoring robots they shouldn’t be contributing significant amounts of traffic if they’re really only pulling data once a day.
Yes, it just downloads the HTML of one page and formats the data into the RSS format with only the information I need.
If the site is getting slowed at times (regardless of whether it is when you scrape), you might want to not scrape at all.
Probably not a good idea to download the whole site, but then that depends upon the site.
- If it is a static site, if you just setup your scraper to not download CSS/JS and images/videos, that should make a difference.
- For a dynamically created site, there’s nothing I can say
- Then again, if you try to reduce your download to what you are using, as much as possible, that might be good enough
- Since sites are originally made for human consumption, you might have considered keeping the link traversal rates similar to that
- The best would be if you could ask the website dev whether they have an API available.
- Even better, ask them to provide an RSS feed.
As far as I know, the website doesn’t have an API but I just download the HTML and format the result with a simple Python script, it makes around 10 to 20 requests, one for each series I’m following at each time.
You can use the cache feature in curl/wget so it does not download the same css, html, twice. Also, can ignore JavaScript, and image files to save on unnecessary requests.
I would reduce the frequency to once every two days to further reduce the impact.
That might/might not be much.
Depends upon the site, I’d say.e.g. If it’s something like Netflix, I wouldn’t think much, because they have the means to serve the requests.
But for some PeerTube instance, even a single request seems to be too heavy for them. So if that server does not respond to my request, I usually wait for an hour or so before refreshing the page.
Seems like an api request would be preferable for the site you’re checking. I don’t imagine they’re unhappy with the traffic if they haven’t blocked it yet
I mean if it’s cms site there may not be an api, this would be the only solution in that case
The problem is these are constant army hordes / datacentres. You have one tool. Sending a few requests from your device wouldn’t even dent a raspberry pi, nevermind a beefier server
I think the intention of traffic is also important. Your tool is so you can consume the content freely provided by the website. Their tool is so they can profit off of the work on the website.
deleted by creator
But html is machine-readable and that absolutely is the point!
So search engines shouldn’t exist? This is absurdly simplistic.
Write TOS that state that crawlers automatically accept a service fee and then send invoices to every crawler owner.
Huawei is Chinese. There’s literally zero chance a European company like Codeberg is going to successfully collect from a company in China over a TOS violation.
It’s not even a company. It’s a non-profit “eingetragener Verein”. They have very limited resources, especially money because they purely live on membership fees and donations.
True, but it can help limit the European AI scrapers too
I really doubt it. Lawsuits are expensive, and proving responsibility is difficult, since plausible deniability is easy. All scrapers need to do is use shared IPs (e.g. cloud providers), preferably owned by a company in a different legal jurisdiction. That could be the case here: a European company could be using Huawei Cloud to mask the source of their traffic.
All scrapers need to do is use shared IPs (e.g. cloud providers),
Simple: just charge the cloud provider.
Once that gets strong enough they’ll start placing terms against scraping in their TOS.
And then they just throw it in the bin because there was never a contract between you and them. What to do then? Sue Microsoft, Amazon and Google
I’m sure Codeberg, a German non-profit Verein, has time and money to do that 🤣.
Sure but that’s a whole different part of the system. Society as a whole has to change (some guillotines would help) and no matter how cool Codeberg is, they can’t do all that on their own.
In the meantime, what the elites visibly respond to and that is more readily accessible is monetary costs. Make it costly (operationally or legally) to scrape sites, and they’ll stop if at least to whine.
They typically don’t include a billing address in the User Agent when crawling 🤣
That’s a technicality. The billing address can be discovered for a nominal fee as well.
I’m sure it can’t, especially for foreign IP addresses, VPNs, and a ton of other situations. Even if directly connect to the internet just via your ISP, many countries in Europe (don’t know about US) have laws that would require you to have very good reasons and a court order to get the info you need from the ISP - for a single(!) case.
If it would be possible to simply get the address of all digital visitors, we wouldn’t have to develop all this anti scrape tech and just sue them.
Cloudflare had a similar idea: Introducing pay per crawl: Enabling content owners to charge AI crawlers for access
If someone just wants to download code from Codeberg for training, it seems like it’d be way more efficient to just clone the git repositories or even just download tarballs of the most-recent releases for software hosted on Codeberg than to even touch the Web UI at all.
I mean, maybe you need the Web UI to get a list of git repos, but I’d think that that’d be about it.
Then they’d have to bother understanding the content and downloading it as appropriate. And you’d think if anyone could understand and parse websites in realtime to make download decisions, it be giant AI companies. But ironically they’re only interested in hoovering up everything as plain web pages to feed into their raw training data.
The same morons scrape Wikipedia instead of downloading the archive files which trivially can be rendered as web pages locally
I just thought that having a client side proof-of-work (or even only a delay) bound to the IP might deter the AI companies to choose to behave instead (because single-visit-per-IP crawlers get too expensive/slow and you can just block normal abusive crawlers). But they already have mind-blowing computing and money ressources and only want your data.
But if there was a simple-to-use integrated solution and every single webpage used this approach?
Believe me, these AI corporations have way too many IPs to make this feasible. I’ve tried per-IP rate limiting. It doesn’t work on these crawlers.
Are you planning to just outright ban IPv6 (and thus half the world)?
Any IP based restriction is useless with IPv6
Not really true, you can block ranges.
Okay, but how does that help? Or are you suggesting just wholesale banning entire ISPs?
What if we had some protocol by which the proof-of-work is transferable? Then not only would there be a cost to using the website, but also the operator would receive that cost as payment.
It’s theoretically viable, but every time that has been tried has failed
There are a lot of practical issues, mainly that it’s functionally identical to a crypto miner malware
Solution was invented long ago. It’s called a captcha.
A little bother for legitimate users, but a good captcha is still hard to bypass even using AI.
And I think for the final user standpoint I prefer to lose 5 seconds in a captcha, than the browser running an unsolicited heavy crypto challenge on my end.
For years, we’ve written that CAPTCHAs drive us crazy. Humans give up on CAPTCHA puzzles approximately 15% of the time and, maddeningly, CAPTCHAs are significantly easier for bots to solve than they are for humans.
https://blog.cloudflare.com/turnstile-ga/
I hate captchas.
AI is better at solving captchas than you.
I tried, and not really.
I had to scrape a site that have some captcha and no AI was able to consistently solve it.
In order to be able to “crack it” I had to replicate the captcha generation algorithm best I could and train a custom model to solve it. Only then I could crack it open. And I was lucky the captcha generation algorithm wasn’t to complex and it was easy to replicate.
This amount of work is a far greater load than Anubis crypto challenges.
Take into account that AI drive ocr drinks from existing examples, if your captcha is novel enough they are going to have a hard time solving it.
It also would drain power, which is the only point of anubis.
There is a difference between you (or me) sitting at home working on this and a team of highly motivated people with unlimited money.
The thing is not that it cannot be done, the thing is that the cost is most likely higher than Anubis.
Begun, the information wars have.
The wars have been fought and lost a while ago tbh
Seems like such a massive waste of bandwidth since it’s the same work being repeated by many different actors to piece together the same dataset bit by bit.
Ah Capitalism! Truly the king of efficiency /s
Are those blocklists publicly available somewhere?
I would hope not. Kinda pointless if they become public
On the contrary. Open community based block lists can be very effective. Everyone can contribute to them and asphyxiate people with malicious intents.
If you think something like, “if the blocklist is available then malicious agents simply won’t use that ips” I don’t think if that makes a lot of sense. As the malicious agent will know any of their IPs being blocked as soon as they use them.
Just to give an example of public lists that are working, I have an IRC server and it’s getting bombarded with spam bots. It’s horrible around the superbowl for some reason, but it just continues year round.
So I added a few public anti spamming lists like dronebl to the config, and the vast majority of the bots are automatically G-Lined/banned.
Im sure there are many but I just learned of Crowdsecs WAF this year which is has a shared ban list. Its pretty cool. Im using it in prod right now. Im not saying it’s the be all end all, but as part of a multilayered approach, it works pretty well.
It doesn’t seem to be free though, right?
I dont know if they have a paid version but their free version has worked for me.
I suppose it depends on whether others contribute or not.
I run my own gitea instance on my own server and within the past week or so I’ve noticed it just getting absolutely nailed. One repo in particular, a Wayland WM I built. Just keeps getting hammered over and over by IPs in China.
Just keeps getting hammered over and over by IPs in China.
Simple solution: Block Chinese IPs!
Are you using Anubis?
Why aren’t you firewalling it to only allow your IP? Are you sharing your code with third parties?
a few repos i’ve made available to the public, the wayland wm for example, so I haven’t gotten around to blocking IPs just yet.
Huh, why does Anubis use SHA256? It’s been optimized to all hell and back.
Ah, they’re looking into it: https://github.com/TecharoHQ/anubis/issues/94
It’s being investigated at least, hopefully a solution can be found. This will probably end up in a constantly escalating battle with the AI companies. https://github.com/TecharoHQ/anubis/issues/978
