Pro@programming.devM to Technology@programming.devEnglish · edit-27 days agoCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comimagemessage-square114linkfedilinkarrow-up1699arrow-down12file-textcross-posted to: technology@lemmy.world
arrow-up1697arrow-down1imageCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comPro@programming.devM to Technology@programming.devEnglish · edit-27 days agomessage-square114linkfedilinkfile-textcross-posted to: technology@lemmy.world
minus-squareS7rauss@discuss.tchncs.delinkfedilinkEnglisharrow-up4·8 days agoI was responding to their question if scraping the site is considered harmful. I would say as long as they are not ignoring robots they shouldn’t be contributing significant amounts of traffic if they’re really only pulling data once a day.
I was responding to their question if scraping the site is considered harmful. I would say as long as they are not ignoring robots they shouldn’t be contributing significant amounts of traffic if they’re really only pulling data once a day.