It’s not. It just doesn’t get enough hits for that 86k to matter. Fun fact: most AI crawlers hit /robots.txt first, they get served a bee movie script, fail to interpret it, and leave, without crawling further. If I’d let them crawl the entire site, that’d result in about two megabytes of traffic. By serving a 86kb file that doesn’t pass as robots.txt and has no links, I actually save bandwidth. Not on a single request, but by preventing a hundred others.

I don’t think serving 86 kilobytes to AI crawlers will make any difference in my bandwidth use :)

That would result in those fediverse servers theoretically requesting 333333 * 114MB = ~38Gigabyte/s.

On the other hand, if the site linked would not serve garbage, and would fit like 1Mb like a normal site, then this would be only ~325mb/s, and while that’s still high, it’s not the end of the world. If it’s a site that actually puts effort into being optimized, and a request fits in ~300kb (still a lot, in my book, for what is essentially a preview, with only tiny parts of the actual content loaded), then we’re looking at 95mb/s.

If said site puts effort into making their previews reasonable, and serve ~30kb, then that’s 9mb/s. It’s 3190 in the Year of Our Lady Discord. A potato can serve that.

I only serve bloat to AI crawlers.

map $http_user_agent $badagent {
  default     0;
  # list of AI crawler user agents in "~crawler 1" format
}

if ($badagent) {
   rewrite ^ /gpt;
}

location /gpt {
  proxy_pass https://courses.cs.washington.edu/courses/cse163/20wi/files/lectures/L04/bee-movie.txt;
}

…is a wonderful thing to put in my nginx config. (you can try curl -Is -H "User-Agent: GPTBot" https://chronicles.mad-scientist.club/robots.txt | grep content-length: to see it in action ;))

…and here I am, running a blog that if it gets 15k hits a second, it won’t even bat an eye, and I could run it on a potato. Probably because I don’t serve hundreds of megabytes of garbage to visitors. (The preview image is also controllable iirc, so just, like, set it to something reasonably sized.)

Ah! My bad.

mumbles something about big corps choosing way too generic names for their stuff

Threads does not interact with the Fediverse in its current form. It’s a horn blasting into the fediverse at best. It’s not participating in the fediverse, it’s shouting into it. As such, it’s correct to not report on how thredsizens participate in the fediverse - they do not, not at this time.

There’s a very easy solution that lets you rest easy that your instance is how you want it to be: don’t do open registration. Vet the people you invite, and job done. If you want to be even safer, don’t post publicly - followers only. If you require follower approval, you can do some basic checks to see that whoever sends a follow request is someone you’re okay interacting with. This works on the microblogging side of the Fediverse quite well, today.

What I’m trying to say is that with registrations requiring admin approval gets you 99% of the way there, without needing anything more complex than that.

…and you think 14-17 year olds won’t circumvent this in mere seconds? Like, they’d just sign up at an instance that doesn’t implement these labels, or doesn’t care about them, or use their parents accounts, or ask them, or an older friend to sign them up, and so on. Even if age verification would be widespread and legally mandated, I highly doubt any sufficiently determined 14-17 year old would have any trouble getting past it.

Nevertheless, as Bluesky grows, there are likely to be multiple professionally-run indexers for various purposes. For example, a company that performs sentiment analysis on social media activity about brands could easily create a whole-network index that provides insights to their clients.

(source)

Is that supposed to be a selling point? Because I’d like to stay far, far away from that, thank you very much.