It’s fair that maybe the architecture of public inbox/outbox protocols aren’t suited for this kind of use (juxtapose with Matrix).
However consider this: Some people on the fediverse simply don’t want to be indexed. It should be opt-in instead of opt-out, for people who explicitly want it. People aren’t against search, they’re against non-consensual search.
I think it’s important for the culture of the fediverse that such civility is encouraged. Because on the fediverse, the community can actually make a difference. By blocking federation with offenders, we can guide the culture of fedi. And it’s better for it.
Running with the idea that people can “technically” do what they want because of the nature of the protocols is counter-productive, because we actually can do something.
All a search implementer has to do is adapt to that culture, and they’ll be fine. So I don’t see why there’s such push-back against this viewpoint.
I had a thought about a week ago about search engines. I think it may be better to move towards a curated-list style of search, instead of an automated indexing one. Think Pinterest, but with a greater emphasis on websites, people, and communities instead of just images.
That way search can be made more human. You can get to know and trust the individuals who curate content, and maybe donate to them if their efforts are valuable to you.
Sounds like this is exactly what you’re looking for: https://fedidevs.org/category/projects
It’s a wiki of sorts started recently by the lead dev of Owncast.