I’m preparing a presentation on how to implement an automated moderation of content on social media. I wanted to talk a bit on how this is done by small forums and Fediverse instances came as an obvious focus of study for me. Is it all done by hand by human moderators, or are there any tools that can filter out the obvious violations of instance’s rules? I’m thinking mostly about images, are those filtered for nudity/violence?
There are some, but they aren’t used very often. I know some instances scan CP and other illegal shit using community tools. But automods? No, I don’t think I’ve ever seen one actively operating (and I don’t think they are even maintained anymore, or at least the ones I know).
And even if they were, the advanced ones are only useful for the host.
So everything is manually moderated by humans at the moment.
That’s why I’m working on a bot with a plugin system. The plugin system will allow users of the bot to implement the logic themselves in one of the supported languages (e.g. Python, JavaScript, Rust) in a sandboxed environment (Wasm). It’s halfway done, but now I’m unsure if I want to create a dedicated Fediverse platform for it. One of the biggest reasons for this is to be free from the limitations of Lemmy and other factors.
The problems with relying on Lemmy right now are:
- You have to provide the bot with a Lemmy database connection (to reduce API usage and for faster response times).
- The above also means you have to be the one hosting the Lemmy instance. The bot doesn’t need persistent storage of posts or comments or other stuff, so the Lemmy database will continue to grow until your server runs out of storage. So you would have to clean the database periodically.
- You have to make Lemmy’s rate limiting practically useless for the bot to function reliably. The more communities use the bot, the more API calls will be made. If the bot gets rate limited, the plugins could be terminated by the plugin runtime. As long as Lemmy doesn’t introduce a way to change or bypass rate limiting for specific people, this will remain a problem.
- The bot uses the
lemmy-clientcrate from the Lemmy devs. Should this ever be discontinued, I would either have to maintain it myself, use another crate, or create one myself. The last two would be painful and require a lot of work. - The bot is as compatible with other Fediverse platforms as Lemmy is. It cannot use the unique features of other platforms (like PieFed or Mbin).
And so I’ve been researching how I could build a lightweight Fediverse platform specifically for the bot. That would eliminate all the problems mentioned above. Since the platform would be in my hands, I could also implement ways to federate with other platforms and even use the unique features of other platforms. But that’s not easy, so it will take some time. I also am not great at web dev, so the frontend will also be a problem.
There are definitely simple text based filters and alert systems, but I haven’t seen more advanced systems that use “AI” or other context aware technology or automated image processing.
I believe there’s an automated filter used by some of the biggest instances to detect and nuke child abuse stuff before a human has to see it
Everything else is human moderation though AFAIK
It’s all done by hand by human moderators
Considering how often I see nsfw or nsfl stuff because it wasn’t tagged… no I don’t think so, at least there doesn’t seem to be anything widely adopted on Lemmy.
For images at least it should be possible to create a bot that downloads any uploaded image, runs it through an API like Sightengine, and then automatically removes posts or bans users. Each instance would then of course decide whether to set up such a bot, so there is not going to be any fediverse-wide automod.
deleted by creator
