Twitter’s former Trust and Safety head details the challenges facing decentralized social platforms
Yoel Roth, the former head of Trust and Safety at Twitter (now X) and currently with Match Group, has voiced significant concerns regarding the future of the open social web. His primary worry centers on the ability of decentralized social platforms to effectively combat prevalent issues such as misinformation, spam, and illegal content, including child sexual abuse material (CSAM).
In a recent interview on the podcast “revolution.social with @Rabble,” Roth highlighted a critical deficit in moderation tools available to the fediverse—a collective of interconnected open social applications like Mastodon, Threads, Pixelfed, and others, as well as platforms such as Bluesky. He emphasized that despite their emphasis on community-based control, these platforms often provide their communities with the least technical resources for policy administration.
Roth also reflected on his tenure at Twitter, recalling pivotal moments such as the platform’s decision to ban President Trump, the pervasive spread of misinformation by Russian bot farms, and instances where even Twitter’s own CEO, Jack Dorsey, fell prey to automated disinformation campaigns. He pointed out a significant “backslide” in transparency and decision legitimacy within the open social web compared to Twitter’s past practices. While Twitter, controversially at times, explained its rationale for content decisions, many current decentralized platforms allow problematic posts to simply vanish without notice, leaving users unaware of the moderation action or even the post’s former existence.
“…looking at Mastodon, looking at other services based on ActivityPub [protocol], looking at Bluesky in its earliest days, and then looking at Threads as Meta started to develop it, what we saw was that a lot of the services that were leaning the hardest into community-based control gave their communities the least technical tools to be able to administer their policies,” Roth stated, questioning the actual success of projects aiming for increased democratic legitimacy if governance transparency regresses.
The Economics of Moderation
A major hurdle, according to Roth, is the financial unsustainability of moderation within the federated model. He cited the recent shutdown of many projects by IFTAS (Independent Federated Trust & Safety), an organization dedicated to building moderation tools for the fediverse, including critical CSAM detection capabilities. IFTAS ceased operations due to a lack of funding in early 2025, underscoring the severe economic challenges.
“We saw it coming two years ago. IFTAS saw it coming. Everybody who’s been working in this space is largely volunteering their time and efforts, and that only goes so far, because at some point, people have families and need to pay bills, and compute costs stack up if you need to run ML models to detect certain types of bad content,” Roth explained. He believes that the economics of this federated approach to trust and safety “never quite added up. And in my opinion, still don’t.”
Bluesky, in contrast, has opted to employ human moderators and invest in trust and safety teams for its own application, alongside offering users tools to customize their moderation preferences. While Roth acknowledges their efforts are generally “doing the right stuff,” he notes that as Bluesky further decentralizes, it will face complex questions about where the responsibility to protect individuals (e.g., from doxxing) lies when moderation is highly user-configured.
Where to Draw the Line on Privacy
Another significant challenge is the inherent tension between user privacy and the requirements for effective moderation. While Twitter aimed to minimize data collection, it still gathered essential information like IP addresses, access times, and device identifiers. This data was crucial for forensic analysis of threats, such as identifying and dismantling Russian troll farms.
In contrast, many fediverse administrators may not collect necessary logs or may hesitate to review them due to privacy concerns. Roth stressed that without such data, distinguishing genuine users from sophisticated bots becomes exceedingly difficult. He recalled an instance from his Twitter days where users would mistakenly label anyone they disagreed with as a “bot,” a behavior he found consistently incorrect after manual review. Even Twitter co-founder Jack Dorsey inadvertently amplified content from a Russian actor masquerading as a U.S. citizen, highlighting the challenge of bot detection without robust data.
The Role of AI
Addressing the evolving digital landscape, Roth discussed the impact of artificial intelligence. He referenced recent Stanford research indicating that large language models (LLMs), when properly tuned, can generate political content more convincingly than humans. This development means that a moderation strategy relying solely on content analysis is insufficient.
Instead, Roth advocated for tracking behavioral signals—such as an entity creating multiple accounts, using automation for posting, or exhibiting unusual posting times corresponding to different time zones. “These are behavioral signals that are latent even in really convincing content. And I think that’s where you have to start this,” Roth advised. He warned that if platforms start only with content analysis, they are in an “arms race against leading AI models and you’ve already lost.”
Roth’s insights underscore the complex and multifaceted challenges facing decentralized social platforms as they strive to balance open, community-driven governance with the vital need for effective content moderation in an increasingly sophisticated digital environment.
