Shouting into the Void

Why Reporting Abuse to Social Media Platforms Is So Hard and How to Fix It

Shouting into the Void

June 29, 2023

Home / Reports / Shouting into the Void

Key Findings:

Writers and journalists regularly face harassment and threats to their free expression online, but when they try to report the abuse to digital platforms, they describe it as “shouting into the void.”

The mechanisms for reporting abuse online are deeply flawed and further traumatize and disempower those facing abuse, failing at every level to safeguard them and hold abusers to account.

This report outlines how platforms can build reporting mechanisms that are more user-friendly, accessible, transparent, efficient, and effective, with clearer and more regular communication between the user and the platform.

PEN America Experts:

Viktorya Vilk

Director, Digital Safety and Free Expression

External Experts:

Kat Lo
Program Manager, Digital Health Lab at Meedan

This report was developed in collaboration with Meedan.

Introduction

When PEN America and Meedan asked writers, journalists, and creators about their experiences reporting online abuse to social media platforms, we heard again and again, over the past three years, about the deep frustration, exasperation, and harm caused by the reporting mechanisms themselves:

I do the reports because I don’t want to not report. That’s even worse. But it feels like shouting into a void. There’s no transparency or accountability.

Jaclyn Friedman, writer and founder, Women, Action & the Media¹

Reporting is the only recourse that we have when abuse happens. It’s a form of accountability…but when people constantly feel like they are wasting their time, they are just going to stop reporting.

Azmina Dhrodia, expert on gender, technology and human rights and former senior policy manager, World Wide Web Foundation²

The experience of using reporting systems produces further feelings of helplessness… Rather than giving people a sense of agency, it compounds the problem.

Claudia Lo, senior design and moderation researcher, Wikimedia³

Online abuse is a massive problem.⁴ According to a 2021 study from the Pew Research Center, nearly half of adults in the U.S. have personally experienced online harassment. The rate of severe harassment—including stalking and sexual harassment—has significantly increased in recent years.⁵

For journalists, writers, and creators who rely on having an online presence to make a living and make their voices heard, the situation is even worse—especially if they belong to groups already marginalized for their actual or perceived identity. In a 2020 global study of women journalists from UNESCO and the International Center for Journalists, 73 percent of respondents said they experienced online abuse. Twenty percent reported that they had been attacked or abused offline in connection with online abuse. Women journalists from diverse racial and ethnic groups cited their identity as the reason they were disproportionately targeted online.⁶ According to Amnesty International’s 2018 report, Toxic Twitter: A Toxic Place for Women, Black women were “84 percent more likely than white women to be mentioned in abusive or problematic tweets.”⁷

Being inundated with hateful slurs, death threats, sexual harassment, and doxing can have dire consequences. On an individual level, online abuse places an enormous strain on mental and physical health. On a systemic level, when creative and media professionals are targeted for what they write and create, it chills free expression and stifles press freedom, deterring participation in public discourse.⁸ Online abuse is often deployed to stifle dissent. Governments and political parties are increasingly using online attacks, alongside physical attacks and trumped-up legal charges, to intimidate and undermine critical voices, including those of journalists and writers.⁹

The technology companies that run social media platforms, where so much of online abuse plays out, are failing to protect and support their users. When the Pew Research Center asked people in the U.S. how well social media companies were doing in addressing online harassment on their platforms, nearly 80 percent said that companies were doing “an only fair or poor job.”¹⁰ According to a 2021 study of online hate and harassment conducted by the Anti-Defamation League and YouGov, 78 percent of Americans specifically want companies to make it easier to report hateful content and behavior, up from 67 percent in 2019.¹¹

Finding product and policy solutions that counter the negative impacts of online abuse without infringing on free expression is challenging, but it’s also doable—with time, resources, and will. In a 2021 report, No Excuse for Abuse, PEN America outlined a series of recommendations that social media platforms could enact to reduce risk, minimize exposure, facilitate response, and deter abusive behavior, while maintaining the space for free and open dialogue. In doing that research, it became clear that the mechanisms for reporting abusive and threatening content to social media platforms were deeply flawed.¹² In this follow-up report, we set out to understand how and why.

On most social media platforms, people can “report” to the company that a piece of content—or an entire account—is violating policies. When a user chooses to report abusive content or accounts, they typically initiate a “reporting flow,” a series of steps they follow to indicate how the content or account violates platform policies. In response, a platform may remove the reported content or account, use other moderation interventions (such as downranking content, issuing a warning, etc.), or take no action at all, depending on the company’s assessment of whether the reported content or account is violative.

For users, reporting content that violates platform policies is one of the primary means of defending themselves, protecting their community, and seeking accountability. For platforms, reporting is a critical part of the larger content moderation process.

To identify abusive content, social media companies use a combination of proactive detection via automation and human moderation and reactive detection via user reporting, which is then adjudicated by automated systems or human moderators. The pandemic accelerated platforms’ increasing reliance on automation, including the algorithmic detection of harmful language. While automated systems help companies operate at scale and lower costs, they are highly imperfect.¹³

Human moderators are better equipped to take the nuances of language, as well as cultural and sociopolitical context, into account. Relying on human moderation to detect abusive content, however, comes with its own challenges, including scalability, implicit bias, and fluency and cultural competency across languages. Moreover, many human moderators—the majority of whom are located in the Global South—are economically exploited and traumatized by the work.¹⁴

Because proactive detection of online abuse, both human and automated, is highly imperfect, reactive user reporting remains a critical part of the larger content moderation process. More effective user reporting, in turn, can also provide the data necessary to better train automated systems. The problem is that when reporting mechanisms do not work properly, that undermines the entire content moderation process, which significantly impedes the ability of social media companies to fulfill their duty of care to protect their users and facilitate the open exchange of ideas.

A poorly functioning moderation process threatens free expression in myriad ways. Content moderation interventions that remove or reduce the reach of user content can undermine free expression, especially when weaponized or abused.¹⁵ At the same time, harassing accounts that are allowed to operate with impunity can chill the expression of the individuals or communities they target.¹⁶

In our research, we found that reporting mechanisms on social media platforms are often profoundly confusing, time-consuming, frustrating, and disappointing. Users frequently do not understand how reporting actually works, including where they are in the process, what to expect after they submit a report, and who will see their report. Additionally, users often do not know if, or why, a decision has been reached regarding their report. They are consistently confused about how platforms define specific harmful tactics and therefore struggle to figure out if a piece of content is violative. Few reporting systems currently take into account coordinated or repeated harassment, leaving users with no choice but to report dozens or even hundreds of abusive comments and messages piecemeal.

On the one hand, the reporting process takes many steps and can feel unduly laborious; on the other, there is rarely the opportunity to provide context or explain why a user may find something abusive. Few platforms offer any kind of accessible or consistent documentation feature, which would allow users to save evidence of online abuse even if it has been deemed abusive and removed. And fewer still enable users to ask their allies for help with reporting, which makes it more difficult to reduce exposure to abuse.

When the reporting process is confusing, users make mistakes. When the reporting process does not leave any room for the addition of context, moderators may lack the information they need to decide whether content is violative. It’s a lose-lose situation—except perhaps for abusive trolls.

For this report, nonprofit organizations PEN America and Meedan joined forces to understand why reporting mechanisms on platforms are often so difficult and frustrating to use, and how they can be improved in concrete, actionable ways. Informed by interviews with nearly two dozen writers, journalists, creators, technologists, and civil society experts, as well as extensive analysis of existing reporting flows on major platforms (Facebook, Instagram, YouTube, Twitter, and TikTok), this report maps out concrete, actionable recommendations for how social media companies can make the reporting process more user-friendly, more effective, and less harmful.

While we discuss the policy implications of our research, our primary goal is to highlight how platform design fails to make existing policies effective in practice. We recognize that reporting mechanisms are only one aspect of content moderation, and changes to reporting mechanisms alone are not sufficient to mitigate the harms of online abuse. Comprehensive platform policies, consistent and transparent policy enforcement, and sophisticated user-centered features are central to more effectively addressing online abuse and protecting users. And yet reporting remains the first line of defense for millions of users worldwide facing online harassment. If social media platforms fail to revamp reporting, as well as put more holistic protections in place, then public discourse in online spaces will remain less inclusive, less equitable, and less free.

1. Create a Dashboard for Tracking Reports, Outcomes, and History

Challenges

Reporting online abuse to tech companies takes time and energy and can amplify the stress, anxiety, and fear of the person experiencing the abuse. Once a person has reported abuse, it is imperative that they receive regular communication and have transparency about the content moderation process, so that they can track outcomes and understand why decisions were made.

After sending a report on Instagram you would get a message saying ‘we’re going to review this,’ but there was no indication of how long the review is going to take or whether they would tell us their decision. When the account got suspended, we found out by going back to the harasser’s page I reported.

Sarah Fathallah, senior research fellow at the research lab Think of Us

At present, however, users often express a sense that they are reporting into a void. In many cases, there is no indication if a content moderation decision has been reached or why.¹⁷ On some platforms, such as Twitter and Twitch, there isn’t even a way to see filed reports once they have been submitted.¹⁸ Sarah Fathallah, senior research fellow at the research lab Think of Us, said, “After sending a report on Instagram you would get a message saying ‘we’re going to review this,’ but there was no indication of how long the review is going to take or whether they would tell us their decision. When the account got suspended, we found out by going back to the harasser’s page I reported.¹⁹

YouTube’s Report History dashboard for content a user has reported as violative. (Screenshot captured by Meedan, June 10, 2023.)

Most platforms do offer some communication about the reporting process and some even have basic inboxes with records of and updates about reports, but the communication is often limited and sporadic and the inboxes can be exceedingly difficult to find and navigate. For example, Instagram and Facebook provide basic inboxes to store reports (see case study below), but users must dig around to find them. TikTok also has an inbox and users are informed about it when they report content, but this inbox is otherwise difficult to find and only accessible via the mobile app. Twitter does not have an inbox or dashboard for reports; instead it sends sporadic updates via the notification feed and users’ emails.

YouTube comes the closest to offering a dashboard for individual users, via its “Report history” section, which is relatively accessible on a desktop, but does not seem to work on mobile. This dashboard allows users to track which videos they have reported, but does not include reported comments, does not indicate where a report is in the review process, and does not include all of the information a user has submitted in the report. Tellingly, it’s actually YouTube’s copyright dashboard, rather than its reporting history dashboard, that provides the kind of information, usability, and accessibility that should exist across all platforms for tracking reports of hate and harassment (for more, see case study below).

Recommendations

Create a dashboard, at the account level, that enables an individual user to track the reports they have personally submitted, the status of each report within the content moderation process, and the outcome for each report. If the user disagrees with the content moderation decision reached in response to the content or account they reported, they should be able to appeal from within the dashboard. This dashboard
- should include reports for all content types, including user profiles, posts, replies, comments, and direct messages;
- should include all key information relevant to reporting, in one place, including the current estimated or average processing times for reviewing reports and whether a human or a bot reviewed the report; and
- could include reports made on behalf of the individual user by authorized third parties (see section 7).
Ensure the dashboard is easily accessible and user-friendly. The reporting dashboard needs to be readily visible and easily accessible from the user settings menu and available across mobile and desktop. And it needs to be easy to use. Platforms should educate users about the existence of the dashboard through prompts at the end of the reporting flow.
Design the dashboard through an identity-aware and trauma-informed lens. Accessible dashboards and notifications are valuable tools, but they could also increase users’ exposure to abusive content. This can be mitigated by employing principles of trauma-informed design, which center on providing users with greater control over when and how they view abusive content.²⁰ To do this, platforms could minimize visibility of the harmful content within the dashboard itself by hiding it and giving users the option to “click to view”; allow users to sort, hide, and archive reports from the dashboard; allow users to indicate whether they want to see the harmful content again when they receive updates about the status of their reports; and/or provide users with the option to receive updates only within the dashboard rather than via emails or pop-up notifications.

Product case study: Facebook’s Support Inbox for reporting

Facebook’s Support Inbox for content a user has reported as violative. (Screenshot captured by Meedan, April 12, 2023.)

For Facebook, Meta has developed a “Support Inbox” that centralizes all of the content a user has reported as violative. The Support Inbox provides several useful features, including the ability to delete the report from the inbox and block the abusive user. In our tests, users were occasionally offered the option to request additional review, but we found this feature to be highly inconsistent and unpredictable. Moreover, users were often not given access within the inbox to the content of the post that they reported. Multiple individuals we interviewed could not actually find the inbox or were not even aware that it existed.²¹ And there were significant issues with keeping users updated, in a clear and timely manner, about the status of their reports. In a formal response, Meta assured us that they “always inform the user we’ve received their report or feedback and follow up accordingly” and inform users “about the final outcome of their reports through a message in their Support Inbox.”²² However, we extensively tested Facebook’s reporting flows and found that the outcomes of reports were presented inconsistently and were sometimes not updated more than a year after the report was submitted, or simply showed error messages.

Product case study: YouTube’s copyright management tool as a model for a reporting dashboard

It is the dashboard for YouTube’s copyright management tool, rather than the platform’s actual reporting system, that provides a model for how platforms can build an effective dashboard for reporting abusive content. If a YouTube user discovers there is unauthorized use of their content on the platform, they can deploy the Copyright Match Tool to identify and protect their intellectual property.²³ This tool provides a centralized dashboard from which users can request removal or monetization of their copyrighted content, including content that has been automatically detected by YouTube’s Content ID system.²⁴ The screenshot below shows this user-friendly and responsive dashboard, which includes key information relevant for reporting, such as type of media; title, description, or preview of the content; description of the relevant violation; and the status of the request. An important aspect of this dashboard interface is that the user can see key information across all of the items at one time in a way that is not overwhelming. It is precisely this kind of centralization, usability, and accessibility that should exist on YouTube itself and across all platforms to help users track their reports of hate and harassment.

YouTube’s copyright management dashboard, which provides a model for how to build an effective dashboard for reporting abusive or otherwise violative content. (Screenshot from YouTube Creators’ “How to Prevent Videos You’ve Taken Down From Being Reuploaded,” accessed April 29, 2022.)

2. Give Users Greater Clarity and Control as They Report Abuse

Challenges

While it is critically important for users to understand what to expect after they have reported abuse, it is equally important that they understand what to expect during the process of reporting abuse. If the actual reporting mechanism is unpredictable and unclear, that can exacerbate the feelings of stress and powerlessness often caused by the abuse itself. Users need to understand what steps they will need to take, where they are in the reporting flow, whether they will be given the opportunity to add context, and who will see their report once they submit it. For many platforms, this is not currently the case.

During walk-throughs and interviews with our research team, multiple participants found reporting mechanisms deeply confusing; two were actually caught by surprise when their reports were submitted before they even realized they had reached the end of the reporting flow.²⁵ For example, as one interviewee was attempting to report abuse to Twitter (before the revamp of its reporting process described below), she was looking for a way to add context and additional tweets, when she suddenly discovered that the report had already been submitted without any of the information she intended to include.²⁶

If users do not know who will ultimately be notified of their report, they can be less likely to report abusive content even on behalf of someone else, because they fear being negatively perceived by others.²⁷ In a 2020 survey of minors who chose not to seek help when confronted with potentially harmful experiences, 24 percent were worried their report would not be anonymous, despite feeling confident in their ability to use platforms’ reporting tools effectively.²⁸ While the studies we cite were focused on youth, we found parallels in our own research with writers, journalists, and creators. And users have every reason to be concerned. In one egregious example, Renée DiResta, research manager at the Stanford Internet Observatory, filed a copyright-based takedown request to stop the use of her photographs of her baby as part of a coordinated harassment campaign; Twitter not only alerted her harassers, but shared her private information, thereby exposing DiResta to further abuse and safety risks.²⁹

For users to feel safe reporting, they need to know if the harasser will be notified that their content has been reported and if the harasser will be notified about the identity of the person who reported their content. Even when platforms provide assurances of anonymity in their help centers, this information is not clearly stated within the reporting flow itself, where users actually need to see it in real time in order to feel comfortable proceeding.

And finally, if users discover that reporting abuse reduces their ability to mitigate that content in other ways, they may choose not to report at all. For example, interviewee Corry Will, an educational science influencer, demonstrated to us in real time that when he attempted to report an abusive comment on Instagram, that comment then automatically disappeared from his view, preventing him from taking any further action on it, such as deleting it or blocking the abusive account; yet everyone else could still see and interact with the abusive comment. In other words, the reporting process effectively prevented him from further protecting himself.³⁰ By contrast, Twitter hides reported comments under a click-through that says “You reported this tweet,” but with the option to view and act on the previously reported content.³¹ In a 2021 survey of creators across social media platforms who had experienced hate and harassment, creators similarly expressed confusion about the trade-offs between reporting and taking other moderation actions, including removing the offending comment, to mitigate harm as quickly as possible.³²

Recommendations

Ensure users understand how the reporting flow works, including what steps they will need to take, whether they will be given the opportunity to add information, who will see the report once they submit it, who will be notified that they reported abuse, if they should expect a platform response and within what timeframe they should expect that response, how they will be notified, and if there’s an appeals process.
Provide a progress bar that helps users understand where they are within the reporting flow, in real time, as they complete the process.
Make it clear to users when their reports are about to be submitted and allow users to review their reports before they are submitted. At the end of the reporting flow, show the user the information that they are submitting—including the categories they selected and any contextual information that they provided—before the user clicks on the button to submit the report. And include a “back” button so that the user can make edits to their report without having to restart it.
Ensure that reporting does not impede or reduce a user’s ability to act on abusive content in other ways, including by blocking, muting, restricting, or documenting the content or the account engaging in harassment.

Product case study: Twitter’s revamped reporting process

In June 2022, Twitter released a substantively revamped user experience for reporting harmful content,³³ which has many features that align with this report’s recommendations. Unfortunately, since Elon Musk purchased the platform in 2022 and gutted its Trust and Safety teams, it is unclear whether this revamped reporting process and the policies it aims to enforce will be kept in place. Moreover, under Musk’s management, Twitter’s larger content moderation process, of which reporting is just one critical component, has become significantly less effective in implementing platform policies, leading to an influx of hate and harassment on the platform.³⁴ That said, the improvements to Twitter’s reporting system from 2022 are worth highlighting, including:

Allowing users to indicate how they believe the content policy that they selected is being violated, including the ability to provide additional context.
Allowing users facing identity-based attacks to indicate what aspect of their identity is being targeted.
Suggesting the category of abuse based on the answers the user provided, with a detailed description and example of what that category entails.
Allowing the user to choose different categories of abuse if they disagree with the category that Twitter proactively suggested.
Allowing users to report multiple abusive tweets from a single account.
Streamlining additional steps by condensing the “Add other tweets” and “Add additional context” options into a menu on a single screen.
Providing an overview of the report for users to review before submitting it.
Informing the user what will happen next, including an estimate of how long the content moderation process will take.

Screenshots of Twitter’s revamped reporting process from December 2021. (Screenshot from Twitter’s Common Thread, “Twitter’s new reporting process centers on a human-first design,” accessed May 2023.)

3. Align Reporting Mechanisms with Platform Policies

Challenges

Platforms are effectively asking users to only report content that violates their policies. In order to do that, however, users need to understand what platforms do—and do not—consider violative. The problem is that users are often confused about how platforms define specific types of harm and harmful tactics, which in turn undermines reporting and the larger content moderation process.³⁵

If there is a comment calling a trans woman a man, is that hate speech or is it harassment? I don’t know. I kind of don’t know what to click and so I don’t do it, and just block.

Natalie Wynn, creator of the YouTube channel ContraPoints

Users who are reporting abuse experience confusion and anxiety about deciding which reporting category is “best” to select so that the abusive content is evaluated under the appropriate policy. Mikki Kendall, an author and diversity consultant who writes about race, feminism, and police violence, points out that some platforms that say they prohibit “hate speech” provide “no examples and no clarity on what counts as hate speech.”³⁶ Natalie Wynn, creator of the YouTube channel ContraPoints, explained: “If there is a comment calling a trans woman a man, is that hate speech or is it harassment? I don’t know. I kind of don’t know what to click and so I don’t do it, and just block.”³⁷ A number of interviewees thought that “bullying and harassment” required multiple persistent actions from one or more individuals and were hesitant to report individual pieces of abusive content, when in fact individual pieces of content can also qualify as harassment under most platform policies.³⁸ In other words, confusion about how platforms define abuse-related terms and tactics can discourage users from reporting abuse altogether.³⁹

To complicate matters, the content that users want to report often violates multiple categories. If users are unable to select all of the relevant categories, they are forced to choose. Leigh Honeywell, founder and CEO of the personal cybersecurity and anti-abuse start-up Tall Poppy, describes the dilemma users often face across multiple platforms: “You can report something as abuse targeting a specific group or you can report it as threatening. It’s up to the user to figure out which of those is going to be more effective. From what I’ve seen, it’s probably the one reporting a threat, but people don’t necessarily know that—and just reporting it as threatening without also flagging the hate aspect of it erases an important part of what makes the threat serious.”⁴⁰

Several interviewees mentioned that they had concerns as to whether the category they selected could negatively impact platform response.⁴¹ According to TikTok and Meta, content reported as abusive or harmful is reviewed against all platform policies, regardless of which category users select in reporting, though we were unable to independently verify this.⁴² If that is the case, users clearly don’t know it.

In a survey of the abuse-reporting tools provided by a range of online service providers, researchers at the Stanford Internet Observatory found that providers do not consistently enable users to report all of the various types of abuse that they may encounter.⁴³ Kristiana de Leon, an activist and city council member in Washington state, found that in some reporting flows on Facebook she was given the option to select multiple categories. In other reporting flows on the same platform, however, she was only given the ability to select one category. When she was unable to select multiple categories in a report, she often reported a single post multiple times under different categories because she felt the post violated multiple policies. But she worried that making multiple reports for the same post could be counterproductive. “Sometimes I would get a note back saying we received your report, but sometimes I only get one, even though I reported maybe four categories,” Kristiana explained. “Is there some sort of hierarchy I’m not aware of? Or maybe it nullified my first one that I thought was the most important.”⁴⁴

One participant expressed concern about whether there could be consequences or penalties for reporting “incorrectly” because they had not properly interpreted what counts as violative. Elizabeth Ballou, writer and narrative designer at Deck Nine Games, who has reported people on Twitter involved in harassment campaigns against marginalized game developers and journalists, expressed a concern about being flagged as an unreliable reporter: “The number of notifications I got from Twitter saying that they had actually banned an account or suspended an account or deleted a tweet were really dropping off and I wonder if Twitter has flagged my account for reporting too many things. Now they see me as a low-quality reporter, maybe they even think that I’m a spammer.”⁴⁵

Recommendations

Ensure that the terms and definitions for harassing and hateful tactics provided in platform policies are clear and closely align with those used in reporting mechanisms. Platform policies need to specify which kinds of tactics are violative, provide definitions with illustrative examples, and use those same terms and definitions within reporting mechanisms.
Ensure that platform policies are easily accessible during the reporting process itself. Users should never have to leave a reporting flow in order to check policies. Platforms should use pop-ups and other methods that are easily navigable, similar to customer support interfaces.
Give users the ability to select multiple categories of harm and indicate which tactics are being deployed. For example, allow a user to select both “harassment” and “hate speech,” and then allow the user to indicate specific tactics, such as “slurs” and/or “violent threats.”

4. Offer Users Two Reporting Options: Expedited and Comprehensive

Challenges

When it comes to reporting abusive content, sometimes users have different—and occasionally competing—needs. In some situations, users want the reporting process to be quicker and easier, especially if they are experiencing a high volume of abuse.⁴⁶ Among the users we interviewed, some opted to block, restrict, or mute abusive accounts, rather than report them—even when they felt the content clearly violated platform policies—because they found the reporting process so cumbersome. Corry Will noticed himself doing this: “When a major amount of harassment started happening, I basically just stopped reporting because it was far less effective than just restricting the accounts.”⁴⁷ One journalist we interviewed, who requested anonymity, found the blocking feature more convenient than going through the reporting process because it was not only more empowering, but much faster.⁴⁸ Users can often block abusive accounts in just one click, whereas reporting takes many clicks. As Claudia Lo, a senior design and moderation researcher at Wikimedia, pointed out, it can take up to six clicks to report abuse on Facebook, “by which time I have probably forgotten what I was trying to report.”⁴⁹

When a major amount of harassment started happening, I basically just stopped reporting because it was far less effective than just restricting the accounts.

Corry Will, educational science influencer

In other situations, users actually want a more comprehensive reporting process that provides room for context, especially if they are being harassed across multiple platforms or in combination with offline threats or abuse, or if the abuse is coded or otherwise requires additional information to understand.⁵⁰ Platforms often fail to appropriately moderate abusive content that violates their policies because their moderators lack sufficient context, including cultural and linguistic nuances, to make an accurate judgment.⁵¹

Users sometimes decide whether or not to report abuse based on how likely they think the platform is to take content down based on the information they believe is submitted in the report. For example, if abusive content contains a person’s deadname, compromising information, or an extremist dog whistle, but avoids using explicitly abusive terms, a user may choose not to report it because they are unable to provide explanatory context and therefore do not think the platform will understand that a policy has been violated.⁵²

One interviewee in Lebanon reported numerous images of two closeted men kissing, which were posted without their consent.⁵³ The danger of revealing LGBTQ+ identities in Lebanon, where homosexuality is socially taboo and criminalized, is significant.⁵⁴ However, the interviewee believed the posts would not be seen as violating Facebook’s community standards because content moderators are not likely to be familiar with the regional or cultural context that makes these posts dangerous.⁵⁵

Influencer Corry Will shared a different example illustrating the importance of being able to provide context when reporting abuse. “There was a post with a number that specifically refers to trans suicide rates, that is well-known as a dog whistle, but you can’t rely on the moderator knowing what that means. To them this person just reported a number. If you can add a little context, then you could say ‘hey, this is an absolute dog whistle.’”⁵⁶

TikTok informed us that they “don’t provide details of previous reports or moderation decisions” to their content moderators “as this may unduly bias the reviewer’s decision.”⁵⁷ While it is understandable to focus moderators on the decision at hand, rather than relitigate past decisions, providing users with space to add context can actually strengthen the ability of moderators to do their jobs more effectively.

There was a post with a number that specifically refers to trans suicide rates, that is well-known as a dog whistle, but you can’t rely on the moderator knowing what that means. To them this person just reported a number. If you can add a little context, then you could say ‘hey, this is an absolute dog whistle.’

Corry Will, educational science influencer

This screenshot indicates the options provided at the end of Facebook’s reporting flow immediately after a user has reported a post on a Facebook page, which includes the ability to “Block” and “Hide all.” (Screenshot captured by PEN America on May 23, 2023.)

Recommendations

Allow users to choose between an expedited and a comprehensive reporting option. Users should be able to select which option they prefer at the beginning of the reporting process.
- Expedited Reporting should not require more than two to three clicks to submit.
- Comprehensive Reporting should allow users to add context to explain why something is abusive and violates policies (e.g., if the harassment is part of a coordinated campaign, tied to events happening offline, etc.). This is particularly important in situations where moderators may lack knowledge of or experience with the relevant cultural or regional context.
Provide prompts to help users understand what kinds of contextual information they need to provide to enable content moderators to assess potential policy violations. Few platforms currently offer users with room to provide context at all and those that do, such as Twitter and YouTube, do so inconsistently and often have open text fields with vague prompts such as “add more context.”
Provide users with options to block, mute, restrict, or otherwise act on abusive content or accounts at the end of the reporting process. While Twitter, Instagram, and Facebook currently offer users an accessible prompt to block, mute, or restrict an abusive account at the end of the reporting flow, other platforms do not yet do this.

Risks and challenges

Enabling users to provide more detailed contextual information in the reporting process can introduce risks and challenges, which will need to be addressed:

When users share personal information in the reporting process to provide context for abuse, that can put them at risk. This is a significant concern for people whose professions, identities, and activism make them vulnerable to government surveillance and persecution, particularly when social media companies collaborate with governments or when security breaches leak personally identifiable information to the public.⁵⁸ Meta assured us that they give their reviewers “the minimal and appropriate amount of information needed to make an accurate decision against our policies,” and have “internal security experts” review all content flows “to ensure user privacy and security concerns are addressed.”⁵⁹ We were not, however, able to independently verify this information. It is important that platforms inform users about how the contextual information in their report will be stored, how it will be associated with their account, and what privacy and other relevant policies apply so users can make informed decisions about what personal information they choose to disclose in a report.
Giving users the ability to provide contextual information as part of the reporting process will create new demands for content moderators. To mitigate the increased volume and strain of evaluating contextual information in reports, platforms could create more structure by offering users specific prompts or more detailed multiple-choice options.

5. Adapt Reporting to Address Coordinated and Repeated Harassment

Challenges

Harassment is often experienced over an extended period of time; in volume, repeatedly, and/or across multiple platforms, types of media (text, image, video, etc.), and types of interactions (DMs, posts, replies, comments, etc.). Yet reporting processes are typically constructed to respond to a single instance from a single user at a single moment in time.

Reporting systems need to reflect online abuse as it actually manifests for users online rather than forcing users to carry the burden not only of abuse, but of conforming to the limitations of systems not designed to serve them.

Few social media platforms explicitly or comprehensively address the phenomenon of networked or mob harassment. In a study analyzing networked harassment on YouTube, researchers at Stanford University and the University of North Carolina at Chapel Hill defined the tactic as “online harassment against a target or set of targets which is encouraged, promoted, or instigated by members of a network, such as an audience or online community.” They concluded that “YouTube’s current policies are insufficient for addressing harassment that relies on amplification and networked audiences.”⁶⁰ Some platforms are starting to fill this gap. For example, in 2021 Facebook launched a new policy to “remove coordinated efforts of mass harassment that target individuals and put them at heightened risk of offline harms,” even if individual pieces of content do not otherwise appear to violate policies.⁶¹ And Discord and Twitch’s policies now include at least some recognition of coordinated abuse.⁶²

Even when platform policies do address networked harassment, few platforms give users the opportunity to indicate that they are experiencing coordinated or repeated harassment or to report that kind of harassment in batches, rather than piecemeal. Twitter, it is worth noting, does allow users to report multiple violating tweets in one batch, including those by a single user. In fact, two interviewees independently praised the introduction of these batch reporting features on Twitter and suggested that it would be useful to have similar features available on other platforms.⁶³ On YouTube, when users report a channel (rather than an individual video), they can attach multiple videos from that channel to provide context, but they cannot actually report multiple videos at once.⁶⁴ While enabling users to report multiple pieces of abusive content—including across content types (such as comments, replies, and DMs) and across multiple users—presents real technical and operational challenges, it is absolutely critical in order to align with lived user-experience and facilitate effective content moderation. Reporting systems need to reflect online abuse as it actually manifests for users online rather than forcing users to carry the burden not only of abuse, but of conforming to the limitations of systems not designed to serve them.

Recommendations

Enable users to submit a single report containing multiple violative pieces of content, across content types such as comments or posts, perpetrated by a single account.
Enable users to report multiple abusive comments and posts from multiple, separate accounts in a single reporting flow.
Enable users to indicate within the reporting process that the abusive content they are experiencing seems to be a part of a coordinated or networked harassment campaign and to request that their report be reviewed holistically in that context.

Risks and challenges

It is important to acknowledge that bulk reporting can be weaponized. In formal comments to PEN America, Meta and TikTok acknowledged this reality, but assured us that mass reporting does not have any impact on their content moderation decisions or response times.⁶⁵ And yet, over the years, there have been ample cases of mass reporting being used maliciously to trigger content takedowns and account suspensions as a tactic to harass and silence.⁶⁶

Allowing users to report multiple pieces of content and accounts at one time may exacerbate this problem. At present, users who have had their content removed or accounts suspended due to malicious reporting can appeal, though the appeals process can be slow and ineffective. And civil society organizations can also escalate incidents of malicious mass reporting to platforms, though again this process is uneven in its efficacy. But in introducing or strengthening the capacity to bulk report content or accounts, more will need to be done to mitigate the risk of weaponization. Platforms may need to limit how many accounts or pieces of content can be reported at a time, or to activate the bulk reporting feature only when automated systems identify an onslaught of harassment that bears hallmarks of coordinated inauthentic activity. Ultimately, risk assessment, consultation with civil society, and user testing during the design process would reveal the most effective mitigation strategies.

6. Integrate Documentation into Reporting

Challenges

A critical step for navigating online abuse is documenting that it happened. Documentation enables users to save evidence and share it with their allies and employers, law enforcement, and legal counsel. People experiencing online abuse can lose evidence if the abuser deletes the abusive content or if the content is reported to platforms and removed.

When you have hundreds coming at you and your Twitter feed changes constantly, you can’t keep up with the speed. The [messages] that were problematic in the beginning, you just lose them because you don’t want to scroll through five different pages to find the tweet.

Jin Ding, chief of staff and operations at the Center for Public Integrity

Documentation should dovetail smoothly with the process of reporting abuse to platforms. At present, however, most platforms do not offer any kind of documentation feature, let alone one that is integrated with the reporting process. In fact, users are sometimes hesitant to report abuse because they are afraid of losing evidence of the abusive behavior. “People are concerned that reporting things will actually result in it being deleted from the platform in such a way that they can’t recover it for law enforcement or litigation purposes,” explains Leigh Honeywell, founder and CEO of Tall Poppy. In the absence of better options, Honeywell often recommends informal documentation methods to those facing abuse.⁶⁷

As a result, users attempting to document abuse often resort to taking screenshots or compiling links in Google Docs⁶⁸, informal methods which may prove inadequate to use as evidence in court or in escalating a case to a social media platform.⁶⁹ Furthermore, users not only have to manually report every individual piece of abusive content they receive, but they also have to manually document it, multiplying the steps they have to take to manage abuse. Jin Ding, chief of staff and operations at the Center for Public Integrity, emphasized the challenge of taking these additional steps to document abuse, especially in the face of a cyber mob. “When you have hundreds coming at you and your Twitter feed changes constantly, you can’t keep up with the speed. The [messages] that were problematic in the beginning, you just lose them because you don’t want to scroll through five different pages to find the tweet.”⁷⁰

Finally, accessible documentation is absolutely critical for newsrooms, publishers, and other institutions to protect the safety of their staff and freelancers and to assess risk. Jareen Imam, currently senior content and editorial manager at Amazon and previously director of social newsgathering at NBC, emphasized the importance of documentation in assessing the potential for escalation to physical attacks. “Documenting this stuff is important because there are sometimes repeated individuals that keep coming back to harass you, and if you don’t have a good sense of who they are, and get some kind of risk assessment, you never know what could happen.”⁷¹

“I don’t know why it’s not rote, built into the systems, that when you press ‘report’ on a comment or a link, that the link or comment isn’t copied and pasted into the report,” says Stephanie Brumsey, a segment producer for MSNBC. “Why is the entire onus on the victim and all the companies that the victims work for?”⁷²

Recommendations

Seamlessly integrate documentation into the reporting process. When users report abusive content, the reporting mechanism should give them the option to automatically collect and save all publicly available information about the abusive content or account that they reported (e.g., time and date the abusive comment was made, the account responsible for the abusive comment, the post on which the comment was made, etc.). The documentation could be emailed to users and/or saved in their reporting dashboard (see section 1).
Enable users to export a structured document with a detailed record and history of the abusive content they reported. Users should be able to share this documentation with their employers, allies, law enforcement, or legal counsel as needed, to show evidence of the harassment they are experiencing.
Provide API access that allows users to submit reports to social media platforms directly via vetted third-party anti-harassment tools. When people use third-party anti-harassment tools to help with blocking, muting, and documentation (see the case study below), they often have to then separately report the abuse directly on the platform itself. Enabling users to submit reports directly from within trusted third-party tools⁷³ can condense documentation and reporting into a single action, rather than requiring users to do twice the work. Additionally, users should be able to enable third-party tools to capture any additional contextual information that they submitted when reporting abuse so that they do not have to reenter that information manually into the third-party tool.

Why is the entire onus on the victim and all the companies that the victims work for?

Stephanie Brumsey, segment producer for MSNBC

Product case study: TRFilter

In 2022 the Thomson Reuters Foundation, in collaboration with Google’s Jigsaw division, launched a third-party tool called TRFilter to help media professionals manage online abuse on Twitter. TRFilter features a dashboard that enables users to document the harassment that they experience and then export that documentation to share with others. The dashboard uses machine learning to identify and highlight potentially abusive tweets or comments, assigns this content a toxicity rating, and tags this content by type of abusive tactic. Users can then select one or more of these toxic tweets or comments—manually or automatically by type of harassment or toxicity rating—and export them in a single document that packages all the key information. The dashboard has customizable features, allowing users to blur content to reduce exposure and trauma. Previously, users could also bulk block and bulk mute accounts and content from within TRFilter, but due to Twitter’s new API licensing rules,⁷³ users can no longer do so.⁷⁴ From within TRFilter, users also cannot actually report abuse; to do that, users have to log into Twitter. TRFilter is a model for the kinds of tools that platforms could integrate directly into their reporting processes or, at the very least, encourage, integrate, and support.

Screenshot of TRFilter, a third-party tool that helps media professionals manage online abuse on Twitter. (Screenshot captured by Meedan, August 26, 2022.)

7. Make It Easier to Access Support When Reporting

Challenges

Facing an influx of abuse—and reporting all of it—can be isolating, exhausting, and time-consuming. Targets of abuse often seek out and benefit from the support of friends, family, and colleagues to cope with the emotional distress they experience, including having friends read and address abusive content for them.⁷⁵ In a study of HeartMob, a third-party platform that offers individuals facing harassment the ability to access peer support, users most frequently requested help reporting abuse to social media platforms; and yet, HeartMob users rarely received that type of support because of the lack of integrated allyship reporting options on social media platforms.⁷⁶ In order to reduce exposure and trauma, people experiencing harassment need to be able to turn to their allies for help reviewing and reporting abusive content.⁷⁷ Azmina Dhrodia, former senior policy manager for gender and data rights at the World Wide Web Foundation, noted the benefit of seeking the support of allies, but pointed out that “It’d be nice to have a way to know that friends are helping you report without having to be in constant conversation across other messaging platforms, which is very time-consuming.”⁷⁸

Sophisticated delegation features that enable gated access to a user’s account—so that they don’t need to share account login credentials with allies—have already been implemented for other purposes, including the Teams feature for TweetDeck,⁷⁹ and channel management features on YouTube.⁸⁰ Third-party apps designed to help users navigate online abuse, such as Block Party, have also created sophisticated delegated access features; Block Party’s “Helper” feature, for example, enables users facing abuse on Twitter to designate helpers that can block and mute on their behalf, but cannot post on their account or access their direct messages.⁸¹ Unfortunately, instead of supporting third-party apps like Block Party and integrating features like delegated access, Twitter and other social media platforms are increasingly making it impossible for such apps and features to survive.⁸²

Finally, it is worth noting that social media platforms sometimes act differently on abusive content that is reported by an ally, as opposed to content reported directly by the impacted user. In some reporting mechanisms, users have to indicate whether they are reporting on behalf of themselves or a third-party. For example, Meta’s policies require users to self-report specific forms of abuse, including “some types of bullying and harassment like unwanted manipulated imagery or positive physical descriptions…because it helps us understand that the person targeted feels bullied or harassed.”⁸³ Policies that require self-reporting can help prevent deliberate and unintentional misuse of reporting mechanisms, such as allies mistakenly reporting content as abusive because they do not understand its context. However, such policies can also further confuse and burden those facing abuse because users may not understand if and when they can ask their allies for help.⁸⁴

Recommendations

Enable users to delegate reporting of abusive content to “ally” accounts—trusted friends or colleagues whom they can authorize to report on their behalf. Ensure that users can specify whether they or their delegate will receive notifications on reporting outcomes. Ensure users can provide an end date to delegation authorization and revoke it at any time.
Communicate clearly to users, in the reporting flow itself, if a report will only be moderated when submitted directly by the target of abuse, rather than by an ally.
When a user—or their delegated ally—reports abusive content, direct them to a set of resources offering: a) additional support for the targeted user, b) guidance for the ally in supporting the target, and c) resources to help the ally to manage their own well-being (such as hotlines, tip sheets, help centers, etc.).
Spotlight and fund the work of civil society organizations that provide resources, training, incident response, and hotline support to individuals facing online abuse.

Product case study: Facebook self-harm support screens/pop-ups

On Facebook, when an ally reports content created by another user that seems to indicate or promote self-harm, the ally is shown a support screen that offers guidance and resources on how to help users engaging in or discussing self-harm. This feature, which was introduced in 2017 and is still active today, serves as a model for how reporting mechanisms can provide allies with just-in-time guidance on supporting those facing abuse.

These screenshots show the reporting flow users see when they report a live stream for self-harm content, which includes a support screen that offers guidance and resources on how to support someone engaging in self-harm. (Screenshot pulled from Building a Safer Community With New Suicide Prevention Tools, accessed May 2023.)

Conclusion

Mechanisms to report abuse to social media platforms are a critical part of the larger content moderation process—and they need significant improvement to better protect users and safeguard free expression online. We need reporting mechanisms that are more user-friendly, accessible, transparent, efficient, and effective, with clearer and more regular communication between the user and the platform.

We need reporting mechanisms that are more user-friendly, accessible, transparent, efficient, and effective, with clearer and more regular communication between the user and the platform.

There have been some improvements to reporting systems in recent years, but these gains are fragile and insufficient. Twitter, for example, has gradually introduced more advanced reporting features, but that progress ground to a halt once Elon Musk bought the platform and—among other actions—drastically reduced the Trust and Safety staff overseeing content moderation and user reporting.⁸⁵ This pattern is playing out across the industry, with companies such as Meta, Google, and Twitter hiring fewer employees to Trust and Safety teams.⁸⁶ At the same time, new platforms and technologies, from Clubhouse and the Metaverse to ChatGPT, are introducing new mediums for abuse, while often failing to implement even the most basic features for reporting.⁸⁷

These inconsistencies across the industry highlight the need for clear standards for minimum viable reporting systems on social media platforms. The protection of users and of free expression online should not be dependent on the decision-making of a single executive or platform. This report maps out how social media companies can collectively begin to reimagine reporting mechanisms to increase transparency, empower users, and reduce the chilling effect of online abuse—but it is up to these companies to take up that challenge.

Methodology

Scope

For this report, PEN America and Meedan set out to better understand how creative and media professionals report online abuse to social media platforms, how reporting mechanisms actually work, and how these mechanisms can be improved. We centered our work on the experiences of those disproportionately attacked online for their identity and profession, specifically writers, journalists, content creators, and human rights activists, especially those who identify as women, LGBTQ+ individuals, people of color, and/or belonging to religious or ethnic minorities.

Interviews were conducted primarily with creative and media professionals based in the United States, as well as several participants based in the United Kingdom and Canada. We fully acknowledge, however, that many of the technology companies analyzed in this report have a global user base, and one of the central challenges to curtailing online abuse is the blanket application of United States–based rules, strategies, and cultural norms globally.⁸⁸ Throughout this report, we strove to take into account the ways that changes to reporting mechanisms on global platforms could play out in regions and geopolitical contexts outside the United States and North America. To that end, we interviewed several experts and creators who work globally, including multiple reporters, activists, and experts based in Lebanon, where Meedan has especially strong partnerships.

We focused our analysis on the reporting mechanisms for Meta (Facebook and Instagram), YouTube, Twitter, and TikTok. We included Meta, Twitter, and YouTube because they consistently come up in studies as the platforms on which creative and media professionals face the most abuse.⁸⁹ We included TikTok because it is a platform with growing significance for creative and media professionals.⁹⁰ We requested comments from Meta, Twitter, YouTube, and TikTok. Meta and TikTok provided detailed responses, which are cited throughout this paper. Twitter and YouTube did not respond to our requests, despite repeated outreach and follow-ups.

Methods

This paper draws on PEN America and Meedan’s extensive experience protecting and supporting creative and media professionals facing online abuse, including by providing resources and training to tens of thousands of writers, journalists, artists, and creators, as well as their allies and employers. In addition, between October 2021 and April 2023, the report’s researchers

conducted 21 semi-structured interviews (one to three hours) with experts, journalists, writers, human rights activists, and content creators, alongside follow-up correspondences, ongoing communication about product ideas, and documentation of new and evolving abuse cases—supplemented with crisis support assistance;
ran user experience walk-throughs for existing social media reporting flows, interviewing participants in real time;
conducted documentation and analysis of reporting flows for Facebook, Instagram, YouTube, Twitter, and TikTok;
reviewed public and private case studies of online abuse where reporting took place, including Facebook Oversight Board decisions and social media posts about reporting;
reviewed social media company policy pages and blog posts about reporting-related feature updates; and
conducted extensive desk research, including a literature review of research and reports on online abuse and on reporting systems.

Acknowledgements

This report was written by Kat Lo, content moderation lead at Meedan, and Viktorya Vilk, director for digital safety and free expression at PEN America. Jazilah Salam, program coordinator for digital safety and free expression at PEN America, managed the project, including editing and research. Azza El-Masri, PhD student at the University of Texas at Austin, conducted additional interviews and research. PEN America’s Chief Program Officer of Free Expression Programs, Summer Lopez, reviewed the report and offered thoughtful feedback, alongside PEN America staff James Tager, Ryan Howzell, Jeje Mohamed, Liz Woolery, and Aashna Agarwal, and Meedan staff Jenna Sherman, Nat Gyenes, and Megan Marrelli. PEN America would also like to thank the fellows whose research, fact-checking, and proofreading made this report possible: Marisol Estrella, Lulia Pan, Lauren Katz, Sara Gronich, Ciara Moezidis, Bryn Carlson, Raya Tarawneh, and Arnold Foda.

PEN America extends special thanks to the following experts for providing invaluable input on this report: Liz Lee, founder of OnlineSOS, and Leigh Honeywell, founder and CEO of Tall Poppy. PEN America is also deeply grateful to the many journalists, writers, creators, scholars, technologists, civil society advocates, and other experts who agreed to be interviewed for this report, including those who are not acknowledged by name. PEN America appreciates the responsiveness of the representatives at Meta, TikTok, and the Thomson Reuters Foundation in our exchanges.

Our deep abiding appreciation goes to the Democracy Fund and Craig Newmark Philanthropies for their support of this project. PEN America also receives financial support from Google and Meta, but those funds did not support the research, writing, or publication of this report.

The report was edited by Carol Balistreri. Kipp Jones at Meedan did graphic design.