No Excuse for Abuse

What Social Media Companies Can Do Now to Combat Online Harassment and Empower Users

No Excuse for Abuse

March 31, 2021

Home / Reports / No Excuse for Abuse

Key Findings

Online abuse strains the mental and physical health of its targets, and can cause people to self-censor, avoid certain subjects, or leave their professions altogether.

When online abuse drives women, LGBTQIA+, BIPOC, and minority writers and journalists to leave industries that are predominantly male, heteronormative, and white, public discourse becomes less open and less free.

Social media platforms should adopt proactive measures that empower users to reduce risk and minimize exposure, reactive measures that facilitate response and alleviate harm; and accountability measures that deter abusive behavior.

Writer and PEN America Trustee Jennifer Finney Boylan draws a connection between Facebook, Twitter, and the World’s Most Dangerous Amusement Park

Read the letter >>

PEN America Experts:

Viktorya Vilk

Director, Digital Safety and Free Expression

Elodie Vialle

Senior Advisor, Online Abuse Defense Program

Download Executive Summary

Introduction

Online abuse—from violent threats and hateful slurs to sexual harassment, impersonation, and doxing—is a pervasive and growing problem.¹ Nearly half of Americans report having experienced it,² and two-thirds say they have witnessed it.³ But not everyone is subjected to the same degree of harassment. Certain groups are disproportionately targeted for their identity and profession. Because writers and journalists conduct so much of their work online and in public, they are especially susceptible to such harassment.⁴ Among writers and journalists, the most targeted are those who identify as women, BIPOC, LGBTQIA+, and/or members of religious or ethnic minorities.⁵ Online abuse is intended to intimidate and censor. When voices are silenced and expression is chilled, public discourse suffers. By reducing the harmful impact of online harassment, platforms like Twitter, Facebook, and Instagram can ensure that social media becomes more open and equitable for all users. In this report, PEN America proposes concrete, actionable changes that social media companies can and should make immediately to the design of their platforms to protect people from online abuse—without jeopardizing free expression.

If you’re going to be a journalist, there is an expectation to be on social media. I feel that I have no choice. The number of followers is something employers look at. This is unfair, because there are not a lot of resources to protect you. No matter what I say about race, there will be some blowback. Even if I say nothing, when my colleague who is a white man takes positions on racism, trolls come after me on social media.

Jami Floyd, senior editor of the Justice and Race Unit at New York Public Radio

The devastating impact of online abuse

Writers and journalists are caught in an increasingly untenable double bind. They often depend on social media platforms—especially Twitter, Facebook, and Instagram—to conduct research, connect with sources, keep up with breaking news, promote and publish their stories, and secure professional opportunities.⁶ Yet their visibility and the very nature of their work—in challenging the status quo, holding the powerful accountable, and sharing analysis and opinions—can make them lightning rods for online abuse, especially if they belong to frequently targeted groups and/or if they cover beats such as feminism, politics, or race.⁷“If you’re going to be a journalist, there is an expectation to be on social media. I feel that I have no choice. The number of followers is something employers look at,” says Jami Floyd, senior editor of the Justice and Race Unit at New York Public Radio. “This is unfair, because there are not a lot of resources to protect you. No matter what I say about race, there will be some blowback. Even if I say nothing, when my colleague who is a white man takes positions on racism, trolls come after me on social media.”⁸

A 2018 study conducted by TrollBusters and the International Women’s Media Foundation (IWMF) found that 63 percent of women media workers in the United States have been threatened or harassed online at least once,⁹ a number significantly higher than the national average for the general population.¹⁰ Often women are targeted in direct response to their identities.¹¹ “I am often harassed online when I cover white nationalism and anti-Semitism, especially in politics or when perpetrated by state actors,” says Laura E. Adkins, a journalist and opinion editor of the Jewish Telegraphic Agency. “My face has even been photoshopped into an image of Jews dying in the gas chambers.”¹² Individuals at the intersection of multiple identities, especially women of color, experience the most abuse—by far.¹³

The consequences are dire. Online abuse strains the mental and physical health of its targets and can lead to stress, anxiety, fear, and depression.¹⁴ In extreme cases, it can escalate to physical violence and even murder.¹⁵ Because the risks to health and safety are very real, online abuse has forced some people to censor themselves, avoid certain subjects, step away from social media,¹⁶ or leave their professions altogether.¹⁷ Dr. Michelle Ferrier, a journalist who founded the anti-harassment nonprofit TrollBusters after facing relentless racist and sexist abuse online, recalls: “I went to management. I went to the police. I went to the FBI, CIA. The Committee to Protect Journalists took my case to the Department of Justice. Nothing changed. But I did. I changed as a person. I became angrier. More wary and withdrawn. I had police patrolling my neighborhood. I quit my job to protect my family and young children.”¹⁸

When online abuse drives women, LGBTQIA+, BIPOC, and minority writers and journalists to leave industries that are predominantly male, heteronormative, and white, public discourse becomes less open and less free.¹⁹ Individual harms have systemic consequences: undermining the advancement of equity and inclusion, constraining press freedom, and chilling free expression.

I went to management. I went to the police. I went to the FBI, CIA. The Committee to Protect Journalists took my case to the Department of Justice. Nothing changed. But I did. I changed as a person. I became angrier. More wary and withdrawn. I had police patrolling my neighborhood. I quit my job to protect my family and young children.

Dr. Michelle Ferrier, founder of TrollBusters and executive director of Media Innovation Collaboratory

Shouting into the void: inadequate platform response

Hate and harassment did not begin with the rise of social media. But because sustaining user attention and maximizing engagement underpins the business model of these platforms, they are built to prioritize immediacy, emotional impact, and virality. As a result, they also amplify abusive behavior.²⁰ In prioritizing engagement over safety, many social media companies were slow to implement even basic features to address online harassment. When Twitter launched in 2006, users could report abuse only by tracking down and filling out a lengthy form for each individual abusive comment. The platform did not integrate a reporting button into the app until 2013;²¹ it offered a block feature (to limit communications with an abuser) early on, but did not provide a mute feature (to hide abusive comments without alerting and possibly antagonizing the abuser) until 2014.²² While Facebook offered integrated reporting, blocking, and unfriending features within several years of its launch in 2004,²³ it has since lagged behind in adding new features designed to address abuse. The platform only enabled users to ignore abusive accounts in direct messages in 2017 and to report abuse on someone else’s behalf in 2018.²⁴When it launched in 2010, Instagram also required users to fill out a separate form to report abuse, and its rudimentary safety guidelines advised users to manually delete any harassing comments.²⁵ Since 2016, the platform has gradually ramped up its efforts to address online harassment, pulling ahead of Facebook and Twitter, although Instagram did not actually introduce a mute button until 2018.²⁶

All of these features were added only after many women, people of color, religious and ethnic minorities, and LGBTQIA+ people, including journalists and politicians, had endured countless high-profile abuse campaigns and spent years advocating for change, applying pressure, and generating public outrage.²⁷This timeline indicates a much broader issue in tech: in an industry that has boasted of its willingness to “move fast and break things,”²⁸ efforts to protect vulnerable users are just not moving fast enough.

Users have noticed. According to a 2021 study from Pew Research Center, nearly 80 percent of Americans believe that social media companies are not doing enough to address online harassment.²⁹ Many of the experts and journalists PEN America consulted for this report concurred. Jaclyn Friedman, a writer and founder of Women, Action & the Media who has advocated with platforms to address abuse, says she often feels like she’s “shouting into a void because there’s no transparency or accountability.”³⁰

Protest against Facebook’s role in spreading online harm in San Francisco in November 2020. Photo by AP/Jeff Chiu

There is a growing international consensus that the private companies that maintain dominant social media platforms have a responsibility, in accordance with international human rights law and principles, to reduce the harmful impact of abuse on their platforms and ensure that they remain conducive to free expression.³¹ According to the United Nations’ Guiding Principles for Business and Human Rights (UNGPs), corporations must “avoid infringing on the human rights of others and should address adverse human rights impacts with which they are involved.”³² In March 2021, Facebook released a Corporate Human Rights Policy rooted in the UNGPs, which makes an explicit commitment to protecting the safety of human rights defenders, including “professional and citizen journalists” and “members of vulnerable groups advocating for their rights,” from online attacks.³³ The UNGPs further mandate that states must ensure that corporations live up to their obligations, which “requires taking appropriate steps to prevent, investigate, punish and redress such abuse through effective policies, legislation, regulations and adjudication.”³⁴

I am often harassed online when I cover white nationalism and anti-Semitism, especially in politics or when perpetrated by state actors. My face has even been photoshopped into an image of Jews dying in the gas chambers.

Laura E. Adkins, journalist and opinion editor of the Jewish Telegraphic Agency

**What can platforms do now to reduce the burden of online abuse?**

In this report, PEN America asks: What can social media companies do now to ensure that users disproportionately impacted by online abuse receive better protection and support? How can social media companies build safer spaces online? How can technology companies, from giants like Facebook and Twitter to small startups, design in-platform features and third-party tools that empower targets of abuse and their allies and disarm abusive users, while preserving free expression? What’s working, what can be improved, and where are the gaps? Our recommendations include both proactive measures that empower users to reduce risk and minimize exposure and reactive measures that facilitate response and alleviate harm; and accountability measures that deter abusive behavior.

Among our principal recommendations, we propose that social media companies should:

Build shields that enable users to proactively filter abusive content (across feeds, threads, comments, replies, direct messages, etc.) and quarantine it in a dashboard, where they can review and address it with the help of trusted allies.
Enable users to assemble rapid response teams and delegate account access, so that trusted allies can jump in to provide targeted assistance, from mobilizing supportive communities to helping document, block, mute, and report abuse.
Create a documentation feature that allows users to quickly and easily record evidence of abuse—capturing screenshots, hyperlinks, and other publicly available data automatically or with one click—which is critical for communicating with employers, engaging with law enforcement, and pursuing legal action.
Create safety modes that make it easier to customize privacy and security settings, visibility snapshots that show how adjusting settings impacts reach, and identities that enable users to draw boundaries between the personal and the professional with just a few clicks.
For extreme or overwhelming abuse, create an SOS button that users could activate to instantly trigger additional in-platform protections and an emergency hotline (phone or chat) that provides personalized, trauma-informed support in real time.
Create a transparent system of escalating penalties for abusive behavior—including warnings, strikes, nudges, temporary functionality limitations, and suspensions, as well as content takedowns and account bans—and spell out these penalties for users every step of the way.

Our proposals are rooted in the experiences of writers and journalists who identify as women, BIPOC, LGBTQIA+, and/or members of religious or ethnic minorities in the United States, where PEN America’s expertise on online abuse is strongest. We recognize, however, that online abuse is a global problem and endeavor to note the risks and ramifications of applying strategies conceived in and for the United States internationally.³⁵ We focus on Twitter, Facebook, and Instagram—because United States-based writers and journalists rely on these platforms most in their work,³⁶ and because it is on these platforms that United States-based users report experiencing the most abuse.³⁷ But our recommendations are relevant to all technology companies that design products to facilitate communication and social interaction.

We draw a distinction between casual and committed abuse: the former is more organic and plays out primarily among individuals; the latter is premeditated, coordinated, and perpetrated by well-resourced groups. We make the case that technology companies need to better protect and support users facing both day-to-day abuse and rapidly escalating threats and harassment campaigns. While we propose tools and features that can both disarm abusers and empower targets and their allies, we recognize that the lines between abuser, target, and ally are not always clear-cut. In a heavily polarized environment, online abuse can be multidirectional. Though abusive trolls are often thought of as a “vocal and antisocial minority,” researchers at Stanford and Cornell Universities stress that “anyone can become a troll.”³⁸ Research conducted in the gaming industry found that the vast majority of toxicity came not from committed repeat abusers but from regular users “just having a bad day.”³⁹Because a user can be either an abuser or a target at any time, tools and features designed to address online abuse must approach it as a behavior—not an identity.

No single strategy to fight online abuse will be perfect or future-proof. Any tool or feature for mitigating online abuse could have unintended consequences or be used in ways counter to its intended purpose. “You have to design all of these abuse reporting tools with the knowledge that they are going to be misused,” explains Leigh Honeywell, co-founder and CEO of Tall Poppy, a company that provides protection for individuals and institutions online.⁴⁰ Ensuring that systems are designed to empower users rather than simply prohibit bad behavior can help mitigate those risks, preserving freedom while also becoming more resilient to evolving threats.

If technology companies are serious about reducing the harm of online abuse, they must prioritize understanding the experiences and meeting the needs of their most targeted users. Every step of the way, platforms need to “center the voices of those who are directly impacted by the outcome of the design process,”⁴¹ argues Dr. Sasha Costanza-Chock, associate professor of civic media at MIT. Moreover, to build features and tools that address the needs of vulnerable communities, technology companies need staff, consultation, and testing efforts that reflect the perspectives and experiences of those communities. Staff with a diverse range of identities and backgrounds need to be represented across the organization—among designers, engineers, product managers, trust and safety teams, etc.—and they need to have the power to make decisions and set priorities. If platforms can build better tools and features to protect writers and journalists who identify as women, BIPOC, LGBTQIA+, and members of religious or ethnic minorities, they can better serve all users who experience abuse.

You can’t have free expression of ideas if people have to worry that they’re going to get doxed or they’re going to get threatened. So if we could focus the conversation on how it is that we can create the conditions for free speech—free speech for reporters, free speech for women, free speech for people of color, free speech for people who are targeted offline—that is the conversation we have to have.

Mary Anne Franks, president of the Cyber Civil Rights Initiative and professor of law at the University of Miami

As an organization of writers committed to defending freedom of expression, PEN America views online abuse as a threat to the very principles we fight to uphold. When people stop speaking out and writing about certain topics due to fear of reprisal, everyone loses. Even more troubling, this threat is most acute when people are trying to engage with some of the most complex, controversial, and urgent questions facing our society—questions about politics, race, religion, gender and sexuality, and domestic and international public policy. Democratic structures depend on a robust, healthy discourse in which every member of society can engage. “You can’t have free expression of ideas if people have to worry that they’re going to get doxed or they’re going to get threatened,” notes Mary Anne Franks, president of the Cyber Civil Rights Initiative and professor of law at the University of Miami. “So if we could focus the conversation on how it is that we can create the conditions for free speech—free speech for reporters, free speech for women, free speech for people of color, free speech for people who are targeted offline—that is the conversation we have to have.”⁴²

At the same time, we are leery of giving private companies unchecked power to police speech. Contentious, combative, and even offensive views often do not rise to the level of speech that should be banned, removed, or suppressed. Content moderation can be a blunt instrument. Efforts to combat online harassment that rely too heavily on taking down content, especially given the challenges of implicit bias in both human and automated moderation, risk sweeping up legitimate disagreement and critique and may further marginalize the very individuals and communities such measures are meant to protect. A post that calls for violence against a group or individual, for instance, should not be treated the same as a post that might use similar language to decry that very behavior.⁴³ Furthermore, some tools that mitigate abuse can be exploited to silence the marginalized and censor dissenting views. More aggressive policing of content by platforms must be accompanied by stepped-up mechanisms that allow users to appeal and achieve timely resolution in instances where they believe that content has been unjustifiably suppressed or removed. Throughout this report, in laying out our recommendations, we address the tensions that can arise in countering abuse while protecting free expression, and propose strategies to mitigate weaponization and unintended consequences. While the challenges and tensions baked into reducing online harms are real, technology companies have the resources and power to find solutions. Writers, journalists, and other vulnerable users have, for too long, endured relentless abuse on the very social media platforms that they need to do their jobs. It’s time for technology companies to step up.

Empowering Targeted Users and Their Allies

In this section we lay out proactive and reactive measures that platforms can take to empower users targeted by online abuse and their allies. Proactive measures protect users from online abuse before it happens or lessen its impact by giving its targets greater control. Unfortunately proactive measures can sometimes be fraught from a free expression standpoint. Sweeping or sloppy implementation, often rooted in algorithmic and human biases abetted by a lack of transparency, can result in censorship, including of creative and journalistic content.⁴⁴ Reactive measures, such as blocking and muting to limit interaction with abusive content, mitigate the harms of online abuse once it is underway but do little to shield targets. Such features sidestep many of the first-order free expression risks associated with proactive measures but are often, on their own, insufficient to protect users from abuse.

It is important to bear in mind that both proactive and reactive measures are themselves susceptible to gaming and weaponization.⁴⁵ In many cases, the difference between an effective strategy and an ineffective or overly restrictive one depends not only on policies but also on the specifics of how tools and features are designed and whom they prioritize and serve. Our recommendations aim to strike a balance between protecting those who are disproportionately targeted by online abuse for their identity and profession and safeguarding free expression.

Proactive measures: Reducing risk and exposure

Proactive measures are often more effective than reactive ones because they can protect users from encountering abusive content—limiting their stress and trauma and empowering them to express themselves more freely. They can also enable users to reduce their risk and calibrate their potential exposure by, for example, fine-tuning their privacy and security settings and creating distinctions between their personal and professional identities online.

Today most major platforms provide some proactive protections, but these are often difficult to find, understand, and use. Many of the writers and journalists PEN America works with, including those interviewed for this report, were unaware of existing features and tools and found themselves scrambling to deal with online harassment only after it had been unleashed. “Young journalists,” says Christina Bellantoni, a professor at the USC Annenberg School for Communication and Journalism, often “don’t familiarize themselves with policies and tools because they don’t predict they will ever face problems. When they do, it’s too late. Tools to help young journalists learn more about privacy settings from the outset would go a long way.”⁴⁶ Social media companies should design and build stronger proactive measures, make them more accessible and user-friendly, and educate users about them.

I wasn’t prepared emotionally for the abuse I saw on my screen and, as a freelancer, received little support from publications. Now I sometimes avoid reporting on certain topics, or I publish pieces, but I just won’t post on social media because I am afraid of the blowback and would rather not deal with it. If I had additional tools to deal with abuse on social media, I would definitely use them. I could cover some topics and post them.

Jasmine Bager, journalist

Safety modes and visibility snapshots: Making it easier to control privacy and security

The challenge: Writers and journalists are especially vulnerable to hacking, impersonation, and other forms of abuse predicated on accessing or exposing private information.⁴⁷ To reduce risks like these, users need to be able to easily fine-tune the privacy and security settings on their social media accounts, especially because platforms’ default settings often maximize the public visibility of content.⁴⁸

Some platforms have gradually given users more granular control over their settings, which is a positive trend.⁴⁹ Providing users with maximum choice and control without overwhelming them is a difficult balancing act.⁵⁰ The usability of these tools is just as important as their sophistication. “Every year —like clockwork —Facebook has responded to criticisms of lackluster security and data exposure by rolling out ‘improvements’ to its privacy offerings,” writes journalist Matthew Keys. “More often than not, Facebook heralds the changes as enabling users to take better control of their data. In reality, the changes lead to confusion and frustration.”⁵¹

Adding to the problem, there is no consistency across platforms in how privacy and security settings work or the language used to describe them. These settings are often buried within apps or separate help centers and are time-consuming and challenging to find and adjust.⁵² Even “Google’s own engineers,” according to Ars Technica, have been “confused” by its privacy settings.⁵³

While many writers and journalists want to maximize their visibility and user engagement, if they find themselves in the midst of an onslaught of abuse—or anticipate one—they need to quickly and easily reduce their visibility until the trouble has passed. Because tightening privacy has real trade-offs, understanding the implications of adjusting specific settings is critically important. As journalist Jareen Imam points out, when users find it “confusing to see what is public and what is not,” they struggle to weigh trade-offs and make informed choices.⁵⁴

Existing features and tools: As some platforms add increasingly granular choices for adjusting settings, they are also experimenting with features to streamline the process. With Twitter’s “protect my tweets” and Instagram’s “private account” features, users can now tighten their privacy with a single click, restricting who can see their content or follow them. But they cannot then customize settings within these privacy modes to maintain at least some visibility and reach.⁵⁵

Facebook’s settings are notoriously complicated, and users don’t have a one-click option to tighten privacy and security throughout an account.⁵⁶ Its users can proactively choose to limit the visibility of individual posts,⁵⁷ but they cannot make certain types of content, such as a profile photo, private,⁵⁸ which can result in the misuse of profile photos for impersonation or non-consensual intimate imagery.⁵⁹ The platform does offer user-friendly, interactive privacy and security checkups.⁶⁰

In trying to comprehend byzantine settings, some users have turned to external sources. Media outlets and nonprofits, including PEN America, offer writers and journalists training and guidance on tightening privacy and security on social media platforms.⁶¹ Third-party tools such as Jumbo and Tall Poppy walk users through adjusting settings step by step.⁶² While external tools and training are useful and badly needed, few writers, journalists, and publishers currently have the resources or awareness to take advantage of them.⁶³ Moreover, the very existence of such tools and training is indicative of the difficulty of navigating privacy and security within the platforms themselves.

Recommendations: Platforms should provide users with robust, intuitive, user-friendly tools to control their privacy and security settings. Specifically, platforms should:

Empower users to create and save “safety modes”—multiple, distinct configurations of privacy and security settings that they can then quickly activate with one click when needed.
- Twitter and Instagram should give users the option to fine-tune existing safety modes (“protect my tweets” and “private account,” respectively) after users activate them. These modes are currently limited in functionality because they are binary (i.e., the account is either private or not).
- Facebook should introduce a safety mode that allows users to go private with just one click, as Twitter and Instagram have already done, while also ensuring that users can then fine-tune specific settings in the new safety mode.
Introduce “visibility snapshots” that clearly communicate to users, in real time, the implications of the changes they are making as they adjust their security and privacy settings. One solution is to provide users with a snapshot of what is publicly visible, as Facebook does with its “view as” feature.^{⁶⁴“How can I see what my profile looks like to people on Facebook I’m not friends with?,” Facebook Help Center, accessed February 19, 2021, facebook.com/help/288066747875915} Another is to provide an estimate of how many or which types of users (followers, public, etc.) will be able to see a post depending on selected settings.
- Twitter and Instagram should add user-friendly, interactive privacy and security checkups, as Facebook has already done, and introduce visibility snapshots.
- Facebook should enable users to make profile photos private.
Regularly prompt users, via nudges and reminders, to review their security and privacy settings and set up the safety modes detailed above. Prompts could proactively encourage users to reconsider including private information that could put them at risk (such as a date of birth or home address).
Convene a multi-stakeholder coalition of technology companies, civil society organizations, and vulnerable users—or deploy a specific existing coalition such as the Global Network Initiative,⁶⁵ Online Abuse Coalition,⁶⁶ or Trust & Safety Professional Association⁶⁷—to coordinate consistent user experiences and terminology for account security and privacy across platforms.

My Instagram is probably the most personal account that I have. And actually, for a long time, it was a private account. But because of the pandemic, it’s impossible to do reporting without having this public. When you have a private account, you also close yourself off to sources that might want to reach out to tell you something really important … So it’s public now.

Jareen Imam, director of social newsgathering at NBC News

Identities: Distinguishing between the personal and the professional

The challenge: For many writers and journalists, having a presence on social media is a professional necessity.⁶⁸ Yet the boundaries between the personal and professional use of social media accounts are often blurred. The importance of engaging with an audience and building a brand encourages the conflation of the professional with the personal.⁶⁹ As journalist Allegra Hobbs wrote in The Guardian: “All the things that invite derision for influencers—self-promotion, fishing for likes, posting about the minutiae of your life for relatability points—are also integral to the career of a writer online.”⁷⁰ A 2017 analysis of how journalists use Twitter found that they “particularly struggle with” when to be “personal or professional, how to balance broadcasting their message with engagement and how to promote themselves strategically.”⁷¹ While writers and reporters may be mindful of the need for privacy, the challenge, as freelance journalist Eileen Truax explains, is that maintaining a social media presence paves the way for professional opportunities: “Many of the invitations I get to participate in projects come to me because they see my activity on Twitter.”⁷²

This fusion of the personal and professional makes writers and journalists vulnerable. Private information found on social media platforms is weaponized to humiliate, discredit, and intimidate users, their friends, and their families. To mitigate risk, Jason Reich, vice president for corporate security at The New York Times, advises journalists to create distinct personal and professional accounts on social media wherever possible, fine-tune privacy and security settings accordingly, and adjust the information they include for each account.⁷³ But following such procedures is challenging because platforms make it difficult to distinguish between personal and professional accounts, to migrate or share audiences between them, and to target specific audiences. While users can theoretically create and manage multiple accounts on most platforms, in practice a user who decides to create a professional account or page separate from an existing personal one has to start over to rebuild an audience.⁷⁴

The COVID-19 pandemic—which has pushed more creative and media professionals into remote and fully digital work—has intensified this dilemma.⁷⁵ “My Instagram is probably the most personal account that I have,” says Jareen Imam, director of social newsgathering at NBC News. “And actually, for a long time, it was a private account. But because of the pandemic, it’s impossible to do reporting without having this public. When you have a private account, you also close yourself off to sources that might want to reach out to tell you something really important.… So it’s public now.”⁷⁶

Existing features and tools: Twitter⁷⁷ and Instagram⁷⁸ allow individual users to create multiple accounts and toggle easily between them. Facebook, on the other hand, does not allow one user to create more than one account⁷⁹ and requires the use of an “authentic name”—that is, the real name that a user is known by offline.⁸⁰ While Facebook enables users to create “fan pages” and “public figure pages,”⁸¹ these have real limitations: By prioritizing the posts of friends and family in users’ feeds, Facebook favors the personal over the professional, curbing the reach of public-facing pages and creating an incentive to invest in personal profiles.⁸²

All of the platforms analyzed in this paper are gradually giving users more control over their audience. Twitter is testing a feature that allow users to specify who can reply to their tweets⁸³ and recently launched a feature that allows users to hide specific replies.⁸⁴ Facebook gives users more control over the visibility of individual posts, allowing users to choose among “public,” “friends,” or “specific friends.”⁸⁵ Instagram has a feature that lets users create customized groups of “close friends” and share stories in a more targeted way, though it has not yet expanded that feature to posts.⁸⁶ But none of these platforms allow individual users to share or migrate friends and followers among multiple accounts or between profiles and pages.⁸⁷

Recommendations: Platforms should make it easier for users to create and maintain boundaries between their personal and professional identities online while retaining the audiences that they have cultivated. There are multiple ways to achieve this:

Empower users to create and save “safety modes”—multiple, distinct configurations of privacy and security settings that they can then quickly activate with one click when needed.
Give users greater control over who can see their individual posts (i.e., friends/followers versus subsets of friends/followers versus the wider public), which is predicated on the ability to group audiences and target individual posts to subsets of audiences. This is distinct from giving users the ability to go private across an entire account (see “Safety modes,” above).
- Like Twitter and Instagram, Facebook should make it possible for users to create multiple accounts and toggle easily between them, a fundamental, urgently needed shift from its current “one identity” approach. Facebook should also ensure that public figure and fan pages offer audience engagement and reach that are comparable to those of personal profiles.
- Like Facebook, Twitter and Instagram should make it easier for users to specify who can see their posts and allow users to migrate or share audiences between personal and professional online identities.

Mitigating risk: While most individual users are entitled to exert control over who can see and interact with their content, for public officials and entities on social media, transparency and accountability are paramount. The courts recently asserted, for example, that during Donald Trump’s presidency, it was unconstitutional for him to block users on Twitter because he was using his “presidential account” for official communications.⁸⁸ There are multiple related cases currently winding their way through the courts, including the ACLU’s lawsuit against state Senator Roy Scott of Colorado for blocking a constituent on Twitter.⁸⁹ It is especially important that public officials and entities be required to uphold the boundaries between the personal and professional use of social media accounts and ensure that any accounts used to communicate professionally remain open to all constituents. Public officials and entities must also adhere to all relevant laws for record keeping in official, public communications, including on social media.

Fine-tuning privacy and security settings on social media is critical to reducing the risk of hacking, impersonation, doxing, and other forms of abuse predicated on accessing or exposing private information.

Account histories: Managing old content

The challenge: Many writers and journalists have been on social media for over a decade.⁹⁰ They joined in the early days, when platforms like Facebook were used primarily in personal life and privacy settings often defaulted to “public” and were not granular or easily accessible.⁹¹ But the ways that creative and media professionals use these platforms has since broadened in scale, scope, and reach. Writers’ and journalists’ long histories of online activity can be mined for old posts that, when resurfaced and taken out of context, can be deployed to try to shame a target or get them reprimanded or fired.⁹²

Existing features and tools: On Twitter and Instagram, users can delete content piecemeal and cannot easily search through or sort old content, which is cumbersome and impractical.⁹³ In June 2020, Facebook launched “manage activity,” a feature that allows users to filter and review old posts by date or in relation to a particular person and to archive or delete posts individually or in bulk.⁹⁴ Manage activity is an important and useful new feature, but it does not allow users to search by keywords and remains difficult to find. There are multiple third-party tools that allow users to search through and delete old tweets and posts en masse;⁹⁵ however, some of them cost money and most require granting third-party access to sensitive accounts, which poses its own safety risks depending on the cybersecurity and privacy practices (and ethics) of the developers.

Recommendations: Platforms should provide users with integrated and robust features to manage their personal account histories, including the ability to search through old posts, review them, make them private, delete them, and archive them—individually and in bulk. Specifically:

Twitter and Instagram should integrate a feature that allows users to search, review, make private, delete, and archive old content—individually and in bulk.
Facebook should expand its new manage activity feature to enable users to search by keywords and should make this feature more visible and easier to access.

Mitigating risk: PEN America believes that users should have control over their own social media account histories. Users already have the ability to delete old content on most platforms and via multiple third-party tools. But giving users the ability to purge account histories, especially in bulk, does have drawbacks. Abusers can delete old posts that would otherwise serve as evidence in cases of harassment, stalking, or other online harms. And by removing old content, public officials and entities using social media accounts in their official capacities may undermine accountability and transparency. There are ways to mitigate these drawbacks. It is vital that people facing online abuse are able to capture evidence of harmful content, which is needed for engaging law enforcement, pursuing legal action, and escalating cases with the platforms. For that reason, this report advocates for a documentation feature that would make it easier for targets to quickly and easily preserve evidence of abuse (see “Documentation,” below). In the case of public officials or entities deleting account histories, tools that archive the internet, such as the Wayback Machine, are critically important resources for investigative journalism.⁹⁶ There are also federal laws—most centrally the Freedom of Information Act—and state-level laws that require public officials to retain records that may be disclosed to the public; these records include their statements made on social media.⁹⁷ Public officials and entities using social media accounts in their official capacities must adhere to applicable record retention laws, which should apply to social media as they do to other forms of communication.

Other people in my friend circle have to take into account that, being Black and queer, I get more negativity than they would. If they’re white or cisgender or heteronormative, they’ll come back and say, ‘You know what? Jordan is getting a lot of flack, so let’s step up to the plate.’

Jordan, blogger (requested to be identified only by their first name)

Rapid response teams and delegated access: Facilitating allyship

The challenge: Online abuse isolates its targets. A 2018 global study from TrollBusters and the IWMF found that 35 percent of women and nonbinary journalists who had experienced threats or harassment reported “feeling distant or cut off from other people.”⁹⁸ Many people targeted by online abuse suffer in silence because of the stigma, shame, and victim blaming surrounding all forms of harassment.⁹⁹ Often targets have no choice but to engage with hateful or harassing content—in order to monitor, mute, report, and document it—which can be overwhelming, exhausting, and traumatizing.¹⁰⁰

Many of the writers and journalists in PEN America’s network emphasize the importance of receiving support from others in recovering from episodes of online abuse. Jordan, a blogger who requested to be identified only by their first name, explained: “Other people in my friend circle have to take into account that, being Black and queer, I get more negativity than they would. If they’re white or cisgender or heteronormative, they’ll come back and say, ‘You know what? Jordan is getting a lot of flack, so let’s step up to the plate.’”¹⁰¹

Existing features and tools: Users can help one another report abuse on Twitter,¹⁰² Facebook,¹⁰³ and Instagram.¹⁰⁴ But for allies to offer more extensive support—such as blocking or checking direct messages (DMs) on a target’s behalf—they need to have access to the target’s account. In-platform features that securely facilitate allyship are rare, and those that exist were not specifically designed for this purpose. As a result, many targets of online abuse either struggle on their own or resort to ad hoc strategies, such as handing over passwords to allies,¹⁰⁵ which undermines their cybersecurity at precisely the moment when they are most vulnerable to attacks.

On Facebook, the owner of a public page can grant other users “admin” privileges, but this feature is not available for personal Facebook profiles.¹⁰⁶ Similarly, Instagram allows users to share access and designate “roles,” but only on business accounts.¹⁰⁷ Twitter comes closest to supporting delegated access with its “teams” feature in TweetDeck, letting users share access to a single account without using the same password and be granted owner, admin, or contributor status.¹⁰⁸

While useful, these features were designed to facilitate professional productivity and collaboration and meant primarily for institutional accounts or pages.¹⁰⁹ The reality is that, like many users, writers and journalists use social media accounts for both personal and professional purposes and need integrated support mechanisms designed specifically to deal with online abuse. Facebook offers a feature that enables users to proactively select a limited number of trusted friends to help them if they get locked out of their account.¹¹⁰ If the company adapted this feature to allow users to proactively select several trusted friends to serve as a rapid response team during episodes of abuse—and added this feature to its new “registration for journalists”¹¹¹—it could serve as an example for other platforms.

Download the graphic >>

Share on Twitter >>

In PEN America’s trainings and resources, we advise writers and journalists to proactively designate a rapid response team—a small network of trusted allies—who can be called upon to rally broader support and provide specific assistance, such as account monitoring or temporary housing in the event of doxing or threats.¹¹² Several third-party tools and networks are trying to fill the glaring gap in peer support. As Lu Ortiz, founder and executive director of the anti-harassment nonprofit Vita Activa, explains: “Peer support groups are revolutionary because they destigmatize the process of asking for help, provide solidarity, and generate resilience and strategic decision making.”¹¹³ The anti-harassment nonprofit TrollBusters coordinates informal, organic support networks for journalists.¹¹⁴ The anti-harassment nonprofit Hollaback! has developed a platform called HeartMob that provides the targets of abuse with support and resources from a community of volunteers.¹¹⁵ “Our goal,” says co-founder Emily May, “is to reduce trauma for people being harassed online by giving them the immediate support they need.”¹¹⁶ Block Party, a tool currently in beta for Twitter, gives users the ability to assign “helpers” to assist with monitoring, muting, or blocking abuse.¹¹⁷ Squadbox lets users designate a “squad” of supporters to directly receive and manage abusive content in email in-boxes.¹¹⁸ These tools and communities provide models for how platforms could integrate peer support.

In PEN America’s trainings and resources, we advise writers and journalists to proactively designate a rapid response team—a small network of trusted allies—who can be called upon to rally broader support and provide specific assistance, such as account monitoring or temporary housing in the event of doxing or threats.¹¹⁹ Several third-party tools and networks are trying to fill the glaring gap in peer support. As Lu Ortiz, founder and executive director of the anti-harassment nonprofit Vita Activa, explains: “Peer support groups are revolutionary because they destigmatize the process of asking for help, provide solidarity, and generate resilience and strategic decision making.”¹²⁰ The anti-harassment nonprofit TrollBusters coordinates informal, organic support networks for journalists.¹²¹ The anti-harassment nonprofit Hollaback! has developed a platform called HeartMob that provides the targets of abuse with support and resources from a community of volunteers.¹²² “Our goal,” says co-founder Emily May, “is to reduce trauma for people being harassed online by giving them the immediate support they need.”¹²³ Block Party, a tool currently in beta for Twitter, gives users the ability to assign “helpers” to assist with monitoring, muting, or blocking abuse.¹²⁴ Squadbox lets users designate a “squad” of supporters to directly receive and manage abusive content in email in-boxes.¹²⁵ These tools and communities provide models for how platforms could integrate peer support.

Recommendations: Platforms should add new features and integrate third-party tools that facilitate peer support and allyship. Specifically, platforms should:

Enable users to proactively designate a limited number of trusted allies to serve as a rapid response team—a small network of trusted allies who can be called upon to work together or individually to monitor, report, and document abuse that is publicly visible and rally a broader online community to help.
Offer users the ability to grant specific members of their rapid response team access to their accounts, akin to the “delegate” system available on Gmail.¹²⁶“Set Up Mail Delegation—Gmail Help,” Google, accessed January 5, 2021, support.google.com/mail/answer/138350?hl=en These delegates could assist with tasks that require direct account access, such as blocking, muting, and reporting abuse in DMs. Users should be able to control the level of access their delegates have (to public feeds versus private DMs, for example).
- Twitter should integrate its “teams” feature, which is currently available only through TweetDeck, more directly into the primary user experience and empower users to specify exactly which anti-harassment features (monitoring, blocking, muting, reporting, etc.) their delegates can access.Instagram should extend its “roles” feature from business accounts to all accounts.Facebook should extend admin privileges from pages to profiles.
Periodically nudge users to create rapid response teams and assign delegates, in tandem with security checkups, potentially when a user reaches a certain follower threshold or immediately after the user has reported online abuse.

Shield and dashboard: Treating online abuse like spam

The challenge: Regularly interacting with hate and harassment is harmful, and users need to have greater control over their exposure to it. As Larry Rosen, professor emeritus of psychology at California State University, explained to Consumer Reports: “You’re going to start feeling more negative, maybe depressed, more stressed, more anxious. The advice I’d give is to identify where the negative stuff is coming from and hide it all.”¹²⁷

Lessons can be learned from the world of email, where managing and metering spam have been a qualified success. In her book The Internet of Garbage, technologist and journalist Sarah Jeong makes the connection between spam and the “garbage” of harassment. “Dealing with garbage is time-consuming and emotionally taxing,” she writes. And while “patterning harassment directly after anti-spam is not the answer,” there are “obvious parallels.”¹²⁸ Taking inspiration from efforts to reduce the volume and visibility of spam, platforms can do more to proactively identify online abuse, filter it, and hide it—from the feeds, notifications, and DMs, of individual users.

Most major social media platforms already rely on a combination of automation and human moderation to proactively identify certain kinds of harmful content in order to reduce its reach, label it, hide it behind screens, or delete it altogether—for all users.¹²⁹ The challenge with online abuse, however, is that it is heavily context-dependent and can often fall into gray areas that both computers and humans have difficulty adjudicating. Shielding individual users from abusive “garbage” and giving them greater control over whether and how they interact with it can provide an alternative to overzealous proactive content moderation, which can severely undermine free expression for all users.

Existing features and tools: The technology to more accurately and effectively identify and filter context-dependent harmful content is currently being built. Twitter,¹³⁰ Facebook,¹³¹ and Instagram¹³² already allow users to hide, mute, and filter some content in feeds, messages, and notifications, but these features are fairly limited in functionality, largely reactive, and rarely quarantine content.

Tools such as Perspective,¹³³ Coral,¹³⁴ L1ght,¹³⁵ and Sentropy¹³⁶ use machine learning to proactively identify and filter harassment at scale. Many newsrooms and other publishers use Coral and Perspective, for example, to automatically identify and quarantine “toxic” content in the comment sections beneath articles, which human content moderators can then evaluate.¹³⁷

Third-party tools that help individual users (rather than institutions) filter and hide abuse—among them Tune, Block Party, Sentropy Protect, and BodyGuard—have also emerged in recent years. Jigsaw’s Tune is an experimental web browser extension that aims to use machine learning to allow users to adjust the toxicity level of the content they interact with, including content on Facebook and Twitter.¹³⁸ Block Party, currently available on Twitter with the goal of expanding to other platforms, aims to proactively identify potentially abusive accounts, automatically block or mute them, and silo related content; users can then choose to review the accounts, report them, and/or unblock and unmute them.¹³⁹ Block Party founder Tracy Chou experienced egregious online abuse as a woman of color in tech and created the tool to serve as a “spam folder.”¹⁴⁰ She explains: “We need to know what people are saying, we need to collect it, we need to keep an eye on it, but we also need to stop seeing it if we want to preserve ourselves.”¹⁴¹ Both Sentropy Protect and BodyGuard use machine learning to proactively identify and silo abusive content, which users can then review and address; the latter is available for multiple languages on Twitter, Twitch, Instagram, and YouTube.¹⁴²

Recommendation: Platforms should create a shield that enables users to proactively filter abusive content (across feeds, threads, comments, replies, direct messages, etc.) and quarantine it in a dashboard, where users could then review and address the content as needed, with the help of trusted allies.

How would the shield work?

The shield would proactively identify abusive content, filter it out (by hiding it from the the targeted user but not from all users), and automatically quarantine it in the dashboard (see below).
Users could turn on the shield with just one click from within the platform’s primary user experience.
Users could fine-tune the shield to adjust the toxicity level of content they filter out.
Users would receive prompts to turn on or fine-tune the shield when the platform detects unusual account activity.

How would the dashboard work? From within the dashboard, users would be able to:

Review quarantined content to block, mute, or report it—and related accounts—including in bulk. Content should be blurred by default, with the option of revealing it for review. Ideally, content should also be labeled with the relevant abusive tactic (hate, slur, threat, doxing, etc.) to help users and their allies prioritize what to review.
Manually add abusive content to the dashboard that was missed by the shield.
Manually release from the dashboard content that was mistakenly filtered as abusive or that the user does not perceive as abusive.
Document the abuse (see “Documentation,” below).
Activate rapid response teams to provide peer support, including giving trusted allies delegated access to the dashboard to help manage abuse (see “Rapid response teams and delegated access,” above).
Access account privacy and security settings (see “Safety modes,” above).
Access account history management tools (see “Managing account histories,” above).
Access personalized support through an SOS button and emergency hotline (see “Creating an SOS button and emergency hotline,” below).
Access external resources such as mental health hotlines, legal counseling, cybersecurity help, and other direct support.

Mitigating risk: The automated filtering of harmful content is an imperfect science—with false positives, rapidly evolving and coded forms of abuse and hate, and challenges analyzing symbols and images.¹⁴³ Platforms should work more closely with one another, with companies that build third-party tools, and with civil society to create and maintain a shared taxonomy of abusive tactics, terms, symbols, etc., and to create publicly available data sets and heuristics for independent review.¹⁴⁴

Download the graphic >>

Share on Twitter >>

Reactive measures: Facilitating response and reducing harm

Platforms currently offer considerably more options for reacting to online abuse than for proactive protection. Most platforms offer some form of blocking, which is intended to cut off further communication between abuser and target. Nearly all of them also offer some form of muting, which hides abuse from its intended target but not from other users. These mechanisms are vitally important, but they are also inconsistent, insufficient, and inherently limited. Blocking, for example, can be taken as a provocation and escalate abuse, while muting can mask serious threats. No platform analyzed in this report provides a mechanism to track or document abusive content. Most platforms enable users to report abuse, but there is widespread consensus that content moderation systems in general—and reporting features specifically—are often ineffective and cannot adequately keep up with the volume of abuse proliferating on platforms.

Documentation: Recording evidence of abuse

The challenge: All users, including writers and journalists, need to have the ability to maintain a detailed, exportable record of the online abuse they’ve been subjected to. Documentation serves as a prerequisite for engaging with law enforcement or pursuing legal action. It can also help targets track online abuse and facilitate communication with allies and employers. The pressing need for a documentation feature is underscored by the fact that targets can lose evidence if, for instance, an abuser deliberately deletes the content as soon as it has been seen or if the content is reported, determined to be a violation of platform policies, and removed.¹⁴⁵

In 2015, the Electronic Frontier Foundation (EFF) advised digital platforms to build “tools that allow targets of harassment to preserve evidence in a way that law enforcement can understand and use. Abuse reports are currently designed for internet companies’ internal processes, not the legal system.”¹⁴⁶ According to lawyer Carrie Goldberg, who founded a law firm specializing in defending the targets of online abuse, a documentation feature that provides evidence that could be used in court in the United States should record “screenshots of abusive content, URLs, the social media platform on which abuse occurred, the abuser’s username/handle and basic account info, a time and date stamp, and the target’s username/handle.”¹⁴⁷ Ideally this information would be encrypted where sensible to protect the privacy of all parties, as well as to protect the integrity of the records for use in court.

A documentation feature needs to have a simple interface that is easily navigable by non-expert users who may be traumatized or in duress. “Imagine if you could select [abusive] mentions and just put them in a report,” says technologist and researcher Caroline Sinders. “Journalists could easily capture all instances of harassment and forward them to an editor or trusted colleague for advice.”¹⁴⁸ Users need to be able to capture all publicly available data for abusive content with just one click. Better still, a documentation feature could automatically record all key data for any content proactively detected by toxicity filters (see “Building a shield and dashboard,” above).

Existing features and tools: Most social media platforms do not offer any tools that facilitate the documentation of online abuse. Twitter comes closest. When users report abusive tweets, they can request that Twitter email them some key information, but users have to proactively report abuse and proactively request this information.¹⁴⁹ In other words, on Facebook, Instagram, and Twitter, users must manually take screenshots, save links, track metadata, and create logs of their abuse, all of which can be time-consuming and retraumatizing.

As for existing third-party tools, PEN America was able to identify only two that facilitate documentation and are specifically designed for abuse: JSafe, an app in beta developed by the Reynolds Journalism Institute with the Coalition for Women in Journalism, and DocuSafe, from the National Network to End Domestic Violence. Both of these apps still require users to manually track and enter data, but they offer a single place to store and organize it.¹⁵⁰Google’s Jigsaw team informed PEN America that they are “experimenting in the space of documentation and reporting to help targets of online harassment manage and take action on the harassment they receive … building on our experience developing Perspective API, which uses machine learning to identify toxic language.”¹⁵¹

Recommendations: Platforms should develop a documentation feature to empower users and their allies to quickly and easily record evidence of abusive content—capturing screenshots, hyperlinks, and other publicly available data automatically or with one click. This feature is especially important for content that is threatening or heightens the risk of physical violence, such as doxing. Specifically, platforms should:

Enable users to automatically capture all relevant publicly available data for content that is flagged as abusive via user muting, restricting, blocking, or reporting, as well as proactive content filtering. To ensure that the feature meets evidentiary requirements for legal proceedings, platforms need to work with civil society, law enforcement, and legal experts to create documentation standards in each country and jurisdiction where they operate.
Enable users to manually document abusive content with one click and to manually add additional data, including context and relevant hashtags, to supplement automatic documentation.
Enable users to download or export documented abuse so it can then be shared with third parties such as nonprofit organizations, employers, support networks, and legal counsel.
Ensure that the documentation feature, which will require at least some engagement with abusive content, is user-friendly and designed using trauma-informed, ethical frameworks.

Muting, blocking, and restricting: Improving and standardizing existing features

The challenge: Users targeted by online abuse, including writers and journalists, need to have the ability to limit contact with an abuser and to control or limit exposure to abusive content. While most platforms are building increasingly sophisticated features for these purposes, there is no consistency across platforms in terms of language, functionality, or the mitigation of potential unintended consequences.¹⁵² Each feature works somewhat differently on each platform, and not every platform offers every feature, making it confusing and time-consuming for users to understand the reactive measures available to them.

Screenshot demonstrating how to restrict an abusive account on Instagram. Photo by Instagram

Existing features and tools:

Blocking allows users to cut off contact and communication with abusers.¹⁵³ But as Pulsar, an audience intelligence company, explained in a 2018 audit of blocking features: “Blocking remains a highly inconsistent experience on different social platforms.”¹⁵⁴ On Facebook, targets have to block the abuser on messenger and then separately block the abuser’s profile in order to ensure that their public-facing content is no longer visible to the blocked abuser.¹⁵⁵ On Twitter, a blocked abuser cannot see any information on the target’s profile (except their profile photo).¹⁵⁶ On Instagram, most of the target’s account (except their name, profile photo, mutual followers, and number of posts) disappears from the blocked abuser’s view.¹⁵⁷ Instagram offers the most robust blocking features, retroactively removing the comments and likes from blocked accounts and enabling users to select up to 25 comments and then block, en masse, all the accounts posting those comments.¹⁵⁸

Muting allows users to make abusive content invisible, but only from their individual perspective (not from all users). The muted material can be a specific piece of content, a specific user, a keyword, or notifications.¹⁵⁹ Like blocking, muting works differently on each platform, and the options can get granular. Furthermore, different platforms use different terms to refer to the act of hiding content. On Twitter, users can “mute” entire accounts, individual tweets, and replies to their tweets, and they can “mute” content by keyword, emoji, or hashtag. But they cannot mute DMs, only the notifications announcing them, and there is no expiration date for muting.¹⁶⁰ On Facebook, there’s no equivalent to muting, but users can “snooze” accounts or groups for 30 days, “mute” other users’ stories, and permanently “unfollow” posts without unfriending accounts.¹⁶¹ Facebook users can also “block” comments by keywords and “filter for profanity,” but only on pages,¹⁶² not on profiles, and the platform’s muting-like features only partly shield targets from abuse in DMs.¹⁶³ Instagram enables users to “filter” comments by keywords or preset filters,¹⁶⁴ “mute” posts or stories, and “mute” accounts entirely.¹⁶⁵ In other words, muting, filtering, and snoozing overlap in functionality but remain distinct, which is profoundly confusing for users interacting with these features across services.

Blocking and muting, while critically useful features, can have serious drawbacks for vulnerable users, especially journalists and writers. These features can make it harder for targets to assess the risk they are truly facing because they can no longer see if abuse is ongoing, or if it has escalated to threats of physical or sexual violence, doxing, etc. When abusers can see that they have been blocked, they often create new accounts and ramp up abuse.¹⁶⁶ As journalist Davey Alba explained in an interview with the Committee to Protect Journalists: “I was blocking a lot of people on Twitter for a while. That ended up being weaponized against me because people started making new accounts saying, ‘Oh, of course you blocked me, you don’t want to hear different points of view.’ So, I switched to muting accounts instead.”¹⁶⁷ Blocking and muting can be impractical for journalists and writers, who need access to their audiences, sources, and subjects. Cutting those people off can mean missing important information.

Download graphic >>

Share on Twitter >>

Restricting, a feature that Instagram introduced in 2019, addresses several of the drawbacks of blocking and muting outlined above. By restricting an abusive account, the targeted user places all comments from that account behind a screen, which targets can then choose to review and decide whether to publish, delete, or leave “pending” indefinitely. What distinguishes restricting from blocking is that abusers are not alerted to the fact that their ability to communicate with a target has been restricted. Restricting is also different from muting in that the abuser is the only person who can see the restricted content—targeted users and all other users cannot.¹⁶⁸ It is worth noting that even within one company, the term “restrict” has different meanings and parameters. On Facebook, users can place an abusive “friend” on a “restricted list” without unfriending that account entirely—which does not correspond to Instagram’s restriction feature.¹⁶⁹

Recommendations: Platforms need to improve and standardize blocking, muting, and restricting features that help users limit their exposure to abusive content and accounts. Specifically, platforms should:

Offer all three mechanisms—blocking, muting, and restricting—and apply each of these mechanisms consistently across all different forms of communication, including comments, DMs, tags, and mentions, and across desktop and mobile apps, regardless of device or browser type.
- Like Twitter and Instagram, Facebook should offer the option to mute by specific kinds of content, such as keywords and emojis.
- Twitter, Facebook, and Instagram should allow users to mute (rather than just block) DMs.
- Like Instagram, Twitter and Facebook should allow users to block, mute, or restrict accounts by flagging comments in bulk.
- Twitter and Facebook should offer functionalities akin to Instagram’s new restrict feature.
Convene a multi-stakeholder coalition of technology companies, civil society organizations, and vulnerable users—or enlist an existing coalition such as the Global Network Initiative,¹⁷⁰ Online Abuse Coalition,¹⁷¹ or Trust & Safety Professional Association¹⁷²—to standardize the terms and language used to describe these features based on their core functionalities and license them openly for reuse.

Mitigating risk: Muting is relatively low stakes, but blocking and restricting are riskier, potentially inhibiting transparency and public discourse. While the intended use of blocking or restricting is to limit exposure to an abusive account, these features can also be used to buffer a user, including a public official, from legitimate criticism or from ideas they do not like. A user who has been blocked by another user on Twitter, for example, can discover that they have been blocked because the two parties can no longer communicate. But a user who has been restricted by another user on Instagram has no way of knowing that their restricted content has been hidden—a highly effective way to curb retaliation and the escalation of abuse, but at the cost of transparency. To mitigate these outcomes, restriction should be strictly confined to limiting the visibility of comments posted by a restricted account in response to posts published by the restricting account. Restriction should not affect a restricted account’s ability to comment anywhere else on the platform. Instagram, which is the only platform to currently offer restriction, got this balance right; other platforms that introduce restriction should follow suit.¹⁷³

From a free expression standpoint, it is important to recognize that blocking does not stop someone from speaking freely on a social media platform but rather prevents one person from communicating directly with another. PEN America believes that most users of individual social media accounts are entitled to limit direct communication with other users, especially those engaging in abusive conduct. From the standpoint of transparency and accountability, however, the situation becomes more complicated for the professional accounts of public officials and government institutions. Some public officials have used the blocking feature to cut off critics, including their constituents.¹⁷⁴ In 2019, the Knight First Amendment Institute filed a lawsuit on behalf of Twitter users blocked by then-President Donald Trump. The court ruled that it was unconstitutional, on First Amendment grounds, for the president to block followers because Twitter is a “designated public forum” and @realDonaldTrump was “a presidential account as opposed to a personal account.”¹⁷⁵

It should be noted, however, that public officials who identify as women, LGBTQIA+, BIPOC, and/or members of religious or ethnic minorities are disproportionately targeted by online abuse and, in some cases, have been driven from public service.¹⁷⁶ While all public officials are subject to the same obligations regarding transparency and accountability, many of the recommendations outlined in this report would significantly reduce the burden of online abuse for public officials and their staff, without sacrificing their obligations to their constituents.

Through the years, we have experienced ups and downs in our relationships with tech companies, and we often still struggle to find the right connections to deal effectively with digital security incidents. We are keen to work with platforms in designing and enabling sustainable and scalable mechanisms to protect some of the most at-risk individuals and organizations in the world.

Daniel Bedoya, director of the global 24-7 Digital Security Helpline for nonprofit Access Now

Reporting: Revamping the user experience

The challenge: The ability of targeted users—and their allies—to report abusive content and accounts is foundational for reducing online harms and a fundamental part of any system of accountability. The goal of reporting is to ensure that abusive content is taken down and that abusive accounts face consequences. But user reporting is just one part of the larger content moderation system, which is often ineffective in ensuring that content that meets the threshold of abuse and violates platform policies is taken down.¹⁷⁷

Content moderation is a complex, imperfect process, and the line between harassment and combative but legitimate critique is often murky. Civil society organizations hear regularly from writers, journalists, artists, and activists about the mistaken removal of creative or journalistic content that does not violate platform policies, while violative content flagged as abuse remains. In addition, the removal process is often opaque. There is an urgent need to reform the larger content moderation process to make it more effective, equitable, and accountable, and multiple excellent reports have provided robust recommendations.¹⁷⁸ Rather than tackling the broader subject of content moderation in this report, PEN America focuses on examining reporting features from the standpoint of the user experiencing the abuse and doing the reporting.

In PEN America’s 2017 survey studying the impact of online harassment on writers and journalists, of the 53 percent of respondents who had alerted social media platforms to the harassment they were experiencing, 71 percent found the platform unhelpful.¹⁷⁹ Three years later, many of the experts and journalists PEN America consulted for this report concurred. “I’d say it’s 50/50 when I report, in terms of whether or not Twitter responds with action,” says journalist Emily Burack.¹⁸⁰ Journalist Jami Floyd has found trying to contact the company “a waste of time.”¹⁸¹ According to a 2020 study of online hate and harassment conducted by the Anti-Defamation League and YouGov, 77 percent of Americans want companies to make it easier to report hateful content and behavior, up from 67 percent in 2019.¹⁸²

Reporting features are often confusing and labor-intensive, placing undue burden on targets of abuse. There is very little consistency across platforms—or even within platforms—about how reporting features work.¹⁸³ All platforms have policies governing acceptable conduct and content, but when users are in the process of reporting, they rarely have quick and easy access to relevant policies.¹⁸⁴ Figuring out if a specific piece of content actually violates a particular policy can be confounding and time-consuming. “Explaining policies would help users interpret criticism versus harassment,” says Jason Reich, vice president of corporate security at The New York Times.¹⁸⁵

These challenges are exacerbated by reporting features that often direct users to select from preset categories of harmful content that rarely align, in language or concept, with specific policies or with the actual experiences of users. For instance, when users report content on Twitter, they must indicate how a specific account is harmful by selecting from preset options that include “targeted harassment,” which is separate from “posting private information,” “directing hate,” “threatening violence,” or “being disrespectful or offensive.” Nowhere in the reporting process—or in its “abusive behavior” policy—does Twitter explain what it means by “targeted harassment” (as opposed to just “harassment” or “abuse”).¹⁸⁶ This is just one example of many that PEN America found across Twitter, Facebook, and Instagram.

Some platforms funnel all reporting through a single channel. Others treat specific abusive tactics separately—distinguishing impersonation, doxing, and blackmail from harassment and requiring users to provide additional evidence to report such cases. Many users find themselves bombarded by hundreds or thousands of abusive messages in the midst of a coordinated campaign, yet few platforms allow them to report abusive content or accounts in bulk, forcing them to manually report individual content and accounts piecemeal.¹⁸⁷ The Facebook Civil Rights Audit, an independent investigation of the platform’s policies and practices through the lens of civil rights impact published in 2020, advocated for the introduction of bulk reporting of harassment, but Facebook has not acted on this recommendation.¹⁸⁸

Research for this report revealed a range of user needs that are not currently being met. Some writers and journalists would like to see a quicker, easier, one-click reporting feature. Others emphasize the importance of being able to add context, including the migration of online threats to other communication channels, which is a red flag for heightened risk. Still others want to be able to indicate when they were being attacked in a coordinated way across platforms (especially platforms owned by the same company) or to clarify, for example, why a particular insult could actually be a coded hateful slur.¹⁸⁹ Technologist Caroline Sinders argues that users should be able to pull up past reports and to draft, reopen, and review reports.¹⁹⁰

Existing features and tools: Twitter, Facebook, and Instagram all offer reporting features. A Facebook or Instagram user whose content has been removed can appeal and then escalate their case to the Facebook Oversight Board, a nominally independent body that launched in 2020 that will review a small subset of content moderation decisions and has the power to reinstate content. But no platform gives users who report abusive content a formal mechanism to appeal or escalate decisions for cases in which the abusive content they reported is not removed.

Twitter comes closest, offering designated communication channels for select newsrooms and nonprofits.¹⁹¹ Facebook has what a spokesperson calls “various, purposefully non-public channels available to specific partners.”¹⁹² Essentially, a system has emerged in which only individuals or institutions, such as newsrooms and nonprofits, with personal connections at social media platforms are able to escalate individual cases. The ad-hoc, informal, and interpersonal nature of these escalation channels is inherently unpredictable, inequitable, and impossible to scale. “Through the years, we have experienced ups and downs in our relationships with tech companies, and we often still struggle to find the right connections to deal effectively with digital security incidents,” says Daniel Bedoya, director of the global 24-7 Digital Security Helpline for nonprofit Access Now. “We are keen to work with platforms in designing and enabling sustainable and scalable mechanisms to protect some of the most at-risk individuals and organizations in the world.”¹⁹³

Recommendations: Platforms should revamp reporting features to be more user-friendly, responsive, and trauma-informed. Specifically, platforms should:

Ensure clarity and consistency between reporting features and policies within platforms. When reporting, users usually select from preset choices to indicate how the content violates the rules. The language used in these preset choices to describe prohibited abusive tactics must be harmonized with the language used in platform policies. When using the reporting features, users should be able to quickly and easily access relevant policies so they can check, in real time, whether the content they are reporting likely violates those policies.
Streamline the reporting process. Platforms should streamline the process by creating a single channel for reporting abusive content or accounts (rather than distinct and divergent channels for harassment, doxing, impersonation, etc.) and requiring as few mandatory steps as possible.

Create a flexible and responsive report management system. For users seeking more in-depth engagement with the reporting process, often because they are facing complex or coordinated abuse, platforms should:
- Allow users to create a draft of a report, add to it later, and combine multiple reports.
- Enable users to see all their past reports and review where they are in the reporting process.
- Offer an option for users to include additional context—to explain cultural or regional nuances in language, for example, or to flag symbols or images being used to abuse or spread hate.
Add bulk reporting. In recognition of the coordinated nature of some harassment campaigns, platforms need to give users the ability to report multiple abusive accounts and content in bulk, to reduce the burden of piecemeal reporting.
Create a formal, publicly known appeals or escalation process. When users report abusive content that they perceive to be in violation of platform policies and that content is not taken down, they should have accessible avenues to appeal or escalate their case. The appeal or escalation process should be formal, public, integrated into the reporting process, and available to all users who report harassment. When users file an appeal or escalation, they should be able to amend their case, adding context or additional related abusive content.
Provide prompts offering additional support. When used in good faith, reporting is a signal that a user is being harmed on a platform. Users who report abuse should be nudged toward additional in-platform features to help mitigate abuse, as well as toward external resources, ideally filtered by identity, location, and needs (see “Anti-abuse help centers,” below).

Mitigating risk: It is important to recognize that bulk reporting can be weaponized. Reporting has been wielded by abusers to trigger account suspensions, shutdowns, and the removal of posts. Offering the ability to report multiple pieces of content and accounts without requiring users to designate specific, violating content for each one may exacerbate this problem.¹⁹⁴ Users whose content has been taken down or whose accounts have been suspended due to malicious reporting can already appeal to platforms to have their content or account reinstated, though the process is not always effective and needs significant improvement (see “Appeals,” below). Users can also turn to civil society organizations that have informal escalation channels with the platforms. However, the introduction of bulk reporting necessitates further mitigation. Platforms may need to limit how many accounts or pieces of content can be reported at a time, or to activate the bulk reporting feature only when automated systems identify an onslaught of harassment that bears hallmarks of coordinated inauthentic activity. Ultimately, risk assessment, consultation with civil society, and user testing during the design process would reveal the most effective mitigation strategies.

When you’re experiencing stress, your body goes into an alarm state. All your key systems slow down so that your emergency system can work, and you either freeze, run away, or prepare to fight back. In an emergency mode you’re not thinking rationally.

Elana Newman, professor of psychology at the University of Tulsa and research director at the Dart Center for Journalism and Trauma

SOS button and emergency hotline: Providing personalized support in real time

The challenge: Some abusive tactics—such as threats of physical or sexual violence or massive coordinated campaigns—can put the targeted user into such an acute state of distress that they have difficulty navigating the attack in the moment. “When you’re experiencing stress, your body goes into an alarm state,” says Elana Newman, professor of psychology at the University of Tulsa and research director at the Dart Center for Journalism and Trauma. “All your key systems slow down so that your emergency system can work, and you either freeze, run away, or prepare to fight back. In an emergency mode, you’re not thinking rationally.”¹⁹⁵

Platforms face a design challenge in meeting the needs of both non-traumatized users seeking to report problematic content and traumatized, actively distressed targets contending with severe abuse, whose needs are quite different and currently poorly supported. The majority of writers, journalists, and other experts we interviewed emphasized the platforms’ lack of responsiveness and stressed the urgent need for customized support in real time.¹⁹⁶ Users need a way to indicate that they are experiencing extreme online abuse and a mechanism for urgently accessing personalized assistance.

Existing features and tools: While Twitter, Facebook, and Instagram provide users with various features to respond to online abuse, many of them discussed in depth throughout this report, these features are rarely designed to reduce trauma and its triggers. All three platforms provide self-guided support experiences, eschewing costlier approaches that would better aid those experiencing ongoing or extreme abuse. None offer customer support staffed by human beings for users experiencing abuse. Facebook users have to dig through over a hundred forms, find the one that most closely corresponds to their problem, fill it out, and wait. Some of these forms, such as those for impersonation or images that violate privacy rights, correspond to specific abusive tactics, but none explicitly deal with online harassment.¹⁹⁷ Twitter also funnels users to forms, though the company does at least group the forms for reporting abuse in one place.¹⁹⁸ Instagram theoretically has a customer support email, but its users are unlikely to receive a reply from a human being.¹⁹⁹

Several platforms have experimented with panic buttons. In 2010, the Child Exploitation and Online Protection Center (CEOP) in the U.K. lobbied Facebook to offer a panic button to instantly provide guidance for children being bullied or threatened online. Facebook resisted but ultimately bowed to public pressure and created a convoluted workaround: a separate app that, once installed, added a tab with CEOP’s logo to a child’s account profile. If clicked, the tab reportedly directed children to Facebook’s “Safety for teens” page and a panic button to report abuse to law enforcement.²⁰⁰ It remains unclear how widely this feature was used or whether it was effective in supporting children; it no longer seems to be available on Facebook.

Panic buttons that are not directly integrated into a platform’s primary user experience are ineffective. As digital safety adviser Ela Stapley warns: “Most of these panic buttons don’t work because users don’t use the app.”²⁰¹ In January 2020, the dating app Tinder launched a panic button integrated directly in its platform, allowing users concerned about their safety to tap the button to alert a third-party company called Noonlight, which “will reach out to check on the user and alert emergency responders if needed.”²⁰² Tinder has not yet published data on the use or efficacy of this promising feature.

Recommendations:

Platforms should create an SOS button that users could activate to instantly trigger additional in-platform protections and to access external resources. Users could proactively set up and customize these protections in advance, including:
- Turning on or turning up toxicity filters (see “Building a shield and dashboard,” above).
- Tightening security and privacy settings (see “Safety modes,” above).
- Documenting abuse with one click or automatically (see “Documentation,” above).
- Activating a rapid response team to provide support (see “Rapid response teams and delegated access,” above).
- Accessing external resources such as emergency mental health counseling, legal counseling, and cybersecurity help (see “Anti-abuse help centers,” below).
Platforms should create an Emergency hotline (phone and/or chat) that provides users facing extreme online abuse (such as cyberstalking or a coordinated mob attack) with personalized, trauma-informed support in real time.
To ensure user-friendliness and accessibility, platforms should fully integrate an SOS button and emergency hotline directly into the primary user experience. Better still, platforms could use automated detection to nudge users about these features, as Facebook does with its suicide prevention efforts.²⁰³

Journalists get frustrated when [an anti-abuse] tool they found useful suddenly stops working because it has run out of funding. They also become disheartened when the tool is buggy or the user experience is bad. They just want a tool that works well and is easy to set up.

Ela Stapley, digital safety adviser

Anti-abuse help centers: Making resources and tools accessible

The challenge: Many of the writers and journalists in PEN America’s network are often unaware of the in-platform features and third-party tools that already exist to mitigate online abuse. In a 2017 survey of writers and journalists who had experienced online abuse, PEN America found that nearly half of respondents reported changing social media settings only after they were harassed.²⁰⁴ As reporter Kate Steinmetz wrote in an article in Time highlighting Instagram’s new anti-harassment features: “The company has launched other tools meant to help users protect themselves from bullies in the past—like a filter that will hide comments containing certain keywords that a user designates—but many remain unaware that such tools exist.”²⁰⁵

Third-party tools that have emerged to counter online abuse (analyzed throughout this report) are urgently needed and very welcome, but they, too, face an array of challenges. Most are still in early stages of development, and few are sufficiently well financed to be as effective as they could be. As digital safety adviser Ela Stapley explains: “Journalists become disheartened when the tool is buggy or the user experience is bad. They also get frustrated when a tool they found useful suddenly stops working because it has run out of funding. They just want a tool that works well and is easy to set up.”²⁰⁶

Some of these tools involve costs for the consumer, an insurmountable obstacle for many staff and freelance journalists in an industry under intense financial pressure. Any anti-abuse tool that must be linked to sensitive accounts, such as social media or email, should be audited by independent cybersecurity experts, but this cost can be prohibitive for the developers. Perhaps the biggest challenge, however, is that social media platforms fail to highlight, integrate, or support the development of these third-party tools.²⁰⁷

Existing features and tools: Twitter, Facebook, and Instagram all have help centers, with some content specifically focused on online abuse. Both Twitter’s multipage “Abuse”²⁰⁸section and Instagram’s “Learn how to address abuse”²⁰⁹ page are relatively easy to find, but the guidance they offer is bare-bones and general. On Twitter’s page headed “How to help someone with online abuse,”²¹⁰ for example, the platform does not explain that users can use TweetDeck to give allies delegated access to their accounts to help manage abuse. Instagram does not explain the strategic differences between blocking, muting, filtering, and restricting. Facebook’s guidance and resources for navigating abuse are more robust yet much harder to find and more confusing to parse. Its “Abuse resources”²¹¹ page, within the help center, is just a brief Q&A that could significantly benefit from more direct, specific, and nuanced guidance. Its section on non-consensual intimate imagery²¹² is more robust but hard to find, and it does not address the myriad other abusive tactics that adult users face. Most of these anti-harassment resources exist outside the primary user experience, and users are not alerted to their existence with proactive nudges. Many of them seem to be primarily aimed at children, parents, and educators rather than adults.²¹³

While platform help centers do not specifically address the needs of journalists, a number of the third-party tools analyzed throughout this report do. Some have been built by research teams within big technology companies.²¹⁴ A handful were developed by the private sector and seem to have viable revenue models.²¹⁵ But the majority of these tools and services have been built with limited resources by universities, nonprofits, and/or technologists launching startups.²¹⁶

Recommendations:

Platforms should build robust, user-friendly, and easily accessible sections within help centers that expressly address online abuse. Specifically, Twitter, Facebook, and Instagram should:
- Outline and adequately explain all the features they already offer to address online abuse.
- Develop content specifically tailored to the needs of writers and journalists, taking into account that they rely on social media to do their work.
- Provide a visually prominent link directly to this information in the main user experience. Like fire alarms, these links can be unobtrusive but should be right at hand in case they are needed.
- Use nudges, sign-on prompts, interactive tip sheets, graphics, videos, and quizzes to regularly educate users about in-platform anti-abuse features.
- Make the information available in multiple languages, as well as in large print and audio, to ensure wider accessibility.
- Invest in training vulnerable users with specific needs, including journalists and writers, on how to use anti-harassment features.
- Direct users to external resources, including online abuse self-defense guides,²¹⁷ cybersecurity help lines and tools,²¹⁸ mental health services,²¹⁹ and peer support.²²⁰ Because direct referral from a global platform will exponentially increase the volume of requests for help, platforms should consult with and support the organizations responsible for staffing and maintaining those resources, as Reddit has done in its partnership with the Crisis Text Line for users at risk of suicide and self-harm.²²¹
Platforms should support the creation of promising new third-party tools designed to counter online abuse—especially those built by and for women, BIPOC, and/or LGBTQIA+ technologists with firsthand experience of online abuse—by investing in R&D and providing access to application programming interfaces (APIs), data, and other relevant information.

I want harassment to be as annoying for my harassers as it is for me to report it.

Talia Lavin, journalist

Disarming Abusive Users

The burden of dealing with online abuse often rests squarely on the shoulders of its targets, and those targets are often women, BIPOC, LGBTQIA+, and/or members of religious and ethnic minorities. Its impact, from the strain on mental health to the chilling effects on speech and career prospects, stands in stark contrast to the point-and-click ease with which abusers inflict it.“I want harassment to be as annoying for my harassers as it is for me to report it,”²²² says journalist Talia Lavin.

Online abuse cannot be addressed solely by creating tools and features that empower the targets of abuse and their allies. Platforms must also actively hold abusive users to account. “By allowing harassers to function with immunity,” they “exacerbate harm,” Soraya Chemaly, executive director of the Representation Project, explained in a 2015 interview with Mic.²²³

As noted in the introduction and throughout this report, efforts to deter abuse must be balanced against competing priorities: to protect critical speech and prevent the silencing of legitimate dissenting viewpoints, which may include heated debate that does not rise to the level of abuse as well as humor, satire, and artistic expression that can be mistaken for abusive content. To that end, PEN America’s recommendations seek to deter abuse without unduly increasing the platforms’ power to police critical speech, which threatens all users’ free expression rights.

A foundational principle of our analysis is that, although the lines can sometimes seem clear, any person can be a target, a bystander, or an abuser, depending on their behavior.²²⁴ Furthermore, it can be useful to distinguish between casual abuse and committed or automated abuse. Casual abuse may include, for example, individuals who are engaged in unfocused nastiness for sport or who get swept up in online anger or vitriol against a target. Committed or automated abuse includes individuals, bots, groups, and state actors engaging in coordinated and premeditated abuse against particular targets, often with particular outcomes in mind. Both are problematic and harmful, but casual abuse warrants a different set of interventions than the strategies being used to battle coordinated inauthentic activity, such as mass content takedowns, rapid account deletions, and forensic investigations.²²⁵ In this section, our recommendations primarily focus on deterring casual abuse.

Nudges: Using design to discourage abuse

The challenge : Social media platforms are built to encourage immediacy, emotional impact, and virality because those characteristics heighten the user engagement that is fundamental to their business models.²²⁶ The result of such incentives, according to Dr. Kent Bausman, a sociology professor at Maryville University, is that social media “has made trolling behavior more pervasive and virulent.”²²⁷ Researchers at the University of Michigan recently highlighted their “concerns about the limitations of existing approaches social media sites use” to curb abuse—namely, restrictive tactics like “removing content and banning users.” They advocate for educating users rather than just penalizing them.²²⁸

One way to do this is through nudges—interventions that encourage, rather than force, changes in behavior by presenting opportunities for feedback and education.²²⁹ For example, a user in the process of drafting a post with abusive language could receive a nudge encouraging them to pause and reconsider. Along these lines, Karen Kornbluth, director of the Digital Innovation and Democracy Initiative at the German Marshall Fund, has called for platforms to counter virality by introducing friction—design elements that nudge users by making certain behaviors less convenient or slower—to “make it harder to spread hate and easier to engage constructively online.”²³⁰ The use of nudges has the major advantage of preserving freedom of expression, giving users an opportunity to make informed and deliberate decisions about how they choose to act.

Screenshots of Twitter and Instagram’s experiments with nudges which pop up to prompt users to reconsider the language of a potentially abusive post.

Screenshot demonstrating how to restrict an abusive account on Instagram. Photo by Instagram

That said, nudges are no panacea. The jury is still out on how effective specific types of nudges actually are,²³¹ particularly in the absence of meaningful experimental validation. Furthermore, nudges that depend on automation can be prone to false positives because algorithms are vulnerable to the biases of their authors and the data sets on which they are trained.²³² In the example offered above, for instance, a nudge could mistakenly flag a post as potentially abusive because it depends on an automated system that has trouble distinguishing between a racial slur and a reclaimed term.

Existing features and tools: In recent years, platforms have piloted the use of nudges to discourage harmful content, especially disinformation. More should be done to apply this approach to reducing harassment and to evaluate its efficacy. Twitter²³³ and Instagram²³⁴ are currently experimenting with automation that proactively identifies harmful language and nudges users to rethink a reply before sending it. In an email to PEN America, Facebook states that it is also piloting nudges for harassing content, though we were unable to verify this statement.²³⁵ While Facebook and Instagram claim that such nudges can help reduce abuse,²³⁶ none of the platforms examined in this report have shared data on the efficacy of these interventions.²³⁷

Recommendations:

Platforms should use nudges to discourage users’ attempts to engage in abusive behavior. One way to do this is to use automation to proactively identify content as potentially abusive and nudge users with a warning that their content may violate platform policies and encourage them to revise it before they post.
Platforms should study the efficacy of nudges to curb abuse and publish these findings. Platforms should also communicate clearly and transparently about how the algorithms that inform many of these interventions are trained, including efforts to curb implicit bias.
Platforms should give outside researchers access to data—both on the efficacy of nudges and on the data on which algorithms are trained to detect harmful language—so they can independently assess success and flag unintended harm, as well as recommend improvements.

Rules in real time: Educating users and making consequences visible

The challenge: Norms governing behavior can only work if they have been clearly communicated, understood, and agreed upon by the members of an online community. However, most social media platforms keep their policies governing user behavior in an area distinct from the primary user experience of their product. In an interview with PEN America, Jillian York, director for international free expression at the EFF, says that Twitter’s rules, for example, “are really difficult to find, so most users aren’t even aware of what they are or how to find them before they violate them.” Moreover, a user might break a rule, “and all they’re told is that they broke the rules, not which rule they broke and why or how.”²³⁸

While it is important for platforms to maintain dedicated areas that display all their policies, it is equally critical to fully integrate the most important and relevant guidelines directly within the primary user experience so users can see this information in real time. When users create a new password, for instance, they should not have to go to a separate page to learn about minimum password complexity requirements; that information is included in the same form or window. Similarly, users should be able to quickly check that content complies with key rules before posting without having to click away and search through a separate website. Integrating platform rules could not only reduce casual abuse and increase transparency, but avert the perception of arbitrary or biased enforcement.

Recent research from Stanford University indicates that making community rules more visible, including at the top of comments and discussions sections, increases newcomers’ compliance with them while simultaneously increasing participation.²³⁹ An internal audit at YouTube found that users actively wanted the platform to create clearer policies to make enforcement more consistent, and to be more transparent about enforcement actions (see “Escalating penalties,” below).²⁴⁰

Existing features: If Twitter users want to review the guidelines governing acceptable behavior, they have to go to the overall menu on the platform’s app or desktop version and intuit that they will find these guidelines within the “help center,” which then takes them to a webpage outside the primary user experience. From there, users have to head to a page called “Rules and policies” and review over a dozen distinct pages that are, for reasons that remain unclear, spread out between two sections titled “Twitter’s Rules and policies” and “General guidelines and policies.”²⁴¹ On Facebook’s app and desktop version, users looking for guidelines have to go to the main menu and embark on a long and winding journey with stops at “help and support,” “help center,” “policies and reporting,” and “about our policies”—none of which actually lead to the platform’s “community standards,” which live on a separate website.²⁴² On Instagram, from within either the app or desktop version, the authors of this report were unable to locate the platform’s “community guidelines” (which live on a separate website²⁴³). Furthermore, as the Facebook Oversight Board asserted, given that Facebook and Instagram belong to the same company, the relationship between Facebook’s extensive “community standards” and Instagram’s shorter “community guidelines” needs to be clarified and their inconsistencies need to be ironed out.²⁴⁴

Recommendations:

Platforms should make their rules—and the consequences for breaking them—easily and directly accessible to users in real time and within the primary user experience.
Platforms should use the full suite of design elements—including nudges, labels, and contextual clues—to spotlight relevant rules.
Platforms should routinely use policy checkups or reminders, akin to existing interactive privacy checkups on Facebook and Google. Whenever platforms make major changes or updates to their rules governing acceptable behavior, they should proactively call attention to these changes and seek affirmative consent from users.

Graphic by Wikicommons

Escalating penalties: Building an accountability system for abusive users

The challenge: To effectively tackle online abuse, especially by committed abusers, decisive measures like account suspensions and bans are sometimes necessary. They are also fraught. As discussed throughout this report, overzealous suspensions or bans can compromise free expression. Content moderation is inherently imperfect, especially when it relies on automation, and it has repeatedly been weaponized to silence writers, journalists, and activists.²⁴⁵ For major and minor infractions alike, however, the consequences for violating rules are not readily visible or clearly communicated to users.

“Platforms should add suggestions of consequences of misconduct in the standards,” says law professor Mary Anne Franks. “They need to have clear rules that say this kind of behavior will result, for instance, in a temporary suspension of an account. They need to say what the enforcement will be. Otherwise, compliance with their standards is a recommendation, not an obligation.”²⁴⁶ Experiments in the gaming industry underscore the efficacy of explaining to abusers exactly which policies their content has violated and why it led to a penalty.²⁴⁷

This is exactly what the Facebook Oversight Board advised in its first set of recommendations, which called for “more transparency and due process for users, to help them understand the platform’s rules,” according to evelyn douek, a lecturer and doctoral candidate at Harvard Law School. “The FOB shows concern that users who have been found to violate their rules simply cannot know what they are doing wrong, whether because Facebook’s policies are not clear or lack detail or are scattered around different websites, or because users are not given an adequate explanation for which rule has been applied in their specific case.”²⁴⁸

Existing features and tools: Some platforms are developing escalating penalties, but these generally remain nascent and poorly publicized to users. They do not actually constitute a transparent and clearly communicated accountability system. Twitter may suspend accounts for violating its rules, including for engaging in abusive behavior, and may require users to verify account ownership or formally appeal to lift a suspension. The company can also place an account in read-only mode, limiting the ability to tweet, retweet, or like content, and may require a user to delete the violating content to restore full functionality. Subsequent violations, it warns, “may result in permanent suspension,” which is a perplexing oxymoron.²⁴⁹

In 2019, Instagram began issuing alerts to users whose posts repeatedly violate community guidelines, informing them that their account may be banned if they persist and providing them with a history of the relevant posts and the reasons for their removal.²⁵⁰ According to Facebook, the platform issues warnings to users who post content or misuse its features in a way that violates community standards.²⁵¹ In an email to PEN America, Facebook elaborated: “Time-bound feature limits are the central penalty used on Facebook users. If Facebook removes multiple pieces of content from a user’s profile, Page, or group within a short period of time, they’ll have a short-term restriction placed on their account. This user will continue to receive additional and longer restrictions as long as they keep on violating our Community Standards. Other less frequent penalties include rate limits, education requirements, audience limitation, loss of certain product features (such as the ability to go Live).”²⁵²

Of the platforms analyzed in this report, only Twitter publicly outlines its penalties in its help center. But to better understand Twitter’s accountability system, a user would have to read several distinct policy pages, all of which make heavy use of the word “may.” PEN America could find only the most minimal information on Facebook’s or Instagram’s help centers that lays out each platform’s penalties for violating their policies; we cobbled together the information above from a news article,²⁵³ an email exchange with both platforms,²⁵⁴ and a corporate blog post from 2018.²⁵⁵ Facebook and Instagram informed PEN America that they do not “provide full visibility into the penalties to avoid gaming.”²⁵⁶ Bottom line: It is exceptionally difficult for users to understand the consequences of violating platform policies, even if they can ascertain what those policies are in the first place.

Beyond the platforms analyzed in this report, YouTube offers a model worth exploring. The platform overhauled its penalties into a system of escalating “strikes” after consulting with its users in 2019.²⁵⁷ When a user violates community guidelines for the first time, they receive a warning that explains what content was removed and which policies were violated, and explains next steps. If a user posts prohibited content a second time, they receive a “strike,” which restricts account functionality for one week. After the second strike, users experience further functionality restrictions for two weeks, and after three strikes within a 90-day period, any channel continuing to post violating content will, according to YouTube, be deleted. Appeals, the platform states, are available at every step of the process.²⁵⁸ The platform claims that “94% of those who do receive a first strike never get a second one,” though it has not released the raw data to back up this assertion.²⁵⁹ While YouTube’s administration of this system remains obscure in places—for example, in its use of automated flagging—it provides a useful template to build on.

Recommendations:

Platforms should create a transparent system of escalating penalties for all users, including warnings, strikes, temporary functionality restrictions, and suspensions, as well as content takedowns and account bans.This accountability system should be fully integrated into the primary user experience and clearly visible alongside policies governing user behavior.
Platforms should use the full suite of design elements (nudges, labels, contextual clues, etc.) to communicate clearly and consistently with users across all available channels (within platform, via email, etc.) about what rule has been violated, current and potential future penalties, and next steps, including how to appeal.
Platforms should convene a coalition of technology companies, civil society organizations, and vulnerable users—potentially leveraging the newly formed Digital Trust & Safety Partnership²⁶⁰Margaret Harding McGill, “Tech giants list principles for handling harmful content,” Axios, February 13, 2021, axios.com/tech-giants-list-principles-for-handling-harmful-content-5c9cfba9-05bc-49ad-846a-baf01abf5976.html?utm_campaign=organic&utm_medium=socialshare&utm_source=twitter—to create a baseline set of escalating penalties that can help establish common expectations, which could include:
- Warnings and strikes. Platforms should adopt a graduated approach to enforcement, issuing warnings and counting strikes before taking the more drastic step of suspending or closing accounts. That said, some violations, like direct incitement to violence, warrant immediate suspension.
- Temporary suspensions and functionality limitations. Platforms should deploy temporary suspensions and functionality limitations—such as preventing accounts from posting but not from browsing, and suspending accounts for several days or weeks.
- Coordinated protocols for adjusting these escalating penalties over time, including in response to evolving patterns of abuse or the weaponization of the penalties themselves.

Appeals: Ensuring a transparent and expeditious process

The challenge: Not all content reported as abuse is actually abusive. Reporting systems are regularly weaponized by abusers seeking to intimidate or defame their targets and trigger the removal of posts and suspension of accounts.²⁶¹ Further, perceptions of what does or does not constitute abuse vary among individuals and communities. Content moderators make decisions that reasonable people can disagree with, and content that is flagged in good faith may fall short of violating platform policies.²⁶² Many users who believe their content was removed as a result of inaccurate or malicious reporting struggle to understand why, have little opportunity to make their case, and can effectively be silenced by the slow restoration of their content.²⁶³

As platforms make much-needed improvements in the ability to flag and remove abusive content, contentious decisions and false positives will inevitably increase. Organizations like the ACLU²⁶⁴ and Ranking Digital Rights²⁶⁵ have argued that a transparent, expeditious appeals process for content takedowns is critical for the preservation of free expression in any content moderation process. PEN America supports the adoption of the Santa Clara Principles. Released in 2018 by a coalition of civil society organizations and academics, this proposal calls for the review of appealed content by human beings, the opportunity for users to provide context during the appeals process, and explicit notification of the final outcome, including a clear explanation of the decision.²⁶⁶

Existing tools and mechanisms: Twitter, Facebook, and Instagram have made improvements to their appeals process in recent years. In 2018, Twitter committed to emailing suspended users to advise them of the content of violating tweets and details about which rule they broke.²⁶⁷ In 2019, the platform integrated appeals directly into its mobile app, rather than requiring users to fill out a separate online form, which Twitter claims improved its response speed by 60 percent.²⁶⁸ In 2020, Instagram also integrated the ability to appeal disabled accounts and content takedowns directly within its app.²⁶⁹ Since 2018, Facebook users have had the ability to appeal the removal of posts, photos, and videos, as well as the removal of groups, profiles, and pages²⁷⁰—though users still have to appeal through a separate online form.²⁷¹ Both Facebook and Instagram now allow users to escalate appeals for content takedowns to a new, purportedly independent oversight board, which will review a small fraction of submissions.²⁷²

Despite recent improvements, the platforms still have a long way to go in the transparency, expeditiousness, and open communication of appeals. As Amnesty International’s “Toxic Twitter” report notes, “A detailed overview of the appeals process, including an explicit commitment to respond to all appeals or a timeframe of when to expect a response is not included in any of Twitter’s policies.”²⁷³ The EFF’s annual “Who Has Your Back” report found that only Facebook has a satisfactory commitment to providing “meaningful notice” regarding content and account takedowns and that neither Twitter, Facebook nor Instagram has a satisfactory commitment to “appeals transparency.”²⁷⁴ Twitter does not include appeals in its transparency report at all.²⁷⁵ While Facebook and Instagram provide information about how much content users appeal and how much of the content is restored, they offer no information about the timeliness of responses or restorations.²⁷⁶

Recommendations: PEN America recommends the implementation of significantly more robust and regularized appeals processes for users whose content or accounts have been taken down, restricted, or suspended. Specifically, platforms should:

Fully and prominently integrate appeals into the primary user experience, and communicate clearly and regularly with users at every step of the appeals process via notifications within the platform’s desktop and mobile app, as well as through secondary communication channels, like email.
Build on the Santa Clara Principles²⁷⁷ to ensure that users can add context during the appeals process and that humans review appealed content.
Create a formal, adequately resourced escalation channel for expediting appeals to address cases of malicious or inaccurate content takedowns and time-sensitive cases where a delay in restoring content or accounts could be harmful, for example during times of urgent political debate or crises. At the very least, this channel should enable an institution, such as a news outlet or civil society organization, to advocate with the platform on behalf of an individual, such as a journalist or human rights defender.
Substantially increase transparency about the way the appeals process works. Specifically, regular transparency reports from platforms should include metrics on how much content users appeal, how much of the appealed content is restored, the timeliness of responses for all appeals, and the timeliness of restoration. Platforms should provide independent researchers with disaggregated data supporting these metrics.

Protesters marching in Washington, D.C. in November 2016. Photo by Lorie Shaull

Methodology

In this report, PEN America lays out the impact of online abuse on the lives, livelihoods, and freedom of expression of writers and journalists and recommends concrete changes that technology companies should make now to better protect all vulnerable people. Our recommendations center on the experiences and needs of United States-based users disproportionately targeted online for their identity and profession and prioritize changes to the design of digital platforms. We base our proposals on in-depth qualitative research, including dozens of interviews and a comprehensive literature review, and on the extensive experience that PEN America has gleaned through its Online Abuse Defense program.

What is online abuse?

PEN America defines online abuse as the “severe or pervasive targeting of an individual or group online with harmful behavior.” “Severe” because a single incident of online abuse, such as a death threat or the publication of a home address, can have serious consequences. “Pervasive” because individual incidents, such as an insult or spamming, may not rise to the level of abuse, but a sustained or coordinated onslaught of incidents like these can cause significant harm. “Harm” can include emotional distress, anxiety, intimidation, humiliation, invasion of privacy, the chilling of expression, professional damage, fear for physical safety, and physical violence.”²⁷⁸

PEN America’s Online Abuse Defense Program

PEN America is a nonprofit that stands at the intersection of literature and human rights to protect free expression in the United States and worldwide. Our mission is to unite writers and their allies to celebrate creative expression and defend the liberties that make it possible. Our Membership consists of over 7,500 journalists, novelists, nonfiction writers, editors, poets, essayists, playwrights, publishers, translators, agents, and other writing professionals, as well as devoted readers and supporters throughout the United States. In 2017, PEN America conducted a survey of over 230 writers and journalists within our network and found that the majority of respondents who had faced online abuse reported fearing for their safety and engaging in self-censorship; this included everything from refraining from publishing their work to deleting their social media accounts.²⁷⁹ In response, in 2018 we launched our Online Abuse Defense program in the United States, which centers on education, research, and advocacy. We develop resources to equip writers and journalists, as well as their allies and employers, with comprehensive strategies to defend themselves against online abuse. Our Field Manual, articles, and tipsheets have reached over 250,000 people.²⁸⁰ The authors of this report have led presentations and workshops on combating online abuse and bolstering digital safety for over 7,000 journalists, writers, editors, academics, lawyers, activists, and others; we also work closely with newsrooms, publishing companies, and professional associations to develop policies, protocols, and training to protect and support writers and journalists. Finally, we conduct research on the impact of online abuse and the solutions to address it, and we advocate for change to reduce online harm so that all creative and media professionals can continue to express themselves freely.

Research Methodology

For this report, PEN America conducted in-depth interviews between April 2020 and February 2021 with over 50 people, including writers and journalists; editors and newsrooms leaders; experts in online abuse and digital safety; researchers and academics who study media, UX, and design; technologists; lawyers; and representatives of technology companies. We conducted a comprehensive cross-disciplinary literature review of over 100 articles, reports, papers, books, and guidelines from academia and civil society, in fields including design, computer science, sociology, human rights, and technology.

We centered our research on the experiences of people disproportionately targeted by online abuse for their identity and/or profession, specifically: 1) writers and journalists whose work requires a public presence online, and 2) women, BIPOC (Black, indigenous, and people of color), LGBTQIA+ (lesbian, gay, bisexual, transgender, queer, intersex, and asexual) people, and/or people who belong to religious or ethnic minorities.²⁸¹ In our recommendations, we prioritize the needs of people at the intersection of these two groups because we know that they experience especially egregious abuse.²⁸² We contend that when technology companies meet the needs of users most vulnerable to online abuse, they will better serve all of their users.

We focus our analysis specifically on Twitter, Facebook, and Instagram because these are the platforms on which United States–based writers and journalists rely most heavily in their work,²⁸³ and the platforms on which United States–based users report experiencing the most abuse.²⁸⁴ We analyzed in-platform features designed to mitigate online abuse (such as blocking, muting, hiding, restricting, and reporting). And we also identified and analyzed relevant third-party tools, some built by private companies and others by universities and nonprofits (such as Block Party, Tall Poppy, BodyGuard, Sentropy Protect, TweetDeleter, Jumbo, Tune, and many others). Although writers and journalists also experience a significant amount of abuse on private messaging platforms (such as email, text messaging, WhatsApp, and Facebook Groups), these platforms carry their own unique privacy and security challenges and fall outside the scope of this report. While our recommendations are rooted in our research on three major social media platforms, we believe they are useful and relevant to all technology companies that design products to facilitate communication and social interaction.

Our research and recommendations focus on the United States, where PEN America’s expertise in online abuse is strongest, but we fully acknowledge that online abuse is a global problem and we understand the urgent need to find locally and regionally relevant solutions. Several of the technology companies analyzed in this report have a global user base, and one of the central challenges to curtailing online abuse is the blanket application of United States–based rules, strategies, and cultural norms internationally.²⁸⁵ Throughout this report, we endeavor to account for the ways that changes to features on global platforms could play out in regions and geopolitical contexts outside the United States.

Acknowledgments

This report was written by Viktorya Vilk, program director for Digital Safety and Free Expression; Elodie Vialle, program consultant for Digital Safety and Free Expression; and Matt Bailey, program director for Digital Freedom at PEN America. PEN America’s senior director for Free Expression Programs, Summer Lopez, reviewed and edited the report, as did CEO Suzanne Nossel. James Tager, Nora Benavidez, Stephen Fee, and Dru Menaker provided thoughtful feedback. PEN America would also like to thank the interns whose research, fact-checking, and proofreading contributed significantly to this report: Jazilah Salam, Margaret Tilley, Hiba Ismail, Sara Gronich, Tarini Krishna, Blythe Drucker, Jordan Pilant, Glynnis Eldridge, and Cheryl Hege.

PEN America extends special thanks to the following experts for providing invaluable input on this report: Jami Floyd, senior editor of the Race and Justice unit at New York Public Radio; Dr. Michelle Ferrier, founder of TrollBusters and executive director of Media Innovation Collaboratory; Kat Lo, content moderation lead at Meedan; Azmina Dhrodia, senior policy manager for gender and data rights at World Wide Web Foundation and adviser at Glitch; Jamia Wilson, vice president and executive editor at Random House; Ela Stapley, digital safety adviser and founder of Siskin Labs; T. Annie Nguyen, lead product designer and design researcher; and Jillian York, director for international freedom of expression at the Electronic Frontier Foundation. PEN America is also deeply grateful to the many journalists, writers, scholars, technologists, psychologists, civil society advocates, lawyers, and other experts who agreed to be interviewed for this report, including those who are not acknowledged by name. PEN America appreciates the responsiveness of the representatives at Twitter, Facebook, Instagram, and Google in our many exchanges, as well as the generosity and openness of the founders and staff of the many third-party tools we examined for this report.

Our deep abiding appreciation goes to the Democracy Fund and Craig Newmark Philanthropies for their support of this project. PEN America also receives financial support from Google and Facebook, but those funds did not support the research, writing, or publication of this report.

The report was edited by Susan Chumsky.