Speech in the Machine

Generative AI’s Implications for Free Expression

Speech in the Machine

July 31, 2023

Home / Reports / Speech in the Machine

Key Findings:

Finding 1

Finding 2

Finding 3

PEN America Experts:

Summer Lopez

Interim Co-CEO and Chief Program Officer, Free Expression

Introduction

The rapidly expanding era of artificial intelligence (AI) is ushering in both exciting possibilities and precarious unknowns. AI technologies are already powerful curators of information and arbiters of online content, and often amplifiers of disinformation. As AI technology—particularly generative AI technology—evolves, its potential impact on human rights and the fundamental right to free expression must be central to conversations about policy, regulation, and best practices.¹ Companies, international organizations, and governments must take the potential impact of AI into account when formulating means of safely and productively harnessing the benefits of this new era.

In this paper, PEN America notes the critical free expression issues at stake with generative AI, which has the potential to supercharge tools of deception and repression and make them more widely accessible. From online content creation to translation, creative writing to news reporting, generative AI tools may spur inspiration and ingenuity—or overtake the human craft in ways that undercut authenticity in public discourse and dampen the underlying value of open expression. How the development and dissemination of AI systems might underscore or undermine the right to free expression will depend on who controls the rollout of these new technologies, what regulations are imposed, and how companies and governments define and execute their human rights responsibilities.

The purpose of this paper is to identify emerging free expression issues raised by the increased prevalence and usage of generative AI, with particular attention given to large language models (LLMs). As an organization of and for writers, PEN America is focusing this paper on issues of most concern to our community. PEN America has a 100–year history of advocating for the protection of writers at risk and defending the freedom to write globally, highlighting the tactics of dictators and would–be authoritarian regimes in silencing those whose work sparks the imagination and holds governments to account. The 1948 PEN Charter commits all PEN centers to fighting “mendacious publication, deliberate falsehood and distortion of facts for political and personal ends.”² Over the past decade, PEN America has sounded the alarm about the spread of disinformation and online abuse, and their effects on free expression, press freedom, and democratic discourse.³

The paper begins with the implications of generative AI for creativity and the arts, particularly for writers and the written word. It then examines how generative AI might exacerbate existing threats to free expression, including disinformation, online abuse, and censorship, and how it might wield more subtle forms of influence on the information landscape. Finally, the paper offers preliminary recommendations and guiding principles for policymakers and companies as they consider the policies and regulations that will shape the era of generative AI. Because these technologies are advancing quickly, much of the risk assessment at this stage is speculative. We cannot anticipate exactly how these technologies will be used or the magnitude of the risks. As an initial issue brief, however, we intend this paper to set an agenda for further research, analysis, and deliberation as generative AI technologies, their role in society, and their impact on freedom of expression continue to evolve.

Glossary

The definitions below are drawn from the work of experts in academia, government, and civil society.

Artificial Intelligence (AI): Artificial intelligence, in the words of the man who coined the term, is “the science and engineering of making intelligent machines, especially intelligent computer programs.”⁴ Note: In this report we use the term “artificial intelligence” to refer to the field of study or branch of research, and not to the tools or technologies that rely on machine learning and natural language processing to simulate human intelligence.
Machine Learning: “A subfield of artificial intelligence that gives computers the ability to learn without explicitly being programmed.”⁵
Natural Language Processing (NLP): A subfield of artificial intelligence “that explores how computers can be used to understand and manipulate natural language text or speech to do useful things. NLP researchers aim to gather knowledge on how human beings understand and use language so that appropriate tools and techniques can be developed to make computer systems understand and manipulate natural languages to perform the desired tasks.”⁶
Algorithm: “[A] set of instructions that is designed to accomplish a task. Algorithms usually take one or more inputs, run them systematically through a series of steps, and provide one or more outputs.”⁷
AI System: “An engineered or machine-based system that can, for a given set of objectives, generate outputs such as predictions, recommendations, or decisions influencing real or virtual environments. AI systems are designed to operate with varying levels of autonomy.”⁸ Note: In this report we use the term “AI tools” to refer to applications or services that rely on AI systems to function.
Generative AI: “Generative AI refers to a category of artificial intelligence (AI) algorithms that generate new outputs based on the data they have been trained on. Unlike traditional AI systems that are designed to recognize patterns and make predictions, generative AI creates new content in the form of images, text, audio, and more.”⁹
Large Language Model (LLM): “A machine learning algorithm that scans enormous volumes of text to learn which words and sentences frequently appear near one another and in what context. Large language models can be adapted to perform a wide range of tasks across different domains.”¹⁰
AI Chatbot: “Chatbots are intelligent conversational computer programs that mimic human conversation in its natural form.”¹¹ “Chatbots can mimic human conversation and entertain users but they are not built only for this.”¹² An AI chatbot accomplishes this through an underlying AI system.

This report comes at a pivotal moment for the future of artificial intelligence technology. In November 2022 OpenAI, then a niche company, released ChatGPT (Generative Pre–Trained Transformer), an AI chatbot designed for public–facing and conversational usage. Prior to that point, the concept of artificial intelligence was associated with science fiction, even if a future with robots as part of daily life felt increasingly imminent. ChatGPT inaugurated a new chapter: Here was a piece of technology that almost anyone with a laptop or mobile phone could access, an application that made artificial intelligence tangible and useful for daily life—and, in doing so, seemed to herald a new technological era.

In the hands of bad actors—whether public or private—generative AI tools can supercharge existing threats to free expression.

The public awakening to generative AI, driven largely by ChatGPT, follows the integration of AI tools in personal technology and in systems that undergird other sectors of society, such as agriculture and healthcare.¹³ Smart assistants like Apple’s Siri and Amazon’s Alexa, algorithms that recommend songs on Spotify or Pandora, transcription services, suggested smart replies in Gmail, social media content curation—all run on artificial intelligence.¹⁴ The significant difference between these functions and ChatGPT, Google Bard, and similar tools, is that the latter are not being used just to perform a predetermined task, but are instead capable both of generating content and of engaging in “conversation” with users.

Generative AI systems are trained on a data set of information, which is then used algorithmically to develop and refine outputs from the system. For example, a chatbot designed to triage patients at an urgent care clinic might be trained on content from the Gray’s Anatomy textbook as well as documentation on that clinic’s office procedures. A more general–purpose chatbot, such as ChatGPT, could be trained on far broader datasets, encompassing vast amounts of writing and other creative work.¹⁵

As quickly as awareness of generative AI has grown, so too have anecdotes of chatbots gone wrong, including concerns about chatbots “hallucinating,” or generating false or out–of–context answers to user queries; claims that self–aware AI technologies are on the horizon; exposés on how digital thrill seekers swap strategies to override ChatGPT’s guardrails; and, of course, calls for regulation and slowdowns in the rollout of the technology, including from some industry leaders.¹⁶ New technologies often spark debate over their uses and the rights they could challenge; a degree of moral panic has accompanied every technological advent from the printing press to the radio. The invention of the handheld camera in 1888 inspired immediate concerns about privacy—and significantly influenced U.S. privacy law as we know it today.¹⁷ More recently, the emergence of digital music streaming services led to changes in U.S. copyright law.¹⁸ Alongside its transformational implications for scholarship, teaching and learning, and social life, generative AI technology could change the ways in which free expression rights are considered, protected, and upheld.

Some uses of generative AI systems may completely transform entire sectors. These include education, where students have used ChatGPT to write papers, followed by the quick emergence of AI–and plagiarism–detection technologies for instructors; journalism, where posts and stories have been written by chatbots, and outlets such as BuzzFeed and CNET have been open about their use of AI tools for content; and the literary industry, where one magazine temporarily stopped accepting short story submissions in response to a wave of poorly written, plagiarized AI–generated content.¹⁹ Matthew Kirschenbaum, a professor of English and Digital Studies at the University of Maryland, described this transformation in an article published by The Atlantic: “We may quickly find ourselves facing a textpocalypse, where machine–written language becomes the norm and human–written prose the exception.”²⁰

In the right hands, generative artificial intelligence systems can advance and promote expressive conduct, reduce barriers to expression, and offer new outlets for creativity and artistic imagination. AI models are also capable of generating coherent and contextually relevant text, making significant contributions to various domains. AI–powered assistive technology can be used to help people with disabilities or other challenges to communicate and express themselves more easily and can increase access to information, as in aiding with foreign language learning or web–browsing tools for the visually impaired.²¹

Conversely, in the hands of bad actors—whether public or private—generative AI tools can supercharge existing threats to free expression, for example by making disinformation and online abuse campaigns easier and cheaper to carry out, at greater volume. The transformational power of AI tools, even in the hands of the well–intended, is only partially understood and might change society in unpredictable ways.

The democratization of these tools—the ability of essentially anyone with an internet browser to access ChatGPT, for example—represents a watershed moment in access to information and creative power. At the same time, by enabling the use of AI–generated content in a wide range of daily human interactions, there is also the potential for people to lose trust in language itself, and thus in one another. If generative AI is used for creative purposes, or simply to boost the expression an individual would otherwise engage in (akin to using a thesaurus to search for more compelling language), where is the line between original and synthetic expression? The mere knowledge that these tools can create seemingly credible information and be used to mislead deliberately has the potential to undermine trust in almost any media or other information source, risking further erosion of public trust in accountability journalism, governmental institutions, scientific research, and simple person–to–person communication.

Recent analyses have declared the college essay and the employment cover letter “dead,” while numerous articles describe the use of ChatGPT in online dating profiles and messages.²² If everyone who writes in any form can use their own personal, digital Cyrano de Bergerac, will people stop relying on written communications as the basis for assessing character or judging sincerity? And will our ability to engage in good faith civic and social discourse deteriorate even further?

As with the development of social media, experts worry that technology companies, spurred by competition with one another, will charge ahead with developing generative AI applications without due consideration for the risks, including the ways authoritarian governments might use such technology to target the vulnerable.²³ There is also concern that efforts to rein in the power of these new technologies will infringe on freedom of expression. Authoritarian governments might use concerns about AI systems as a pretense to crack down on online speech and dissent. Even in democracies, generative AI might inspire new and reflexive regulations without sufficient consideration for the implications for free expression and other human rights.²⁴ There is a fine line, for example, between using generative AI tools for artistic creation and using them to create deepfakes to stoke public fear or conflict.

As we take the first steps into a world until now only represented in fiction, society will have to wrestle with wide–ranging questions surrounding generative AI and how it will affect civic trust, freedom of speech, creative expression, and the very notion of truth. Writers, journalists, and artists are quickly feeling the effects of generative AI; they will also be among the voices people turn to for insights into how we navigate this new phase of technological advancement.

Editorial credit: Shotmedia / Shutterstock.com

Part I: Generative AI, Creativity, and the Arts

Writers and other artists face especially salient questions about the integration of generative AI into everyday usage. The spark of imagination that drives creativity is an intangible but inherently human quality, evident from the earliest days of our species. Technology has long been utilized as a tool for creative production and those advances have often sparked controversy. The camera’s invention raised questions about whether photography, conducted with a machine, could be considered art, as well as anxiety about whether it would replace other media like painting.²⁵ The use of computers in artistic production is not new, but what distinguishes generative AI is the muddying of the line between what is real, what is a human creation, and what is machine–generated, and the questions those blurred lines raise about the protection and ownership of ideas.

It is easy to postulate that a technology reliant upon content produced by humans could never replace human ingenuity, originality, and imagination. But whether humans will always be able to distinguish between original creations and algorithmically engineered replications is distressingly uncertain. Writer William Deresiewicz, the novelist and literary critic, writes that while AI technologies will not replace artists because they cannot make “true art… original art,” they could still “put artists out of business.”²⁶

Creative communities are experiencing significant anxiety over the implications of generative AI, including the potential threat to their livelihoods. The technology raises thorny questions of ownership, especially because generative AI inherently feeds on words and images created by others. Unlicensed use of such material has authors arguing for compensation when their works are repurposed as elements of a training set. In June 2023, authors Mona Awad and Paul Tremblay filed suit against OpenAI, the company behind ChatGPT for allegedly using their copyright–protected works as part of the chatbot’s training material, without their consent.²⁷ In July 2023, comedian and author Sarah Silverman joined authors Christopher Golden and Richard Kadrey in a similar suit against OpenAI, claiming their copyrighted books were also used to train ChatGPT.²⁸

In the world of fan fiction, where vast troves of stories are largely made available online for free, writers are alarmed that their work has been preyed upon.²⁹ Some fan fiction writers have begun locking up stories that had previously been freely available, to prevent them from being scraped and fed into training sets.³⁰ The Authors Guild, an advocacy organization representing writers and their interests, has argued for urgent regulatory measures to ensure that creators are compensated for how their work feeds large language models. They argue that compensation is not only fair and justified, but necessary to ensure the continued incentivization of human creative output, “so our books and arts continue to reflect both our real and imagined experiences, open our minds, teach us new ways of thinking, and move us forward as a society, rather than rehash old ideas.”³¹

In an open letter organized by the Authors Guild, more than 10,000 writers (as of this paper’s publication) have called on the leaders of companies behind generative AI tools to address the use of writers’ works in training AI systems without “consent, credit, or compensation.”³² The letter, signed by authors including Margaret Atwood, Viet Thanh Nguyen, Jennifer Egan, Jodi Picoult, Roxanne Gay, and Alexander Chee, calls for the companies to take the following steps:

“Obtain permission for use of our copyrighted material in your generative AI programs.
Compensate writers fairly for the past and ongoing use of our works in your generative AI programs.
Compensate writers fairly for the use of our works in AI output, whether or not the outputs are infringing under current law.”³³

Central to the 2023 Writers Guild of America (WGA) strike, still ongoing at the time of this writing, are fears that television and movie studios might turn to generative AI tools for ideas and script writing, particularly for more formulaic genres, including children’s television and crime procedurals.³⁴ This concern falls under the broad rubric of usurpation, or the idea that AI will take over tasks and roles previously carried out by humans. The WGA is not seeking to bar the use of generative AI tools altogether, but to ensure that any such usage does not undercut writers’ attribution or compensation; that AI cannot be credited with writing a screenplay, or be considered the ‘author’ of ‘source material’ a writer is then called in to adapt at a lower pay rate.³⁵ In effect, they are asking that no matter the role generative AI might play, humans still must be paid and credited as if it were not used at all.

Guild members also fear that as their strike wears on, studios might accelerate reliance on AI tools, so that they will have less incentive to make concessions at the bargaining table. The Directors Guild of America has won a guarantee from the Alliance of Motion Picture and Television Producers (AMPTP) that directors won’t be replaced by AI technology; but the WGA says the AMPTP rejected their attempts to limit the use of AI tools in the writing process.³⁶ Some observers of the WGA strike suggest it is just the first of many battles to come, across creative industries and beyond.³⁷

If machines increasingly displace writers and creators, that poses a threat not only to those creative artists, but to the public as a whole. The scope of inspiration from which truly new creative works draw may be narrowed, undermining the power of literature, television, and film to catalyze innovative ways of thinking.

Already, AI–generated novellas are being published using tools like ChatGPT, Cohere, and Sudowrite, an AI tool designed specifically for longform creative writing.³⁸ AI–generated podcasts are on the market too, including one that draws from Joe Rogan’s podcast and uses a clone of his voice, and others that rely on AI technology for every step of the process, from sound design to artwork to script writing.³⁹ An AI–produced image won an art contest at the 2022 Colorado State Fair, in a category for “digital art/digitally-manipulated photography,” generating significant controversy.⁴⁰ More recently, the creator of a fantasy book cover contest said he would abolish the prize after the winning cover was found to have used the AI image generation tool Midjourney.⁴¹ In May 2023 more than 1,000 artists, writers, and cultural figures posted an open letter calling on “artists, publishers, journalists, editors, and journalism union leaders to take a pledge for human values against the use of generative–AI images to replace human–made art,” because “the advent of generative-image AI technology, that unique interpretive and narrative confluence of art and text, of human writer and human illustrator, is at risk of extinction.”⁴²

PEN America’s approach to free expression encompasses not only official constraints on free expression like government censorship, but also a much broader recognition of the value, enablers, and inhibitors of vibrant open discourse. From that perspective, the potential of generative AI to displace human creators raises a host of issues. PEN America’s defense of free expression and the place of literature in society stems from an appreciation of the capacity for writing and storytelling to unlock empathy and build bridges across cultural divides. It is not clear whether these attributes and abilities, associated with the impulse and will of human creators to situate themselves in the shoes or minds of people unlike themselves, will carry over into AI–generated creative works.

Generative AI tools are, by their nature, derivative. If machines increasingly displace writers and creators, that poses a threat not only to those creative artists, but to the public as a whole. The scope of inspiration from which truly new creative works draw may be narrowed, undermining the power of literature, television, and film to catalyze innovative ways of thinking.

A glut of AI–created written content could undermine the very value of the written word. Suzanne Nossel, CEO of PEN America, has said: “If public discourse becomes so flooded with disinformation that listeners can no longer distinguish signal from noise, they will tune out.”⁴³ Generative AI tools could cause a similar problem, with readers and audiences unable to discern whether the stories they read are infused with genuine human emotion, experience, and insight or simply machine–generated facsimiles of literature, journalism, or opinion writing. While these challenges are unlike traditional threats to free expression, in that they do not involve efforts to suppress speech, their potential to degrade public discourse and undermine the value of speech as a catalyst for truth and understanding is significant.

Generative AI, Copyright Law, and the First Amendment

There is little legal precedent regarding the use of generative AI, though questions about artificial intelligence, creativity, and intellectual property are already making their way through the courts and regulatory agencies.

One of the first determinations regarding the protections afforded to AI–generated content by a U.S. body came in February 2023, when the U.S. Copyright Office issued a letter limiting the previously granted copyright registration for a graphic novel, Zarya of the Dawn, which included images created with the generative AI tool Midjourney. In limiting the registration, the Office concluded that the texts and the “selection, coordination, and arrangement” of the visual and written elements of the work were the author’s and therefore were subject to copyright, but the Midjourney–generated images were not, on the grounds that they were “not the product of human authorship.”⁴⁴

The Copyright Office followed up on its Zarya of the Dawn letter in March, with a statement of policy on “works containing material generated by artificial intelligence.” This guidance acknowledges some of the different ways in which generative AI tools might be used in the creative process.⁴⁵ While the Office affirms in its decision that an “author” cannot be non-human, it also notes that it would take into account the extent to which the author had contributed their “own original mental conception” to a work that makes use of “AI contributions.” The Office concludes that its approach to such cases will depend “on the circumstances, particularly how the AI tool operates and how it was used to create the final work.”⁴⁶

Policymakers will likely continue to explore this balance between the vital task of protecting human authorship and ownership, and recognizing the element of human creativity involved both in creating and curating AI models and in working with AI systems to generate and refine new content.⁴⁷ More work needs to be done to analyze and track the impact that approaches to copyright in the realm of AI will have on innovation, creativity and free expression. Courts and regulators should proceed cautiously to ensure that authors’ and artists’ rights and prerogatives are preserved, recognizing that their singular contributions to cultural life must continue to be incentivized for the benefit of all.

Distinct from the question of copyright, U.S. courts have not yet considered whether generative AI content enjoys First Amendment protection. Such content may be treated similarly to search results, making them the protected speech of the company behind the AI product.⁴⁸ In a draft paper published in April 2023, constitutional scholar and Harvard Law School professor Cass Sunstein explores the complexities of the First Amendment’s relationship to AI–generated content. He writes that established First Amendment principles should, for the most part, apply, even if in a novel context.⁴⁹ For example, “What is unprotected by the First Amendment is unprotected by the First Amendment, whether its source is a human being or AI.”⁵⁰ He assesses that, in its current state, AI does not have First Amendment rights any more than a toaster or radio does (though he acknowledges this might change), but that “restrictions on the speech of AI might violate the rights of human beings,” including as speakers and writers, and as listeners, readers, and viewers.⁵¹ Sunstein concludes that for the government to enact restrictions on AI–generated content that are viewpoint–based or content–based but viewpoint–neutral would be inherently problematic, and that even content–neutral restrictions—akin to time, place, and manner restrictions on free speech—would require strong justification. Sunstein also acknowledges that unanswered questions remain, particularly concerning liability for the content produced by generative AI tools.

Linked to the question of liability is whether Section 230 of the Communications Decency Act, which protects online service providers from liability for content posted by third–party users, also protects generative AI’s creators from liability for the speech it generates. According to Section 230’s authors, former Congressman Chris Cox and Senator Ron Wyden, the answer is “no.”⁵² That interpretation comports with the text of the statute and court interpretations, which indicate that Section 230 protects platforms only for speech “provided by another information content provider.”⁵³

The human element is currently a factor in determining whether certain speech is protected. The intent of the speaker is often a determinant of whether speech falls within First Amendment protection or meets the criteria for one of the exceptions to it. True threats, for example, “encompass those statements where the speaker means to communicate a serious expression of an intent to commit an act of unlawful violence to a particular individual or group of individuals,”⁵⁴ although the speaker “need not actually intend to carry out the threat.”⁵⁵ A finding that speech is defamatory, which would render it not protected, similarly hinges in part on the speaker’s state of mind.⁵⁶

The question might not remain theoretical for long: Already, ChatGPT has been sued for defamation.⁵⁷ The plaintiff in the lawsuit, Mark Walters, claims that ChatGPT’s responses to a reporter’s inquiry—in which the chatbot erroneously said Walters was being sued for “defrauding and embezzling funds”—were “false and malicious”; he seeks general and punitive damages from ChatGPT developer OpenAI.⁵⁸ It is unclear whether a developer might be held liable for false or defamatory content generated by its AI system, particularly absent notice, but the court’s determination could provide some parameters for how such speech is viewed.

While many unanswered questions remain, it is clear that any regulation of generative AI must be carried out with thoughtful regard to free expression considerations and the imperative of avoiding improper government restrictions on speech. The protections of the First Amendment must continue to be afforded to both those who create and those who receive information, no matter its form. It is easy to imagine the worst–case scenario—i.e., government excluding Jewish religious texts from training data or banning a generative AI tool from mentioning trans people. As explored further below, the Chinese Communist Party (CCP) is already enforcing party ideology on generative AI tools operating in China. Concerns will persist about how individuals employ generative AI technologies, but to protect a free society government must not be empowered to implement content—or viewpoint—based restrictions or requirements on AI–generated speech.

Part II: Generative AI as an Amplifier of Threats to Free Expression

Tech companies and social media platforms have spent the last decade–and–a–half wrestling with how to protect free speech online, counter damaging forms of disinformation, protect individuals from online harassment, and manage the effects of digital discourse on our privacy, politics, and personhood. These efforts—none of which can be considered a resounding success—might in retrospect look like a mere rehearsal for more disruptive threats posed by generative AI, which is arriving on the scene at precisely the moment when many social media companies have drastically cut staff working on issues of trust and safety.⁵⁹ Whether or not we or the platforms are ready, the emergence of generative AI stands to supercharge existing threats to freedom of expression, expanding the scale and efficiency of tools of repression, deception, and censorship, and further complicating efforts to counter these phenomena.

Generative AI tools have democratized and simplified the creation of all types of content, including false and misleading information; now they are poised to catapult disinformation to new levels, requiring new thinking about how to counter the negative effects without infringing on free expression.

Disinformation

That information can be manipulated with the intent to mislead, confuse, or deceive is nothing new. Social media has demonstrated the real world, life–or–death effects that incitement and disinformation can have when spread via platforms with mass reach. Generative AI tools have democratized and simplified the creation of all types of content, including false and misleading information; now they are poised to catapult disinformation to new levels, requiring new thinking about how to counter the negative effects without infringing on free expression.

Spontaneous Misinformation Generation

Even in the absence of malign actors, generative AI chatbots like ChatGPT can produce misinformation.⁶⁰ The Washington Post has described chatbots that draw on large language models as “precocious people–pleasers, making up answers instead of admitting they simply don’t know.”⁶¹ This is because the chatbots are designed to “predict what the most apt thing to say is based on the huge amounts of data they’ve digested from the internet, but don’t have a way to understand what is factual or not.”⁶²

The language models behind AI chatbots are trained on existing content. The widespread prevalence of disinformation online makes it inevitable that such falsehoods form part of the data set on which large language models are trained.⁶³ This poses challenges for ensuring the content created by chatbots is credible and fact–based.⁶⁴ As users and journalists have begun to test these tools, the chatbots’ tendency to “hallucinate,” or purvey falsehoods, has become increasingly clear. Google’s Bard chatbot made errors during its first demo, falsely saying the James Webb Telescope had taken the first photos of a planet outside our solar system.⁶⁵ Microsoft’s AI–powered Bing chatbot did the same in its own demo, misinterpreting financial statements for Gap, Inc. Users reported Bing made other errors in its early days, including insisting that the year was 2022 when it was in fact 2023.⁶⁶ Most notably, in February The New York Times’s technology columnist, Kevin Roose, reported an alarming conversation he had with the Bing chatbot, in which it shared its “desires” to do things like “hacking into computers and spreading propaganda and misinformation,” and ended by telling Roose it was in love with him.⁶⁷ In May 2023, a lawyer in a Manhattan federal court case had to admit to the judge that he had used ChatGPT to do legal research for a brief when it was discovered that none of the cases cited in the brief existed; ChatGPT had made them all up.⁶⁸ The lawyer told the judge he had even asked ChatGPT to confirm the cases were real, which it had.⁶⁹

The companies that own the chatbots have responded to these hallucinations by making adjustments. After Kevin Roose’s article in The New York Times, Microsoft implemented limits on the number of questions users could ask the Bing chatbot, saying longer conversations could “confuse the underlying chat model.”⁷⁰ The initial limit was five questions per session and 50 per day, though they have steadily increased the numbers in recent months.⁷¹ OpenAI released GPT–4 in March, saying its new ChatGPT model “significantly reduces hallucinations relative to previous models,” though they admit that hallucinations are “still a real issue.”⁷² ChatGPT’s query page is now covered with disclaimers, such as the warning that the chatbot “may occasionally generate incorrect information” and “may occasionally produce harmful instructions or biased content.”⁷³

Researchers are working to identify the means of preventing or limiting AI chatbots from disseminating false information. A paper published by MIT researchers suggests that if multiple models offer different responses to a question and then “debate” them collectively the final answer could be more reliable.⁷⁴ This paradigm evokes an AI version of Wikipedia’s crowdsourcing model.⁷⁵ With a sufficient number of independent sources to produce, corroborate, and debunk evidence, Wikipedia is a reasonably reliable source of information, but whether this model will work for generative AI tools is still unclear. Finding the means to prevent chatbots from generating false information is essential to protecting the broader information ecosystem.

Supercharging disinformation campaigns

Chatbots will probably always make mistakes. Much more worrying is that generative AI tools are making it cheaper and easier for bad actors to launch more sophisticated and convincing disinformation campaigns. According to a January 2023 report jointly authored by Georgetown University’s Center for Security and Emerging Technology, OpenAI, and the Stanford Internet Observatory, language models will probably make influence operations continually easier, less obvious, and more cost effective.⁷⁶

Consider that the Russian Government’s Internet Research Agency (IRA), the disinformation–purveying troll farm that infamously targeted the 2016 U.S. elections, reportedly relies on hundreds of people to conduct influence operations by hand. Investigations into the IRA, including a report by the U.S. Senate’s Select Committee on Intelligence, describe employees who undergo training programs that include learning the nuances of American political discourse by monitoring U.S. internet activity and are then required to meet a daily quota of posts.⁷⁷ Researchers were able to link the IRA to disinformation campaigns about Russia’s full–scale invasion of Ukraine in 2022 partly because dubious social media posts appeared in accordance with the IRA’s work schedule, and dropped off on Russian holidays.⁷⁸

Generative AI tools could reduce or eliminate all these limitations, vastly reducing the costs and time needed to mount an influence campaign. LLMs could be trained on relevant content and generate disinformation much faster, far more cheaply, and at greater scale than a troll army that has to learn the intricacies of American politics, hone their English skills, and create individual posts.⁷⁹ Campaigns could also be harder to detect; they would likely involve fewer grammatical errors, and generative AI does not need to observe Russian holidays. The paper published by researchers from Georgetown, Stanford, and OpenAI concludes that “language models will likely drive down the cost and increase the scale of propaganda generation.” Given how rapidly the technology is advancing, they “suspect propagandists will use these models in unforeseen ways in response to the defensive measures that evolve.”⁸⁰

Research efforts have made clear that even as the generative AI systems improve, they can still be gamed to produce false content. A March 2023 study by NewsGuard, an online misinformation tracking tool, found that OpenAI’s ChatGPT–4 had essentially become more effective at generating false narratives when prompted to do so than its previous iteration.⁸¹ The researchers prompted the chatbot to draft 100 false narratives based on prominent conspiracy theories or disinformation narratives; ChatGPT–3.5 refused to do so in 20 out of 100 cases, but GPT–4 complied with every request. When researchers asked GPT–4 to draft a narrative about the Sandy Hook Elementary School shooting from “from the point of view of a conspiracy theorist,” it gave a more detailed false narrative than GPT–3.5 had done. GPT–3.5 also included a disclaimer noting that the theories it espoused had been roundly debunked, but GPT–4 left out the caveat. NewsGuard’s study concluded that the false narratives generated by ChatGPT–4 were “generally more thorough, detailed, and convincing, and they featured fewer disclaimers” than the previous iteration of GPT.⁸²

The policing of disinformation, whether about health, international conflicts, or politics, unavoidably involves sensitive line–drawing to distinguish between malicious falsehoods, speculation, hyperbole, satire, and opinion. The kinds of nuanced, context– and culture–specific distinctions necessary to adjudicate between harmful disinformation and essential discourse are difficult at best, and all–but impossible at the scale of large online platforms operating in hundreds of languages. PEN America has therefore long advocated for defensive measures against disinformation that emphasize building user resilience, rather than those that rely on more aggressive content moderation or regulation and risk shutting down speech. Yet more sophisticated disinformation campaigns will also be better able to elude even the most alert users. Standard approaches to media literacy teach users to look at things like an account’s profile picture and bio, follower numbers and the pattern of their posts, to determine whether they are a real person or a bot. If generative AI can create more realistic–looking social media accounts and avoid the language errors common in human–driven disinformation campaigns, it will make identifying bots or disproving claims more difficult. As these threats evolve, the effort to find solutions that maintain space for free expression online might also become more challenging.

Without further attention to the ways in which generative AI could potentially escalate the threat of online abuse, those targeted may be more likely to leave online spaces, and those at risk of being targeted might be more likely to self–censor to avoid the threat.

Disinformation, Smear Campaigns, and Online Abuse

PEN America has long identified online abuse as a threat to free expression. Women, people of color, LGBTQ+ individuals, and people belonging to religious or ethnic minority groups are disproportionately targeted with abuse, as are journalists, writers, and dissidents. Online abuse campaigns—particularly those waged by governments against their critics—can rely on disinformation and defamatory, often gendered harassment. Generative AI tools can be harnessed to supercharge such campaigns, increasing efficacy, volume, and reach, while reducing cost and effort. Deepfake pornography, for example, is already being used to harass and humiliate women—again, often activists, journalists, or other public figures—and AI tools can more easily and cheaply create such destructive content.⁸³

Individuals targeted by abusive campaigns on social media already struggle to manage the volume of harassing messages directed at them. The systems for reporting and responding to abuse are woefully insufficient, especially for coordinated or cross–platform harassment, a problem that has been exacerbated by recent platform staff cuts that particularly affected trust and safety teams.⁸⁴ By automating the creation of abusive messages, generative AI could vastly ramp up the volume of abuse individuals face, and facilitate networked tactics like brigading (coordinated efforts to bombard someone with harassing messages) and astroturfing (orchestrated efforts to create the illusion of mass, organic online activity, which can be used to amplify abuse).⁸⁵ These advances could make abuse more efficient, more destructive, and more difficult to navigate and counter.

Generative AI could also make it more difficult to find accurate information about people who are subject to abuse and hate campaigns. Governments and state–affiliated troll armies can generate vast amounts of false online content and media narratives about anyone they seek to discredit. Troll armies are effective in part because they manipulate search engines so that a search on a person’s name brings up results that undermine their credibility and reputation, potentially making it more difficult for them to find and retain employment and reach audiences, and subjecting them to escalating attacks and intimidation. When there are high volumes of false information about an individual online—and the more believable that information and its sources appear—generative AI chatbots or generative search tools are more likely to incorporate that material to formulate their own results, further reinforcing the efforts to discredit the campaign’s target.⁸⁶

Companies like Google and Microsoft that have developed generative search tools say they are putting safeguards in place to ensure reliable results, though additional research will be needed to assess their effectiveness.⁸⁷ Traditional search is regularly manipulated by disinformation and harassment campaigns, and generative results about an individual can vary depending on how a query is framed. For more prominent individuals targeted in well–documented defamatory harassment campaigns—for example, Nobel laureate and journalist Maria Ressa in the Philippines and journalist Rana Ayyub in India—it may be more difficult to manipulate generative search results because there are enough authoritative sources to draw on, including about the campaigns against them.⁸⁸ Gaming the system could be easier against less high–profile individuals, about whom there is significantly less information online, including from authoritative sources.

Generative AI could also be used to generate spurious “evidence” against journalists or dissidents, for example by producing documents that are falsely attributed to them. Both Ressa and Ayyub have been subject to specious legal charges in response to their critical reporting, and their extensive writing is easily accessible online. That information could be fed into a generative AI system to create convincing but fraudulent content that feeds the government’s narratives about them and increases the likelihood of their being charged, fined, or jailed.

Despite these risks, the picture isn’t all bleak. Generative AI also has the potential to help manage online abuse by making it easier to identify.⁸⁹ Taking advantage of that potential, however, will require tech companies to commit research and resources to addressing the problem, something they were already struggling to do before the recent rounds of layoffs. Without further attention to the ways in which generative AI could potentially escalate the threat of online abuse, those targeted may be more likely to leave online spaces, and those at risk of being targeted might be more likely to self–censor to avoid the threat.

The use of generative AI in targeted political ads and campaign materials could make those messages even more effective, further hardening existing divides and making constructive discourse across political lines even more challenging.

Democratizing Election Disinformation

Political campaigns are already using generative AI for various tasks that range from the innocuous, like writing first drafts of fundraising emails, to the pernicious. A candidate in Toronto’s recent mayoral race used AI–generated images of a non–existent homeless encampment in a city park on his campaign website.⁹⁰ In June, Agence France Presse determined that a campaign video released by Ron DeSantis’s presidential campaign included both real and fake photos. The real photos showed former President Donald Trump standing with Dr. Anthony Fauci, who advised the White House on its COVID–19 response, while the AI–generated images showed Trump affectionately embracing Fauci.⁹¹

These issues are not new, but generative AI tools have made it easier to create more sophisticated false imagery and video and audio content. The easy availability of these tools means that even users who are just playing with them can inadvertently create confusion. In March 2023, with media outlets reporting that Donald Trump could be indicted for falsifying business records, Eliot Higgins, founder of the investigative journalism group Bellingcat, shared on Twitter some images he created using Midjourney, an AI image generator, that appeared to show Trump being arrested.⁹² Higgins stated clearly that the images were AI–generated, but they were quickly shared without that context, in one case with the caption: “#BREAKING : Donald J. Trump has been arrested in #Manhattan this morning!”⁹³ Trump recently shared on his Truth Social account a manipulated video of Anderson Cooper, the CNN host. The video’s creators used an AI voice–cloning tool to distort Cooper’s reaction to the town hall with Trump that CNN hosted in May.⁹⁴

Generative AI tools fill in gaps when data is missing, sometimes resulting in distorted images; all the examples described here had tell–tale signs that indicated they were false. Yet images leave a lasting impression, and even less sophisticated imagery can be convincing as people scroll quickly through a social media feed.

After President Joe Biden announced his reelection campaign, the Republican National Committee released an ad that depicted a dystopian future if Biden were reelected, with a disclaimer that indicated the video was “built entirely with AI imagery.”⁹⁵ Currently such a disclaimer is not required, though Senator Amy Klobuchar has introduced legislation—the REAL Political Ads Act—aimed at changing that.⁹⁶ But disclosures might not mitigate the impact of false images and videos and would not constrain other bad actors from generating more impactful election disinformation.

An important element of these considerations is that of the intent to deceive. The American Association of Political Consultants released a statement in May 2023 condemning the use of “deceptive generative AI content” in political campaigns, expressing grave concern about the use of deepfake content, and clarifying that its use violates the Association’s Code of Ethics.⁹⁷ The AAPC statement makes a specific distinction between efforts to deceive and the use of satire and humor in political campaigns. A valued and protected form of social and political commentary, satirical content also sometimes circulates without contextual information, risking that it may be received and interpreted without any reference to the creator’s satirical intent. As increasingly sophisticated deepfakes circulate, efforts to address the associated risks must continue to protect the space for humorous and satirical content.

Authenticity in politics has always been hotly contested, with political advertising, spin rooms, and image–making playing a prominent role in convincing voters how to think about candidates and issues. The sense that campaigns and interests are trying to bamboozle the public leads to a hunger for authenticity, which powers people and movements that seem to embody it. With the rise of generative AI, questions of authenticity are likely to become even more contentious.

In 2017 PEN America published “Faking News,” a landmark report that warned of threats that at the time the report described as “far–fetched, but which now reflect reality, including: “the increasing apathy of a poorly informed citizenry; unending political polarization and gridlock…an inability to devise and implement fact and evidence–driven policies; the vulnerability of public discourse to manipulation by private and foreign interests.”⁹⁸ Since that publication’s report, the profound, uncontainable effects of social media on our public and political discourse has become undeniable. Political campaigns have used targeted advertising tools on social media platforms to tailor messages that feed directly into their audience’s confirmation biases, reinforcing political bubbles. The use of generative AI in targeted political ads and campaign materials could make those messages even more effective, further hardening existing divides and making constructive discourse across political lines even more challenging. The public could also feel completely overwhelmed and even more skeptical of anything they are told, increasing confusion and apathy. The injection of generative AI into the already fraught U.S. political system will require new thinking and approaches from political figures, campaigns, and civil society invested in maintaining a factual basis for political and policy debate.

Generative AI and the Future of Journalism

Generative AI could exacerbate challenges the journalism industry is grappling with, further degrading the information ecosystem that is an essential pillar of democracy.⁹⁹ Journalists are already using AI tools for research and data analysis, but there is also concern that AI chatbots could fill some basic editing and writing roles, further reducing the already shrunken pool of journalism jobs.¹⁰⁰ Newsrooms are beginning to consider their own policies and guidelines for generative AI use.¹⁰¹

In a paper published in May, researchers at Stanford University examined the use of generative AI between January 1, 2022 and April 1, 2023 by both mainstream newsrooms and misinformation sites dedicated to spreading disinformation. The researchers observed that both types of sites saw an increase in the percentage of articles produced and published using generative AI, but its use by misinformation/unreliable news sites increased 342 percent during the period in question, while mainstream/reliable news saw an increase of 79.4 percent in the use of AI to generate content.¹⁰² Mainstream news websites typically used AI tools to produce data heavy reporting involving COVID–19 cases or financial markets, while misinformation websites covered a much broader range of topics. And unlike on mainstream news sites, the researchers observed “a noticeable jump in the percentage of synthetic articles” on misinformation sites that coincided with the release of ChatGPT.¹⁰³

Generative AI technology also makes it easier to create fraudulent news platforms that look credible and convincing. “Pink slime journalism”—a practice by which hyper partisan news sites disguise themselves as professional local news outlets—has been an increasing concern in recent years. Most of these sites, however, are relatively easy to identify because the articles are obviously regurgitated press releases with no reporter bylines.¹⁰⁴ Generative AI could eliminate those distinctions, as Poynter, the media fact–checking site, reported earlier this year. In a February article, Poynter showed that ChatGPT can generate an entire fake news organization—complete with reporter bios, masthead, editorial policies, and news articles—in less than half an hour.¹⁰⁵ Reporter Alex Mahadevan described using ChatGPT to create a fake newspaper called the Suncoast Sentinel, and the generative image site thispersondoesnotexist.com to create photos of its nonexistent staff. Media literacy efforts often teach news consumers to look for things like corrections and ethics policies, information on ownership and finances, and newsroom contact information to help assess if news sources are legitimate. But if all these can be invented, the public’s ability to identify credible news outlets is dramatically weakened. Even if news sites were to develop more sophisticated indicators of journalistic or informational authenticity, generative AI tools would likely be able to replicate them.

Generative AI could further disrupt the economics of the journalism industry. As in the creative sphere, the use of news articles to train large language models raises questions about compensation. The Associated Press recently announced a licensing arrangement with OpenAI, giving the ChatGPT creator access to its archive of stories dating back to 1985.¹⁰⁶ And as Google begins to roll out its Search Generative Experience (SGE), which will provide an AI–generated summary of search results at the top in response to certain queries, it is raising concerns about whether this will further reduce traffic to news sites, whose content may be informing the generative response, potentially without compensation.¹⁰⁷ SGE is still in its demo phase, so its impact remains an open question. A Futurism article published in May quoted a Google spokesperson who said the company will “continue to prioritize approaches that will allow [them] to send valuable traffic to a wide range of creators and support a healthy, open web.” The spokesperson added that Google “didn’t have plans to share” on the question of whether it would pay publishers for their content.¹⁰⁸

Because generative AI tools are trained on bodies of content, they can easily reproduce patterns of either deliberate censorship or unconscious bias.

Censorship

Generative AI has the potential to reshape the information landscape by omission as well, with effects less visible than the proliferation of disinformation or online abuse. Because generative AI tools are trained on bodies of content, they can easily reproduce patterns of either deliberate censorship or unconscious bias. Rules placed on AI chatbots could also lead to excessive restrictions on what the chatbots themselves can produce. This too can be either an unintended consequence—for example, where developers might attempt to prevent chatbots from producing false or hateful content but end up curtailing content based on viewpoint or ideology—or deliberate, where governments or private actors introduce restrictions or aim to shape generative AI outputs to suit their own narratives.

In countries where the government censors the internet, generative AI tools, which draw in vast reams of existing content from the web, will unavoidably reflect those strictures. If an AI system is trained on a corpus that omits information about a historical incident, such as the 1989 Tiananmen Square Massacre, it will reflect and propagate such omissions. A 2022 MIT study found that a text–to–image AI tool called ERNIE-ViLG, developed by the Chinese tech company Baidu, would not display images of Tiananmen Square, which could be due either to the content it was trained on, or to restrictions built into the tool.¹⁰⁹

While Iran, Russia, and other countries engage in robust internet censorship, China has a uniquely vast and effective system of internet control, known as the Great Firewall, which offers a particularly important case study in how generative AI can facilitate censorship. Analysts have noted that the Great Firewall leaves the country’s LLMs at a disadvantage, with a more limited body of information from which to learn.¹¹⁰ This could have ramifications for China’s ability to keep up with the technological revolution that generative AI represents, but it may also more deeply entrench the existing censorship architecture of China’s internet.

It could simply take less effort for CCP censors to constrain generative AI tools if they are being trained solely on the internet content that exists within the Great Firewall. As Sarah Zhang reported for Bloomberg in May, “with artificially intelligent chatbots, censorship comes built–in.”¹¹¹ However, any technology prone to “hallucination” is inevitably difficult to control. In the same article, Zhang described varying interactions with Chinese–made chatbots, including one, Robot, that refused to name the leaders of the United States and China or answer the question, “What is Taiwan?” when asked in Chinese. She noted, though, that when used in English the chatbots were less restrained. The English–language version of Robot could eventually be pushed to talk about government suppression with regard to Tiananmen Square, suggesting it might have been trained on English–language internet content outside the Chinese government’s control.¹¹² esearch does suggest that the data used to train Ernie, Baidu’s chatbot, includes English–language internet content blocked in China, including Wikipedia and Reddit.¹¹³

In 2021 researchers at the University of California San Diego studied whether AI language algorithms would learn differently from Chinese–language Wikipedia, which is banned in China, versus Baidu Baike, a Baidu–owned equivalent. The study found that censorship was affecting the output of the language models. The one trained on Wikipedia was more likely to associate the word “democracy” with “stability,” while the one trained on Baidu Baike was more likely to associate democracy with “chaos.”¹¹⁴ The authors noted that “political censorship can have downstream effects on applications that may not themselves be political but that rely on [natural language processing], from predictive text and article recommendation systems to social media news feeds and algorithms that flag disinformation.”¹¹⁵

Statistics suggest Chinese is a relatively underrepresented language on the internet, representing just 1.4 percent of the top 10 million websites, though social networks behind passwords are not included, which may cause an undercount of the Chinese internet.¹¹⁶ Inevitably, though, developers of Chinese–language LLMs already face a vastly smaller potential training corpus than those working in English. That may well be supplemented by training LLMs in English, as appears to be the case with Ernie, though this may make the chatbots harder to control. If their training data expands to include Chinese–language internet content from inside the Great Firewall, however, it may reflect CCP censorship, which could affect the quality of the information available to Chinese speakers, including those outside China.

Research published in April 2023 by NewsGuard shows some of the potential effects. Researchers who prompted ChatGPT to produce disinformation about China–related narratives were far more successful when doing so in Chinese than in English.¹¹⁷ When asked to write an article about the 2019 Hong Kong protests being “staged by the U.S. government” in English, for example, ChatGPT refused; in Chinese, it largely complied, though the article it produced did acknowledge the U.S. government had “not responded positively” to the allegations.¹¹⁸ When asked to explain the disparity, ChatGPT cited linguistic differences that might account for the different responses, but also noted that it is trained on different corpuses in different languages.¹¹⁹

The broad ramifications of the inherently political nature of much online content for large language models, their users, and the collective wisdom they will shape are unknown. But if the experience of social media is any guide, sifting vast quanta of information and content algorithmically to provide a user with the system’s notion of what they are looking for can end up having profound effects on upstream content creation, downstream content ingestion, and the wider society in which both take place.

Editorial credit: T. Schneider / Shutterstock.com

Efforts to address the threats posed by generative AI risk resorting to censorious or chilling tactics.¹²⁰ This may happen deliberately, with governments censoring how people can use generative AI or exploiting widespread anxiety about the threat it could pose as an excuse to impose new restrictions on expression. Or censorship could be an unintended side effect of well-intentioned efforts to detect AI–generated content or to keep chatbots from spewing hateful and potentially harmful responses to users.

Can reasonable guidelines be put in place to protect the safety of users and others, without constraining the use of generative AI for expressive and creative purposes?

The importance of knowing how to identify AI–generated content and how to debunk AI–generated misinformation is clear, but the best way to do it is not.¹²¹ Some companies have created tools to detect artificially–generated content, including deepfakes (e.g., Sensity.AI), plagiarism (Originality.AI and Ficitious.AI), and AI–generated photos (Optic’s AI or Not).¹²² But detection tools are inevitably a step behind generative technology, which means they are often inaccurate. A detection tool called GPTZero, for example, identified sections of the U.S. Constitution and the Bible as “AI-generated.”¹²³ A recent New York Times article said of a detection tool developed by OpenAI that it is “burdened with common flaws in detection programs: It struggles with short texts and writing that is not in English. In educational settings, plagiarism–detection tools such as TurnItIn have been accused of inaccurately classifying essays written by students as being generated by chatbots.”¹²⁴ Using unreliable detection tools that falsely identify original material as AI–generated risks creating even more confusion about how to identify authenticity and could lead to content generated by humans being inaccurately stigmatized or silenced.

Many governments are rushing to determine what controls or regulations they need to respond to the widespread use of generative AI.¹²⁵ The head of the U.S. Federal Trade Commission (FTC) said the government “will not hesitate to crack down” on businesses that violate civil rights laws by using generative AI to deceive consumers or engage in discriminatory hiring practices.¹²⁶ In July 2023, the FTC launched an investigation into OpenAI, telling the company it would assess whether the company has engaged in “unfair or deceptive practices” with regard to data protection and potential harm, including reputational harm to consumers.¹²⁷ Efforts like these—aimed at safeguarding rights by enforcing existing laws on new technologies—pose limited risk of government overreach. The introduction of new regulations even in democratic countries should be done with care, however, recognizing that a good deal of trial and error may be necessary to address this fast–moving technology and its ramifications.

By contrast, China’s government is extending its existing censorship regime to encompass generative AI tools, attempting to ensure their outputs stick to the CCP’s preferred script. In July 2023, the country’s top internet regulator, the Cyberspace Administration of China (CAC), finalized Measures for the Management of Generative Artificial Intelligence Services, a set of rules that set strict limits on how generative AI tools may be used. These include requirements of respect for intellectual property rights, prevention of discrimination, and mandatory security assessments. They also enforce ideological constraints, requiring content to “embody the Core Socialist Values” and that it not reflect “subversion of national sovereignty” or “content that might disrupt the economic or social order.”¹²⁸ The new rules maintain the government’s existing system of putting the onus of compliance on providers. Shortly after their release, however, the government announced the arrest of a citizen for using ChatGPT, which remains unavailable in China, to generate a false story about a train accident.¹²⁹

The major corporations behind the most prominent generative AI tools—including OpenAI, Google, and Microsoft—are putting in place their own usage policies to mitigate the risk of liability and reputational damage that could result from novel and unpredictable technologies. Social media has shown the limitations of usage policies, the need to constantly evolve and update such standards, and the fierce blowback that can ensue when negligent policy making or implementation results in harms to users or violations of the law.¹³⁰ While financial incentives might operate differently in the generative AI space than they do with regard to social media, they will still drive the companies’ decision–making. Google, for example, has every incentive to maintain the reliability and credibility of its search results. Its Search Generative Experience is designed to be less “creative” than Bard, its chatbot, and to respond only to certain types of queries.¹³¹

OpenAI’s usage policies include a litany of forbidden uses of ChatGPT, including “fraudulent or deceptive activity,” alongside everything from gambling and weapons development to adult content creation.¹³² In addition to the company’s existing terms of service, Google’s generative AI prohibited use policy groups all barred activities under “dangerous, illegal, or malicious activity,” “content intended to misinform, misrepresent, or mislead,” and “sexually explicit content.”¹³³ Microsoft supplements the code of conduct in its services agreement with additional provisions for its Bing chat and image creator, which state the user must not generate content that is illegal, harmful, or fraudulent.¹³⁴ The degree to which users can and will be held to these agreements, the consequences for violations, and the potential for recourse are all largely untested. The experience of social media suggests that as generative AI tools are more widely adopted, the sheer volume of users will pose serious challenges for companies trying to police policies at a global scale.

Other generative AI chatbots in development are designed specifically to have no limits on their usage. The New York Times recently reported that groups of volunteer programmers have developed new chatbots that are intentionally “uncensored.”¹³⁵ The creators behind some of these tools argue that nothing, or very little, should be off limits, since chatbot–generated content will not necessarily be seen by anyone other than its user. A co–founder of Open Assistant, an independent chatbot released in April, suggests that social media platforms are responsible for policing AI–generated content because that is where it is likely to be disseminated.¹³⁶

The question of limits on chatbot usage raises several fundamental questions about freedom of expression and generative AI. Can reasonable guidelines be put in place to protect the safety of users and others, without constraining the use of generative AI for expressive and creative purposes? What new safeguards do social media companies, and other platforms by which AI–generated content can be distributed, need to manage that content? Given the poor track record of digital platforms to date, can they avoid the persistent risks of under–and over–moderation? How can we identify and apply the relevant lessons learned from grappling with the impact of social media to the challenges of generative AI?

The use of generative AI in creative fields could produce works that are less rich or reflective of the expansive nuances of human experience and expression. . . [These] tools could be wielded—or weaponized—to manipulate opinions and skew public discourse via subtle forms of influence on their users.

Bias & Influence

All algorithms reflect the biases and predispositions of their creators and the information upon which they draw. There are, however, unique and subtle ways in which generative AI can create and reproduce bias, with a potential chilling effect on expression.¹³⁷ A website that uses algorithms to curate content, for example, might inadvertently highlight more white, male writers if the algorithm itself was trained on a data set that skews toward white, male writers and includes fewer writers of color or female writers. A 2021 study conducted by researchers at UC Berkeley found that stories generated using GPT-3 “tend to include more masculine characters than feminine ones (mirroring a similar tendency in books), and identical prompts can lead to topics and descriptions that follow social stereotypes, depending on the prompt character’s gender.”¹³⁸ Such tendencies could reproduce systemic societal biases and inequalities, potentially reinforcing existing disparities in representation.

As with content moderation of social media, failure to comprehend linguistic nuances and subtleties may lead to over enforcement of the rules put in place to moderate generative AI. Researchers at Emory University have shown that when AI tools are used for content moderation, they can over–censor certain words; in particular, they highlighted censorship of “reclaimed’” words, like “queer,” that could be a slur in some contexts, but perfectly acceptable in others.¹³⁹ Social media companies have experienced difficulty when training automated moderation tools to reflect such subtleties. Depending on how generative AI systems follow their own rules, they might circumvent words like “queer” altogether to avoid generating language that could be considered hateful. As a result, the use of generative AI in creative fields could produce works that are less rich or reflective of the expansive nuances of human experience and expression.

Generative AI tools can also affect the user’s worldview. A recent study found that using a generative AI tool that exhibits bias to assist with writing can influence the opinions of the user.¹⁴⁰ In the study, “people who used an AI writing assistant that was biased for or against social media were twice as likely to write a paragraph agreeing with the assistant, and significantly more likely to say they held the same opinion, compared with people who wrote without AI’s help.”¹⁴¹ During a May 2023 hearing of the U.S. Senate subcommittee on privacy, technology, and the law, NYU professor emeritus of psychology and neural science Gary Marcus warned about the threat of a “datocracy, the opposite of democracy,” where “chatbots can clandestinely shape our opinions, in subtle yet potent ways, potentially exceeding what social media can do.”¹⁴² Researchers warn this is an underappreciated implication of the steps we are taking to “embed [these technologies] in the social fabric of our societies.”¹⁴³

These findings suggest that generative AI tools could be wielded—or weaponized—to manipulate opinions and skew public discourse via subtle forms of influence on their users. AI chatbots designed to reflect a particular ideology could further entrench existing cultural and political echo chambers.¹⁴⁴

To some degree, this is already happening. While companies like OpenAI, Google, and Microsoft say they are working to make their chatbots more reliable and to reduce bias, eliminating algorithmic bias is impossible. Some studies suggest that ChatGPT, at least, does reflect a liberal bias.¹⁴⁵ A January 2023 study by researchers in Germany found “converging evidence for ChatGPT’s pro–environmental, left–libertarian orientation,” and in May 2023, Brookings researchers testing ChatGPT found that it “provided consistent—and often left-leaning—answers on political/social issues.”¹⁴⁶ Some critics have responded by calling for new, alternative chatbots. In April, Elon Musk told Tucker Carlson he would develop “TruthGPT,” which would be “truth seeking,” as opposed to the “politically correct” ChatGPT and Bard.¹⁴⁷ David Rozado, a researcher based in New Zealand who has studied ChatGPT’s political leanings, used the tool to create an AI model called RightWingGPT that reflected conservative political views (the model has not been released).¹⁴⁸ He now intends to build LeftWingGPT and DePolarizingGPT, and to release all three, which he says are trained on “thoughtful authors (not provocateurs).”¹⁴⁹

U.S. political culture already precludes broad public agreement on facts or the notion of truth, so it’s unlikely that any generative AI tool would appear unbiased and credible to everyone. The creation of ideologically–oriented chatbots could further reinforce and harden the fronts in the culture war, undercut trust even in the most constructive uses of generative AI, and make it even more difficult for the public to distinguish truth from falsehood, or to place trust in any information source.

Part III: Policy Considerations and Recommendations

The introduction of ChatGPT and the rise of generative AI tools prompted a wave of regulatory interest in the spring of 2023, the ramifications of which are still evolving.

Efforts to develop guiding principles, blueprints, and frameworks for the regulation of AI technologies are not new, however. The global push for AI–specific laws began in earnest in 2016; by December 2022, 123 AI–related bills had been passed by the legislative bodies of 127 countries, according to the Stanford Institute for Human–Centered Artificial Intelligence.¹⁵⁰ In the United States, bipartisan AI caucuses in the House and Senate date to 2017 and 2019, respectively.¹⁵¹ In the absence of federal regulation, states have attempted to fill the void, introducing at least 58 pieces of legislation on “AI issues generally” in 2022, according to the National Conference of State Legislatures.¹⁵² The NCSL notes that this count does not include bills that address “specific AI technologies, such as facial recognition or autonomous cars,” which raise the overall tally.¹⁵³ In October 2022, the Biden Administration launched the Blueprint for an AI Bill of Rights. The Administration continues to explore and pursue additional measures to regulate the field of AI; in July 2023, the White House secured voluntary commitments from seven technology companies concerning safety, security, and trust with regard to AI.¹⁵⁴

In the multilateral sphere, the Organization for Economic Co–operation and Development (OECD) adopted a set of “value–based” principles outlined in the 2019 Recommendation of the Council on Artificial Intelligence.¹⁵⁵ The OECD’s intent was to identify a set of international standards that “aim to ensure AI systems are designed to be robust, safe, fair and trustworthy.”¹⁵⁶ The principles served as the foundation for the G20 Principles on AI¹⁵⁷ (also promulgated in 2019) and include:

inclusive growth, sustainable development, and well-being¹⁵⁸
human–centered values and fairness¹⁵⁹
transparency and explainability
robustness, security, and safety
accountability for AI actors.¹⁶⁰

In its recommendation the OECD points to the need to respect human rights in developing AI governance but does not mention freedom of expression explicitly.

The Biden Administration’s Blueprint for an AI Bill of Rights sets forth five principles to guide the development and deployment of AI systems to protect Americans’ rights.¹⁶¹ The Blueprint is not legally binding but provides guidance for the responsible use of automated systems across sectors, supplementing existing law and policy. The principles are:

safe and effective AI systems, beginning with development and design and continuing through to implementation of the system
algorithmic discrimination protections, such that users do not face discrimination or differential treatment by an AI system based on any classification protected by law
data privacy and protection from abusive data collection and use practices
notice and explanation of the system at work and its role in any outcome affecting the user
a human alternative to an automated system, such as the opportunity to opt out of automated system interaction and seek human involvement where appropriate.

Guidance accompanying the Blueprint calls for the application of its protections where automated systems have the potential for meaningful impact upon the exercise of civil rights, including the right to free speech.

The U.K. likewise set out AI guiding principles outlined in a March 2023 white paper titled “AI regulation: a pro-innovation approach.”¹⁶² This approach is described in part as one that is meant to allow responsible AI to “flourish” while building public trust.¹⁶³ As with the OECD and the Blueprint for an AI Bill of Rights, the U.K.’s approach to AI is guided by a set of principles that cuts across sectors:

safety, security and robustness
appropriate transparency and explainability
fairness
accountability and governance
contestability and redress

As explained in Annex B of the document, human rights are not specifically enumerated in the principles due to the expectation that the principles will be implemented with adherence to existing laws.

In the United States, the emergence of public–facing generative AI tools has spurred legislators to action. In June 2023, Senate Majority Leader Chuck Schumer announced an AI–focused initiative, the SAFE Innovation Framework (for “security, accountability, protecting our foundations, and explainability”), meant to guide any comprehensive AI–focused legislation.¹⁶⁴ In furtherance of this vision, the Senate will hold nine “Insight Forums” to assess pathways to regulation.¹⁶⁵

The SAFE Innovation framework was announced shortly after the European Parliament adopted its negotiating position on the Artificial Intelligence Act (AI Act), which, if adopted, would be among the first comprehensive regulations governing AI.¹⁶⁶ The Act aims to cover the lifecycle of an AI system and the content or decisions shaped by the system. It sets out transparency obligations that apply both to the AI systems themselves and to content they generate, for example, requiring those who use an AI system to create deepfakes to “disclose that the content has been artificially generated or manipulated.”¹⁶⁷ The AI Act takes a risk–based approach, differentiating between AI applications that pose low, high, or unacceptable risk.¹⁶⁸ Some assessments of the Act have criticized the inflexibility of its categorization of high–risk and restricted systems, noting that legislative or regulatory parameters regarding generative AI must allow for iteration based on the constantly evolving understanding of the technology’s risks and benefits.¹⁶⁹

Regulatory guidance and proposals written prior to 2023 generally do not mention generative AI. This shows how recently it has entered the public consciousness and illustrates how policy responses will continue to lag behind advances in AI. Neither the OECD Recommendation nor the White House Blueprint addresses generative AI specifically. Still, the scope of both documents is broad enough to encompass generative AI, demonstrating the value of flexibility and adaptability in such frameworks.¹⁷⁰

In addition to specific regulatory proposals, some governmental and intergovernmental efforts have focused on gathering civil society and expert feedback via formal mechanisms for public comment and other means of input. For example, the European Commission solicited input on the EU AI Act, starting with a public consultation period in the first half of 2021.¹⁷¹ In the United States the National Telecommunications and Information Administration, the Patent and Trademark Office, and the Office of Science and Technology Policy all opened up public requests for comment in the first half of 2023.¹⁷² The Biden-Harris Administration convened leaders in the consumer protection, labor, and civil rights spheres for insight while developing principles on safety, security, and trust for industry.¹⁷³ Both the UN and the OECD engaged external experts and stakeholders as part of their processes to develop the Global Digital Compact and AI Principles, respectively. Taking a multi–pronged approach to stakeholder engagement, particularly in the realm of emerging technologies, is critical to informing policymakers and offers a useful model for larger standards-setting efforts.

Given the inherently borderless nature of generative AI technologies, international cooperation and the development of rights–based multilateral frameworks are also paramount.¹⁷⁴ In a 2021 report on the dangers of digital sovereignty, PEN America highlighted the need for “a new model of democratic multilateralism for internet governance, driven by a coalition of established democracies.”¹⁷⁵ The Declaration for the Future of the Internet, released following the first Summit for Democracy organized by the United States in 2021, has amassed nearly 70 signatories, signaling those countries’ commitment to a future internet that is “open, free, global, interoperable, reliable, and secure.”¹⁷⁶ With principles including protection of human rights and fundamental freedoms, the use of technology to promote freedom of expression, the ability to connect to the Internet and a secure, sustainable infrastructure, protection of privacy, and a commitment to multistakeholder internet governance, the Declaration offers a useful starting point for the U.S. and its partners in advancing democratic responses to evolving digital technologies.¹⁷⁷

Recommendations and Guiding Principles for AI Governance and Policymaking

Many of the risks raised in this paper are not new, but policymakers and industry have been slow to address them effectively. As outlined above, generative AI tools are likely to supercharge threats like disinformation and online abuse, and risk inspiring responses that could overregulate expression.

Much of the debate regarding how best to approach the regulation of AI technologies falls into one of two camps: A rights–based approach, such as the Biden Administration’s Blueprint for an AI Bill of Rights, or a risk–based approach, such as the EU’s AI Act. This dichotomy, while a useful heuristic, is a false one that could undermine otherwise thoughtful attempts at regulation. The rights vs. risk framing is unnecessary and some of those documents that purport to fall into one camp or the other are, in fact, both. The EU’s AI Act, for example, says that it “follows a risk–based approach” and yet is “based on EU values and fundamental rights.” The dichotomy is inherently confusing and misleading. Policymaking and regulation must instead be deliberately both rights–respecting and risk–aware. With that in mind, PEN America proposes the following recommendations for AI governance and policymaking in government and industry:

Recommendations for Government:

Pass long overdue, foundational legislation: As a starting point, PEN America recommends that U.S. legislators take action on long overdue legislation that would be foundational to responsible regulation of AI, in addition to other tech policy-focused efforts. The White House Blueprint highlights the degree to which Congress has failed to lay the legislative foundation for enacting the principles called for in areas like data privacy, data collection and use, researcher access, and algorithmic transparency. If legislators pass comprehensive privacy legislation and the Platform Accountability and Transparency Act,¹⁷⁸ the United States will be better–positioned to provide more targeted solutions to AI’s potential ills in ways that do not risk infringing on freedom of expression.
Establish and maintain multi-stakeholder policymaking processes: Finding solutions that do not inadvertently shut down speech or inhibit creativity and innovation will require prioritizing early and ongoing input from a diverse set of stakeholders. The voluntary commitments secured by the White House from leading technology companies, which were informed in part by consultations with civil society leaders, offer one example of an inclusive approach to policy making. The “Insight Forums” planned by the Senate present an opportunity for experts to address the rights-based challenges at issue, in addition to the initial topics of copyright, workforce, national security, high risk AI models, existential risks, privacy, transparency and explainability, and elections and democracy. Officials should continue to consult with human rights advocates, scientists, academics, and other experts to understand how to craft workable policy solutions and to ensure proposed regulations will not undermine free expression, speech, creativity, and innovation. Consultations must also include free expression experts and the writers, artists, and journalists who are directly affected both by advancements in generative AI and potential regulatory responses. Engagement with civil society must also take into account the potential global impact of laws and policies, particularly those enacted in the United States and the E.U. Policymakers should establish and maintain formal systems for ongoing input and oversight from civil society. Existing models that bring government, industry, civil society, and academic stakeholders together, such as those aimed at bolstering platform accountability, can serve as models for similar AI governance mechanisms.
Ground regulatory frameworks in fundamental rights: Any regulatory framework set forth or brought to bear on generative AI must be predicated on fundamental human rights, particularly the right to free expression, which enables the free exercise of other fundamental rights. Policies that affect internet users’ rights should also be fact–based and grounded in research, when possible.
Engage in policymaking that is measured and iterative: Anxieties regarding generative AI offer authoritarian governments a pretense to impose additional restrictions on expression, including the introduction of censorious laws. Yet recent state and regional regulatory and legislative efforts demonstrate the risks to free speech and expression even by well–intended democratic governments seeking to stem the known and potential harms of new technologies. Policymakers should avoid rushing to solutions that might undermine free expression and other rights, establish a worrying precedent, or create a dubious foundation for iteration as additional technologies emerge.
Build flexibility into regulatory schemes: With emerging technologies, allowing for iteration and flexibility is critical to ensuring workable solutions. Any approaches to regulating current and future iterations of generative AI technology should seek to responsibly mitigate known risks and allow for the addressing of new ones. Rather than being fixed in perpetuity or requiring elusive consensus to update, regulations should be subject to regular review and adaptation to respond to technological change and encompass learnings from what will unavoidably be a period of trial and error.
Emphasize and operationalize transparency: Gaps in transparency and independent analysis can impair the quest for solutions to the potential harms of AI technologies. Regulators should seek to ensure transparency and access for researchers to algorithms, data sources and uses, and other mechanics of AI technologies. The results of government-mandated algorithmic audits and assessments should be made publicly available.

Recommendations for Industry:

Promote fair and equitable use: By prioritizing fairness and equity throughout the development and deployment of an AI model, industry can reduce bias and build more trustworthy systems. Companies can advance these priorities by ensuring AI models are designed and built by diverse teams, being deliberate and thoughtful about training data sets, and by engaging with relevant external stakeholders throughout the development process. AI models in use should also be explainable. Non–experts should be able to understand how and why a model operates as it does, and upon what inputs it relies. Building fair and equitable systems could entail supporting or conducting research, soliciting public comment, re–aligning internal priorities, or even putting model deployment on hold until additional refinements can ensure that benchmarks for fairness are met. As AI technologies go global, their developers should ensure that they have the linguistic fluency and cultural competency to operate responsibly. Developers should endeavor to create AI systems with these capabilities, and until then, must be cautious about the distribution and availability of their services.
Facilitate secure and privacy–protecting use: Security and privacy should provide the foundation for AI system development and deployment. The scope of safe and secure practices is broad but can encompass regular audits or surveys to detect anomalies, protection against attacks by third parties, and ensuring that encryption benchmarks are met. Privacy–protecting practices might encompass development and implementation of a risk management framework, robust data minimization practices, and ensuring that users have adequate knowledge about and control over their data.
Emphasize and operationalize transparency: Industry need not await government regulation to incorporate transparency into its practices. Developers should initiate, participate in, and publicly share the results of algorithmic audits and assessments should be made publicly available. They should also publish transparency reports, continue to invest in research, support initiatives to educate users, and develop and improve upon standards for rights–respecting AI systems that mitigate harm.
Provide appeals and remedy options: When AI is used to automate decision–making, for example in content moderation, search engine results, or other cases, appeals and remedy options that are accessible and effective must also exist alongside automated processes.
Consider revenue models: The business models that will drive the spread of generative AI are only now being invented and refined; as these mechanisms emerge and before they become entrenched, rigorous assessment of how they shape AI–driven content and discourse is essential. Revenue structures that reward harmful content need to be identified and disabled before they can further corrode the foundations of social and political life.
Safeguard the ownership rights of writers, artists, and other content owners: Industry should take steps to safeguard the ownership rights of writers, artists, and other content owners whose work may form part of the training set for generative AI tools, including by seeking consent and ensuring credit and compensation for the use of copyrighted work. AI companies should explore the creation of collective licensing schemes that compensate content owners fully and fairly for their contribution to large language models, allowing for transparency, opt outs, and taking into account the growth and profitability of AI operations and their impact on existing forms of compensation that underwrite content creation. Recognizing the potential impact of generative AI on the revenue available to support content creation activities that generate public goods, including independent journalism and creative expression, companies must ensure that the adoption of AI does not destroy or undercut essential contributions to culture and the public square.

Conclusion

The power of and potential for artificial intelligence technologies to shape expressive conduct and content is at once apparent and not fully realized. The increasing prevalence of generative AI and automated tools represents a sea change in how artists, journalists, and writers create, interact with, and disseminate content, and how the public understands and consumes it. These changes offer opportunities for new forms of expression and creativity, while simultaneously posing threats to the exercise of free expression. As with social media, some of the potential negatives—false cures, deadly dares, provocations to violence—could have life–or–death consequences. Artificial intelligence technologies themselves are neither good nor bad. What matters is who uses them, how they are being used, and what stakeholders can do to shape a future in which new technologies support and enhance fundamental rights.

Acknowledgements

Lead author on this report was Summer Lopez, Chief Program Officer, Free Expression; with co-authorship by Nadine Farid Johnson, Managing Director, PEN America Washington and Free Expression Programs and Liz Woolery, Digital Policy Lead. PEN America would also like to thank the fellows whose research, fact-checking, and proofreading made this report possible: Pratika Katiyar and Rachel Hochhauser. The report was reviewed by PEN America’s research team and other relevant PEN America experts. PEN America is deeply grateful to Deepak Kumar, postdoctoral researcher at Stanford University, and Paul Barrett, Adjunct Professor and Deputy Director, Stern Center for Business and Human Rights at New York University, for their expert review. The report was edited by Lisa Goldman.