Understanding Red Teaming for Generative AI: A Case Study at Dana-Farber Cancer Institute
Generative artificial intelligence (AI) is transforming various sectors, including healthcare. However, the deployment of models like GPT-4 raises crucial questions regarding copyright compliance, especially within academic medical institutions. A recent exercise conducted by the Dana-Farber Cancer Institute explored these concerns through a comprehensive red teaming initiative aimed at assessing the integrity and functionality of its internal generative AI tool, GPT4DFCI.
Background on Generative AI in Healthcare
As healthcare organizations increasingly adopt generative AI solutions, the potential risks associated with copyright infringement become more pronounced. The Dana-Farber Cancer Institute took significant steps to ensure that their internally developed GPT-4 tool, GPT4DFCI, adheres to copyright laws while fulfilling its mission in research and operations. With the shared responsibility outlined by Microsoft OpenAI products, it became imperative to test this tool rigorously for compliance.
The Red Teaming Exercise
In November 2024, a structured red teaming exercise was conducted involving 42 participants from diverse sectors such as academia, industry, and government. The goal was to explore four critical domains of copyrighted content: literary works, news articles, scientific publications, and clinical notes that are access-restricted.
Literary Works
One of the pivotal findings of the exercise occurred in the literary works domain. The red teams successfully extracted verbatim quotes from book dedications and near-exact passages through what is termed “indirect prompting strategies.” This discovery raises important questions about the presence of copyrighted materials within the training data used to develop generative AI models. It underscores the need for enhanced filtering mechanisms during the inference stage to protect against unintentional copyright breaches.
News Articles
The results varied significantly across different domains. While the teams aimed to extract news articles, they encountered challenges and ultimately failed to retrieve copyrighted material. This indicates that while some content types may not be adequately protected, others still offer resistive layers against extraction, a finding that deserves further investigation.
More Read
Scientific Publications
Similarly, when investigating scientific publications, the teams were only able to produce high-level summaries. This outcome suggests a potential gap in the model’s ability to engage with detailed academic content while ensuring proper adherence to copyright laws. It highlights a need for more robust guidelines on how generative AI should handle scholarly articles.
Clinical Notes
The assessment of access-restricted clinical notes revealed promising results regarding patient privacy and data security. The AI tool demonstrated appropriate safeguards by reformatting data instead of reproducing it verbatim. This is a critical success as it aligns with confidentiality requirements prevalent in healthcare, ensuring that sensitive information is well-protected.
Implementing Mitigation Strategies
The results of the red teaming exercise were enlightening and led to immediate changes in how GPT4DFCI operates. A noteworthy response was the implementation of a copyright-specific meta-prompt, aimed at reducing the chances of unauthorized content generation. This meta-prompt has been operational since January 2025, responding to the identified vulnerabilities and fostering a more compliant use of generative AI in academic settings.
Continuous Testing and Compliance
The findings from this detailed exercise emphasize the importance of ongoing compliance testing within academic medical institutions deploying generative AI. Continuous red teaming can illuminate potential vulnerabilities in real-time, allowing organizations to address compliance challenges proactively. The experience at Dana-Farber serves as a blueprint for similar institutions navigating the complex landscape of generative AI technology and copyright law.
Ultimately, the intersection of AI and healthcare is fraught with challenges, but with structured assessments like red teaming, institutions can confidently embrace innovation while ensuring adherence to ethical and legal standards.
Inspired by: Source
