Resources

Tech Policy Resources

Last updated: December 4, 2024

• Newsletters
  • Tech policy in general
  • AI focus
• Podcasts
• Research Center
• Conferences
  • AI-related conferences
  • AI ethics
  • Trust and safety
• Events
• Selected AI reading recommendations
  • Understanding sources of bias
  • Thinking about risk
  • Positive use cases
  • Approaches to alignment
  • Evaluation challenges
  • Benchmarks and evaluations
  • Persuasion
  • Deceptive campaigns and misinformation
  • Language and its impact
  • AI incident trackers
  • Model transparency
  • Perceptions of generative AI
  • AI ethics classics
  • AI policy overviews
• AI-safety related organizations
• Stanford groups
• Job/internship opportunities

Newsletters

Tech policy in general

• Tech Policy Watch newsletter
• Stanford Cyber Policy Center newsletter
• Harvard Berkman Klein Center Newsletter

AI focus

• Import AI
• AI Safety Newsletter
• Georgetown CSET newsletter
• ChinAI

Podcasts

• Hard Fork
• The AI Policy Podcast
• POLITICO Tech
• Arbiters of Truth
• Moderated Content
• Google DeepMind: The Podcast
• Microsoft Research Podcast

Research Centers

• Stanford HAI
  • Center for Research on Foundation Models (CFRM)
  • RegLab
• Stanford McCoy Family Center for Ethics in Society
• Stanford Cyber Policy Center (has multiple relevant centers like the Stanford Internet Observatory or Governance of Emerging Technologies and hosts many events and a lunch speaker series)
• Georgetown Center for Security and Emerging Technology (CSET)

Conferences

AI-related conferences

• International Conference on Learning Representations (ICLR)
• Conference on Empirical Methods in Natural Language Processing (EMNLP)
• Knowledge Discovery and Data Mining (KDD)
• The Web Conference (The Web Conference (WWW))
• IEEE / CVF Computer Vision and Pattern Recognition Conference (CVPR)
• Conference and Workshop on Neural Information Processing Systems (NeurIPS)
• International Conference on Machine Learning (ICML)
• Conference on Computer and Communications Security (CCS)
• IEEE Symposium on Security and Privacy (IEEE S&P)
• Conference on Computer-Supported Cooperative Work and Social Computing (CSCW)
• Conference on Computing and Sustainable Societies (COMPASS)
• Annual Meeting of the Association for Computational Linguistics (ACL)
• CHI conference on Human Factors in Computing Systems (CHI)
• International Conference for Computational Social Science (IC2S2)
• Conference on Designing Interactive Systems (DIS)

AI ethics

• ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT)
• EAAMO (Equity and Access in Algorithms, Mechanisms, and Optimization)

Trust and safety

• Trust and Safety Research Conference
• TrustCon

Events

• GenAI Events in the Bay Area (this links to an amazing, almost overwhelming spreadsheet)

Selected AI reading recommendations

Understanding sources of bias

• Suresh, H., & Guttag, J. V. (2021). A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle. In EAAMO 2021: Equity and Access in Algorithms, Mechanisms, and Optimization.
• Ferrara, E. (2023). Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models.

Risk overviews and taxonomies

• Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.-S., … Gabriel, I. (2021). Ethical and social risks of harm from Language Models.
• Marchal, N., Xu, R., Elasmar, R., Gabriel, I., Goldberg, B., & Isaac, W. (2024). Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data.
• Gabriel, I., Manzini, A., Keeling, G., Hendricks, L. A., Rieser, V., Iqbal, H., … Research, G. (2024). The Ethics of Advanced AI Assistants.

Thinking about risk

• Kapoor, S., Bommasani, R., Klyman, K., Longpre, S., Ramaswami, A., Cihon, P., … Engler, A. (2024). On the Societal Impact of Open Foundation Models.
  • Presents helpful framework to analyze risks, and highlights the need to account for context and assess marginal risk
  • Summary:
    • Risks of LLMs should be evaluated in terms of their marginal risks over existing technologies
    • Framework to analyze risks: 1) identify threat 2) evaluate existing risks (without open foundation models) 3) evaluating existing defenses (without open foundation models) 4) assess evidence of marginal risks through open foundation models 5) assess potential for open foundation models in assisting defense against risks 6) describe assumptions and uncertainties of assessment
    • Open foundation models have several benefits, including giving users the power to shape acceptable model behavior, increasing innovation, accelerating science, enabling transparency and mitigating market concentration.
• Narayanan, A., & Kapoor, S. (2024). AI safety is not a model property.
  • Highlights importance of considering context
• Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2023). Auditing Large Language Models: A Three-Layered Approach. AI and Ethics. Springer International Publishing.
• Yang, E., & Roberts, M. E. (2023). The Authoritarian Data Problem. Journal of Democracy, 34(4), 141–150.

Positive use cases

• Argyle, L. P., Busby, E., Gubler, J., Bail, C., Howe, T., Rytting, C., & Wingate, D. (2023). AI Chat Assistants can Improve Conversations about Divisive Topics.
• Costello, T. H., Pennycook, G., & Rand, D. G. (2024). Durably reducing conspiracy beliefs through dialogues with AI. Science, 385(6714), eadq1814.
• Tessler, M. H., Bakker, M. A., Jarrett, D., Sheahan, H., Chadwick, M. J., Koster, R., … Summerfield, C. (2024). AI can help humans find common ground in democratic deliberation. Science, 386(6719).

Approaches to alignment

• Anthropic. (2023). Collective Constitutional AI: Aligning a Language Model with Public Input.
• Perrigo, B. (2024, February). Inside OpenAI’s Plan to Make AI More ‘Democratic.’ TIME.
• Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., … Kaplan, J. (2022). Constitutional AI: Harmlessness from AI Feedback.
• Ganguli, D., Askell, A., Schiefer, N., Liao, T. I., Lukošiūtė, K., Chen, A., … Kaplan, J. (2023). The Capacity for Moral Self-Correction in Large Language Models.
• Robinson, D. G. (2022). The Kidney Transplant Algorithm’s Surprising Lessons for Ethical A.I.
  • Summary:
    • Kidney organ donation algorithm as an example of how AI can be governed more democratically Recognizing the moral decisions behind technical ones
    • Broad public input, not only expert decision-making
    • Transparent decision-making
    • External auditing
    • Ability to forecast what changes in the system would mean

Evaluation challenges

• Anthropic. (2023). Challenges in evaluating AI systems.
• Friedler, S., Singh, R., Blili-Hamelin, B., Metcalf, J., & Chen, B. J. (2023). AI Red-Teaming Is Not a One-Stop Solution to AI Harms: Recommendations for Using Red-Teaming for AI Accountability.

Benchmarks and evaluations

• Bommasani, R., Liang, P., & Lee, T. (2022). Holistic Evaluation of Language Models.
• Durmus, E., Nyugen, K., Liao, T. I., Schiefer, N., Askell, A., Bakhtin, A., … Ganguli, D. (2023). Towards Measuring the Representation of Subjective Global Opinions in Language Models.
• Röttger, P., Hofmann, V., Pyatkin, V., Hinck, M., Kirk, H. R., Schütze, H., & Hovy, D. (2024). Political Compass or Spinning Arrow? Towards More Meaningful Evaluations for Values and Opinions in Large Language Models.
  • Interesting critique of standard survey-based model evaluations, evidence of political bias
• Hofmann, V., Kalluri, P. R., Jurafsky, D., & King, S. (2024). AI generates covertly racist decisions about people based on their dialect. Nature, 633(February).

Persuasion

• Anthropic. (2024). Measuring the Persuasiveness of Language Models.
• Bai, H., Voelkel, J. G., Eichstaedt, J. C., & Willer, R. (2023). Artificial Intelligence Can Persuade Humans on Political Issues.
• Matz, S. C., Teeny, J. D., Vaid, S. S., Harari, G. M., & Cerf, M. (2024). The potential of generative AI for personalized persuasion at scale. Nature Scientific Reports, 14(4692), 1–16.
• Kobi Hackenburg, & Margetts, H. (2024). Evaluating the persuasive influence of political microtargeting with large language models. Proceedings of the National Academy of Sciences, 120.
• Salvi, F., Ribeiro, M. H., Gallotti, R., West, R., & Mar, C. Y. (2024). On the Conversational Persuasiveness of Large Language Models: A Randomized Controlled Trial.
• Jakesch, M., Bhat, A., Buschek, D., Zalmanson, L., & Naaman, M. (2023). Co-Writing with Opinionated Language Models Affects Users’ Views. In Conference on Human Factors in Computing Systems.
• Williams-Ceci, S., Jakesch, M., Bhat, A., Kadoma, K., Zalmanson, L., & Naaman, M. (2024). Bias in AI Autocomplete Suggestions Leads to Attitude Shift on Societal Issues.
• Fisher, J., Feng, S., Aron, R., Richardson, T., Choi, Y., Fisher, D. W., … Reinecke, K. (2024). Biased AI can Influence Political Decision-Making.

Deceptive campaigns and misinformation

• Goldstein, J. A., Sastry, G., Musser, M., DiResta, R., Gentzel, M., & Sedova, K. (2023). Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations.
• Goldstein, J. A., Chao, J., & Grossman, S. (2024). How persuasive is AI-generated propaganda? PNAS Nexus, 3(2), 1–7.
• Marcellino, W., Beauchamp-Mustafaga, N., Kerrigan, A., Navarre Chao, L., & Smith, J. (2023). The Rise of Generative AI and the Coming Era of Social Media Manipulation 3.0: Next-Generation Chinese Astroturfing and Coping with Ubiquitous AI.
• Microsoft. (2024). Staying ahead of threat actors in the age of AI.
• Harbarth, K. (2024). Guide to the 2024 Elections.
  • The risk of GenAI for elections may be overestimated. The real risk may be the narrative that GenAI could produce harm
• Dufour, N., Pathak, A., Samangouei, P., Hariri, N., Deshetti, S., Dudfield, A., … Bregler, C. (2024). AMMeBa: A Large-Scale Survey and Dataset of Media-Based Misinformation In-The-Wild

Language and its impact

• Nicholas, G., & Bhatia, A. (2023). Lost in Translation: Large Language Models in Non-English Content Analysis.

AI incident trackers

• OECD AI Incidents Monitor (AIM)

Model transparency

• Bommasani, R., Klyman, K., Kapoor, S., Longpre, S., Xiong, B., Maslej, N., & Liang, P. (2024). The Foundation Model Transparency Index v1.1.
• Bommasani, R., Klyman, K., Longpre, S., Kapoor, S., Maslej, N., Xiong, B., … Liang, P. (2023). The Foundation Model Transparency Index.

Perceptions of generative AI

• Jakesch, M., Hancock, J. T., & Naaman, M. (2023). Human heuristics for AI-generated language are flawed. Proceedings of the National Academy of Sciences, 120(11).
• Begum Celiktutana, Romain Cadarioa, & Morewedge, C. K. (2017). People see more of their biases in algorithms. Proceedings of the National Academy of Sciences, 120.
• Altay, S., & Gilardi, F. (2024). People are skeptical of headlines labeled as AI-generated, even if true or human-made, because they assume full AI automation. PNAS Nexus. 3, 1-11.
• Ejaz, W., Fletcher, R., Nielsen, R. K., & McGregor, S. C. (2024). What Do People Want? Views on Platforms and the Digital Public Sphere in Eight Countries. Reuters Institute Report, 1-52.

AI ethics classics

• Bender, E. M., Gebru, T., Mcmillan-Major, A., & Shmitchell, S. (2021). On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? Conference on Fairness, Accountability, and Transparency (FAccT ‘21), March 310, 2021, Virtual Event, Canada (Vol. 1). Association for Computing Machinery.

AI policy overviews

• G’sell (2024). Regulating under Uncertainty: Governance Options for Generative AI
  • Very comprehensive overview of current AI governance frameworks
• Anthropic’s Responsible Scaling Policy
• DeepMind’s Frontier Safety Framework
• OpenAI’s Preparedness Framework

AI-safety related organizations

• AI Safety Institutes and Offices
  • US AI Safety Institute
  • UK AI Safety Institute
  • EU AI Office
• The Frontier Model Forum is a group of leading AI companies that exchanges on AI safety
• The Collective Intelligence Project partners with AI companies on gathering democratic input for AI
• Mozilla’s AI Intersections Database lists a range of organizations that work on the societal impact of AI
• Emerging Tech Policy Careers lists valuable advice and career resources for those interested in public service careers focused on emerging tech policy
• METR provides model evaluation and threat research
• Distributed AI Research Institute (DAIR) founded by Timnit Gebru
• There are also a range of AI consultancies, including Malo Santo, Humane Intelligence, Ethical Intelligence and Avanade

Mailing lists

• AI Safety Google Group
• Stanford University
  • Stanford AI Club for students interested in AI safety
  • Stanford Wonks & Techies mailing list for students and faculty interested in tech policy

Job/internship opportunities

• Trust and Safety Professional Association Job Board
• 80,000 hours job board (filtered for AI safety & policy)
• Stanford Public Interest Technology Jobs Newsletter
• Tech Congress Newsletter
• Stanford Fellowships, including
  • Tech Ethics & Policy Rising Scholars Program
  • Stanford Global Policy Internships
  • Public Policy Summer Internship Fellowship
  • Stanford in Government Fellowships and Stipends
  • Global Studies Internship Program
  • Tech Ethics & Policy Summer Fellowships