91��Ƭ��

John Borek is a faculty lecturer in the National Security Intelligence Analysis (NSIA) master’s program, and Seth Marcotte is a recent NSIA graduate. Together, they were published in the International Journal of Intelligence and CounterIntelligence. 

John Borek

Supported by the Office of the Director of National Intelligence (ODNI), “An Evaluation of Large Language Models Using AnalyticTradecraft Standards,” by John Borek and Seth Marcotte reflects the strength of the National Security Intelligence Analysis program at the 91��Ƭ��. It also demonstrates the sustained connection between faculty, students, and graduates. 

, the authors evaluate the output of Google’s Gemini and Microsoft’s Copilot against ODNI analytic tradecraft standards the ODNI Rating Scale for Evaluating Analytic Tradecraft Standards, and compare those results to human analysts. 

Their evaluation found large language models struggled with tasks requiring disaggregation of information, particularly describing source quality and credibility, expressing uncertainties, and distinguishing between facts, assumptions, and judgments. The models achieved highest ratings on communication-related tasks including analysis of alternatives, relevance and implications, and clear and logical argumentation. 

Learn more about the NSIA program, its faculty, and research: 

Explore Our M.S. National Security Intelligence Analysis Program