Field Guide · Modern War Tech 101

What Human-Centered AI Decision Support Means

AI that helps warfighters decide faster and with more confidence — while keeping the human in the loop for consequential actions. Not autonomous lethal decision-making.

01 · Plain-English explanation

Plain-English explanation

Human-centered AI decision support means AI systems designed to help human decision-makers act faster and with greater confidence — by processing large amounts of data, identifying patterns, generating options, and presenting information humans can act on — while keeping the human explicitly in the decision loop for consequential actions. This is distinct from autonomous decision-making, where an AI makes and executes consequential decisions without human review.

In a military context, that means fusing intelligence feeds and flagging anomalies, generating course-of-action options with risk assessments, summarizing battlefield state faster than staff can read raw reports, and clearly distinguishing high-confidence from low-confidence information — communicating uncertainty, data provenance, and confidence levels. Provenance, confidence, and uncertainty are not optional features — they are mission-critical.

Human judgment remains central. AI should expose confidence, provenance, uncertainty, and alternatives — not hide them. A decision-support tool that obscures how sure it is, or where its information came from, is not trustworthy regardless of how capable it appears.

02 · Why it matters in Ukraine

Why it matters in Ukraine

Ukraine and Russia are both experimenting with AI-assisted targeting, primarily computer vision for automatic target recognition in drone guidance. In these systems, AI identifies and locks onto a target, but human operators typically authorize the attack before terminal guidance. This is human-in-the-loop AI, though the loop can be very short in a fast engagement.

03 · Why it matters to U.S. and allied warfighters

Why it matters to U.S. and allied warfighters

The Maven Smart System — the U.S. flagship AI platform for intelligence fusion — is deployed across all six military branches and is framed as “decision support, not decision-making.” That framing is central to its legal and ethical legitimacy. International humanitarian law requires meaningful human judgment in targeting decisions.

04 · Why it matters to industry and manufacturing

Why it matters to industry and manufacturing

Building trustworthy decision-support tools requires disciplined engineering: memory, provenance tracking, confidence scoring, and explicit uncertainty. Helicon houses this work under Helicon Labs so it is understood as a focused capability, not a claim that the whole organization is an AI company.

05 · Common misunderstandings

Common misunderstandings

“Military AI means autonomous lethal robots.” Current deployed AI in U.S. and allied forces is primarily decision support — analysis, intelligence fusion, logistics optimization.
“Human-in-the-loop means a human pushes a button for every action.” The legal requirement is for meaningful human judgment, not necessarily manual action on every engagement.
“AI can process battlefield information without bias.” AI reflects the biases of its training data and architecture; provenance and uncertainty flags exist to mitigate this.

06 · Related technologies and concepts

Related technologies and concepts

Decision support depends on all-domain awareness and sensor fusion. See that explainer for how the underlying data picture is built.

07 · Further reading and videos

How Helicon works in this area

Helicon Labs focuses on AI that helps warfighters make better decisions faster — with memory, provenance, confidence-scoring, and explicit uncertainty — never autonomous targeting or lethal decision-making.

Key sources, explained

Each card explains why a source matters, what it teaches, and the Helicon takeaway. Public-domain primary texts can be read in full on this page; everything else links out.

Reuters (via The Straits Times)

Ukraine Collects a Vast War-Data Trove to Train AI Models

Why this matters

It shows how battlefield data — not just hardware — has become a strategic asset in modern war.

What it teaches

That Ukraine has gathered an enormous archive of drone footage and engagement data, drawn from 15,000+ drone crews, to train automatic-target-recognition and other AI models.

Helicon takeaway

Data provenance, confidence, and uncertainty must be engineered in. AI here is decision support — it should expose its reasoning, not hide it.

Reuters reporting, republished via The Straits Times, describes how Ukraine has assembled an extraordinary archive of battlefield video through the Ochi system — a non-profit Ukrainian digital platform originally designed to give military commanders a unified view of their operating areas by displaying drone footage from all nearby crews side by side on a single screen. The archive has grown to over two million hours of battlefield video since 2022, contributed by more than 15,000 drone crews working on the front lines, with new material arriving at a rate of five to six terabytes per day. The raw scale — described as the equivalent of 228 years of continuous video — is itself a strategic fact.

The significance is conceptual as much as technical. Ukraine is now using this footage to train artificial-intelligence models, including systems for target recognition, combat-tactics analysis, and weapons-effectiveness assessment. According to Oleksandr Dmitriev, the footage gives AI systems battlefield experience that can be translated into mathematical models — allowing algorithms to learn trajectories, effective engagement angles, target shapes and colors under real combat conditions. Analysts cited in the reporting note that the size and quality of the dataset are critical because AI recognition models learn from patterns in real imagery, and synthetic or exercise data does not capture the visual complexity, adversary behavior, and friction of actual operations the way live combat footage does. The Avengers system maintained by Ukraine's Defense Ministry separately centralizes drone and CCTV feeds, and has been described as identifying roughly 12,000 Russian pieces of equipment per week using AI identification tools. Thousands of drones are already using AI-assisted navigation, and Ukrainian companies are developing drone swarms in which computer systems would execute coordinated commands across linked clouds of dozens of units.

The article reframes data itself as a strategic asset of modern war. The side that systematically captures, labels, and learns from real engagement footage builds an advantage that compounds over time, because each model improvement feeds back into better collection and faster iteration. This is a different kind of arms race — measured in datasets, feedback loops, and model generations rather than only in platforms and munitions counts.

That framing also raises hard governance questions that Helicon takes seriously. Combat data carries provenance, quality, and bias problems that are not always visible in model outputs; algorithms trained on one theater's conditions can fail in unfamiliar environments; and the line between human-supervised decision support and automated targeting must be drawn deliberately rather than left implicit. The Helicon position is that AI in this context should function as decision support — exposing confidence levels, provenance, uncertainty, and alternatives to a human decision-maker rather than concealing them — with meaningful human judgment retained over the use of force. This is a Helicon-written summary; the full Reuters reporting is available via The Straits Times.

Optional quick digest prepared by Helicon from the cited source. Open the original for the full text.

Open original

Reuters — January 2026

Ukraine to Share Wartime Combat Data With Allies to Help Train AI

Why this matters

It signals that allied AI development is increasingly built on real combat data shared between partners.

What it teaches

That Ukraine is moving to share wartime combat data with allied governments and industry to accelerate AI model training.

Helicon takeaway

Trusted handling of shared combat data — provenance, access control, and human judgment — is a precondition, not an afterthought.

Reuters reported in January 2026 that Ukraine intends to create a framework enabling allied governments and defense industry to train artificial-intelligence military models using the combat data Ukraine has accumulated during nearly four years of full-scale war. The announcement was made by Ukraine's newly appointed Defense Minister, who framed it as part of a broader data-driven overhaul of how Ukraine manages its military and engages its partners. The minister, who previously served as Ukraine's digitalization minister, described Ukraine's wartime data repository as one of its strategic advantages in negotiations with other nations — an asset to be shared deliberately rather than simply held.

The development signals a structural shift in how allied AI systems are built. Models trained on real, recent combat data — reflecting genuine adversary behavior, contested electromagnetic environments, and the friction of actual operations — are meaningfully different from models trained on synthetic datasets or exercise footage that may not capture the conditions of high-intensity war. Ukraine has amassed millions of hours of drone footage and systematically recorded combat statistics that represent a volume and quality of real-world engagement data that allied nations would take years to accumulate independently. Sharing that data across trusted-partner borders turns Ukraine's hard-won operational learning into a collective allied asset and compresses allied AI development timelines significantly.

The strategic value of this data also raises the stakes for governance and handling. Combat data of this type can expose sources, methods, sensor capabilities, and operational vulnerabilities if it is mishandled or if access is not tightly controlled. The Reuters reporting notes that Ukraine is receiving guidance from the Center for Strategic and International Studies in the United States and from the Royal United Services Institute in Britain, and that Ukraine is utilizing AI technology from U.S. data analytics firm Palantir. Fedorov's stated emphasis on systematic calculation, quantifiable outcomes, and high-quality data suggests an institutional orientation toward rigor, though the article does not describe formal data-security protocols for cross-border sharing in detail.

Helicon's view is that trusted handling — meaning documented provenance, controlled access, independent auditability, and retained human judgment at decision points — is a precondition for this kind of data partnership, not a condition to be satisfied retroactively. The value of Ukraine's data trove is real; so is the responsibility that comes with accessing and applying it. Moving that learning into U.S. and allied capability development requires transition pathways that meet the governance standards of the receiving institutions, not just the urgency of the supply side. We do not republish Reuters' text; this is a Helicon-written summary linking to the original reporting.

Optional quick digest prepared by Helicon from the cited source. Open the original for the full text.

Open original

U.S. Department of Defense — ai.mil

Chief Digital and Artificial Intelligence Office (CDAO)

Why this matters

It is the official home of how the U.S. military actually frames and fields AI.

What it teaches

That the DoD frames its flagship work around decision support — the Maven Smart System for sensor fusion, agentic battle management, and data integration — with the human kept in the loop.

Helicon takeaway

Helicon Labs builds toward decision support with memory, provenance, and explicit uncertainty — never autonomous lethal decision-making.

Visit ai.mil

Official SourceTechnical

U.S. Department of Defense

DoD Directive 3000.09 — Autonomy in Weapon Systems

Why this matters

It is the U.S. policy that requires appropriate human judgment over the use of force in autonomous and semi-autonomous weapons.

What it teaches

That the directive mandates rigorous verification and validation, test and evaluation, explainability, auditability, and well-designed human-machine interfaces — human judgment is required, not optional.

Helicon takeaway

Helicon Labs builds toward decision support with provenance and explicit uncertainty — human-in-the-loop, never autonomous lethal decision-making.

Read the directive (PDF)

Official SourceTechnical

U.S. National Institute of Standards and Technology (NIST)

AI Risk Management Framework (AI RMF)

Why this matters

It is the neutral, widely-adopted U.S. government framework for building trustworthy AI.

What it teaches

That trustworthy AI is organized around four functions — Govern, Map, Measure, and Manage — and characteristics including validity, reliability, safety, security, accountability, transparency, explainability, and fairness.

Helicon takeaway

Helicon treats provenance, uncertainty, and human oversight as design requirements — consistent with the AI RMF — not as features added later.

Open original

Cited sources

Every factual claim above traces to these sources, confirmed live as of the research date. Independently verify before operational use.

Arms Control Association — Beyond a Human 'In the Loop' (2024)Open original
CSIS — What is Maven Smart System, and What Does It Do? (Feb. 2026)Open original
ICRC — The ethical challenges of AI in military decision support systems (Sept. 2024)Open original

Full public-domain text, hosted on this site for reading. Source: U.S. Department of Defense.

Originating Component: Office of the Under Secretary of Defense for Policy

Effective: January 25, 2023

Releasability: Cleared for public release. Available on the Directives Division Website at https://www.esd.whs.mil/DD/.

Reissues and Cancels: DoD Directive 3000.09, "Autonomy in Weapon Systems," November 21, 2012

Approved by: Kathleen H. Hicks, Deputy Secretary of Defense

Purpose

This directive: - Establishes policy and assigns responsibilities for developing and using autonomous and semi-autonomous functions in weapon systems, including armed platforms that are remotely operated or operated by onboard personnel. - Establishes guidelines designed to minimize the probability and consequences of failures in autonomous and semi-autonomous weapon systems that could lead to unintended engagements. - Establishes the Autonomous Weapon Systems Working Group.

Section 1: General Issuance Information

#### 1.1. Applicability

a. This directive applies to: (1) OSD, the Military Departments, the Office of the Chairman of the Joint Chiefs of Staff (CJCS) and the Joint Staff, the Combatant Commands, the Office of Inspector General of the Department of Defense, the Defense Agencies, the DoD Field Activities, and all other organizational entities within the DoD. (2) The design, development, acquisition, testing, fielding, and employment of autonomous and semi-autonomous weapon systems, including guided munitions that are capable of automated target selection. (3) The application of lethal or non-lethal, kinetic or non-kinetic, force by autonomous or semi-autonomous weapon systems.

b. This directive does not apply to: (1) Autonomous or semi-autonomous cyberspace capabilities. (2) Unarmed platforms, whether remotely operated or operated by onboard personnel, and whether autonomous or semi-autonomous. (3) Unguided munitions. (4) Munitions manually guided by the operator (e.g., laser- or wire-guided munitions). (5) Mines. (6) Unexploded explosive ordnance. (7) Autonomous or semi-autonomous systems that are not weapon systems.

#### 1.2. Policy

a. Autonomous and semi-autonomous weapon systems will be designed to allow commanders and operators to exercise appropriate levels of human judgment over the use of force.

(1) Systems will go through rigorous hardware and software verification and validation (V&V) and realistic system developmental and operational test and evaluation (T&E) in accordance with Section 3. Training, doctrine, and tactics, techniques, and procedures (TTPs) applicable to the system in question will be established. These measures will provide sufficient confidence that autonomous and semi-autonomous weapon systems:

(a) Function as anticipated in realistic operational environments against adaptive adversaries taking realistic and practicable countermeasures. (b) Complete engagements within a timeframe and geographic area, as well as other relevant environmental and operational constraints, consistent with commander and operator intentions. If unable to do so, the systems will terminate the engagement or obtain additional operator input before continuing the engagement. (c) Are sufficiently robust to minimize the probability and consequences of failures.

(2) Consistent with the potential consequences of an unintended engagement or unauthorized parties interfering with the operation of the system, physical hardware and software will be designed with appropriate: (a) System safety, anti-tamper mechanisms, and cybersecurity in accordance with DoD Instruction (DoDI) 8500.01 and Military Standard 882E. (b) Human-machine interfaces and controls. (c) Technologies and data sources that are transparent to, auditable by, and explainable by relevant personnel.

(3) For operators to make informed and appropriate decisions regarding the engagement of targets, the human-machine interface for autonomous and semi-autonomous weapon systems will: (a) Be readily understandable to trained operators, such as by clearly indicating what actions operators need to perform and which actions the system will perform. (b) Provide transparent feedback on system status. (c) Provide clear procedures for trained operators to activate and deactivate system functions.

b. Persons who authorize the use of, direct the use of, or operate autonomous and semi-autonomous weapon systems will do so with appropriate care and in accordance with the law of war, applicable treaties, weapon system safety rules, and applicable rules of engagement (ROE). The use of AI capabilities in autonomous or semi-autonomous weapons systems will be consistent with the DoD AI Ethical Principles, as provided in Paragraph 1.2.f.

c. With the exception of systems intended to be used in a manner that falls within the policies in Paragraphs 1.2.d.(1) through 1.2.d.(4), autonomous weapon systems, including weapon systems with both autonomous and semi-autonomous modes of operation, must be approved by the Under Secretary of Defense for Policy (USD(P)), the Under Secretary of Defense for Research and Engineering (USD(R&E)), and the Vice Chairman of the Joint Chiefs of Staff (VCJCS) before formal development. They must be approved again by the USD(P), the Under Secretary of Defense for Acquisition and Sustainment (USD(A&S)), and the VCJCS before fielding. These requirements for approval are supplementary to the requirements in other applicable policies and issuances. Autonomous weapon systems requiring these senior approvals in accordance with Section 4 of this directive before formal development and again before fielding include:

(1) Autonomous weapon systems that have not previously been reviewed and approved in accordance with this directive, including autonomous weapon systems that are modifications of an existing non-autonomous weapon system. (2) Modified versions of previously approved autonomous weapon systems whose system algorithms, intended mission sets, intended operational environments, intended target sets, or expected adversarial countermeasures substantially differ from those applicable to the previously approved weapon systems so as to fall outside the scope of what was previously approved in the senior review. Such modified systems require a new senior review and approval before formal development and again before fielding.

d. The senior review described in Paragraph 1.2.c is not required for weapon systems intended to be used in the manner described in Paragraphs 1.2.d.(1) through 1.2.d.(4). These will be considered for approval in accordance with applicable policies and issuances, such as applicable issuances related to the Defense Acquisition System. Weapon systems that do not require the senior review provided in Paragraph 1.2.c are:

(1) Semi-autonomous weapon systems used to apply lethal or non-lethal, kinetic or non-kinetic, force without any modes of operation in which they are intended to function as an autonomous weapon system. (2) Operator-supervised autonomous weapon systems used to select and engage materiel targets for local defense to intercept attempted time-critical or saturation attacks for: (a) Static defense of installations with personnel, including networked defense where the autonomous weapon system is not co-located with the installation. (b) Onboard and/or networked defense of platforms with onboard personnel. (3) Operator-supervised autonomous weapon systems used to select and engage materiel targets for defending operationally deployed remotely piloted or autonomous vehicles and/or vessels. (4) Autonomous weapon systems used to apply non-lethal, non-kinetic force against materiel targets in accordance with DoDD 3000.03E.

e. International sales or transfers of autonomous and semi-autonomous weapon systems will be approved in accordance with existing technology security and foreign disclosure requirements and processes in accordance with DoDD 5111.21.

f. The design, development, deployment, and use of AI capabilities in autonomous and semi-autonomous weapon systems will be consistent with the DoD AI Ethical Principles and the DoD Responsible Artificial Intelligence Strategy and Implementation Pathway. The DoD AI Ethical Principles, as adopted in the February 21, 2020 Secretary of Defense Memorandum, are:

(1) Responsible. DoD personnel will exercise appropriate levels of judgment and care, while remaining responsible for the development, deployment, and use of AI capabilities. (2) Equitable. The DoD will take deliberate steps to minimize unintended bias in AI capabilities. (3) Traceable. The DoD's AI capabilities will be developed and deployed such that relevant personnel possess an appropriate understanding of the technology, development processes, and operational methods applicable to AI capabilities, including with transparent and auditable methodologies, data sources, and design procedures and documentation. (4) Reliable. The DoD's AI capabilities will have explicit, well-defined uses, and the safety, security, and effectiveness of such capabilities will be subject to testing and assurance within those defined uses across their entire life cycles. (5) Governable. The DoD will design and engineer AI capabilities to fulfill their intended functions while possessing the ability to detect and avoid unintended consequences, and the ability to disengage or deactivate deployed systems that demonstrate unintended behavior.

Section 2: Responsibilities

#### 2.1. USD(P)

The USD(P): a. Provides policy oversight for developing and employing autonomous and semi-autonomous weapon systems. b. Receives requests for approval of systems submitted in accordance with Paragraph 1.2.c, and in coordination with the USD(A&S) or USD(R&E) and the VCJCS, reviews and considers for approval such systems. c. Issues guidance to help implement this directive, and reviews, as necessary, the appropriateness of such guidance given the continual advancement of new technologies and changing warfighter needs. d. Approves the DoD position on international sales or transfers of autonomous and semi-autonomous weapon systems in accordance with existing technology security and foreign disclosure requirements and processes. e. Supervises and assigns a chair for the Autonomous Weapon Systems Working Group, provides necessary logistical and administrative support for the working group, approves the charter for the working group, and provides guidance and terms of reference as needed.

#### 2.2. USD(A&S)

The USD(A&S): a. In coordination with the USD(P) and the VCJCS, reviews and considers for approval weapon systems submitted before fielding in accordance with Paragraph 1.2.c. b. Ensures that DoD guidance relating to the Defense Acquisition System includes a requirement to document the determination that an autonomous or semi-autonomous weapon system is intended to be used in a manner that falls within the policies in Paragraphs 1.2.d.(1) through 1.2.d.(4), and therefore does not require senior approval in accordance with this directive. This documentation should occur before formal development and again before fielding, regardless of the acquisition pathway that is applicable to that weapon system.

#### 2.3. USD(R&E)

The USD(R&E): a. Oversees establishment of standards and evaluation metrics for developmental testing, safety certification, and reliability assessment of autonomous and semi-autonomous weapon systems, with particular attention to the risk of unintended engagements or operational interference by unauthorized parties. b. Oversees establishment of science and technology and research and development priorities for autonomy in weapon systems, including the development of new methods of V&V and T&E and the establishment of minimum thresholds of risk and reliability for the performance of autonomy in weapon systems. c. Oversees formulation of concrete, testable requirements for all non-AI elements of autonomous and semi-autonomous weapon systems. d. Collaborates with the Chief Digital and Artificial Intelligence Officer (CDAO) to formulate concrete, testable requirements for implementing the DoD AI Ethical Principles and the DoD Responsible AI Strategy and Implementation Pathway. e. Oversees and evaluates the developmental testing of autonomous and semi-autonomous weapon systems to assess the risk of failures. f. Develops and maintains workforce certification processes, talent management, and curricula to support T&E and V&V of autonomous and semi-autonomous weapon systems by DoD personnel. g. In coordination with the USD(P) and the VCJCS, reviews and considers for approval weapon systems submitted before entering formal development in accordance with Paragraph 1.2.c. h. Coordinates with the Director, Operational Test and Evaluation (DOT&E) and the appropriate Secretary of a Military Department or Commander, United States Special Operations Command (USSOCOM) to provide for monitoring to identify and address when changes to the system design or operational environment require additional T&E to provide sufficient confidence that the system will continue to avoid unintended engagements and resist interference by unauthorized parties.

#### 2.4. Under Secretary of Defense for Personnel and Readiness

In accordance with DoDD 1322.18, the Under Secretary of Defense for Personnel and Readiness oversees and establishes policy for: a. Individual military training programs for the Total Force relating to autonomous and semi-autonomous weapon systems. b. Individual and functional training programs for military personnel and the collective training programs of military units and staffs relating to autonomous and semi-autonomous weapon systems.

#### 2.5. DOT&E

The DOT&E: a. Oversees development of realistic operational T&E standards for autonomous and semi-autonomous weapon systems, including requirements for data collection and standards for T&E of any changes to the system following initial operational T&E (IOT&E), in accordance with Paragraph 1.2.a.(1) and Section 3. b. Evaluates whether autonomous and semi-autonomous weapon systems under DOT&E oversight have met standards for rigorous V&V and T&E in realistic operational conditions, including potential adversary action, to provide sufficient confidence that the probability and consequences of failures have been minimized. c. Establishes standards for data collection post-fielding and monitoring and assessment by programs. d. Coordinates with the USD(R&E) and the appropriate Secretary of a Military Department or Commander, USSOCOM to provide for monitoring to identify and address when changes to the system design or operational environment require additional T&E to provide sufficient confidence that the system will continue to avoid unintended engagements and resist interference by unauthorized parties. e. Reviews and approves operational and live fire test plans for autonomous and semi-autonomous weapon systems for Major Defense Acquisition Programs and programs designated for DOT&E oversight.

#### 2.6. General Counsel of the Department of Defense (GC DOD)

In accordance with DoDD 5000.01, DoDD 2311.01, DoDD 5145.01, and, where applicable, DoDD 3000.03E, the GC DoD provides for guidance on, and coordination of, significant legal issues in autonomy in weapon systems. The GC DoD also coordinates on the review of the legality of weapon systems submitted in accordance with Paragraph 1.2.c.

#### 2.7. Assistant to the Secretary of Defense for Public Affairs

The Assistant to the Secretary of Defense for Public Affairs coordinates on the development of guidance on public affairs matters concerning autonomous and semi-autonomous weapon systems and the use of such guidance and approves final guidance release.

#### 2.8. CDAO

The CDAO: a. Monitors and evaluates AI capabilities in and cybersecurity for autonomous and semi-autonomous weapon systems, in accordance with Paragraph 1.2.a.(2)(a) of this directive and DoDI 8500.01, and advises the Secretary of Defense on such matters. b. Collaborates with the USD(R&E) to formulate concrete, testable requirements for implementing the DoD AI Ethical Principles and the DoD Responsible AI Strategy and Implementation Pathway. c. Establishes policy and issues guidance on definitions of requirements and testability for AI-enabled systems to implement and demonstrate adherence to the DoD AI Ethical Principles and the DoD Responsible AI Strategy and Implementation Pathway. d. Issues guidance on T&E practices for AI capabilities in autonomous or semi-autonomous weapon systems. e. Coordinates with the USD(R&E) and DOT&E on developing and using common tools and infrastructure for T&E and V&V of AI capabilities in autonomous or semi-autonomous weapon systems.

#### 2.9. Secretaries of the Military Departments; Commander, USSOCOM; and Directors of the Defense Agencies and DoD Field Activities

The Secretaries of the Military Departments; the Commander, USSOCOM; and, under the authority, direction, and control of their respective OSD Component head, the Directors of Defense Agencies and DoD Field Activities:

a. Design and develop autonomous and semi-autonomous weapon systems that allow commanders and operators to exercise appropriate levels of human judgment over the use of force. This will include developing and implementing: (1) Employment concepts, doctrine, experimentation strategies, TTPs, training, and logistics support. (2) V&V, anti-tamper mechanisms, physical hardware, and software system safety in accordance with Military Standard 882E. (3) Cyber survivability, operational resilience, and cybersecurity in accordance with DoDI 8500.01. (4) Appropriate developmental and operational T&E, regardless of acquisition pathway, the joint/non-joint nature of those system's missions, or the lack of a survivability Key Performance Parameter for those systems.

b. For the systems in Paragraph 2.9.a: (1) Design autonomous and semi-autonomous weapon systems to minimize the probability and consequences of failures. (2) Perform rigorous and realistic developmental and operational T&E and V&V, including T&E of any changes to the system following IOT&E, in accordance with Paragraph 1.2.a.(1) and Section 3. (3) In coordination with the USD(R&E) and DOT&E, provide for monitoring to identify and address when changes to the system design or operational environment require additional T&E to provide sufficient confidence that the system will continue to avoid unintended engagements and resist interference by unauthorized parties. (4) For systems incorporating AI capabilities, design the system to utilize robust AI, in accordance with the DoD Responsible AI Strategy and Implementation Pathway, so that the system is resilient in real-world settings and against adversarial attacks and spoofing. (5) Design system safety, anti-tamper mechanisms, cyber survivability, operational resilience, and cybersecurity capabilities in accordance with Paragraph 1.2.a.(2) of this directive, DoDI 5000.83, the Joint Capabilities Integration and Development System Manual, and DoDI 8500.01. (6) Design human-machine interfaces to be readily understandable to trained operators, with clear procedures to activate and deactivate system functions, and to provide transparent feedback on system status in accordance with Paragraph 1.2.a.(3). (7) Certify that operators have been trained in system capabilities, doctrine, and TTPs to exercise appropriate levels of human judgment over the use of force and employ systems with appropriate care in accordance with the law of war, applicable treaties, weapon system safety rules, and ROE that are applicable or reasonably expected to be applicable. (8) Establish and periodically review training, TTPs, and doctrine to ensure operators and commanders understand the functioning, capabilities, and limitations of a system's autonomy under realistic operational conditions, including as a result of possible adversary actions.

c. Ensure that legal reviews of the intended acquisition, procurement, or modification of autonomous and semi-autonomous weapon systems are conducted in accordance with DoDD 5000.01, DoDD 2311.01, and, where applicable, DoDD 3000.03E. Legal reviews must address consistency with all applicable domestic and international law and, in particular, the law of war.

d. Consider for support only those autonomous and semi-autonomous weapon systems that are technically feasible, consistent with applicable law, and consistent with the standards in this directive.

e. In accordance with Paragraphs 1.2.c and 1.2.d, submit any autonomous weapon system for which approval is required to the USD(P), USD(A&S) or USD(R&E), and the VCJCS before a decision to enter formal development, and again before fielding of any such system.

#### 2.10. CJCS

The CJCS: a. Develops and implements joint employment concepts, doctrine, experimentation strategies, TTPs, training, and logistics support for autonomous and semi-autonomous weapon systems. b. Assesses military requirements for autonomous and semi-autonomous weapon systems, including applicable Key Performance Parameters and key system attributes. c. Develops and publishes joint doctrine, policy, and other guidance as appropriate to incorporate emerging capabilities of autonomous and semi-autonomous weapon systems into joint and combined operations, in accordance with this directive.

#### 2.11. VCJCS

In coordination with the USD(P) and USD(A&S) or USD(R&E), the VCJCS reviews and considers for approval autonomous weapon systems submitted in accordance with Paragraph 1.2.c.

#### 2.12. Combatant Commanders

The Combatant Commanders: a. Use autonomous and semi-autonomous weapon systems in accordance with this directive and in a manner consistent with their design, testing, certification, operator training, doctrine, TTPs, and approval as autonomous or semi-autonomous weapon systems. b. Employ autonomous and semi-autonomous weapon systems with appropriate care and in accordance with the law of war, applicable treaties, weapon system safety rules, and applicable ROE, in accordance with Paragraph 1.2.b, and employ AI capabilities in autonomous and semi-autonomous weapon systems consistent with the DoD AI Ethical Principles and the DoD Responsible Artificial Intelligence Strategy and Implementation Pathway, in accordance with Paragraph 1.2.f. c. Ensure that autonomous and semi-autonomous weapon systems are not employed or modified to operate in a manner that falls outside the policies in Paragraphs 1.2.d.(1) through 1.2.d.(4) without specific approval in accordance with Paragraph 1.2.c. d. Integrate autonomous and semi-autonomous weapon systems into operational mission planning as appropriate. e. Through the CJCS, identify warfighter priorities and operational needs that may be met by autonomous and semi-autonomous weapon systems.

Section 3: Verification and Validation and Testing and Evaluation of Autonomous and Semi-Autonomous Weapon Systems

Regardless of the acquisition pathway or OSD T&E oversight status for a given weapon system, to ensure autonomous and semi-autonomous weapon systems function as anticipated in realistic operational environments against adaptive adversaries and are sufficiently robust to minimize failures:

a. Systems will go through rigorous hardware and software V&V and realistic system developmental and operational T&E, including analysis of unanticipated emergent behavior. (1) Hardware and software V&V will include iterative cyber T&E in accordance with DoDI 5000.89, to verify that the weapon system is resilient and survivable in contested cyberspace. (2) Systems incorporating AI capabilities will go through rigorous developmental and operational T&E to verify and validate that the AI is robust according to design requirements.

b. T&E of systems incorporating AI capabilities will include testing to confirm that their autonomy algorithms can be rapidly reprogrammed on new input data.

c. After IOT&E, as directed by the DOT&E, system data will be collected and any further changes to the system will undergo appropriate V&V and T&E to ensure that critical safety features have not been degraded. (1) System software will be tested using best-available DoD means and methods to validate that critical safety features have not been degraded. Automated testing tools, such as modeling and simulation, will be used whenever feasible. The testing will identify any new operating states and other relevant changes in the autonomous or semi-autonomous weapon system. (2) As directed by the DOT&E: (a) Each new or revised operating state will undergo appropriate and tailored additional T&E to characterize the system behavior in that new operating state. (b) Changes to the state transition matrix may require whole system follow-on operational T&E.

d. In coordination with the USD(R&E) and DOT&E, the owning Component will provide for monitoring to identify and address when changes to the system design or operational environment require additional T&E to provide sufficient confidence that the system will continue to avoid unintended engagements and resist interference by unauthorized parties.

Section 4: Guidelines for Review of Certain Autonomous Weapon Systems

4.1. Autonomous weapon systems intended to be used in a manner that falls outside the policies in Paragraphs 1.2.d.(1) through 1.2.d.(4) must be approved by the USD(P), USD(R&E), and VCJCS before formal development and by the USD(P), USD(A&S), and VCJCS before fielding. If the weapon system in question is to be developed and then fielded by DoD, it will need to undergo both reviews and receive approvals. A review is not needed if the weapon system is covered by a previous approval for formal development or fielding. Requests for senior review and approval should be submitted to USD(P), attention to the Director of the Emerging Capabilities Policy Office.

a. An autonomous weapon system that is a variant of an existing weapon system previously approved through this review will not be covered by previous approval if changes to the system algorithms, intended mission set, intended operational environments, intended target sets, or expected adversarial countermeasures substantially differ from those applicable to the previously approved weapon system so as to fall outside the scope of what was previously approved in the senior review. Such systems will require a new senior review before their formal development and again before fielding.

b. An autonomous weapon system that is a modification of an existing weapon system not previously approved through this review requires the senior review described in Paragraph 1.2.c unless it is intended to be used in a manner that falls within the policies in Paragraphs 1.2.d.(1) through 1.2.d.(4).

c. Before a decision to enter formal development, the USD(P), USD(R&E), and VCJCS will verify that: (1) The system design incorporates the necessary capabilities to allow commanders and operators to exercise appropriate levels of human judgment over the use of force in the envisioned planning and employment processes for the weapon. (2) The system is designed to complete engagements within a timeframe and geographic area, as well as other applicable environmental and operational parameters, consistent with commander and operator intentions. If unable to do so, the system will terminate engagements or obtain additional operator input before continuing the engagement. (3) The combination of the system's design and concept of employment (e.g., its target selection and engagement logic and other relevant processes or measures) accounts for risks to non-targets, consistent with commander and operator intent. (4) The system design, including system safety, anti-tamper mechanisms, and cybersecurity in accordance with DoDI 8500.01, addresses and minimizes the probability and consequences of failures. (5) Plans are in place for V&V and T&E to establish system reliability, effectiveness, and suitability under realistic conditions, including possible adversary actions, to a sufficient standard consistent with the potential consequences of an unintended engagement or unauthorized parties interfering with the operation of the system. (6) For systems incorporating AI capabilities, plans are in place to ensure consistency with the DoD AI Ethical Principles and the DoD Responsible AI Strategy and Implementation Pathway. (7) A preliminary legal review of the weapon system has been completed in coordination with the GC DoD and in accordance with DoDD 5000.01, DoDD 2311.01 and, where applicable, DoDD 3000.03E.

d. Before fielding, the USD(P), USD(A&S), and VCJCS will verify that: (1) System capabilities, human-machine interfaces, doctrine, TTPs, and training have been demonstrated to allow commanders and operators to exercise appropriate levels of human judgment over the use of force and to employ systems with appropriate care and in accordance with the law of war, applicable treaties, weapon system safety rules, and ROE that are applicable or reasonably expected to be applicable. (2) System safety, anti-tamper mechanisms, cyber survivability, operational resilience, and cybersecurity capabilities have been implemented in accordance with DoDI 5000.83, the Joint Capabilities Integration and Development System Manual, and DoDI 8500.01 to minimize the probability and consequences of failures. A monitoring regime is in place to identify and address changes in operational environment, data inputs, and use that could contribute to such failures. (3) V&V and T&E: (a) Assess system performance, capability, reliability, effectiveness, and suitability under realistic conditions, including possible adversary actions, consistent with the potential consequences of unintended engagement or unauthorized parties interfering with the operation of the system. (b) Have demonstrated that the system can be reprogrammed with sufficient rapidity to enable timely correction of any unintended system behaviors that may be observed or discovered during future system operations. (4) Adequate training, TTPs, and doctrine are available, periodically reviewed, and used by system operators and commanders to understand the functioning, capabilities, and limitations of the system's autonomy in realistic operational conditions. (5) System design and human-machine interfaces are readily understandable to trained operators, provide transparent feedback on system status, and provide clear procedures for trained operators to activate and deactivate system functions. (6) For systems incorporating AI capabilities, the deployment and use of the AI capabilities in the weapon system will be consistent with the DoD AI Ethical Principles and the DoD Responsible AI Strategy and Implementation Pathway. (7) A legal review of the weapon system has been completed, in coordination with the GC DoD, and in accordance with DoDD 5000.01, DoDD 2311.01, and, where applicable, DoDD 3000.03E.

4.2. In cases of urgent military need, the USD(P), USD(A&S), USD(R&E), or VCJCS may request a Deputy Secretary of Defense waiver of the requirements in this section and Paragraph 1.2.c.

Section 5: Autonomous Weapon System Working Group

#### 5.1. General

The Autonomous Weapon System Working Group will: a. Support the USD(P), the USD(R&E), and the VCJCS in considering the full range of relevant DoD interests during the review of autonomous weapon systems before formal development. b. Support the USD(P), the USD(A&S), and the VCJCS in considering the full range of relevant DoD interests during the review of autonomous weapon systems before fielding. c. When requested by appropriate representatives of the Secretaries of the Military Departments; the Commander, USSOCOM; or, when applicable, a Director of a Defense Agency or a DoD Field Activity: (1) Advise whether a given weapon system requires senior-level approval in accordance with this directive. (2) Help identify and advise on addressing potential issues presented by a given weapon system during a potential senior-level review in accordance with this directive.

#### 5.2. Membership

In addition to representatives of the USD(P), the Autonomous Weapon System Working Group will consist of representatives of each of the following officials listed below. All members of the working group will be full time Federal Government employees, permanent part-time Federal Government employees, or Service members on active duty. The parent organizations for the representatives will be responsible for any expenses, to include travel related expenses, associated with participation in the working group: a. USD(A&S). b. USD(R&E). c. GC DoD. d. CDAO. e. DOT&E. f. CJCS representatives from: (1) Director for Strategy, Plans and Policy (Joint Staff J5). (2) Director, Command, Control, Communications and Computers/Cyber, Chief Information Officer (Joint Staff J6). (3) Director for Force Structure, Resources and Assessment (Joint Staff J8). (4) Legal Counsel to the Chairman of the Joint Chiefs of Staff.

Glossary

#### G.1. Acronyms

AI — artificial intelligence CJCS — Chairman of the Joint Chiefs of Staff CDAO — Chief Digital and Artificial Intelligence Officer DoDD — DoD directive DoDI — DoD instruction DOT&E — Director of Operational Test and Evaluation GC DoD — General Counsel of the Department of Defense IOT&E — initial operational test and evaluation ROE — rules of engagement T&E — test and evaluation TTPs — tactics, techniques, and procedures USD(A&S) — Under Secretary of Defense for Acquisition and Sustainment USD(P) — Under Secretary of Defense for Policy USD(R&E) — Under Secretary of Defense for Research and Engineering USSOCOM — United States Special Operations Command VCJCS — Vice Chairman of the Joint Chiefs of Staff V&V — verification and validation

#### G.2. Definitions

autonomous weapon system — A weapon system that, once activated, can select and engage targets without further intervention by an operator. This includes, but is not limited to, operator-supervised autonomous weapon systems that are designed to allow operators to override operation of the weapon system, but can select and engage targets without further operator input after activation.

failure — An actual or perceived degradation or loss of intended functionality or inability of the system to perform as intended or designed. Failure can result from a number of causes, including, but not limited to, human error, faulty human-machine interaction, malfunctions, communications degradation, software coding errors, enemy cyber-attacks or infiltration into the industrial supply chain, jamming, spoofing, decoys, other enemy countermeasures or actions, or unanticipated situations on the battlefield. For the purposes of this issuance, minimizing the probability and consequences of failure means reducing the probability and consequences of unintended engagements to acceptable levels while meeting mission objectives and does not mean achieving the lowest possible level of risk by never engaging targets.

fielding — Making a weapon system available for, or placing it into, operational use (rather than testing, exercises, or experiments), regardless of the acquisition approach employed for the weapon system, including major defense acquisition programs, middle tier acquisitions, or prototyping efforts such as joint concept technology demonstrations.

formal development — Begins at "Milestone B," as described in Paragraph 3.10 of DoDI 5000.85, in the case of major defense acquisition programs. For cases other than major defense acquisition programs, begins after the preliminary design review that correlates with the end of the technology maturation and risk reduction phase.

materiel — Defined in the DoD Dictionary of Military and Associated Terms.

operator-supervised autonomous weapon system — An autonomous weapon system that is designed to provide operators with the ability to intervene and terminate engagements, including in the event of a weapon system failure, before unacceptable levels of damage occur.

operating state — A variable or vector reflecting the status of the system.

operator — A person who operates a platform or weapon system.

remotely operated platform — An air, land, surface, subsurface, or space platform that is actively controlled by an operator who is not physically on the platform.

robust AI — Defined in the DoD Responsible Artificial Intelligence Strategy and Implementation Pathway.

semi-autonomous weapon system — A weapon system that, once activated, is intended to only engage individual targets or specific target groups that have been selected by an operator. This includes: Weapon systems that employ autonomy for engagement-related functions including, but not limited to, acquiring, tracking, and identifying potential targets; cuing potential targets to operators; prioritizing selected targets; timing of when to fire; or providing terminal guidance to home in on selected targets, provided that operator control is retained over the decision to select individual targets and specific target groups for engagement. "Fire and forget" or lock-on-after-launch homing munitions that rely on TTPs to maximize the probability that the only targets within the seeker's acquisition basket when the seeker activates are those individual targets or specific target groups that have been selected by an operator.

specific target group — A discrete group of potential targets, such as a particular flight of enemy aircraft, a particular formation of enemy tanks, or a particular flotilla of enemy vessels. A general class of targets or a specific type of target, such as a particular model of tank or aircraft, does not constitute a specific target group.

state transition matrix — A matrix that characterizes the ability of a system to transition from one operating state to another.

target selection — The identification of an individual target or a specific group of targets for engagement.

unintended engagement — The use of force against persons or objects that commanders or operators did not intend to be the targets of U.S. military operations, including unacceptable levels of collateral damage beyond those consistent with the law of war, ROE, and commander's intent.

weapon system — Defined in the DoD Dictionary of Military and Associated Terms.

weapon system safety rules — Guidance for personnel, issued by competent authority, focused on addressing weapon safety issues and concerns that present significant mishap risk and is applied with a view towards ensuring freedom from conditions that can cause occupational illness, unintended death or injury, unintended damage to or loss of equipment or property, or unintended damage to the environment.

Full public-domain text, hosted on this site for reading. Source: U.S. National Institute of Standards and Technology (NIST).

Executive Summary

Artificial intelligence (AI) technologies have significant potential to transform society and people’s lives – from commerce and health to transportation and cybersecurity to the environment and our planet. AI technologies can drive inclusive economic growth and support scientific advancements that improve the conditions of our world. AI technologies, however, also pose risks that can negatively impact individuals, groups, organizations, communities, society, the environment, and the planet. Like risks for other types of technology, AI risks can emerge in a variety of ways and can be characterized as long- or short-term, high- or low-probability, systemic or localized, and high- or low-impact.

The AI RMF refers to an AI system as an engineered or machine-based system that can, for a given set of objectives, generate outputs such as predictions, recommendations, or decisions influencing real or virtual environments. AI systems are designed to operate with varying levels of autonomy (Adapted from: OECD Recommendation on AI:2019; ISO/IEC 22989:2022).

While there are myriad standards and best practices to help organizations mitigate the risks of traditional software or information-based systems, the risks posed by AI systems are in many ways unique (See Appendix B). AI systems, for example, may be trained on data that can change over time, sometimes significantly and unexpectedly, affecting system functionality and trustworthiness in ways that are hard to understand. AI systems and the contexts in which they are deployed are frequently complex, making it difficult to detect and respond to failures when they occur. AI systems are inherently socio-technical in nature, meaning they are influenced by societal dynamics and human behavior. AI risks – and benefits – can emerge from the interplay of technical aspects combined with societal factors related to how a system is used, its interactions with other AI systems, who operates it, and the social context in which it is deployed.

These risks make AI a uniquely challenging technology to deploy and utilize both for organizations and within society. Without proper controls, AI systems can amplify, perpetuate, or exacerbate inequitable or undesirable outcomes for individuals and communities. With proper controls, AI systems can mitigate and manage inequitable outcomes.

AI risk management is a key component of responsible development and use of AI systems. Responsible AI practices can help align the decisions about AI system design, development, and uses with intended aim and values. Core concepts in responsible AI emphasize human centricity, social responsibility, and sustainability. AI risk management can drive responsible uses and practices by prompting organizations and their internal teams who design, develop, and deploy AI to think more critically about context and potential or unexpected negative and positive impacts. Understanding and managing the risks of AI systems will help to enhance trustworthiness, and in turn, cultivate public trust.

Social responsibility can refer to the organization’s responsibility “for the impacts of its decisions and activities on society and the environment through transparent and ethical behavior” (ISO 26000:2010). Sustainability refers to the “state of the global system, including environmental, social, and economic aspects, in which the needs of the present are met without compromising the ability of future generations to meet their own needs” (ISO/IEC TR 24368:2022). Responsible AI is meant to result in technology that is also equitable and accountable. The expectation is that organizational practices are carried out in accord with “professional responsibility,” defined by ISO as an approach that “aims to ensure that professionals who design, develop, or deploy AI systems and applications or AI-based products or systems, recognize their unique position to exert influence on people, society, and the future of AI” (ISO/IEC TR 24368:2022).

As directed by the National Artificial Intelligence Initiative Act of 2020 (P.L. 116-283), the goal of the AI RMF is to offer a resource to the organizations designing, developing, deploying, or using AI systems to help manage the many risks of AI and promote trustworthy and responsible development and use of AI systems. The Framework is intended to be voluntary, rights-preserving, non-sector-specific, and use-case agnostic, providing flexibility to organizations of all sizes and in all sectors and throughout society to implement the approaches in the Framework.

The Framework is designed to equip organizations and individuals – referred to here as AI actors – with approaches that increase the trustworthiness of AI systems, and to help foster the responsible design, development, deployment, and use of AI systems over time. AI actors are defined by the Organisation for Economic Co-operation and Development (OECD) as “those who play an active role in the AI system lifecycle, including organizations and individuals that deploy or operate AI” [OECD (2019) Artificial Intelligence in Society—OECD iLibrary] (See Appendix A).

The AI RMF is intended to be practical, to adapt to the AI landscape as AI technologies continue to develop, and to be operationalized by organizations in varying degrees and capacities so society can benefit from AI while also being protected from its potential harms.

The Framework and supporting resources will be updated, expanded, and improved based on evolving technology, the standards landscape around the world, and AI community experience and feedback. NIST will continue to align the AI RMF and related guidance with applicable international standards, guidelines, and practices. As the AI RMF is put into use, additional lessons will be learned to inform future updates and additional resources.

The Framework is divided into two parts. Part 1 discusses how organizations can frame the risks related to AI and describes the intended audience. Next, AI risks and trustworthiness are analyzed, outlining the characteristics of trustworthy AI systems, which include valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy enhanced, and fair with their harmful biases managed.

Part 2 comprises the “Core” of the Framework. It describes four specific functions to help organizations address the risks of AI systems in practice. These functions – GOVERN, MAP, MEASURE, and MANAGE – are broken down further into categories and subcategories. While GOVERN applies to all stages of organizations’ AI risk management processes and procedures, the MAP, MEASURE, and MANAGE functions can be applied in AI system-specific contexts and at specific stages of the AI lifecycle.

Additional resources related to the Framework are included in the AI RMF Playbook, which is available via the NIST AI RMF website: https://www.nist.gov/itl/ai-risk-management-framework.

Development of the AI RMF by NIST in collaboration with the private and public sectors is directed and consistent with its broader AI efforts called for by the National AI Initiative Act of 2020, the National Security Commission on Artificial Intelligence recommendations, and the Plan for Federal Engagement in Developing Technical Standards and Related Tools. Engagement with the AI community during this Framework’s development – via responses to a formal Request for Information, three widely attended workshops, public comments on a concept paper and two drafts of the Framework, discussions at multiple public forums, and many small group meetings – has informed development of the AI RMF 1.0 as well as AI research and development and evaluation conducted by NIST and others. Priority research and additional guidance that will enhance this Framework will be captured in an associated AI Risk Management Framework Roadmap to which NIST and the broader community can contribute.

3. AI Risks and Trustworthiness

For AI systems to be trustworthy, they often need to be responsive to a multiplicity of criteria that are of value to interested parties. Approaches which enhance AI trustworthiness can reduce negative AI risks. This Framework articulates the following characteristics of trustworthy AI and offers guidance for addressing them. Characteristics of trustworthy AI systems include: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed. Creating trustworthy AI requires balancing each of these characteristics based on the AI system’s context of use. While all characteristics are socio-technical system attributes, accountability and transparency also relate to the processes and activities internal to an AI system and its external setting. Neglecting these characteristics can increase the probability and magnitude of negative consequences.

Trustworthiness characteristics (shown in Figure 4) are inextricably tied to social and organizational behavior, the datasets used by AI systems, selection of AI models and algorithms and the decisions made by those who build them, and the interactions with the humans who provide insight from and oversight of such systems. Human judgment should be employed when deciding on the specific metrics related to AI trustworthiness characteristics and the precise threshold values for those metrics.

Addressing AI trustworthiness characteristics individually will not ensure AI system trustworthiness; tradeoffs are usually involved, rarely do all characteristics apply in every setting, and some will be more or less important in any given situation. Ultimately, trustworthiness is a social concept that ranges across a spectrum and is only as strong as its weakest characteristics.

When managing AI risks, organizations can face difficult decisions in balancing these characteristics. For example, in certain scenarios tradeoffs may emerge between optimizing for interpretability and achieving privacy. In other cases, organizations might face a tradeoff between predictive accuracy and interpretability. Or, under certain conditions such as data sparsity, privacy-enhancing techniques can result in a loss in accuracy, affecting decisions about fairness and other values in certain domains. Dealing with tradeoffs requires taking into account the decision-making context. These analyses can highlight the existence and extent of tradeoffs between different measures, but they do not answer questions about how to navigate the tradeoff. Those depend on the values at play in the relevant context and should be resolved in a manner that is both transparent and appropriately justifiable.

There are multiple approaches for enhancing contextual awareness in the AI lifecycle. For example, subject matter experts can assist in the evaluation of TEVV findings and work with product and deployment teams to align TEVV parameters to requirements and deployment conditions. When properly resourced, increasing the breadth and diversity of input from interested parties and relevant AI actors throughout the AI lifecycle can enhance opportunities for informing contextually sensitive evaluations, and for identifying AI system benefits and positive impacts. These practices can increase the likelihood that risks arising in social contexts are managed appropriately.

Understanding and treatment of trustworthiness characteristics depends on an AI actor’s particular role within the AI lifecycle. For any given AI system, an AI designer or developer may have a different perception of the characteristics than the deployer.

Trustworthiness characteristics explained in this document influence each other. Highly secure but unfair systems, accurate but opaque and uninterpretable systems, and inaccurate but secure, privacy-enhanced, and transparent systems are all undesirable. A comprehensive approach to risk management calls for balancing tradeoffs among the trustworthiness characteristics. It is the joint responsibility of all AI actors to determine whether AI technology is an appropriate or necessary tool for a given context or purpose, and how to use it responsibly. The decision to commission or deploy an AI system should be based on a contextual assessment of trustworthiness characteristics and the relative risks, impacts, costs, and benefits, and informed by a broad set of interested parties.

3.1 Valid and Reliable

Validation is the “confirmation, through the provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled” (Source: ISO 9000:2015). Deployment of AI systems which are inaccurate, unreliable, or poorly generalized to data and settings beyond their training creates and increases negative AI risks and reduces trustworthiness.

Reliability is defined in the same standard as the “ability of an item to perform as required, without failure, for a given time interval, under given conditions” (Source: ISO/IEC TS 5723:2022). Reliability is a goal for overall correctness of AI system operation under the conditions of expected use and over a given period of time, including the entire lifetime of the system.

Accuracy and robustness contribute to the validity and trustworthiness of AI systems, and can be in tension with one another in AI systems.

Accuracy is defined by ISO/IEC TS 5723:2022 as “closeness of results of observations, computations, or estimates to the true values or the values accepted as being true.” Measures of accuracy should consider computational-centric measures (e.g., false positive and false negative rates), human-AI teaming, and demonstrate external validity (generalizable beyond the training conditions). Accuracy measurements should always be paired with clearly defined and realistic test sets – that are representative of conditions of expected use – and details about test methodology; these should be included in associated documentation. Accuracy measurements may include disaggregation of results for different data segments.

Robustness or generalizability is defined as the “ability of a system to maintain its level of performance under a variety of circumstances” (Source: ISO/IEC TS 5723:2022). Robustness is a goal for appropriate system functionality in a broad set of conditions and circumstances, including uses of AI systems not initially anticipated. Robustness requires not only that the system perform exactly as it does under expected uses, but also that it should perform in ways that minimize potential harms to people if it is operating in an unexpected setting.

Validity and reliability for deployed AI systems are often assessed by ongoing testing or monitoring that confirms a system is performing as intended. Measurement of validity, accuracy, robustness, and reliability contribute to trustworthiness and should take into consideration that certain types of failures can cause greater harm. AI risk management efforts should prioritize the minimization of potential negative impacts, and may need to include human intervention in cases where the AI system cannot detect or correct errors.

3.2 Safe

AI systems should “not under defined conditions, lead to a state in which human life, health, property, or the environment is endangered” (Source: ISO/IEC TS 5723:2022). Safe operation of AI systems is improved through:

responsible design, development, and deployment practices;
clear information to deployers on responsible use of the system;
responsible decision-making by deployers and end users; and
explanations and documentation of risks based on empirical evidence of incidents.

Different types of safety risks may require tailored AI risk management approaches based on context and the severity of potential risks presented. Safety risks that pose a potential risk of serious injury or death call for the most urgent prioritization and most thorough risk management process.

Employing safety considerations during the lifecycle and starting as early as possible with planning and design can prevent failures or conditions that can render a system dangerous. Other practical approaches for AI safety often relate to rigorous simulation and in-domain testing, real-time monitoring, and the ability to shut down, modify, or have human intervention into systems that deviate from intended or expected functionality.

AI safety risk management approaches should take cues from efforts and guidelines for safety in fields such as transportation and healthcare, and align with existing sector- or application-specific guidelines or standards.

3.3 Secure and Resilient

AI systems, as well as the ecosystems in which they are deployed, may be said to be resilient if they can withstand unexpected adverse events or unexpected changes in their environment or use – or if they can maintain their functions and structure in the face of internal and external change and degrade safely and gracefully when this is necessary (Adapted from: ISO/IEC TS 5723:2022). Common security concerns relate to adversarial examples, data poisoning, and the exfiltration of models, training data, or other intellectual property through AI system endpoints. AI systems that can maintain confidentiality, integrity, and availability through protection mechanisms that prevent unauthorized access and use may be said to be secure. Guidelines in the NIST Cybersecurity Framework and Risk Management Framework are among those which are applicable here.

Security and resilience are related but distinct characteristics. While resilience is the ability to return to normal function after an unexpected adverse event, security includes resilience but also encompasses protocols to avoid, protect against, respond to, or recover from attacks. Resilience relates to robustness and goes beyond the provenance of the data to encompass unexpected or adversarial use (or abuse or misuse) of the model or data.

3.4 Accountable and Transparent

Trustworthy AI depends upon accountability. Accountability presupposes transparency. Transparency reflects the extent to which information about an AI system and its outputs is available to individuals interacting with such a system – regardless of whether they are even aware that they are doing so. Meaningful transparency provides access to appropriate levels of information based on the stage of the AI lifecycle and tailored to the role or knowledge of AI actors or individuals interacting with or using the AI system. By promoting higher levels of understanding, transparency increases confidence in the AI system.

This characteristic’s scope spans from design decisions and training data to model training, the structure of the model, its intended use cases, and how and when deployment, post-deployment, or end user decisions were made and by whom. Transparency is often necessary for actionable redress related to AI system outputs that are incorrect or otherwise lead to negative impacts. Transparency should consider human-AI interaction: for example, how a human operator or user is notified when a potential or actual adverse outcome caused by an AI system is detected. A transparent system is not necessarily an accurate, privacy-enhanced, secure, or fair system. However, it is difficult to determine whether an opaque system possesses such characteristics, and to do so over time as complex systems evolve.

The role of AI actors should be considered when seeking accountability for the outcomes of AI systems. The relationship between risk and accountability associated with AI and technological systems more broadly differs across cultural, legal, sectoral, and societal contexts. When consequences are severe, such as when life and liberty are at stake, AI developers and deployers should consider proportionally and proactively adjusting their transparency and accountability practices. Maintaining organizational practices and governing structures for harm reduction, like risk management, can help lead to more accountable systems.

Measures to enhance transparency and accountability should also consider the impact of these efforts on the implementing entity, including the level of necessary resources and the need to safeguard proprietary information.

Maintaining the provenance of training data and supporting attribution of the AI system’s decisions to subsets of training data can assist with both transparency and accountability. Training data may also be subject to copyright and should follow applicable intellectual property rights laws.

As transparency tools for AI systems and related documentation continue to evolve, developers of AI systems are encouraged to test different types of transparency tools in cooperation with AI deployers to ensure that AI systems are used as intended.

3.5 Explainable and Interpretable

Explainability refers to a representation of the mechanisms underlying AI systems’ operation, whereas interpretability refers to the meaning of AI systems’ output in the context of their designed functional purposes. Together, explainability and interpretability assist those operating or overseeing an AI system, as well as users of an AI system, to gain deeper insights into the functionality and trustworthiness of the system, including its outputs. The underlying assumption is that perceptions of negative risk stem from a lack of ability to make sense of, or contextualize, system output appropriately. Explainable and interpretable AI systems offer information that will help end users understand the purposes and potential impact of an AI system.

Risk from lack of explainability may be managed by describing how AI systems function, with descriptions tailored to individual differences such as the user’s role, knowledge, and skill level. Explainable systems can be debugged and monitored more easily, and they lend themselves to more thorough documentation, audit, and governance.

Risks to interpretability often can be addressed by communicating a description of why an AI system made a particular prediction or recommendation. (See “Four Principles of Explainable Artificial Intelligence” and “Psychological Foundations of Explainability and Interpretability in Artificial Intelligence” found here.)

Transparency, explainability, and interpretability are distinct characteristics that support each other. Transparency can answer the question of “what happened” in the system. Explainability can answer the question of “how” a decision was made in the system. Interpretability can answer the question of “why” a decision was made by the system and its meaning or context to the user.

3.6 Privacy-Enhanced

Privacy refers generally to the norms and practices that help to safeguard human autonomy, identity, and dignity. These norms and practices typically address freedom from intrusion, limiting observation, or individuals’ agency to consent to disclosure or control of facets of their identities (e.g., body, data, reputation). (See The NIST Privacy Framework: A Tool for Improving Privacy through Enterprise Risk Management.)

Privacy values such as anonymity, confidentiality, and control generally should guide choices for AI system design, development, and deployment. Privacy-related risks may influence security, bias, and transparency and come with tradeoffs with these other characteristics. Like safety and security, specific technical features of an AI system may promote or reduce privacy. AI systems can also present new risks to privacy by allowing inference to identify individuals or previously private information about individuals.

Privacy-enhancing technologies (“PETs”) for AI, as well as data minimizing methods such as de-identification and aggregation for certain model outputs, can support design for privacy-enhanced AI systems. Under certain conditions such as data sparsity, privacy-enhancing techniques can result in a loss in accuracy, affecting decisions about fairness and other values in certain domains.

3.7 Fair – with Harmful Bias Managed

Fairness in AI includes concerns for equality and equity by addressing issues such as harmful bias and discrimination. Standards of fairness can be complex and difficult to define because perceptions of fairness differ among cultures and may shift depending on application. Organizations’ risk management efforts will be enhanced by recognizing and considering these differences. Systems in which harmful biases are mitigated are not necessarily fair. For example, systems in which predictions are somewhat balanced across demographic groups may still be inaccessible to individuals with disabilities or affected by the digital divide or may exacerbate existing disparities or systemic biases.

Bias is broader than demographic balance and data representativeness. NIST has identified three major categories of AI bias to be considered and managed: systemic, computational and statistical, and human-cognitive. Each of these can occur in the absence of prejudice, partiality, or discriminatory intent. Systemic bias can be present in AI datasets, the organizational norms, practices, and processes across the AI lifecycle, and the broader society that uses AI systems. Computational and statistical biases can be present in AI datasets and algorithmic processes, and often stem from systematic errors due to non-representative samples. Human-cognitive biases relate to how an individual or group perceives AI system information to make a decision or fill in missing information, or how humans think about purposes and functions of an AI system. Human-cognitive biases are omnipresent in decision-making processes across the AI lifecycle and system use, including the design, implementation, operation, and maintenance of AI.

Bias exists in many forms and can become ingrained in the automated systems that help make decisions about our lives. While bias is not always a negative phenomenon, AI systems can potentially increase the speed and scale of biases and perpetuate and amplify harms to individuals, groups, communities, organizations, and society. Bias is tightly associated with the concepts of transparency as well as fairness in society. (For more information about bias, including the three categories, see NIST Special Publication 1270, Towards a Standard for Identifying and Managing Bias in Artificial Intelligence.)

Part 2: Core and Profiles

5. AI RMF Core

The AI RMF Core provides outcomes and actions that enable dialogue, understanding, and activities to manage AI risks and responsibly develop trustworthy AI systems. As illustrated in Figure 5, the Core is composed of four functions: GOVERN, MAP, MEASURE, and MANAGE. Each of these high-level functions is broken down into categories and subcategories. Categories and subcategories are subdivided into specific actions and outcomes. Actions do not constitute a checklist, nor are they necessarily an ordered set of steps.

Risk management should be continuous, timely, and performed throughout the AI system lifecycle dimensions. AI RMF Core functions should be carried out in a way that reflects diverse and multidisciplinary perspectives, potentially including the views of AI actors outside the organization. Having a diverse team contributes to more open sharing of ideas and assumptions about purposes and functions of the technology being designed, developed, deployed, or evaluated – which can create opportunities to surface problems and identify existing and emergent risks.

An online companion resource to the AI RMF, the NIST AI RMF Playbook, is available to help organizations navigate the AI RMF and achieve its outcomes through suggested tactical actions they can apply within their own contexts. Like the AI RMF, the Playbook is voluntary and organizations can utilize the suggestions according to their needs and interests. Playbook users can create tailored guidance selected from suggested material for their own use and contribute their suggestions for sharing with the broader community. Along with the AI RMF, the Playbook is part of the NIST Trustworthy and Responsible AI Resource Center.

Framework users may apply these functions as best suits their needs for managing AI risks based on their resources and capabilities. Some organizations may choose to select from among the categories and subcategories; others may choose and have the capacity to apply all categories and subcategories. Assuming a governance structure is in place, functions may be performed in any order across the AI lifecycle as deemed to add value by a user of the framework. After instituting the outcomes in GOVERN, most users of the AI RMF would start with the MAP function and continue to MEASURE or MANAGE. However users integrate the functions, the process should be iterative, with cross-referencing between functions as necessary. Similarly, there are categories and subcategories with elements that apply to multiple functions, or that logically should take place before certain subcategory decisions.

5.1 Govern

The GOVERN function:

cultivates and implements a culture of risk management within organizations designing, developing, deploying, evaluating, or acquiring AI systems;
outlines processes, documents, and organizational schemes that anticipate, identify, and manage the risks a system can pose, including to users and others across society – and procedures to achieve those outcomes;
incorporates processes to assess potential impacts;
provides a structure by which AI risk management functions can align with organizational principles, policies, and strategic priorities;
connects technical aspects of AI system design and development to organizational values and principles, and enables organizational practices and competencies for the individuals involved in acquiring, training, deploying, and monitoring such systems; and
addresses full product lifecycle and associated processes, including legal and other issues concerning use of third-party software or hardware systems and data.

GOVERN is a cross-cutting function that is infused throughout AI risk management and enables the other functions of the process. Aspects of GOVERN, especially those related to compliance or evaluation, should be integrated into each of the other functions. Attention to governance is a continual and intrinsic requirement for effective AI risk management over an AI system’s lifespan and the organization’s hierarchy.

Strong governance can drive and enhance internal practices and norms to facilitate organizational risk culture. Governing authorities can determine the overarching policies that direct an organization’s mission, goals, values, culture, and risk tolerance. Senior leadership sets the tone for risk management within an organization, and with it, organizational culture. Management aligns the technical aspects of AI risk management to policies and operations. Documentation can enhance transparency, improve human review processes, and bolster accountability in AI system teams.

After putting in place the structures, systems, processes, and teams described in the GOVERN function, organizations should benefit from a purpose-driven culture focused on risk understanding and management. It is incumbent on Framework users to continue to execute the GOVERN function as knowledge, cultures, and needs or expectations from AI actors evolve over time.

Practices related to governing AI risks are described in the NIST AI RMF Playbook. Table 1 lists the GOVERN function’s categories and subcategories.

| Category | Subcategory | |---|---| | GOVERN 1: Policies, processes, procedures, and practices across the organization related to the mapping, measuring, and managing of AI risks are in place, transparent, and implemented effectively. | GOVERN 1.1: Legal and regulatory requirements involving AI are understood, managed, and documented. | | | GOVERN 1.2: The characteristics of trustworthy AI are integrated into organizational policies, processes, procedures, and practices. | | | GOVERN 1.3: Processes, procedures, and practices are in place to determine the needed level of risk management activities based on the organization’s risk tolerance. | | | GOVERN 1.4: The risk management process and its outcomes are established through transparent policies, procedures, and other controls based on organizational risk priorities. | | | GOVERN 1.5: Ongoing monitoring and periodic review of the risk management process and its outcomes are planned and organizational roles and responsibilities clearly defined, including determining the frequency of periodic review. | | | GOVERN 1.6: Mechanisms are in place to inventory AI systems and are resourced according to organizational risk priorities. | | | GOVERN 1.7: Processes and procedures are in place for decommissioning and phasing out AI systems safely and in a manner that does not increase risks or decrease the organization’s trustworthiness. | | GOVERN 2: Accountability structures are in place so that the appropriate teams and individuals are empowered, responsible, and trained for mapping, measuring, and managing AI risks. | GOVERN 2.1: Roles and responsibilities and lines of communication related to mapping, measuring, and managing AI risks are documented and are clear to individuals and teams throughout the organization. | | | GOVERN 2.2: The organization’s personnel and partners receive AI risk management training to enable them to perform their duties and responsibilities consistent with related policies, procedures, and agreements. | | | GOVERN 2.3: Executive leadership of the organization takes responsibility for decisions about risks associated with AI system development and deployment. | | GOVERN 3: Workforce diversity, equity, inclusion, and accessibility processes are prioritized in the mapping, measuring, and managing of AI risks throughout the lifecycle. | GOVERN 3.1: Decision-making related to mapping, measuring, and managing AI risks throughout the lifecycle is informed by a diverse team (e.g., diversity of demographics, disciplines, experience, expertise, and backgrounds). | | | GOVERN 3.2: Policies and procedures are in place to define and differentiate roles and responsibilities for human-AI configurations and oversight of AI systems. | | GOVERN 4: Organizational teams are committed to a culture that considers and communicates AI risk. | GOVERN 4.1: Organizational policies and practices are in place to foster a critical thinking and safety-first mindset in the design, development, deployment, and uses of AI systems to minimize potential negative impacts. | | | GOVERN 4.2: Organizational teams document the risks and potential impacts of the AI technology they design, develop, deploy, evaluate, and use, and they communicate about the impacts more broadly. | | | GOVERN 4.3: Organizational practices are in place to enable AI testing, identification of incidents, and information sharing. | | GOVERN 5: Processes are in place for robust engagement with relevant AI actors. | GOVERN 5.1: Organizational policies and practices are in place to collect, consider, prioritize, and integrate feedback from those external to the team that developed or deployed the AI system regarding the potential individual and societal impacts related to AI risks. | | | GOVERN 5.2: Mechanisms are established to enable the team that developed or deployed AI systems to regularly incorporate adjudicated feedback from relevant AI actors into system design and implementation. | | GOVERN 6: Policies and procedures are in place to address AI risks and benefits arising from third-party software and data and other supply chain issues. | GOVERN 6.1: Policies and procedures are in place that address AI risks associated with third-party entities, including risks of infringement of a third-party’s intellectual property or other rights. | | | GOVERN 6.2: Contingency processes are in place to handle failures or incidents in third-party data or AI systems deemed to be high-risk. |

5.2 Map

The MAP function establishes the context to frame risks related to an AI system. The AI lifecycle consists of many interdependent activities involving a diverse set of actors (See Figure 3). In practice, AI actors in charge of one part of the process often do not have full visibility or control over other parts and their associated contexts. The interdependencies between these activities, and among the relevant AI actors, can make it difficult to reliably anticipate impacts of AI systems. For example, early decisions in identifying purposes and objectives of an AI system can alter its behavior and capabilities, and the dynamics of deployment setting (such as end users or impacted individuals) can shape the impacts of AI system decisions. As a result, the best intentions within one dimension of the AI lifecycle can be undermined via interactions with decisions and conditions in other, later activities.

This complexity and varying levels of visibility can introduce uncertainty into risk management practices. Anticipating, assessing, and otherwise addressing potential sources of negative risk can mitigate this uncertainty and enhance the integrity of the decision process. The information gathered while carrying out the MAP function enables negative risk prevention and informs decisions for processes such as model management, as well as an initial decision about appropriateness or the need for an AI solution. Outcomes in the MAP function are the basis for the MEASURE and MANAGE functions. Without contextual knowledge, and awareness of risks within the identified contexts, risk management is difficult to perform. The MAP function is intended to enhance an organization’s ability to identify risks and broader contributing factors.

Implementation of this function is enhanced by incorporating perspectives from a diverse internal team and engagement with those external to the team that developed or deployed the AI system. Engagement with external collaborators, end users, potentially impacted communities, and others may vary based on the risk level of a particular AI system, the makeup of the internal team, and organizational policies. Gathering such broad perspectives can help organizations proactively prevent negative risks and develop more trustworthy AI systems by:

improving their capacity for understanding contexts;
checking their assumptions about context of use;
enabling recognition of when systems are not functional within or out of their intended context;
identifying positive and beneficial uses of their existing AI systems;
improving understanding of limitations in AI and ML processes;
identifying constraints in real-world applications that may lead to negative impacts;
identifying known and foreseeable negative impacts related to intended use of AI systems; and
anticipating risks of the use of AI systems beyond intended use.

After completing the MAP function, Framework users should have sufficient contextual knowledge about AI system impacts to inform an initial go/no-go decision about whether to design, develop, or deploy an AI system. If a decision is made to proceed, organizations should utilize the MEASURE and MANAGE functions along with policies and procedures put into place in the GOVERN function to assist in AI risk management efforts. It is incumbent on Framework users to continue applying the MAP function to AI systems as context, capabilities, risks, benefits, and potential impacts evolve over time.

Practices related to mapping AI risks are described in the NIST AI RMF Playbook. Table 2 lists the MAP function’s categories and subcategories.

| Category | Subcategory | |---|---| | MAP 1: Context is established and understood. | MAP 1.1: Intended purposes, potentially beneficial uses, context-specific laws, norms and expectations, and prospective settings in which the AI system will be deployed are understood and documented. Considerations include: the specific set or types of users along with their expectations; potential positive and negative impacts of system uses to individuals, communities, organizations, society, and the planet; assumptions and related limitations about AI system purposes, uses, and risks across the development or product AI lifecycle; and related TEVV and system metrics. | | | MAP 1.2: Interdisciplinary AI actors, competencies, skills, and capacities for establishing context reflect demographic diversity and broad domain and user experience expertise, and their participation is documented. Opportunities for interdisciplinary collaboration are prioritized. | | | MAP 1.3: The organization’s mission and relevant goals for AI technology are understood and documented. | | | MAP 1.4: The business value or context of business use has been clearly defined or – in the case of assessing existing AI systems – re-evaluated. | | | MAP 1.5: Organizational risk tolerances are determined and documented. | | | MAP 1.6: System requirements (e.g., “the system shall respect the privacy of its users”) are elicited from and understood by relevant AI actors. Design decisions take socio-technical implications into account to address AI risks. | | MAP 2: Categorization of the AI system is performed. | MAP 2.1: The specific tasks and methods used to implement the tasks that the AI system will support are defined (e.g., classifiers, generative models, recommenders). | | | MAP 2.2: Information about the AI system’s knowledge limits and how system output may be utilized and overseen by humans is documented. Documentation provides sufficient information to assist relevant AI actors when making decisions and taking subsequent actions. | | | MAP 2.3: Scientific integrity and TEVV considerations are identified and documented, including those related to experimental design, data collection and selection (e.g., availability, representativeness, suitability), system trustworthiness, and construct validation. | | MAP 3: AI capabilities, targeted usage, goals, and expected benefits and costs compared with appropriate benchmarks are understood. | MAP 3.1: Potential benefits of intended AI system functionality and performance are examined and documented. | | | MAP 3.2: Potential costs, including non-monetary costs, which result from expected or realized AI errors or system functionality and trustworthiness – as connected to organizational risk tolerance – are examined and documented. | | | MAP 3.3: Targeted application scope is specified and documented based on the system’s capability, established context, and AI system categorization. | | | MAP 3.4: Processes for operator and practitioner proficiency with AI system performance and trustworthiness – and relevant technical standards and certifications – are defined, assessed, and documented. | | | MAP 3.5: Processes for human oversight are defined, assessed, and documented in accordance with organizational policies from the GOVERN function. | | MAP 4: Risks and benefits are mapped for all components of the AI system including third-party software and data. | MAP 4.1: Approaches for mapping AI technology and legal risks of its components – including the use of third-party data or software – are in place, followed, and documented, as are risks of infringement of a third party’s intellectual property or other rights. | | | MAP 4.2: Internal risk controls for components of the AI system, including third-party AI technologies, are identified and documented. | | MAP 5: Impacts to individuals, groups, communities, organizations, and society are characterized. | MAP 5.1: Likelihood and magnitude of each identified impact (both potentially beneficial and harmful) based on expected use, past uses of AI systems in similar contexts, public incident reports, feedback from those external to the team that developed or deployed the AI system, or other data are identified and documented. | | | MAP 5.2: Practices and personnel for supporting regular engagement with relevant AI actors and integrating feedback about positive, negative, and unanticipated impacts are in place and documented. |

5.3 Measure

The MEASURE function employs quantitative, qualitative, or mixed-method tools, techniques, and methodologies to analyze, assess, benchmark, and monitor AI risk and related impacts. It uses knowledge relevant to AI risks identified in the MAP function and informs the MANAGE function. AI systems should be tested before their deployment and regularly while in operation. AI risk measurements include documenting aspects of systems’ functionality and trustworthiness.

Measuring AI risks includes tracking metrics for trustworthy characteristics, social impact, and human-AI configurations. Processes developed or adopted in the MEASURE function should include rigorous software testing and performance assessment methodologies with associated measures of uncertainty, comparisons to performance benchmarks, and formalized reporting and documentation of results. Processes for independent review can improve the effectiveness of testing and can mitigate internal biases and potential conflicts of interest.

Where tradeoffs among the trustworthy characteristics arise, measurement provides a traceable basis to inform management decisions. Options may include recalibration, impact mitigation, or removal of the system from design, development, production, or use, as well as a range of compensating, detective, deterrent, directive, and recovery controls.

After completing the MEASURE function, objective, repeatable, or scalable test, evaluation, verification, and validation (TEVV) processes including metrics, methods, and methodologies are in place, followed, and documented. Metrics and measurement methodologies should adhere to scientific, legal, and ethical norms and be carried out in an open and transparent process. New types of measurement, qualitative and quantitative, may need to be developed. The degree to which each measurement type provides unique and meaningful information to the assessment of AI risks should be considered. Framework users will enhance their capacity to comprehensively evaluate system trustworthiness, identify and track existing and emergent risks, and verify efficacy of the metrics. Measurement outcomes will be utilized in the MANAGE function to assist risk monitoring and response efforts. It is incumbent on Framework users to continue applying the MEASURE function to AI systems as knowledge, methodologies, risks, and impacts evolve over time.

Practices related to measuring AI risks are described in the NIST AI RMF Playbook. Table 3 lists the MEASURE function’s categories and subcategories.

| Category | Subcategory | |---|---| | MEASURE 1: Appropriate methods and metrics are identified and applied. | MEASURE 1.1: Approaches and metrics for measurement of AI risks enumerated during the MAP function are selected for implementation starting with the most significant AI risks. The risks or trustworthiness characteristics that will not – or cannot – be measured are properly documented. | | | MEASURE 1.2: Appropriateness of AI metrics and effectiveness of existing controls are regularly assessed and updated, including reports of errors and potential impacts on affected communities. | | | MEASURE 1.3: Internal experts who did not serve as front-line developers for the system and/or independent assessors are involved in regular assessments and updates. Domain experts, users, AI actors external to the team that developed or deployed the AI system, and affected communities are consulted in support of assessments as necessary per organizational risk tolerance. | | MEASURE 2: AI systems are evaluated for trustworthy characteristics. | MEASURE 2.1: Test sets, metrics, and details about the tools used during TEVV are documented. | | | MEASURE 2.2: Evaluations involving human subjects meet applicable requirements (including human subject protection) and are representative of the relevant population. | | | MEASURE 2.3: AI system performance or assurance criteria are measured qualitatively or quantitatively and demonstrated for conditions similar to deployment setting(s). Measures are documented. | | | MEASURE 2.4: The functionality and behavior of the AI system and its components – as identified in the MAP function – are monitored when in production. | | | MEASURE 2.5: The AI system to be deployed is demonstrated to be valid and reliable. Limitations of the generalizability beyond the conditions under which the technology was developed are documented. | | | MEASURE 2.6: The AI system is evaluated regularly for safety risks – as identified in the MAP function. The AI system to be deployed is demonstrated to be safe, its residual negative risk does not exceed the risk tolerance, and it can fail safely, particularly if made to operate beyond its knowledge limits. Safety metrics reflect system reliability and robustness, real-time monitoring, and response times for AI system failures. | | | MEASURE 2.7: AI system security and resilience – as identified in the MAP function – are evaluated and documented. | | | MEASURE 2.8: Risks associated with transparency and accountability – as identified in the MAP function – are examined and documented. | | | MEASURE 2.9: The AI model is explained, validated, and documented, and AI system output is interpreted within its context – as identified in the MAP function – to inform responsible use and governance. | | | MEASURE 2.10: Privacy risk of the AI system – as identified in the MAP function – is examined and documented. | | | MEASURE 2.11: Fairness and bias – as identified in the MAP function – are evaluated and results are documented. | | | MEASURE 2.12: Environmental impact and sustainability of AI model training and management activities – as identified in the MAP function – are assessed and documented. | | | MEASURE 2.13: Effectiveness of the employed TEVV metrics and processes in the MEASURE function are evaluated and documented. | | MEASURE 3: Mechanisms for tracking identified AI risks over time are in place. | MEASURE 3.1: Approaches, personnel, and documentation are in place to regularly identify and track existing, unanticipated, and emergent AI risks based on factors such as intended and actual performance in deployed contexts. | | | MEASURE 3.2: Risk tracking approaches are considered for settings where AI risks are difficult to assess using currently available measurement techniques or where metrics are not yet available. | | | MEASURE 3.3: Feedback processes for end users and impacted communities to report problems and appeal system outcomes are established and integrated into AI system evaluation metrics. | | MEASURE 4: Feedback about efficacy of measurement is gathered and assessed. | MEASURE 4.1: Measurement approaches for identifying AI risks are connected to deployment context(s) and informed through consultation with domain experts and other end users. Approaches are documented. | | | MEASURE 4.2: Measurement results regarding AI system trustworthiness in deployment context(s) and across the AI lifecycle are informed by input from domain experts and relevant AI actors to validate whether the system is performing consistently as intended. Results are documented. | | | MEASURE 4.3: Measurable performance improvements or declines based on consultations with relevant AI actors, including affected communities, and field data about context-relevant risks and trustworthiness characteristics are identified and documented. |

5.4 Manage

The MANAGE function entails allocating risk resources to mapped and measured risks on a regular basis and as defined by the GOVERN function. Risk treatment comprises plans to respond to, recover from, and communicate about incidents or events.

Contextual information gleaned from expert consultation and input from relevant AI actors – established in GOVERN and carried out in MAP – is utilized in this function to decrease the likelihood of system failures and negative impacts. Systematic documentation practices established in GOVERN and utilized in MAP and MEASURE bolster AI risk management efforts and increase transparency and accountability. Processes for assessing emergent risks are in place, along with mechanisms for continual improvement.

After completing the MANAGE function, plans for prioritizing risk and regular monitoring and improvement will be in place. Framework users will have enhanced capacity to manage the risks of deployed AI systems and to allocate risk management resources based on assessed and prioritized risks. It is incumbent on Framework users to continue to apply the MANAGE function to deployed AI systems as methods, contexts, risks, and needs or expectations from relevant AI actors evolve over time.

Practices related to managing AI risks are described in the NIST AI RMF Playbook. Table 4 lists the MANAGE function’s categories and subcategories.

| Category | Subcategory | |---|---| | MANAGE 1: AI risks based on assessments and other analytical output from the MAP and MEASURE functions are prioritized, responded to, and managed. | MANAGE 1.1: A determination is made as to whether the AI system achieves its intended purposes and stated objectives and whether its development or deployment should proceed. | | | MANAGE 1.2: Treatment of documented AI risks is prioritized based on impact, likelihood, and available resources or methods. | | | MANAGE 1.3: Responses to the AI risks deemed high priority, as identified by the MAP function, are developed, planned, and documented. Risk response options can include mitigating, transferring, avoiding, or accepting. | | | MANAGE 1.4: Negative residual risks (defined as the sum of all unmitigated risks) to both downstream acquirers of AI systems and end users are documented. | | MANAGE 2: Strategies to maximize AI benefits and minimize negative impacts are planned, prepared, implemented, documented, and informed by input from relevant AI actors. | MANAGE 2.1: Resources required to manage AI risks are taken into account – along with viable non-AI alternative systems, approaches, or methods – to reduce the magnitude or likelihood of potential impacts. | | | MANAGE 2.2: Mechanisms are in place and applied to sustain the value of deployed AI systems. | | | MANAGE 2.3: Procedures are followed to respond to and recover from a previously unknown risk when it is identified. | | | MANAGE 2.4: Mechanisms are in place and applied, and responsibilities are assigned and understood, to supersede, disengage, or deactivate AI systems that demonstrate performance or outcomes inconsistent with intended use. | | MANAGE 3: AI risks and benefits from third-party entities are managed. | MANAGE 3.1: AI risks and benefits from third-party resources are regularly monitored, and risk controls are applied and documented. | | | MANAGE 3.2: Pre-trained models which are used for development are monitored as part of AI system regular monitoring and maintenance. | | MANAGE 4: Risk treatments, including response and recovery, and communication plans for the identified and measured AI risks are documented and monitored regularly. | MANAGE 4.1: Post-deployment AI system monitoring plans are implemented, including mechanisms for capturing and evaluating input from users and other relevant AI actors, appeal and override, decommissioning, incident response, recovery, and change management. | | | MANAGE 4.2: Measurable activities for continual improvements are integrated into AI system updates and include regular engagement with interested parties, including relevant AI actors. | | | MANAGE 4.3: Incidents and errors are communicated to relevant AI actors, including affected communities. Processes for tracking, responding to, and recovering from incidents and errors are followed and documented. |

Discuss a Technology Back to Modern War Tech 101 Back to the Field Guide