AI governance and management learning resources

Governing AI models focuses on reducing model output risks, such as inaccuracies, hallucinations, bias, inappropriate content, and data leaks. Governing AI agents centers on managing potentially harmful agent actions, like unauthorized system calls, costly runaway loops, and privilege escalation. Agent governance requires assigning unique non-human identities to agents, continuously enforcing runtime controls, and keeping humans in the loop for important decisions or actions.

The EU AI Act defines four levels of risk for AI systems: unacceptable risk, high risk, transparency risk, and minimal or no risk. Certain unacceptable-risk AI practices are prohibited, while minimal- or no-risk uses are generally not subject to additional rules under the Act. The U.S. National Institute of Standards and Technology (NIST) AI Risk Management Framework (RMF) is a voluntary framework organized by four functions: Govern, Map, Measure, and Manage. The EU AI Act aims to ensure safe, human-centric, and trustworthy AI while protecting rights and supporting innovation. The NIST AI RMF focuses more narrowly on enhancing AI system trustworthiness.

Accountability and liability are central concerns with agentic AI. Organizations must establish accountability for AI agents as part of their governance program. In the event of serious errors or financial losses, liability will depend on the applicable legal framework, contractual arrangements, and the roles of the organizations involved. Responsibility may sit with the organization deploying the agent, the provider or developer, or other parties depending on the circumstances.

Assigning non-human identities (NHIs) to agents strengthens security by enabling organizations to restrict agent actions with granular, least-privilege access policies. Effective NHI management helps limit agent sprawl. By allowing only verified identities to execute system calls, for example, organizations reduce the number of shadow agents (which wouldn’t be able to execute calls). Organizations can also use short-lived identity tokens to reduce the security risks associated with orphaned or inactive agents.

AI agent swarms occur when multiple agents autonomously collaborate, sharing context or invoking external tools to complete multi-step tasks. Open source tools and frameworks can help address swarms. For example, using the OpenTelemetry (OTel) framework with tools such as Jaeger or Grafana Tempo organizations can monitor and audit AI agent swarms. OTel captures agent telemetry data while Jaeger or Tempo generates a visual map to facilitate auditing.

Establishing an enterprise AI governance committee should begin with assembling cross-functional senior leadership, with members from engineering, cybersecurity, legal, financial, and product groups. Once responsibilities are defined, the team should create policies and then operationalize those policies across the enterprise, possibly through a center of excellence (CoE). The CoE, staffed with technical and risk practitioners, can build standardized processes to discover, evaluate, and classify AI systems; streamline policy-compliant development processes; deploy identity controls; and implement tracing and testing.