AI Risk Management
Purpose
This document describes how the AI Conveyor identifies, assesses, and controls the risks of AI-based solutions: machine learning models, language models, knowledge assistants, automations, and agentic scenarios.
The aim of risk management is not to ban AI, but to prevent a situation where a solution affects a business process, customers, money, or regulatory obligations without clear quality, an owner, and a way to stop it.
Core ideas
- Risk depends on the impact of the solution. An internal assistant and a solution that affects a customer credit limit require different levels of rigor.
- Human in the loop is a management decision. You need to explicitly define where AI advises, where it prepares a draft, and where an action is performed automatically.
- Quality must be measurable. You cannot launch a solution if it is unclear how the accuracy, completeness, stability, or usefulness of the result is verified.
- Constraints must be visible to the user. If a solution is not suitable for legally significant conclusions, this must be explicitly recorded.
- The stop must be described in advance. A working solution must have conditions for review, shutdown, or rollback.
How it works
AI risk management is embedded in the stage gates:
| Stage | What is checked | Decision |
|---|---|---|
| New | whether there is an obviously high risk | send to assessment, reject, or request security |
| Assessment | risk class, data, process impact, human in the loop | choose the delivery route and the level of control |
| Delivery | quality, testing, constraints, logging, rollback | allow launch or return for rework |
| Awaiting impact | actual quality and impact on the process | confirm the impact, adjust, or stop |
| Support | degradation, incidents, changes in data and process | continue, reconsider, or shut down |
Related sections: AI governance model, data governance, stage gate model.
Main types of risk
| Risk | What can go wrong | How to control it |
|---|---|---|
| Result error | the solution gives a wrong answer, forecast, or recommendation | test sets, manual review, quality thresholds |
| Hallucinations | a language model confidently invents facts | references to sources, context restriction, verification of critical answers |
| Bias | the result is systematically worse for some customers, products, or processes | sample analysis, fairness checks, usage constraints |
| Data leakage | sensitive information ends up in an unsuitable environment | data classification, masking, access control |
| Uncontrolled automatic action | the solution performs an action that a human should confirm | human in the loop, limits, action log |
| Quality degradation | the data or process changed, but the solution keeps working the old way | monitoring, periodic review, stop signals |
| Non-reproducibility | it is impossible to understand why a result was produced | versions of data, model, prompt, rules, and parameters |
| Reputational risk | the solution looks unfair, opaque, or unsafe | explainability, usage rules, communication of constraints |
Impact classes
Minimum classification by impact:
| Class | Description | Requirements |
|---|---|---|
| Hint | AI helps the user but does not make the decision | a warning about constraints, basic quality check |
| Draft | AI prepares a text, document, calculation, or recommendation for human review | mandatory human confirmation, logging |
| Recommendation | AI influences a choice, priority, or action | quality metrics, explanation of factors, decision owner |
| Automatic action | AI triggers an action without manual confirmation | strict limits, rollback, audit, extended sign-off |
| Critical impact | the solution affects money, customers, risk, security, or legally significant processes | independent review, human in the loop, regular review |
The higher the impact class, the more requirements there are for verification, monitoring, and documentation.
What to record on the initiative card
For initiatives with medium and high risk, you need to record:
- the impact class;
- the type of decision;
- the data used;
- the business decision owner;
- the technical solution owner;
- usage constraints;
- quality metrics;
- verification results;
- the mode of human involvement;
- stop conditions;
- the rollback plan;
- the date of the next review.
This data can be stored as card fields, artifacts, or tasks, but it must be available when passing through the stage gates.
Pre-launch check
Before moving an initiative to awaiting impact, you need to answer the questions:
- on what data the solution was tested;
- which quality metrics were achieved;
- what counts as an error;
- who reviewed the results;
- which constraints are known;
- what the user will see in the interface;
- who is responsible for incidents;
- how to shut down the solution;
- what happens if the model, data, or external service is unavailable.
If there are no answers to these questions, the solution may be a prototype, but not a governed adoption.
Post-launch monitoring
After launch you need to track:
- result quality;
- usage volume;
- the share of manual corrections;
- complaints and incidents;
- changes in input data;
- integration latency and errors;
- the cost of running the solution;
- the actual impact on business metrics.
For high-impact solutions, a review must be set: for example, once a quarter, after a data source changes, or after a significant incident.
The role of the AI assistant
The AI assistant can:
- assemble a description of the risks;
- prepare a draft security review;
- point out constraints;
- create a list of questions for the data owner;
- help with the initiative report;
- create quality check tasks.
But the AI assistant must not approve risky transitions on its own, give the final security sign-off, or bypass access rights.
Stop conditions
A solution must be stopped or reviewed if:
- quality is below the agreed threshold;
- a leak or access violation is detected;
- the data source has changed;
- users are correcting the result en masse;
- the operating cost has grown;
- the solution is used for unintended purposes;
- the business impact is not confirmed;
- new regulatory or internal constraints have appeared.
Anti-patterns
Bad risk management:
- "we'll check quality after launch";
- a language model answers without references to sources in a critical process;
- automation performs an action without a log and rollback;
- constraints are written in a document but not visible to the user;
- there is no incident owner;
- there is no review date;
- risk does not affect the transition decision.
Good risk management makes risk part of the initiative's route, not a late emergency brake.