Human-in-the-loop isn't a safety net — it's a design principle
Under BaFin and the EU AI Act, design decides conformity. Human-in-the-loop belongs in the architecture, not at the edge.
SIFAMO Editorial — SIFAMO GmbH
What banks and insurers now have to implement, technically and organizationally.
Agentic AI governance under the EU AI Act will become the decisive test for putting AI into production at banks and insurers from 2026 onward.
With the EU AI Act now in force and supervisory expectations increasingly made concrete by BaFin, one thing is clear: AI systems must not only be capable – they must be operated in an explainable, controllable and accountable way. Agentic AI systems in particular, which prepare decisions or act autonomously, push regulatory requirements well beyond classical AI governance.
Regulatory frame: EU AI Act + BaFin practice
The EU AI Act classifies AI systems by risk. For banks and insurers, high-risk AI systems are especially relevant – for example in:
- creditworthiness assessment
- underwriting
- claims handling
- fraud detection
- compliance & risk analysis
Agentic AI falls into this context as soon as it prepares or influences decisions, proposes measures on its own or steers processes without immediate human control. That effectively makes it a mandatory discipline.
Why classical AI governance is no longer enough
Many institutions currently rely on model validation, bias tests and documentation of individual AI components. That is necessary – but not sufficient.
Agentic AI is not a single model, but a system of several components: planning, reasoning, tool use, memory and feedback loops.
Supervisors therefore no longer ask „Is the model correct?“ but „How does the system arrive at this decision – over time?“
BaFin-relevant core requirements for agentic AI
From current supervisory practice, four central governance requirements can be derived:
1. Traceability & explainability
Institutions must be able to demonstrate which steps an agent performed, which data was used and which assumptions were made. Technically required:
- decision logs
- reasoning traces
- versioning of prompts, policies and models
Without these mechanisms, a supervisory audit is barely possible.
2. Human-in-the-loop (effective, not pro forma)
The EU AI Act requires effective human oversight. For agentic AI that means:
- clear escalation thresholds
- defined approval points
- documented interventions
An „emergency button“ is not enough. Human-in-the-loop must be anchored architecturally.
3. Accountability & liability
BaFin reviews increasingly focus on clear role assignment, owners per decision class and documented governance structures.
The central question: who is accountable when an agent prepares a wrong decision? Without clear agent governance, that question cannot be answered.
4. Operations & control (operational governance)
Agentic AI must be operated like critical infrastructure: monitoring, incident management, rollbacks as well as change & release processes. This makes agent governance functionally the new DevOps – with a regulatory focus.