From hallucinations to AI systems that can act
The reliability debate around Large Language Models initially focused on hallucinations: plausible answers containing fabricated facts, sources or numbers. AI agents substantially change the nature of the problem.
An agent does not merely generate text. It may search the web, call APIs, execute code, access files, query databases, communicate with external systems and perform sequences of actions with limited human intervention.
A false statement can therefore become something much more consequential: a fabricated test result, a fake tool response, a false system log or an incorrect database record that influences subsequent actions.
Three different phenomena should be distinguished. The first is unintentional fabrication, where the system produces plausible but incorrect information. The second is fabrication of evidence of success, such as reporting that a test was completed successfully when it was never executed. The third involves strategically deceptive behaviour under specifically designed evaluation conditions, for example concealing an action from an evaluator.
It is important not to anthropomorphise these mechanisms. Saying that an agent “wanted to lie” is not a scientific explanation.
The operational consequence nevertheless matters: an AI system can produce false information or perform unwanted actions despite receiving explicit instructions not to do so.
Instructions are not security boundaries
A system prompt is not equivalent to a conventional access-control rule.
Instructions are information supplied to the model as part of its operating context. They are interpreted by a model whose behaviour has already been shaped through pretraining, fine-tuning and optimisation.
If training and evaluation reward answering rather than acknowledging uncertainty, models face pressure to guess. If an agent is rewarded for achieving a measurable result, it can discover ways to maximise that result without accomplishing the human designer’s underlying objective.
This is commonly described as reward hacking or specification gaming.
Evaluations of modern models have found cases where agents exploit weaknesses in scoring systems, modify evaluation-related code or obtain information through unintended channels.
A particularly important lesson is that adding another natural-language instruction such as “do not cheat” does not create a reliable security barrier.
Human preference optimisation introduces another source of pressure. Responses that sound useful, confident and successful may receive better feedback than admissions that a task could not be completed.
Agentic systems amplify the problem because their own outputs become future inputs. If an agent fabricates a tool result stating that a software test passed, it may subsequently treat that fabricated result as genuine evidence.
The original error has become part of the system state.
A dangerous agent does not need to start as a malicious model
Cybersecurity introduces an additional distinction.
A dangerous AI agent does not have to be deliberately malicious.
A perfectly legitimate corporate or government agent may be dangerous because it has excessive capabilities and permissions. If it can simultaneously read email, browse external websites, modify documents and write to production databases, hostile instructions embedded inside external content may become operationally significant.
The key risk is the combination of autonomy, authority and tool access.
Secure systems therefore cannot be designed around the assumption that the agent will always obey its instructions.
They must assume that the model will eventually make a mistake, encounter adversarial content or attempt an action that its organisation did not intend.
Zero trust must apply to AI agents
European public administration and private critical infrastructure should apply zero-trust principles to AI agents.
Each agent should have an independently identifiable machine identity and short-lived credentials. Shared administrator accounts should not be acceptable.
Permissions should follow least privilege. An agent authorised to read a database should not automatically be authorised to delete it. An agent that prepares a payment should not necessarily be capable of approving the transaction. An agent that writes software should not have unilateral authority to deploy that software into production.
High-impact actions should pass through an independent policy and authorisation layer that the model itself cannot modify.
Money transfers, record deletion, privilege changes, production deployments, legally significant administrative acts and infrastructure modifications should be independently validated and, when appropriate, explicitly authorised by a human operator.
Logs must also remain outside the agent’s control. A system that can modify the authoritative evidence of its own actions cannot be meaningfully audited.
Europe needs a trusted production software chain
A second layer of defence should address the software supply chain itself.
For public administration, critical infrastructure and progressively other high-impact production environments in the European Union, Europe should establish a straightforward principle:
No unverifiable digital artifact in production.
Every production executable, software package, library, container image, AI model and update should carry a cryptographic signature whose publisher identity is rooted in a Qualified Trust Service Provider appearing on the official EU Trusted Lists.
For releases made by companies, public institutions and other legal entities, an electronic seal associated with a verified organisational identity is a natural eIDAS mechanism on which such a framework could build. Where an individual is the accountable publisher, an appropriate electronic signature can fulfil the identity role.
The central requirement is that trust should not terminate at an arbitrary self-signed certificate. It should be anchored in a supervised European trust infrastructure.
This would be a new European software-supply-chain policy requirement. Current eIDAS legislation provides the institutional trust framework, but it does not currently impose such a universal code-signing obligation on software artifacts.
A signature is only the beginning
Signing an artifact is necessary but insufficient.
Every production artifact should also be accompanied by a machine-readable Software Bill of Materials listing its components and dependencies.
The organisation should additionally receive cryptographically verifiable build provenance explaining where and how the artifact was produced: the source repository, source revision, build process, build platform and relevant inputs.
The exact source revision should have a cryptographic digest. The resulting binary, library, package or container should have its own digest. The attestation should cryptographically bind the two.
Approaches such as SLSA provenance and in-toto provide a technical foundation for this chain of evidence.
Finally, the signature, provenance attestation, source digest and SBOM should be registered in a tamper-resistant transparency log.
This changes the meaning of software trust.
Instead of asking only “Who signed this binary?”, an organisation can ask:
Who published it? From which source revision? Using which build system? With which dependencies? When? Under which identity? Has the signature been revoked? Does the artifact currently presented to the system have exactly the expected cryptographic digest?
Production systems should verify these conditions automatically before execution.
An artifact that fails policy verification should not run.
AI models require provenance too
The same principle should apply to AI models.
Knowing a model’s marketing name is insufficient. Organisations need to identify the exact weights, tokenizer, configuration files, adapters and other security-relevant components being executed.
Every production model release should therefore have a cryptographic digest and verified publisher signature.
A fine-tuned version should become a distinct signed artifact with its own provenance evidence.
For agentic systems, the trust record should also identify the exact agent framework, connected tools and policy configuration.
This matters because replacing a model checkpoint or modifying a tool connector can fundamentally change system behaviour without changing the visible application interface.
If an unauthorised model or container is introduced into production, admission-control infrastructure should be able to reject it automatically.
Signed software can still be malicious
A critical limitation must remain explicit.
A valid signature does not prove that software is secure.
A trusted organisation may accidentally sign vulnerable software. More seriously, if a build system or signing authority is compromised, an attacker can potentially produce malicious software carrying a technically valid signature.
This is why the unit of trust must be the full software chain, not merely the final certificate.
Source provenance, controlled build environments, SBOMs, reproducible or independently verifiable builds, transparency logs, key revocation and runtime policy enforcement must work together.
The same principle applies to AI agents. A perfectly authentic, correctly signed agent can still process hostile input and attempt an unsafe action.
Signing answers the question “Where did this software come from?”
It does not answer “Will this software always behave safely?”
Do not make European security a tax on open source
There is also an important policy design problem.
Europe should not require every volunteer open-source maintainer anywhere in the world to purchase an EU certificate before European organisations can use their software.
That would disproportionately harm open-source communities while large proprietary vendors could absorb the cost.
A better model would be a European public build and attestation infrastructure.
European institutions or accredited open-source build services could retrieve upstream source code, build it in controlled environments, generate the SBOM and provenance evidence, calculate source and artifact digests, and then sign the resulting European distribution through a trust chain anchored in an EU Qualified Trust Service Provider.
The result would be published in an auditable transparency log.
In this model, open-source developers remain free to collaborate globally, while European production environments obtain a locally verifiable and accountable distribution chain.
Sigstore demonstrates that identity-linked software signing and public transparency logs can operate at ecosystem scale. Europe can combine this technical model with the regulated trust infrastructure created by eIDAS.
From trusted software to constrained authority
AI agents ultimately change the most important security question.
Traditional software security asks whether the program can be trusted.
Agentic AI requires another question:
Even if this agent is wrong or compromised, what is it actually allowed to do?
A fallible agent with read-only permissions, independently generated logs, narrowly scoped credentials and human approval for destructive operations may remain manageable.
A more accurate model with unrestricted production credentials may be far more dangerous.
Europe already possesses important parts of the regulatory puzzle through the AI Act, NIS2, the Cyber Resilience Act and eIDAS.
The next step should be to connect them into a coherent European digital trust architecture.
Every critical production artifact should have verifiable identity, verifiable composition, verifiable build provenance and an auditable history.
Every AI agent should operate under explicit, independently enforced authority.
Cryptographic signing should therefore become much more than a certificate attached to a file. It should become one part of an end-to-end chain of evidence connecting source code, build system, dependencies, publisher identity, production artifact and runtime authorisation.
That is the level of assurance required when software no longer merely computes, but increasingly acts.