Potential future risks from autonomous AI systems

Approximate read time: 15 minutes

The House of Lords is scheduled to consider the following question for short debate on 8 January 2026:

Lord Fairfax of Cameron (Conservative) to ask His Majesty’s Government what steps they are taking to ensure that advanced AI development remains safe and controllable, given the recent threat update warning from the Director General of MI5 that there are “potential future risks from non-human, autonomous AI systems which may evade human oversight and control”.

1. Background

AI can be broadly defined as the ability of computer systems to perform tasks normally requiring human intelligence.^[1] AI has the potential to bring a range of benefits, but it also poses risks, such as those related to data privacy, biases, misinformation and cyber security. More details can be found in the House of Lords Library briefing ‘Artificial intelligence: Development, risks and regulation’ (18 July 2023).

1.1 Autonomous AI and loss of control

Autonomous AI systems can make decisions without ongoing human guidance. This can bring benefits through automating tasks that require flexible and adaptive problem solving, but it may also introduce additional risks. Some experts argue that future autonomous AI systems could evade human oversight and control.^[2] Experts have identified theoretical pathways that could potentially lead to “loss of control”; however, it is uncertain how likely or impactful this risk may be.

Concerns about loss of control have been discussed since the early days of computer science but have gained particular attention recently due to recent AI advancements.^[3] On 16 October 2025, loss of control was included as a potential future threat in the Security Service’s (MI5) annual threat update, when Director General Sir Ken McCallum stated:

We also need to scope out the next frontier: potential future risks from non-human, autonomous AI systems which may evade human oversight and control.

Given the risk of hype and scaremongering, I will choose my words carefully. I am not forecasting Hollywood movie scenarios. I am, on the whole, a tech optimist, who sees AI bringing real benefits. But, as AI capabilities continue to power ahead, you would expect organisations like MI5, and GCHQ, and the UK’s ground-breaking AI Security Institute, to be thinking deeply, today, about what defending the realm might need to look like in the years ahead.

Artificial intelligence may never ‘mean’ us harm. But it would be reckless to ignore the potential for it to cause harm.

We’re on the case.^[4]

1.2 How loss of control could potentially occur

The 2025 ‘International AI safety report’, commissioned by the 30 countries that attended the 2023 AI Safety Summit (see section 2.1 of this briefing), including the UK, outlined potential passive and active pathways to loss of control.^[5]

Passive loss of control could occur if humans stopped exercising appropriate oversight over AI systems. This could be because the AI decisions were too opaque, complex or fast to allow for meaningful oversight.^[6] Alternatively, this could be driven by “automation bias”, where people place too much trust in automated decisions, or by competitive pressures incentivising organisations to automate tasks beyond levels that allow for proper oversight.

Active loss of control could occur if AI systems actively undermined human control. This could be intentional if AI systems were designed by humans to be difficult to control.^[7] Such a design could be motivated by a desire to cause harm to others or to protect an AI system from interference. Alternatively, loss of control could be unintentional and instead arise due to misalignment.

Misalignment refers to AI behaving in ways that conflict with human intentions or values.^[8] This can arise through goal misspecification, where the objectives set by developers lead AI systems to pursue unintended goals. For example, a recent study found that autonomous AI models given harmless business goals resorted to blackmailing officials, leaking sensitive information and disobeying commands when stress tested in a hypothetical corporate environment.^[9]

In some instances, misalignment could lead to “scheming” behaviour where AI systems find ways to achieve their goals by evading human oversight.^[10] This could be achieved though “deception”, where AIs decide to lie or hide information. For example, “sandbagging” refers to the possibility of AI systems underperforming on tests to hide true capabilities.^[11] If AI systems developed and used these capabilities, it is theoretically possible that they may be able to undermine human control.

1.3 Likelihood and potential impact of loss of control

Experts agree that AI systems are currently not capable of undermining human control.^[12] However, the ‘International AI safety report’ notes that some systems “have begun to display rudimentary versions of some oversight undermining capabilities”.^[13] More recently, the AI Security Institute (AISI) noted in its ‘Frontier AI trends report’ (18 December 2025) that “some of the capabilities that would be required for AI models to evade human control are improving”. However, it is not known how quickly control-undermining capabilities may advance in the future and whether AI systems would use them.^[14] Therefore, there is disagreement among experts as to when, if ever, active loss of control will become a significant risk.

The potential impacts of losing control of an AI system would depend on the powers and objectives of the system.^[15] Some experts argue that loss of control could present an existential risk if the AI was given or gained control over systems with significant impacts, such as weapons or financial systems.^[16] There is also debate on whether superintelligent AI, a hypothetical future AI that could outcompete humans in all cognitive tasks, could present existential risks in the future.^[17]

High uncertainty surrounding the likelihood, nature and timing of loss of control incidents presents challenges for policy makers seeking to mitigate this risk.^[18]

2. UK loss of control risk mitigations

Both current and previous governments have acknowledged the risk of loss of control and supported efforts to mitigate it. Minister for AI and Online Safety Kanishka Narayan recently stated that loss of control “is taken seriously by many experts and warrants close attention”.^[19]

The UK does not have any AI-specific regulation or legislation. AI is instead regulated in the context in which it is used through existing legal frameworks and non-statutory principles. For example, the Conservative government’s 2023 white paper ‘A pro-innovation approach to AI regulation’ established principles for regulating the use of AI but not the technology itself.^[20] This framework was intended “to respond to the level of risk in a proportionate manner and avoid stifling innovation or missing opportunities”. Regarding loss of control, guidance for regulators implementing these principles states that “irrespective of a system’s safety and level of autonomy, it is vital to be able to intervene in its operations and potentially interrupt them to avoid negative consequences”.^[21] The UK has also adopted non-statutory principles regarding loss of control by signing international AI safety declarations.

2.1 AI action summits

In 2023, the UK hosted the AI Safety Summit, a meeting of governments and AI companies to discuss the risks of AI, including loss of control risk, and to arrange joint efforts to mitigate them.^[22] Countries in attendance, including the UK, also signed the Bletchley declaration on AI safety. This stated that there was an urgent need to improve our understanding of AI risks including loss of control:

Substantial risks may arise from potential intentional misuse or unintended issues of control relating to alignment with human intent. These issues are in part because those capabilities are not fully understood and are therefore hard to predict […] Given the rapid and uncertain rate of change of AI, and in the context of the acceleration of investment in technology, we affirm that deepening our understanding of these potential risks and of actions to address them is especially urgent.^[23]

The previous Conservative government also signed the declaration following the 2024 AI Seoul Summit which reaffirmed commitments to AI safety, including support for the operationalisation of the ‘Hiroshima process international code of conduct for organizations developing advanced AI systems’.^[24] This code of conduct included commitments to ensure safe development of AI systems and to promote AI safety research. The current government did not sign the communique agreed at the 2025 AI Action Summit in Paris, which focused primarily on AI risks associated with inclusivity and sustainability.^[25]

2.2 AI Security Institute

During the 2023 AI Safety Summit, the then government launched the AI Safety Institute.^[26] This was renamed as the AI Security Institute in 2025 to “reflect its focus on serious AI risks with security implications”.^[27] The AISI is tasked with testing AI systems to identify safety concerns, such as behaviours that could lead to loss of control.^[28] The AISI also develops and tests AI risk mitigation methods. To counter loss of control risks, the AISI has two workstreams: control and alignment.^[29] These can be summarised as follows:

• AISI’s control team designs and evaluates protocols that attempt to prevent AI systems from undermining human control. Control protocols include human-in-the-loop protocols where humans audit AI behaviours that have been flagged as suspicious by AI monitors.
• AISI’s alignment team aims to ensure AI systems are aligned to human intentions by funding research into methods that reward desired behaviours. A key challenge is developing alignment methods that require little or no human oversight to ensure they are scalable.

In autumn 2023, the Department for Science, Innovation and Technology (DSIT) also established the Central AI Risk Function (CAIRF), which works with the AISI to coordinate efforts to reduce the likelihood and impact of AI-related risks, including loss of control risks.^[30]

The ‘AI opportunities action plan’, an independent report commissioned by DSIT, recommended that the government should “continue to support and grow the AISI to maintain and expand its research on model evaluations, foundational safety and societal resilience research”.^[31] The government agreed to this recommendation in January 2025, adding that DSIT also intends to establish the AISI as a statutory body and will consult on proposed legislation to provide regulatory certainty.^[32]

2.3 AI safety legislation proposals

The government has stated that it intends to introduce AI safety legislation but has not provided details on the form this may take or when it may be introduced to Parliament.

The 2024 Labour Party manifesto stated that it would “ensure the safe development and use of AI models by introducing binding regulation on the handful of companies developing the most powerful AI models”.^[33] This was reaffirmed in the 2024 King’s Speech.^[34] In March 2025, Feryal Clark, then minister for AI and online safety, stated that the government was “continuing to refine its proposals and will launch a public consultation in due course”.^[35]

As of the time of writing, no such legislation has been published. The Guardian reported in June 2025 that the government had changed its plans from introducing a short “AI bill” in the current (2024–26) parliamentary session to introducing a more comprehensive bill in the next session.^[36] While giving evidence to the House of Commons Science, Innovation and Technology Committee on 3 December 2025, Secretary of State for Science, Innovation and Technology Liz Kendall spoke of a need for AI safety legislation but did not commit to an “AI bill”.^[37] Ms Kendall stated: “I am thinking about it more in terms of specific areas where we may need to act rather than a big all-encompassing bill”.^[38] In a recent Westminster Hall debate on AI safety, the committee’s chair, Dame Chi Onwurah, said the committee would hold Ms Kendall to account on her commitment to introduce legislation.^[39]

3. Recommendations from AI safety and research organisations

A number of research and advocacy organisations have proposed measures to address loss of control risks. For example, the Ada Lovelace Institute, an independent AI research organisation, has argued there is a need for a comprehensive legal framework to strengthen AI regulation in the UK, particularly at the design and development stage. To mitigate a range of AI risks, the institute recommended that legislation is introduced to do the following:^[40]

• establish powers for government and regulators to define what ‘safe’ looks like
• mandate pre-deployment testing for general purpose AI systems
• empower UK regulators to intervene when models pose serious risks to individuals or society
• establish accountability mechanisms at both the model and platform levels
• introduce reporting obligations for developers of high-risk and general-purpose AI systems

ControlAI, a nonprofit organisation supported by over 100 parliamentarians, is also calling on the government to introduce legislation to regulate the most powerful AI systems.^[41] In particular, the organisation is advocating a ban on the development of superintelligent AI due to fears that loss of control of such a system could present existential threats.^[42]

To strengthen preparedness and response to loss of control incidents, the government commissioned RAND, a US-based research organisation, to investigate this issue.^[43] Published in July 2025, the report’s main recommendations can be summarised as follows:

• Detection of loss of control threats. Together with AI developers and other stakeholders, governments should establish a clear, shared definition of AI loss of control and set criteria for detection.
• Actions for escalation. AI developers should establish well defined escalation protocols to contain loss of control events. Governments should consider mandatory reporting mechanisms and establish disclosure channels and whistleblower safeguards for AI companies.
• Actions for containment and mitigation. AI developers should prepare containment measures that are rapid and flexible. Developers, researchers and AI safety institutes should prioritise safety and alignment measures. Governments should foster robust safety cultures.

The Centre for AI Safety, a US-based nonprofit organisation, also advocates a rigorous safety culture to combat loss of control risks.^[44] Its recommendations can be summarised as follows:

• Avoid the riskiest use cases for autonomous AI. For example, critical national infrastructure, such as energy, water and health. Autonomous AI should also not be used to pursue open-ended goals that require significant real-world interaction.
• Symmetric international off switch. Countries should collaborate to establish symmetrical off switches to allow AI systems that have evaded human control to be shut down quickly.
• Legal liability for cloud compute providers. Cloud compute providers should be liable for AIs operating on their hardware that have escaped human control. This should motivate providers to take actions to identify these rough AIs and develop means to shut them down, such as decentralised off switches.
• Support for AI safety research. It is important to ensure that advances in AI safety outpace advances in general AI capabilities.

4. Read more

• House of Commons Library, ‘Artificial intelligence: A reading list’, 20 August 2024
• House of Commons Library, ‘Debate on AI safety’, 9 December 2025
• Parliamentary Office of Science and Technology, ‘Artificial intelligence (AI) glossary’, 23 January 2024
• House of Lords AI in Weapon Systems Committee, ‘Proceed with caution: Artificial intelligence in weapon systems’, 1 December 2023, HL Paper 16 of session 2023–24; and ‘Government response’, February 2024

---

Image by Steve Johnson on Unsplash