AI and the ‘Spectrum of Autonomy’

Like other sectors of the economy, financial services firms are adopting AI tools and systems at a staggering rate. In one sense, this is unsurprising, given the potential opportunities offered by AI’s ever-increasing capabilities. However, it presents a clear challenge for governments and regulators globally, namely, are the existing legal and regulatory frameworks fit for purpose in an increasingly AI-enabled world?

In this second article in our series on the FCA’s Mills Review, Nigel Sydenham, Director of Compliance Training, unpacks the significance of the ‘spectrum of autonomy’, a key concept within the review, and one that has direct implications for how firms evidence robust governance frameworks.

Read the first article in the series here.

The FCA’s current approach to AI

The FCA’s current approach is to argue that the various components of its regulatory regime (such as the Principles for Businesses, SMR and Consumer Duty) are sufficient to manage the risks arising from increasing adoption of AI. Indeed, one of the key arguments for principles or outcomes-focused approaches to regulation has been that they are better able to adjust to changing contexts and technologies than more prescriptive, rules-based, approaches. However, the rapid development and deployment of AI is testing that argument, particularly as AI systems become increasingly autonomous.    

The transformative power of AI

The Mills Review is clear about the potential for AI to change the face of financial services, with corresponding implications for the required regulatory framework:

“AI will transform financial services markets by 2030 and beyond”

Central to its approach is the 'spectrum of autonomy', a framework for understanding the transition from AI that a human employee primarily uses as a tool, to AI systems that are increasingly able to act with minimal (or no) human intervention.

What is the AI 'spectrum of autonomy'?

The spectrum, as set out in the review, has five stages, with each stage representing increased autonomy for the AI agent. The stages are defined in terms of the changing role of the human. Hence the stages describe the human as:

1. Operator – the AI supports on demand
2. Collaborator – the human and AI plan and act together
3. Consultant – the AI leads, the human gives guidance
4. Approver – the AI acts, the human signs off key steps
5. Observer – the AI executes, the human monitors

The spectrum is not a prediction that all AI tools or systems will reach Stage 5, or that the use of AI within firms will inevitably, and uniformly, move through each stage. Rather, it seems likely that some use cases will move to higher stages, as technological developments allow, while others will continue to operate at lower levels.

The main benefit of the spectrum as an analytical tool is that it encourages us to reflect on the impact of AI’s increasing capability to act as an autonomous agent:

“The spectrum should be understood through the changing role of the human: as AI systems move along this spectrum, people’s roles change, from performing and directing individual tasks towards setting boundaries, granting permissions and overseeing and observing outcomes.” 

The spectrum of autonomy is not the only way to understand how AI might develop, but assuming we accept it as a useful analytical tool, what are its practical implications for both firms and regulators?

The accountability challenge for firms

If, as we might expect, we do see a transition towards increasing AI autonomy, it will pose significant issues for firms. Effective governance frameworks are built on oversight and accountability, with senior management being held ultimately accountable for the actions taken within their areas of responsibility.

Consider a firm in which AI systems are operating at the higher end of the spectrum. If employees are acting as ‘approvers’ of AI actions (Stage 4), senior managers could presumably have oversight of these approval decisions. However, it’s not clear that the same is true where employees are solely monitoring actions which are being executed by an autonomous AI (Stage 5). Will a senior manager be in a position to maintain oversight of an AI agent, in the same way that they oversee the activities of human employees?

To put it bluntly – who is accountable when autonomous AI gets things wrong?

The first step for firms

What should firms be doing in response to this challenge to oversight and governance?

One response may be that, since current AI use cases are typically lower down the autonomy spectrum, the issue of accountability for highly autonomous AI is tomorrow’s problem. However, the reality is that most firms will already be using AI in some form – their employees certainly will, even if the firm has not officially approved or sanctioned its use.

The starting point for any firm is, therefore, to understand its existing use of AI, and how this use fits within the firm’s current governance framework. To take a simple example, if an AI tool is being used in a front-office function, is its use (or misuse) the responsibility of the Head of Information Technology (or equivalent) or of the Head of the business unit in question? If a firm is unable to answer this question, it already has an accountability gap in its governance framework, even if it remains at Stage 1 of the autonomy spectrum.

The evolving regulatory response

Of course, the question of accountability for AI is not only one for firms – it also poses a serious challenge to the regulatory framework.

To take one example, the SMR was implemented in the aftermath of the global financial crisis of 2007-08, in order to enhance the accountability of senior managers. Prior to this, regulators had found it difficult to hold individual senior managers personally responsible for misconduct or other regulatory breaches within their areas of responsibility. The SMR has fundamentally shifted expectations around personal accountability, but the rise of autonomous AI will undoubtedly test the limits of the regime.

The review argues that the FCA’s existing approach continues to be appropriate:

“We find that the overall regulatory framework remains sound. Its principles and outcomes-based approach, including the Consumer Duty, Senior Managers Regime (SMR), operational resilience and other key features, was designed to flex across changing business models.”

However, the review goes on to observe the understandable desire of firms for greater clarity regarding the practical application of the existing framework within the context of AI:

“Respondents [to the review’s consultation process] did not seek changes to this system. They wanted clarity on how to interpret and govern increasing use of AI within the existing regime.”

This desire for greater clarity from the regulator is aligned with the call by the UK Treasury Committee for the FCA to publish further practical guidance for firms by the end of 2026.

While most firms will welcome such guidance, it seems clear that, at least in the short term, the challenge of maintaining accountability for increasingly autonomous AI tools will not be resolved by the implementation of new rules. Rather, it will be shaped by the way existing obligations are interpreted, applied and evidenced.

Upcoming webinar

If you found this article useful, join us for our upcoming webinar, where we'll explore the current state of play with AI in financial services, the principal risks, and how the regulatory response is evolving.

Date: Tuesday 15 September
Time: 2:00 – 2:45 pm BST
Delivery: Virtual, Zoom
Presenters: Nigel Sydenham and Michael Sacks

Nigel specialises in training boards, senior executives and other staff on the impact of regulation and regulatory change.

He is a CFA Charterholder and Chartered Fellow of the CISI, with over 20 years' of industry experience.

With a background in compliance in private banking and wealth management, Nigel has a particular interest in effective corporate governance and the management of compliance and regulatory risk. His interests also include issues relating to ESG and climate risk, conduct and culture (including non-financial misconduct), and all aspects of financial crime prevention, as well as the impact of fintech on compliance and regulation.

Recent assignments have included briefing multiple boards and executive teams on the Consumer Duty, delivering compliance and ethics training for senior managers and front-office staff and creating a user-friendly risk and compliance handbook for a major bank.