By Richard Batstone

Data or decision? A practical test for supervising AI

Almost all firms now have access to general-purpose AI tools, and have policies around their use. Far fewer, however, have solved the challenge of how to supervise the work AI produces at an operational level. How, on a live matter with a deadline looming, does a supervising lawyer actually check AI work product? This gap between policy and practice is where risk builds, both for firms and individuals.

Data or decision? A practical test for supervising AI

The profession has noticed. The Law Society's latest practice note makes clear that professional responsibility doesn't pass to the machine, and SuperTech WM's new AI Adoption & Evaluation Toolkit explicitly assesses whether a tool builds a lawyer's oversight capability, or erodes it. For those of us developing legal tech, the message is just as clear: if lawyers can't supervise a tool, the tool isn't suitable.

In this article, we share a simple test that firms can use to decide where AI belongs in a workflow and where a lawyer’s decision needs to stay explicit: is this step data, or a decision? We show how that question shaped AI import for Clarilis questionnaires (including what we deliberately chose not to build) and how you can put it to work in your own firm.

From policy to practice

One of the conversations we’ve had most often this year is about how firms can implement AI governance in practice: writing the policy is one thing, but turning principles into outcomes is much harder.

The Law Society’s “Using artificial intelligence in legal practice” October practice note sets out the challenge clearly:

“The use of AI agents and agentic AI does not transfer or diminish professional responsibility from you or your organisation.

Appropriate human oversight becomes increasingly important with the use of more autonomous AI systems such as AI agents and agentic AI.

You remain responsible for verifying final outputs and ensuring compliance with your professional, regulatory and legal obligations.”

And, more pointedly:

“It will likely not be sufficient to instruct managers to supervise AI-assisted work without providing the time, training, tools and escalation routes needed to do so effectively.”

This second point echoes something we have heard from more than one senior lawyer recently: rather than attempting to supervise junior lawyers in the use of general-purpose AI tools, they simply tell them not to use AI at all. Or to put it another way, lawyers do not feel equipped to provide the level of supervision and oversight of AI enabled tools that their professional obligations demand.

A technology partner's role

As a builder of AI legal tech, that’s alarming to hear, and we think it puts the onus on legal tech to do better. Pushing the supervision requirement to the end user isn’t sustainable if the user doesn't also have mechanisms for identifying and quantifying risk. The tools themselves have to be designed with supervision in mind.

This was one of the core themes emerging from SuperTech WM’s LawTech Open Innovation Lab which, over the course of the year and with insight from law firms, academics and legal techs (Clarilis among them), developed the AI Adoption & Evaluation Toolkit. The toolkit evaluates technology against three KPI pillars:

  1. Efficiency & Productivity
  2. Quality, Accuracy & Risk
  3. Professional Development & Human Capital

Pillar 3 explicitly considers a given tool’s impact on junior lawyers’ learning and the extent to which senior lawyers must re-work outputs. As the toolkit cautions:

“[Pillar 3] is the leading indicator of whether an AI rollout would actually be beneficial. A tool that scores well on Pillars 1 and 2 but shows no Junior Learning Impact is producing output without building capability, which leaves firms dependent on the tool rather than developing its people.”(Emphasis added)

So, the role of a technology partner in meeting this challenge is clear and, in our experience, that is as much about where you choose not to use AI as where you do. Below, we explain how we drew that line in one of our own products, and then set out a simple test, drawn from that work, that firms can apply to their own use of AI.

Case study: data or decision?

Clarilis works by having users complete a dynamic questionnaire. The platform then applies rules, written by knowledge lawyers, to the underlying templates, building a risk-reviewed first draft document. The questionnaire is where the deal is described and is a natural place to use AI to reduce the manual effort needed to complete it. It is also, for the same reason, the place where it is most important to be careful.

When we designed our AI import tools (which let users upload unstructured material, such as a share capital table, and have the relevant details extracted into the questionnaire) we started from a simple observation: not every input to an automation is the same kind of thing, and they shouldn’t all be treated the same way.

This mirrors one of the LawTech Open Innovation Lab’s central findings that AI “should be viewed as a supervised productivity tool rather than an autonomous decision-maker”. Moving information from one place to another in an efficient way (e.g. without rekeying) is a productivity task. Choosing how a deal should be documented is a decision.

Inputs that are data

A company name, a registered address, a list of directors, the shareholdings on a cap table. These just need to be correct, and they usually already exist somewhere and just needs to be carried across/ transposed on Companies House, in a firm or client system, or in a document the team has already put together. Here, we make it as easy as possible to bring that information in from messy sources. Rather than retyping it, AI presents extracted data for the user to review, before importing it directly into the questionnaire. Transposing by hand is slow, and every re-keyed detail is another chance for an error to creep in.

Inputs that are deal terms or judgement

Is this facility secured? Should the tenant have a break right? Is completion conditional?

These answers shape the document and reflect a decision someone has taken. We are deliberately more cautious here, and we don’t use AI import to hide that complexity.

Why we don’t use AI to make every selection

It would be possible to have AI read a term sheet and pre-answer the whole questionnaire. It would demo well. We chose not to, for four reasons:

1. The fastest workflow isn’t always the one with the most AI

Many deal-term inputs are simple selections: a tick box, a dropdown, or a choice between clearly defined alternatives. Our product development work showed that, for these inputs, making the selection directly is often faster than asking AI to infer an answer from a document and then checking whether it got it right.

If it’s one click for the lawyer, it shouldn’t be a guess for AI.

2. An incorrect assumption can create more work than it saves

Questionnaire answers drive what comes next. If AI wrongly interprets a transaction as conditional, the user may be taken through an entire set of unnecessary questions. More importantly, an error that goes unnoticed could lead to inappropriate drafting, and one that is buried in a pre-populated questionnaire or worse still, an assumption or inference that is not presented back to the user is very easy to miss.

By keeping substantive selections explicit, Clarilis keeps users in control of the assumptions driving their documents.

An assumption you can’t see is a risk you can’t manage.

3. A term sheet isn’t a substitute for legal judgement

A term sheet rarely translates directly into a first draft. It may record provisional commercial intention, omit important legal positions, contain inconsistencies, or reflect negotiations that have since moved on. The lawyer’s role is to interpret that information, resolve the outstanding points and decide how the deal should be reflected in the documentation.

That is valuable professional work, not administrative overhead to be eliminated. It is also exactly what the toolkit warns against displacing: the question, as it puts it, is whether a tool is good enough “…with the supervision you can realistically give it - and without displacing your professional judgement or regulatory obligations”.

4. Good automation should strengthen supervision, not bypass it

One of the core strengths of document automation is supervised delegation. Less experienced lawyers can produce sophisticated documents from their firm’s risk-reviewed precedents, while supervisors retain visibility over the inputs and decisions behind the drafting.

That model works best when the person completing the questionnaire understands the choices being made and can spot the points that need further thought. Encouraging users to accept AI-inferred legal positions risks weakening that understanding, and with it the junior lawyer’s development.

The LawTech Open Innovation Lab found that juniors tend to rate fluent, confident output more highly than seniors do, because seniors catch the gaps beneath it. A fully pre-answered questionnaire would be that “fluency trap” in another form: it would look complete to the person least placed to question it.

By keeping substantive decisions visible and deliberate, we aim to support better delegation, more effective review and greater confidence in the resulting documents.

Applying the test in your firm

Each of these four considerations is really a question, and their application is broader than automation questionnaires. In fact, we think they are most useful in exactly the area the Law Society’s guidance is aimed at: the everyday use of AI tools, general-purpose and specialist, where supervision is hardest.

Before AI takes on any step in a workflow, ask:

  1. The value test. Allowing for the time it takes to check the output, is AI actually faster than a person doing this directly?
  2. The visibility test. If the AI got this wrong, would anyone / a busy lawyer notice? Can a supervisor see what was AI-produced, and tell a guess from a validated source?
  3. The judgement test. Is this where the professional value sits: interpreting instructions, resolving ambiguity, guiding the client? AI can help prepare that work, but it shouldn’t do it out of sight.
  4. The learning test. Is this a decision a junior lawyer should be learning to make? Taking it away might save time this year and cost capability in following years.

Data steps usually pass all four. These are the places to use AI with confidence, with checks built in where the information lands. Decision steps usually fail at least one. They should stay explicit, with any AI input clearly presented as a suggestion rather than an answer.

Governance, built in 

The Law Society is right that supervising AI-assisted work takes more than an instruction to managers. But that burden shouldn’t fall on firms alone. If senior lawyers are telling juniors not to use AI because they can’t supervise it, that is a design problem, and the people building the tools should treat it as one.

Asking “data or decision?” won’t resolve every governance question, but it does turn a broad principle into something a fee earner can apply to a task in front of them, and something a supervisor can check. A tool that is hard to supervise isn’t really efficient: the time comes back later as review, usually at a more senior level. A tool that is easy to supervise is one that lawyers will feel confident using.

That is what we mean by governance by design. AI should take the user and the platform further, without compromising on transparency or supervision. Sometimes that means building more. Sometimes it means deliberately building less.

Clarilis was pleased to take part in SuperTech WM’s LawTech Open Innovation Lab, and we’d encourage any firm working through its AI governance to download the AI Adoption & Evaluation Toolkit. If you’re working out where to draw the line between data and decisions in your own firm, we’d be glad to compare notes. To find out more about AI import, visit the Clarilis website or get in touch with the team.

 

Subscribe

Subscribe to blog

To stay up to date with all the latest news and insights, enter your email address.
(Privacy policy)

 

“The use of AI agents and agentic AI does not transfer or diminish professional responsibility from you or your organisation."

The Law Society’s “Using artificial intelligence in legal practice”

 

 

Writing the policy is one thing, but turning principles into outcomes is much harder. 

 

 

Pushing the supervision requirement to the end user isn’t sustainable if the user doesn't also have mechanisms for identifying and quantifying risk. The tools themselves have to be designed with supervision in mind.

 

 

The role of a technology partner is as much about where you choose not to use AI as where you do.

 

 

If senior lawyers are telling juniors not to use AI because they can’t supervise it, that is a design problem.

 

If it’s one click for the lawyer, it shouldn’t be a guess for AI.

 

An assumption you can’t see is a risk you can’t manage.

 

A term sheet isn’t a substitute for legal judgement.

 

 

AI should take the user and the platform further, without compromising on transparency or supervision.

 

AI Draft - notice generation

 

AI Draft Logo White INCLUDED

Clarilis AI Draft enables lawyers to use generative AI to draft definitions, clauses, schedules and additional documents not covered by the underlying template, so you get a more complete first draft, faster.

Every AI-generated clause is highlighted in your document so you know exactly what to review.