Prompting, Fine-Tuning and Rules in High-Stakes Workflows

Before approving an AI implementation for consequential work, separate two decisions: whether its output is acceptable and whether it has permission to act. Specify what the system may propose, what it may execute and who owns an incorrect outcome. Assess prompting, fine-tuning and rules engines within that boundary.

The available evidence distinguishes properties worth evaluating, but establishes no universal winner. The recommendations below are architecture-review questions, not validated thresholds for selecting a mechanism.

Define the workflow boundary first

Start with the consequential action and work backwards through workflow states, permitted transitions, approvals and exception paths. Name the owner of each decision, including the decision to stop processing when a case cannot be resolved safely.

Document which conditions must block execution and which outputs require human review. Carry those requirements into each candidate's evaluation. A change of model or implementation should require a separate approval if it also changes the system's authority.

Compare system properties

The UK Department for Education's guidance on automated marking distinguishes deterministic rule-based engines, LLMs fine-tuned on human-marked responses and LLMs prompted without task-specific training. It identifies differences in explainability, consistency, training-data reliance and susceptibility to bias. That is a useful evaluation lens, bounded to marking rather than a general ranking of production architectures.

Apply the same scrutiny across candidates: what explanation can a reviewer inspect, how consistent are repeated decisions, which data dependencies need ownership, and how are errors distributed across relevant case groups? Do not treat the absence of task-specific training as an exemption from reviewing data dependence.

Then examine changeability. Ask each team what a policy revision would require: edits to logic, revised instructions, new task-specific training or a combination. Include validation, approval and rollback in that assessment. Make the work of maintaining the implementation part of the decision to adopt it.

Keep explicit policy traceable to logic

The MIT Computational Law Report describes rules as code as converting legal rules into programming-language code so computers can analyse and implement them. This establishes a bounded role for explicit logic, not a preference for rules engines across all policy-driven or non-legal workflows.

In architecture review, distinguish an agreed rule from an unresolved interpretation. For a codifiable requirement, document its authority, intended meaning, required inputs and exception path. If interpretation remains contested, assign a decision owner before approving the implementation.

As a governance measure, require traceability from the approved rule to the logic and its tests. Keep policy approval separate from code approval, with named owners for changes to each. Preserve deterministic controls where explicit constraints are required, while keeping disputed interpretations and exceptions subject to accountable judgment.

Challenge historical fit and the quality measure

A 2019 University of Chicago Law School legal-theory working paper argues that machine learning is less effective when past patterns do not resemble future conditions, including when new variables appear. It also argues for less trust where overfitting, Simpson's paradox or omitted variables make measurement difficult. These are conditional analytical arguments in legal applications, not production benchmarks or operational cutoffs.

Bring two questions to deployment review: does the evaluation represent the cases the system will face, and does the quality measure capture the errors the organisation cannot accept? Examine changes in policy, inputs and case mix. Assess consequential errors separately rather than approving on an aggregate score alone.

Agree which changes should trigger re-evaluation, restricted authority or manual handling. Where evaluation cannot answer a material risk question, record the uncertainty and limit deployment accordingly. Leave room to withdraw an earlier approval.

Make human control operational

The same 2019 paper argues that a human-AI combination can be superior to AI alone where measurement is difficult. It does not establish that adding a reviewer improves every system. Treat human review as part of the workflow to evaluate, rather than sufficient evidence that the workflow is controlled.

• — For production approval, require explicit answers:
• — Authority: which outputs may prepare, classify or flag work, and which require approval before action?
• — Review: what inputs, applicable rules and system outputs can the reviewer inspect?
• — Escalation: who resolves ambiguous cases, overrides and disagreements?
• — Auditability: what record connects the output, implementation version, reviewer decision and resulting action?
• — Reversibility: who can suspend the mechanism, and how will unresolved work be handled?

Test the effort required to inspect an output, resolve an exception and recover from an incorrect action. Give the accountable owner authority to intervene, and verify that the intervention path works before production approval. Include that review and recovery effort in the operating-cost assessment.

Validate locally before setting a default

The research considered here offers no shared cost, latency, reliability or maintenance comparison that settles the choice. It establishes neither a production-scale threshold for moving from prompting to fine-tuning nor a build-versus-API rule.

Compare viable candidates against the same workflow requirements and relevant cases. Set acceptance criteria with the accountable operational owner, covering reliability, consistency, error consequences, review effort, auditability, integration constraints and operating cost. Include the work required to change or remove the mechanism, and specify how unresolved exceptions will be handled.

Approve a bounded use with stated conditions: what it may do, which evidence supports that authority, what requires another review and how to withdraw it. Make that approval discipline the standard before making any mechanism the default.

Written by the studio, from work in delivery