How Should Legal AI Collect Attorney Feedback? It is a UX Problem.

The Problem

A legal AI system can produce a useful legal work product, but difficult legal work still requires attorney judgment.

The question is therefore not simply whether an attorney should review the AI's work. The more interesting question is how that review should happen.

Today, attorneys are used to reviewing documents. A senior attorney may receive an initial draft, redline it, add comments, rewrite a paragraph, and send it back to a junior attorney. The junior attorney then interprets that feedback and revises the initial draft accordingly.

This process makes sense when a human attorney is doing the work. However, when an AI system is doing the work, the same feedback may be more difficult to interpret. A redline can show what changed, but it may not explain why the attorney made the change or what the AI should do differently next time.

There are other possibilities. The AI system can ask the attorney to provide structured feedback. It can also generate a set of suggested questions (e.g., semantically related questions) or actions based on the initial draft itself.

Each approach creates a different tradeoff between what is easy for the attorney and what is useful for the AI system. A good feedback collection interface needs to work for both.

The question is therefore a UX problem: how should we design the review experience for an AI system such that attorney feedback is easy to express and also meaningful to the AI system?

(UX, or user experience, is how a person feels and interacts when using a product, system, or service. It typically involves comprehensive user studies for understanding users' behaviors, preferences, and pain points.)

The Approach

Our current approach is to treat attorney feedback as part of the workflow rather than as something that happens only at the end.

There are several possible ways to do this.

Traditional redline and comments. An attorney can review the work product in a familiar document interface, make changes, and leave comments. This requires little change to existing legal practice and may work especially well when the work product is already close to a final document.

Structured feedback. An attorney can provide feedback through a separate interface or feedback file. The feedback can identify, in each section of the work product, what is wrong, what should change, and why. This makes the attorney's instructions more explicit and easier for the AI system to process.

AI-generated review questions. Instead of asking an attorney to start with a blank feedback box, the AI system can examine the work product and suggest possible questions or actions. For example:

• Reconsider the conclusion.
• Double check the citations.
• Analyze the opposing argument.
• Explain this reasoning.
• Narrow the analysis.
• Compare alternative interpretations.

These three example approaches do not have to be mutually exclusive. A simple approval may be enough for one work product. Another may require a detailed redline. Additionally, a difficult legal analysis may benefit from a few targeted questions before the attorney decides whether a full revision is necessary.

Our goal is not to force attorneys into one feedback format. Rather, it is to make the appropriate level of feedback easy to provide.

What We Are Investigating

We are investigating how different feedback interfaces affect both attorney experience and the AI workflow.

For example:

• When is a traditional redline better than structured feedback?
• When is a simple approve or reject action good enough?
• When should an AI system suggest questions or actions?
• What level of feedback should an attorney provide?
• How should freeform comments become actionable AI instructions?
• How should feedback be connected to a specific workflow stage?
• How should the AI system distinguish between a factual correction, a legal disagreement, and a change in strategy?
• How much should the AI system infer from an attorney's feedback?

Everything comes with a tradeoff.

For example, if an AI system suggests five questions for an attorney to consider, it may reduce the attorney's effort required to review the work product. Nevertheless, the attorney may choose from those five questions without realizing that there is actually a sixth problem the AI system did not suggest.

We are interested in that tradeoff: making attorney review easier without allowing the AI system to narrow the attorney's judgment.

Why It Matters

Attorney review is often described as keeping a "human in the loop." That description is useful, but incomplete.

The important question is not simply whether a human is present. It is whether an AI system gives the attorney a useful way to exercise judgment and whether that judgment can actually affect what the AI system does next.

Feedback Itself Is a Work Product

This changes the role of feedback.

In a traditional workflow, a redline is mainly a way for one person to communicate changes to another person. In a legal AI system, however, feedback can become part of the workflow itself. It can tell the AI system what went wrong, what should change, and what to do next.

This gives us an opportunity to design new forms of review. An AI system can present targeted questions, preserve structured feedback, connect comments to specific stages of the workflow, and ask for more or less attorney input depending on what is being reviewed.

The goal should not be to make attorneys adapt to the AI system. Instead, it should be to make attorney judgment easy to express, easy for the AI system to act on, and visible as part of the work that follows.

Conclusion

AI may change not only how legal work is produced, but also how attorneys interact with that work.

The traditional redline is one useful UX pattern, but it was designed primarily for communication between people. An AI system gives us other possibilities: structured feedback, targeted review questions, simple approval actions, and interfaces that adapt to the work being reviewed.

Each approach creates a different user experience. The challenge is to reduce the friction and cognitive load of attorney review without making the feedback so structured that it becomes burdensome or so AI-driven that it narrows the attorney judgment.

That is what we are investigating: how to design feedback interfaces that are usable for attorneys, meaningful for AI systems to act on, and flexible enough to preserve human judgment throughout the workflow.