Designing human-agent responsibility, not just AI interfaces

The first generation of generative-AI products made conversation the dominant metaphor.

A box. A prompt. A response.

That interface was useful because it gave people an intuitive way to interact with an unfamiliar capability.

But as AI systems begin doing more than generating responses, the chat box becomes a very small part of the design problem.

An agent may retrieve information, make plans, call tools, update systems, communicate with other agents, recommend decisions, or take actions in the world.

At that point the central design question changes.

It is no longer:

"What should the AI interface look like?"

It becomes:

"Who is responsible for what?"

Capability is not the same as authority

An AI system may be capable of taking an action.

That does not mean it should be authorized to take it.

Consider a system that can:

draft an email, send the email, issue a refund, change a customer record, recommend a financial decision, approve an application, schedule an employee, publish content, or modify production data.

The technical question is whether the system can perform the action.

The Product question is whether it should.

That depends on dimensions such as:

risk, reversibility, confidence, customer expectation, regulation, financial consequence, privacy, and the cost of being wrong.

A useful agentic experience therefore needs an explicit responsibility model.

Some actions may be automated.

Some may be proposed.

Some may require approval.

Some may be prohibited.

And those boundaries may change with context.

"Human in the loop" is not a design

It is tempting to solve uncertainty by putting a human approval step everywhere.

That feels safe.

It can also destroy the value of automation.

If a person must approve every low-risk action, the human becomes the agent's mechanical appendage.

The system technically automates the workflow while functionally creating another inbox.

The better question is:

Where does human judgment create enough value to justify intervention?

A responsibility model might consider two variables:

Low-consequence, easily reversible actions may support greater autonomy.

High-consequence, difficult-to-reverse actions may require explicit human authority.

But even that is not enough.

Confidence matters.

Context matters.

Policy matters.

The user's own preferences may matter.

The design problem is therefore not simply inserting approval gates.

It is designing an intelligible system of authority.

Visibility becomes part of the experience

Traditional interfaces often make system state visible through screens.

Agentic systems may operate across multiple tools and over longer periods of time.

The user may not watch every action happen.

That creates a new experience requirement: visibility without constant supervision.

A person should be able to understand:

what the agent is doing, why it is doing it, what information it used, what it has already changed, what it plans to do next, where it is uncertain, and when it needs help.

This does not mean exposing chain-of-thought or flooding the user with technical detail.

It means designing useful operational transparency.

The right level of visibility depends on the task.

A low-risk personal assistant may need little explanation.

A system acting on customer accounts, money, regulated information, or business operations may need substantially more.

Trust is contextual.

The interface should reflect that.

Failure is part of the product

AI Product teams naturally spend a great deal of energy on the successful path.

But probabilistic systems make the unsuccessful path unusually important.

The system may misunderstand the request.

The retrieved information may be incomplete.

A tool may fail.

Permissions may block an action.

Two sources may conflict.

The model may be uncertain without recognizing that uncertainty.

The world may change between planning and execution.

A mature AI product therefore needs a designed failure model.

These are experience decisions as much as technical ones.

Evaluation belongs in Product

When software was largely deterministic, Product teams could often rely on functional testing plus customer metrics.

AI introduces another layer.

The system can technically work while producing inconsistent quality.

That means Product leaders need to become comfortable with evaluation as a product discipline.

Not necessarily building every evaluation system themselves, but understanding:

what "good" means, which failures matter, what representative test cases look like, how quality changes over time, which dimensions trade off against each other, and what production behavior tells us that offline testing missed.

An evaluation framework might combine:

task success, accuracy or groundedness, policy adherence, human review, latency, cost, customer outcome, and business outcome.

The correct mix depends on the product.

The important shift is conceptual:

Model quality is not merely an Engineering concern.

It is part of the customer experience.

The organization is also part of the loop

Responsibility does not end with the user and the agent.

Enterprise AI systems operate inside organizations.

Someone owns the policy.

Someone owns the data.

Someone decides which tools the agent can access.

Someone monitors performance.

Someone handles exceptions.

Someone changes the workflow when the system creates new possibilities.

That means human-agent design eventually becomes organizational design.

The Product leader needs to understand not only:

What does the user do?

and:

What does the agent do?

but also:

What does the organization now need to do differently?

This is especially important when AI moves from assistance to action.

A copilot can often fit inside an existing workflow.

An agent may change the workflow itself.

Design the responsibility map first

Before designing the detailed interface for an agentic product, I find a more useful starting point is to map responsibility.

For each meaningful step:

Only then does the interface problem become clear.

Sometimes the answer is conversation.

Sometimes it is a dashboard.

Sometimes it is an approval queue.

Sometimes it is a notification.

Sometimes the best interface is almost invisible because the system reliably handles low-risk work in the background.

The experience should emerge from the responsibility model, not the other way around.

The next generation of AI design

The most important AI experiences may not be distinguished by more impressive chat interfaces.

They may be distinguished by better decisions about autonomy.

Better visibility.

Better recovery.

Better evaluation.

Better boundaries.

And better understanding of where human judgment remains essential.

As AI systems become capable of doing more, Product and Design leaders have a new responsibility of their own:

not merely deciding what the machine can do,

but deciding what the machine should be allowed to do—and designing the system around that answer.