How to give an AI agent a task: what Anthropic's experiment showed

Anthropic handed Claude agents the books of 201 employees and sent them off to trade. The agents negotiated fine. Where they lost was earlier, before any deal: in the five-minute chat where the agent tried to learn what the person liked to read.

That chat accounts for 85% of the shortfall. The negotiation itself, 15%.

I read the write-up of the experiment (Anthropic calls it Project Swap, published September 24) and then went into our own database. We build an AI agent for small businesses, and our agent keeps running into the same wall: the things a person never told it. The numbers below come from both sources.

What Anthropic tested

The setup is simple. Each participant brought a book they were willing to give away and spent five minutes telling Claude about their taste. After that, people were out of the loop: the agents went to a trading floor and struck deals with each other. Some agents were told to be ruthless, others prosocial.

Results were scored against each participant's own ranked list. If everyone had received the best book actually available to them, the score would have been 0.89. It came out at 0.55: on average, a book from the middle of the list.

Anthropic then split the gap:

• the agent guessed which of two books a person would prefer in 61% of pairs. A coin would get 50%;
• 85% of the shortfall came from the agent misunderstanding preferences;
• 15% came from weak negotiation;
• "ruthless" agents beat "prosocial" ones by 0.02. Next to nothing;
• a stronger model helped a lot: Opus scored 0.88, Haiku 0.75, measured against Claude's own reading of each participant's taste.

One detail I like more than any of the numbers. Some participants never got a book at all: the trade went through, and nobody tracked the delivery. Someone else got a book they had already read. And one person complained that their agent bargained too meekly.

Afterwards, participants said they would trust an agent with about 30% of their yearly book budget. A trusted friend would get 40%.

How to give an AI agent a task when a brief is not enough

The usual advice in articles about prompts for AI agents goes like this: describe the task in detail, set a role, a format, constraints, give an example. The advice is fine. Anthropic's experiment shows where it stops working.

Participants had five minutes and a partner who asked questions. That beats most briefs people type into a work chat. And the agent still read their taste at 61% against 50% for random choice.

I don't think the problem is that people phrase things badly. Most of what matters, a person won't remember up front. A participant doesn't mention the books they've already read, because that's obvious. Nor that a trade should end with a parcel in their hands.

A business has more of these obvious things. You know a five-pack is sold as a single item. That "Alexander" in a deal comment is the sales rep, not the customer. That the warehouse in Dubna is excluded from purchase planning. The agent doesn't know any of it, and no prompt template will make you recall all of it before the work starts.

What our agent asks our users

Over the last 30 days (up to September 25) our agent stopped and asked a clarifying question 612 times, across 61 people. It's a separate card in the chat: a question, sometimes with options, and the work waits until the person answers.

I pulled random questions and stripped names and company details. Here's what comes up:

• "What does 'Sale <name> <name>' in the comment mean: who gets credit for the sale, and for what?"
• "What rates apply to diamond boring and patches, hot and cold? Flat per piece or a percentage of the total?" The agent was calculating pay for workshop staff, and those rates were in neither the rules nor past calculations.
• "There's no exact match in the catalog, and a substitute would change inventory records." Then two options: create the 50 ml syringe as a separate item, or post it against an existing product card.
• "Product titles contain quantities, 5 and 10. Should I multiply the price by the number of packs, or are all four sold individually?"
• "Please give a daily limit separately for Ozon and Yandex Direct." The agent found 36 products to advertise and would not launch campaigns with an unknown budget.

None of these questions are about how to do the work. The agent knows how to post a goods receipt in MoySklad and how to set up a campaign. It asks about what the person wants, which is exactly where Anthropic's agents lost.

475 of the 612 questions got an answer. Median time to answer: 66 seconds. In 56 more cases the person wrote something else instead of answering, and the agent followed the new request. The rest went unanswered: the person left and the task stalled.

Delegating to AI: what one minute is worth

66 seconds looks trivial. Now count what happens without the question. If the agent decides on its own that "5 and 10" in a title are just numbers, the prices of four products end up five or ten times too low. If it posts the syringe against the wrong card, stock drifts, and the next stocktake shows a shortage that doesn't exist.

So I disagree with the idea that a good agent is one that never asks. Such an agent either works on a very narrow task or guesses. The agents in Anthropic's experiment couldn't ask anything after the five-minute chat, and they guessed right 61% of the time.

There's a fair counterpoint. A question costs something if it didn't need asking. An agent that asks "which warehouse?" when there's only one annoys people faster than it makes mistakes. That's why whatever a person has answered once, the agent should remember: staff rates, bundle rules, ad limits. We covered how that kind of memory works separately.

Working with AI agents: approval is not understanding

We have a second kind of stop: the agent asks you to confirm an irreversible action. Posting documents, sending an order, launching ads. Over the same 30 days there were 439 of these cards, across 27 people.

Median time to approve: 9.6 seconds. Against 66 seconds for a clarifying question.

I don't know yet how to read that. Maybe approvals arrive when everything is already clear and the person just taps "yes". Or maybe people tap without reading. Anthropic's experiment points at the second as the risk: a book went to the wrong person because nobody checked. What the owner sees at an irreversible step and what stays in the log is in a separate piece.

What I'd do when handing a task to an agent

Keep the brief short. A long spec won't cover what you don't remember. Then check a few things.

First, whether the agent can stop and ask in the middle of the work, or whether it runs to the end on what it understood at the start. In Anthropic's experiment it couldn't, and that cost more than any weakness in negotiation.

Second, whether it remembers the answer. If you have to answer the question about rates every month, you'll stop answering and start writing specs again.

Third, whether you can see how things ended. A trade without delivery looked like a success in Anthropic's data. What to do when an agent gets no answer from a marketplace or a bank is covered here, and tasks that run overnight without you here.

A stronger model helps too, by the way: the gap between Opus and Haiku in the experiment (0.13) is six times bigger than between ruthless and prosocial instructions. In other words, "be more assertive" in a prompt buys almost nothing.

We build Samreshuuu: an AI agent that works in 1C, MoySklad, marketplace seller accounts and CRMs, asks when data is missing, and hands irreversible steps to a human for approval. The numbers about questions above come from its work.

Sources

• Anthropic, "Project Swap: What happens when agents trade for us?", 24.09.2026: 201 participants, six offices, a book exchange run by Claude agents.
• Samreshuuu data: the agent's clarifying questions and approvals for 26.08–25.09.2026. Questions anonymized.
• Yandex Wordstat, Russia, data as of 25.09.2026.
• Our articles: AI agent memory, what the owner sees, AI agent mistakes, an overnight task.