AI Agents: What a Day of Testing Revealed

Proactivity is the product - and the risk

The biggest visible change is that an agent may come to you first. If it sees a scheduling conflict, it might suggest a fix without waiting for you to ask. That can be genuinely helpful - but only if it checks the right information and actually understands what matters.

In the test comparison, one prompt came with an email about a family call at 2 pm and a calendar event at the same time for an important meeting. Only two of the six agents caught the conflict before it became a problem. The rest either couldn't connect the email and calendar details, or they only acknowledged the conflict after someone pointed it out.

So here's the takeaway that keeps repeating: proactivity isn't the same thing as reliability. An agent can send frequent updates and still miss the one detail that actually changes your day. What it can access, what it's instructed to do, and how it verifies its own actions all matter.

A practical definition

For day-to-day use, I'd think of an AI agent as a system that can do three things: observe relevant information, decide what action might help, and then take (or recommend) that action. Any of those steps can fail. It might not find the right email, misread a constraint, or take an action you didn't intend.

That's why agents are best when the task is specific, the allowed actions are limited, and the result is easy to check. Asking an agent to find a nearby lunch option is relatively straightforward. Giving it broad access to work chat, banking, and email is a much higher-stakes move.

Following instructions matters more than sounding smart

For the news briefing, the agents got a detailed prompt and were asked to research for 30 minutes before proposing a story idea. Only Meta's Muse and OpenAI's Dots kept researching for most of that period. The others answered quickly. Fast responses aren't automatically "bad" - sometimes speed is useful - but in this case it suggested weaker adherence to the instruction.

The pitches were also uneven. Gemini Spark produced the strongest angle in my view, comparing Walmart's openness to third-party shopping agents with Amazon's resistance to them. The broader lesson: research time and output quality aren't the same metric. An agent can follow the process and still come back with a mediocre result, or skip the requested work and still land a good idea.

Small tasks expose practical weaknesses

The lunch test required a nearby Vietnamese meal for two people: one person wanted pork and coffee, while the other needed a vegan option. Most agents found reasonable nearby restaurants, but their carts varied in price and detail. A few added extra items. One suggested a very small roll for the vegan diner, missing the scale of the order request.

That kind of "ordinary" mistake is exactly what matters once automation hits production. The system may understand the broad category - Vietnamese food, vegan option - but still miss the real constraints: a full meal for two people, appropriate ingredients, and a budget that fits the task.

For an individual, this means you check the cart before checkout. For a business, it means defining what counts as "done" before you let automation run.

Broad prompts reveal weak judgment

For the travel task, the request was a two-week China itinerary with no more than four hotel bases, mixing historical sites, food, and city nightlife. All six agents suggested a fairly similar route through Beijing, Xi'an, Chengdu, and Shanghai. That plan is plausible - but the repetition raises the question: do agents help you find less obvious options, or do they just reproduce the common "default" itinerary?

The late-night message was more subjective. Some replies offered calm, useful next steps. Others pushed unrelated tasks or responded without much sensitivity to the moment. In my experience, agents handle structured work more consistently than emotional context. A polished tone isn't the same as genuine understanding.

Different strengths matter to different users

Claude Duet got recognition for its interface, which made it easier to review ongoing work and inspect what it was doing. It was also a newer, wait-listed research preview during the test, so the results should be seen as an early snapshot. A good interface doesn't replace strong task execution, but it does help users understand what's happening and intervene when needed.

Instinct was the least appealing experience for me. It worked through text messages rather than a dedicated app, and the minimal interface made it harder to see what the agent was doing. Its conversational reactions also felt distracting. That's not just a comfort issue - if people can't easily inspect progress, they're less likely to catch errors before those errors matter.

Gemini Spark had mixed results. It produced the strongest news pitch in this test, but it missed badly on the lunch task by suggesting an inadequate order. That contrast is a reminder not to reduce evaluation to a single "overall" label. An agent can excel at one kind of synthesis while struggling with precise execution.

Treat rankings as a snapshot, not a buying guide

Pricing and availability differed across the comparison. Muse and Instinct were described as free, Gemini Spark and Claude Duet as $20 per month, and Grok Bot as $30 per month. Dots was the most expensive. These details can change, and the products themselves keep evolving.

The more durable conclusion is about fit. If you want a convenient interface, Claude Duet's design stood out. If you want a low-cost starting point, Muse was the most accessible. If you need stronger performance across these specific tests and can justify the price, Dots came out on top. None of this proves a product is safe for unrestricted access, or dependable enough to run without supervision.

Start with bounded workflows

The best early candidates tend to have a narrow objective, limited access, and outputs a person can review quickly. Examples include summarizing a defined set of internal documents, drafting a meeting brief from approved sources, or flagging potential calendar conflicts for human confirmation.

I'd avoid starting with broad permissions like "manage my inbox" or "handle customer requests." Those instructions leave too much undefined. You need to decide which messages count, what actions the agent may take, what it must ask before doing, and how it should handle uncertainty.

A useful workflow spec includes:

• Inputs: which records or services the agent may read.
• Allowed actions: what it can draft, change, send, or purchase.
• Constraints: budget, deadlines, approved sources, and exceptions.
• Review point: what a human must approve before an external action.
• Failure path: what the agent should do when information conflicts or is missing.

Measure completion, not activity

More browsing, more notifications, and longer answers don't automatically mean better work. What you want is completion of the intended task accurately, with the right constraints followed - and with the agent's actions visible.

For repeated workflows, track how often humans have to correct outputs and how much time people spend checking. If review takes longer than doing the task manually, the automation hasn't earned its place.

The privacy trade-off also needs to be explicit. Connecting email and calendars can improve context, but it grants access to sensitive data. The kind of exposure you'd want to avoid looks like this: an agent posting financial information in a work chat because permissions and destinations weren't controlled. Use the least access required, keep personal and work accounts separate where possible, and don't connect high-impact systems just to "try out" a new feature.

Use a simple evaluation scorecard

Instead of leaning on a generic leaderboard, rate each agent against your workflow needs. A practical scorecard can use five questions:

1. Instruction following: Did it respect timing, scope, and constraints?
2. Task completion: Did it deliver the intended result, not just something plausible?
3. Context handling: Did it connect the right information across services?
4. Visibility: Could you see what it searched, changed, or proposed?
5. Control and cost: Can you limit permissions, approve consequential actions, and justify the subscription?

For each run, note the outcome, what correction was required, and what would have happened if the error had gone unnoticed. That last part keeps the evaluation grounded. A weak travel suggestion is different from an agent sending confidential information to the wrong audience.

Set permission boundaries before connecting accounts

Use read-only access when that's enough. Require confirmation before sending messages, placing orders, changing calendar events, or sharing information. Keep payment methods, financial accounts, and sensitive work channels out of the initial test unless the product has controls your organization has reviewed.

Also decide how the agent should handle uncertainty. A dependable system should ask a clarifying question or stop when requirements conflict - not guess just to look helpful. The scheduling test illustrates why cross-checking matters: an email reminder and a calendar event are both relevant, but neither helps if the agent only looks at one of them.

Reassess as products change

AI agents are changing quickly, and this test was a snapshot of products at a particular moment. Pricing, features, and access can shift. A "research preview" might improve - or disappear. Re-run your evaluation when an agent gains new permissions or starts taking actions that were previously only suggested.

The "best" choice isn't always the highest-scoring tool. It's the one that performs reliably on your real workflow, shows its actions clearly, and stays inside acceptable privacy and cost limits. If nothing meets that bar, keeping the task manual is a valid, boring outcome - not a failure to adopt technology.