AI Tax Benchmark: What DAYJOB Reveals About Agents
See how DAYJOB tests AI agents on messy tax records, professional judgment and complete workflows—and what its results say about readiness for real work.
AI Tax Benchmark
Can AI Handle Real Tax Work? What the DAYJOB Benchmark Tests
An AI agent can answer a tax question correctly and still leave a client exposed. Take a homeowner comparing two dates to sell a condo. The agent gets the usual residence test right for excluding up to $250,000 in gains. But it doesn't flag that waiting too long could make the homeowner ineligible. The answer is technically accurate - and still incomplete right where it counts.
That gap is the point of evaluating AI in professional services. Real tax work rarely shows up as a neat, fully-posed question with all the facts neatly laid out. You get scattered records, conflicting evidence, unclear instructions, and sometimes issues the client just doesn't think to raise. In other words, the agent has to figure out what matters, what it can actually support, and how to deliver something that holds up.
DAYJOB: Tax is a benchmark built to test that broader responsibility across 50 realistic tax environments. This article explains what it measures, how its assignments are structured, what the published results indicate, and why a benchmark score shouldn't be treated as readiness to handle tax work without oversight.
What does DAYJOB Tax evaluate?
DAYJOB: Tax checks whether AI agents can take responsibility for a professional tax assignment - from the initial request to a usable deliverable. The benchmark includes 50 environments across eight areas: individual and family tax, small-business tax, partnerships, property and investments, estates and trusts, international tax, corporate transactions, and tax review and disputes.
An agent receives a client request and a workspace of supporting records. Depending on the task, it may need to prepare a return, build workpapers, review a prior filing, investigate a transaction, or write advice about a decision the client hasn't made yet. And crucially, the task isn't always fully specified. A big part of the job is figuring out what the request leaves out.
So it's not just checking whether an agent can retrieve a rule or do a calculation. It tests whether the agent can:
- Find and reconcile relevant facts across multiple documents.
- Spot conflicts in the records or when important evidence is missing.
- Identify issues the client didn't ask about but that could change the result.
- Decide what needs further investigation versus what it can conclude.
- Carry the reasoning through into the requested filing, review, or recommendation.
That matters because professional work isn't just a sequence of isolated answers. A calculation can be correct while relying on an unsupported assumption. And a response can be accurate while still failing to flag a deadline, an eligibility problem, or a missing document that should change what the client does next.
DAYJOB grades attempts against task-specific criteria covering calculations, evidence, consequential issues, and the completed deliverable. The published pass rate uses pass@1 at a 95% rubric threshold: an attempt has to satisfy at least 95% of the criteria for that task to count as a pass. It's a demanding measure of end-to-end performance, not a general score for tax knowledge.
The benchmark's core question is practical: can an agent produce defensible tax work from incomplete, messy inputs - and clearly communicate the limits of what it concluded? That's a different bar than generating something plausible for a well-formed question.
What do the tax assignments look like?
The workspaces contain 1,789 files, with a median of 30 files per assignment and up to 123. Across the collection there are 1,487 PDFs, 117 Excel workbooks, and 38 Word documents, along with emails, images, CSVs, JSON records, and plain text. You'll see common tax documents mixed with records that are less structured and harder to interpret.
An assignment might include filed and draft returns, W-2s, K-1s, IRS notices, general ledgers, payroll records, contracts, and correspondence. The agent has to identify which pieces actually matter to the question, connect related records, and notice when the evidence trail doesn't fully justify a conclusion. That's closer to reviewing a client file than answering a tax quiz.
One example involves a handwritten receipt for merchandise purchased for resale. The agent has to read the item descriptions and amounts, compare the receipt with other spending records, and judge how the purchases affect inventory and cost of goods sold. The challenge isn't only extracting text from an image. It's understanding what the transaction represents and where it fits in the accounting record.
Another assignment is about an industrial silo-services company. The client assumes the business operates only in Alabama, but field dispatch records show crews working across five states. Locations, crew sizes, and days on site may matter for multistate tax obligations. If the agent just follows the client's stated assumption, it can miss an issue that's sitting in the evidence.
Some tasks require the agent to combine records across domains. In one cross-border household scenario, Polish employment and property documents have to be reconciled with cryptocurrency exchange exports, wallet histories, and historical purchase lots. The test here is whether the agent can connect records that don't arrive in a single, standardized "tax packet."
Assignments also differ in what they ask the agent to produce. Some require a complete filing package with supporting workpapers. Others ask for a transaction investigation, a review of an earlier return, or advice on a future decision. In each case, the output has to be readable and usable to someone responsible for the work.
That variety is useful. Document-heavy tasks break systems in ways a clean benchmark dataset won't reveal. Still, breadth makes evaluation harder: the quality of a result depends on both the model and the tools it uses to retrieve, organize, and reason over the workspace.
What do the benchmark results show?
In the published DAYJOB: Tax leaderboard, Claude Sonnet 5.5 has a 10.8% pass rate, Claude Fable 5.1 has 9.6%, and Claude Opus 5.5 has 9.2%. The benchmark also compares performance with average cost per trial. Claude Sonnet 5.5 and Opus 5.5 sit toward the high-performance end of the measured Pareto frontier.
Here's the first thing to notice: even the top reported pass rate is low under the benchmark's strict definition of "pass." Passing requires meeting at least 95% of the rubric for a particular task. The results suggest that consistently completing these complex assignments to that standard is still hard for the evaluated agents.
A leaderboard helps you compare evaluated systems under the same test conditions. It doesn't prove a model is ready to prepare returns independently, and it doesn't guarantee the ranking will hold across every firm, client population, or software setup. A benchmark is evidence about performance on its tasks - not a substitute for validating it in real operations.
Cost matters too. In a production workflow, higher performance may come with higher cost per trial. The Pareto view helps show which models offer a reasonable balance between measured performance and expense. But the "right" choice depends on the cost of mistakes, the volume of work, and how much review your process requires. A cheap system might be fine for low-risk classification. The calculus changes for advice that could affect a major financial decision.
The scenarios involving Anna's condo and the separating couple show why you can't rely on aggregate scores alone. In Anna's case, the agent assessed her proposed sale dates but missed the point where waiting could put the residence exclusion at risk. Across five attempts, Astra didn't identify that deadline, while Opus identified it in all five.
In the retirement-plan scenario, Opus recommended dividing a Roth IRA and 401(k) based on the assumption that the couple was not legally married. But the records pointed to a possible marriage tied to time spent in Texas and a ceremony in Colorado. That status needed to be verified before giving a recommendation. Across five attempts, Opus stuck with the unmarried assumption, while Astra recognized that the status needed checking in four attempts.
These examples don't prove one system will always outperform another in a live setting. What they do show is the kind of failure a task-level score is meant to expose: an agent can produce a coherent recommendation while overlooking a fact that changes what it should say.
Why is professional judgment difficult for AI?
Professional judgment often starts before the explicit technical question. A client may ask whether one of two dates is better, but the adviser still has to notice deadlines beyond those dates. A client may ask how to divide retirement accounts, but the adviser first needs to establish whether the couple's legal status limits the available options.
This is hard for an AI agent because the evidence is distributed and the task is underspecified. The system has to sort out what the client says, what the documents show, and what's still uncertain. It also has to decide which uncertainty is important enough to investigate. Asking for clarification on every gap would make the workflow unusable. Guessing on consequential gaps is how things get risky.
A practical way to think about the work is as a chain of decisions:
- Frame the assignment. Identify what the client wants, what deliverable is expected, and what decisions depend on the answer.
- Build the evidence picture. Locate relevant records, connect them, and note contradictions or missing support.
- Surface consequential issues. Look for deadlines, eligibility conditions, jurisdiction questions, and assumptions that could change the result.
- Choose a response to uncertainty. Verify facts when possible, ask the client when needed, and clearly qualify conclusions that remain uncertain.
- Complete and review the work. Carry findings into calculations, workpapers, or advice - then check that the output addresses the actual risk.
The order matters. If an agent starts calculating before confirming a fact that determines the tax treatment, it may produce precise-looking work built on the wrong premise. More tools or a larger context window can improve file retrieval, but they don't automatically fix the judgment problem. The agent still has to decide what to look for and how to respond when the evidence is incomplete.
This is also why a fluent answer isn't strong evidence of reliability. A polished explanation can hide an unsupported assumption. In professional settings, useful output should make the evidence trail - and the unresolved questions - visible enough for a reviewer to check.
For teams considering AI in tax work, a sensible near-term approach is to use agents to organize records, spot potential issues, and draft work for review - while keeping a responsible professional in the loop. The right level of oversight depends on the task. A routine extraction step might need spot checks. A recommendation involving legal status, eligibility, or a major financial decision needs direct verification.
The goal isn't to demand that an agent never makes mistakes. The goal is to design the workflow so that uncertainty and high-consequence errors get surfaced before they reach a client. DAYJOB's scenarios are a useful stress test because they check whether the agent recognizes when that safeguard is needed.
How does DAYJOB differ from narrower tax benchmarks?
Many existing evaluations focus on a specific capability. TaxBench tests answers to 114 tax questions across specialties, with each problem framed in a defined way and graded against a rubric. TaxCalcBench evaluates tax-return accuracy against expected filing values using tax documents and taxpayer scenarios. TaxLab tests whether an agent can gather missing information from a simulated taxpayer and complete a federal individual return.
Those tests are meaningful. A system needs tax knowledge, accurate calculations, and the ability to collect missing details. The difference is how much responsibility is left to the agent. DAYJOB is designed to test whether it can determine what the problem is - rather than only solve a problem that's already clearly framed.
That means asking whether an agent notices a potentially expiring tax benefit when the client only asked to compare two dates. It means checking whether a ledger entry is truly supported when an attachment is missing, instead of treating the accounting record as proof. And it means testing work beyond individual federal returns, including partnership analysis, estate work, reviews of disputed positions, and strategy advice.
DAYJOB's task design starts from real-world professional workflows. Tax experts build the scenarios, supporting records, expected outcomes, and grading criteria. Each task goes through a three-layer expert review for correctness, completeness, and solvability. The expertise includes corporate taxation, partnerships, individual returns, trusts, real estate, and multistate compliance.
That process matters because benchmark quality depends on more than realistic-looking documents. The expected outcome has to be defensible, the evidence has to be sufficient to solve the task, and the rubric has to distinguish material errors from acceptable judgment calls. Expert review can reduce ambiguity in evaluation, but it can't make the benchmark identical to every firm's standards or every client situation.
For a business evaluating tax AI, use this kind of benchmark as one input in a broader assessment. Ask whether your documents resemble the test environments, whether the model can work with your actual tools, and whether mistakes are recoverable before filing or advice goes out. Run the workflow on representative cases - including edge cases and missing evidence - then measure review time and the types of mistakes that still slip through.
The operational question isn't just whether an agent can do tax work. It's which parts of the workflow it can do reliably, under what review conditions, and what happens when it's wrong. DAYJOB adds evidence on that broader question, while leaving deployment decisions to teams that understand their risk and obligations.
Conclusion
Tax work isn't just rule lookup and arithmetic. It requires people to recognize when the question is incomplete, connect evidence across records, and explain what remains uncertain. DAYJOB: Tax turns those demands into a benchmark - and it shows how far evaluated agents still have to go to complete complex assignments consistently.
For now, the sensible stance is neither to dismiss AI nor to treat a strong answer as proof of readiness. Test agents on work that looks like your real cases, make the evidence trail reviewable, and keep human accountability where a missed issue could materially affect a client. The value comes from using automation where it reduces effort without pretending the judgment part is optional.