What Workplace Studies Measure

In Generative AI at Work, Erik Brynjolfsson, Danielle Li, and Lindsey Raymond studied a phased rollout to 5,172 customer support agents. Their published analysis reports that agents resolved 15% more issues per hour on average, although gains differed across workers. That outcome measures productivity in one service setting. A claim about payroll or return on investment would need separate evidence. [1]

Eleanor Dillon and colleagues conducted a field experiment with 7,137 knowledge workers across 66 firms. Among employees who received access to the tool, 80% used it. The authors report that those users spent two fewer hours on email each week during the second half of the six-month study. The researchers did not detect changes in the quantity or composition of tasks from providing AI to individual workers. The distinction matters when an organization expects time savings to produce additional output. [2]

METR found a different result in a 2025 randomized trial involving 16 experienced developers and 246 tasks in familiar open source projects. Allowing AI tools from early 2025 increased completion time by 19%. In a February 2026 follow-up, METR reported that participant selection and difficulties measuring concurrent agent use made its newer estimates unreliable indicators of the current effect. METR expects those problems to understate AI’s benefit, because developers who rely most on AI declined to participate or withheld tasks suited to it. Neither result supplies a universal multiplier for a business forecast. [3] [4]

These studies examine different people, tasks, tools, and outcomes. Each organization needs to test whether the findings apply to its own employees and workflows. Averaging the studies’ headline percentages would obscure what each study measured.

The Baseline and Comparison Group

The baseline describes what would happen without the proposed AI deployment. It might involve the current workflow, conventional automation, or additional staff to meet rising demand. A sound comparison covers feasible alternatives over the same period and documents expected changes in workload, prices, and service standards.

A simple comparison of results before and after adoption can assign unrelated improvements to AI. Suppose employees in an AI group complete ten cases per hour before adoption and thirteen afterward, while a comparison group increases its rate from ten to eleven. All cases meet the same quality standard. Subtracting the changes produces an estimated effect of two additional cases per hour. The full increase of three includes the improvement that the comparison group also achieved.

Figure 1

Observed Change and the Comparison Trend

Where feasible, an organization can randomize access or rollout timing. Researchers can assign individuals or teams, depending on whether employees might share AI outputs with colleagues in the comparison group. Outcomes should be defined in advance, with a plan for missing data and nonuse. HM Treasury’s AI evaluation guidance recommends considering experimental methods early and explaining the comparison clearly. [5]

When randomization is unavailable, difference in differences compares outcome changes across groups. Its causal interpretation requires assumptions about how outcomes would have evolved without treatment. Similar trends before treatment can inform that judgment, but they cannot prove it. For rollouts across several dates, methods such as Callaway and Sant’Anna’s estimate a separate effect for each adoption group and period. [6]

Three Models an Organization Can Test

Evaluators need to link assignment and usage records to task logs, quality reviews, invoices, and sales records. Each record should match the workers, teams, or business units in the study. Each proposed benefit needs an observable outcome and a defined measurement period.

A spreadsheet can organize the financial calculation. The empirical model tests whether AI changes an outcome and estimates the size of that change. The resulting estimate, its uncertainty, and explicit rollout assumptions then become inputs to the spreadsheet. The calculation cannot establish causation on its own.

For a sales experiment, contribution per eligible lead includes leads that never purchase. Comparing only customers who bought would change the composition of the groups and could distort the result. Returns, discounts, fulfillment costs, and displacement of existing sales also affect the financial outcome.

Statistical significance is only one consideration. Even when researchers estimate a gain precisely, that gain may be too small to cover deployment costs. A wider uncertainty interval may leave both a worthwhile benefit and a loss plausible. Managers need to weigh that uncertainty against the cost of expanding the deployment.

How Teams Use the Time AI Saves

Time savings give managers an opportunity to change production or spending. Whether the organization can use that opportunity depends on demand, scheduling, staff capabilities, and constraints elsewhere in the workflow. A team that saves five minutes on each task may still need the same number of employees to cover its shifts.

In this hypothetical example, AI frees 200 staff hours each month after employees complete review and rework. Managers use 50 hours to replace contractor work, 100 hours to support additional sales, and 50 hours to reduce a backlog. Figure 2 assigns every hour once.

Figure 2

Allocation of Staff Time

If the organization would otherwise pay a contractor $40 per hour, it saves $2,000. Suppose a separate sales evaluation supports $3,000 of additional contribution after incremental fulfillment costs, and employees can handle that business within the 100 hours available. The example records backlog reduction as a service outcome without assigning it a dollar value.

At $40 per hour, analysts could value all 200 hours at $8,000. That estimate expresses the value of staff time, and the organization receives no additional $8,000 in cash. Adding that amount to the contractor saving and sales contribution would count the same operational improvement more than once. The government’s digital benefits framework also warns against counting overlapping benefits. [7]

Figure 3

Monthly Net Cash Benefit

Existing staff time still has an opportunity cost. An appraisal can estimate the value of work that employees forgo when they use their time elsewhere. That valuation needs an explanation and a check on whether the output measures already capture it. A cash forecast records when the organization pays or receives money. An economic appraisal can also include costs and benefits that involve no cash payment.

Adoption and the Timing of Returns

A pilot’s effect may change as employees learn, the workload expands, or the organization introduces new model versions. A forecast should specify who can use AI, what share of eligible work they use it for, and how effective that use becomes. License purchases alone do not answer those questions.

The example now includes an $18,000 initial investment. At full adoption, the organization gains $5,000 each month before paying $2,000 in operating costs. The calculation assumes that benefits increase in proportion to adoption. Monthly costs comprise $1,000 fixed plus $1,000 multiplied by the adoption rate. These assumptions describe hypothetical scenarios.

Figure 4

Cumulative Returns Under Three Adoption Paths

Immediate full adoption produces an $18,000 net cash benefit by month twelve. Gradual uptake produces $7,600, while the slower path leaves a $12,000 shortfall. A report that shows only the monthly result at full adoption would conceal that difference.

In practice, benefits may not increase proportionately. A reduction in contractor spending may require a contract renewal, and payroll savings may require a staffing threshold. Faster production may also generate no additional sales when demand is weak. A good model represents those dependencies directly and tests the assumptions that change the decision.

For longer horizons, analysts should discount future cash flows consistently with the organization’s appraisal method. Their cost estimates should include transition work, evaluation, training, and any additional security checks or human review that the deployment requires. Adding depreciation to a cash model that already records the asset purchase would count the same cost twice.

Forecast Validation and Measured Results

A forecast combines evidence with assumptions about future conditions. Analysts should preserve the original forecast and label which inputs came from experiments, observed operations, vendor estimates, or judgment. Predictions can then be compared with later data from periods or business units outside the data used to fit the model.

Prediction checks and causal evaluation answer different questions. A model can predict spending accurately without establishing that AI caused a saving. Conversely, a credible pilot can identify a local effect that fails to transfer to a larger rollout.

When results differ from the forecast, analysts should investigate adoption, task mix, review time, demand, and model changes separately. Invoices and payroll show whether savings actually occurred, and sales records show whether additional sales cover the incremental costs of serving customers. Recording the tool version and workflow helps analysts distinguish changes in the technology from changes in how employees use it.

A financial calculation may leave out outcomes that matter to employees and customers. Evaluators should also examine workload, customer experience, service failures, and access to services. Reports can present those outcomes alongside the cash estimate even when no defensible price exists for them. Employees completing tasks faster does not, by itself, establish that customers receive better service.

Managers can use this evidence to decide whether to expand the deployment or change the workflow. If the expected benefits do not materialize, the original forecast and subsequent measurements help them identify which assumptions failed and whether continued spending has a defensible basis.