AI Helps Experienced Lawyers More Than Juniors, Study Finds

Artificial intelligence may not be the great skill equalizer that many young workers were promised, according to new research from MIT economist David Autor and six co-authors. In a three-month experiment with patent lawyers, AI improved the quality of legal drafts across the board, but when the tool was removed and lawyers were tested on independent judgment, only experienced lawyers showed measurable gains. Early-career lawyers did not.

«Our results challenge the idea of AI being an automatic skill equalizer,» Autor said in written responses to Fortune. «Our data suggest that it’s a performance equalizer, but a skill-disequalizer, in that only practitioners who already had foundational mental models were able to level up their underlying skill sets.»

The study, published as a National Bureau of Economic Research working paper, has not been peer-reviewed. It enrolled 133 lawyers at 11 U.S. intellectual-property law firms that have ongoing patent-drafting relationships with Google. Researchers randomly assigned access to a custom AI patent-drafting assistant, a then-unreleased Google Labs tool. Two-thirds of the lawyers received access; the rest went without until the study ended.

Google’s involvement went beyond providing the tool. The paper lists Autor’s six co-authors as Google employees. It also states that Google conducted the study and paid its direct costs, and that MIT’s human-subjects committee determined MIT was not engaged in the research. Google representatives described the paper as independent research Autor did as part of his fellowship.

Lawyers drafted patents from simulated inventor materials after 10 days and again after 90 days. Patent attorneys at an independent law firm graded the work without knowing which lawyers had used AI, scoring enforceability, accuracy, strategic ambiguity, completeness and clarity. After 90 days, AI access raised drafting scores by 0.38 standard deviations. In a Google blog post accompanying the paper, Autor and co-author Tanya Rodchenko described that as an 11-percentile-point gain relative to control scores. The improvement came from less weak work, not more excellent work.

The harder test came at the end. Lawyers had to mark up a hypothetical patent containing many substantive and stylistic errors, a task patent lawyers routinely perform unaided. No AI was allowed. Senior lawyers who had AI access beat their control-group peers by 0.45 standard deviations. Junior lawyers showed no average gain.

The paper reports several caveats. Only 91 of the 133 lawyers finished the AI-free test. The firms declined a skills test at the outset, so the researchers inferred skill gains from random assignment instead of measuring before and after. The authors also flagged 15 of the 91 markups as possibly AI-assisted despite the ban, and the flags turned up about as often among control lawyers. With those 15 excluded, the senior advantage shrinks to 0.39 standard deviations and holds up, though at a looser statistical threshold. The overall effect does not.

The juniors’ scores split, the paper found. Some improved, but others did not, suggesting that AI may create an illusion of competence for less experienced workers. The study sorted lawyers by experience, not age. «Senior» meant seven or more years in practice, and the researchers did not study Gen Z as a group. But early-career workers are the people the findings bear on most directly.

Autor is head of MIT’s economics department and a Google Technology and Society Visiting Fellow. He is best known for the «China shock,» developed with economists David Dorn and Gordon Hanson. Their research showed that Chinese import competition did lasting damage to American manufacturing communities, with wages and labor-force participation depressed for at least a decade in heavily exposed labor markets.

More recently, Autor and Hanson have warned of a «China shock 2.0»: Chinese competition in advanced industries that threatens American technological leadership and high-wage jobs. Some economists, including Apollo Global Management’s Torsten Slok, have warned that the «AI shock» could resemble the China shock in its impact on the U.S. labor force. Autor has pushed back, saying AI «will not be, in any sense, a repeat of the China trade shock» because that was experienced by U.S. firms as a «pure negative competitive shock,» while AI will have a «very different texture» in boosting productivity.

Autor said his worry about AI is not mass job loss. «The danger that we may face with AI is not that it eliminates large numbers of jobs in net, but that it eliminates some areas of specialty while creating new ones simultaneously,» he said. «That sounds like a reasonable deal, but the workers losing those old careers are generally not the ones who will be able to take advantage of the new roles that emerge.»

5Views

Author

Delaney Sawyer

Society Reporter

Delaney Sawyer covers public affairs, politics, business, culture and daily news for Core Memo. The role focuses on verification, context, and clear explanations for readers.

All articles by this author