If you read only one thing about AI/LLMs this week, make it this:
September 21, 2023
If you read only one thing about AI/LLMs this week, make it this: "Centaurs and cyborgs on the jagged frontier" by Ethan Mollick
• In a study involving BCG consultants, those who used AI/LLMs to complete a set of 18 realistic tasks outperformed those who didn’t — they completed more tasks, completed them more quickly, and did so with higher quality. LLMs also level the playing field — lower/more junior performers got the biggest boost.
• It’s not always obvious which tasks are well suited to LLM assistance — this “jagged frontier” of what LLMs can and can’t do well is a key knowledge gap. This also creates an “asleep at the wheel” effect where consultants without AI did better on ill-fitting tasks simply because they didn’t have the AI leading them astray.
• The study identified two successful models for how to work with AI/LLM tools: Centaur (half-human, half-machine -- where certain tasks are delegated to AI) and Cyborg (human and machine are intertwined on most or all tasks). Both models work better than delegating all tasks to the AI, which tends to produce mediocre results.
• We have been waiting for this kind of data — this study was huge, involving 758 consultants, and shows conclusively that LLMs improve knowledge worker productivity. Full stop. No caveats, and even the negative effects (the drop in quality on tasks outside the frontier) wash out in the sheer volume of positive impact.
• The real question for leaders is how to come up with their own list of 18 tasks for their own people, and how to identify the limits of the “jagged frontier” in their own organizational context. LLMs are weird enough that you need to learn by doing, which requires a substantial investment by both people and teams.
• Malcolm Gladwell popularized the concept (I am oversimplifying here) that it takes 10,000 hours of practice to become great at anything. LLMs are no different! Learning the nuances of how to use them to accomplish your own goals is THE key skill for all knowledge workers going forward.
• This applies even (maybe especially) to people who write code — becoming a Cyborg is the best way to avoid an avalanche of bad software, written by robots who don’t know any better.