Recently, I was lucky enough to be invited to give a short presentation on blended teams, and I wanted to share some of my thoughts with you, dear reader!

What is a “blended team”? I am defining it as a team that has both human and AI workers who are collaborating on the same goals and tasks.

As you know, AI is good at some things, and not so good at others. You could say “when it’s good, it’s very, very good; but when it’s bad, it’s very, very bad” (wink to Henry Wadsworth Longfellow).

Vaccaro et al., Nature Human Behaviour, 2024 · When combinations of humans and AI are useful: A systematic review and meta-analysisSystematic review and meta-analysis synthesized 106 experiments and found that human – AI combinations underperformed the stronger standalone performer on average. The combination wins only when the task, evidence standard, reviewer, and escalation point are explicit.

The diagrams below, from arXiv:2507.09089v2 [cs.AI] Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity – Joel Becker, Nate Rush, Elizabeth Barnes, David Rein, present an interesting finding from the domain of software development: on average, experts tend to dramatically over-estimate how much AI assistance will speed up development. This study found that the estimates ranged from a 10% to 50% reduction in time – but the actual observed results were that tasks took up to 40% longer when AI assistants were used.

While developers estimated that AI assistance would enable them to reduce the time required for a task, as shown in the diagram below from the same study, the result was that they in fact took significantly longer when using AI.

Why is this the case? What’s going on here?

I’d like to offer the hypothesis that AI (and Generative AI in particular) is very good at doing things that have been done before, but much worse at doing something novel. This makes a lot of sense (to me, at least) if you consider that these models have been trained on very large corpora of existing code, and that they work by probabalistic prediction of the next token. So a model that has seen many examples of how to implement a particular type of requirement is likely to do a very good job of implementing that requirement. But, if you ask a model to implement something new, that it has not seen any, or many, examples of – it’s not going to do as good a job on that.

Another important thing to consider is the quality of the code the models were trained on. Charles Babbage called it way back in 1820 – garbage in, garbage out. And every developer knows, this is still very relevant today. If you consider that most of the code that the models were trained on is not what we would consider “good” – then you might not be surprised that the code they generate is also not “good”. Here’s some charts I pulled out of a paper showing distributions of some measurements of goodness across a wide sample of open-source code. The whole paper is very interesting and well worth a read.

Jin, Siyuan; Zhang, Mianmian; Guo, Yekai; He, Yuejiang; Li, Ziyuan; Chen, Bichao; Zhu, Bing; and Xia, Yong. “Software Code Quality Measurement: Implications from Metric Distributions.” 2023 IEEE 23rd International Conference on Software Quality, Reliability, and Security (QRS), 2023, pp. 488–496. https://doi.org/10.1109/QRS60937.2023.00054

Consider another example – AI assisted detection of security vulnerabilities. Models like Mythos and Fable can detect many more vulnerabilities that we ever did before them using human review and our pre-existing tools. But on analysis, we find that a lot of those issues detected (at least in my experience) were poor quality, false positives, or just duplicates of issues that we found with our existing tools. An enormous amount of human work is needed to review the output of these models and separate the signal from the noise.

With the widespread use of AI in this area, there also came a massive amount of work for software developers to review, categorize and address these issues. We’ve seen a spike in CVEs since the availability of these models, as shown in the diagram below, created from public CVE data reported by cve.org – note that this data set is up to end of June 2026. No one really knows if that trend will continue upwards – my educated guess is that all of the activity by security researchers and developers will result in a lot of issues being fixed, and we will possibly see that level off and start to drop again, as the industry as a whole gets more focused on fixing issues, and adapts our development, testing and release processes to catch more issues before shipping code.

There’s a lot of talk about AI replacing jobs, but I think it is much better to think about tasks that AI can do, not whole jobs. We all know that the job description rarely, if ever, actually describes the real job. There are a lot of tasks that people do that don’t make it into the job description. I don’t like to think about using AI to replace a person’s job, but instead to think about which individual tasks we could use AI for – tasks where we would get an improvement (in time, quality, etc.) and which would let the person focus more time on the more difficult (less automatable) tasks.

So what tasks are AI models good at? Here’s a synthesis from a number of studies (see the citations at the end of this article):

In my own personal experience, I’ve found that AI is really good at things like doing research, fact checking, writing code (for known patterns, and with strong guardrails), scanning for vulerabilities, and less good at things like writing prose, creating graphics, making decisions, and so on. Human verification is essential for all AI outputs, but especially whenever errors are costly, context is hidded or judgement defines success.

Here’s a finding from a study that looked at usage of Copilot for various tasks and how successful that usage was for different kinds of tasks.

arXiv:2507.07935v6 [cs.AI]  Working with AI: Measuring the Applicability of Generative AI to Occupations – Kiran Tomlinson, Sonia Jaffe, Will Wang, Scott Counts, Siddharth Suri

A very interesting study by Becker et al clearly demonstrates how the amount of time spent of different tasks changes when AI is used. Interestingly, the total amount of work is not necessarily the same, this looks only at the percentages, not the total.

arXiv:2507.09089v2 [cs.AI] Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity – Joel Becker, Nate Rush, Elizabeth Barnes, David Rein

This is supported by other studies which found that the total amount of time increased. Here are some findings from one such study in the domain of open-source software development:

And here are some findings from a study on professional writing – total time reduced but the amount of time spent of each part of the work changed:

Even if AI can reduce the amount of time it takes to write code from weeks to minutes – that does not mean I can ship the code and pass on that time gain to my end users. I still need to have people review the code, and I still need to do all the other security, compliance, and release management tasks. So while an individual task might be faster, my wall clock time is not necessarily impacted.

I love a poster I saw at a ballet school which read “Hard work is not for everyone” – I think that’s a very true statement. Using AI is easy, anyone can do it. But using AI well is a very different thing – and I think it is fair to say that most people probably won’t invest the time and effort to learn how to use AI well. This means that a lot of AI applications are going to be sub-par. You can easily find evidence for this – from the hype merchants publishing YouTube videos claiming to show you how to build an AI application in “five minutes”, to the high visibility failures like those showcased in “12 famous AI disasters”, Thor Olavsrud, CIO Magazine https://www.cio.com/article/190888/5-famous-analytics-and-ai-disasters.html and I suspect that you have personal experience of talking to an AI and finding its ability to actually “help” to be very poor.

Trust is vitally important in AI, and reputation damage and financial penalties are very real risks.

Using AI “well” means things like understanding how the models work, comparing the performance of different models and hyper parameter settings for your specific use case and possibly corpora, implementing guardrails and retries, using techniques like planning and reflection, building and using evaluations, knowing how and when to use different kinds of memory, using skills, and so on. No matter how good you are, I’d argue that you won’t get it right the first time, and you need a continuous improvement loop.

It’s important to collect inputs and outputs, and then to do human reviews, and provide feedback about what is good and what bad. This data can be used to implement improvements, so that in essence the application ‘learns’ over time how to do the task right.

You need a mechanism to route and track tasks. AI can work faster than people. You need a way to queue up tasks so that it can pick form the queue based on whatever criteria are appropirate, maybe the token quota for example. If a task is not clear, it should be able to ask questions before starting work – so you need a way to allow it to assign a task back to a human to get more information. You should use techniques like guardrails and adversial reviews (ideally with a different model) to control quality.

So when is AI going to excel?

I think that this is a good litmus test to apply when trying to identify tasks that AI will be able to excel at:

• Smaller, well-defined tasks with clear context, strong guardrails, and a strong definition of “right”
• Tasks that do not require novel thought or complex decision making
• Clean boundaries and handoffs between tasks
• Repeatable, high-volume tasks
• Continuous improvement feedback loop

We have a huge amount of opportunity ahead of us. Here’s an interesting finding from an Anthropic study that compares the opportunity in various industries to the actual observed usage today:

Labor market impacts of AI: A new measure and early evidence, Massenkoff, McCrory 2026 https://www.anthropic.com/research/labor-market-impacts

There’s lots of very active research in this area I’ve shared a number of papers that I’ve read myself as well as some of my conclusions from my own personal experience. I’d love to hear your thoughts and experience, please hit me up in the comments.

[1] Noy, Shakked, and Whitney Zhang. “Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence.” Science 381(6654), 2023. https://doi.org/10.1126/science.adh2586

[2] Brynjolfsson, Erik, Danielle Li, and Lindsey R. Raymond. “Generative AI at Work.” Quarterly Journal of Economics 140(2), 2025. First circulated as NBER Working Paper 31161, 2023. https://www.nber.org/papers/w31161

[3] Ishikane, Masahiro, Yuki Kataoka, Yasushi Tsujimoto, Yuki Moriyama, Yukimasa Matsuzawa, and Norio Ohmagari. “Efficacy of a Large Language Model Data Extraction System in Evidence Reviews for Emerging Infectious Diseases: A Randomized Crossover Trial.” Open Forum Infectious Diseases 13(7), 2026. https://pmc.ncbi.nlm.nih.gov/articles/PMC13393321/

[4] Khraisha, Qusai, Sophie Put, Johanna Kappenberg, Azza Warraitch, and Kristin Hadfield. “Can Large Language Models Replace Humans in Systematic Reviews? Evaluating GPT-4’s Efficacy in Screening and Extracting Data from Peer-Reviewed and Grey Literature in Multiple Languages.” Research Synthesis Methods 15(4), 2024. https://doi.org/10.1002/jrsm.1715

[5] Kalliamvakou, Eirini. “Research: Quantifying GitHub Copilot’s Impact on Developer Productivity and Happiness.” GitHub Blog, 2022 (updated 2024). Experiment conducted with Sida Peng and Aadharsh Kannan. https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/

[6] Becker, Joel, Nate Rush, Elizabeth “Beth” Barnes, and David Rein. “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity.” METR / arXiv preprint, 2025. https://arxiv.org/abs/2507.09089

[7] Nadra, Samir, and James P. Purdy. “Is genAI a Good Editor of Academic Writing?” Computers and Composition 80, 2026. https://doi.org/10.1016/j.compcom.2026.102993

[8] Kumar, Deepak. “SWE-PRBench: Benchmarking AI Code Review Quality Against Pull Request Feedback.” arXiv preprint, 2026. https://arxiv.org/abs/2603.26130

[9] Ribalta-Albado, Maria, and Pere-Pau Vázquez. “Evaluating LLMs’ Abilities to Create Charts, a Systematic Approach.” Computers & Graphics 135, 2026. https://doi.org/10.1016/j.cag.2026.104544

[10] Collier, Zachary A., Richard J. Gruss, and Alan S. Abrahams. “How Good Are Large Language Models at Product Risk Assessment?” Risk Analysis 45(4), 2025 (published online in 2024). https://doi.org/10.1111/risa.14351