Why AI Doesn't Replace Your ScrumMaster (and probably never will)

AI’s superpower is speed—its ability to churn out text faster than any human. But there’s a catch: it sounds too confident for what it actually “knows”. ScrumMasters, Product Owners and Developers frequently ask me how they can use GenAI effectively, and they wonder if it’s going to automate their job out of existence. Most questions come from misunderstandings of how the current generation of tools work.

GenAI Doesn’t Think

Marketing hype for the current generation of tools that we call AIs makes it seem like these tools are on the verge of replacing people at work, implying that you won’t need a ScrumMaster because the AI will be your coach, and there’s no need for a Product Owner when User Stories write themselves.

The hype claims that Generative AIs (or GenAI), such as Google Gemini, Claude Sonnet, and ChatGPT, can think and reason, but don’t be fooled by the marketing language. Here’s the reality:

• Learning - They don’t learn, they’re limited to the data they were trained on.
• Reasoning and Thinking - Even when asked to think step-by-step, it’s just faking the process. It doesn’t actually understand.
• Context Window - The models we use today have limited memory for the current conversation. Once that memory is exceeded, they forget the beginning of the conversation.
• Training Data - AIs have been trained on all of the text on the internet. So their output is just the average of the internet which, of course, is unregulated.

GenAI (properly called Large Language Models) takes the text you give it in the prompt and predicts the next word that should come in the conversation. That word is added to the text after your original prompt, and the revised text is fed back to the model to predict the next word. This continues, building responses one word at a time, until it predicts that enough words have been generated.

This is not how a human works. We consider what we want to express and then find the best way to say it. By giving GenAI human attributes—reason, thinking, conversation—we build a model in our heads that implies they’re like humans. Except they’re not, so we must keep clear in mind their strengths and weaknesses.

GenAI Strengths and Weaknesses

Their core strength is dealing quickly with a large amount of information. Their weaknesses centre around errors. The greatest danger is that, as demonstrated in “The Impact of Generative AI on Critical Thinking” [1], the more confident the AI sounds, the more likely we are to believe it [2] even though there is no correlation between that and whether or not the information is good.

Strengths

• Sifting through a large quantity of information quickly
• Finding patterns in data
• Generating a large number of different ideas quickly in response to a question (also a weakness if done too early in our work process, as it will bias our creative thinking and the likelihood of spotting missed cases)
• Summarization
• Analyzing our existing work for what we might have missed

Weaknesses

• Patterns - false positives, it will find a signal in noise even when there is no signal
• Sounds confident even when providing inaccurate information
• Sycophantic – The tool’s response to a prompt often starts with “That’s a great idea”
• Risk of errors (from misinterpretations) and hallucination (completely false information) with new models may worsen as new models are increasingly trained with limited supervision on “synthetic data” (generated by other models)

GenAI can be beneficial, however, we need guardrails.

Guidelines for GenAI Use

Key considerations:

1. Collaboration - Much of Scrum’s success is getting team members to collaborate
2. Bottlenecks - The theory of constraints shows that improvements that aren’t at a system’s bottleneck are waste (usually making the bottleneck worse).

Scrum and Agile teams can use the following questions to help distinguish between healthy and harmful uses of GenAI.

Evaluating Tools Using Our Questions

I’ve surveyed many tools across the internet to find a representative sample of the promises the tool vendors make about helping your team. But there’s an inherent problem as I offer my opinions. No website or marketing copy captures what a product actually does behind the scenes. I interpret content and then share the information with you, and you will in turn interpret what you think I’m saying. Think of the child’s game of “Telephone”. Communication, no matter how careful and responsible, leaves room for error.

AI for a Sprint Retrospectives

Many GenAI models promise to connect with tools like Jira, Slack, GitHub, and calendars to automatically gather and analyze sprint data (e.g. completed tasks, communication patterns, roadblocks). They pitch:

• AI-Suggested Talking Points: Generates relevant discussion topics based on the analyzed data, helping teams focus on impactful issues.
• AI Grouping and AI generated Action Items: Help you easily streamline your retrospective.
• Sentiment Analysis: Analyzes team communications and feedback to gauge sentiment and identify underlying emotions or concerns.
• Facilitation Assistance: The AI can act as a co-facilitator, guiding the conversation and covering all key areas.
• Team Members Accountability: Helps identify, assign, and track action items that arise from the retrospective.

My Take

This is a bit of a mix.

The good: Automated data collection makes it more likely that valuable data comes to the Retrospective.

Only ‘okay’: Analysis, if used to augment and not replace human thinking, might spot hidden patterns. Sentiment Analysis of the team’s conversations might spot overlooked patterns. However, the harm caused to Psychological Safety, knowing that an AI overlord always looks over our shoulders, exceeds the value.

Outright bad: Grouping exercises in Retrospectives are designed to start a conversation among team members about how problems are related. So AI Grouping and, worse, AI generated Action Items are replacing human insight, which is ironic since the humans still need to understand and solve the problem.

Baked into the tools, I looked at? The assumption that all of the meaningful interactions occur inside a JIRA, Asana or KanbanZone. Email, Slack, or real-world conversations are far more important in most teams.

AI Daily Scrum

You just can’t make this up. One vendor promises to replace Daily Scrum with text, voice and video messages, then run reports on a schedule and share the reports with multiple Admins.

My Take

A Daily Scrum is an event to give the team time to understand what their peers are working on, hear about impediments others face, and to spark collaboration. None of this is served by an automated tool. This tool has nothing to do with Scrum, except that it stole the language. I can only imagine that a team using a tool like this will have low psychological safety and also be micromanaged to the brink of extinction.

Not surprisingly, this same vendor, who completely misses the point of Scrum, pitched the ability of the tool to also run Retrospectives on autopilot by gathering notes and feedback from individual team member submissions and “Replace retrospective meetings with scheduled team surveys. …. Run retrospectives consistently on schedule”. Is replacing human interaction with writing to an AI the way to build effective teams and positive work environments? Is keeping to a schedule the most important thing in a Retrospective?

Generate Meeting Agendas and Take Notes

Several tools are offered to generate meeting agendas (planning, reviewing, and retrospectives) based on the current sprint’s needs, and claim to make meetings more purposeful, efficient, and less time-consuming.

An AI meeting assistant auto joins, auto shares, and auto summarizes meetings. AI-powered meeting assistants help save professionals and teams an average of 4 hours a week and increase productivity by automatically generating action items, summaries, and follow-up emails.

My Take

• How much time do we spend writing agendas? Minutes? Speed-ups in creating agendas seem to be of low value.
• Meeting summary and action items might be useful. It saves time, and the risk of errors seems low as long as the participants review the notes/actions immediately afterwards.

Conclusion

So many tools out there promise to provide AI, making promises to solve all of your team’s problems. I could write another five pages reviewing tools. Like any other Scrum-related improvement, take the time to breathe. Before we rush to adopt AI, take the time to look:

• Where is your team bottlenecked?
• Where would applying a GenAI help make the lives of all team members easier?
• Have we considered all the consequences, not just the hype the tool vendor is selling us?

GenAI is a valuable tool. However, it is not the only tool in a ScrumMaster’s tool chest.

Good tools help you better understand your team and current circumstance. The real work of understanding a team and their work can’t be automated.

Where have you found GenAI helping or harming your team?