AI Chatbots for Agile Coaches: Why They Fail

AI used everywhere is a wave that is sweeping the internet, and now it is coming for Agile Coaches. There have been a number of Coaches using (or promising to use) AI Chatbots “trained” in their writing to act as a proxy for themselves.

As with everything in the world of GenAI, it is more complicated and more dangerous than it looks at first.

First, we need to clear up a misconception: unless you’re doing a serious amount of heavy lifting, you’re not training the model. Outside of the major labs, no one is training a model from scratch.

So, what can you do to build an AI chatbot using your materials?

Technique	Costs	What is it?
Fine Tuning	Large	A base model is trained on a dataset to better at that specific task
Reinforcement Learning from Human Feedback (RLHF)	Large	Human rates answers to thousands of questions
Retrieval Augmented Generation (RAG)	Cheap	The LLM decides when the query is run, which documents (or parts) are relevant to answering the question

Fine Tuning, RLHF and RAG

It sounds geeky. Why does this all matter?

If you spent the time and money to do a combination of Fine Tuning and RLHF, in most cases, we would get a more reliable system. However, even that system would have its content fixed when the training was complete.

So most Chatbots we see are based on RAG, and in the right context, that’s ok. The problem is that RAG is good at finding documents that closely match the query. If the writing comes close to matching the question, the AI will pull that document.

Problems with RAG Chatbots and Agile Coaching

The core problem is that the questions that people ask about Scrum, Kanban and Agile don’t map well to the writing in our libraries. As part of my work on “Effective Scrum”, I compiled a database of thousands of questions people have asked online. (My friends have called me the human LLM; unlike the tool vendors, I didn’t just send a bot out to scrape the internet. I spent days reading Reddit, Quora and StackOverflow.)

Here is a sampling of the questions I found:

• Do we need to update time estimates when we estimate based on story points?
• Numbering Sprints across products – should they reset or continue ad infinitum?
• How to handle disagreements about story-point estimates in Scrum?
• Whose responsibility is it to add stories to JIRA?
• …

In a RAG system built against my own content, some of these questions are answered well, and others well the system made up an answer. That is the core of the problem right there. The person asking the chatbot a question likely doesn’t have the expertise to evaluate the answer.

Other Risks to be aware of with AI Chatbots

Not all models are good at sticking to your content​. If it doesn’t find the answer in your content, many models will draw on their training data to answer the question. I can promise you that the average training data, from the internet, doesn’t have good advice for Agile teams.

AI vendors have reducee hallucinations (i.e. making stuff up) recently, but randomness and, therefore, hallucinations are inherent to the technology. More subtly, sometimes the chatbot just gets confused and forgetful. For example, I’m planning a hiking vacation in the Canary Islands, and the AI tool I’m using keeps assuming that I will park a rental car for 6 days at the ferry terminal in Los Cristianos while I’m on the island of La Gomera.

Agile teams and coaching on especially hard subjects for AI tools are especially hard because the subject area is so vast. In my office, I have two shelves of books related to Agile and according to DevonThink, another 20GB on my laptop. The more I learn about this subject, the more I realize most answers to questions should be “it depends” and “tell me more”.

Critical Thinking is key with these tools, and yet, as Microsoft demonstrated with a 2025 paper[^1], the better the tool sounds, the more likely people are to believe the output.

But I still want to build an AI Chatbot

Alaska’s court system is developing an AI chatbot to assist with all the forms involved in probate (i.e. dealing with property after someone dies). From all accounts, they’re doing the hard work to make it happen. Yet an activity estimated to take 3 months has already taken 15 months and is being scaled back[^2]. I would expect that dealing with probate is easier than dealing with the complexity of Agile teams.

Even Air Canada’s chatbot, which is just a glorified FAQ, has epic-fail moments that result in a lawsuit[^3]. It told a grieving family member that they needed to pay full fare, when in fact Air Canada has bereavement fares.

There are many effective ways to use GenAI in Agile teams, but a chatbot on a professional website to give advice is not one of them. Can you afford for your clients to subtly wrong advice? Agile is focused on people and relationships, let’s keep it that way.

[^1:] The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers

[^2:] Alaska’s court system built an AI chatbot. It didn’t go smoothly.

[^3:] Air Canada found liable for chatbot’s bad advice on plane tickets