Is AI Making Your Organization Fragile or More Resilient

“We must adopt AI, it’s an arms race” - I hear this refrain from clients all too often. We seem to have forgotten that AI is just a tool. Like any tool, it can be used well, making us more effective. But it also works rapidly and has an outsized effect on our work, so caution and discretion are needed.

Remember to consider and ask, when we change a system, what other effects will this have? What are the second-order effects? (There is a whole body of work called Systems Thinking that helps explore this in far more depth.)

I’ve created a few simple questions to help decide if we’re using AI in the right place.

• Are we helping at the bottleneck? Inside any team or organization, there will be a bottleneck to delivering value. (Theory of Constraints). If we speed up work ahead of the bottleneck, we just make the bottleneck worse.
• AIs are creative (that’s good), but that same creativity will lead to errors. Do we have enough safeguards in place to catch the mistakes? Can we afford errors in the part of our system where we use these tools?
• Catching the errors requires increased critical thinking and more focused time to do that thinking. Are we allocating enough time for this so it won’t be stolen back for another purpose?
• Does our use of AI improve Simplicity, or make things harder to understand? One of the key reasons for the 2008 financial crisis was that the lending system had become too Complex, so no one understood everything that was happening.
• Is the AI usage increasing Collaboration or harming it? Agile approaches bake in the principle of Collaboration, but using AI can reduce it if we’re not careful.

Why AI might be harmful

One of the primary things that makes organizations fragile is adopting new technology without understanding its impact throughout the system.

Technical example

Data from a report by GitClear[1] indicates an increased reliance on AI for code generation. Code duplication is up, refactoring is down. AI allows us to go faster now but increases the technical debt we have to pay down later. Further, the DORA metrics[2] showed that deployment rollbacks increased in 2024. This is likely tied to an increase in defects. When reading the DORA report[3], I was shocked that it didn’t make the obvious linkage with the increased use of AI to write code.

Although we will never see a smoking gun, I can already see AIs killing products and companies by burying the team in technical debt. Using AI to write code today will give technical coaches good work for years.

An increase in code duplication, defects and rollbacks means the AI generated code is making the system more fragile. Generating technical debt faster is not a good thing.

Drug Discovery example

The Globe and Mail [4] reported that, in several Canadian companies, AI made it faster to find a new molecule. However, did that solve the real problem?

I’m not doubting that it can get us there a little bit quicker,” said Brian Bloom, CEO of Bloom Burton & Co., a Toronto investment bank focused on health care. “But in the 10-year journey of drug development, you may be saving five months.

In pharmaceuticals, the bottleneck in the system is proving that the drug works, not the speed at which they find candidates. Even if the AI model was better than people at finding a candidate drug, it still requires 10 years of human trials. Further, for the drugs described in the article, most of them may be no better than the ones people find without using an AI.

Deep Genomics was trying to use 40 different AI models in their process. It became messy and unsustainable, making their system fragile.

In the above examples, AI has been used where there wasn’t any bottleneck. At a minimum it has wasted time and energy that could have been used to make improvements at the bottleneck. If they had considered the second-order effects, they either wouldn’t have used AI at all, would have put safeguards in place (to avoid duplication, defects etc), or better, they would have used it to help at the real bottleneck. In the software development world, the safeguards to avoid these problems don’t currently exist and so we rely on the individual team member to engage in more critical thinking. There is no evidence to show that is working well.

It’s not all bad. Let’s turn to the good.

What is Generative AI good at?

Finding patterns

Generative AIs are large-scale pattern recognition tools. They’re helpful at spotting patterns that we humans miss. Caveat: beware the false positives and incomplete answers. AIs can find a pattern that isn’t supported by the data, building relationships with things that aren’t related. When conducting weekly and quarterly review sessions, I use these tools to analyze my personal notes. It always makes connections between unrelated items. It also misses patterns that I can clearly see. So why use it? Because it often spots something I missed in my notes. At Product Camp Ottawa, several attendees reported using a GenAI to transcribe and summarize interviews. These people still took the time to read the transcripts, however, the tool saved them several hours of typing to compile their notes like they would have done in the past.

Building a Rapid Prototype

A Product Owner I met recently showed a prototype of his next great feature. He hypothesized that he could speed onboarding for new clients by retrieving key marketing assets directly from their websites. He used GenAI in several places.

First he used Claude to generate prototype grade requirements from a rough idea. Then he used Cursor - an AI programming tool - to generate the app. This enabled him to take his feature from idea to working prototype in only 3-4 days of work. Instead of talking theory, he used the prototype to show his leadership the value of the feature. He got a green light and the engineering team have since built the feature with the care needed to make it sustainable.

Creating Automations

Many small repetitive tasks deserve to be automated, but creating the automation takes time and effort. Generative AI can be very useful at creating automations, especially ones you can easily verify. The AI was useful at taking the drudgery out of creating the automation, and the risks are low because we can verify the result with little effort.

In these cases, the Generative AI was useful because:

• Critical thinking was involved
• Results could be verified
• Duplication and Technical Debt were acceptable (e.g. the prototype)

Bottom Line

The promise of these tools to speed up our work is enticing. But we need to ensure that we’re always asking questions about whether their use will make our systems more resilient and not more fragile. As with anything Agile, we should be assessing the effect of the tool(s) on our system and constantly checking second-order effects, and not just succumb to the promises of the shiny tools vendors.