Stelian ROMANProject Manager| MicroSafetyCarlingford, New South Wales, Australia
Artificial Intelligence (AI) is transforming businesses across industries, but as models become more complex, understanding their decisions becomes increasingly challenging. Explainable AI (XAI) aims to address this by making AI systems’ behaviour and outputs transparent and interpretable. However, integrating XAI requirements within Agile workflows—characterized by rapid, iterative development—poses unique difficulties. This blog post explores the intersection of XAI and Agile, identifies key challenges, and offers practical recommendations for teams looking to build transparent, trustworthy AI in fast-paced environments.
What challenges has your team faced in implementing XAI requirements within Agile process?
Which XAI tools or techniques have you found most effective for iterative development?
How do you measure the success of explainability initiatives in your organization?
Blog post: Explainable AI (XAI) Requirements in Agile Workflows
Luis BrancoCEO| Business Insight, Consultores de Gestão, LdªCarcavelos, Lisboa, Portugal
One question I keep coming back to is whether we are measuring the success of XAI in the right way.
Many initiatives assess explainability by the quality or completeness of the explanations produced. Yet explanations only create value when they improve human understanding, strengthen judgment and lead to better decisions.
Perhaps the ultimate measure of Explainable AI is not how well a model explains itself, but how much better people are able to question, trust and act on its outputs. After all, the purpose of explainability is not explanation itself. It is better human decision-making.
...
1 reply by Stelian ROMAN
Sep 08, 2026 6:47 PM
Stelian ROMAN
...
"How much better people are able to question, trust, and act on its outputs". Your statement brings back painful memories. We developed analytical reports that management couldn't understand due to a lack of knowledge and experience. I see the same problem with AI output. Some managers will accept it because it looks nice and iot is credible, not because they understand how it was actually done.
Program Manager| HARPER SRLSanto Domingo / Distrito Nacional, Dominican Republic
I haven't implemented formal XAI requirements yet, but I see the biggest challenge as balancing explainability with delivery speed, especially in Agile environments. I think explainability should be considered from the beginning rather than added later. That makes it easier to validate assumptions, support governance, and build stakeholder trust as the solution evolves.
...
1 reply by Stelian ROMAN
Sep 08, 2026 6:50 PM
Stelian ROMAN
...
I agree, explainability should be part of the project scope. "To give the contractor free rein between requirement definition and operation is inviting trouble." W. Royce, 1970. We can replace the contractor with "AI Agent " :)
Saving Changes...
Stelian ROMANProject Manager| MicroSafetyCarlingford, New South Wales, Australia
Jun 29, 2026 7:38 AM
Replying to Luis Branco
...
One question I keep coming back to is whether we are measuring the success of XAI in the right way.
Many initiatives assess explainability by the quality or completeness of the explanations produced. Yet explanations only create value when they improve human understanding, strengthen judgment and lead to better decisions.
Perhaps the ultimate measure of Explainable AI is not how well a model explains itself, but how much better people are able to question, trust and act on its outputs. After all, the purpose of explainability is not explanation itself. It is better human decision-making.
"How much better people are able to question, trust, and act on its outputs". Your statement brings back painful memories. We developed analytical reports that management couldn't understand due to a lack of knowledge and experience. I see the same problem with AI output. Some managers will accept it because it looks nice and iot is credible, not because they understand how it was actually done. Saving Changes...
Stelian ROMANProject Manager| MicroSafetyCarlingford, New South Wales, Australia
Jun 30, 2026 8:42 PM
Replying to Lissette Indhira Pimentel Sosa
...
I haven't implemented formal XAI requirements yet, but I see the biggest challenge as balancing explainability with delivery speed, especially in Agile environments. I think explainability should be considered from the beginning rather than added later. That makes it easier to validate assumptions, support governance, and build stakeholder trust as the solution evolves.
I agree, explainability should be part of the project scope. "To give the contractor free rein between requirement definition and operation is inviting trouble." W. Royce, 1970. We can replace the contractor with "AI Agent " :) Saving Changes...
Community Champion
