KPMG recently released a report on the influence of artificial intelligence within the consulting industry, stirring controversy due to the validity of its findings. Titled "Total Experience: Redefining Excellence in the Age of Agentic AI," this report aimed to investigate how AI transforms customer experience. However, an investigation revealed a significant issue. Most of the sources cited in the report were found to be fabricated, misattributed, or unverifiable.
The analysis conducted by AI-detection firm GPTZero exposed severe inaccuracies, with only 5 out of 45 citations deemed fully accurate, corresponding to an accuracy rate of just 11%. The remaining citations suffered from issues such as paraphrasing errors, fictional authors, and vague statements that could not be substantiated. Notably, this raised alarms about the exaggerated claims made regarding AI implementation at well-known firms like UBS and various healthcare providers.
This problem isn’t exclusive to KPMG. The consulting industry is witnessing a pattern of AI-generated inaccuracies. Similar concerns have emerged at EY, which withdrew a report amid issues with erroneous AI-generated citations, and Deloitte Australia has experienced similar challenges with fabricated references. Legal consequences have also been witnessed in other sectors, such as courts sanctioning lawyers for AI-generated citation errors.
Why is this concerning for consulting research consumers? Many companies rely on research from prominent consulting firms like the Big Four to inform major business decisions, including strategic shifts and technology investments. If the case studies that portray AI success stories turn out to be flawed or fictional, executives are basing key decisions on distorted information. If three of the Big Four have already been caught with substantial inaccuracies, it raises questions about other reports that clients have originally relied upon. This scenario illustrates not just a failure in quality control but underscores the critical need for diligence in evaluating the credibility of consulting reports.
As the consulting industry grapples with the ramifications of AI inaccuracies, it’s vital for business leaders to critically assess the information they consume. This ongoing situation serves as a reminder to approach consulting insights with caution and to prioritize verification of sources to ensure actionable and reliable information in today’s rapidly evolving digital landscape.