Building trust in artificial intelligence for audit

Blog post header; blue cubes moving on a conveyor belt towards a green gate.

With the advent and growth of artificial intelligence (AI) in audit, the topic of trust comes up repeatedly in discussions. Auditors have always relied on their credibility to build and maintain relationships with clients. Auditors must build their own confidence in AI technologies before convincing clients and regulators that they have achieved the same level of assurance, if not more, than traditional methods.

At MindBridge, we have been asking ourselves for some time: How can we build this confidence for our customers?

To support auditors in their assessment of AI as a viable option, we commissioned a third-party audit of the algorithms used in our risk discovery platform. This independent assessment by UCLC (University College London Consultants) is an industry first, providing a high level of transparency to any user of MindBridge technology and assurance that our AI algorithms operate as expected.

While the independent report is only available to customers, we’ll summarize the activities and results here.

Ethical AI and MindBridge

AI and machine learning (ML) are the most influential and transformative technologies of our time, leading to legitimate questions around the creation and application of these systems. Will AI-based algorithms influence potentially life-altering decisions? How are these systems secured? Are audit firms required to prove the credibility of their AI tools?

The ethics of AI sees continual press and social media coverage because the technology shapes how we interact with the world, and defines how aspects of the world interact with us.

“AI ethics is a set of values, principles, and techniques that employ widely accepted standards of right and wrong to guide moral conduct in the development and use of AI technologies.”

The biggest companies, from Amazon to Google to Microsoft, recognize the ethical issues that arise from the collection, analysis, and use of massive sets of data. The ICAEW, in its “Understanding the impact of technology in audit and finance” paper, says that it is “crucial for the regulators to develop their capabilities to be in a position to effectively regulate these sectors in the face of advances in technology.”

MindBridge has long realized that transparency and explainability are critical for the safe and effective adoption of AI, with a demonstrated commitment to the ethical development of our technology.

Why third-party validation matters

“80% of respondents say auditors should use bigger samples and more sophisticated technologies for data gathering and analysis in their day-to-day work. Nearly half say auditors should perform a deeper analysis in the areas they already cover.”

Audit 2025: The Future is Now, Forbes Insights/KPMG

The most significant difference between traditional audit approaches and an AI-based one is in the effectiveness of the auditors’ time. An AI audit analytics platform can search 100% of a client’s financial data such that auditors can avoid large samples in low risk areas, focusing their time instead on areas of high judgement and audit risk. 

Due to the increased effectiveness of AI-based tools, regulators, audit firms, and their clients now consider data analytics an essential part of the industry’s business operations. With such a widespread impact, no one should blindly trust technology that has the potential for misinterpretation or misuse.

Auditors are known for assessing risk and gaining reasonable assurance. AI is just another example where skilled, third-party technology validation must be performed.

How the audit of MindBridge algorithms was achieved

The third-party audit of MindBridge algorithms was performed by UCLC, a leading provider of academic consultancy services supported by the prestigious UCL (University College London). UCLC’s knowledge base draws from over 6,500 academic and research staff covering a broad range of disciplines, and includes clients from international organizations, multinational enterprises, and all levels of government. UCL’s reputation as a world leader in artificial intelligence meant they were the right partner for MindBridge in completing this audit.

The goal of the audit was to verify that the algorithms used by our automated risk discovery platform are sufficient in three areas:

  1. The algorithms work as designed
  2. What the algorithms do while operating
  3. Sufficiency of MindBridge processes with regards to algorithm performance review, the implementation of new algorithms, and algorithm test coverage

For auditors and the accounting industry, the importance of this type of report cannot be understated:

“Auditors have to document their approach to risk assessment in a way that meets the auditing standards. They are also required to clearly document conclusions they make over the populations of transactions considered through the course of audit. Where this analysis is being conducted through MindBridge, there is therefore a burden of proof to demonstrate that the software is acting within the parameters understood by the auditor.”

– 3rd Party Conformity Review (Algorithm Audit) for MindBridge, UCLC

The first step was for the UCLC auditors to identify potential risks in the MindBridge algorithms across four sets of criteria:

Robustness of the algorithm

Split into correctness and resilience, this set of criteria validates that the algorithms perform as expected, react well to change, and are documented well. Generally, these are rated against the ability of the algorithm to score risk properly and work properly across a range of inputs and situations.

For example, many of the techniques employed by MindBridge audit analytics are used to identify unusual financial patterns, such as those required by the ISA 240 standard. One set of these techniques falls under “outlier detection,” a form of ML that doesn’t require pre-labelled training data and as such, reduces the potential to bring bias into the analysis.

The limitation of this unsupervised ML is that it has no deep or specific knowledge of accounting practices. MindBridge adopts the concept of an ensemble, or augmenting with different techniques, to bring domain expertise into the analysis. Called Expert Score, this allows the analysis to identify the relative risk of unusual patterns by combining human expert understanding of business processes and financial monetary flows with the outlier detection.

AI explainability

Split into documentation and interface components, this set of criteria validates that the algorithms and their purposes are easy to understand for users. This is critical for financial auditors in providing clear and meaningful expectations and key to building confidence in their clients.

“When making use of third-party tools, audit trails are vital, and auditors should ensure that they are able to obtain from providers clear explanations of a tools function, including how it manipulates input data to generate insights, so that they are able to document an audit trail as robust as one that could be created with any internally developed tool.”

Response to Technological Resources: Using technology to enhance audit quality, Financial Reporting Council

Privacy

These criteria apply to the effectiveness of controls relevant to the security, availability, and processing integrity of the system used to process client information. Privacy is closely linked to the prevention of harm, whether financial or reputational. AI systems must guarantee privacy and data protection throughout the lifecycle and be measured against the potential for malicious attacks.

Bias and discrimination

This applies to the effectiveness of controls in place to prevent unfairness in AI-based decision making. As MindBridge algorithms don’t use or impact data from identifiable individuals, this category presents limited risk.

Methodology

The assessment was done by conducting numerous tests and research to grade performance along a scale from “faulty” to “working as intended or passed the tests” for the sets of criteria above. This included:

  • Using different settings and data as input into the AI algorithms and recording the results
  • Comparing the results of validation code against the results of the AI platform code
  • Conducting interviews with the CTO and key data science, software development, and infrastructure personnel to determine processes and controls for systems development, operationalization, security, and testing
  • Assessing the data science and software development expertise of the MindBridge team

The UCLC auditors were granted a level 7, or “Glass-box,” access to the algorithms. This is the most transparent level available to an assessment and allowed the audit to cover all details of the algorithms.

Graph showing 7 levels for information concealed versus feedback detail trade-off curve

All MindBridge algorithms passed the assessment and the auditor’s report is available upon request to customers, regulators, and others who must rely on our algorithms.

Conclusion

With completion of the independent, third-party audit of its algorithms, MindBridge demonstrates clear evidence for AI-based tools to support the financial audit process safely and effectively. Through this assessment, MindBridge further enables the audit of the future by helping firms build confidence with their clients on the value of making AI and audit analytics an essential part of business operations.

For auditors, this announcement makes it easier to place further reliance on the results of the MindBridge artificial intelligence, allowing auditors to sample fewer items and spend more time where it matters most. It’s a key stepping stone in building credibility for AI in audit, and we hope that such third-party algorithm audits become the standard across the sector.

To learn about MindBridge’s most recent verification journey with Holistic AI visit our blog.

For more information on how AI and automated risk discovery supports your firm, download this free eBook now:

Automated risk discovery: What is it, and how firms can achieve it

Audit standards don’t need to change. Our approach to innovation does.

Audit standards - Innovation in the approach

If there is one thing that COVID-19 taught us, the audit industry can implement wide-reaching change at scale. In a matter of weeks we witnessed audit partners driving the adoption of video calling software, new headsets appearing, and people’s home desks being upgraded. It wasn’t long before palm-tree laden beaches were appearing as backgrounds on Zoom calls, something I couldn’t have imagined a few years ago.

Following the onset of COVID-19, many of these tools became a lifeline for accounting firms, who had to quickly change the way they operate, shifting to remote digital practices.

Despite the fast pace of change for these operational parts of audit, little has changed for the core ways that auditors generate assurance. Whilst the impact and take-up of data analytics and artificial intelligence is accelerating, there are still some oft-quoted roadblocks to wide-spread reliance on these techniques. “As a profession and including our regulators, we must up the ante on making analytics a mandatory part of the audit process” recently noted Becky Shields, Head of Digital Transformation at Moore Kingston Smith. 

Often the finger is pointed at the audit standards or regulators as major blockers of innovation. “They do not allow or require new technologies” is a perspective I’ve heard a few times, and comes in a few guises. Either in pointing specifically to the audit standards, or in a general comment that such innovation isn’t required by the PPC, a checklist commonly used by audit firms in the US.

Is it true that the standards and regulations themselves are standing the way of innovation? Or do we as an industry struggle to see value from innovation and change the way we think about assurance?

 

The audit standards – fit for purpose

Let’s take a quick look at the audit standards. All the checklists and expectations for a good quality audit start with the standards, after all.

The Public Company Accounting Oversight Board (PCAOB), which regulates auditors of publicly traded companies in the United States, recently explored whether there is a need for changes to standards, guidance, or other regulatory actions in light of technological advancements like AI. 

In their report, the PCAOB stated that its “auditing standards are not precluding or detracting from firms’ ability to use technology-based tools in ways that could enhance audit quality (for example, to perform more thorough and better-informed risk assessments).” The PCAOB did, however, acknowledge that the current standards “do not explicitly encourage” the use of technology-based audit tools, which isn’t a surprising revelation. 

Encouragingly, the PCAOB praised the capabilities of technology-based audit tools, noting that they can “enhance the auditor’s ability to efficiently and effectively analyze larger volumes of data, and in more depth, than when using manual audit technologies alone.” The Board also stated that technology-based tools could assist auditors with addressing the requirements in PCAOB risk assessment standards, a view that we at MindBridge share.

The PCAOB is not alone in its assessments. Across the pond, the UK’s Financial Reporting Council (FRC) recently consulted stakeholders regarding the use of technology to enhance audit quality. In a December 2020 report outlining its findings, the FRC noted: “Respondents also agreed that whilst additional application material and guidance would be beneficial, the current assurance model and audit standards do not represent a significant impediment to the development and deployment of technology in audit.” The report also noted that nearly all the respondents agreed that technology use could “significantly improve audit quality.”

These are views that are largely shared by the international standards setters at the IAASB. In a June 2020 update from the Technology Working Group they stated “The ISAs are flexible in terms of how audit procedures may be performed – manually, involving the use of , or a combination of both.” Whilst there has been fewer statements of this kind from the AICPA, the convergence between the AICPA and the ISA’s will likely mean that those following the American Clarified Statements on Auditing Standards will be in a similar position.

Whilst not a look at the standards, Chartered Professional Accountants of Canada (CPA Canada) has issued publications which encourage their members to adopt new technologies into their practices to stay relevant: “Firms left behind during the ADA implementation are more likely to see their situation deteriorate further with each new wave of emerging technologies. Therefore, there is a pressing need for accounting firms to develop and implement a long-term ADA adoption and use strategy that will allow them to continue with the next generation of analytics using Big Data and prepare for the use of analytics related to the .”

It’s the opinion of the standards-setters, whether in the US or internationally, that audit standards are fit for purpose. Reading the standards themselves, this isn’t hugely surprising.The standards are principles based and lay out a framework under which firms are free to rely on data-driven techniques for evidence. Regulators, methodology providers, and firms must interpret these standards to create their own workflows.

If the standards are fine, perhaps it is the regulators that are standing in the way of innovation?

 

Speak softly and carry a big stick – the role of the regulator

Whilst often the standards setters and regulators are the same body, it’s worth making the distinction between these two roles. The standards are high level and principles based. The regulators, whether it is the PCAOB, the FRC, or any of the others, are entrusted with the interpretation and enforcement of the standards. The goal is to maintain a level of audit quality amongst audit firms, and to make sure the audit is delivering on its mission to society as a whole.

Largely, regulators focus on after-the-fact punitive measures to ensure audit quality. Whether through the reputational damage caused by releasing their public inspection results, fines or action taken against individual audit partners, regulators and the regulatory process focus their attention on completed audit files.

When looking at these audit files, auditors and regulators are applying a shared sense of what good looks like. There is a cultural norm for audit files. These expectations change from market to market, and from sector to sector. The audit file for a large bank will look nothing like the audit file for the small non-profit down the road. With new technologies changing the way that assurance is generated, the expectations of those that enforce quality have to keep pace with the rest of the market. How do regulators know what good looks like when assessing a revolutionary new technique?

As firms innovate, they change their audit file and assurances models. They may do in a way that does not fit with the regulator’s or peer’s expectations for what a good quality audit looks like. This dynamic makes firms rightly nervous about putting together an audit file that looks nothing like other files in the same sector. The fear of the regulator is a real deterrent to trying new things – why stick your head above the parapet when the checklist is accepted.

We often see auditors request for greater levels of guidance from the regulators. It’s a theme that was echoed in much of the recent research from the regulatory bodies, and would be surely welcome. But the regulators are in a difficult position, as innovation often requires an agile, experimental and iterative approach. How can the regulator’s possibly create guidance for techniques that are under drastic evolution from one year to the next?

 

Safe spaces and dialogue

Regulators across the US, Canada, and the UK have a major role to play in encouraging and fostering innovation. Dialogue with auditors is a good place to start, and in our experience regulators are open to speaking with firms innovating on their audit approach. Even better would be the establishment of safe spaces for firms and the regulators to collaborate on innovative techniques. It’s an approach that the UK financial regulator, the FCA, has implemented well with its Digital Sandbox and TechSprints

This kind of collaboration between firms and regulators would allow both to explore new ways of creating assurance, and to receive feedback in a low pressure environment. It’s a classic approach for exploring new ways of thinking. It also has the potential to enable firms to innovate on real data; another key requirement to truly exploring new ways to generate assurance. Given the opportunity for artificial intelligence to change the way that the industry works, now seems like as good a time as any.

There’s also an opportunity for firms to create their own safe-spaces for innovation. Asking a separate team to try a new technique or technology alongside the traditional checklist is one such approach. This allows a side by side comparison, and an opportunity to assess what level of assurance both are generating. It provides a place where auditors can ask ‘what do I actually learn about my client from doing it this way’, without the regulators breathing down their neck. The firm can always take these experimental working papers to their regulator if they want feedback.

 

Where’s the carrot? Firms need incentives to innovate

In the end, firms need to gain a competitive advantage in the market from innovation. If they didn’t, there is no point in them innovating. Often innovation drives improved audit quality, so the firm must translate this into bigger and better client wins, or eliminate work they were previously conducting. 

Publicly released inspection reports from the regulators are one key public facing measure of audit quality, but it is hard for everyone involved to tell whether their particular audit is one that is deficient in some way. The large delay between the audit work happening and the release of these public results also undermines the value that these inspections play. 

Whilst the regulators could consider ways to make the audit process and its outputs more transparent, firms also have the opportunity to talk about the results of their audit much more openly with their clients and prospective clients. Greater transparency into the audit process and its outputs make it more likely that innovative firms will reap the rewards from pushing the boundary. 

The capabilities that technology-based tools can bring to the audit process mean that clients expect more from their auditors than ever before. Rather than a hindsight perspective, clients want a “forward-looking view”, with deeper insights that add value. Clients no longer want a financial checklist, according to Audit 2025: The Future is Now, a report from Forbes Insights and KPMG. Firms that maintain the “if it’s not broken, don’t fix it” attitude about their methodology are increasingly losing ground in the market.

“Clients want their auditors to take things to the next level to weigh and prioritize risks and opportunities based on their in-depth knowledge of the organization, its controls and processes, so more-informed decisions can be made to guide the organizations forward,” the report stated. Openly presenting the findings, particularly aided by visualisations, cements the auditor’s position as a fountain of knowledge about a company.

That forward-looking expectation also means that more clients expect their auditor to stay current with technology and adapt as new tools become available. As a result, clients “rightly believe that technology has improved the quality of audit,” noted the KPMG/Forbes Insights report. 

So it’s no surprise that 78% of KPMG/Forbes Insights respondents believe auditors should use more sophisticated technologies for gathering data, and nearly 80% think auditors should analyze bigger samples.

 

A wider range of skills needed for tomorrow’s audit

The pace of change is accelerating in the audit market, and it seems clear that today’s innovations will define how we generate assurance in tomorrow’s audit.

The skill set of auditors has to expand in order to facilitate the audit process of tomorrow. That shift necessitates an adjustment in attitudes and mindsets to benefit the industry at large. In order for these changes to be successful, change needs to happen at every level of the industry, from the junior associates joining our firms, through the partners and regulators. 

Among the sought-after skills that clients want their auditor to have, the KPMG/Forbes Insights report found that 67% of clients are looking for increased technology skills, while 66% want better communication skills, 65% for critical thinking skills, and 59% for investigative financial skills. They are not looking for the old-school, number-crunching accountant any more. Considering they are obsessed with learning patterns and identifying anomalies in data, It is time for us to learn from what the data scientists are doing.

Given the massive skills and talent gap that the audit industry is facing, It’s also worth pointing out that teaching these skills is an opportunity for firms to keep their best staff around for longer.  Many like-minded organizations, including MindBridge, have also taken an interest in preparing tomorrow’s accountants and auditors with the tools and knowledge they need to succeed before they reach the workforce.

 

The data revolution is an opportunity for auditors

The audit sector is known for being stuck in its ways. We’re the number-crunching, adhering to regulations, checklist-oriented type. Sometimes, we follow those traits to a fault. As technology proliferates and data volumes grow, modern businesses are putting data front and center. This introduces both risk and complexity, but also an opportunity for data savvy auditors to provide leadership in the data driven world.

By encouraging more curiosity, creativity, and experimenting across the industry, we can hopefully adjust attitudes and mindsets regarding technology-based tools and audit standards. 

The future of audit is being reshaped by technology, there is no dismissing that. However, the industry must ensure it is keeping pace with the broader economy. Instead of looking for excuses to resist change, collectively, we need to find ways around any obstacles so that the accounting industry becomes a leader in innovation, resilience, and agility. We can’t wait to be told by regulators what tools to use ​​ it’s crucial to change attitudes and embrace the changes that are happening now. 

A study by the Harvard Business Review summed it up perfectly: “If more and more companies methodically dismantle blockers to innovation and encourage employees to experiment, perhaps we will finally see the gap close between leaders’ innovation goals and reality.”