From Data Checks to Smarter Benchmarking

At Cerise+STPF, we often talk about data quality, reliability, and usefulness. But behind those words lies a practical challenge: how can we review more audits, more consistently, and generate insights that are useful for analysis and benchmarking?

This is where AI (artificial intelligence) and automation can make a real difference.

We have been developing automated review approaches to support the analysis of SPI Online audits. As the number of audits grows, reviewing every response, comment, and inconsistency manually becomes harder to scale. We envision AI and automation as a way to strengthen expert review, not replace it. Our objective is not simply to automate controls and save time: it is to improve audit quality, strengthen data analysis, and produce more relevant benchmarks.

To that end, we have developed a review workflow for SPI Online audits that combines structured checks, historical comparison, contextual information about the people involved in the audit process, and AI-supported analysis to help us assess the quality and reliability of an audit more effectively.

How does this approach work?

Automation helps reduce repetitive work, while AI can add an extra layer of interpretation where needed.

The review workflow does not assess an audit in isolation. It gathers audit answers, comments, and contextual information from different sources, then performs a series of automated checks on key items such as portfolio figures, borrower totals, staff breakdowns, payroll data, deposits, branches, exchange rates, and average loan values.

It also compares each audit with the institution’s historical data, which helps us determine whether the current results are in line with past submissions or whether a sudden shift may require closer review. This makes the analysis much more robust: the question is not only whether the data is coherent in itself, but whether it is coherent for this institution over time.

On top of that, this approach adds useful context around the audit process itself. For example, it can identify whether people from our SEPM Pros Network are involved in the audit process, and whether users linked to the institution or the audit process have completed the e-learning courses we offer. This gives us additional context when interpreting audit quality. It helps us better understand the environment in which the audit was produced, the likely level of familiarity with the methodology, and the type of support or capacity that may already exist.

More than quantitative checks: reasonable ranges and analytical controls.

An important feature of this work is that the review logic goes beyond simple yes-or-no validation. It also applies reasonableness checks and analytical calculations to assess whether reported values are plausible and internally consistent.

For example, the workflow verifies whether certain figures fall within expected ranges, such as exchange rates compared with current reference values, average loan size, APR, board size, or average annual pay. It also compares breakdowns with totals and checks whether related values align within an acceptable margin. In addition, the analysis looks at the coherence of responses across practices within the Universal Standards.

For instance, if an institution does not report output or outcome indicators under Dimension 1 on Social Strategy, it would be inconsistent to report that management is evaluated on social achievements under Dimension 2 on Committed Leadership. These controls help identify data that may be technically complete but still unlikely, inconsistent, or difficult to interpret.

Some calculations go further. In the payroll review, for example, the workflow calculates not only overall average pay, but also average pay for women and men separately. It can then derive additional indicators, such as the gender pay gap and an average pay ratio based on standard deviation.

This type of quantitative review allows us to move beyond basic validation and toward a more meaningful analysis of patterns, coherence, and potential anomalies in the data.

An AI-supported qualitative review

The most interesting layer is not only quantitative checking, but also qualitative review.

In addition to the calculations, many audits include comments that explain or justify an answer. Those comments are extremely useful for each audit to interpret them and use them for decision-making.

The AI component helps assess how informative those comments really are. It looks at whether a comment explains institutional practices, provides useful context, refers to documents or interviews, or simply repeats the indicator. This makes it possible to distinguish between audits’scores that are well justified and audits that may appear complete on the surface but provide limited explanatory value.

In other words, the tool checks: "Is the explanation meaningful?"

A tool for better human analysis, not less.

The tool does not replace human judgement at the final stage.

It suggests a quality score, highlights issues, and surfaces signals that deserve attention, but that assessment is then reviewed and validated by a human expert from the team.

That human step is essential. Automation helps us apply consistent logic, save time on repetitive checks, and focus attention where it matters most; expert review ensures that the final interpretation remains nuanced, grounded, and credible. That is where AI becomes truly useful for Cerise+SPTF: not as a substitute for expertise, but as a way to reinforce it and make it more scalable, more consistent, and more impactful. This helps us move away from a purely static reading of audit results and toward a more informed interpretation of quality, comparability, and reliability.

A pivotal approach to improving benchmark

With systematic review of quality of the audits, the value for benchmarking becomes clear.

If we want benchmarks to be relevant, we need confidence in the underlying data. We also need context. By combining automated checks, historical comparisons, qualitative review, information about training and network involvement and expert review, we can select quality audits to produce benchmarks, analysis and state of practices that are nuanced, reliable and useful for management.