Continuous Compliance Testing and How to Benchmark Screening Providers
In our latest Fireside Chat, Castellum.AI’s Peter Piatetsky spoke with Howard Spieler, Business Sanctions Risk Officer at Citi; Marlon Oxley, Sanctions Tech Innovations Lead at Rabobank; Perla Stoekert, CCO at Tipalti; and Jose Caldera, CEO of Yanez Compliance.
The discussion focused on a critical challenge in sanctions compliance: the risks of outdated screening systems. The panel explored why a “set it and forget it” approach no longer works, how to implement continuous testing and catch blind spots before regulators do, how to validate screening models for accuracy and speed, and best practices for benchmarking screening providers.
Key Takeaways
A “Set It and Forget It” approach no longer works
“While false positives are easily identified, the real concern is the undetected false negatives—risks that slip through unnoticed. Without proactive testing, organizations remain unaware of these vulnerabilities until they become serious compliance failures.”
Many organizations assume that once their compliance screening systems are configured, they will continue to function effectively. However, this static approach, where companies check their screening systems once a quarter or even once a year, is increasingly risky. Screening systems often develop blind spots that go unnoticed until an audit reveals gaps or a regulatory issue arises.
With regulatory requirements and sanctions lists in constant flux, risk profiles can shift overnight. A compliance system that worked yesterday may fall short today. Regular benchmarking and system calibration help businesses proactively identify gaps in their compliance programs. By continuously testing against known risk scenarios organizations ensure their screening systems are accurate and effective against emerging threats.
Above-the-Line (ATL) vs. Below-the-Line (BTL) testing
Effective compliance programs require two complementary types of testing:
Above-the-line testing examines what is being flagged, ensuring that high-risk transactions and entities are correctly identified.
Below-the-line testing focuses on what is not being flagged. It helps to detect potential blind spots where risky transactions may be slipping through unnoticed.
"Compliance failures don’t come from what you flag, they come from what you miss. If you're not doing below-the-line testing, you’re running a compliance program in the dark—gaps will go unnoticed until it’s too late”
Continuous compliance testing for global payment companies
For global payment companies, compliance is a complex balancing act. They must maintain the speed customers expect while navigating diverse regulatory requirements across multiple jurisdictions, constantly evolving sanctions lists, regional naming conventions and linguistic variations.
Name screening: A high-stakes challenge
Name screening is particularly difficult when processing cross-border payments. Different regions structure names in different ways, and if screening tools don’t account for these variations, false negatives become a serious risk.
“Payment companies must train their teams to understand these naming conventions and configure their systems periodically to recognize linguistic nuances and avoid missing high-risk entities.”
Future of compliance in global payments
Regulators worldwide are tightening expectations, making it clear that businesses must take a proactive approach. A system that only works in one jurisdiction, or under specific conditions, won’t be effective for a company operating globally.
With the growing volume of financial transactions, especially with instant payments, slow, inefficient and error-prone manual compliance checks are becoming impractical. AI-driven, automated compliance solutions that enable real-time screening not only improve accuracy but also help financial institutions meet shifting regulatory expectations, like the EU’s Instant Payment Regulations (IPR), that now mandate real-time sanctions screening to be a standard part of compliance programs. As one panelist summed it up,
“By implementing a unified testing framework that leverages automation, continuously refines testing methodologies and ensures systems are fine-tuned to capture regional variations, payment companies can build a compliance program that is both effective and scalable.”
A smarter approach to benchmarking compliance systems
To improve compliance screening, organizations must adopt a structured benchmarking approach. This involves:
Regularly testing screening systems against real-world scenarios
Analyzing detection capabilities for emerging risks and adapting controls accordingly
Aligning compliance efforts with regulatory expectations to demonstrate a proactive stance on risk management
Benchmarking screening providers: Key considerations
Initial tuning matters: An out-of-the-box compliance solution often produces misleading results if not properly calibrated. Organizations frequently underestimate the time required for initial setup. While ongoing testing is essential, the initial configuration is equally important to ensure risk thresholds and screening parameters align with a company’s specific risk exposure.
Peer benchmarking: Comparing screening practices with similar organizations (i.e. with organizations that have similar customer base, transaction types and jurisdictional risks) can be just as valuable as formal benchmarking studies.
Risk-aligned testing for vendor selection: When assessing compliance providers, organizations should design test scenarios that accurately reflect their unique risk profile.
Objective data representation: If the data used for benchmarking is incomplete or skewed, it can create gaps in compliance coverage. To avoid lapses, screening evaluations should be based on a clear, comprehensive and unbiased dataset.
Result accuracy: An effective system must reliably identify high-risk entities while minimizing false positives that create operational bottlenecks and analyst fatigue.
Update speed and data freshness: Sanctions lists and risk data change frequently. Providers must offer real-time or near-real-time updates to ensure organizations are always working with the latest information.
How to build trust with regulators
Regulators understand that compliance systems are not flawless, but they expect firms to justify their decision-making processes. Even if a firm’s compliance approach is imperfect, well-documented processes and approvals can demonstrate diligence and provide some leniency from regulators.
“The goal is not just compliance but a defensible, transparent compliance strategy backed by evidence.”
Documentation should clearly outline:
The rationale behind compliance decisions
The steps taken for model validation and testing
How risks are identified and managed
How the program is continuously improved
How AI is used in compliance workflows
Final thoughts
The regulatory environment is evolving at a breakneck speed, rendering traditional compliance methods increasingly obsolete. To stay ahead, companies must:
Adopt continuous testing to detect blind spots before regulators do
Document compliance decisions to build regulator confidence
Leverage automation to improve screening accuracy and efficiency
Benchmark screening systems to ensure they meet industry standards