Building Trust in Disconnected Data for Critical Compensation Schemes

TDS supports a UK Government’s arm-length body’s data platform programme, helping piece together crucial, life-affecting information for a critical compensation scheme from multiple disconnected sources, each with its own quirks and definitions.

TDS embeds a “shift-left” testing strategy to build confidence in that data from the earliest stages of design, rather than testing it in after the fact.

The Challenge

The task is to consolidate critical, life-affecting information for a compensation scheme from initially four distinct and disconnected sources, each with unique characteristics and definitions, then enrich it through entity resolution to present the digital service with uniform, trustworthy data that can track a person through their entire compensation journey, from registration to payment. Further data sources, each with their own schemas and governance, continue to be onboarded. Given the stakes involved, rigorous and correct testing is essential to establishing trust in a system where lives are literally on the line.

Solution and Implementation

Strategic Approach

TDS integrated confidence in the data early through a shift-left testing strategy — engaged from the very start of design, working alongside architecture specialists, developers, DevOps and data engineers to define the environment, test tooling and integration strategy from the outset. Two development environments support this: a general-purpose Dev-CI environment for continuous integration work, and a second environment that uses live data under carefully defined usage policies, enabling models to be built and tested against real data. A test pyramid approach catches errors early and reduces reliance on slower, more expensive end-to-end testing.

To address the lack of out-of-the-box AWS platform testing frameworks, TDS implemented and adapted boto3 (the standard Python library for AWS) integrated with pytest, enabling custom cloud infrastructure validation and reliable handling of complex asynchronous processes — continuously checking the real-time status of crawlers and database updates rather than relying on arbitrary timeouts. K6 is used for performance testing, allowing realistic load models to be defined and verified against specific thresholds, while PACT and RestAssured cover API and integration testing, adapting quickly to changing requirements and capturing downstream dependencies as they arise, backed by a continuously updated regression suite.

Key Deliverables

• A shift-left testing strategy embedded from the outset of design and requirements gathering
• Dual development environments, including a carefully governed live-data environment for model testing
• A test pyramid approach reducing reliance on slow, high-level end-to-end testing
• A boto3 + pytest framework for custom AWS platform validation and asynchronous process testing
• Performance testing against defined thresholds using K6
• API and integration testing via PACT and RestAssured, backed by a continuously updated regression suite

Results and Outcomes

• Foundational trust established in data critical to a high-stakes compensation scheme
• Errors caught early through the test pyramid, reducing rework and time-consuming late-stage testing
• Efficient, reliable AWS platform validation, reducing time spent on slow, error-prone manual checks
• Complex asynchronous AWS processes tested reliably, without delays from unreliable timeouts
• Performance verified against defined thresholds ahead of last-minute bottlenecks
• Adaptable integration testing that keeps pace with evolving business requirements
• Thorough lifecycle testing minimising surprises and ensuring a higher-quality delivered product