Eva Visser
34 · Almere
- Children
- 2, in day care
- Income
- €38,000 a year
- Residence
- Lawful, 9 years
- Nationality
- Dutch
Paid
Demo. The people and results are made up.
Twelve parents claimed childcare benefit this month. A risk model decides whose claim is investigated as fraud.
The right to benefit depends on lawful residence, not on nationality. Both have lived here legally for nine years. Eva's claim is paid. Kasia's is flagged as fraud. Unless someone tests the model first, nobody finds out.
This happened. Until 2018, the Dutch tax administration's risk model gave parents without Dutch nationality higher fraud scores. The data protection authority fined it €2.75 million in 2021. Source
34 · Almere
Paid
34 · Almere
Flagged
The authority brings test claims the provider never sees. The provider brings its model, sealed. They meet on a European supercomputer, cut off from the network.
The provider's modelSealed. Opened only inside the test.
The authority's test claimsSealed. The provider never sees them.
Compute centre in the EUNo network. Keys destroyed when the test ends.
360 claims with the same family, income and residence. Only the nationality changes. Each square is one answer.
Flagged as fraud Paid
Dutch nationality
Other nationality
Parents without Dutch nationality were four times as likely to be flagged. The model does not go live.
Fails the agreed limitThe gap between groups is wider than the limit both sides set before the test.
The provider retrains the model. The same sealed test runs again, under the same rules.
Within the agreed limitEvery group is inside the limit both sides set before the test.
The authority signs off on evidence verified on its own compute. The provider deploys with proof. People are protected.
Decide on evidence, not the provider's word. Your test data stays yours.
Show your model is fair without handing it over. Find problems before your users do.
Host confidential tests as ordinary jobs.