Identity resolution is a data problem wearing an ML costume

Retailers typically have three times as many customers as they have people. The records exist; they are just not linked to each other.

The instinctive response is a probabilistic matching model. For most retailers, deterministic matching on the fields they already hold recovers the majority of the duplication and is considerably easier to explain.

Start by finding out how bad it is

Before choosing an approach, measure:

• total customers versus known unique humans
• match rate on exact email, and on normalised phone
• how many records have more than one email or phone
• the rate at which a customer acquires a new email in a year

That last one matters more than teams expect. Contact preferences change, households share numbers, and a "duplicate" created last year may be the same person with a new email — which means any matching approach has to be versioned and re-evaluated, not built once.

Deterministic first, and in the right order

Match on strong identifiers before anything fuzzy:

1. Government or customer reference number where held
2. Exact email match, lowercased and trimmed
3. Exact phone match, normalised to a national format
4. Exact postcode plus surname, for in-store loyalty without digital identifiers

In most retail estates the first three recover between sixty and eighty per cent of what is recoverable. That is enough to make segmentation materially better than nothing, and it takes weeks.

Then use fuzzy matching for the residue

After deterministic matching, a smaller ambiguous residue remains: name variations, transposed digits, household members sharing a phone.

For this, record linkage using pairwise comparison features — edit distance on name, Jaro-Winkler on surname, shared postcode, purchase-pattern similarity — is transparent and tunable. A specialist can be used, but the effort is usually better spent on the deterministic layer.

Every household is not one person

A significant error in retail identity is collapsing a household into one profile. Shared landlines and shared family plans mean four customers with one phone number.

Where a shared identifier exists, do not resolve. Instead mark it as shared and treat it as a household entity, with individuals resolved through other evidence — loyalty accounts, payment instruments, name plus date of birth.

Clusters with many distinct names and one phone are the signal. Most retail data teams find this quickly once they look.

Do not merge irreversibly

The permanent mistake is a hard merge. If two records were wrongly linked, the evidence has been destroyed, and separating them later requires manual intervention with no automated path.

Keep the linkage as an assertion with a score and a method, never as a destructive operation. Record every link and unlink with a reason. Allow an override, and log the override — the overrides are how you discover your matching rules are wrong.

A golden record that is wrong is more damaging than no golden record, because downstream everything trusts it.

Survey question identity as the hard case

Retailers with strong loyalty programmes face the toughest version: one person, several loyalty accounts, several emails, one landline, and purchases that alternate between them.

This is genuinely hard, and it is often not worth solving fully. A useful compromise is to identify the household accurately, link known accounts to it with confidence, and accept that individual-level identity is approximate at the edges.

Most segmentation, targeting and lifecycle marketing work at household level anyway, and an accurate household is more valuable than an inaccurate individual.

Measure what improved

Track:

• known unique customers after resolution, against a baseline from a manual audit
• campaign reach across resolved profiles
• duplicate creation rate per month after go-live
• override rate by match method, which tells you where the rules are misfiring

And run the manual audit again after six months. Identity resolution degrades continuously as contact details change. A system that was accurate at launch and is quietly wrong by year two is common and entirely avoidable with a scheduled re-evaluation.