Most organizations hold the same customer, supplier or product several times: once in the CRM, again in billing, again in a marketing tool, each with slightly different details. Data pipelines that collect information from many outside sources face the same problem at a larger scale. Without entity resolution, reports double-count and AI systems give different answers depending on which copy they find.

Entity resolution works when records are standardized first, candidate pairs are narrowed by blocking, each pair is scored on several fields, matches are grouped into clusters and the result is measured against a labeled sample, because no single matching rule handles every variation in real data.

This article explains the entity resolution process step by step, the main matching techniques, how language models help, which tools to consider, how to measure quality and how a threat intelligence pipeline resolves entities from hundreds of sources. It does not cover master data management governance programs.

Key takeaways

  • Entity resolution links records that describe the same thing when there is no shared, reliable identifier.
  • The process has six steps: standardize, block, compare, classify, cluster and create a canonical record.
  • Blocking is essential at scale. Comparing every pair of one million records means about 500 billion comparisons.
  • Matching can use rules, probabilistic models such as Fellegi-Sunter, machine learning or language models, and most production systems combine several.
  • Measure precision and recall on a labeled sample. False merges are often more harmful than missed matches.
  • In a delivered intelligence pipeline, URL and content-hash deduplication plus entity resolution into a versioned graph turned hundreds of sources into one consistent view of vendors, products and security events.

Deduplication, record linkage and entity resolution

TermMeaning
DeduplicationFinding duplicates within one dataset, such as one CRM
Record linkageMatching records across two or more datasets, such as CRM and billing
Entity resolutionThe broader process of identifying, linking and merging records into real-world entities, within and across datasets
Master data managementThe organizational program that keeps a trusted version of key entities over time, with ownership and governance

The entity resolution process

1. Standardize

Bring fields into consistent forms before comparing them: lowercase text, remove punctuation and legal suffixes such as Inc. or Ltd., format phone numbers internationally, parse addresses into parts and normalize dates. Much of the matching quality is won or lost here.

2. Block

Comparing every record with every other record does not scale. The number of pairs is n × (n − 1) ÷ 2, so 1 million records produce about 500 billion pairs. Blocking groups records by a cheap key, such as the first letters of a name plus a postcode, or the email domain, and only compares records within the same block. Good blocking cuts comparisons by several orders of magnitude while keeping almost all true matches.

3. Compare

For each candidate pair, compute similarity per field: exact match for IDs, edit distance or Jaro-Winkler for names, token overlap for addresses, numeric difference for dates and amounts, and embedding similarity for free text.

4. Classify

Decide whether each pair is a match, a non-match or uncertain, using rules, a probabilistic model, a trained classifier or a language model. Uncertain pairs go to human review.

5. Cluster

Matches form chains: if A matches B and B matches C, all three may be one entity. Clustering groups them, with checks that stop a single weak link from merging two different entities.

6. Create the canonical record

Build one entity from each cluster with survivorship rules, such as the most recent address, the most complete company name or the value from the most trusted source, and keep links to every source record.

A six-step entity resolution pipeline with record counts falling at each stage

Matching techniques compared

TechniqueHow it decidesStrengthsLimits
Deterministic rulesExact or near-exact match on chosen fields, such as email or tax IDFast, transparent, easy to auditMisses variations and typos
Probabilistic (Fellegi-Sunter)Weighs agreement and disagreement on each field by how informative it isHandles messy data, can be trained without labelsNeeds careful field choices and tuning
Machine learning classifiersLearn match decisions from labeled pairsHigh accuracy with good training dataLabeled data takes time to build
EmbeddingsCompare meaning, not spelling, using vector similarityCatches synonyms and different wordingCan match things that are similar but not the same
Language modelsRead two records and judge whether they describe the same entityStrong on ambiguous text and unstructured sourcesSlower and costlier per pair, need evaluation

The probabilistic approach dates back to Ivan Fellegi and Alan Sunter's 1969 theory of record linkage and remains the basis of widely used open-source tools.

How language models help

Language models are useful in two places. First, extraction: they pull structured fields, such as company names, product names and versions, out of articles, emails and documents, so unstructured sources can enter the matching process. Second, adjudication: for the small share of pairs that scores cannot decide, a model can read both records and explain whether they match. Running a model on every pair is expensive and unnecessary, so keep rules and scores for the easy cases and send only the uncertain ones to the model.

Tools to consider

ToolTypeNotes
SplinkOpen source, PythonProbabilistic linkage based on Fellegi-Sunter, runs on DuckDB, Spark and other backends
dedupeOpen source, PythonActive learning: asks a person to label a few pairs, then learns
ZinggOpen sourceMachine learning entity resolution on Spark
AWS Entity ResolutionManaged serviceRule-based and machine learning matching within AWS
Custom pipelineYour codeFits when entities are specific to your domain, such as vulnerabilities or legal parties

According to its documentation, Splink can link about a million records on a laptop in around a minute and scale to over 100 million records on Spark, and it is used by public bodies including the UK Office for National Statistics and NHS England.

Duplicate customers, suppliers or products spread across systems? We build entity resolution pipelines that merge them into records you can trust.

Explore staff augmentation

How to measure entity resolution quality

  • Precision. Of the pairs the system matched, how many are true matches? Low precision means false merges.
  • Recall. Of the true matches, how many did the system find? Low recall means duplicates remain.
  • A labeled sample. Label a few hundred pairs, including hard cases near the decision threshold, and measure against them after every change.
  • Cluster checks. Review the largest clusters. An entity with thousands of linked records often means one bad rule merged unrelated records.

Decide which error costs more. Merging two different customers can send one person's data to another, while missing a duplicate usually means a second marketing email. Most teams set thresholds to favor precision and send borderline pairs to review.

Entity resolution in a threat intelligence pipeline

A US cybersecurity media company needed a platform that collects news and advisories from hundreds of open sources and turns them into structured data about vendors, products and security events. The same vendor or vulnerability appears in many articles under different names, and the same story can appear at several addresses.

A .wrk team of six to eight engineers built resolution into the pipeline.

  • Deduplication before extraction. Items are matched by URL and by a hash of their content, so the same story is processed once.
  • Language model extraction. Models pull vendors, products and events out of each article into structured fields.
  • Entity resolution into a graph. A graph worker resolves each mention to an existing entity or creates a new one, linking vendors, products and security events.
  • Temporal versioning. Entities keep earlier versions, so the team can see how information about a vendor or product changed over time.
  • Idempotent processing. A processing ledger and dead-letter queue make reprocessing safe, so a repeated article updates entities instead of duplicating them.

The structured data now powers a public security events section on the company's website. Read the NEXUS case study.

A vendor node linked to products and security events, with version history for one entity

Entity resolution for CRM and marketing data

The same principles apply to customer data, where the usual problem is the same person or company entered several times with different spellings and job titles. Our guide to CRM data cleansing covers the standardization step for B2B fields such as company size, industry and role, and our article on medallion architecture shows where resolved entities sit in a data platform. For pipelines that must survive retries and replays, see our guide to idempotency.

Common mistakes

  • Matching before standardizing. Inconsistent formats hide true matches and create false ones.
  • No blocking. Comparing everything with everything works in a test and fails in production.
  • One threshold for everything. Different entity types and sources need different thresholds.
  • Merging without keeping source links. Once records are merged without links, a wrong merge cannot be undone.
  • No human review. A small review queue for uncertain pairs improves both quality and the training data for the next version.
  • Never re-measuring. New sources and data drift change quality. Measure on a schedule.
FAQ

Frequently asked questions

What is entity resolution?

Entity resolution is the process of identifying records that refer to the same real-world entity, such as a person, company or product, across one or more datasets, and linking or merging them into one entity, even when the records differ in spelling, format or identifiers.

What is the difference between entity resolution and deduplication?

Deduplication removes duplicate records within one dataset. Entity resolution is broader: it links records within and across datasets, groups them into entities and usually builds a canonical record while keeping links to the sources.

What are common entity resolution techniques?

Common techniques include deterministic rules, probabilistic matching based on the Fellegi-Sunter model, machine learning classifiers trained on labeled pairs, embedding similarity and language models for ambiguous cases. Most systems combine them with blocking and clustering.

Can large language models do entity resolution?

Language models are useful for extracting entities from unstructured text and for deciding difficult pairs. Using them for every comparison is slow and expensive, so they work best alongside rules and similarity scores that handle the easy cases.

What tools are used for entity resolution in Python?

Popular Python tools include Splink for probabilistic record linkage and dedupe for active learning. Zingg runs on Spark, and managed services such as AWS Entity Resolution suit teams that prefer not to run their own pipeline.

To sum up

Entity resolution turns scattered, inconsistent records into entities you can trust. Standardize first, block to keep comparisons manageable, score pairs on several fields, cluster carefully, keep links to every source and measure precision and recall on labeled examples. Use language models where text is unstructured or pairs are ambiguous, and rules and scores everywhere else.

.wrk builds data pipelines, entity resolution and data platforms for companies in the US and Europe, with engineers who join your team or deliver the pipeline end to end.

Need one trusted view of your customers, suppliers or products? Tell us about your sources, and we will propose an entity resolution approach and timeline.

Explore staff augmentation