Most organizations hold the same customer, supplier or product several times: once in the CRM, again in billing, again in a marketing tool, each with slightly different details. Data pipelines that collect information from many outside sources face the same problem at a larger scale. Without entity resolution, reports double-count and AI systems give different answers depending on which copy they find.
Entity resolution works when records are standardized first, candidate pairs are narrowed by blocking, each pair is scored on several fields, matches are grouped into clusters and the result is measured against a labeled sample, because no single matching rule handles every variation in real data.
This article explains the entity resolution process step by step, the main matching techniques, how language models help, which tools to consider, how to measure quality and how a threat intelligence pipeline resolves entities from hundreds of sources. It does not cover master data management governance programs.
Key takeaways
- Entity resolution links records that describe the same thing when there is no shared, reliable identifier.
- The process has six steps: standardize, block, compare, classify, cluster and create a canonical record.
- Blocking is essential at scale. Comparing every pair of one million records means about 500 billion comparisons.
- Matching can use rules, probabilistic models such as Fellegi-Sunter, machine learning or language models, and most production systems combine several.
- Measure precision and recall on a labeled sample. False merges are often more harmful than missed matches.
- In a delivered intelligence pipeline, URL and content-hash deduplication plus entity resolution into a versioned graph turned hundreds of sources into one consistent view of vendors, products and security events.
Deduplication, record linkage and entity resolution
| Term | Meaning |
|---|---|
| Deduplication | Finding duplicates within one dataset, such as one CRM |
| Record linkage | Matching records across two or more datasets, such as CRM and billing |
| Entity resolution | The broader process of identifying, linking and merging records into real-world entities, within and across datasets |
| Master data management | The organizational program that keeps a trusted version of key entities over time, with ownership and governance |
The entity resolution process
1. Standardize
Bring fields into consistent forms before comparing them: lowercase text, remove punctuation and legal suffixes such as Inc. or Ltd., format phone numbers internationally, parse addresses into parts and normalize dates. Much of the matching quality is won or lost here.
2. Block
Comparing every record with every other record does not scale. The number of pairs is n × (n − 1) ÷ 2, so 1 million records produce about 500 billion pairs. Blocking groups records by a cheap key, such as the first letters of a name plus a postcode, or the email domain, and only compares records within the same block. Good blocking cuts comparisons by several orders of magnitude while keeping almost all true matches.
3. Compare
For each candidate pair, compute similarity per field: exact match for IDs, edit distance or Jaro-Winkler for names, token overlap for addresses, numeric difference for dates and amounts, and embedding similarity for free text.
4. Classify
Decide whether each pair is a match, a non-match or uncertain, using rules, a probabilistic model, a trained classifier or a language model. Uncertain pairs go to human review.
5. Cluster
Matches form chains: if A matches B and B matches C, all three may be one entity. Clustering groups them, with checks that stop a single weak link from merging two different entities.
6. Create the canonical record
Build one entity from each cluster with survivorship rules, such as the most recent address, the most complete company name or the value from the most trusted source, and keep links to every source record.

Matching techniques compared
| Technique | How it decides | Strengths | Limits |
|---|---|---|---|
| Deterministic rules | Exact or near-exact match on chosen fields, such as email or tax ID | Fast, transparent, easy to audit | Misses variations and typos |
| Probabilistic (Fellegi-Sunter) | Weighs agreement and disagreement on each field by how informative it is | Handles messy data, can be trained without labels | Needs careful field choices and tuning |
| Machine learning classifiers | Learn match decisions from labeled pairs | High accuracy with good training data | Labeled data takes time to build |
| Embeddings | Compare meaning, not spelling, using vector similarity | Catches synonyms and different wording | Can match things that are similar but not the same |
| Language models | Read two records and judge whether they describe the same entity | Strong on ambiguous text and unstructured sources | Slower and costlier per pair, need evaluation |
The probabilistic approach dates back to Ivan Fellegi and Alan Sunter's 1969 theory of record linkage and remains the basis of widely used open-source tools.
How language models help
Language models are useful in two places. First, extraction: they pull structured fields, such as company names, product names and versions, out of articles, emails and documents, so unstructured sources can enter the matching process. Second, adjudication: for the small share of pairs that scores cannot decide, a model can read both records and explain whether they match. Running a model on every pair is expensive and unnecessary, so keep rules and scores for the easy cases and send only the uncertain ones to the model.
Tools to consider
| Tool | Type | Notes |
|---|---|---|
| Splink | Open source, Python | Probabilistic linkage based on Fellegi-Sunter, runs on DuckDB, Spark and other backends |
| dedupe | Open source, Python | Active learning: asks a person to label a few pairs, then learns |
| Zingg | Open source | Machine learning entity resolution on Spark |
| AWS Entity Resolution | Managed service | Rule-based and machine learning matching within AWS |
| Custom pipeline | Your code | Fits when entities are specific to your domain, such as vulnerabilities or legal parties |
According to its documentation, Splink can link about a million records on a laptop in around a minute and scale to over 100 million records on Spark, and it is used by public bodies including the UK Office for National Statistics and NHS England.
Duplicate customers, suppliers or products spread across systems? We build entity resolution pipelines that merge them into records you can trust.
Explore staff augmentationHow to measure entity resolution quality
- Precision. Of the pairs the system matched, how many are true matches? Low precision means false merges.
- Recall. Of the true matches, how many did the system find? Low recall means duplicates remain.
- A labeled sample. Label a few hundred pairs, including hard cases near the decision threshold, and measure against them after every change.
- Cluster checks. Review the largest clusters. An entity with thousands of linked records often means one bad rule merged unrelated records.
Decide which error costs more. Merging two different customers can send one person's data to another, while missing a duplicate usually means a second marketing email. Most teams set thresholds to favor precision and send borderline pairs to review.
Entity resolution in a threat intelligence pipeline
A US cybersecurity media company needed a platform that collects news and advisories from hundreds of open sources and turns them into structured data about vendors, products and security events. The same vendor or vulnerability appears in many articles under different names, and the same story can appear at several addresses.
A .wrk team of six to eight engineers built resolution into the pipeline.
- Deduplication before extraction. Items are matched by URL and by a hash of their content, so the same story is processed once.
- Language model extraction. Models pull vendors, products and events out of each article into structured fields.
- Entity resolution into a graph. A graph worker resolves each mention to an existing entity or creates a new one, linking vendors, products and security events.
- Temporal versioning. Entities keep earlier versions, so the team can see how information about a vendor or product changed over time.
- Idempotent processing. A processing ledger and dead-letter queue make reprocessing safe, so a repeated article updates entities instead of duplicating them.
The structured data now powers a public security events section on the company's website. Read the NEXUS case study.

Entity resolution for CRM and marketing data
The same principles apply to customer data, where the usual problem is the same person or company entered several times with different spellings and job titles. Our guide to CRM data cleansing covers the standardization step for B2B fields such as company size, industry and role, and our article on medallion architecture shows where resolved entities sit in a data platform. For pipelines that must survive retries and replays, see our guide to idempotency.
Common mistakes
- Matching before standardizing. Inconsistent formats hide true matches and create false ones.
- No blocking. Comparing everything with everything works in a test and fails in production.
- One threshold for everything. Different entity types and sources need different thresholds.
- Merging without keeping source links. Once records are merged without links, a wrong merge cannot be undone.
- No human review. A small review queue for uncertain pairs improves both quality and the training data for the next version.
- Never re-measuring. New sources and data drift change quality. Measure on a schedule.
Frequently asked questions
What is entity resolution?
Entity resolution is the process of identifying records that refer to the same real-world entity, such as a person, company or product, across one or more datasets, and linking or merging them into one entity, even when the records differ in spelling, format or identifiers.
What is the difference between entity resolution and deduplication?
Deduplication removes duplicate records within one dataset. Entity resolution is broader: it links records within and across datasets, groups them into entities and usually builds a canonical record while keeping links to the sources.
What are common entity resolution techniques?
Common techniques include deterministic rules, probabilistic matching based on the Fellegi-Sunter model, machine learning classifiers trained on labeled pairs, embedding similarity and language models for ambiguous cases. Most systems combine them with blocking and clustering.
Can large language models do entity resolution?
Language models are useful for extracting entities from unstructured text and for deciding difficult pairs. Using them for every comparison is slow and expensive, so they work best alongside rules and similarity scores that handle the easy cases.
What tools are used for entity resolution in Python?
Popular Python tools include Splink for probabilistic record linkage and dedupe for active learning. Zingg runs on Spark, and managed services such as AWS Entity Resolution suit teams that prefer not to run their own pipeline.
To sum up
Entity resolution turns scattered, inconsistent records into entities you can trust. Standardize first, block to keep comparisons manageable, score pairs on several fields, cluster carefully, keep links to every source and measure precision and recall on labeled examples. Use language models where text is unstructured or pairs are ambiguous, and rules and scores everywhere else.
.wrk builds data pipelines, entity resolution and data platforms for companies in the US and Europe, with engineers who join your team or deliver the pipeline end to end.
Need one trusted view of your customers, suppliers or products? Tell us about your sources, and we will propose an entity resolution approach and timeline.
Explore staff augmentation