Every team that touches job data writes a deduplicator, and every one of them underestimates it. The naive version takes an afternoon and is wrong in ways you only discover months later, in aggregate numbers you have already reported.
Why exact matching fails
The same role reaches different boards through different pipelines. Titles get suffixes: Senior Engineer, Senior Engineer (Remote), Senior Engineer - Berlin. Descriptions get truncated at different lengths. HTML gets stripped differently. Salary appears in one copy and not another.
Hashing the description finds the small share of copies that travelled without modification. Everything else slips through.
Why fuzzy matching over-merges
Loosen the criteria and you merge things that should stay apart:
- A company hiring three Backend Engineers in one city — three requisitions, near-identical text.
- The same title at the same company in different offices, which is genuinely different work.
- A role reposted after six months, which is a new requisition, not a duplicate.
- Agency and employer versions of one role, where merging loses the fact that an agency is involved.
What actually works
A composite key, applied in order, with the strongest evidence first:
- Source identity. If two records share an applicant tracking system id, they are the same posting. Free and certain when available.
- Apply URL. Normalise it — strip tracking parameters — and identical destinations are the same role.
- Employer + normalised title + normalised location. Strong, but needs a tie-break for genuine multi-seat requisitions.
- Description similarity. Shingled hashing over the description, as confirmation rather than as primary evidence.
- Time window. Two matching records six months apart are two requisitions, not one.
The multi-seat problem
There is no clean answer to a company advertising four identical seats. Merging understates demand; keeping them separate overstates it if they are actually one posting duplicated by a board. The honest approach is to pick a rule, publish it, and stay consistent — so at least your series is comparable with itself.
Ours: identical employer, title and location inside a short window collapse to one record, with every source retained. It undercounts multi-seat hiring. We would rather undercount predictably than overcount invisibly.
How to know yours is wrong
Take one large employer you can verify by hand. Count what your pipeline says they have open. Count what their careers page says. If those differ by more than a few percent, the difference is your deduplication, and it is applying to every number you produce.