The hardest part of sanctions screening is not comparing strings — it is finding candidates when the query and the list entry share almost no characters. Here is the pipeline that does it, end to end, on a single PostgreSQL instance.
Normalise everything once
At load time every name is transliterated to Latin, lower-cased and stripped of diacritics; legal forms and stop-words are removed into a separate «key»; each name part is tagged with dictionary symbols — language-independent identifiers where the Russian «Ким», the English Kim and the Korean 김 resolve to the same symbol.
Retrieve through four channels
Identifiers first (a passport or IMO hit is definitive), then shared name symbols, then trigram similarity on the cleaned key, then phonetics as a fallback. Twenty candidates go to the scorer — raising that number past twenty bought us nothing on recall and doubled latency.
Score with logic-v2
We did not invent a scorer. logic-v2 from the nomenklatura library (MIT, the engine behind OpenSanctions) weighs name similarity across scripts, rewards exact identifiers and penalises conflicting birth years and countries. Every response carries the fired signals in a features object.
The numbers
On a public set of 400 generated cases: recall@5 0.99, zero false «likely» on clean names, median 66 ms, p95 142 ms — measured cold. Both misses are typos inside legal forms; they are listed on the methodology page.
The same pipeline powers every plan, including the free one.
