I was validating 442 translated titles before writing them to a database, and the duplicate check lied to me. I concluded the file was corrupt. The file was fine.
The check reported 397 duplicates that did not exist
Here is the uniq pipeline I ran against the list of Japanese titles:
cut -f2 titles.tsv | sort | uniq -c | sort -rn | head -3 397 街で人を眺める
12 夜に1時間本を読む
8 朝5時に起きる397 identical titles out of 442 meant the translation had collapsed. So I counted that exact string directly:
grep -c '街で人を眺める' titles.tsv1One occurrence. Loading the file into a Python set() gave 442 distinct values. The data was correct; the counter was not.
The culprit is uniq, not sort
My first guess was sort, some ordering quirk pushing unrelated lines together. But sort prints them just fine:
printf '가나다\n라마바\n사아자\n' > k.txt
sort k.txt가나다
라마바
사아자Pipe those same three lines into uniq -c:
sort k.txt | uniq -c 3 가나다Three distinct lines, collapsed into one group. uniq compares adjacent lines, and that comparison is locale-aware. My shell had:
locale | grep LC_COLLATELC_COLLATE="en_US.UTF-8"The comparison skips the Korean entirely
To find the rule, I fed in controlled inputs. Every line in each row is a distinct string; the numbers are how many groups uniq produced.
| Input lines | Default locale | LC_ALL=C |
|---|---|---|
가나다 라마바 사아자 | 1 | 3 |
가나다1 라마바2 사아자3 | 3 | 3 |
a가나다 a라마바 a사아자 | 1 | 3 |
가a나 나a가 | 1 | 2 |
apple banana cherry | 3 | 3 |
Row three is the tell. a가나다 and a라마바 share only the leading a, and everything after it differs, yet they compare equal. The Korean characters carry no collation weight in this locale, so they are simply not read. Row four makes it blunt: 가a나 and 나a가 both reduce to a.
That also explains row two. The digits 1 2 3 do have weight, so they alone separate the lines. And it explains my file: those 397 titles contained no ASCII at all, so they all reduced to the empty key. The other 45 happened to contain a digit like 1 or 5, which is the only thing that split them.
It is not just Korean
| Input lines | Default locale | LC_ALL=C |
|---|---|---|
日本 中国 | 1 | 2 |
αβγ δεζ | 1 | 2 |
Привет Мир | 1 | 2 |
🍎 🍊 | 1 | 2 |
café cafe | 2 | 2 |
straße strasse | 2 | 2 |
Han characters, Greek, Cyrillic, and emoji all vanish from the comparison. Accented Latin does not. café and cafe stay distinct, as do straße and strasse.
That asymmetry is why this never shows up in English-language pipelines. sort -u uses the same comparison and is wrong in the same way.
The fix is one line
cut -f2 titles.tsv | LC_ALL=C sort | LC_ALL=C uniq -dYou need it on both commands. LC_ALL=C replaces collation rules with byte comparison, so different strings compare as different. My file reported zero duplicates under it, which was the correct answer.
One caveat if you also need the sorted output: byte order is not dictionary order, so LC_ALL=C sort will not order non-Latin text the way a reader expects. Deduplication and human-facing ordering are different jobs, so use it for the former only.
What this cost
The whole problem is that nothing failed. uniq exits 0 and returns a plausible-looking number. I read 397, concluded the translation was broken, and came close to discarding correct output.
I have hit this shape before in the same repo: a type check that was silent rather than passing. There too the problem was not that the checker was wrong. It was that nobody had checked the checker.
I no longer do duplicate checks in the shell. The validator now counts with a dictionary inside the script that consumes the data. Shell pipelines are short and pleasant, but short is not a reason to trust one without checking it against a second method.
Why this is easy to miss
Nothing about the failure looks like a failure. There is no warning on stderr, no non-zero exit, no hint that a comparison was performed on a reduced version of your input. The number that comes back is a real count of real groups; it is just counting groups that were formed under a rule you did not know was in effect.
It is also invisible in every place you would normally look for it. The man page for uniq does not mention collation. The strings look correct when you print them, because printing does not go through the comparison. And the moment you reach for grep or a scripting language to confirm, you get the right answer, which makes the shell result look like corruption in the data rather than a defect in the pipeline.
That combination is what makes it expensive. A check that fails loudly costs you a minute. A check that returns a confident wrong number costs you however long you spend acting on it, and in my case that nearly meant throwing away 442 correct translations and regenerating them.
Scope
Verified on macOS 26.6 (build 25G72) with sort 2.3-Apple (199), in a default shell with LC_COLLATE=en_US.UTF-8.
I did not test Linux. GNU coreutils ships a different uniq and glibc ships different locale data, so the result may differ there. I also did not test other locales such as ko_KR.UTF-8. The one combination above is what I measured.