Remove Duplicate Lines — Remove Duplicate Lines from Text
Remove duplicate lines from text. Case-sensitive or case-insensitive, keep first or last occurrence — runs in your browser, no upload.
Remove Duplicate Lines
Remove duplicate lines from text. Works entirely in your browser.
Duplicate lines are the silent noise in every data pipeline. A CSV export that accumulated redundant rows across multiple runs. A log file that repeated the same error a thousand times. A list of email addresses where the same entry was pasted twice. Manual deduplication — scanning thousands of lines by eye — is slow and error-prone. The line deduplicator eliminates that noise in one operation and tells you how many duplicates it removed so you know the scope of the problem.
The distinction between keep-first and keep-last matters more often than it seems. Keep-first preserves the earliest occurrence, which is correct for chronologically accumulated data where the first entry is the original. Keep-last preserves the most recent occurrence, which is correct for lists that have been updated over time and the newest version of a row supersedes the old one. The case-insensitive toggle is for the common situation where the same data was entered with inconsistent casing — ‘John Smith’ and ‘john smith’ in an email list that was manually typed.
The line-count statistics are what turn this from a black-box operation into a transparent one. The tool shows the input line count, the output line count, and the number of duplicates removed. A 10,000-line file that produces 3,000 unique lines tells you that 70% of the file was redundant — which is worth investigating upstream. A 10,000-line file that produces 9,997 unique lines tells you that three rogue duplicates snuck in, and the deduplication was a safety net, not a cleanup. The statistics panel makes the deduplication auditable, which a simple sort -u command does not.
How to use
Paste your text
Paste or type multi-line text into the input. The tool scans every line, identifies duplicates, and removes them according to the chosen mode. Line count statistics update as the text changes.
Choose a duplicate mode
Keep first occurrence (default — the earliest instance survives) or keep last occurrence. Toggle case-insensitive mode to treat 'Hello' and 'hello' as duplicates. Toggle trim whitespace to ignore leading/trailing spaces during comparison.
Copy or download the result
Copy the deduplicated text to your clipboard, or download it as a .txt file. The removal is non-destructive — the original input is preserved in a separate panel for comparison.
Frequently asked
How are blank lines handled?
By default blank lines are treated as regular lines and deduplicated. Toggle 'preserve blank lines' to keep all blank lines regardless of deduplication. This is useful for text files where blank lines separate sections and removing them would collapse the structure.
Does line order matter for duplicate detection?
Yes. The tool preserves the original line order and removes duplicates based on that order. Keep-first mode deletes all subsequent occurrences of a line; keep-last mode deletes all earlier occurrences and keeps only the final one. The order of the surviving lines matches the order of the first or last occurrence.
Can I deduplicate based on a field within each line?
No — the tool compares entire lines. For field-level deduplication (e.g., CSV deduplication by the email column), convert to JSON with the CSV-to-JSON tool, deduplicate programmatically, and convert back. The line deduplicator is for flat text lists.
What counts as a 'line' in the input?
Lines are split on newline characters (LF, CRLF, or CR). A trailing newline at the end of the file does not count as an extra empty line. The line count shown in the statistics panel reflects the parsed line count, not the raw character count.
Will it handle very large files?
The deduplication is in-memory and scales to files with hundreds of thousands of lines in a modern browser tab. For million-line files, expect a brief pause during processing. The memory footprint is proportional to the number of unique lines.
Limitations
- Full-line comparison onlyField-level and partial-line deduplication are not supported. For deduplication by a specific column or substring, pre-process the text with a text editor or a script before using this tool.
- No fuzzy matchingLines are compared for exact equality only. Near-duplicates (typos, whitespace variations, encoding differences) are treated as distinct lines. Use a fuzzy-matching tool for approximate deduplication.
- Encoding is assumed to be UTF-8The tool treats the input as UTF-8. Files in other encodings may produce unexpected results if the same character is encoded differently in different sections of the file.
Platform notes
- macOS
- For command-line deduplication, `sort -u` removes duplicates but reorders lines. `awk '!seen[$0]++'` preserves order. The browser tool is the right pick for interactive deduplication with the keep-first/keep-last toggle.
- Linux
- The `uniq` command removes only adjacent duplicates. The browser tool detects duplicates anywhere in the text, not just adjacent lines. Combine `uniq` with `sort` for the same effect on the command line.
- Web
- Runs entirely client-side. Works offline once loaded.