Remove duplicate lines
Remove repeated lines and keep the first occurrence, in the original order.
Runs in your browser. Your input is not uploaded.
What it does
Removing duplicates keeps the first time each line appears and drops later copies. The remaining lines stay in their original order, unlike sort -u, which sorts as well.
How to use it
- Paste the lines.
- Choose whether case matters and whether to trim spaces before comparing.
- Copy the result. The count of removed lines is shown under it.
Example
Input
apple banana apple Apple
Output
apple banana
Things to watch for
- With case-insensitive matching, the first spelling wins:
Appleis removed ifapplecame first. - Blank lines are removed as duplicates of each other unless you tick Keep blank lines.
Questions
- Does it keep the original order?
- Yes. Use Sort lines if you also want the list sorted.
- What is the command-line equivalent?
awk '!seen[$0]++'keeps the first occurrence in order.sort -uremoves duplicates but also sorts.- Are trailing spaces ignored?
- Only when Trim is ticked. Otherwise
aandaare different lines. - Is there a size limit?
- No fixed limit. Hundreds of thousands of lines work; the practical limit is your browser’s memory.