Remove duplicate lines

Remove repeated lines and keep the first occurrence, in the original order.

Runs in your browser. Your input is not uploaded.

What it does

Removing duplicates keeps the first time each line appears and drops later copies. The remaining lines stay in their original order, unlike sort -u, which sorts as well.

How to use it

  1. Paste the lines.
  2. Choose whether case matters and whether to trim spaces before comparing.
  3. Copy the result. The count of removed lines is shown under it.

Example

Input
apple
banana
apple
Apple
Output
apple
banana

Things to watch for

  • With case-insensitive matching, the first spelling wins: Apple is removed if apple came first.
  • Blank lines are removed as duplicates of each other unless you tick Keep blank lines.

Questions

Does it keep the original order?
Yes. Use Sort lines if you also want the list sorted.
What is the command-line equivalent?
awk '!seen[$0]++' keeps the first occurrence in order. sort -u removes duplicates but also sorts.
Are trailing spaces ignored?
Only when Trim is ticked. Otherwise a and a are different lines.
Is there a size limit?
No fixed limit. Hundreds of thousands of lines work; the practical limit is your browser’s memory.