Toolsel

Remove Duplicate Lines

Runs in your browser

Strip duplicate lines from a list while keeping the order you choose.

Report issue

Input

Result

What is a Remove Duplicate Lines?

A duplicate line remover is useful for exactly the kind of line-based data that piles up messily in real work: repeated lines in a block of text, a list merged from two sources, URLs pulled from a crawl or backlink report, email addresses collected from more than one signup form, keyword lists exported from research tools, a single column copied from a spreadsheet, or log lines that repeat on every retry. This tool compares each line against the others using the settings you choose, keeps one copy of every distinct line, and discards the rest — the surviving lines stay in their original order unless you turn sorting on. Everything runs in your browser; nothing you paste is sent anywhere.

Two settings control what counts as a duplicate. Case-sensitive comparison, on by default, treats "Apple" and "apple" as different lines — turn it off if you want matches regardless of letter case, which is often what you want for URLs, emails, or tags. Trim whitespace, also on by default, ignores leading and trailing spaces when comparing, so a line with an invisible trailing space still matches its otherwise-identical twin instead of surviving as a false "unique" line.

Two more settings control what the output looks like rather than what counts as a match. Keep last occurrence keeps the text of the most recent duplicate instead of the first one found, useful when a later entry in your list is the more up-to-date version — the line still appears where the first occurrence was, only its content changes. Sort alphabetically re-orders the final, deduplicated list; leave it off to preserve your original ordering.

Where can I use a duplicate line remover?

Use caseWhy deduplicate it
URL listsCrawl exports, backlink reports, and scraped link lists often repeat the same URL.
Email listsContact exports pulled from more than one signup form or system commonly repeat an address.
Keyword listsKeyword research tools frequently export overlapping terms across reports.
Spreadsheet columnsCopy a single column from a spreadsheet — each cell becomes its own line — to spot repeated entries.
LogsRepeated log lines from retries or polling can be collapsed down to the distinct events.
General text listsTags, IDs, or names where re-pasting or merging two files introduced repeats.

How to use it

  1. Paste your list, one item per line.
  2. Toggle case-sensitive comparison and whitespace trimming to match what should count as a duplicate.
  3. Choose whether to keep the first or last occurrence of each duplicate.
  4. Optionally sort the result alphabetically.
  5. Copy the deduplicated list — the count of lines removed is shown above the result.

Examples

Default settings

Input
apple
banana
Apple
apple
cherry
Output
apple
banana
Apple
cherry

With case-insensitive matching

Input
apple
banana
Apple
apple
cherry
Output
apple
banana
cherry

Frequently asked questions

What counts as a duplicate by default?
Exact, case-sensitive matches after trimming leading and trailing whitespace from each line — that combination catches the most common real-world case (a re-pasted list with inconsistent spacing) without silently merging lines that are genuinely different, like differently-cased words that matter in your context.
Does the tool preserve my original order?
Yes, by default. Each surviving line stays at the position of its first occurrence; only turning on "Sort alphabetically" changes the order.
What does "keep last occurrence" do?
Instead of keeping the text from the first time a line appears, it keeps the text from the last matching line found. The surviving line still appears at the position of the first occurrence — only its exact content is replaced with the later version. Useful when a later duplicate is the more up-to-date entry, such as an updated row in a merged list.
Can I remove duplicate URLs, emails, or keywords?
Yes — the tool works on any plain-text list, one item per line, so a list of URLs, email addresses, or keywords dedupes exactly the same way a word list does. It doesn't validate that a line is a properly-formed URL or email; it only compares each line's text against the others.
Are blank lines treated as duplicates of each other?
Yes — every empty line is identical to every other empty line, so with default settings only the first (or last) blank line survives, the rest are removed along with any other duplicate.
Are my pasted lines uploaded anywhere?
No. The comparison and deduplication happen entirely in your browser — nothing you paste is sent to a server.