# List Deduplicator & Sorter

Canonical URL: https://offlineutils.com/list-deduplicator
Category: Text & Content
Published: 2026-06-24
Publisher: offlineutils.com
Runs entirely in the browser; nothing entered into the tool is uploaded.

> A List Deduplicator removes duplicate lines from a pasted list, optionally trims whitespace, normalises case for comparison, and sorts the result alphabetically, by length, or in reverse — all in the browser using native JavaScript Set and Intl.Collator.

Paste a list (one item per line) — email addresses, log entries, research notes, anything. Toggle case-insensitive matching, whitespace trimming, and empty-line removal. Pick a sort order and the cleaned list updates instantly with a summary of how many duplicates were removed. The dedupe runs in O(N) via a Set; sorting uses the locale-aware Intl.Collator for natural ordering.

## Features

- O(N) deduplication via native Set
- Locale-aware sorting via Intl.Collator (natural order)
- Case-insensitive matching toggle
- Whitespace-trim and empty-line filters
- Sort options: A–Z, Z–A, shortest, longest, shuffle (CSPRNG)
- Summary line showing how many duplicates were removed

## Use cases

- Cleaning a list of email addresses pulled from multiple sources
- Deduplicating tags pasted from a CMS export
- Alphabetizing an ingredient list before grouping a shopping run
- Shuffling a list to randomize a draw or assignment

## How to use it

1. **Paste your list** — Paste a list with one item per line.
2. **Configure cleaning** — Toggle case-insensitive matching, whitespace trimming, and empty-line removal.
3. **Pick a sort** — A–Z, Z–A, length-based, or shuffle. Or leave it off to keep original order.
4. **Copy the cleaned list** — Click Copy to grab the deduplicated, sorted result.

## When to use it, and when to use something else

- **vs `sort -u` or `awk '!seen[$0]++'` in the terminal** — use List Deduplicator & Sorter when you want a paste-and-go UI with case-insensitive toggles. Use `sort -u` or `awk '!seen[$0]++'` in the terminal when you are scripting and want this in a pipeline.
- **vs Excel 'Remove Duplicates'** — use List Deduplicator & Sorter when the list is one column of plain text. Use Excel 'Remove Duplicates' when the data is already in a multi-column spreadsheet.

## Frequently asked questions

### Is the original order preserved when I don't sort?

Yes. With sorting off, deduplication keeps the first occurrence of each line in original order — the same behavior as awk '!seen[$0]++' from a shell.

### How does case-insensitive matching work?

When enabled, 'Apple' and 'apple' are treated as duplicates. The output uses the casing of the first occurrence.

### Can I sort by length or randomise?

Yes — sort options include alphabetical (locale-aware), reverse alphabetical, shortest-first, longest-first, and shuffle (uses crypto.getRandomValues for an unbiased random order).

## Related tools

- https://offlineutils.com/case-converter
- https://offlineutils.com/word-counter
- https://offlineutils.com/text-diff