Bash: How to Find Duplicate Lines in File


Often you may want to use Bash to find duplicate lines in a file.

You can use the following methods to do so:

Method 1: Display Duplicate Lines in File

sort cities.txt | uniq -d

This particular example will display the lines that occur more than once in the file named cities.txt.

Method 2: Display Duplicate Lines with Occurrences in File

sort cities.txt | uniq -dc

This particular example will display the lines that occur more than once along with their number of occurrences in the file named cities.txt.

Here is what the various commands do:

  • uniq: Extracts only unique lines from the file
  • -d: Only outputs lines that are repeated in the file
  • -c: Displays the number of occurrences of each line in the file

The following examples show how to use each method in practice with the file named cities.txt, which contains the following content:

The file contains 13 total lines in which each line contains the name of a particular U.S. city.

Example 1: Display Duplicate Lines in File Using Bash

We can use the following syntax to display the duplicate lines (the lines that occur more than once) in the cities.txt file:

sort cities.txt | uniq -d

The following screenshot shows how to use this syntax in practice:

Bash find duplicate lines in file

This returns the following lines from the file:

  • Houston
  • Miami

These are the two lines that occur more than once in the file.

Example 2: Display Duplicate Lines with Occurrences Using Bash

We can use the following syntax to display the duplicate lines along with their number of occurrences in the cities.txt file:

sort cities.txt | uniq -dc

The following screenshot shows how to use this syntax in practice:

Bash find duplicate lines with occurrences

This returns the following output:

  • 3 Houston
  • 2 Miami

This tells us that “Houston” occurs 3 times and “Miami” occurs 2 times in the file.

Related Tutorials

The following tutorials explain how to perform other common tasks in Bash:

Bash: How to List All Files in Folder by Size
Bash: How to List All Files in Folder Without Extension
Bash: How to List All Files Matching a Specific Pattern

Leave a Reply