Bash: How to Count Number of Unique Lines in File


Often you may want to use Bash to count the number of unique lines in a specific file.

You can use the following methods to do so:

Method 1: Count Total Unique Lines in File

sort cities.txt | uniq -c | wc -l

This particular example will return the total number of unique lines in the file named cities.txt.

Method 2: Count Occurrences of Each Unique Line in File

sort cities.txt | uniq -c | sort -bgr

This particular example will return the count of each unique line in the file named cities.txt.

The following examples show how to use each method in practice with the file named cities.txt, which contains the following content:

The file contains 13 total lines in which each line contains the name of a particular U.S. city.

Example 1: Count Total Unique Lines in File Using Bash

We can use the following syntax to count the total number of unique lines in the file:

sort cities.txt | uniq -c | wc -l

The following screenshot shows how to use this syntax in practice:

Bash count unique lines in file

This returns a value of 10, which tells us that there are 10 unique lines in the file.

Note that if you simply wanted to know the total number of lines in the file, you could use the following syntax instead:

sort cities.txt | wc -l

Note: The wc command stands for “word count” and the -l operator specifies that we would like to count the number of “lines” in the file.

Example 2: Count Occurrences of Each Unique Line in File Using Bash

We can use the following syntax to count the total number of occurrences of each unique line in the file:

sort cities.txt | uniq -c | sort -bgr

The following screenshot shows how to use this syntax in practice:

Bash count occurrences of each unique line in file

This returns the count of each unique line in the file.

For example, we can see:

  • The line with the text Houston occurs 3 times in the file.
  • The line with the text Miami occurs 2 times in the file.
  • The line with the text Sacramento occurs 1 time in the file.

And so on.

Related Tutorials

The following tutorials explain how to perform other common tasks in Bash:

Bash: How to List All Files in Folder by Size
Bash: How to List All Files in Folder Without Extension
Bash: How to List All Files Matching a Specific Pattern

Leave a Reply