Often you may want to use Bash to find duplicate lines in a file.
You can use the following methods to do so:
Method 1: Display Duplicate Lines in File
sort cities.txt | uniq -d
This particular example will display the lines that occur more than once in the file named cities.txt.
Method 2: Display Duplicate Lines with Occurrences in File
sort cities.txt | uniq -dc
This particular example will display the lines that occur more than once along with their number of occurrences in the file named cities.txt.
Here is what the various commands do:
- uniq: Extracts only unique lines from the file
- -d: Only outputs lines that are repeated in the file
- -c: Displays the number of occurrences of each line in the file
The following examples show how to use each method in practice with the file named cities.txt, which contains the following content:
The file contains 13 total lines in which each line contains the name of a particular U.S. city.
Example 1: Display Duplicate Lines in File Using Bash
We can use the following syntax to display the duplicate lines (the lines that occur more than once) in the cities.txt file:
sort cities.txt | uniq -d
The following screenshot shows how to use this syntax in practice:
This returns the following lines from the file:
- Houston
- Miami
These are the two lines that occur more than once in the file.
Example 2: Display Duplicate Lines with Occurrences Using Bash
We can use the following syntax to display the duplicate lines along with their number of occurrences in the cities.txt file:
sort cities.txt | uniq -dc
The following screenshot shows how to use this syntax in practice:
This returns the following output:
- 3 Houston
- 2 Miami
This tells us that “Houston” occurs 3 times and “Miami” occurs 2 times in the file.
Related Tutorials
The following tutorials explain how to perform other common tasks in Bash:
Bash: How to List All Files in Folder by Size
Bash: How to List All Files in Folder Without Extension
Bash: How to List All Files Matching a Specific Pattern
