Often you may want to use PowerShell to find duplicate lines in a file.
You can use the following methods to do so:
Method 1: Display Duplicate Lines in File
Get-Content cities.txt | Group-Object | Where-Object { $_.Count -gt 1 } | Select -Expand Name
This particular example will display the lines that occur more than once in the file named cities.txt.
Method 2: Display Duplicate Lines with Occurrences in File
Get-Content cities.txt | Group-Object | Where-Object { $_.Count -gt 1 } | Select Name, Count
This particular example will display the lines that occur more than once along with their number of occurrences in the file named cities.txt.
Note that in both examples we use the Where-Object cmdlet to filter the lines of the file to only display the lines that have a count greater than ( -gt ) 1.
The following examples show how to use each method in practice with the file named cities.txt, which contains the names of various U.S. cities:
Example 1: Display Duplicate Lines in File
We can use the following syntax to display the duplicate lines (the lines that occur more than once) in the cities.txt file:
Get-Content cities.txt | Group-Object | Where-Object { $_.Count -gt 1 } | Select -Expand Name
The following screenshot shows how to use this syntax in practice:
This returns the following lines from the file:
- Dallas
- Miami
These are the two lines that occur more than once in the file.
Example 2: Display Duplicate Lines with Occurrences
We can use the following syntax to display the duplicate lines along with their number of occurrences in the cities.txt file:
Get-Content cities.txt | Group-Object | Where-Object { $_.Count -gt 1 } | Select Name, Count
The following screenshot shows how to use this syntax in practice:
From the output we can see that “Dallas” occurs 3 times and “Miami” occurs 2 times in the file.
Related Tutorials
The following tutorials explain how to perform other common tasks in PowerShell:
PowerShell: How to Count Specific Words in a File
PowerShell: How to Find Duplicate Values in Array
PowerShell: How to Remove Duplicate Values from Array
