PowerShell: How to Find Duplicate Lines in File


Often you may want to use PowerShell to find duplicate lines in a file.

You can use the following methods to do so:

Method 1: Display Duplicate Lines in File

Get-Content cities.txt | Group-Object | Where-Object { $_.Count -gt 1 } | Select -Expand Name

This particular example will display the lines that occur more than once in the file named cities.txt.

Method 2: Display Duplicate Lines with Occurrences in File

Get-Content cities.txt | Group-Object | Where-Object { $_.Count -gt 1 } | Select Name, Count 

This particular example will display the lines that occur more than once along with their number of occurrences in the file named cities.txt.

Note that in both examples we use the Where-Object cmdlet to filter the lines of the file to only display the lines that have a count greater than ( -gt ) 1.

The following examples show how to use each method in practice with the file named cities.txt, which contains the names of various U.S. cities:

Example 1: Display Duplicate Lines in File

We can use the following syntax to display the duplicate lines (the lines that occur more than once) in the cities.txt file:

Get-Content cities.txt | Group-Object | Where-Object { $_.Count -gt 1 } | Select -Expand Name 

The following screenshot shows how to use this syntax in practice:

PowerShell display duplicate lines in file

This returns the following lines from the file:

  • Dallas
  • Miami

These are the two lines that occur more than once in the file.

Example 2: Display Duplicate Lines with Occurrences

We can use the following syntax to display the duplicate lines along with their number of occurrences in the cities.txt file:

Get-Content cities.txt | Group-Object | Where-Object { $_.Count -gt 1 } | Select Name, Count

The following screenshot shows how to use this syntax in practice:

PowerShell find duplicate lines and count occurrences in file

From the output we can see that “Dallas” occurs 3 times and “Miami” occurs 2 times in the file.

Related Tutorials

The following tutorials explain how to perform other common tasks in PowerShell:

PowerShell: How to Count Specific Words in a File
PowerShell: How to Find Duplicate Values in Array
PowerShell: How to Remove Duplicate Values from Array

Leave a Reply