uniq Command in Linux with Examples

Description :

uniq prints the unique lines in a sorted file, retaining only one of a run of matching lines. Optionally, it can show only lines that appear exactly once, or lines that appear more than once. uniq requires sorted input since it compares only consecutive lines.

Usage :

uniq [OPTION]… [INPUT [OUTPUT]]

Options :

-u, –unique
Only print unique lines.
-d, –repeated
Only print duplicate lines.
-c, –count
Print the number of times each line occurred along with the line.
-number, -f, –skip-fields=number
In this option, number is an integer representing the number of fields to skip over before checking for uniqueness. The first number fields, along with any blanks found before number fields is reached, are skipped over and not counted. Fields are defined as a strings of non-space, non-tab characters, that are separated from each other by spaces and tabs.
+number, -s, –skip-chars=number
In this option, number is an integer representing the number of characters to skip over before checking for uniqueness. The first number characters, along with any blanks found before number characters is reached, are skipped over and not counted. If you use both the field and character skipping options, fields are skipped over first.
-w, –check-chars=number
Specify the number of characters to compare in the lines, after skipping any specified fields and characters. Normally the entire rest of the lines are compared.
–help
Print a usage message and exit with a status code indicating success.
–version
Print version information on standard output then exit.

Examples :

Note:
The following test file is used in some of the example to understand how uniq command works.

$ cat test
aa
aa
bb
bb
bb
xx

1. uniq command run without any option

It removes duplicate lines and displays unique lines as shown below.

$ uniq test
aa
bb
xx

2. Print only Duplicate Lines using -d option

This option is to print only duplicate repeated lines in file. As you see below, this didn’t display the line “xx”, as it is not duplicate in the test file.

$ uniq -d test
aa
bb

3. Print the duplicated lines original number of times

The above example displayed all the duplicate lines, but only once. But, this -D option will print all duplicate lines in file.

$ uniq -D test
aa
aa
bb
bb
bb

4. Count Number of Occurrences using -c option

This option is to count occurrence of lines in file.

$ uniq -c test
      2 aa
      3 bb
      1 xx

5. Print only Unique Lines using -u option

$ uniq -u test
xx

6. Limit Comparison to ‘N’ characters using -w option

For this example, use the following test input file.

$ cat test
hi Linux
hi LinuxU
hi LinuxUnix
hi Unix

The following uniq command using option ‘w’ is compares first 8 characters of lines in file, and then using ‘D’ option prints all duplicate lines of file.

$ uniq -D -w 8 test
hi Linux
hi LinuxU
hi LinuxUnix

7. Avoid Comparing first ‘N’ Characters using -s option

For this example, use the following test3 input file.

$ cat test
aabb
xxbb
bbc
bbd

The following uniq command using option ‘s’ skips comparing first 2 characters of lines in file, and then using ‘D’ option prints all duplicate lines of file.

$ uniq -D -s 2 test3
aabb
xxbb

8. Avoid Comparing first ‘N’ Fields using -f option

Consider the following file

$ cat test2
hi hello Linux
hi friend Linux
hi hello LinuxUnix

The following uniq command using option ‘f’ skips comparing first 2 fields of lines in file, and then using ‘D’ option prints all duplicate lines of file.

$ uniq -D -f 2 test2
hi hello Linux
hi friend Linux