This tutorial explains Linux “uniq” command, options and its usage with examples.
uniq prints the unique lines in a sorted file, retaining only one of a run of matching lines. Optionally, it can show only lines that appear exactly once, or lines that appear more than once. uniq requires sorted input since it compares only consecutive lines.
uniq [OPTION]… [INPUT [OUTPUT]]
Only print unique lines.
Only print duplicate lines.
Print the number of times each line occurred along with the line.
-number, -f, –skip-fields=number
In this option, number is an integer representing the number of fields to skip over before checking for uniqueness. The first number fields, along with any blanks found before number fields is reached, are skipped over and not counted. Fields are defined as a strings of non-space, non-tab characters, that are separated from each other by spaces and tabs.
+number, -s, –skip-chars=number
In this option, number is an integer representing the number of characters to skip over before checking for uniqueness. The first number characters, along with any blanks found before number characters is reached, are skipped over and not counted. If you use both the field and character skipping options, fields are skipped over first.
Specify the number of characters to compare in the lines, after skipping any specified fields and characters. Normally the entire rest of the lines are compared.
Print a usage message and exit with a status code indicating success.
Print version information on standard output then exit.
The following test file is used in some of the example to understand how uniq command works.
$ cat test aa aa bb bb bb xx
1. uniq command run without any option
It removes duplicate lines and displays unique lines as shown below.
$ uniq test aa bb xx
2. Print only Duplicate Lines using -d option
This option is to print only duplicate repeated lines in file. As you see below, this didn’t display the line “xx”, as it is not duplicate in the test file.
$ uniq -d test aa bb
3. Print the duplicated lines original number of times
The above example displayed all the duplicate lines, but only once. But, this -D option will print all duplicate lines in file.
$ uniq -D test aa aa bb bb bb
4. Count Number of Occurrences using -c option
This option is to count occurrence of lines in file.
$ uniq -c test 2 aa 3 bb 1 xx
5. Print only Unique Lines using -u option
$ uniq -u test xx
6. Limit Comparison to ‘N’ characters using -w option
For this example, use the following test input file.
$ cat test hi Linux hi LinuxU hi LinuxUnix hi Unix
The following uniq command using option ‘w’ is compares first 8 characters of lines in file, and then using ‘D’ option prints all duplicate lines of file.
$ uniq -D -w 8 test hi Linux hi LinuxU hi LinuxUnix
7. Avoid Comparing first ‘N’ Characters using -s option
For this example, use the following test3 input file.
$ cat test aabb xxbb bbc bbd
The following uniq command using option ‘s’ skips comparing first 2 characters of lines in file, and then using ‘D’ option prints all duplicate lines of file.
$ uniq -D -s 2 test3 aabb xxbb
8. Avoid Comparing first ‘N’ Fields using -f option
Consider the following file
$ cat test2 hi hello Linux hi friend Linux hi hello LinuxUnix
The following uniq command using option ‘f’ skips comparing first 2 fields of lines in file, and then using ‘D’ option prints all duplicate lines of file.
$ uniq -D -f 2 test2 hi hello Linux hi friend Linux
Sanfoundry Global Education & Learning Series – 1000 Linux Tutorials.