uniq Command in Linux with Examples

The uniq command in Linux eliminates duplicate lines from a file. It prints only the unique lines from a sorted file, preserving only one of a sequence of matching lines. Optionally, it can display only lines that appear exactly once or lines that appear multiple times.

Syntax:
The basic syntax of the uniq command in Linux is:

uniq [OPTION]... [INPUT [OUTPUT]]

• Option: Various options that modify the behavior of uniq.
• Input: The file containing the lines to be processed. If not specified, uniq reads from standard input (stdin).
• Output: The file where the processed data will be written. If not specified, uniq writes to standard output (stdout).

Options:

• -u, –unique: Only print unique lines.
• -d, –repeated: Only print duplicate lines.
• -c, –count: Print the number of times each line occurred along with the line.
• -number, -f, –skip-fields=number: It specifies the number of fields to skip before checking for uniqueness. These fields, along with any preceding blank spaces, are ignored. Fields are defined as strings of non-whitespace characters separated by spaces or tabs.
• +number, -s, –skip-chars=number: It specifies the number of characters to skip before checking for uniqueness. These characters, along with any preceding blank spaces, are ignored. If both field and character skipping are used, fields are skipped first.
• -w, –check-chars=number: It specifies the number of characters to compare in each line, after skipping any specified fields and characters. By default, the entire remaining portion of each line is compared.
• –help : It displays a usage message and exits with a success code.
• –version: Prints version information to the standard output and then exits.

uniq Command Examples:

Note:
The following test file is used in some of the example to understand how uniq command works.

$ cat test
aa
aa
bb
bb
bb
xx

Example 1: uniq command without any option

$ uniq test

Running uniq without any option on the test file removes duplicate lines and displays only the unique lines.

Output:

aa
bb
xx

Example 2: Print only Duplicate Lines using -d option

$ uniq -d test

This option is to print only duplicate repeated lines in file. As you see below, this didn’t display the line “xx”, as it is not duplicate in the test file.

Output:

aa
bb

Example 3: Print the duplicated lines original number of times

$ uniq -D test

The -D option displays all duplicate lines in the file as they appear (with their original count).

Output:

aa
aa
bb
bb
bb

Example 4: Count Number of Occurrences using -c option

$ uniq -c test

The -c option counts the occurrences of each line in the file and displays them along with the line.

Output:

      2 aa
      3 bb
      1 xx

Example 5: Print only Unique Lines using -u option

$ uniq -u test

The -u option prints only the lines that are unique (non-duplicate).

Output:

xx

Example 6: Limit Comparison to ‘N’ characters using -w option

For this example, use the following test input file.

$ cat test
hi Linux
hi LinuxU
hi LinuxUnix
hi Unix

The following uniq command using option ‘w’ is compares first 8 characters of lines in file, and then using ‘D’ option prints all duplicate lines of file.

$ uniq -D -w 8 test

Output:

hi Linux
hi LinuxU
hi LinuxUnix

Example 7: Avoid Comparing first ‘N’ Characters using -s option

For this example, use the following test3 input file.

$ cat test
aabb
xxbb
bbc
bbd

The following uniq command using option ‘s’ skips comparing first 2 characters of lines in file, and then using ‘D’ option prints all duplicate lines of file.

$ uniq -D -s 2 test3

Output:

aabb
xxbb

Example 8: Avoid Comparing first ‘N’ Fields using -f option

Consider the following file

$ cat test2
hi hello Linux
hi friend Linux
hi hello LinuxUnix

The following uniq command using option ‘f’ skips comparing first 2 fields of lines in file, and then using ‘D’ option prints all duplicate lines of file.

$ uniq -D -f 2 test2

Output:

hi hello Linux
hi friend Linux

Sanfoundry Global Education & Learning Series – 1000 Linux Tutorials.