Tokens in C

Tokens are the basic parts of a C program. Understanding them is important for learning the language. In this tutorial, you will learn about different types of tokens, how they help define C’s rules, and see simple examples. By the end of this tutorial, you’ll have a clear grasp of tokens and how they contribute to writing structured C programs.

Contents:

1. What are Tokens in C?
2. Types of Tokens in C
3. Keywords in C
4. Identifiers in C
5. Constants in C
6. Strings in C
7. Operators in C
8. Special Symbols in C
9. How Tokens Define C’s Syntax
10. Example of Tokens in C
11. How Tokens Help in Code Compilation
12. FAQs on Tokens in C

What are Tokens in C?

Tokens are the smallest units in a C program that are meaningful to the compiler. They form the building blocks of a C program, allowing it to function as intended. In simple terms, every word and symbol in a C program is considered a token.

Types of Tokens in C

There are six types of tokens in C:

• Keywords – Reserved words with special meanings (e.g., int, return, if).
• Identifiers – Names for variables, functions, and arrays (e.g., sum, main).
• Constants – Fixed values that do not change (e.g., 10, ‘A’, 3.14).
• Strings – A sequence of characters inside double quotes (e.g., “Hello”).
• Operators – Symbols for performing operations (e.g., +, -, *, /).
• Special Symbols – Characters with special purposes (e.g., {}, ;, []).

Keywords in C

In C, keywords are reserved words that have predefined meanings and cannot be used as identifiers (variable names, function names, etc.). These words help define the structure and functionality of a C program.

List of Keywords in C:

auto       break      case       char       const
continue   default    do         double     else
enum       extern     float      for        goto
if         int        long       register   return
short      signed     sizeof     static     struct
switch     typedef    union      unsigned   void
volatile   while

Rules for Using Keywords:

• Keywords cannot be used as variable names, function names, or any other identifiers.
• They are case-sensitive in C.
• All keywords must be written in lowercase.

Example Using Keywords:

#include <stdio.h>
 
int main() {               // 'int' and 'main' are keywords
    int score = 90;        // 'int' is a keyword
    char grade = 'A';      // 'char' is a keyword
 
    if (score >= 50) {     // 'if' is a keyword
        printf("Passed\n");
    } else {               // 'else' is a keyword
        printf("Failed\n");
    }
 
    return 0;              // 'return' is a keyword
}

Output:

Passed

Identifiers in C

Identifiers are names used for variables, functions, and arrays. They are user-defined and must follow naming rules.

Rules for Identifiers:

• Can have letters (A-Z, a-z), numbers (0-9), and _ (underscore).
• Must start with a letter or an underscore.
• Cannot be a keyword (like int, return).
• Case-sensitive (Score and score are different).

Example:

int score = 85;  // 'score' is an identifier  
float point = 99.99;  // 'point' is an identifier

Constants in C

• Fixed values that do not change during the execution of a program.
• Can be of different types such as integer, float, character, or string.
• Syntax:

const int score = 100;  // Integer constant
const float marks = 95.5;     // Float constant
const char grade = 'A';       // Character constant

• Examples: 10, 3.14, ‘A’, “Hello”

Strings in C

A string is a sequence of characters enclosed in double quotes “…”. Strings end with a null character (\0).

Example:

#include <stdio.h>
 
int main() {
    char sanfoundry[20] = "C Programming"; // String initialization
 
    printf("Welcome to %s\n", sanfoundry);
 
    return 0;
}

Output:

Welcome to C Programming

Operators in C

Symbols that perform operations on operands.

Syntax:

int a = 5, b = 10;
int sum = a + b;  // '+' is an operator

Examples:

• Arithmetic Operators: +, -, *, /, %
• Relational Operators: ==, !=, >, <, >=, <=
• Logical Operators: &&, ||, !
• Bitwise Operators: – &, |, ^, ~, <<, >>
• Assignment Operators: – =, +=, -=, *=, /=, %=

Special Symbols in C

Special symbols, also known as punctuators, are symbols with a special meaning in C. They help define the structure of a program.

Common Special Symbols in C:

Example:

#include <stdio.h>  // '#' is a special symbol
 
int main()
{
    // '[]' for array, '{}' for initialization
    int arr[3] = {10, 20, 30};
 
    // '()', ';' are special symbols
    printf("Value: %d\n", arr[1]); 
    return 0;  // ';' ends the statement
}

How Tokens Define C’s Syntax

Tokens are important parts of a C program. They help define the rules of the language, so the compiler can understand and run the program correctly. Every C statement follows specific grammar rules made up of tokens like keywords, identifiers, constants, and operators.

For example:

int score = 10 + 20;

• int (Keyword) defines the variable type.
• score (Identifier) is the variable name.
• = (Operator) assigns a value.
• 10 and 20 (Constants) are numbers.
• + (Operator) adds the values.
• ; (Special Symbol) marks the end of the statement.

Each token must be used correctly to follow C’s syntax rules.

Example of Tokens in C

#include <stdio.h>          // Preprocessor Directive (Special Symbol and Header File)
 
int main() {                // Keyword, Special Symbols
    int sanfoundry = 10;    // Keyword, Identifier, Constant
    float marks = 95.5;     // Keyword, Identifier, Constant
    char grade = 'A';       // Keyword, Identifier, Constant (Character)
 
    // Operator and String in printf
    printf("Sanfoundry Marks: %d\n", sanfoundry);  
    printf("Marks obtained: %.2f\n", marks);       
    printf("Grade: %c\n", grade);                 
 
    // Conditional Statement with Operator and Identifier
    if (sanfoundry >= 50) {  
        printf("Result: Passed\n"); // String and Function
    } else {
        printf("Result: Failed\n");
    }
 
    return 0;    // Keyword and Constant
}

Output:

Sanfoundry Marks: 10
Marks obtained: 95.50
Grade: A
Result: Passed

Explanation:

• Keywords: int, float, char, if, else, return
• Identifiers: main, sanfoundry, marks, grade
• Constants: 10, 95.5, ‘A’, 0
• Strings: “Sanfoundry Marks: %d\n”, “Result: Passed\n”
• Operators: =, >=
• Special Symbols: {, }, ;, (), #

How Tokens Help in Code Compilation

Tokens are important when compiling a C program. The compiler processes them in four steps:

• Breaking the Code (Lexical Analysis) – The compiler scans the program and splits it into small parts (tokens).
• Checking the Rules (Syntax Analysis) – It checks if the tokens follow C’s grammar.
• Checking the Meaning (Semantic Analysis) – It makes sure the tokens are used correctly (e.g., the right data types).
• Creating Machine Code (Code Generation) – It converts tokens into machine code so the computer can run the program.

If tokens are not used correctly, the compiler gives errors. For example:

int 5score = 10; // A variable name cannot start with a number.

Tokens help structure the program correctly, so it compiles without errors.

FAQs on Tokens in C

1. What are tokens in C?
Tokens are the smallest units of a C program. The compiler breaks the source code into tokens to understand and process it.

2. What are the types of tokens in C?
There are six types of tokens in C: keywords, identifiers, constants, strings, operators, and special symbols. Each type plays a specific role in defining the syntax and behavior of a program.

3. Why are tokens important in C?
Tokens help define the syntax of a C program. The compiler uses them to understand and execute the code correctly.

4. What is the role of special symbols in C?
Special symbols help define the program structure. For example:

• {} – Defines a block of code.
• ; – Marks the end of a statement.
• # – Used for preprocessor directives like #include.

5. How do tokens help in code compilation?
During compilation, the compiler first scans the program and breaks it into tokens. It then checks their correctness before converting them into machine code. This process ensures that the program follows the rules of the C language.

6. How are string constants different from character constants?
A string constant is a sequence of characters enclosed in double quotes and ends with a null character. A character constant is a single character enclosed in single quotes and represents a single value.