Understanding Tokens in C++

In this lesson, you will learn

• Tokens in C++
• Examples

Tokens in C++

In C++, tokens are the smallest units of a program that have a specific meaning to the compiler.

Tokens are categorized into several types, each playing a different role in the language. Here’s a detailed explanation of each type of token with examples.

1. Identifiers

• Identifiers are names given to variables, methods, classes, objects, arrays, and other program elements.
• Identifiers must follow certain rules and conventions.

Rule to make identifiers in C++

The following are the rules for forming valid identifiers.

1. An identifier can have uppercase and lowercase letters(A-Z or a-z), numbers(0-9), underscore( _ ), and dollar-sign($) characters.
2. Identifiers must not begin with digits.
3. It can be of any length.
4. Keywords can’t be used for naming identifiers.
5. Java is a case-sensitive language i.e. VALUE and value are different identifiers.

Examples: Valid identifiers

Mybooks, DATE_06_12, B1T019, AvgValue, result, $1_23_$a, _value, PRINTRESULT, _PRICE, _MY_JK_2, temp$, isAlphabet

Examples: Invalid identifiers

1. 23Date – Invalid because Starting with a number
2. Item-Price, yes/no, print.file – Invalid because it contains a character other than A-Z, a-z, digit, underscore( _ ), or dollar($)
3. Print Result – Invalid because it contains whitespace.

Example: Identifier

#include<iostream>

int main() {
    int age = 30; // 'age' is an identifier
    std::cout << "Age: " << age << std::endl;
    return 0;
}

Identifiers: age, main, cout, variableName, etc.

2. Keywords

C++ keywords are reserved words that have special meanings in the language. They cannot be used as identifiers (e.g., variable names, function names) because they are fundamental to the syntax and structure of the language.

Here’s a summary table of the C++ keywords:

3. Constants or Literals

In C++, constants and literals are values that do not change during the execution of a program. They represent fixed data that can be used directly in code.

Here’s a detailed explanation of different types of constants and literals in C++, with examples.

1. Integer Literals

Integer literals represent whole numbers and can be expressed in decimal, octal, or hexadecimal format.

Examples: Integer Literals

#include<iostream>

int main() {
    int decimal = 42;    // Decimal integer literal
    int octal = 052;     // Octal integer literal (base 8, prefixed with 0)
    int hexadecimal = 0x2A; // Hexadecimal integer literal (base 16, prefixed with 0x)
    
    std::cout << "Decimal: " << decimal << std::endl;
    std::cout << "Octal: " << octal << std::endl;
    std::cout << "Hexadecimal: " << hexadecimal << std::endl;

    return 0;
}

Output:

Decimal: 42
Octal: 42
Hexadecimal: 42

2. Floating-Point Literals

Floating-point literals represent numbers with decimal points and can be expressed in decimal or scientific notation.

Examples: Floating-Points Literals

#include<iostream>

int main() {
    float pi = 3.14159f;       // Decimal floating-point literal (float)
    double e = 2.718281828459; // Decimal floating-point literal (double)
    double scientific = 1.23e4; // Scientific notation (1.23 &times; 10^4)
    
    std::cout << "Pi: " << pi << std::endl;
    std::cout << "E: " << e << std::endl;
    std::cout << "Scientific notation: " << scientific << std::endl;

    return 0;
}

Output

Pi: 3.14159
E: 2.71828
Scientific notation: 12300

3. Character Literals

Character literals represent individual characters enclosed in single quotes.

Examples: Character Literals

#include<iostream>

int main() {
    char letter = 'A';       // Single character literal
    char digit = '7';        // Single character literal
    char newline = '\n';    // Escape sequence for newline
    char tab = '\t';        // Escape sequence for horizontal tab

    std::cout << "Letter: " << letter << std::endl;
    std::cout << "Digit: " << digit << std::endl;
    std::cout << "Newline: " << newline << "Next line" << std::endl;
    std::cout << "Tab: " << tab << "Tabbed text" << std::endl;

    return 0;
}

Output:

Letter: A
Digit: 7
Newline:
Next line
Tab:    Tabbed text

4. String Literals

String literals represent sequences of characters enclosed in double-quotes.

Examples: String Literals

#include<iostream>

int main() {
    const char* greeting = "Hello, World!"; // String literal
    const char* multiline = "This is a\nmultiline\nstring"; // String with newline characters

    std::cout << "Greeting: " << greeting << std::endl;
    std::cout << "Multiline: " << multiline << std::endl;

    return 0;
}

Output:

Greeting: Hello, World!
Multiline: This is a
multiline
string

5. Wide Character Literals

Wide character literals represent characters in wide format, enclosed in single quotes with an L prefix.

Examples: Wide Character Literals

#include<iostream>

int main() {
    wchar_t wideChar = L'汉';     // Wide character literal
    wchar_t wideString[] = L"你好"; // Wide string literal

    std::wcout << L"Wide character: " << wideChar << std::endl;
    std::wcout << L"Wide string: " << wideString << std::endl;

    return 0;
}

Output:

Wide character: 汉
Wide string: 你好

6. Boolean Literals

Boolean literals represent truth values and can be true or false.

Examples: Boolean Literals

#include<iostream> 

int main() {
    bool isTrue = true;   // Boolean literal true
    bool isFalse = false; // Boolean literal false

    std::cout << "isTrue: " << isTrue << std::endl;
    std::cout << "isFalse: " << isFalse << std::endl;

    return 0;
}

Output:

isTrue: 1
isFalse: 0

7. Null Pointer Literals

Null pointer literals represent a null pointer constant. In modern C++, the nullptr keyword is used.

Examples: Null Pointer Literals

#include<iostream>

int main() {
    int* ptr = nullptr; // Null pointer literal
    
    if (ptr == nullptr) {
        std::cout << "Pointer is null." << std::endl;
    }

    return 0;
}

Output:

Pointer is null.

Summary

• Integer Literals: Represent whole numbers (e.g., 42, 052, 0x2A).
• Floating-Point Literals: Represent numbers with decimal points (e.g., 3.14159, 1.23e4).
• Character Literals: Represent individual characters (e.g., 'A', '7', '\n').
• String Literals: Represent sequences of characters (e.g., "Hello, World!").
• Wide Character Literals: Represent wide characters (e.g., L'汉', L"你好").
• Boolean Literals: Represent truth values (true, false).
• Null Pointer Literals: Represent null pointers (nullptr).

These constants and literals are fundamental in C++ programming, providing fixed values that can be used throughout your code.

4. Punctuation (Delimeters)

In C++, punctuation (also known as delimiters) are special symbols that help define the structure and organization of the code.

They are used to separate and group tokens, mark the end of statements, define blocks of code, and more.

Here’s a summary table of the punctuation (delimiters) used in C++ with examples:

5. Operators