Java StreamTokenizer Class - Set 1

Last Updated : 02 Jun, 2025

In Java, the StreamTokenizer class is present in the java.io package. It is used to parse an input stream by breaking it into small chunks known as tokens, these tokens make the processing easier. A token can be a word, a number, or any specific symbol. Stream Tokenizer can recognize numbers, quoted strings, and various comment styles.

Features of the StreamTokenizer Class:

The key features of the StreamTokenizer class are listed below:

It breaks the input streams into tokens like symbols, words, and numbers.
It supports tracking line numbers.
It can treat end-of-line characters as tokens.
It can also convert word tokens to lowercase automatically.

Declaration of StreamTokenizer Class

The declaration of the StreamTokenizer class is:

public class StreamTokenizer extends Object implements Serializable

Note: It extends Object and implements Serializable.

Constructors of the StreamTokenizer Class

This class consists of two constructors, with the help of which we can create objects of this class in different ways. The following are the constructors available in this class:

1. StreamTokenizer(InputStream is): This constructor is deprecated. It is an older way to create a tokenizer directly from a byte stream.

Syntax:

StreamTokenizer(InputStream is)

Note: This is not recommended because it works on bytes not characters.

2. StreamTokenizer(Reader r): This is the best way to create tokenizer, it uses a character stream which handles the text properly.

Syntax:

StreamTokenizer(Reader r)

Example:

Java

// Demonstrating the working 
// of StreamTokenizer(Reader r) 
import java.io.*;

public class Geeks {
    
    public static void main(String[] args) throws IOException {
        
        Reader r = new StringReader("Hello 123");
        StreamTokenizer t = new StreamTokenizer(r);  
        
        int token;
        while ((token = t.nextToken()) != StreamTokenizer.TT_EOF) {
            if (token == StreamTokenizer.TT_WORD) {
                System.out.println("Word: " + t.sval);
            } else if (token == StreamTokenizer.TT_NUMBER) {
                System.out.println("Number: " + t.nval);
            }
        }
    }
}

Output

Word: Hello
Number: 123.0

Java StreamTokenizer Methods

The table below demonstrates the methods of this class.

Method	Description
commentChar()	Specifies that the character ch starts a single-line comment. All characters from the comment character to the end of the line are ignored.
lineno()	Returns the current line number of the input stream.
toString()	Returns a string representation of the current stream token and the line number it occurs.
eolIsSignificant(boolean flag)	Determines whether end-of-line characters are treated as significant tokens. If true, end-of-line characters are returned as tokens.
ordinaryChar(int ch)	Specifies that the character ch is treated as an ordinary character, not as a word, number, or comment character.
nextToken()	Parses the next token from the input stream and returns its type.
lowerCaseMode()	Determines whether word tokens are automatically converted to lowercase.
ordinaryChar()	Specifies that the character ch is treated as an ordinary character.
ordinaryChars()	Specifies that all characters in the range low to high are treated as ordinary characters.

Now, we are going to discuss about each method one by one in detail:

1. commentChar(): This method is used to specify the the character ch which is starts at single line comment and all the character from this character to the end of the line are not recognized by StreamTokenizer.

Syntax:

public void commentChar(int ch)

Parameter: This method takes a single integer value ch after that all the characters are ingnored
Return Type: This method does not return anything.

Example:

Java

// Demonstrating the working of commentChar() method
import java.io.*;

public class Geeks {
    public static void main(String[] args) throws IOException {
        FileReader reader = new FileReader("ABC.txt");
        BufferedReader bufferread = new BufferedReader(reader);
        StreamTokenizer token = new StreamTokenizer(bufferread);

        // Use of commentChar() method
        token.commentChar('a');  

        int t;
        while ((t = token.nextToken()) != StreamTokenizer.TT_EOF) {
            switch (t) {
                case StreamTokenizer.TT_NUMBER:
                    System.out.println("Number : " + token.nval);
                    break;
                case StreamTokenizer.TT_WORD:
                    System.out.println("Word : " + token.sval);
                    break;
            }
        }
    }
}

Output:

Word : Programmers
Number : 1.0
Number : 2.0
Number : 3.0
Word : Geeks
Word : Hello
Word : a
Word : Program
Word : is
Word : explained
Word : here
Word : my
Word : friends.

Note: This program will not run sucessfully because the file "ABC.txt" does not exist. If we want to test the code on the system, we just simply need to create file name ABC.txt.

Create a file ABC.txt with the following content which is listed below:

Programmers 1 2 3 Geeks Hello a Program is explained here my friends.

2. lineno(): This method returns the current line number, which is processed by the StreamTokenizer. This method is very useful when we want to check how the processing works, debug the program and we can also track the line numbers during the time of tokenization.

Syntax:

public int lineno()

Parameter: This method does not take any parameter.
Return Type: This method return an int value, the line number of the current input stream.

Example:

Java

// Demonstrating the use of lineno() method

import java.io.*;
public class Geeks
{

    public static void main(String[] args) throws InterruptedException, 
                                                  FileNotFoundException, IOException
    {
        FileReader reader = new FileReader("ABC.txt");
        BufferedReader bufferread = new BufferedReader(reader);
        StreamTokenizer token = new StreamTokenizer(bufferread);
        
        token.eolIsSignificant(true);
        // Use of lineno() method 
        // to get current line no.
        System.out.println("Line Number:" + token.lineno());

        token.commentChar('a');
        int t;
        while ((t = token.nextToken()) != StreamTokenizer.TT_EOF)
        {
            switch (t)
            {
            case StreamTokenizer.TT_EOL:
                System.out.println("");
                System.out.println("Line No. : " + token.lineno());
                break;
            case StreamTokenizer.TT_NUMBER:
                System.out.println("Number : " + token.nval);
                break;
            case StreamTokenizer.TT_WORD:
                System.out.println("Word : " + token.sval);
                break;

            }
        }
    }
}

Output:

Line Number:1
Word : Progr

Line No. : 2
Number : 1.0

Line No. : 3
Number : 2.0

Line No. : 4
Number : 3.0

Line No. : 5
Word : Geeks

Line No. : 6
Word : Hello

Line No. : 7
Word : This
Word : is

3. toString(): This method returns a string which represents the current stream token with the token value and the line number it is currently using.

Syntax:

public String toString()

Parameter: This method does not take any parameter.
Return Type: This method returns a string value representing the current stream token with the line number.

Example:

Java

// Demonstrating the use of toString() method
import java.io.*;

public class Geeks {
    public static void main(String[] args) throws IOException {
        FileReader reader = new FileReader("ABC.txt");
        BufferedReader bufferread = new BufferedReader(reader);
        StreamTokenizer token = new StreamTokenizer(bufferread);

        int t;
        while ((t = token.nextToken()) != StreamTokenizer.TT_EOF) {
            switch (t) {
                case StreamTokenizer.TT_NUMBER:
                    System.out.println("Number : " + token.nval);
                    break;
                case StreamTokenizer.TT_WORD:
                    System.out.println("Word : " + token.toString());  
                    break;
            }
        }
    }
}

Output:

Word : Token[Programmers], line 1
Number : 1.0
Number : 2.0
Number : 3.0
Word : Token[Geeks], line 5
Word : Token[Hello], line 6
Word : Token[a], line 7
Word : Token[Program], line 7
Word : Token[is], line 7
Word : Token[explained], line 7
Word : Token[here], line 7
Word : Token[my], line 7
Word : Token[friends.], line 7

4. eolIsSignificant(): This method does not return anything but is used to check whether the EOL (End of Line) character should be tokenize. If the flag is true, then each end-of-line character is treated as a token and assigned the token type TT_EOL, the eol character is ignored are treated as whitespace.

Syntax:

public void eolIsSignificant(boolean flag)

Parameter: This method takes a boolean flag if the it is true then end of the line character is treated as the a token or ignored as whitespace.
Return Type: This method does not return anything.

Example:

Java

// Demonstrating the use of eolIsSignificant() method

import java.io.*;
public class Geeks
{
    public static void main(String[] args) throws InterruptedException,
    FileNotFoundException, IOException
    {
        FileReader reader = new FileReader("ABC.txt");
        BufferedReader bufferread = new BufferedReader(reader);
        StreamTokenizer token = new StreamTokenizer(bufferread);

        boolean arg = true;
        // Use of eolIsSignificant() method
        token.eolIsSignificant(arg);
        // Here the 'arg' is set true, so EOL is treated as a token

        int t;
        while ((t = token.nextToken()) != StreamTokenizer.TT_EOF)
        {
            switch (t)
            {
            case StreamTokenizer.TT_EOL:
                System.out.println("End of Line encountered.");
                break;
            case StreamTokenizer.TT_NUMBER:
                System.out.println("Number : " + token.nval);
                break;
            case StreamTokenizer.TT_WORD:
                System.out.println("Word : " + token.sval);
                break;

            }
        }
    }
}

Output:

Number : 1.0
End of Line encountered.
Word : Geeks
End of Line encountered.
Number : 2.0
End of Line encountered.
Word : For
End of Line encountered.
Number : 3.0
End of Line encountered.
Word : Geeks

Note: This program will not run sucessfully because the file "ABC.txt" does not exist. If we want to test the code on the system, we just simply need to create file name ABC.txt.

Create a file ABC.txxt with the following content which is listed below:

1 Geeks 2 For 3 Geeks

5. nextToken(): This method reads the next token from the input stream and returns the type. The type of the token is stored in the ttype field. It returns the type as an integer value, which can be TT_WORD, TT_NUMBER and TT_EOL and etc.

Syntax:

public int nextToken()

Parameter: This method does not take any parameter.
Return Type: This method return the int value of the token type.

Example:

Java

// Demonstrating the use of nextToken() method

import java.io.*;
public class Geeks
{
    public static void main(String[] args) throws InterruptedException,
    FileNotFoundException, IOException
    {
        FileReader reader = new FileReader("ABC.txt");
        BufferedReader bufferread = new BufferedReader(reader);
        StreamTokenizer token = new StreamTokenizer(bufferread);

        // Use of nextToken() method to parse Next Token from the Input Stream
        int t = token.nextToken();
        while ((t = token.nextToken()) != StreamTokenizer.TT_EOF)
        {
            switch (t)
            {
            case StreamTokenizer.TT_NUMBER:
                System.out.println("Number : " + token.nval);
                break;
            case StreamTokenizer.TT_WORD:
                System.out.println("Word : " + token.sval);
                break;

            }
        }
    }
}

Output:

Word : This
Word : program
Word : tells
Number : 2.0
Word : about
Word : use
Word : of
Number : 3.0
Word : next
Word : token
Word : method

Note: This program will not run sucessfully because the file "ABC.txt" does not exist. If we want to test the code on the system, we just simply need to create file name ABC.txt.

Create a file ABC.txt with the following content which is listed below:

1 This program tells 2 about use of 3 next token() method

6. lowerCaseMod(): This method takes a boolean flag value and checks whether the token should automatically be converted to lowercase. If the flag is true, then all words of the token are converted to lowercase or otherwise, the tokens are set as it is and don't want to convert them.

Syntax:

public void lowerCaseMode(boolean flag)

Parameter: It takes a boolean flag value. If it is true then all the tokens are converted to lowercase, and if false, then they will not be converted.
Return Type: This method does not return anything.

Example:

Java

// Demonstrating the use of lowerCaseMode() method

import java.io.*;
public class NewClass
{
    public static void main(String[] args) throws InterruptedException,
    FileNotFoundException, IOException
    {
        FileReader reader = new FileReader("ABC.txt");
        BufferedReader bufferread = new BufferedReader(reader);
        StreamTokenizer token = new StreamTokenizer(bufferread);

        // Use of lowerCaseMode() method to
        //Here, the we have set the Lower Case Mode ON
    
        boolean arg = true;
        token.lowerCaseMode(arg);

        int t;
        while ((t = token.nextToken()) != StreamTokenizer.TT_EOF)
        {
            switch (t)
            {
            case StreamTokenizer.TT_NUMBER:
                System.out.println("Number : " + token.nval);
                break;
            case StreamTokenizer.TT_WORD:
                System.out.println("Word : " + token.sval);
                break;

            }
        }
    }
}

Output:

Word : hello
Word : geeks
Word : this
Word : is
Word : about
Word : lowercasemode

Note: This program will not run sucessfully because the file "ABC.txt" does not exist. If we want to test the code on the system, we just simply need to create file name ABC.txt.

Create a file ABC.txt with the following content which is listed below:

Hello Geeks This Is About LowerCaseMode()

7. ordinaryChar(): This method takes an int value, ch should be treated as a character. Using this method, we can treat a charactera as special character like a number, word or whitespace.

Syntax:

public void ordinaryChar(int ch)

Parameter: This method takes a single int ch value, which will be treated as a character.
Return Type: This method does not return anything.

Example:

Java

// Demonstrating the use of ordinaryChar() method

import java.io.*;
public class Geeks
{
    public static void main(String[] args) throws InterruptedException,
    FileNotFoundException, IOException
    {
        FileReader reader = new FileReader("ABC.txt");
        BufferedReader bufferread = new BufferedReader(reader);
        StreamTokenizer token = new StreamTokenizer(bufferread);

        // Use of ordinaryChar() method 
        // Here we have taken 's' as an ordinary character
         token.ordinaryChar('s');

        int t;
        while ((t = token.nextToken()) != StreamTokenizer.TT_EOF)
        {
            switch (t)
            {
            case StreamTokenizer.TT_NUMBER:
                System.out.println("Number : " + token.nval);
                break;
            case StreamTokenizer.TT_WORD:
                System.out.println("Word : " + token.sval);
                break;

            }
        }
    }
}

Output:

Word : Hello
Word : Geek
Word : Thi
Word : I
Word : zz
Word : About
Word : ordinaryChar

Note: This program will not run sucessfully because the file "ABC.txt" does not exist. If we want to test the code on the system, we just simply need to create file name ABC.txt.

Create a file ABC.txt with the following content which is listed below:

Hello Geeks Thissss Issszz About ordinaryChar() This method has remove 's' from the entire Stream

8. ordinaryChars(): This method specifies that all characters in the range from the low to high(inclusive) will be treated as ordinary characters and after calling this method, the characters will no longer be treated as special characters.

Syntax:

public void ordinaryChars(int low, int high)

Parameter: This method takes two integer values low and high (inclusive), the range of the character that is converted into a special character.
Return Type: This method does not return anything.

Example:

Java

// Demonstrating the use of ordinaryChars() method

import java.io.*;
public class Geeks
{
    public static void main(String[] args) throws InterruptedException,
    FileNotFoundException, IOException
    {
        FileReader reader = new FileReader("ABC.txt");
        BufferedReader bufferread = new BufferedReader(reader);
        StreamTokenizer token = new StreamTokenizer(bufferread);

        // Use of ordinaryChars() method 
        // Here we have taken low = 'a' and high = 'c' 
         token.ordinaryChars('a','c');

        int t;
        while ((t = token.nextToken()) != StreamTokenizer.TT_EOF)
        {
            switch (t)
            {
            case StreamTokenizer.TT_NUMBER:
                System.out.println("Number : " + token.nval);
                break;
            case StreamTokenizer.TT_WORD:
                System.out.println("Word : " + token.sval);
                break;

            }
        }
    }
}

Output:

Word : Hello
Word : Geeks
Word : This
Word : is
Word : out
Word : ordin
Word : ryCh
Word : rs

Note: This program will not run sucessfully because the file "ABC.txt" does not exist. If we want to test the code on the system, we just simply need to create file name ABC.txt.

Create a file ABC.txt with the following content which is listed below:

Hello Geeks This is about ordinaryChars()

Using StreamTokenizer To Tokenize A Text File

The StreamTokenizer class is also used to tokenized the text file and here we are using the methods of Tokenizer class methos.

Step 1: First create a text file with .txt extension in the same root directory. Here, we created it with the name Geeks.txt.

Step 2: Now create a Java file and write the code to tokenized the text data present in the text file.

Geeks.java file:

Java

// Java program to Tokenized the text 
// file data using StreamTokenizer methods
import java.io.*; 
public class Geeks
{ 
    public static void main(String[] args) throws InterruptedException, 
    FileNotFoundException, IOException 
    { 
        FileReader reader = new FileReader("Geeks.txt"); 
        BufferedReader bufferread = new BufferedReader(reader); 
        StreamTokenizer token = new StreamTokenizer(bufferread); 
  
        // Use of ordinaryChar() method  
        // Here we have taken 's' as an ordinary character 
         token.ordinaryChar('s'); 
  
        int t; 
        while ((t = token.nextToken()) != StreamTokenizer.TT_EOF) 
        { 
            switch (t) 
            { 
            case StreamTokenizer.TT_NUMBER: 
                System.out.println("Number : " + token.nval); 
                break; 
            case StreamTokenizer.TT_WORD: 
                System.out.println("Word : " + token.sval); 
                break; 
  
            } 
        } 
    } 
}