Scribd
Upload
176 views3 pages

Program No. - 3: Write A Program To Find Different Tokens in A Program

The program reads a file and identifies different tokens within it. It counts the number of keywords, identifiers, operators, and delimiters. It uses functions to check each string read from the file against the token types and return a value. In the end, it prints the number of tokens of each type that were identified.

Uploaded by

Learning ID
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
176 views3 pages

Program No. - 3: Write A Program To Find Different Tokens in A Program

The program reads a file and identifies different tokens within it. It counts the number of keywords, identifiers, operators, and delimiters. It uses functions to check each string read from the file against the token types and return a value. In the end, it prints the number of tokens of each type that were identified.

Uploaded by

Learning ID
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 3

PROGRAM NO.

- 3

Aim:
Write a program to find different tokens in a program.

Theory:
Lexical Analysis is the first phase of compiler also known as scanner. It converts the input
program into a sequence of Tokens.
Lexical Analysis can be implemented with the Deterministic finite Automata.
Token-
A lexical token is a sequence of characters that can be treated as a unit in the grammar of
the programming languages.
Example of tokens:
 Type token (id, number, real, . . . )
 Punctuation tokens (IF, void, return, . . . )
 Alphabetic tokens (keywords)
Keywords; Examples-for, while, if etc.Identifier; Examples-Variable name, function name
etc.Operators; Examples '+', '++', '-' etc.Separators; Examples ',' ';' etc.
Example of Non-Tokens:
 Comments, preprocessor directive, macros, blanks, tabs, newline etc.

Algorithm:
1. A file is read single string at a time until the end of file.
2. Each string is made to check against rules and strings of keywords, identifiers,
operators and delimiters.
3. If the string is matched with one of the tokens, that particular function returs 1 else it
returns 0.
4. The tokens identified are then counted in counter variables.
5. The file is closed when all the tokens are identified and counted.

Code:
#include<iostream>
#include<conio.h>
#include<stdio.h>
#include<string.h>
using namespace std;

int keywords(char []);


int identifiers(char []);
int operators(char []);
int delimiters(char []);

int main()
{
int k=0,id=0,op=0,d=0;
char s[100];
FILE *fp;
fp=fopen("text.txt","r");
while(fscanf(fp,"%s",s)!=EOF)
if(keywords(s)){
k++;}
else if(operators(s)){
op++;}
else if(delimiters(s)){
d++;}
else if(identifiers(s)){
id++;}
fclose(fp);
cout<<"No. of keywords = "<<k<<endl;
cout<<"No. of identifiers = "<<id<<endl;
cout<<"No. of operators = "<<op<<endl;
cout<<"No. of delimiters = "<<d<<endl;
return 0;
}

int keywords(char s[])


{
if(!strcmp(s, "if") || !strcmp(s, "else") || !strcmp(s, "while") || !strcmp(s, "do") ||
!strcmp(s, "break") || !strcmp(s, "continue") || !strcmp(s, "int")
|| !strcmp(s, "double") || !strcmp(s, "float") || !strcmp(s, "print")
|| !strcmp(s, "return") || !strcmp(s, "case") || !strcmp(s, "char") || !strcmp(s, "switch"))
return 1;
return 0;
}
int identifiers(char s[])
{
if(strcmp(s, "0") || strcmp(s, "1") || strcmp(s, "2") || strcmp(s, "3") ||
strcmp(s, "4") || strcmp(s, "5") || strcmp(s, "6") || strcmp(s, "7") ||
strcmp(s, "8") || strcmp(s, "9"))
return 1;
return 0;
}
int operators(char s[])
{
if(!strcmp(s, "+") || !strcmp(s, "-") || !strcmp(s, "*") ||
!strcmp(s, "/") || !strcmp(s, "<") || !strcmp(s, ">") ||
!strcmp(s, "="))
return 1;
return 0;
}
int delimiters(char s[])
{
if(!strcmp(s, ",") || !strcmp(s, ";") || !strcmp(s, "(") ||
!strcmp(s, ")") || !strcmp(s, "{") || !strcmp(s, "}") ||
!strcmp(s, "[") || !strcmp(s, "]"))
return 1;
return 0;
}

Output:

Learnings:
Tokens are the most essential part of compilers. These are the basic units that are needed
for higher level operations of compiler. Every program is converted into tokens defined by a
programming language. Some of the most common tokens are keywords, identifiers,
numbers, strings, operators, constants, delimiters.

You might also like