Lexical Analyzer Generator

Lex (Flex in recent implementation)

Samy Said Mohamed Eshaish

Pre-Masters student, Department of Computer Science

2012-2013

Compiler Design 2
Contents:-

1- Lex
 What is Lex
 Structure of Lex Program
 Lex Predefined Variables
 Lex Library Routines

2- Set up the Tool

 Install on Windows
 Install on Linux (Ubuntu)

3- Running the Tool

 Run on Windows
 Run on Linux (Ubuntu)

4- Decaf Example

 Example Code
 Run The Example

5- References
  References

1. Lex
 What is Lex?
• The main job of a lexical analyzer (scanner) is to break up an
input stream into tokens (tokenize input streams).

• Ex :-

a = b + c * d;

ID ASSIGN ID PLUS ID MULT ID SEMI

• Lex is an utility to help you rapidly generate your scanners

 Structure of Lex Program

• Lex source is separated into three sections by %% delimiters

• The general format of Lex source is

 • The absolute minimum Lex program is thus

 Definitions

• Declarations of ordinary C variables, constants and Libraries.

%{

#include <math.h>

#include <stdio.h>

#include <stdlib.h>

%}

• flex definitions :- name definition

Digit [0-9] (Regular Definition)

 Operators  “\[]^-?.*+|()$/{}%<>

• If they are to be used as text characters, an escape should be used

\$ = “$”

\\ = “\”

• Every character but blank, tab (\t), newline (\n) and the list above is
always a text character

 Translation Rules

• The form of rules are:

Pattern { action }

• The actions are C/C++ code.

 [0-9]+ { return(Integer); } // RE

{DIGIT}+ { return(Integer); } // RD

 User Subroutines Section

• You can use your Lex routines in the same ways you use routines in
other programming languages (Create functions,
identifiers).

• The section where main() is placed

 Lex Predefined Variables

• yytext -- a string containing the lexeme

• yyleng -- the length of the lexeme

• yyin -- the input stream pointer

• the default input of default main() is stdin

• yyout -- the output stream pointer

 • the default output of default main() is stdout.

 Lex Library Routines

• yylex()

– The default main() contains a call of yylex()

• yymore()

– return the next token

• yyless(n)

– retain the first n characters in yytext

• yywarp()

– is called whenever Lex reaches an end-of-file

– The default yywarp() always returns 1

2. Set up the Tool

 Windows
1. Download the Windows version of FLEX

2. Download a C/C++ Compiler DevCPP or Code::Blocks

3. Install all. It's recommended to install in folders WITHOUT spaces in their

names. I use 'C: \GNUWin32' for FLEX and 'C: \ DevCPP'

4. Now add the BIN folders of both folders into your PATH variable. In case
you don't know how to go about this, see how to do it on Windows
 XP ,Windows Vista and Windows 7. You will have to add ';
C:\GNUWin32\bin;C:\Dev-Cpp' to the end of your PATH

5. Open up a CMD prompt and type in the following

C:\flex --version
flex version 2.5.4

C:\>gcc --version
gcc (GCC) 3.4.2 (mingw-special)
Copyright (C) 2004 Free Software Foundation, Inc.
this is free software; see the source for copying
conditions.

There is NO warranty; not even for MERCHANTABILITY

Or FITNESS FOR A PARTICULAR PURPOSE.

 Linux (Ubuntu)
NOTE: You have to install m4 (general purpose macro processor) before
installing Flex:-

 Downloading m4:

Run the command below,

wget ftp://ftp.gnu.org/gnu/m4/m4-1.4.10.tar.gz

 Extracting files from the downloaded package:

tar -xvzf m4-1.4.10.tar.gz

Now, enter the directory where the package is extracted.

cd m4-1.4.10

 Configuring m4:

./configure

 Compiling m4:

make

Note: check for any error message.

 Installing m4:

As root (for privileges on destination directory), run the following.

sudo make install

Note: check for any error messages.

 Downloading Flex (The fast lexical analyzer):

Run the command below,

wget http://prdownloads.sourceforge.net/flex/flex-.5.33.tar.gz?download

Note: Replace 2.5.33 with your version number.

 Extracting files from the downloaded package:

tar -xvzf flex-2.5.33.tar.gz

Now, enter the directory where the package is extracted.

cd flex-2.5.33

 Configuring flex before installation:

PATH=$PATH: /usr/local/m4

NOTE: Replace '/usr/local/m4/bin' with the location of m4 installation

directory. Now, configure the source code before installation.

./configure

 Compiling flex:

make

Note: check for any error message.

 Installing flex:

As root (for privileges on destination directory), run the following.

sudo make install

 Note: check for any error messages.

3. Running the Tool

 Windows
 Open CMD

 First Go to Directory Contains Files With CD Command

 To run Lex on a source file, type  flex (lex source file.l) Command

 It produces a file named lex.yy.c which is a C program for the lexical analyzer.

 To compile lex.yy.c, type  gcc lex.yy.c Command

 To run the lexical analyzer program, type  a.exe < input file > output file Command

 Linux (Ubuntu)
 Open Terminal

 First Go to Directory Contains Files With CD Command

 To run Lex on a source file, type  flex (lex source file.l) Command

 It produces a file named lex.yy.c which is a C program for the lexical analyzer.
 To compile lex.yy.c, type  gcc lex.yy.c Command

 To run the lexical analyzer program, type  ./a.out < input file > output file Command

4. Decaf Example
 CODE
//Definitions Section

%{

/* need this for the call to atof() below */

#include <math.h>

#include <stdio.h>

#include <stdlib.h>

%}

DIGIT [0-9]

LITTER [a-zA-Z]

C [//]

S []

L [\n]

ID {LITTER}({LITTER}|{DIGIT}|_)*

double {DIGIT}+(\.{DIGIT}+)?([Ee][+-]?{DIGIT}+)?

hex (0[xX])({DIGIT}|[a-fA-F])*

String (\"({LITTER}|{S})*\")

Sym [~!@#$%^&*()_+|><""''{}.,]

Comment (({LITTER}|{DIGIT}|{Sym}|{S}|{L})*)

Comment2 ({C}+({LITTER}|{DIGIT}|{Sym}|{S})*{L})

%%

//Rule Section

{DIGIT}+ {

printf( "An integer: %s (%d)\n", yytext,atoi( yytext ) );

{double} {
 printf( "A double: %s (%g)\n", yytext,

atof( yytext ) ); }

{hex} {

printf( "A Hexadecimal: %s \n", yytext ); }

{String} {

printf( "A String: %s \n", yytext ); }

"/*"{Comment}+"*/" /* eat up one-line comments */

{Comment2} /* eat up one-line comments */

void|int|double|bool|string|class|interface|null|this|extends|implements|for|while|if|else|

return|break|new|NewArray|Print|ReadInteger|ReadLine|true|false

{ printf( "A keyword: %s\n", yytext ); }

{ID} { printf( "An identifier: %s\n", yytext ); }

"+"|"-"|"*"|"/"|"="|"=="|"<"|"<="|">"|">="|"!="|"&&"|"||"|"!"|";"|","|"."|"["|"]"|"("|")"|"{"|"}"

{ printf( "An operator: %s\n", yytext ); }

"{"[^}\n]*"}" /* eat up one-line comments */

[ \t\n]+ /* eat up whitespace */

. {printf( "Unrecognized character: %s\n", yytext ); }

%%

//User Code Section

main( argc, argv )

int argc;

char **argv;

++argv, --argc; /* skip over program name */

if ( argc > 0 )

yyin = fopen( argv[0], "r" );

else

yyin = stdin;

yylex();

}
 int yywrap(){

return 1; }

 Run the CODE

 Save file with name “decaf.l”

 Open CMD or Terminal then go to directory contains file with cd command

 Type this command  flex decaf.l

 Type this command  gcc lex.yy.c

 Type this command  a.exe<source prog file .txt> output file.txt (Windows)
or ./ a.out <source prog file .txt> output file.txt (Ubunto)

5. References
 lex & yacc 2nd_edition John R. Levine, Tony
Mason, Doug Brown, O’reilly

 Mastering Regular Expressions by Jeffrey E.F.Friedl

O’Reilly