SYLLABUS Compiler Design - 3170701 ESBee SF CONTENTS of2.2.1 Strings and Langvaee, 2.22 Operations on Language 2.23 Regular Set. “ 2.24 Regular Expressions. ‘2.3 Recognition of Tokens. 24 Input Buffering... 2.5 Elementary Scanner Design and 25, Structure of LEX 252 LEX Programs. a 2.6 Applying Concepts of Fnite Automata for Recognition of Tokens: 2.6.1 Construction ofa NFA from Regular Expression Ys Construction) 2.7 Design of Lexical Analyzer Generat 2.7. Transtion Diagrams for Programming Constuts.. 2.8 Optimization of DFA. a sti Frite Automata (OFA). 1 DFA Conversion 2.9 Short Questions and Answers... 2.10 Multiple Choice Questions with Answers ‘Chapter-3 Syntax Analyse —g-ys ease 3.1 Understanding Parser and CFG(Context Free Grammars) Implementation (Les 3.2 Top Down and Bottom Up Parsing Algorithms.. 3.3 Top-Down Parsing 33.1 Problems with Top - Down Parsing, 3.3.2 Recursive Descent Pars 3.3.3 Predictive U1) Parser. 3.4.1 Shift Reduce Parser 3.5 Operator-Precedence Parse 3.5:1 Operator Precedence Pars 3.5.2 Precedence Functions 3.6 LR Parsers 3.9 LALR Parser 3.10 Comparison of LR Parsers. 3.11 Using Ambiguous Grammars. 3.12 Parser Generators, 43,13 Automatic Generation of Par 3.14 Short Questions and Answers 3.15 Multiple Choice Questions with Answers. Chapter-4 Syntax Directed Translation 4,1 Introduction. 4.2 Syntax Directed Definitions (SDD). 4.3 Construction of Syntax Trees. - 4.3.1 Construction of Syntax Tree for Expresso 4.4 Bottom Up Evaluation of S-Attributed Definitions 71) to (4-38) 4-2 4-2 4-18 4-18 4-21 4.4.1 Synthesized Attributes on the Parser Stac vo 22 45 L-Attributed Definitos 2 4-26 4.6 Syntax Directed Definitions and Translation Schemes 4-28 4.6.1 Guideline for Designing the sone 33 4.7 Short Questions and Answers. 4-34 4-36 4.8 Multiple Choice Questions with Answers£85 Loops in Flow Graph: 3.6 Next Use Information a6, Storage! 447 The DAG Representation of Basi BIOGAS 187.1 Agorithm for Construction of DAG. £4.72 Aoplcations of DAG 4,73 0NG based Local Optimization {8.8 Machine Dependent Optimization {881 Characersies of Peephole Optimization 8.9 Simple Code Generator.. location and Assignment. Jabal Register Alocation.. 10.2 Usage Count ment for Outer Loop 8.10.4 Graph Coloring for Register Assignment 8.11 More Examples on Code Generatior 8.12 Machine independent Optimization 48.13 Few Selected Optimizations. 8.33.1 Compile Time Evaluation 8.13.2 Common Sub Expression Elimination, 8.13.3 Variable Propagat 8.134 Code Movement... 8.135 Strength Reducti 8.13.6 Dead Code Eiminat ‘8.16 Multiple Choice Questions with Answers. ‘Ghepier-9 Instruction - Level Parallelism 19)» 716) eer 9"2 9.1 Processor Architectures. 9.1.1 Instruction Pipelines and Branch Delays. 9.1.2 Pipelined Execution 9.1.3 Mult 9.2 Code-Schedul 9.2. Data Dependence... 9.22 Finding Dependences Among Memory Access. 9.23 Tradeoft between Register U ction issue. and Parallelism 9.24 Phase ordering between Register Allocation and Code Scheduling. 9.25 Control Dependence. 19.2.7 Basic Machine Mode! 9.3 Basic - Block Scheduling. 19.3.1 Data Dependence Grap 19.3.2 Ust Scheduling Algorithm. 9.4 Pass Structure of Assemble1 Overview of the Compiler and its Structure Syllabus 4.5 The Science of Builiing Compilers 1.8 Applications of Language Processors 1.7 Basie Understanding of interpreter and Assembier 1.8 Difference between Interpreter ‘and Compiler cssescs, Winter-f7, 20, Summert9, --+ Marks 4 1.9 Compilation of Source Code into Target Language Wintor-3, Marks 8 4.10 Cousins of Compiler ................Maye12, Wintor-18, Marks 4 1.41 Types of Compiir 4.42. Short Questions and Answers 1.13. Multple Choice QuestionsSeen. BBE Language Processor “pefation 1 A warsator one Kind of PYOBAT, Defnaton +A warn er form, The input program i cilled source as input and converts {s called target language; — ee sembly language or a high Tanguae as! he et Para he source language can be Tow level languns Tike a ear que nb ow er ings or ane gst Fig, 4.14 Translator Types of Translator ‘There are two types of translators compiler and assembler. 2 The translator should convert the source program to a target machine code in such 1 way thatthe generated target code should-be easy-to understand. —_ 4. The translator should preserve the meaning ofthe source code. 4 The translator should teport errors that occur during compilation to its users. 5. The translation must be done efficiently, CUE EEA Definition of Compiler In this section we will discuss two things : “What is compiler ? And “Why to write compiler Let ws start with “What is compiler?” Compiler a program which takes one language (Source progam) as input — snd rasates ito an eualent another language (arg Progra). During this process of tansaion if some emors ae encountered ten compiler sliplays them as enor messages. Te basic model of compiler can be represented as Fig. 12 Compiler Compter Seinen 1-9 Orentowof te Compt and te Sucre The compiler takes a sour © Program as higher level languages such as FORTRAN snd coves itn ow eel ngage oes see ene assembl guage. cenae (DEED compte and ests are to treats. 1 ransators convert one form of program into another form: Compiler concerts high lcel language to machine level language thle asembler concerts assembly language program to machine lt! lmguage. Factors that Affect the Design of Compiler aro - 1. The choice of source language camplier. Also list major functions done by compiler. CEES PRINS NTRS TECHNCAL PUBLCATIONS uptempo TECHNICAL PUBLICATIONS® - an upto hnowtedgeove 4. Lenial Ana go called scanning ys 1 Th es a aon in whch the complete Source code i cad gy «+s the phase of omPA up into group of strings called token, ae aracers having a collective meaning. For example total = count + ate * 20 ‘hen in leva aalss phase ths statement 6 broken up into series of token x follows 1. The enti otal 2. The assignment symbol 5. The identifier count 6, The multiplication sign 7. The constant number 10 The blank characters which are used in the programming statement are eliminated during the lexical analysis phase. 2. Syntax Analysis + The syntax analysis is also called parsing. ‘+ Im this phase the tokens generated by the lexical analyser are grouped together to zoos 7 \ + The syntax analysis determines the gmx te, For ‘the expression tolal= count + rate #10 the parse tree can be generated as follows io sa Fig. 4.4.1 Parse tro for total = count + rato “10 TEOMUCAL PUBLICATIONS? «on wa '* Once the syntax is checked in the syntax analyser phase the next phase ie. the semantic analysis determines the meaning of the source string ‘+ For example meaning of source string means matching of pare expression, or matching of if else statements or performing arithm of the expressions that are type compatible, or checking the scope of For example, Fig. 1.42 Semantic analysis TECHNICAL PUBLICATIONS? - an ups for knowledge‘Tus these three phases are performing the tas of analysis, After these phases an intermediate code gets generated. ‘4 Intermediate Code Generation {sa kind of code which is easy to generate and this code to target code. This code is in variety of forms such ay triple, pos. most three operands. For example, 0) 5, Code Optimization «© The code optimization phase attempts to improve the intermediate code. 1+ This is necessary to have a faster executing code or less consumption of memory. ‘+ ‘Thus by optimizing the code the overall running time of the target program can bbe improved. 6. Code Generation ‘+ In code generation phase the target code gets generated. ‘+ The intermediate code instructions are translated into sequence of machine ‘instructions. MOV rate, RI ‘MUL #1000, RA ‘MOV Rt, total Example - Show hov an input a = b+c 60 get processed in compiler. Show the ‘output at each stage of compiler. Also show the contents of symbol table. TECINCAL PLBLCATIONS® wn tina er montage Comper Desi 47 —1:7_Qvvin ct npr an Sire Input processing in compler Output ‘a= bree 60 Sytaxtece Semantic ee4 sca conamate HMO 82 Se a Symbal uble MARSEERESL er a symbol table i¢ maintained. The task o oo sappst thse pes z eRe identifiers cused in the program. informal fe usualy its type, its scope, . The information about the subroutine, number o vretod of passing these arguments(nay be call by value or call by rs return type if any «Basically symbol table is a data structure used to store the information about ‘entities +The symbol table allows us to find the record for each identifier quickly and to store or retrieve data fom that record efficiently. + During compilation the lexical analyzer detecis the identifier and makes is entry in the symbol table. However lexical analyzer can not determine all the attributes of an identifier and therefore the attributes are entered by remaining phases of compiler “Various phases can use the symbol table in various ways. For example while doing the semantic analysis and intermediate code generation, we need to know what type of identifiers are. Then during code generation typically information about how much storage is allocated to identifier is seen. Error detection and handling + In compilation, each phase detects errors, These errors must be reported to error handler whose tsk is to handle the errors so thatthe compilation can proceed. ‘+ Nonnally the errors are reported inthe form of message. {arge number of errors can be detected in syntax analysis phase, Such errors ae ' ale a syntax ers, During semantic analysis; type mismatch kind o! ‘ror is usually detected, ee TEGAMEAL PUBLICATIONS®- wp at ee Cemetery tt Fig. 1.43 Phases of compilor GENERERD Descrite the output for the various phases of compiler with respect to flowing ‘position ‘= initial + Rato + 60 Solution : Phase result for position : = ini tial + Rate + 60 ‘The output at each phase during compilation is as given below ~ TECHNICAL PUBLICATIONS® - an upttrust br nowa ‘overview ofthe Compiler and Stacy Comper Deion Token stream Optimizes cose Syntace Machine code tie ouput of al phases (except of an optinzation phase) of «compiler. Asune © of type fost asatbec*2: See Fig. 144 on next page. leElE 3. Explain diferent phases of compiler 4, Explain analysis phase of source program with example TECHNICAL PUBLICATIONS® - an up-hrust for knowledge TEOWCAL PUBLICATIONS an opinOvenew ofthe COMBHer a 8 Srey, Ten en , syrtastee > A, 2 — ae eat i 2 _ Intermediate code 3 7 cr ntmzeie hin contain any programming, cons machine code. While translating the code source language EXE Modeting in Compitor Design and Implementation + In the compiler design we must design right design model and must choose right algorithm. Both the algorithms and design must be simple and efficient. + Many compilers make use of the most fundamental mode State phase for identifying Then context free languages are also used to describe the syntax of the program. [EEE] Code Optimization + The desiring feature of any compiler is to produce optimized code. Optimized ‘code means the code that executes efficiently on the machine. For the compiler the optimized code becomes complex and important. * For optimization following are the compiler design objectives that are used - 1. When optimization is done then the meaning of the source program must be preserved. This is called correct optimization. 2. Due to optimization the performance of the program must 3, Compiler time required to execute the optimized code must be reasonable. 4, The efforts required due to optimization must be manageable. [EES Applications of Language Processors KEKE optimization for High Level Programming Languages ‘+ Compiler takes the higher ‘most commonly addressed programming language: rimization is one important OP! pages ce yam ei uma nd rds te ove : Soa esha systems: ‘of two basic techniques mons Se ene ™ Pn ee cin tin as = ear from the programmer. The compilers ¢é range the memory Bercy means 8 : tr aa em ld 1 the proce fates but eal Si see and Te empass is gen for making the memory Nerecy cit EI pesignng of New Computer Architecture features of these architectures, the compilers aze developed in processor design stage its then run on simulators and then used to evaluate the architectural features. The RISC, CISC architectures are highly influenced by the compilers. «Various specialized architectures are getting invented. For instance - SIMD (Single Instruction Multiple Data), VLIW (Very Long. Instruction Word) machines, symbolic arrays, multiprocessors with distributed shared memory. The evelopment in these architecture leads the development and improvement in compiler technologies. EEE] Program Translation The cmplton tchoology inves translation ofthe program from high level © ‘machine level language. Following are some important applic a ‘translation techniques - Bertaeens, Be TEGIICA PLBLCATONS® on want weeps omni Devan Ovonow ofthe Compt and ts Strctre Binary Translation Compiler are used to translate binary code of one machine to run it on another machine. Thus due to binary translation the one machine code can be run 0 another machine. ‘Query Interpreter + The SQL (Structured Query Languages) are used to search database. The SQL Interpreters compile of interpret the queries given at the command prompt ra 3 important elements of any software systems. The errors in the program can cause crashing of the entire system. Hence testing is done to locate errors in the program. Data flow analysis technique is used to locate errors along the execution path. This technique used in testing is mainly derived in compilers. Other important techniques of locating errors are - ‘Type Checking ‘The type checking is an effective tool used to catch the inconsistencies in the data types. When operation with wrong types is carried out then this technique identifies the . is a technique in which array index boundary can be checked using bounds check technique. This technique is used to check the overflow of the buffers in the program. TECHNICAL PUBLICATIONS® - an unit fr nowedge> Coveniow ofthe Compiler and its Stace, “0 (ei ae compen amor taraement TOO anpteofmemary management 101 Thee mq FE ifferonce between interprete + Garbage collector a8 88 Ty memory leaks which are mostly occur in C o, memory management €or fe programmer to find out the mem me sous tools are developed fo help the P ig The analysis phase of interpreter and and semantic analysis is performed management errors, lysis is per id Assembler Rese =. a Ed Basic Understanding of Interpreter an ry ae oe Find of translator which produces the result direct Defnton + An interpreter is a kind of = a is given to it as input. 7 rn the source language and data is i ‘The source program gets interpreted every In the process of compilation the program time its to-be executed, and every tine the is analyzed only once and then the code is suse program analyzed. Hence ‘generated, Hence compiler i efScent than interpretation i less efficient than Compiler. Interpreter 2. The interpreters do not produce object code. The compilers produce object code 3 Mert: Demerit : + It does not produce the object ‘code rather each time the program needs execution Fetes ie nse pert ines amen ea are Pee shown in. 171 GET crrvie i eyo rpc ea ese pectin wees ‘© Languages suchas BASIC, fe peeprten ‘SNOBOL, ‘can be translated. using inter s. JAVA also uses interpreter. Merit : ‘Demerit : us moe PO) lets empire ge uw Ta ae cp geo ‘+ The process of interpretation can be carried out in following phases. 1. Lexi analysis 2. Syntax analysis 3. Semantic analysis 4, Direct execution ‘Advantages : ‘+ Modification of user program can be easily made and implemented as execution ‘improved debugging environment. requires lage amount of memory. proceeds. ‘+ Type of object that denotes a variable may change dynamically. + Debugging a and is simpli ram used for tear Poet and fining ere i simplified task fora program used + The interpreter forthe language makes it machi Disadvantages : semeching indspendent, ¥ ‘The execution of the program is slower. ‘* Memory consumption is more. Te Compiler is a kind of mech ngs ad ane gaa tl ource progam nse te tram ito the machine language Ba ieeearin a level sours Ure Debesgeribastely a grphica __nadand C compiler ot Tcbo C camper rample of high level language is C4, p Source level SN cnn ak nope he repens wen nC orc EI 8 Ce HSA wt to _ SS Se TROON PU aa TECHNICAL PUBLICATIONS? - an upstrust orknomodge4 Preprocessors - The output of preprocessors may be given as the input to compiles. Te tasks performed by the preprocessors are given as below instracons which cn be used repeatedly ithe progam. Thos macro prepoceing task is be done by preprocesors anc immer sho alows wer 0 include the header es which may be rue by ar By this statement the header file stdioh can be included and user can make use of the functions defined in this header fle. This task of preprocessor is called file inclusion.ww ofthe Compllor andy veri Sra tructions. Whenever 19 @ props small set of ins eres by macro definit a Moc Pd then tat name 1 7 ed PP oad en tal —- vo kinds of statements Macro ocean ae use. The veer a malo ey so si ge ons ae EST a spot else bY vale 3.1 ‘Whenever the “PI” is en ode as output tuce the assembly cc itput which ig 2 tnt Se coe cerber Kind of wanhtor Whey hig jven to the assem ve code as output. br nb program as input and produces the machine re ‘An assembly code is a mnemonic version of machine code, The typical assembly instructions are as given below. MOV a, RL MUL #51 ‘The assembler converts these instructions in the binary language which can be understood by the machine. Such a binary code is often called as machine code. Ths ‘machine code is a relocttale machine code that can be passed directly to the loader linker for execution purpose. ‘The assembler converts the assembly program to low level machine language using fo passes. A pass means one complete scan of the input program. The end of second assis the relocatable machine code, 3. Loaders and Link Editors - Loader is a program which performs two functions: ‘Loading and link editing, Loading is a process in which the relocatable machine code is Loacing oe be Unk eating "ECiwCHPUBUCATONS® nwa Comper Design EEN types of Compilor In this section we will discuss various types of compilers. Incremental Compiler Incremental compiler is a compiler which performs the recompilation of only modified source rather than compiling the whole source program. The basic features of incremental compiler are, 1. It tracks the depends ween output and the source program. Cross Compiler Basically there exists three types of languages 1. Source language ie. the application program. 2. Target language in which machine code is written. 43, The Implementation i ‘There may be a case th: ‘may be a compiler which machine. Such a compiler i technique platform independency can be achieved. TECHNICAL PUBLICATIONS® - an upmus for owiedpea Lovernew oF ine Lompier an we To pet 8 OM ie a as target and | as Implementation tn ciogram wth a8 source, T co ig. 414017 or source language L the target language N gets generated which runs on macy M For example : For the frst version of EQN compiler, the compiler is written in C and the commit are generated for TROFF, which is as shown in Fig, 1.9.1. ‘The cross compiler for EQN can be obtained by running it on PDP-11 through C copiler, which produces output for PDP-11 as shown below - TROFF ‘dow-: Translator is a kindof program which converts one form of the source lags! into another form ofthe target language G2 _ Enlist the two coms monly st Purpose ? ‘sed translators in operating system. What i Fee ee rice Gompteroewgn __1-23__oventew tthe Compton ts Sucre compiler converts the source program machine code. language into equivalent machine co ‘written in high level language into the equ What is compllor 7 9: The compiler is a kind of translator which converts the high level language into tine level language. The examples of high level languages are C, Fortran, C++, cal and 50 on, are the phases of complior 7 es of compiler are lexical analysis, syntax analysis or parsing, «typically operates. les, loaders and i directly when duce the object code rather each time the program needs execution. land back end model of compiler 7 ing the front end and back end model of the What Is the advantage of * By Keeping the same front end and attaching different back ends one can produce a compiler for same source language on different machines. ferent front ends and same back end one can compile several 8 What aro machine dependant and machine independent phases ? ‘Ans. : The machine dependent phases are code generation and ct The machine independent phases are lexical analyzers, synt analyzers. 9 What are the factors affecting number of passes in compiler ? ‘Ans. : Various factors affecting the number of passes in compiler are 1, Forward reference 2. Storage limitations 3, Optimization. Q.10 Define the term cross com Ans. : There may be a cot code for another machine 11 What are the ph ‘end produce ? led cross compiler. of a compiler ? What doe TECHNICAL PUBLICATIONS® - an upthrust or knowsSome eaton tc a oo i tetie ane ii) Macro preprocessor lil) Assem! ee 4) Preprocessor “b Compiler isa program that converts high level source program ino the machine code. Compiler isa program that converts low level source program into the machine code 4 Compile is « general purpose application program. 22 What are the stages of compilation process 7 4 Requirement analysis, design, implementation, testing and maintenance Documentation, coding and testing Testing and quality assurance 4. Lexical analysis, syntax analysis, intermediate code generation, code ‘generation and code optimization, 2 The languages such a8 C\C+*, PASCAL and FORTRAN are referred 25 _— a) databases high level programming languages. © Tow level programming languages, (A middle level programming SSS. TECHNICAL PUBLIGATIONS® an upatva! ev noniocge 4:25 ___Ovenew of ne Compter and ts Stctre rogram, Which is implemented from the design. ine by line as program runs ‘| C, PASCAL, FORTRAN. | Assembly language Expert system, knowledge based system. 6 Translation of low level language to machine code is done by interpreter d. loader a) compiler © assembler Q7 Cross compiler is a compiler ‘that runs on one machine but produces object code for another machine 1b which is written in a language that is different from the source program. € is writen in the same language of source program |) generates the object code for the host machine only. Q.8 Incremental compiler is 4 that runs on one machine but produces object code for another machine 1b which is written in a language that is different from the source program. ] is written in the same language of source program d_ that allows a modified portion of the program to be recompiled, 9 An ideal compiler is that takes less time for compilation which converts the high level source program to machine level language which produces the object code which is smaller in size and exceute faste d_ All of the above. Q.10 Intorproter is preferred than compiler because less time to execute, ful in inital phases of program development process. TECHNICAL PUBLICATIONS® - an uptivst for knowledge= dabogging faster and exsir- “dit requres less number of resources. rogram that 44 An interpreter 18 3 P : o a> covet the high evel language into a machine level language by poy, “© aufomates the translation of assembly language into machi “@._places the source program in the memory and prepares fo 2.12 Compiler is preferred than interpreter because is helpful in initial phases of program development process, € debugging is faster and easier. 4. it roquires less number of resources. 2.43 Compiler is. preferred than interpreter because 42 converts the high level language into a machine level languag the object code Compler Design G46 Syntax directed translation engines are a) loaders compilers tar —____1:27 __oveniow of ne Comper and its Stture b. operating system Q.7 Storage mapping is done by a loader | compiler Q.18 The front end and back end model of compiler Is beneficial because 6 d linker operating system Q.19 The external references are resolved by a) loader © compilers link editors d_ assemblers compiler construction tools written in different languages can be compiled by the same Q.20 Compilers are generally writen by 2 computer users. _ professional programmers automates the translation of assembly language into machine l € database administrators places the source program in the memory and prepares project managers Answer Keys for Multiple Choice Questions 214 A system program that places an executable program into the memory ‘1 for execution Ie fore os « lon) + law a a) cometaer : 2d gy 8 an! s Qn a compiler 2 2 ; o 2 & © loader (linker Qs > os Qis & 2415 Which of the followi into | 4 © Qo a a8 Program rts the bly language th system converts the assombly ea Q@5 > Quo >» 4 Assembler | Sinaia = > Compiler © Loader =e Linker ——— ee SA PURLCATONS® on pt or owe TECHNICAL PUBLICATIONS? = an uth fr knowl go0ae (Overview ofthe Compiler, a ae 2 Lexical Analysis The Roe ofthe Leia Anaer, Speciation of Toke ——————— ‘Syllabus Contents dll = (2.1 The Role of the Lexical Analyzer. en 23 Recognition of Tokens a 24 Input Butforing tii ™ 29 Short Questions and Answers 2.10 Muttiple Choice Questions‘oters Fig, 24-4 Role of lxlcl analyzer the tokens it is also vical analyzer scans the source program to recognize ale a oat Apart fom token Identification lexical, analyzer also performs folowing functions Functions of lexical analyzor es blank and comments. tes symbol table which stores the information about identifiers, constants works in two phases In first phase it performs scan and inthe levcal analysis; mean it generates the series of tokens. REI Tokens, Patterns, Lexemes Let us lam some terminologies, which are frequently used when we talk about the activity of lexical analysis Tokens : It describes the dass or category of input string. For example, identifies, keywords, constants are called tokens. Pattrns Set of rules that describe the token. Loxames : Sequence of Sequence of characters in the source program that are matched with the Palle ofthe token, For example inti num, ans, choice, le. TEOINICAL PUBLICATIONS®- on wpa ri Rial ie ie: TECHNICAL PUBLICATIONS? - Wwe submit this program to produces sequence of tokens ‘operator keyword identifier ‘operator operator keyword an uta fr imoutorings sa ei oe Following fem rc Meaning, ae ing zero or more tail symbols, oblained By removing tne prefix could be ‘Hindy reat sing AS ig Hindustan he Pe For camp, sng zero or more lading symbols sping sting cn rtatan the ux could be tan ssi Fereame and suffix ofa given ig prefix and A ing cand or ample for ting Hindustan Substring sing’ ae ae subsing removing Zero of OE Not Necessarily ceotaving yang ome BY TOI 20 ae of ating Por the conan canbe sequence oF NB. EET operations on Language that the language is a collection of strings. There are various an be performed on the language E aT operation Description ‘Union of two languages LIUL2 = ot of strings in LI and strings in 12). CConateaton of yo LHL = (et of strings in L1 followed by set of strings ingunges end 12 in 2h Ken dooue tL 10 Lt denotes sero or more concatenation of Pave sium ofr. Oy v4 For example, Let L be the set of alphabets such as L= (A, B, C ...Z, a, b, be the set of digits such as D = { 0, 1, 2..9} then by performing various opera discussed above new languages can be generated as follows - + Ly Disa set of letters and digits. + LDisa set of strings consisting of letters followed by digits. + Lisa set of strings having length of 5 each. + Us a set of strings having all the strings including e, 1 Eis et of sings having all the strings including TECHNICAL PUBLICATIONS® an pti for knowledge Comper Dasign ar Lexical Anaysis # Lt is a set of strings except e ZED re, Set are mathematical symbolisms which describe the set of strings of rovides convenient and useful notation for representing tokens Here are some rules that describe definition of the regular expressions over the input set denoted by 5 ressions then R = Rl + also regular expression which represents union represented as R = Ri ‘operation. If RI and R2 are jons then R = RIR2 is also a regular expression 4. TERI is a regular exp: represents kleen closu also a regular expression which A language denoted by regular expressions is said to be a regular set or a regular Ianguage. Let us see some examples of regular expressions. engi Solution: RE. {identifier is a combination of letters or letter a letter always. Hence RE. can be der TECHNICALand digit = 0 1.29) for the language a will string. So we Can aon ‘This set indicates t rae ae EEE re cm ore a any mre imguage containing all the strings wih soltion: The RE. will be RE = (0+) ‘The set fr this RE willbe ~ L = fe a-aa,ab,b, ba, bab, abab, ..any combination of a and b] The (a+) means any combination of a and b even a mull string EERIE coco rer expression for the Tanguge cotning all ston hang ay number of a's and B's except the mull string. Solution : RE. = (a+b)? ‘This gular expression will give the set of strings of any combination of a's and bs except a nll string. REIN ite 2 reguer expression fora recognizing identifier. (Construct the RE. forthe language accepting all the strings which are ending with the set E= (01 Solution : The RE. has to be formed in which at the end there should be RE. = (Any combination of O's and 1's) 00 RE = (0+) 0 Thus the valid strings are 10,0100, 1000... we have all strings ending. with 00 See TECHNICAL PUBLICATIONS? «an upthruat for knowledge 29. Lexical Anaiysis accepting the strings which are starting with 1 ‘Land the last symbol should be 0. ‘Note that the condition followed by keeping starting and ending symbols correctly. In between them there can be any combination of 0 and 1 including null string. Write @ regular expression to denote the language L over 5", where ‘in tohich every string will be such that any number of a’s followed by any umber of Bs followed by any number of c's any number of b's and any number of Solution : Any number of a's means ma’ b' c’, ‘¢s means b” and c’. So the regular exp: 3° character from right end of the string is always a Solution =| anymumber of! | a Sand bs 3 2 " RE. = (a+b) a(atd)(a+b) ‘Thus the valid strings are babb, baaa or abb, baba or aaab and s0 on. of exactly two b's over the any number of a's) & any number of a's |b | any number of a's RE. = a" baba’ TECHNICAL PUBLICATIONS® - an up-tustfor krowindpecomer De eee ee Solution : The for odd number lar expres Complar Design - Lance Analysis scion tat contains even number of O and odd numberof, to symb dar expression corresponding to given st poe program and ge CO fe neti kk Om a at 2 The lnguage of al strings containing 0's and 1's both are even |3. The linguge of ll strings containing atmost one pir of consecutive 1's, 44. The language ofall strings that do not end with 02. SEI Token Code Value Solution: 1 Wirite down the regular expression forthe binary strings with even length Preto smal Eo constant = «6 = Par to symbol bebasbo)* ‘able Write @ regular definition for = r - 1. The language of ll strings tht donot end with OL = : 2. AI srngs of dig tht contain no lading 0, Eos f i : Solution a f ; a; 5 4 1 3 2 5 1 ° 2 EB] Recognition of Tokens For a programming language there are various types of tokens such as eywords, constants and operators an token type and token pce" and 50 on. The token is usualy represented Consider, a program TECHIICAL PUBLICATIONS? «an upinnt or outage TECHNICAL PUBLICATIONS® - an upstust er knowledge2.12 token stream. following 307, 10, (6107), 2, te will be, ipl table for identifiers and constants ‘The comesponding symbol 6 a © Location TyPe ‘Value above exam} izes “if” as a keyword Je, scanner scans the input string and recognizes k and vatons then type as 1 since in given encoding code 1 indicates keyword i lexical analyzer does not insert it into the table instead it returns the location wt resent ZA input Butter a The lexical analyzer scans the input string from left to right one character at uses two pointers begin_ptr (bp) and forward_ptr (fp) to keep track of the port ‘input scanned, Initally both the pointers point to the first character of the input string as shown below ~ EEE F244 Ini! configuration TECHNCAL PUBLCATIONS® nuts or nomage to t t ® Comptier Design 219 E Leica Anais Soto tata An ‘The forward_ptr moves ahead to search for end of lexeme. As soon as the blank space is encountered, it indicates end of oxen. In forwardpt fp) encounters a blank space the levee “in” bp LTT The fp will be moved ahead at white space. When fp encounters white space, it ignore and moves ahead. Then both the begin_ptr (bp) and forward ptr (fp) are set at next token i The input character is thus read from secondary storage. But reading in 1. One butfer scheme In this one buffer scheme, only one buffer is used fo store the input string. But the problem with this scheme is that if lexeme is very long then it crosses the buffer boundary, to scan rest of the lexeme the buffer has to be refilled, that makes overwriting the first part of lexeme. 2. Two butfer scheme To overcome the prob in this method two buff input string. The first but scanned alternately. When end of current buffer is is filed. The only problem if length of the lexeme is buffer then scanning input ‘cannot be scanned completely. TECHNICAL PUBLICATIONS? - an up-tnat for knowledge above example as soon as is identified 6 Fig. 2.42 Input buffering Fig. 24.3 Input buffering from secondary storage is costly. Hence buffering technique is used. A block of lata is first read into a buffer, and then scanned by le Used in this context : one buffer scheme and two buffer scheme. 1 analyzer. There are two Fig. 24.4 One butter scheme storing Tout string Fig. 24.5 Two buffer scheme storin " input tring :oth the bp and wan ee the string fist unt character of ipa eee ns, Write the two methods used in lexical analyzer for buffering sed for speing up the lexial analyzer. the input Which Compier Design 215 Lovcal Anaya [EE] Elementary Scanner Design and its Implementation (Lex) CE (Lexieal analyzer rogram g 3 TBxectabe program 1 = put ae Sveam of stings tokens ‘+ A LEX lexer is very much faster in finding the tokens as compared to the handwritten LEX program in C. ‘+ LEX scans the source program in order to get the stream of tokens and these tokens are related together so that various programming constructs such as ‘expressions, block statements, procedures, control structures can be realised, ‘+ The LEX specification file can be created using the extension (often pronounced 4s dot L). For example, the specification file can be x ‘+ This x file is then given to LEX compiler to produce lexyy.« + This lexyy.c. is a C program which is actually a lexical analyzer program. The LEX specification file stores the regular expressions for the tokens and the lex.yy.<. file consists of the tabular representation of the transition diagrams constructed for the regular expression. The lexemes can be recognized with the help of this tabular representation of transition diagram. * Finally the compiler compiles this generated lexyy.c and produces an object rogram a.out. When some input stream is given to a.out then sequence of tokens {get generated. The above described scenario can be modelled below. (EGSI structure of Lex Now the question arises how do we write the specification file? Well, the LEX rogram consists of three parts - 1, Declaration section 2. Rule section and __—_3. Procedure section. TECHNICAL PUBLICATIONS” - an uptvst for knowledgebs doe Son a I et ages anne este wing cea. There sin be po ‘© The section starting and ending with %, { and %) respectively is a definition section. +The section stating with %9% is called rule section. This section is closed by "5%,Then press either control Notations used in Regular E [peaeeeaee | | | For example [ez] matches with any alphabet in lower ese * The string writen in quotes matches literal, For eampl, E ee eed in ait hare. ‘Matches with one or more occurences of preceeding expression ‘Matches zero or one occurence of preceding regula expression “Matches the beginning fa line os fst character. al Used as for negation. For example, [ero] means except verb UY 5 om wh anfthing le. Group of elt exresions ogee Pu it ne nla (et fee cried ney erate Ipoh arr sander rule cary aoe > | ee | yin Of the type FILE". This point tothe curent ile being parse by the lex I's Standard input file that stores input source program. : yout OF the type FILE". This point to the location where the output of ‘rien By deft bol yin ad yet polt sanded pet yytet The teat ofthe matched patter i stored in this variable (har) Le. matches or recognizes the token from input token the leeme stored in terminated string called yytext. Thus eurent token is returned by this variable, TECHIGAL PUBLICATIONS? - an wpa fr iowa Compior Design 219 Lexical Anatyie eng Gives he length ofthe mt pate. The value in lng is same a tan ypylineno Provides current line number information, ylval This is global variable used to store the value of any foken Built-in Functions yyled) This sa starting point of lex from which scanning of source program starts yywrap) This function i called when end of files encountered. I yywrap retums 0 the sanner countinues scanning if returns I the sean does ot return tokens xyylessfintn) This function can be used to push back all frst n characters of the token being read -yymore) This function ells lever to attach next tokens to curren token. For displaying error messages, this function is used yperor) wt {nt Char_Cnt=0,Word_Cnt=0,Line Cat=0; yylexd) Drintf("\nThe character count= %d",Char_Cnt}: print(("\nThe word count= %d",Word_ Ci); Drint("\nThe line count=%d" Line Cat}; rine("\n");Laval Anais “Tag command line interface isthe interface which allows the user to interact with te computer by typing the commands, TG we can pass these parameters to the main function in the form of chance TEGHIICAL PUBLICATIONS® an uth or kowindpe{brint(\nwss is a LESS THAN fs OPERATOR’ yytext EQUAL Af opened file “be” will then be assinged (0 the standard i, 2 na am ae a Sarg dre iN . ‘ist a\tio ls a GREATER THAN EQUAL a TOR’ yytext);} {Gruen 8 EQUAL TOcompte Desi 2s Lancet anatras eee san [EE Applying Concepts of Finite Automata for Recognition of Tokens ‘There is a close relationship between a finite automata and the regular expression. We can show this relation int Fig. 26.1. To construct NFA from regular expression r the Thompson's construction is used. The input string of + is parsed and following constructs are used to build an equivalent NFA.Sa... conte a ny4y then NFA can be drawn 29° a ©” D F Here Nie repretents NFA for regular expression r, andl N(r) represents Fas regular expression 4. When # = 1) fp then NFA can be drawn as = Fig. 26.6 Thus these are use constructs are useful for creating NFA equivalent to regular express"? TECHNICAL PUBLICATIONS® - an upto inonedyo | TEEHCAL LBUEATION® ‘Then we wil build = (2+ Bs (Refer Fig. 268 ()) Fig. 26.8 (b) TECHNICAL PUBLICATIONS® - an up-thrust for knowledge Complr Design 2.29 Lexical Anatais EA Design of Lexical Analyzer Generator # To design Ie designed first. ‘+ These patterns are for recognizing various tokens form input string analyzer generator, the pattern of expressions are © From these patterns it is easy to design a Non-deterministic Finite Automata (NFA). ‘© But the simulation of DFA is easier by program. Hence we convert the NFA drawn from these patterns to DFA. Fig. 27.1 Building of lexical analyzer generator Let us understand the process of pattern matching with the help of some suitable example (EEEEERAD 41x progr i gen bl - ‘Auualllary Definitions Implement the LEX Program as DFA. Solution : We will implement the given LEX program as DFA in following steps Stop 1: For each regular expression first we will build the Finite Automata (FA), Pattern 1 : Patton 2: TECHNICAL PUBLICATIONS® - an up-trst for knowing2190 Ay Compo Des 2-34 Levcal Anatsis Patton 3: String aab matches with patter 3. 7 \ . ze patterns by designing NFA with e 3] (oa Le Step 2: Now we wil combine al these pattems EY +f aaa ‘Zoe os the combined NFA to DFA, Fig. 27.2 Combined NFA ‘Step 3: The above drawn combined NFA can recognize the pattern in the given 5, a Fer instance Roepe: a eee 7] (es E 3 fF i oe susie aig Sting abb matches wth pattern 2 8((27], 8) = (8(2)u 8 (4)U8(7,6)) =] =@uUsu8) eS 4 = ty = [58] > newstate. Eq ‘Thus we will obtain the input transitions for the new states that are getting ic ae generated, Finally the transition table can be - _ states. TECHNICAL PUBLICATIONS® - an up-tt fr nowtedge TECHNICAL PUBLICATIONS® - an up-thrust for knowledge‘compiler Di" Lexical Anaiysis [EREI Transition Diagrams: for Programming Constructs: EET rte gtr eres for erie ae eyword and design a transis agra fort Solution: re =leter (etter + digit)® LEER rite «regular eqression for defining. constants unsigned mutes Design the transition graphs for ther. Solution : The constant can be defined by three different ways - i) re = digit” NERD iste mers are ee rege tions for the above me Solution: Refer example 273, (GQEMEERAD Di ie site rarsition diagram for the signed numbers Solution : Refer example 27.3 TECHNICAL PUBLICATIONS® - an upatust for rowiedge TECHNICAL PUBLCATIONS®- an uta or oweam ————_——— OO cam Give regula defsiton for signed and unsigned bers. Er Solution : ‘The finite automata is called determini ‘automata if there is only one path for a specific input , from current state to next state. For example, the DFA input ‘a’ there is only one path, larly from Sy there is only one path for described in the definition of FSM. Fig. 28.1 Deterministic fits Defrition of DFA 'A deterministic finite automation is a collection of following things - 1) The finite set of states which can be denoted by Q. 2) The finite set of input symbols 13) The start state gp such that qy € Q. 4) Ase of final states F such that Fe Q. 5) The mapping function or transition function denoted by 6. Two paramete’ ‘passed to this transition function : One is current state and other is input sy" The transition function returns a state which can be called as next state anton = 8 Ge 8) means fom cent sate dy with input the nes In short the DFA is a five tuple notation denoted as : A= QE 8qF) ~Mhename of DEA is/A which in a collection of above described five elements TEGHCAL PURLICATIONS® «an ypontioninowvae ° 1 REED Wit ie sor eres = (1A, Bl (0,1), 8, A, (B1), where end with odd mui EEE wr A to DFA Conversion sert NFA to DFA. 1, Using subset construction method 2. Direct method 3, Using DFA tree method construction method,Levee Arey ees contr ns +. Mod for converting NFA wit «19 DFA with e. We have to convert this yp, (@E.8.ap.F) #2 NFA with a 1 comer M = en Caen DFA denoted PY Mp = (Qpr®-p: 9or ¥) Then obtain becomes a star e-closure(qo) = {Pi/P2/P30™ Pa} then [Pi P2/P3” Pp ] become DE Now [p1-P2/P3~ Pa] € Qo transitions on [pt P2-P3-~- Pa ] for each input. = € closure (6(P1-8)U8(P212)¥~-8(Ps/ a) ‘stop 2: We will obtain 5 8p (lP1-P2e~ Pala) = (Je dosure5(pi,2) ied where ais input € ©. “Paros Pa] € Qn- The states containing final so Step 3: The states obtained [ps {in p; isa final state in DFA. Definition of etosure The -closre (p) sa set of al states which are reachable from state pon easton such that: 1) ¢-cosue (p) =p where pe. ii) If there exists €- closure (p) = {q} and 8(q,¢) = r then € - closure (p) = {4 11 EET ine - closure for he fing NFA with. f { a OQ a Fig. 284 closure (qo) = {40,4 1-42} means self state + ¢ - reachable states. dose (q;) = {41,42} means qj i self state and qa isa state obtained from a, with ¢ input - dlosure (q2) = {q2} IO en TECINCAL PUBLICATIONS? - an prt ioe losure of each state e- closure (q2) = { Now we will obtain 8 transition. Let - closure (qo) = {40-41-42} call it as state A. 3(A,0) = €- closure = e- closure {5( (41.9 v 5(42.0} = e- closure {qo} = {40-4 BA.) = &- closure fe state A »} ) U Bq) U B(q2 AV} =e dlosure Call it as state B 41-43).2} = closure {5(40.2 U 8141.2) B(a2.2)} Bache {as} = {a2} ‘Thus we have obtained a(A.2) canine su c ay, O-O : RO aed eabre ax vane 2 eC br Boe © Hence (8,0) = e- closure {8(q1/4: Fig. 285 TECHNICAL PUBLICATIONS® an upitst for hnowedge5B. = = e- dlosure (5(91/2)Y 5(42/2)} Hence the DFA is as shown in . 2} in which final state q, lies B= {gaz} the state qy lies hence B is also final state in C = {qo}, the state 4, lies hence C is also a final state. (GEEEIERD constrict the NEA using thompson’s notation for the following rx ‘expression and then concert it to DFA a* (c|d)b*£# EE Fig. 2.8.8 )a) = e- closure (5 (aga) 5(41-)) itasB 3(A.b) = e-closu 8(q1.b)) = e-closure (@ =¢ BA) = 6 BAA) = 6 BAD = 6 U8(a6-a), UKa7,a)) ) Call as state C n uptint fr oweTECHNICAL PUBLICATIONS® - an up-hrust for knowledge Compl Design 2oat Levent Anaiyaie 13} Call it as state E Ha 5,2) 817 a) (as -b)V Hag-b)v 8q7,b)) e-dlosure(6(q3,45, 45,46 97)°) = e-dlosurel qs} =D = 9 44, 45,96,97)2)) losure(8((q3,94,9,96,97)9) of = o ,a) = e-closure(8(qs, 430, 411/413)-2) TECHNICAL PUBLICATIONS® - an upstst for owedSS - a Aen a) = Comper Design (an) 2 a) Call it as State F See aye ais ie. State G 3Ea) losure(8((q9~ 410,411, 9139) bE) =F BE,0 = SE a= % 8E09=6 ga) = e-ctosure(@((ai1- 412,433) 9)) HERI Coie the NEA for fing = 5G) = 8G.) =8(G.d) = 1G, =¢ == Seas = e-closure (1s) ie, State “os 2H a -B(A.a) = e-dlosure (6 = e-dosure (5 49,8) 8(ay, a) Ha, 844/20 8147 9) = e-closure (q3 as) = Cal ita state B 5(Aa) = B (a,b) = €-dlonure ( (go, 91-42-44°97)0)) at = Call tas state C aA.) = 8'B.a) = e~closure((B(Q)-42/49-44/9648-99)4) 92-43/94/46/48/49)-b)) “ Laven Anaya E e-closure ((5(4 1-42/44/45-4.),a)) = € closure (q 3) ~dlosure(q 2} +95 -46-5132)-2)) 92-44/45-467412)-b)) = e-closure (qs) Bb) = C TECHNICAL PUBLICATIONS® - an up for knowledge= (qo/4i-42-44-47-48 } Call it A -elosure (8(40, 41,42, 44,47, 48)-4)) ~ closure (43 qo) = &-closure (q3) U €- closure(qg) 42 43-44, 46-47-98]1¥ (49-410 422/413) 92+ 43-44r Go-47/98-49- 410" 412, dis} > B state - closure (8(40, 41, 42, 44, 47, 48)-b)) ~ closure (q5) 2-44 45+46/ 47/48! = Call it as state CLexie ng 2.50 Ara a= = e-cloeure (ay, 4243.44.96, 97. 5a = B = e-closure (qs) SE.) = e-closure (6(41 42, 94,95,46,47, 48411-4142 413)-b)) = e-closure (qs) = C 516) = c ————__._ —VvVw TECHNICAL PUBLIGATIONS®- an upstrat for imoutedge —— compar Design as The transition table is Fig. 2.8.11 2. Direct Method In this method the NFA with ¢ is built using Thompson's Construction Method. The € fs eliminated and then the NFA without ¢ is converted to DPA. Let us understand this ‘method with the help of exampl ttomaton to accept the regular expression 41)" is as follows - TECHNICAL PUBLICATIONS® «an upstrst for knowtodgeNow no new state getting generated. Hence we will write transition table as ~ oe ‘Step 5: Now we will eliminate € transitions and the FA which we will obtain will be follows Step 6: We now design transition table from Hel ad ead fm - i. - a @ 4 a faa) aa) Fal (4.43) (aad ed * [ordeal 195.45. 0 fo: a sseumed 0 be @ a new state [49,41] and {q9/q2} is assumed to be 2 ‘Hence we will find input transitions on these states. First we wl Ifwe assume A=[q0].B= ‘The minimize DFA will then be - fal ma) apa Qi) Bo. 4p Sad feel a1 leaal + [e884 9 96.9. [40-41] C= [40-42] P= [90-41-442-56 Lote poy, “aA fom glen regular expression 10+ (0-+12)0°1 ‘will construct the transition diagram for given regular expression 10 a? on Solution : First we CERIEEN cic 4 aang se ings of Hy dis wich oe _— tumbers. [CTU : Summer20, Marks 7 | TECHNICAL PUBLICATIONS® - an ups or knowledgeFig 20:13 ‘The rules for lastpos are same as the rules of firstpos except that a and b are reversed. “The strings that end with 0 are the even numbers, The computation of followpos function wil be using these rules - Rule 4: If node n represents concatenation with let child a and right child b and x is ‘3. DFA Tree Method aren rest retion m npc OF folk RS are infllowpos (0 2a eh ann” pe Rule 2: If nis a Kleen closure node and x isa poston in lastpos(n) then all positions a el aa comes in ips () are in fllowpos(o teitt “The firstpos and lastpos and fllowpos fr each node are as follows - = 02.3) o Solution : The grammar is sid to be an augmented grammar if iis a YG We will construct a syntax tree T for (ft ie (a[b)"¥ and then compute four “pa functions - 3 sae 2 at aa ly we wl construct DFA from fllowpos, aa oa ) * aa ‘Step 1: Construction of syntax tree ae ~ rN. F set (1,2, 3) =the 1 and 3 represents input ie. A. From set {1, 2,3) - the 2 represents ‘As no new states are getting generated, the DFA will be = Fig. 28.14 DFA for alt)" Lo DFA. 12)0%a PTUs Winter19, Marks 7 |Fain edad va nya wel cosas? vmAssume first pos (root) = {1} call it as state A we will obtain or transition on state A. Dtran [A, a] = followpos (1) = (23,4, 5) = State B * Diran IA, al = B _Dtran {A, b] = ¢, similarly Dtran (A, ¢] = Dtran (A, d] = Dtran [A, fl = 6. Dtran [B, a] means find out positions of a from B = (2, 3, 4, 5}. The node 2 denotes thefespos ©) fr mllable (C2) oF fstpos (C2) rr [seme cing) | Tasos 6) _ oe pos 2) seeCcomptr Design an Letcal Anaysis UB 6 7= (1, 2 3,4,5,6,7) Dtran (, ‘As no new state getting generated, we can construct DFA using above comp! as follows. As state B= {1,2,3,4,5,6, 7}, ie. It contains 7 a final state, we will consider B as ZI short Questions and Answers (G1 What aro the functions of lexical analyzer ? ‘Ana, : 1. It produces stream of tokens, 2. It eliminates blank and comments 3. It generates symbol table which stores the information about identifiers, 4 test on end of source language. One can av buffer using sentinels atthe end of buffer. For example: ‘eof. 3 What Is recognizer 7 ‘Ans. : Recognizers are machines. These are the machines which accepts the strings belonging to certain la the valid strings of such languages are accepted by the corresponding language is accepted by that machine, ample - 1. The finite state machines are recognizers for regular expressions or regular Soe ‘Dtran [B,b] = FOLLOWPOS(2) 2. The push down automata are recognizers for context free languages. = 112,3,4}0 (1, 2,3,4,5,6,7) Q4 What is the role of lexical analyzer 7 Dinan [8,6] = B Bite The Lexical analyzer scar th source program and separates out the lokans from Dtran [B,c] = FOLLOWPOS() U FOLLOPOS(6) i = * a = TECHNICAL PUBLICATIONS® - an uptust or knowledge TECNICA, PUBLICATIONS nwt! r moeapeidentifier. identifier will be ~ Ans. + The reg re. = letter (etter + digit! ‘The transition diagram will be a6 What are the possible error recovery actions in lexical analyzer ? ‘Ana. : Various possible error recovery actions in lexical analyzer a {)_ Deleting extra characters that might appear in the source statement. 41) Inserting missing character fi) Interchanging two adjacent characters. {v) Replacing an incorrect character by the correct character, 127 What isthe input to lexical analyzer generator ? What Is Its output? ‘Ane. The lexical analyzer generator takes the source program as input and g the stream of tokens as output. 8 Write the Regular expressions ‘Ans. 1. Regular expression for identifier is REsleter(letersdigi)* 2 Regular expression for integer number is RE=digitdigitt G2 Define tokens, pattems and lexemes. ‘Ans: Token : It describes the dass or category of input string, For exanpl: deniers, keywords, constants are called tokens. Patter : The set of rules that describe the token. Lexemes It represents the sequence of characters in the source program tha! & ‘matched with pattem of token, 10 Write the regular expression for the identifier and whitespace. ‘Ans: Refer answer of Q8 for regular expression of identifier Regular expression for white space is SS TECHRICAL PLBLICATIONS®- an updrut or omieage {for identifier and number. Ccompler Design Leica Anatysis “Qik Why Is buffering buffering methods 7 lysis 7 What are the commonly used ‘The one buffer and two buffer schemes ai 12 Write regular expression to describe a I consists of strings made of ) syntax analysers d) code generators Whether the token Is formed from the given patter or not is dependant upon 2) compiler ) target language ©) source language operating system a3 ‘Keeping track of the line numbers is a task of ) lexical analyser 4) syntax analyser ©) semantic analyser d) none of the above Q4 Following is a regular expression for identifier = a) letter b) letter Digit © Digit ) letter(ettersDigity™ QS Lexical analyser stores the input in @ input butfer fe) linked list Q5 A pattern can be represented by a) context free grammar~ [run time “E both at compile time and run time ‘d_ none of the above a1 In ex a) regul bl regular © yser generator sxpression are converted to equivalent NFA “pressions are converted to equivalent regular grammar A is converted to regular grammar. no conversion takes place (0.22 What is true about LEX _- fa) LEX is a lexical analyser generator LEX isa compiler © LEX is a database. |d_ LEXis a specification file used for generating token ‘Answer Keys for Multiple Choice Questions “= as Q2 a6 goa Syntax Analysis ‘Syllabus Understanding Poser ond CFG (Context Free Grammars) Tap Down and Bottom Up Parsing Algontoms Top-Down Parsing Bottom Up Parsing Operator Precedence Parsor LR Parsers ‘Simple LR Parsing (SLR) LAW Parser LAR Parser ‘Comparison of LR Parsers Using Ambiguous Grammars Parser Generators ‘Automatic Generation of Parsers Short Questons and Answers utile Choe QuestonsIEEE understanding Parser and cFG(Context Free Grammars) jon. The 5} x analysis isthe second phase in compilation, The 0 for the syntax of the language. A syntax analy ich a way that some Progra van be recognized. After grouping the tokens if at all any synt "hen apntctic error wil be generated. This overall process is checking of the language, inition = io which takes the arpaeer: A pring ov sts arlsis &a process i Dettion of part dur «pane ove (yas acre) F generates ey ‘The sbove programming statement is first given to lexical analyzer. The lexia ‘analyzer will divide it into group of tokens, The syntax analyzer takes the tokens x input, and generates a tree ike structure called parse tree, ‘ Concept of Context Free Grammar Aa NS Definition | The context free grammar can be formally defined as a set denoted by Fig, 34-1 Parse too for a= b + 10 G=(VLP,S) where V and T are set of non-terminals and terminals respectively. P is “he parse tee drawn above is for some programming statement. It shows how t# |_| $8 OF production rules, where each production rue is in the form of statement gets parsed according to their syntactic specification. | -Ron-terminal -> non-terminals REI note of Parser | Pemiemin rin S is a start symbol Jn the proces of compilation the parser and lexical analyzer work together. T= S85 means, when parser requires string of tokens it invokes lexical analyzer. In tur. * For example, Jexical analyzer supplies tokens to syntax analyzer (parser) P=(Sss+s Sases $36) $54) es "RICA PBLENTON cipal Se sions wane emmoecor enote the non-tenminals. to denote the terminals. We are using following 4. The capital letters are 2. The lower cas letters are used : (QED consi the CA forte regular expression (0+) Solution : The CFG can be given By, P= ($05 | 15 sae} of O's and 1's with the start symbol. Sin in this sete is a string, So in the rules w« AsaA | bA |e} ‘The rule § + AsAsA is something in which the two a's are maintained si to's should be there in the strings. And A —> aA |bA\e gives any combi ‘and bs ie. this rules gives the strings of (a +b) ‘Thus the logic fo this example will be {any thing) «(any thing) « (any thing) So before «or after a there could be any combination of a's and b's. Construct a grammar generating UES wew! where we { a,d}* Solution : The strings which can be generated for given L is {ancan, bb, abcbe, b> ‘The grammar could be SaSa Sobsb Sse Since the language L= w cw where w € (a + b)* Hence $+ a § a or $b S b. The string abcba can be generated from s°° Production rules as s aSa abSb abcba ————— TECHNICAL PUBLICATIONS® - an upstrt for moniedge $= > S5+ |SS*|a ‘Show thatthe string aava* can be generated bythe grammar. Construct the parse tree for The given grammar is not amt siven string fi) The production rules for producing strings having equal equal number of 1's is - S0B1A TECHNICAL PUBLICATIONS® «an up-to knoeior Daan a7 Syntax Anaysit pifference between Top down and Bottom up Parser Bottom up parser Algorithms: ‘As we know, there ae two parsing techniques, these parsing techniques work om by || arse tree can be built from root to leaves. Parse tee i built from leaves to toot H | ple to implement ‘Tis complex to to implement 2 This is simple to imp complex to BE ie ee eticent ping echnigpes vines te otom vp pres ends testis cure op cae naan principle, | down technique ae ambiguity left meee scans the input string from left to right and identifies that uy leftmost o rightmost | tis applicable o anal clas of languages tis applicable to abroad class of oS ngunges arin paring techniques ae 1) Recuive Various pring techniques ae 3) Sift Ascent parer 2) Preditve poner rence 3 Opetoe petedence 38 paver appropriate rules for .e can be constructed from root and expanded ‘When the parse ree can be constructed from leaves to root, then such type of pas: fs called as bottom-up parser. Thus the parse tree is built in bottom up manner. leftmost derivation matches this requirement. The ms find the appropriate production rule in order to produ SE ‘will understand the process of top-down parsing with the Consider a grammar. SoxPz Fig. 3.24 Parsing techniques: - TECHNICAL PUBLICATIONS® - an up-nt for knowing TECICAL PUBLEATIONS™ wept: mpegs Consider the input sting x72 as shown below, aainyaiea + Input ber Now we will construct the parse tree for above grammar deriving the given string, And for this derivation we will make use of top down approach. ‘Step 1: ‘The first leftmost leaf of the parse {input pointer. The next leaf node is P. We have to expand the node P. After expansion we get the node y which ‘matches with the input symbol y. Fig. 33.1 (8) Step 2: Now the next node is w which is ‘not matching with the input symbol. Fig. 33.1 (0) "EOWA FALCATON?- on wenciivhcoman compl Desin a9 Stop 3: We halt and declare that the parsing is ‘completed successfully ing luction ‘unsuitable for the string match then in that case the parse tree cannot be built. EERE Problems with Top - Down Parsin; 4) Backtracking Possibilities. This increases lot of Decomes necessary to eliminate the —A Lancair amma a grammar which given blow AtAa steps. can be non-terin vere £, means deriving te input in one oF more sept, The A ca sone some input string, If left recursion i present in the grammar the ae eerie problems. Because of left recursion the teplown parser can ener» {finite loop. This i as shown in the Fig, 333. Fig. 33.3 Loft recursion ‘Thus expansion of A causes further expansion of A only and due to generation of § ‘Aa, Aca, Acar, .., the input pointer will not be advanced. This causes major probi= r Parsing and therefore elimination of let recursion is a must a ee Tet recursion we need fo modify the grammar Let Ge a cone! Fer example : Consider the grammar EoE+T|T Wislesniesap Gls grammar with the re A A al B- Then ecg equation (2) we can say, Tren ws clint lft ecurion by rewring the production rule a = ay a=+T pine +-@) Ber Then the rule becomes, new symbol A is introduced _| ale verify, whether ‘modi srammar is equivalent to gia ent, We 8 abo verity, n er i fowtedge a — = au ee TECHNICAL PUBLICATIONS® - an updhrst fr kowedge‘compler Design a coer ter etn We map this grammar with the rule A-+ Aap B+ Bejb an! A AaB Sita hogy, use the rule A—+BA‘ AGA’, A’-+8.. Then we get Bo bp’ B+ eB’ Boe To summarize, the grammar without left recursion will be Asaa’ A‘ BAA’ EE oo Pe: B bB’ Solution : Consider the rule, ‘A-> ABd|Aala ee He eine gma wh oe le A> A ns Fr te slic oylng tne pend, ue 2 Sic cae aa ; SIAC sf te Ay aA’ =» At > BAA a lp From the following gremner and rewrite the grammar. tik es St wine Port Sumcntz nar} A le fSckaion nn Lee 54 Aalb A AaB 1 This left recursion can be eliminated as - A Ac|Sdle A = BA’ A > aA’ be the given grammar. If we replace $ by Aa|b in the rule A + Sd then the A’ > aA’ = A’ aa’ grammar becomes. ee Aro A Ac|Aad|bale io ume This rae is left recursive. We will map A —> Ac | Ac |B; |B to this rule AABd|Aala => A — gd’ ‘This gives A + B,A'|B2A' and A'>a,A'|aA'|e Avo Bia‘an” Bevis A ile As cA'lada'le B Be|b ———— = >| ae TECHNICAL PUBLICATIONS® TECHNICAL PUBLICATIONS® - an up-trst fr knowege + an upstust fr knowiedgo3 Compier Desay once ater eliminating the left recursion the grammar becomes $3 Aalb AS bdA'LA’ x oeailada'le CEERI eto nace it ite ee int fom te prone lms let recursive from flloing grammar. Eo mar which is as given below : Selution +The ef recursive grammar is 2g at se ‘To eliminate eft recursion we apply following rule. compiler Design 216 Syntac Arays ‘Solution : The rule for removal of let recursion is as follows - If A> Aqy| Acza|B,/Bp isa left recursive rule the, to eliminate let A> PiA‘|B2A” : Aro ayA’JazA je From given grammar, only one non-terminal ic gives left recursive grammar = B Bole We will map the above production rule with left recursive rule as BB ble fs tt A A a:By For eliminating left recursion BoB Br be" To summerize, the grammar on eliminating recursion, S$ aBDn Boa’ Br bb’ DEF Egle Fofle 3) Left factoring If the grammar is left factored then it bec table for the use. Basically fact it becomes suitable for the use ly left factoring is used when it is smatives is used to expand the non-terminal. By left te the production in ‘which the decision can be deferred until enough of the input is seen to make the rightIn general if Aat|oB2 then itis not possible for us to take a decision whether t0 choose fay SS re ch a situation the above grammar can be left factored as rule or second. In suct Asaa’ A> BilB2 For example : Consider the following, grammar. SEIS | iBtSeS | a E>b EEEIEEDS © factoring in te jotowing grammar — (A= 2AB|2A|a es Solution : If the rule is A + cf /a2|... is a production then the grammar needs to be lef factored. Consider We have to convert it to Asaa Asaa’ A-*bilba A’>ABl Ale a> slo boyy yay gate A ahab ree Scilly To summarize the grammar with lef factor operation will be - Aaa" A’ ABlAle B be’ Bo Ble Ee TEIWCAL PUBLCATONS® nwt br comin DOS” Bn am Perform the tft factoring of aa ablateb aioe Seaton: Fr ef fatigue A «pa — factoring we ction Stax Analysis Asan AS BilBa Consider ‘Then Asan Alo d]b|bele Hence we get Asaald A> dlb|bcle 4) Ambiguity ‘The ambiguous grammar is not desirable in top-down parsing. Hence we need to remove the ambiguity from the grammar if tis present. For example : ESE+E| E+E lid is an ambiguous grammar. We will design the parse tree for id + id * id as follows. © © Koo Oo & 608 © @) o © © (0) Parse tree ¢ (0) Parse toe 2 Fig. 23.5 Ambiguous grammar TECHNICAL PUBLICATIONS® - an upd fr knowdye— an 18 es OR = is one rule : If the grammar te seeing embi lY e e M Fi doe opertor (euch 09417) % recur “ ae were associative operator (uch Gparator (exponential operator) then induce th 5, Cos ANY ‘grammar has right fae fT SL Meet Ot ° ‘The unambiguous grammar f a EET EST ToT ‘Unambiguous production rules are as given below. The ambiguity arise due to use nly one non terminal Le. 5. We will ue non leminal A tat donuts snatch statement a ‘and B denotes unmatched statement. The unambiguous grammar will be ~ a SAB oe one thing that the grammar 1s unambiguous but itis left recuse mj! 4p BtAeA La cumini of such left econ i gain 8 st Aiea 4b CRETE eck wither folowing grammar is ambiguous or not ‘The parsing of string ibtibtaea 830s Sse Solution : Consider the input string (())() for which we will draw the parse tree. s Left Factoring - Refer section 3.3.1 (3) ‘As only one parse tree can be generated for given string, we can say that g6 ee | > ee sey emer coal ae of tsa ne Ibe canbe pared. Following parse tees show the p= TEGRICAL PLBUCATIONG® on et money TECNICA PUBLCATIONS® ws apa orcompler Design 3.21 eee Iylldthe recursive routines, The RHS. ofthe proiution mule ls directly converted to a procedure (code) 8 RHLS. ofthe corresponding wosteviny Basie stops for construction of RD parser The RHS. ofthe mle is czety converted ito program code symbol by symbol the non-terminal is made. 2. Ifthe input symbol is terminal then i s matched with the lookahead from input The lookahead pointer has tobe advanced on matching ofthe input symbel is é OBSE+ alte to ‘combined into a single body of procedure. 4, The parser should be activated by a procedure corresponding to the start symbol. Let us take one example to understand the construction of RD parser. Consider the ‘grammar having start symbol E. E Fig. 33.6 Types top - down parsers, ‘here are two types by which the top-down parsing can be performed. 1. Backtracking, A parser that uses col of ing ea lection of recursive procedures for parsing the given Descent (RD) Paner. In this type of parser the CFG is used 3° - an up-thrust for knowtedge hz = : TECHNICAL PUBLICATIONS” PUBLICATIONS - an uptua for Knowle program. For each nonderminal a separate prosture is writen and body of the 1 If the input symbol is non-terminal then a call to the procedure corresponding 10“riplectthe following grammar using recite decent SED ee Ee A eS A ee TEGHNIGAL PUBLCATIONS® - an uous fr Kowlee =rex | ‘Advantages of recursive descent parser 1. Recursive descent parsers are simple f° build. aceasta vecomeooea cee" imation of recursive descent parse specie decent pares ae Tat ‘cia oe one I a ome np ease pare can nt provide ood ever es i is ancl to pene te sing if lookahead symbol is atbltrerly long very efficient as compared to other pariy FEED Provictive LL(1) Parser | ths top-down parsing algorithm i of nonrecusve Pe: £ «In this type of parsing a table is built. ieee tae eat emt. | © er table MUA entry at MAA er eport Err. «For LL() the first L means the input is scanned from left to right, The second tatens it uses leftmost derivation for input string. And the number 1 in the ive ‘Symbol means it uses only one input symbol lookahead) to predict the pus process. “The simple block diagram for LL(1) parser is as shown in Fig. 33.8. Refer ble MAA] i a Aeerpektenpeaoapar | hence this type of parsing method is * The configuration of LL() p: lookahead token. + One by one configuration is performed and the inp opt [Te token PTs paner reaches the hating configuration. tele Output eae crtemin a) a Panning ble steps A 3 C For construction of Predictive LL() parser we have to follow the folowing steps - Fig, 338 Model for LL) parser Step 1 + Computation of FIRST and FOLLOW func TECHNICAL PUBLICATIONS® - an upxtnt for knowndpe Si ase Ce peo stop 2; conotct he Predictive Parsing Table using FIST and FOLLOW fet ive Parsing Table compiler D0" 0 solution : Fo red tere mbal # RIS. is T. Now T- FT in which the tomb ft RHLS. is F and ther isa mile for Fas F-+ ya FIRST(E) = FIRST(T) = FIRST(R) Parse the input string with the help of Predict ‘a set of terminal symbols that are first symbols appearing if ase then € is also in FIRST (a). Foid | TEP pce FbyRHS rule ARSE) = FIRSTOD = FST = (Ga) eg FIRST(E) = (+7 ©) FOLLOW (A) is defined as the set of terminal symbols that appear immediately | the right of A. In other words ro FOLLOW(A) = (a | Sa Aa B where a and are some grammar symbols terminal or non-terminal. Bimmer ase SUL teeny wpb - ‘The first terminal symbol appearing at RH.S. of production rule for T’ is added in the FIRST function. Now we will compute FOLLOW function. FOLLOWEE) - 4) As there is a rule F-+(E) the symbol‘ appears immediately to the right of E. Hence ‘Y will be in FOLLOWEE). 5 Te computation rules eee ae a aa =( B=EB=). FOLLOW) = FIRST) = FIST) =()) FOLLOWE) = {)} Since E is a start symbol, add § to follow of E eee bs) | = BS cepa ni'2 everything in 1B) ie. everything in FOLLOW(E) is in FOLLOW(E). TECHNICAL PUBLICATONS® - an uett for knowincompte basin 3:22 __ Siriertoany 4) E> TE the computational ules A #036. he tea a +t, B= &, Pe then by computational rule 3 everthing iy * OLLOW(A) isin FOLLOWG) Le. everything in FOLLOW(E) is in FOLLOW) FOLLOWE) = (05 J] 4Te’ 8) Consider [E> TE’ we will map it with A >a BB A=E,a-e,B=T,=B=E by computational rule 2 FOLLOW(B) = (FIRST () -¢. ‘That is FOLLOW(T) = (FIRST(E) ~€) 4i) Consider E’-> +TE’ we will map it with A 0. BB A=E,a=+,B=T,B=E’ by computational rule 3 FOLLOW(A) = FOLLOW) We will map this rule with A—aBB then A=T, a=F, BeT’, B=e the FOLLOW(T) = FOLLOW(T) = [+,)) T3r We will map this rule with AaB then) A=T, F ay has, Bar, Bae’ te FOLLOW(T) = FOLLOW( a="F, B=T’, 6: Algorithm for predictive parsing tab! competence FOLLOW(F) essen ental re 2 ae rie =p | acaBB Ponce Act ase BaF BT FOLLOWG) = (FIRST () el | FOLLOWG) = (FIRST) ~ 2) | FOLLOW) = (") Consider T'-> *FT’ by computational rule 3 roar AsaBB AzT,a=*B=FB=T A=T,as' Bar pet FOLLOWG) = (FIRST) —e) FOLLOWG) = (FIRST) ~e} FOLLOWE) =") Hence[FOLLOW(T) = Bhs construction of predictive parsing table is an important activity in predictive Sa Parsing method. This algorithm requires FIRST and FOLLO' 7" TECHNICAL PUBLICATIONS® ~ on ipa fr a —_ on igual tr wont = TECHNICAL PUBLICATIONS® - an upsAlgorithm : For the mule Aa of grammar G |. For each a in FIRST(a) create entry M{A.a] = A -»a where a i terminal symbol, amebrisih renege ve computed FIRST and FOLLOW of all the non-terminal symbols. ‘Now we will fill up the entries in the tab ‘consider each rule one by one. e using the above given algorithm. For that37 Syras kraiyss complerenn star Ara [Now the input string id + id + 1d $ can be parsed using above able. At the initial canton the stack wil ena stk abo he np fer opt ingS— BoBa Ase Boe [Now we will compute FIRST and FOLLOW functions FIRST(S) = {a,b} vera ' rad 7, ee. ae | IRST(B) = The first terminal sym ‘AaAb ‘appearing on RHS. ‘of production rule for B S>aAb When Ae inst@®) = {b,e}Sa compte 0069 comer oseen__ (CEEEIDREDDD coi st sromrsr s A bare teming FOLLOW for given nonterminals. FOLLOWG) = {eS} FOLLOW(A) = {e/$} FOLLOW) = {t} arsing table will be ce =. t i s Sica rary ae Ane eS cob Lo aaa ‘As we have got multiple entries in M[A,e] given grammar is not LL(Q) gramma The (it, ¢,a,b) are the terminal symbols because they do not derive any product rule Construct the predictive parser forthe following grammar. Sala RE LoL s|s Solution : As the given grammar is left recursive because of Lousis. ‘We will first eliminate left recursion . As A > A aijB can be converted as ASB A salle We can write L +L, §|$ as Losu Uo sue TECHNICAL PUBLICATIONS® an uptst for gamplerbesien gy Now the grammar taken for predictive parsing - sola Losi’ V5 ,sU'le Now we will compute FIRST and FOLLOW of non - terminals ee Vor vase (BREMBR ct i hi of teprer ote sen, 2) wing he grammar “Specified above. S > (L)\a Solution : “As we have constructed a predictive parsing table in Example 33.19 we will Parse the string (a, a) using that table as shown below. Sa ne | sec sow oe Lasu’ [sus soa [awe nas fees oe TECHNICAL PUBLICATIONS® - an vp-tvst or knwoSisto, CERERED cone RST ent FOLLOW fo al notin 0 ey rn Sst Bel Be hod Bod Solution : FIRST(S) = first terminal symbol appearing on RHS. Consider the rule $+ Aa which can be mapped with AaB B. Tht with ane A BB ‘Then FOLLOW(A) = FIRST (f) = FIRST(c) = {o} 4: FOLLOWIA) = {a} TECHNICAL PUBLICATIONS® an upstrust fo knomiedge ear rata Consider the rule, porte A aBB FOLLOW(B) = FIRST(S) = FIRST(a) = {a} ees A BB FOLLOW(B) = FIRST(®) = FIRST(c) = {c} = FOLLOW() = {a,<} with ame GEISEREESD @ tie fottowring grammar suitable for LU) parsing ? if not make it suitable for BED peng Compute FIRST and FOLLOW ses. Core he paring ble > AB B~ BaAC This can be liminated by rewriting the production rle as AnBa’ Bock AoA = Bs aAce Roe a TECHNICAL PUBLICATIONS® - an ups for Arowecve —Solution : Let the grammar be S > AcB|1AB|e TAC | 0c rite or Wale driven pare forthe given grammar. 1s the Basel Coc EEN — A AaB Ix B4BCb Icy e3ce lsee mug nies inca Y column, given grammar is not LL (1) Tint | float poet ie ig rama ig tle ren parser end check we Se nat S> aBDh, B-> eC, C> C/N, D> EF, E> 9!" F> 2) Find first and fellow for eck non-termial for resultant grammarEERE cresting groner gL) or at 7 Sable Bricle csi Solution Let, Sable Bobcle cassie ‘The FIRST and FOLLOW are computed as Symbol FIRST FOLLOW 3 ke 6 a be | ce wo ‘The parsing table will be oo /boabaelre s 5 |S—aB Soe 8 Boe Boe © C58) cor ‘simulation = Consider derivation of abe. State ss $B so Ss (GENER strict 210) parsing tate or he towing grammar EGTU Winter-18, Marks 7] Solution + Let, the given grammar is - ES ET|T ToT FF F5®)la ‘The given grammar has some rules that are recursion, using the rule. If A > Aa, | Act2 |B, |B> is a rule then AS BIA'IB2A A’ a,A"|az4%e Hence on eliminating let recursion we get Bote FE’ 4TE’e TH Torre Fo @iid For constructing LL) parsing table for above grammar, unple 33.11, GRRE fers) parsing tate fr ftowing grammar. Check wheter tie ri LD or AaB Asx BoBC Bocy Cace —— - — TECHNICAL PUBLICATIONS - an uprutforkrowodge e. We will remove left8 ty como asin meee r- ‘We wit find FIRST and FOLLOW fr amine tution : We wl , FIRST FOLLOW ra ws is B fey toy c ed bend ‘he predictive parsing table will be» 2 » x y ‘ ; a hoe eae Be ‘B+ BCb i Baty aor eae ig Cac c ‘As there are multiple entries in MIB, y] the given grammar is not LL(1) Wie a agri for nating Uf con (Refer Section 33.40 rer the Bock gram of a 5. Write oon he agorth orf tring. Refer Section 3.3.19) [ro wine 13.19. Na Expansive! pce parang, (Refer Secon 3.321 Go 5: Wat sf facing and of recurson? Expcn it with state example. ES 6 Wats ambiguous grammar? Desrie with example Co EZ] Bottom Up Parsing EEE + In bottom-up parsing method, the input string is taken first and w this string with the help of grammar and try to obtain the start * The process of parsing halts successfully as soon as we reach to 2. Explain non recursive TECHNICAL PUBLICATIONS® - an up-hust for knowiedy? Step 1 : We will star from leaf node conpier Deion =o 3-85 a =, The parse te i constructed from botiom to up ‘ In this process, the input symbols are placed parsing, «The bottom-up parse tr reduced further to inter eventually a root node is ‘The internal nodes are created ‘Thus the crucial but prime tash that can be used for reduction, * The bottom-up parse tree construction process in. derivations are to be done in reverse order. For example Consider the grammar for decl SoTL; To int Lou statement, ‘The input string is float id,idid; Parse Troe Step 2: S Step 3: Reducing foat to 7.7 + oat Stop 4 : Read next sting from input