lecture #5 began here

Avoid These Top 10 Homework #1 Bugs in Your Homework #2

1. yytext or yyinput were not declared global

2. main() does not have its required argc, argv parameters!
3. main() does not call yylex() in a loop or check its return value
4. getc() EOF handling is missing or wrong!
5. opened files not (all) closed! file handle leak!
6. end-of-comment code doesn't check for */
7. yylex() is not doing the file reading
8. yylex() does not skip multiple spaces, mishandles spaces at the front of input, or requires
certain spaces in order to function OK
9. extra or bogus output not in assignment spec
10. = instead of ==

Introductory Comments on BASIC

We are doing (a large subset of) TRS-80 Color Computer Extended BASIC. Compared with last
semester, you will get more help from me on getting started with each assignment, and be
asked to do a little more.
Thema inl a ng uager e
fe re
nc emanualsareonourc l
as swe bpage
http:
// www. c s .
nms u.
edu/~je f
fery/
cours
es/370/
Ia mwr itingmyownc omme ntar
yoni t
http:
// www. c s .
nms u.
edu/~je f
fery/
cours
es/370/basi
c.
html wheremos tofy ourformal
speci
ficationswi llappe ar.
Iha vema deas t
artatsome
pseudoc ode ht tp: //www. cs.nms u.
edu/~j
effer
y/cours
es/37 0/basi
c.
icnforthe" inter
pret
e rloop"
thatwi l
lwr apa roundy ourc ompile
r.

Some BASIC Regular Expression Examples

I
napr evi
ouslecturewes awreg
ularex
press
ions,t
hepref
err
ednotati
onforspe
c i
fyi
ngpat
terns
ofc
ha ra
ctersthatdef
inetokencat
egori
es.Thebestwa
yt ogetafe
elforreg
ularexpre
ssi
onsis
toseee xampl
e s
.Notet hatr
egula
re x
press
ionsformthebasi
sforpatt
ernma tc
hinginmany
UNI
Xt oolssuc
ha sgrep,
a wk,
perl
,etc
.

Wea r
edoingasubse
tofBASICinthiscour
se,s
owhati
sthereg
ula
rexpre
ssi
onforeachoft
he
di
ff
erentle
xic
ali
temsthatappearinBAS I
Cprogr
ams
?T hi
sist
hefir
stofmanything
swhere
BASI
Cissomewhateas
iert
ode a
lwiththanC.

lexical
BASIC notes, and comparison with C
category
 Foroperat
orstha tareregularexpressi
onoper at
or swenee d
ma r
kthemwi thdoubl equot esorbacks l
ashest
oi ndica
te
t
hecha
ract
ers youme anthecha racter,
nott heregularexpres
sion
ope
rat
ors
t
hemse
lves operat
or.Notes everaloperatorshaveac ommonpr efi
x.The
l
exical
analyze
rne e
dst olooka headtot el
lwhethe ran=is
anassi
gnme nt,orisfoll
owe dbya nother=f orexample.
t
heconcat
enat
ionof
r
eserv
ed Reser
vedwordsar
eal
somat
chedbyther
egula
rexpr
ess
ion
c
harac
ters
;ca
se
words fori
dent
if
ier
s,s
oadis
ambi
guat
ingr
ulei
sneeded.
i
nse
nsit
iv
e
no_ ;
$a tendsof
some;2s i
gni
fic
ant
i
dent
if
ier
s [
a-z
A-Z
_][
a-z
A-Z
0-9]
*
l
e t
ter
s!?;cas
e
i
nsensit
ive
i
ntsandreal
s,
numbe
rs 0x
[0-
9a-
fA-
F]+e
tc.
s
tart
ingwit
h[0-9]
+
c
omme
nts RE
M.* C'
scomme
ntsa
ret
ri
ckyr
egex
p's
al
most".
*";
no
s
tri
ngs es
cape
dquot
es
es
capes
wha
tel
se?

lecture #6 began here

lex(1) and flex(1)

The seprogramsg eneral

lytakealex
icals
pecif
ic
ati
ong i
veni
na. lf
il
eandc reat
eac orr
esponding
Cla nguagelexicalanal
yzerinafil
ena medlex.
yy.c
.Thelexi
calanal
yzeristhenl
inkedwi t
ht he
restofy ourcompi l
er.
TheCc odeg enerat
edbyl exhasthefoll
owingpubli
cint
erf
ace.Notetheus eofg
lobalv a
ria
bles
i
nste adofpa ramet er
s,andt heuseoft heprefi
xyyt odi
sti
nguis
hs cannername sfrom your
prog r
amna me s
.Thisprefi
xisal
sousedintheYACCpa rs
ergener
ator.

F
ILE*yyi
n; /
*setthi
svari
ablepri
ortoc al
l
ingyy
lex
()*/
i
ntyy
lex
(); /
*cal
lthisf
uncti
ononc eforeacht
oken*/
c
haryyt
ext[
]; /
*yyl
ex (
)wri
testhetoken'sl
exemetoana rr
ay*/
/
*note:wit
hflex
, I
be l
i
evee xt
erndecl
ara
tionsmustr
ead
e
xter
ncha
r*y
ytex
t;*/

T
he.lf
il
eformatc
onsi
st
sofamixt
ureofl
exsy
nta
xa ndCcodefr
agment
s.T
hepe r
cents
ign(
%)
i
sus
edtos i
gni
fyl
exe
lement
s.Thewhol
efi
lei
sdi
videdi
ntothr
eesec
ti
onssepa
rat
edby%%:

hea
der
%%
body
%%
hel
perf
unc
ti
ons

T
hehea de
rconsi
st
sofCcodefra
gmentsencl
osedin%{and%}a swella
sma c
rode f
ini
ti
ons
c
ons
ist
ingofanameandaregul
arexpre
ssi
ondenotedbythatname.l
exmacr
osa r
einvoked
e
xpl
i
cit
lybyencl
osi
ngt
hemacronameincur
lybrac
es.F
oll
owingar
esomeexampl
elexma c
ros.

l
ett
er [
a -
zA-
Z]
di
gi
t [
0-9]
i
dent {
lett
er}
({
let
ter
}|{
dig
it}
)*

Thebodyc onsist
sofofas e
que nceofr egularexpr
essi
onsf ordiffer
enttokenc at
egor
iesa nd
otherlexi
calenti
ties
.Eachreg ul
arexpr e
s s
ionca nhaveaCc odefragmentenc l
osedinc url
y
bracesthate xecut
eswhe nt ha tregulare xpr
essi
on i
sma t
ched.F ormos toft heregular
expres
s i
onsthi
sc odefrag
me nt(als
oc all
edasemantic action consi
stsofret
urninganinteger
thati
dentif
iesthetokencategorytother estofthecompil
e r,parti
cul
arl
yforusebyt hepa rs
er
tochecksyntax
. S
omet ypi
calregula
rex pres
s i
onsandsemant i
ca cti
onsmighti
nclude:

"" {/*no-op,disc
ardwhi
tespa
ce*/}
{i
dent} {ret
urnIDENT I
FIE
R;}
"*" {ret
urnAST ERI
SK;}
".
" {ret
urnPERIOD; }
Youa l
soneedr
egul
arexpres
sionsforl
exi
caler
ror
ssuc
hasunt
ermi
nat
edc
har
act
erc
ons
tant
s,
oril
l
egalcha
rac
ter
s.

Thehel
perf
uncti
onsi
nalexfi
let
ypic
all
ycomput
ele
xic
ala
ttr
ibut
es,s
ucha
sthea
ctua
lint
ege
r
orst
ri
ngval
uesdenot
edbyl
it
eral
s.

A Short Comment on Lexing C Reals

Cfloatanddoubl econs tantsha vetohaveatlea

stonedi
git
,eit
herbef
oreoraf
terther
equi
red
decimal
. Thi
sisapa in:
([
0-9]+.
[0-9]
*|[ 0-9]*.
[0-9]+)...
Youmi ghtal
mos tbeha ppierifyouwrote
([
0-9]*.
[0-9]
*) {r etur
n( strcmp(y y
text
,"
.")
)?REAL:PERI
OD; }
You-al
lknowC' sternarye 1? e2: e3opera
tor,
don'ty
a?It
sanif-
then-
els
eexpr
essi
on,ve
rysl
i
ck.

Lex extended regular expressions

L
exfur
therext
endsther
egul
are
xpr
ess
ionswi
ths
eve
ralhel
pfulope
rat
ors
.Le
x'sr
egul
ar
e
xpr
ess
ionsi
ncl
ude:
c
nor
mal
cha
rac
ter
sme
ant
hems
elv
es
\
c
back
sla
shesc
apesr
emovethemeani
ngfrommostoper
atorc
har
act
ers
.Ins
idec
har
act
er
set
sandquotes
,ba
cks
las
hperf
ormsC-s
tylee
sca
pes.
"
s"
Doubl
equot
esme a
ntomat
chtheCs
tri
nggi
vena
sit
sel
f.T
hisi
spar
ti
cul
arl
yusef
ulf
or
mult
i-
byt
eoper
atorsa
ndmaybemor
ereada
blet
hanus
ingbac
ksl
ashmult
ipl
eti
mes.
[
s]
T
hisc
har
act
ers
etope
rat
orma
tche
sanyonec
har
act
era
mongt
hos
eins
.
[
^s]
Aneg
ate
d-s
etma
tche
sanyonec
har
act
ernota
mongt
hos
eins
.
.
T
hedotope
rat
orma
tche
sanyonec
har
act
ere
xce
ptne
wli
ne:
[^\
n]
r
*
ma
tchr0ormor
eti
mes
.
r
+
ma
tchr1ormor
eti
mes
.
r
?
ma t
chr0or1t ime.
r
{m,n}
ma t
chrbe twe e
nma ndnt ime s.
r
1r2
concatenat
ion.ma t
c hr
1followedbyr 2
r|r
1 2
al
ternati
on.ma tchr1orr2
(
r)
parenthese
ss peci
fyprecede ncebutdonotma tc
ha ny
t hi
ng
r/r
1 2
l
ook ahead.ma t
chr1wh enr 2fol
lows,wit
houtconsumingr2
^
r
ma t
chronl ywhe nitocc
ur sa tt
hebeg i
nni
ngofaline
r
$
ma t
chronl ywhe nitocc
ur sa tt
heendofal i
ne

Lexical Attributes and Token Objects

Bes i
de sthet oken'sca teg
ory,ther es toft
hec ompil
ermayneedsevera
lpie
cesofinf
ormat
ion
aboutat ok eni nor de rtope rf
orms ema
nti
ca naly
sis
,codegene
rati
on,anderrorhandl
i
ng.
T hesea res toredi nanobj ectinstanc eofcl
assToken,ori
nC,astr
uct.Thefi
el
dsaregener
all
y
some thingl ike:
structtok en{
intc a
te gory ;
cha r*text;
intl
inenumber ;
intcolumn;
char*f i
lename ;
unionl i
teralvalue;
}
T heuni onl i
teralwi llhol dc omput edv al
ue sofi nteg ers,realnumbe rs,ands t
rings.In your
homework assignment, I am requiring you to compute column #'s; not all compilers require
them, but they are easy. Also: in our compiler project we are not worrying about optimizing our
use of memory, so am not requiring you to use a union.

Flex ManpageExamplefest

ToreadaUNI X"manpage",ormanualpa
ge ,
y outype"mancommand"whe r
ecomma ndisthe
UNIXprogr
am orli
brar
yfuncti
onyouneedinformati
onon. Readthema npageformantolearn
moreadvanceduses(
"manma n")
.
I
tturnsoutthefl
exma npageisi
ntendedtobepr ettycomplet
e,e noughsothatwecandra w
ourexamplesfr
om it
.Perhapswhatyoushoul df
igureoutfrom theseexamplesi
sthatfl
exis
act
uall
y.
..f
lexi
bl
e.Thefi
rsts
e v
era
lexamplesus ef
lexasafi
lte
rfrom standa
rdinputt
os t
andard
out
put.

• s
neak
ys t
ri
ngremovaltool:
• %%
• "z
apme "
• e
xce
sswhites
pacetr
immer
• %%
• [\t
]+ put char('');
• [\t
]+$ / *ignorethi
stoke
n* /
• s
neak
ys t
ri
ngsubst
it
utiontool:
• %%
• user
name printf("
%s "
,get
log
in(
));