$'HHS/HDUQLQJ)DFLDO([SUHVVLRQ5HFRJQLWLRQ

EDVHG6FRULQJ6\VWHPIRU5HVWDXUDQWV
:DQ-XQJ&KDQJ 0LULDP6FKPHO]HU)ORULDQ.RSS&KLD+DR+VX
-LDQ3LQJ6X/LDQJ%L&KHQDQG0LQJ&KH&KHQ

'HSDUWPHQWRI(OHFWURQLF(QJLQHHULQJ6RXWKHUQ7DLZDQ8QLYHUVLW\RI6FLHQFHDQG7HFKQRORJ\7DLQDQ7DLZDQ

'HSDUWPHQWRI&RPSXWHU6FLHQFH8QLYHUVLW\RI$SSOLHG6FLHQFHV&RQVWDQFH+7:*&RQVWDQFH.RQVWDQ]*HUPDQ\

'HSDUWPHQWRI(OHFWULFDO(QJLQHHULQJ8QLYHUVLW\RI$SSOLHG6FLHQFHV&RQVWDQFH+7:*&RQVWDQFH.RQVWDQ]*HUPDQ\

$UWLILFLDO,QWHOOLJHQFHRYHU,QWHUQHWRI7KLQJV$SSOLHG5HVHDUFK&HQWHU$,R7&HQWHU
6RXWKHUQ7DLZDQ8QLYHUVLW\RI6FLHQFHDQG7HFKQRORJ\7DLQDQ7DLZDQ
^ DOOHQFKDQJOLDQJELFKHQMHUU\KDWD`#VWXVWHGXWZ^PLULDPVFKPHO]HUIORULDQNRSS`#KWZJNRQVWDQ]GH

Abstract—Recently, the popularity of automated and

unmanned restaurants has increased. Due to the absence of staff,
there is no direct perception of the customers' impressions in
order to find out what their experiences with the restaurant
concept are like. For this purpose, this paper presents a rating
system based on facial expression recognition with pre-trained
convolutional neural network (CNN) models. It is composed of an
Android mobile application, a web server, and a pre-trained AI-
server. Both the food and the environment are supposed to be
rated. Currently, three expressions (satisfied, neutral and
disappointed) are provided by the scoring system.

Keywords—automated restaurants; deep learning; facial

recognition; restaurant scoring; unmanned restaurants; Internet of
things (IoT)

, ,1752'8&7,21
,Q UHFHQW \HDUV WKH UDSLG GHYHORSPHQW RI LQIRUPDWLRQ DQG
FRPPXQLFDWLRQWHFKQRORJ\,&7WKH,QWHUQHWRI7KLQJV,R7
DQG DUWLILFLDO LQWHOOLJHQFH $, KDV UHVXOWHG LQ DQ LQFUHDVLQJ D
QXPEHURIDSSOLFDWLRQVEDVHGRQWKHVHWHFKQRORJLHV)ROORZLQJ
WKLV WUHQG WKH SRSXODULW\ RI DXWRPDWHG DQG XQPDQQHG
UHVWDXUDQWVFRQWLQXHVWRJURZ3DUWLFXODUO\LQ-DSDQDQG7DLZDQ
WKH QXPEHU RI DXWRPDWHG RU XQPDQQHG UHVWDXUDQWV WKDW UXQ
VXFFHVVIXOO\LVJURZLQJ,Q)LJH[DPSOHVRIWZRUHVWDXUDQWV
RIWKLVW\SHDUHVKRZQ
6DHHG et al. >@ SURSRVHG D FORXGEDVHG VPDUW UHVWDXUDQW
PDQDJHPHQWV\VWHPZKLFKLVFRPSRVHGRIDQ$QGURLGPRELOH
DSSOLFDWLRQIRUFXVWRPHUVDQGDZHEDSSOLFDWLRQIRUUHVWDXUDQW
VWDII PHPEHUV 7KH PRELOH DSSOLFDWLRQ SURYLGHV WKH FXVWRPHU
DPRQJRWKHUWKLQJVDQLQWHUDFWLYHPHQXIRURUGHULQJGLVKHVDQG
SD\LQJWKHLUELOO
$VWKHUHLVQRVWDIIDYDLODEOHLQXQPDQQHGUHVWDXUDQWVLWLV
GLIILFXOW IRU WKH UHVWDXUDQW PDQDJHPHQW WR HVWLPDWH KRZ WKH
FRQFHSW DQG WKH IRRG LV H[SHULHQFHG E\ WKH FXVWRPHUV ,Q
DGGLWLRQ LW LV QRW SRVVLEOH WR HVWLPDWH WKH DJH DQG JHQGHU
FRPSRVLWLRQ RI FXVWRPHUV WR VXFFHVVIXOO\ SHUIRUP WDUJHWHG
PDUNHWLQJ ([LVWLQJ UDWLQJ V\VWHPV VXFK DV *RRJOH DQG
E
7ULS$GYLVRU RQO\ SDUWLDOO\ VROYH WKLV SUREOHP DV WKH\ RQO\ )LJ  7ZR DXWRPDWHGXQPDQQHG UHVWDXUDQWV H[DPSOHV D $Q DXWRPDWHG
FRYHUDSDUWRI WKHFXVWRPHUV¶ RSLQLRQV7KHVHUDWLQJ V\VWHPV UHVWDXUDQW RSHDUWHG LQ 7RN\R -DSDQ E $ XQPDQQHG UHVWDXUDQW RSHDUWHG LQ
DUH RQO\ XVHG E\ D VXEVHW RI WKH FXVWRPHUV ZKR UDWH WKH .DRKVLXQJ7DLZDQ3KRWRJUDSKVFRXUWHV\RI,QWHUQHW

7KLVZRUNLVVXSSRUWHGLQSDUWE\WKH0LQLVWU\RI6FLHQFHDQG7HFKQRORJ\
0R67 7DLZDQ XQGHU JUDQW 0267 (.*Corresponding
author: Prof. Wan-Jung Chang.

‹,(((  ,&$,,&

UHVWDXUDQW RQ LQGHSHQGHQW UDWLQJ SODWIRUPV RQ WKHLU RZQ
LQLWLDWLYH 7KLV DSSOLHV PDLQO\ WR FXVWRPHUV ZKR H[SHULHQFH
WKHLUYLVLWDVYHU\SRVLWLYHRUQHJDWLYH
,Q RUGHU WR VROYH WKLV SUREOHP DOO FXVWRPHUV PXVW EH
PRWLYDWHG WR JLYH DUDWLQJ7KLVSDSHU LQWURGXFHVDQDSSURDFK
IRU D UHVWDXUDQW UDWLQJ V\VWHP WKDW DVNV HYHU\ FXVWRPHU IRU D
UDWLQJDIWHUWKHLUYLVLWWRLQFUHDVHWKHQXPEHURIUDWLQJVDVPXFK
DVSRVVLEOH,WUXQVZLWKLQWKHIUDPHZRUNRIDQ$QGURLGPRELOH
DSSOLFDWLRQ UXQQLQJ RQD PRELOHGHYLFHWKDWLV PRVWO\ XVHG LQ
XQPDQQHGUHVWDXUDQWVDQ\ZD\DVVHHQIRUH[DPSOHLQ>@7KH
VFRULQJ V\VWHP LV EDVHG RQ IDFLDO H[SUHVVLRQ GHWHFWLRQ XVLQJ
SUHWUDLQHG FRQYROXWLRQDO QHXUDO QHWZRUN &11 PRGHOV ,W
DOORZVWKHFXVWRPHUWRUDWHWKHIRRGDVZHOODVWKHHQYLURQPHQW
E\WDNLQJDSLFWXUHRIKLVIDFHWKDWUHIOHFWVWKHFRUUHVSRQGLQJ
IHHOLQJV
&RPSDUHG WR WH[WEDVHGUDWLQJ V\VWHPV WKHUH LV PXFK OHVV
LQIRUPDWLRQ DQG QR LQGLYLGXDO H[SHULHQFH UHSRUWV FROOHFWHG
+RZHYHU WKLV VLPSOH IDVW DQG SOD\IXO UDWLQJ V\VWHP VKRXOG
JLYH D ZLGHU UDQJH RI RSLQLRQV DERXW WKH H[SHULHQFHV RI WKH
FXVWRPHUV ZLWK WKH UHVWDXUDQW FRQFHSW ,Q DGGLWLRQ VWDWLVWLFV
DERXW WKH HVWLPDWHG DJH DQG JHQGHU FRPSRVLWLRQ RI WKH
FXVWRPHUVFDQEHJHQHUDWHG )LJ$SSOLFDWLRQVFHQDULRRIWKHUHVWDXUDQWVFRULQJV\VWHP

,, $5&+,7(&785($1''(6,*1
IURP WKH ZHE VHUYHU DV VRRQDV WKHUHLV RQHDYDLODEOH7KLV LV
)LJ  VKRZV DQ DSSOLFDWLRQ VFHQDULR RI WKH SURSRVHG GRQHWKURXJKSROOLQJWKHZHEVHUYHU$VORQJDVQRUHVXOWRIWKH
VFRULQJV\VWHP$IWHUWKHFXVWRPHUKDVILQLVKHGHDWLQJDQGSDLG IDFLDOH[SUHVVLRQGHWHFWLRQLVDYDLODEOHDKLQWLVGLVSOD\HGWKDW
KHFRXOGEHDVNHGWRUDWHWKHIRRGDVZHOODVWKHHQYLURQPHQW WKHUHFRJQLWLRQLVEHLQJZDLWHGIRU
RQWKHVDPHWDEOHWWKDWKDVDOUHDG\EHHQXVHGIRURUGHULQJWKH
IRRGDQGSD\PHQW ,Q RUGHU WR LQWHUFHSW SRWHQWLDO LQFRUUHFW GHWHFWLRQV RI WKH
IDFLDO H[SUHVVLRQ GHWHFWLRQ DOJRULWKP DQG WKXV WR JHW VWDWLVWLFV
7KHSURSRVHGUHVWDXUDQWVFRULQJV\VWHPLVFRPSRVHGRIDQ DERXW LWV DFFXUDF\ WKH FXVWRPHU DGGLWLRQDOO\ LV DVNHG WR UDWH
$QGURLGPRELOHDSSOLFDWLRQDZHEVHUYHUDQGDSUHWUDLQHG$, WKHIRRGPDQXDOO\LQWKLVH[SHULPHQWDOYHUVLRQ3UHVHQWO\WKHUH
VHUYHU 7KH PRELOH DSSOLFDWLRQ IRUPV WKH LQWHUIDFH WKURXJK DUH WKUHH GLIIHUHQW IDFLDO H[SUHVVLRQV VDWLVILHG - QHXWUDO .
ZKLFK WKH FXVWRPHU LQWHUDFWV ZLWK WKH VFRULQJ V\VWHP 7KH DQG GLVDSSRLQWHG / SURYLGHG IRU WKH UDWLQJ %\ FOLFNLQJ WKH
IDFLDO H[SUHVVLRQ UHFRJQLWLRQ LV SHUIRUPHG RQ WKH $, VHUYHU ³1H[W´EXWWRQWKHPDQXDOUDWLQJLVXSORDGHGWRWKHZHEVHUYHU
7KHPRELOHDSSOLFDWLRQDQGWKH$,VHUYHUGRQRWFRPPXQLFDWH DQG WKH FXVWRPHU LV UHGLUHFWHG WR WKH QH[W SDJH ZKHUH KH LV
GLUHFWO\ ZLWK HDFK RWKHU ,QVWHDG WKH HQWLUH GDWD WUDQVIHU DVNHGWRUDWHWKHHQYLURQPHQW7KHSURFHGXUHLVWKHVDPHDVIRU
EHWZHHQWKHVHWZRUXQVYLDDZHEVHUYHURQZKLFKWKHGDWDLV WKH IRRG UDWLQJ 7KH FRUUHVSRQGLQJ VFUHHQVKRWV DUH VKRZQ LQ
VWRUHGLQDGDWDEDVH )LJFDQG)LJG
,Q )LJ  D IORZFKDUW LV VKRZQ ZKLFK UHSUHVHQWV WKH
H[HFXWLRQ VWHSV RI WKH VFRULQJ V\VWHP ,W LV GLYLGHG LQWR WKUHH B. Web Server
DFWLYLWLHV HDFK UHSUHVHQWLQJ WKH IORZ RI RQH RI WKH WKUHH $ ZHE VHUYHU ZDV FKRVHQ RYHU D GLUHFW FRPPXQLFDWLRQ
FRPSRQHQWVPRELOHDSSOLFDWLRQZHEVHUYHUDQG$,VHUYHURI EHWZHHQ WKH PRELOH DSSOLFDWLRQ DQG WKH $, VHUYHU DV WKLV
WKHVFRULQJV\VWHP)LJGHSLFWVWKHFRUUHVSRQGLQJVHTXHQFH PDNHVLWHDVLHUWRUHSODFHDQGDGDSWLQGLYLGXDOFRPSRQHQWVRI
GLDJUDP ,W LOOXVWUDWHV WKH FRPPXQLFDWLRQ VHTXHQFHV EHWZHHQ WKHV\VWHP3+3DQG0\64/DUHXVHGRQWKHZHEVHUYHUVLGH
WKHV\VWHPFRPSRQHQWVLQFOXGLQJWKHFXVWRPHU7KHGHVLJQRI $Q$3,ZDVZULWWHQLQ3+3VFULSWLQJODQJXDJHZKLFKVHUYHVDV
DOO WKUHH FRPSRQHQWV DV ZHOO DV WKHLU H[HFXWLRQ VWHSV DUH DQLQWHUIDFHWRERWKWKHPRELOHDSSOLFDWLRQDQGWKH$,VHUYHU,W
GHVFULEHGEHORZ PDQDJHV WKH UHDGLQJ DQG ZULWLQJ RI GDWD IURP DQG WR WKH
0\64/ GDWDEDVH ,Q WKH GDWDEDVH WDEOH WKH LPDJH 85/ DQG
A. Android mobile application QDPH WKH FXVWRPHU DQG WKH $, VHUYHU UDWLQJ WKH FXVWRPHU¶V
)LJVKRZVVFUHHQVKRWVRIWKHPRELOHDSSOLFDWLRQ2QWKH HVWLPDWHG DJH DQG JHQGHU DV ZHOO DV WKH WLPH DQG GDWH RI WKH
ILUVWLQ)LJXUHDWKHFXVWRPHULVDVNHGWRWDNHDSLFWXUHRIKLV UDWLQJVDUHVWRUHG
IDFH LQ ZKLFK WKH H[SUHVVLRQ VKRXOG UDWH WKH IRRG :KHQ WKH
SLFWXUH LV DFFHSWHGE\ WKH FXVWRPHULW LV XSORDGHG WR WKH ZHE C. AI Server
VHUYHU7KHDSSGLUHFWO\GLVSOD\VWKHQH[WSDJHZKLFKLVVKRZQ 7KH FXVWRPHUV UDWLQJ LPDJH LV GRZQORDGHG IURP WKH ZHE
LQ ILJXUH E 2Q WKLV SDJH \RX FDQ VHH WKH FXVWRPHU
V UDWLQJ VHUYHUDVVRRQDVLWLVDYDLODEOH7KLVLVGRQHE\SROOLQJWKHZHE
LPDJH XQGHU ZKLFK WKH UHVXOW RI WKH $, VHUYHU
V IDFLDO VHUYHU ,Q RUGHU WR DFKLHYHWKH IDFLDO H[SUHVVLRQ GHWHFWLRQZH
H[SUHVVLRQUHFRJQLWLRQLVGLVSOD\HG7KLVUHVXOWLVGRZQORDGHG FRPELQHWKH)DFH1HW>@ZLWKWKHPRELOH1HW>@7KH)DFH1HW


LV D OD\HU &11EDVHG DSSURDFK ZKLFK FDQ GHWHFW IDFH ,QDQH[WVWHSWKHV\VWHPFRXOGEHFRPELQHGZLWKH[LVWLQJ
ORFDWLRQVRQDQLPDJH$IWHUWKDWWKHPRELOH1HWLVXVHGLQWKH WH[WEDVHGUDWLQJSODWIRUPVOLNH*RRJOHUDWLQJWREULQJWRJHWKHU
VDPHZD\DVLQ>@IRUWKHIDFLDOH[SUHVVLRQUHFRJQLWLRQ WKH DGYDQWDJHV RI ERWK V\VWHPV $ IXUWKHU GHYHORSPHQW FRXOG
OHDG WR D V\VWHP ZKHUH WKHFXVWRPHU FDQUDWH WRXFKOHVV LQ WKH
,W LV D OLJKW ZHLJKW GHHS QHXUDO QHWZRUN IRU PRELOH DQG UHVWDXUDQW)RUWKLVLWPXVWEHHQVXUHGWKDWWKHDFFXUDF\RIWKH
HPEHGGHG YLVLRQ DSSOLFDWLRQV ,W ZDV FKRVHQ EHFDXVH LW FDQ IDFLDOH[SUHVVLRQUHFRJQLWLRQLVKLJKHQRXJK,WLVDOVRDQLGHD
DOVR DFKLHYH IDVW DQG SUHFLVH UHVXOWV RQ ORZFRVW KDUGZDUH WR H[WHQG WKH LPDJHEDVHG UDWLQJ V\VWHP ZLWK D VSHHFK
V\VWHPVVXFKDVWKHXVHG1YLGLD-HWVRQ7;,WLVDEOHWRGHWHFW UHFRJQLWLRQ IHDWXUH 7KH FXVWRPHU FRXOG H[SUHVV KLV RSLQLRQ
VHYHQGLIIHUHQWIDFLDOH[SUHVVLRQV DQGLPSUHVVLRQVYHUEDOO\RUPDNHVXJJHVWLRQVIRULPSURYHPHQW
)RUWKLVXVHFDVHLWLVPRGLILHGVRWKDWWKHVHH[SUHVVLRQVDUH OLNHLWLVDOUHDG\GRQHZLWK*RRJOHUDWLQJV
PDSSHG WR WKH WKUHH H[SUHVVLRQV XVHG LQ RXU VFRULQJ V\VWHP ,W LV SODQQHG WR H[WHQG WKH V\VWHP ZLWK D ZHE DSSOLFDWLRQ
7KH QHJDWLYH H[SUHVVLRQV VDG IHDU GLVJXVW DQG DQJHU DUH WKDW ZLOO HQDEOH WKH UHVWDXUDQW PDQDJHPHQW WR JHW D TXLFN
PDSSHG WR GLVDSSRLQWHG KDSS\ DQG VXUSULVHG WR VDWLVILHG DQG JUDSKLFDORYHUYLHZDQGHDV\LQVLJKWLQWRWKHVWDWLVWLFV7KHDLP
QHXWUDO UHPDLQV QHXWUDO $V VRRQ DV D UHVXOW RI WKH IDFLDO LV WR LQWHJUDWH WKH SURSRVHG UHVWDXUDQW UDWLQJ V\VWHP LQWR
H[SUHVVLRQ UHFRJQLWLRQ LV DYDLODEOH LW LV XSORDGHG WR WKH ZHE H[LVWLQJXQPDQQHGUHVWDXUDQWV
VHUYHU
7KH $, VHUYHU UXQV RQ DQ 1YLGLD -HWVRQ 7; 7KLV LV DQ 5()(5(1&(6
HPEHGGHG $,FRPSXWLQJ GHYLFH ZLWK VPDOO IRUP IDFWRU DQG
>@ +DVVDLQ 6DHHG $OL 6KRXPDQ 0DLV (OIDU 0RVWDID 6KDEND 6KLNKDUHVK
ORZ HQHUJ\ FRQVXPSWLRQ ,W LV RSWLPL]HG IRU $, DSSOLFDWLRQV 0DMXPGDUDQG&KXQJ+RUQJ/XQJ³1HDUILHOGFRPPXQLFDWLRQVHQVRUV
DQGSRZHUHIILFLHQW DQGFORXGEDVHGVPDUWUHVWDXUDQWPDQDJHPHQWV\VWHP´LQProceedings
of the 2016 IEEE 3rd World Forum on Internet of Things (WF-IoT)SS

,,, &21&/86,21
>@ )ORULDQ 6FKURII 'PLWU\ .DOHQLFKHQNR DQG -DPHV 3KLOELQ ³)DFH1HW D
,QWKLVSDSHUDUHVWDXUDQWVFRULQJV\VWHPZKLFKLVEDVHGRQ XQLILHG HPEHGGLQJ IRU IDFH UHFRJQLWLRQ DQG FOXVWHULQJ´ Q Proceedings
IDFLDOH[SUHVVLRQUHFRJQLWLRQLVSURSRVHG,WLVSRVVLEOHWRJHWD of the 2015 IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)SS
ZLGHU UDQJH RI FXVWRPHU RSLQLRQV FRPSDUHG WR LQGHSHQGHQW
VFRULQJSODWIRUPVE\PDNLQJDGLUHFWUHTXHVWDWWKHHQGRIWKH >@ $QGUHZ * +RZDUG 0HQJORQJ =KX %R &KHQ 'PLWU\ .DOHQLFKHQNR
:HLMXQ :DQJ 7RELDV :H\DQG 0DUFR $QGUHHWWR DQG +DUWZLJ $GDP
YLVLW WR WKH FXVWRPHU %XW WKHUH LV RQO\ D URXJK LQVLJKW VLQFH ³0RELOH1HWV HIILFLHQWFRQYROXWLRQDOQHXUDOQHWZRUNVIRUPRELOH YLVLRQ
RQO\ WZR UDWLQJV DUH UHTXHVWHG %HFDXVH IDFLDO H[SUHVVLRQ DSSOLFDWLRQV´DU;LYSUHSULQWDU;LY
UHFRJQLWLRQ LV D PRGHUQ WHFKQRORJ\ WKDW LV XVHG LQ D SOD\IXO >@ -DQQH7RPPROD3HGUDP*KD]L%LVKZR$GKLNDUDDQG+HLNNL+XWWXQHQ
VHWWLQJ IRU WKH VFRULQJ V\VWHP WKH LQWHUHVW RI WKH FXVWRPHU WR ³5HDO WLPH V\VWHP IRU IDFLDO DQDO\VLV´ LQ Proceedings of the 7th
JLYHDUDWLQJLVDURXVHG European Workshop on Visual Information Processing (EUVIP’18)


D E F G

)LJ   6FUHHQVKRWV RI WKH $QGURLG PRELOH DSSOLFDWLRQ D 5HTXHVW WR UDWH WKH IRRG E 0RRG UHFRJQLWLRQ DQG UDWLQJ RI WKH IRRG F 5HTXHVW WR UDWH WKH
HQYLURQPHQWG0RRGUHFRJQLWLRQDQGUDWLQJRIWKHHQYLURQPHQW


)LJ)ORZFKDUWRIWKHUHVWDXUDQWVFRULQJV\VWHP

)LJ6HTXHQFHGLDJUDPIRUWKHUHVWDXUDQWVFRULQJV\VWHP