Hindusthan College of Engineering And Technology

Approved by AICTE, New Delhi, Accredited with ‘A’ Grade by

NAAC (An Autonomous Institution, Affiliated to Anna University,
Chennai) Coimbatore – 641 032

AICTE – RPS RESEARCH PROJECT

ANNUAL PROGRESS REPORT

ON

“Augmented Reality (AR) using Natural User Interface (NUI)

Based Physical Therapy Rehabilitation Framework with Mobile
Decision Support System (MDSS)”

Submitted to

ALL INDIA COUNCIL FOR TECHNICAL EDUCATION

Submitted By

Dr.S.Shankar
Professor & Head
Department of Computer Science & Engineering
Hindusthan College of Engineering and Technology
Coimbatore 641 032

April 2021
 TABLE OF CONTENTS
PAGE
S. NO. TITLE
NO
List of Tables iii
List of Figures iv
List of Abbreviations vi
1 Principal Investigator 1
2 Project Title 1
3 Date of Commencement of the Project 1
4 Amount Sanctioned by AICTE 1
5 Amount Released by AICTE 1
6 Details of Expenditure 2
7 Objectives of the Project 3
8 Salient Research Achievements 3
Abstract 3
8.1 Introduction 4
8.2 Motivation 5
8.3 Plan of Research 6
8.4 Summary of Progress 6
8.4.1 Release of Grant 6
8.4.2 PEC Formation 6
8.4.3 Purchase of Equipment 7
8.4.4 Literature Study 7
8.4.4.1 Stroke Survivors in India 7
8.4.4.2 Physical Therapy Rehabilitation
9
for Stroke Patients
8.4.4.3 Treatment Methods for Stroke
Patients Upper Limb 10
Rehabilitation
8.4.4.4 Gesture Recognition for
13
Upper Limb Stroke
 Rehabilitation
8.4.4.5 Augmented Reality for Upper
15
Limb Stroke Rehabilitation
8.4.4.6 Techniques of Natural User
Interface for Stroke 17
Rehabilitation
8.4.4.7 ADL Based Rehab
19
Gamification System
8.4.4.8 Mobile Decision Support
System for Stroke 20
Rehabilitation
8.4.5 Development Environment Setup 22
8.4.5.1 Microsoft © Kinect Sensor 22
8.4.5.2 Leap Motion Controller 24
8.4.6 Sample Collection 26
8.5 New Findings / Achievements 26
8.5.1 Research Gap 26
8.5.2 Proposed Framework 27
8.5.3 ADL Rehab Gamification System 29
8.5.4 Algorithm of ADL Rehab Game Play 30
8.5.5 Activities of Daily Living Rehab
Gamification System As An Intelligent 32
Agent
8.5.6 Implementation of the System 39
8.5.6.1 System Requirements 39
8.5.6.2 Unity 3D gaming engine 39
8.5.6.3 Exercise based Game Scenario
40
Development for ADL Training
8.5.7 Gesture Angle Capturing using Kinect 41
8.5.8 Hand Gesture Recognition using Leap
46
Motion Controller
8.5.9 Mobile Decision Support System 49
8.5.9.1 Proposed MDSS Architecture 49
8.5.9.2 Decision Network for Patient 54
i
 Performance and Improvement
Classification
8.5.9.3 Implementation of the System 59
8.6 Innovation Brought in 65
8.7 Potential for Application 65
9 List of Publications Arising from the Project 66
10 Patent(s) Filed/to be Filed 66
11 References 67

LIST OF TABLES

ii
TABLE NO. TITLE PAGE NO
8.1 Population survey of stroke incidence in India 4
Reviews of Physical Therapy Rehabilitation for
8.4.4.1 10
stroke patients
Reviews of Treatment methods for stroke patients
8.4.4.2 12
Upper Limb Rehabilitation
Reviews of Gesture Recognition for Upper Limb
8.4.4.3 14
Stroke Rehabilitation
Reviews of Augmented Reality for Upper Limb
8.4.4.4 16
Stroke Rehabilitation
Reviews of Techniques of Natural User Interface
8.4.4.5 18
for stroke rehabilitation
Reviews of ADL based Rehab Gamification
8.4.4.6 20
System
Reviews of Mobile Decision Support System for
8.4.4.7 22
stroke rehabilitation
8.4.6.1 Inclusion Criteria for stroke patient identification 26
8.5.5.1 PAGE explanation of Intelligent Agent of ADL
32
Rehab Game Play
8.5.5.2 Environment Properties of Intelligent Agent of
33
ADL Rehab Gamification
8.5.5.3 Sampling Technique Employed 35
8.5.5.4 Subjects Profile Summary (C and E Group) 36
8.5.7.1 Comparison of Normal RoM and Kinect RoM -
44
UL angle measurement
8.5.9.1 Patient Rehabilitation Database (PRD) Details 51
8.5.9.2 CPT for P (improvement/Training score) 56
8.5.9.3 CPT for P (improvement/Training frequency) 56
8.5.9.4 Utility Table (UT) of the Decision Network for
56
Patient Improvement
8.5.9.5 Expected Utility Table (EUT) with Threshold
57
Value for Improvement Classification
8.5.9.6 Improvement Classification of a Subject for 5
58
days Training Period

iii
 LIST OF FIGURES

PAGE
FIGURE NO. TITLE
NO
8.3.1 Yearwise Plan of Research 6
8.4.4.1 Stroke related deaths from 2010 to 2013 8
8.4.5.1 Microsoft @ Kinect V2 components 23
8.4.5.2 Leap Motion Controller 25
8.4.5.3 Leap Motion Controller Internal Structure 25
Proposed system of Augmented Reality-Natural
8.5.1 User Interface-Rehabilitation Mobile Decision 28
Support System
Activities of Daily Living Rehab Gamification
8.5.2 30
Environment
8.5.5.1 Experimental Clinical Setup 33
8.5.5.2 Methodology for Research Experiment 34
UI and Illustration of Reach and Touch
8.5.6.1 40
Scenario
UI and Illustration of Hold and Release
8.5.6.2 41
Scenario

8.5.6.3 UI and Illustration of Activity of Eating 41

Upper limb gesture data capturing and joint
8.5.7.1 42
angle measurement
Kinect - Skeletal joints for upper limb angle
8.5.7.2 42
measurement

8.5.7.3 (a) Skeletal joint captured using Kinect 43

8.5.7.3 (b) Elbow Angle Calculation using Kinect 43

Upper Limb RoM a) Shoulder Flexion b)
8.5.7.4 44
Elbow Flexion and Extension

8.5.8.1 Leap Motion Hand Skeleton 47

iv
8.5.8.2 Object Capture using Leap Motion Controller 49
Architecture of Mobile Decision Support
8.5.9.1 51
System

8.5.9.2 Decision Network for patient classification 54

MDSS user interface (a) Login (b) View
8.5.9.3.1 61
Patients
MDSS user interface (a) View Patients (b)
8.5.9.3.2 62
Personalize Treatment Plan

8.5.9.3.3 MDSS user interface for Performance Review 63

MDSS user interface for Instant Messaging
8.5.9.3.4 64
Service (IMS)

v
 LIST OF ABBREVIATIONS
ADL - Activities of Daily Living
AI - Artificial Intelligence
API - Application Program Interface
AR - Augmented Reality
ARAMIS - Automatic Recovery Arm
Motility Integrated System AR-NUI-REHAB-MDSS -
Augmented Reality-Natural User
Interface-
Rehabilitation-Mobile Decision Support System
DSS - Decision Support System
FMA - Fugl Meyer Assessment
GR - Gesture Recognition
IR - Infrared Radiation
ISB - International Biomedical Standards
LED - Light Emitting Diode
LMC - Leap Motion Controller
LMIC - Low and Middle Income Countries
MDSS - Mobile Decision Support System
MMSE - Mini Mental State Examination
MTBT - Mobile based Tablet Therapies
NUI - Natural User Interface
PEC - Project Evaluation Committee
PRD - Patient Rehabilitation Database
RGB - Red, Green, Blue
RGBD - Red, Green, Blue, Depth
SDK - Software Development Kit
SWA - Smart Wearable Armband
TAGER - Targeted Acquiring Exercise
TULIA - Test of Upper Limb Apraxia

vi
UL - Upper-Limb
VR - Virtual Reality
WHO - World Health Organisation

vi
i