MATEC Web of Conferences 255, 01003 (2019) https://doi.org/10.

1051/matecconf/201925501003
EAAI Conference 2018

Automatic Image Annotation for Small and Ad hoc Intelligent

Applications using Raspberry Pi
Syed Muslim Jameel1,*, Manzoor Ahmed Hashmani1, Syed Sajjad Hussain Rizvi 3, Vali Uddin 3, and Mobashar Rehman 2
1 Department of Computer and Information Sciences, University Technology Petronas, Malaysia
2 Department of Computer Science, University Tunku Abdul Rahman, Malaysia
3 Faculty of Engineering Sciences and Technology, Hamdard University, Pakistan

Abstract. The cutting-edge technology Machine Learning (ML) is successfully applied for Business
Intelligence. Among the various pre-processing steps of ML, Automatic Image Annotation (also known as
automatic image tagging or linguistic indexing) is the process in which a computer system automatically
assigns metadata in the form of captioning or keywords to a digital image. Automatic Image Annotation
(AIA) methods (which have appeared during the last several years) make a large use of many ML
approaches. Clustering and classification methods are most frequently applied to annotate images. In
addition, these proposed solutions require a high computational infrastructure. However, certain real-time
applications (small and ad-hoc intelligent applications) for example, autonomous small robots, gadgets,
drone etc. have limited computational processing capacity. These small and ad-hoc applications demand a
more dynamic and portable way to automatically annotate data and then perform ML tasks (Classification,
clustering etc.) in real time using limited computational power and hardware resources. Through a
comprehensive literature study we found that most image pre-processing algorithms and ML tasks are
computationally intensive, and it can be challenging to run them on an embedded platform with acceptable
frame rates. However, Raspberry Pi is sufficient for AIA and ML tasks that are relevant to small and ad-hoc
intelligent applications. In addition, few critical intelligent applications (which require high computational
resources, for example, Deep Learning using huge dataset) are only feasible to run on more powerful
hardware resources. In this study, we present the framework of “Automatic Image Annotation for Small and
Ad-hoc Intelligent Application using Raspberry Pi” and propose the low-cost infrastructures (single node
and multi node using Raspberry Pi) and software module (for Raspberry Pi) to perform AIA and ML tasks
in real time for small and ad-hoc intelligent applications. The integration of both AIA and ML tasks in a
single software module (with in Raspberry Pi) is challenging. This study will helpful towards the
improvement in various practical applications areas relevant to small intelligent autonomous systems.

Keywords: Automatic Image Annotation, Real-Time Machine Learning, Big Data Annotation,
Small and Ad-hoc Intelligent Application.

Abbreviations: Automatic Image Annotation (AIA), Machine Learning (ML), Big Data Annotation
(BDA), Small and Ad-hoc Intelligent Application (SAIA).

1 Introduction Image Annotation techniques can be categorized into

three (3) different approaches.
1.1 Related Work 1. Text-based approach
The state-of-art Machine Learning (ML) algorithms are 2. Content-Based Image Retrieval (CBIR)
contributing to several sectors of human life through Big approach
Data [01]. However, explosive growth of Big Data 3. Automatic Image Annotation (AIA)
generates several types of Big Data, in which visual data
(images) is much informative and complex to perform Text-based approach manually annotates (through
analytics using ML algorithms. Image data requires more human) the images and hence inappropriate in the
systematic pre-processing steps (Image Annotation) for current digitization scenario [3-4]. Content-Based Image
analytics then non-imaging data [02]. In the literature, Retrieval approach automatically retrieves and index

*
Corresponding author: [email protected]

© The Authors, published by EDP Sciences. This is an open access article distributed under the terms of the Creative Commons Attribution License 4.0
(http://creativecommons.org/licenses/by/4.0/).
MATEC Web of Conferences 255, 01003 (2019) https://doi.org/10.1051/matecconf/201925501003
EAAI Conference 2018

different low-level features (colour, shape and texture) 2 AIA for small and ad-hoc intelligent
[5-6]. The need for large-scale image dataset annotation applications.
introduced the concept of Automatically Image
Annotation (AIA) [7-10]. The AIA technique contains
the good characteristics (advantages) from both 2.1 Research Methodology
traditional (text based and CBIR) annotated techniques
through the keyword searching based on image content. In this research study, our aim is to propose a novel
In AIA, the semantic concept model automatically learns approach for small and ad-hoc based intelligent system
from the large number of visual data. Several studies architecture through cost-effective and less
already discussed AIA advantages over traditional computational resources. However, our main concern is
approaches [11-14]. . Few recent surveys, also discussed to develop the Automatic Image Annotation Systems
the cutting edge Deep Learning approaches for AIA [15- which will capture the real-time images and
16]. The issue of AIA is it requires a massive automatically annotate them for ML analytics. To
(expensive) infrastructure to annotate the large-scale achieve our required goal, we followed below objectives;
images in real time because these approaches mainly use 1. A comprehensive literature review of existing
high computational systems Therefore, in this study, we AIA approaches.
proposed a cost-effective AIA approach to generate and 2. Propose the complete AIA framework for real-
annotate the massive amount of dataset. time image capturing, pre-processing
(annotation) and analytics.
3. To design, implement and validate the proposed
approach.

Along with a systematic literature review (to achieve the

first objective). This study will follow Design Science
(DS) research process to achieve the goal. Design
Science is being considered as the most efficient process
for the development of successful models or artifacts.
Proposal and evaluation of model can include a variety
of algorithms, techniques, methods, tools or framework,
which could create various hindrances in terms of
strategy, creation, and control which could be avoided by
adopting Design Science process.
Fig. 1. Typical centralized approach for real-time Automatic
Image Annotation.

1.2 Research Motivation

The current Machine Learning(ML) architecture is

centralize in nature for real-time AIA and ML task.
Typically, centralized server is responsible to execute all
AIA and ML tasks and forward response (output) to
point of action. This approach is quite optimal when Fig. 2. Proposed single node approach for real-time Automatic
using a moderate or large type of Intelligent Systems, as Image Annotation (AIA).
shown in Fig. 1. Beside high cost this approach also
upfront security-related issues too [17]. On contrary, for
the small and ad-hoc intelligent systems, we need quick
and on spot, data pre-processing to perform intelligent
tasks. Therefore, there is need of automatically annotate
data (images) on spot (for creating a database) and
utilize them for quick prediction or classification. In this
study, we present the idea of real time AIA for ML tasks
using Raspberry Pi. We propose single node approach
for representing the small autonomous intelligent
systems and multiple nodes integration approach towards
decentralized distributive small and ad-hoc intelligent
systems (which could behave as a single network) to
perform some high computational processing jobs
through object migration, as shown in Fig. 4.
Fig. 3. Software module for real-time AIA and ML tasks.

2
MATEC Web of Conferences 255, 01003 (2019) https://doi.org/10.1051/matecconf/201925501003
EAAI Conference 2018

2.2 How it works? substantially reduces the cost (through low-cost devices)
and distributes the computational processing (through
Fig. 02 shows the single node approach, using a small multiple nodes) among various small units (single
camera (for capturing real-time images of the natural nodes). Interestingly, the proposed small ad-hoc
environment), attached with a small Raspberry Pi intelligent systems contain the capability to run the
(hardware). This cheap and efficient hardware machine learning algorithms to perform automatic image
infrastructure can capture and annotate the real-time annotation and classification. Certainly, the further
images automatically through the machine learning advancement in these individual small and ad-hoc
algorithm. For that purpose, we design the software intelligent systems towards the sharing of computational
module (for each raspberry pie) as shown in Fig. 3. This resources (through migration of objects) can make it
software is responsible for performing basic and feasible on the situations where large-scale
advanced level annotation task using a machine learning infrastructures are necessary. In this paper, we present
algorithm (3 channel pixels to matrix conversion, object the idea and propose the framework for Automatic
identification, and object labeling). Fundamentally, this Image Annotation for small and ad-hoc intelligent
software is based on two major components; application using Raspberry Pi. In the future, we will
implement the proposed approach to validate this
1. Automatic Image Annotation Module approach.
2. Image Analytic Module In the implementation phase, we will design the
necessary tools and applications (for Raspberry Pi) using
The Automatic Image Annotation module is capable to MATLAB (Machine Learning toolbox). For evaluation
receive the raw image data (real-time capturing through purpose, we will first implement the proposed solution in
the camera) and automatically convert it into an a single Raspberry Pi based intelligent node as shown in
annotated format (by tagging and indexing). Annotated Fig. 2. (For validating the autonomous behavior of a
image transfers to the image repository. Image repository single node). Later we will setup the Raspberry Pi
keeps track of all the indexed and tagging images with intelligent network with multiple intelligent nodes, as
their respective labels to train Image Analytic Module, shown in Fig. 4. (To validate the combined behavior of
as shown in Fig. 3. multiple nodes).
The single node performs some less computational
processing tasks due to its hardware limitations. Acknowledgment
However, multiple single nodes may share their
computing resources and form a decentralized
This research study is a part of Big Data collection and
distributed network environment to perform some high
annotation phase of the funded project under matching
computational ML tasks, as shown in Fig. 4. These
grant scheme supported by University Technology
architectures will be less expensive, fast and feasible for
Petronas (UTP), Malaysia and Hamdard University,
on spot decision making.
Pakistan.

References
1. SM. Jameel, MA. Hashmani, H. Alhussain, A.
Budiman. A Fully Adaptive Image Classification
Approach for Industrial Revolution 4.0. IRICT
2018. pp. 311-321. 2018.
2. Li, Jia, and Z. James. Real-time computerized
annotation of pictures. Proceedings of the 14th
ACM international conference on Multimedia.
ACM, 2006.
3. Tamura, Hideyuki, and Y. Naokazu. Image database
systems: A survey. Pattern recognition 17.1. pp. 29-
43 (1984).
4. Chang, S-K., and H. Arding. Image information
Fig. 4. Multiple nodes integration to perform high
systems: where do we go from here?. IEEE
computational Automatic Image Annotation tasks. transactions on Knowledge and Data Engineering
4.5, pp. 431-442 (1992).
5. N. Rasiwasia, N. Vasconcelos, P.J. Moreno. Query
3 Conclusion and Future Work by semantic example. Proceedings of the Fifth
Instant preparation (annotation) of image dataset from International Conference on Image and Video
Retrieval, 4071, pp. 51–60 (2006).
the real-time data collection is critical and cost-effective
in typical AIA (centralized) approach. Whereas the
decentralized approach possesses the potential to

3
MATEC Web of Conferences 255, 01003 (2019) https://doi.org/10.1051/matecconf/201925501003
EAAI Conference 2018

6. N. Vasconcelos. Minimum probability of error

image retrieval. IEEE Transactions on Signal
Processing 52 (8), pp. 2322–2336 (2004).
7. Uricchio, Tiberio. Automatic image annotation via
label transfer in the semantic space Pattern
Recognition 71, pp. 144-157(2017).
8. Zhang, Pengyu. Automatic Image Annotation Based
on Multi-Auxiliary Information. IEEE Access 5, pp.
18402-18411 (2017).
9. Pobar, Miran, and I Marina. Automatic image
annotation refinement. MIPRO, 2016 39th
International Convention on. IEEE (2016).
10. Hao, Zhangang, Ge. Hongwei, Gu.Tianpeng.
Automatic Image Annotation Based on Particle
Swarm Optimization and Support Vector Clustering.
Mathematical Problems in Engineering (2017).
11. Liu, Ying. A survey of content-based image
retrieval with high-level semantics." Pattern
recognition 40.1, pp.. 262-282 (2007).
12. Lew, S. Michael. Content-based multimedia
information retrieval: State of the art and challenges.
ACM Transactions on Multimedia Computing,
Communications, and Applications (TOMM) 2.1,
pp. 1-19 (2006).
13. Vasconcelos, Nuno. From pixels to semantic spaces:
Advances in content-based image retrieval.
Computer 40.7 (2007).
14. Datta, Ritendra. Image retrieval: Ideas, influences,
and trends of the new age. ACM Computing
Surveys (Csur) 40.2, pp. 5 (2008).
15. Cheng, Qimin. A survey and analysis on automatic
image annotation. Pattern Recognition 79, pp. 242-
259 (2018).
16. Srivastava, Gargi, R. Srivastava. A Survey on
Automatic Image Captioning. International
Conference on Mathematics and Computing.
Springer, Singapore. (2018).
17. MA. Hashmani, SM. Jameel. An Ensemble
Approach to Big Data Security (Cyber Security)
(IJACSA), 9.9(2018).
http://dx.doi.org/10.14569/IJACSA.2018.090910.