Volume 8, Issue 12, December 2023 International Journal of Innovative Science and Research Technology

ISSN No:-2456-2165

User Recommendation System on Text Based Images

Abishek Karthik; Adarsh V.
RV College of Engineering, Karnataka, India

Abstract:- With the growth of social media, online and saves the data like size, object analysis and OCR in the
shopping, and e-commerce, images have become an database.
integral part of our daily lives. The keyword search
system for text-based images is an important project in D. "Knowledge Graph-Based Image Classification”
today's world because of the vast amount of digital Authors: DEHAI ZHANG MENGLONG CUI YUN:
images that are generated and shared every day. This paper talks about the biologically inspired ideas that
However, finding a specific image from this vast are important in image processing. The usage of human
collection can be a daunting task, especially when we brain reasoning mechanism to present the image knowledge
have to search through thousands or even millions of graph(IKG) as a biological vision mechanism to improve the
images. So, Our System will help users to easily fetch the performance of the image classification.
images based on the Keyword Search. A GNN is used for
user recommendation. E. "Semantic-Based Search Engine System for Graph
Images in Academic Literatures by Use of Semantic
Keywords:- Keyword Search, Text-based images. Relationships"
Authors: Sarunya Kanjanawattana and Masaomi
I. INTRODUCTION Kimura: This paper talks about information retrieval which
is a baseline of search engine systems. There is a very large
With the growth of social media, online shopping, and amount of data published on the Internet that cannot be
e-commerce, images have become an integral part of our manually searched. However, search engine systems should
daily lives. The keyword search system for text-based not only present relevant results but also obtain new
images is an important project in today's world because of knowledge from the user's searches. The main idea of this
the vast amount of digital images that are generated and research was to propose methods for extracting information
shared every day. However, finding a specific image from from graphical and linguistic representations as well as
this vast collection can be a daunting task, especially when utilise them to express explicit and implicit knowledge.
we have to search through thousands or even millions of
images.So, our system will help users to easily fetch the F. A User-Centred Approach for Information Retrieval"
images based on the Keyword Search. A recommendation is Authors: Antonio Picariello Antonio M. Rinaldi: This
also made to the user based on the previous searches they paper talks about Information retrievals which can take great
have made. advantages and improvements considering users’ feedback.
Therefore, the user dimension is a relevant component that
II. LITERATURE SURVEY must be taken into account while planning and
implementing real information retrieval systems. In
A. "Content-based Image Retrieval using Tesseract OCR particular, combining the Semantic information from a
Engine and Levenshtein Algorithm" general knowledge base with statistical information using
Authors: Charles Adjetey Kofi Sarpong Adu-Manu relevance feedback. Several experiments and results are
(2021): This paper provides a technique for obtaining a full presented using a test set constituted of Web pages.
image document given that the user has some portions of the
document under search. A combination of Optical Character G. "An Empirical Study of Important Keyword Extraction
Recognition (OCR) engine and an improved text matching Techniques from Documents"
algorithm was used in the system implementation. The Authors: H. M. Mahedi Hasan, Falguni Sanyal,
Tesseract OCR engine and Levenshtein Algorithm were Dipankar Chaki: This paper explores keyword extraction
integrated to perform the image search. that compiles a list of phrases that give a general overview
of the material.Keyword gives information regarding
B. "A New Content-Based Image Retrieval Method on the identifying a specific document. Keyword extraction will be
Google Cloud Vision API" the main strategy, to analyse a large number of documents
Authors: Shih-Hsin Chena Yi-Hui Chen (2017): This and extract the pertinent information. This strategy will
paper investigates the effectiveness of Google Cloud Vision enable us to comprehend its depth even before we read it.
API compared with some efficient Machine learning Two or more strategies used- statistical, machine-learning,
algorithms in the literature. semantic data.
C. "Smart Gallery using Google Vision" H. "Explainable Graph-based Search for Lessons-Learned
Authors: Shalva Thakurdesai Shubham Vira Gouri Documents in the Semiconductor Industry"
Kanitkar Dr. Jagruti Save (2021): This paper comprises a Authors: Hasan Abu-Rasheed , Christian Weber,
system which helps intelligent gallery management. For
Johannes Zenkert , Roland Krumm, Madjid Fathi (July
this, a server, mobile interface and web interface has been 2021):This paper focuses on knowledge graphs.Industrial
made. The mobile interface uploads photos to the server and processes produce a considerable volume of data and thus
ensures privacy of photos. The server processes the photos

IJISRT23DEC1664 www.ijisrt.com 2264

Volume 8, Issue 12, December 2023 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
information. Whether it is structured sensory data or semi- L. "A Comprehensive Survey on Graph Neural Networks"
to unstructured textual data, the knowledge that can be Authors: Zonghan Wu; Shirui Pan; Fengwen Chen;
derived from it is critical to the sustainable development of Guodong Long; Chengqi Zhang; Philip S. Yu: This
the industrial process. A search engine is developed and paper explores the concept of deep learning has
applied to answer queries.In contrast to mere keyword-based revolutionised many machine learning tasks in recent
searching, the searchability of the knowledge graph offers years, ranging from image classification and video
enhanced search results beyond direct matches and acts as a
processing to speech recognition and natural language
mean for generating explainable results and result
recommendations understanding. The data in these tasks are typically
represented in the Euclidean space. However, there is
I. "Deep Learning on Knowledge Graph for Recommender also an increasing number of applications in GNN
System: A Survey" especially in data mining.
Authors: YANG GAO and YI-FAN LI: This paper
talks about recommender systems (RS) which are used to
help users make decisions about products or services. III. OBJECTIVE OF STUDY
Traditional RS are content-based or collaborative-filtering
based, but they are not effective for all applications.  The main purpose of the keyword search system for text-
Knowledge graphs (KGs) can encode high-order relations based images is to allow users to search for images
between objects, and graph neural networks (GNNs) can be based on the presence of specific keywords or phrases
used to extract information from KGs. GNNs have been within the text that appears in the image.
shown to be effective for a variety of RS tasks, such as item  By using the proposed keyword search system, users can
recommendation, user profiling, and cold-start easily locate images that contain specific text that is
recommendation. relevant to their needs, such as product names,
descriptive text or keywords associated with a particular
J. "Graph neural networks: A review of methods and theme or topic. This can help to streamline the image
applications" search process and make it easier to find the right image
Authors: Zhou Ganqu Cui Shengding Hu Zhengyan quickly and efficiently.
Zhang: This paper uses Graph neural networks (GNNs)  To implement a GNN to map images to which users
which are neural models that capture the dependence of would be interested in them.
graphs via message passing between the nodes of graphs. In  To calculate the F1 score and checking if the model’s
recent years, variants of GNNs such as graph convolutional score is above 0.8
network (GCN), graph attention network (GAT), graph
recurrent networks (GRN) have demonstrated ground- IV. METHODOLOGY
breaking performances on many deep learning tasks. In this
survey, we propose a general design pipeline for GNN  Data Collection: Collect data on real time images and
models and discuss the variants of each component, store them in a database or a file system.
systematically categorise the applications, and propose four  Data Preprocessing: Resize the images, convert them to
open problems for future research a format that the GNN can understand, and remove any
noise from the images. Implement using a variety of
K. "The Graph Neural Network Model" technologies, such as Python libraries or machine
Authors: Franco Scarselli, Marco Gori, Ah Chung learning frameworks.
Tsoi, Markus Hagenbuchner, Gabriele Monfardini: This  Inference: Select the relevant features from the data that
paper talks about the many underlying relationships among are most likely to impact the recommendation system.
data in several areas of science and engineering, e.g., The inference layer can be implemented using a variety
computer vision, molecular chemistry, molecular biology, of technologies, such as Python libraries or machine
pattern recognition, and data mining, which can be learning frameworks.
represented in terms of graphs. In this paper, we propose a  Model Selection: Choose an appropriate neural network
new neural network model, called graph neural network model for user recommendation based on the selected
(GNN) model, that extends existing neural network methods features. This can include GNN algorithms.
for processing the data represented in graph domains. This  Model Training: Train the machine learning model
GNN model, which can directly process most of the using real-time data using TensorFlow or PyTorch.
practically useful types of graphs, e.g., acyclic, cyclic,  Model Evaluation: Evaluate the performance of the
directed, and undirected, implements a function tau(G,n) is model using metrics such as precision, recall and F1
in IR m that maps a graph G and one of its nodes n into an m score.
-dimensional Euclidean space.
 Deployment: Deploy the trained model into a web
application or mobile app, where users can input their
details and receive a recommendation.
 Continuous Learning: Continuously update the model
with new data to improve its accuracy over time.

IJISRT23DEC1664 www.ijisrt.com 2265

Volume 8, Issue 12, December 2023 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165

Fig. 1: Methodology

V. IMPLEMENTATION

The user enters the values of the different parameters necessary for inputting of data and required Keyword Search.

Fig. 2: Home Page

Fig. 3: Neo4j Cloud Console

IJISRT23DEC1664 www.ijisrt.com 2266

Volume 8, Issue 12, December 2023 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165

Fig. 4. Search Page

 Linkword nodes enable users to establish connections recommendation system incorporates feedback and
between nodes that might not otherwise be connected, continuously learns from user interactions. By collecting
allowing for the creation of complex networks of user feedback on the recommendations and iteratively
relationships in a graph database. refining the algorithms and models, the system improves
 Linkword nodes can represent any concept or entity that over time and provides increasingly accurate and relevant
has a relationship with other nodes in the database, recommendations.
including chemical elements, scientific concepts, or any
other type of data. B. Conclusions:
 By using Linkword nodes to connect nodes that might In conclusion, the system for text-based images can
not otherwise be connected, users can perform more enhance user engagement and satisfaction by providing
comprehensive analysis of the relationships between accurate and relevant recommendations based on keyword
different entities in their graph database. search verified by 0.9 F1 score. By analysing the content of
images, extracting keywords, and leveraging Graph Neural
 Linkword nodes can be created with various properties,
Networks (GNNs) and other algorithms, the system
such as a unique ID or a type label, which can help to
streamline querying and data analysis. generates personalised recommendations and helps users
discover new content. The system's ability to deliver
 The use of Linkword nodes in a graph database can help
accurate recommendations, increase user engagement, and
to reveal previously unseen relationships between
continuously improve through user feedback contributes to a
different entities, providing valuable insights for
positive user experience. Regular monitoring and iteration
scientific research or other types of data analysis
are crucial to ensure the system remains aligned with user
VI. ANALYSIS preferences and expectations, providing ongoing value to
both users and the platform
After experimenting with different GNN algorithms
and evaluating their performance on relevant metrics, such VIII. FUTURE ENHANCEMENTS
as recommendation accuracy and user satisfaction, the most
suitable algorithm is used.  The keyword search system for text-based images
utilises Tesseract OCR technology to extract text from
VII. RESULTS AND CONCLUSION images and create a knowledge graph based on extracted
keywords.
A. Results:  While the system has shown promising results, there is
The recommendation system provides accurate and room for improvement in terms of accuracy, supporting
relevant recommendations to users. When the user performs multiple languages, enhancing the user interface, and
a search using keywords related to text-based images, the supporting more image formats.
system analyses the keywords, compares them with the  Improving accuracy is a crucial area for future work,
indexed data, and generates recommendations that align which could be done by exploring new OCR
with the user's interests. The quality of the recommendations technologies or developing better image pre processing
evaluated shows an F1 score of 0.9.The recommendation techniques.
system enhances user engagement by suggesting images that  Supporting multiple languages is essential for making
match the user's interests. The user recommendation system the system more globally accessible, and enhancing the
based on keyword search offers personalised user interface can improve the user experience.
recommendations tailored to individual user preferences. By  Supporting more image formats can make the system
analysing the keywords and understanding user behaviour, more versatile, and continued efforts to improve the
the system learns and adapts to each user's interests, system can make it an invaluable tool for various
providing a personalised experience. Additionally, the industries.
system also helps users discover new and relevant images or
content they might not have encountered otherwise. The

IJISRT23DEC1664 www.ijisrt.com 2267

Volume 8, Issue 12, December 2023 International Journal of Innovative Science and Research Technology
ISSN No:-2456-2165
ACKNOWLEDGMENT [9]. B. Wang, X. Zhang, and N. Li, “Relevance feedback
technique for content-based image retrieval using
At the conclusion of this thrilling journey, We would neural network learning,” in 2006 International
like to sincerely thank a few important persons who made it Conference on Machine Learning and Cybernetics.
possible. We would first like to express our gratitude to Dr. IEEE, 2006, pp. 3692–3696.
K N Subramanya, the principal of RV College of [10]. A. Rashno and E. Rashno, “Content-based image
Engineering, for giving us the encouragement and retrieval system with most relevant features among
opportunities to further our studies and follow our passion wavelet and colour features,” arXiv preprint
for computer science. Your unshakable confidence in our arXiv:1902.02059, 2019.
skills has been a constant source of encouragement. [11]. R. R. Saritha, V. Paul, and P. G. Kumar, “Content
based image retrieval using deep learning process,”
We also want to express my gratitude to Dr. Cluster Computing, vol. 22, no. 2, pp. 4187–4200,
Ramakanth Kumar P, the head of the computer science 2019.
department, who has provided us with advice and assistance [12]. K. Zagoris, S. A. Chatzichristofis, N. Papamarkos, and
throughout our academic career. Our development into the Y. S. Boutalis, “img (anaktisi): A web content based
person We are now would not have been possible without image retrieval system,” in 2009 Second International
your unfailing encouragement and support. Workshop on Similarity Search and Applications.
Our deepest gratitude also extends to Prof. Shanta R, IEEE, 2009, pp. 154–155.
our professor, for her crucial contributions to this effort. [13]. K.-M. Wong, K.-W. Cheung, and L.-M. Po, “Mirror:
This event has been made absolutely remarkable by your an interactive content based image retrieval system,” in
professional expertise and knowledge as well as your 2005 IEEE International Symposium on Circuits and
enthusiasm for education. Systems. IEEE, 2005, pp. 1541–1544.
[14]. F. Karamzadeh and M. A. Azgomi, “An automated
Finally, We would like to provide a sincere thank you system for search and retrieval of trademarks,” in
to all of the wonderful people who have contributed to the Proceedings of the 11th International Conference on
impact and memorability of this voyage. We count Electronic Commerce, 2009, pp. 374–377.
ourselves extremely fortunate to have had their support and [15]. B. Girod, V. Chandrasekhar, R. Grzeszczuk, and Y. A.
direction, and We aim to continue doing them proud in all Reznik, “Mobile visual search: Architectures,
that We do in the future. technologies, and the emerging.

REFERENCES

[1]. K. Pustu-Iren, G. Bruns, and R. Ewerth, “A

multimodal approach for semantic patent image
retrieval,” 2021.
[2]. P. Ghadekar, S. Kaneri, A. Undre, and A. Jagtap,
“Digital image retrieval based on selective conceptual
based features for important documents,” in
Evolutionary Computing and Mobile Sustainable
Networks. Springer, 2021, pp. 569–579.
[3]. K. M. Lakshmi et al., “An efficient telugu word image
retrieval system using deep cluster,” Turkish Journal of
Computer and Mathematics Education (TURCOMAT),
vol. 12, no. 11, pp. 3247–3255, 2021.
[4]. L. Yang, M. Gong, and V. K. Asari, “Diagram image
retrieval and analysis: Challenges and opportunities,”
in Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition Workshops,
2020, pp. 180–181.
[5]. W. Zhou, H. Li, and Q. Tian, “Recent advance in
content-based image retrieval: A literature survey,”
arXiv preprint arXiv:1706.06064, 2017.
[6]. A. Jain, “The role and importance of search engine and
search engine optimization,” International Journal of
emerging trends & technology in computer science,
vol. 2, no. 3, pp. 99–102, 2013.
[7]. N. Hochst ¨ otter and D. Lewandowski, “What users
see–structures in ¨ search engine results pages,”
Information Sciences, vol. 179, no. 12, pp. 1796–1812,
2009.
[8]. A. V. Singh, “Content-based image retrieval using
deep learning,” 2015.

IJISRT23DEC1664 www.ijisrt.com 2268