Designing Text Summarization
Model with BERT
Presented by Under the Guidance of
Floride Tuyisenge(20BCP286) Dr. Hiren Thakkar
Abraham Wari (20BC282)
ABSTRACT
• Our project focuses on developing a text summarization tool using BERT for
extractive summarization technique.
• The extractive summarization will capture key ideas, scoring the sentences and
compose summary by selective sample sentences from given text.
• We utilize the combination of models and libraries for training and fine-tuning our
summarization model.
• Our text summarization tool can be applied in different domains such as
education, research centers, and social media platform.
• We intend to advance the accessibility and efficiency of the text summarization by
making important insights easier to find and use.
INTRODUCTION
• Summarization condenses different types of information
(text, audio, video) while capturing its main points, like a mini
version highlighting the essentials.
• Text summarization focuses on the written content while
Audio summarization focuses on the spoken content, and
Video summarization processes video content.
• BERT, a bidirectional transformer model, is designed to pre-
train unlabeled text and be fine-tuned to perform different
tasks.
• BERT uses Transformer to learn contextual relation between
words and is bidirectional model as it analyzes context of
words in both directions(left-to right and right to left
LITERATURE SURVEY
● There are two existing methods for text summarization task at present: abstractive and extractive.
( Wang et. al., 2019).
● (Yang et. al., 2021). To the best knowledge the first extractive and abstractive summarization systems reported are the first such systems for Hungarian. The task of summarization can be categorized into
two methods, extractive and abstractive.
● (Liu & Lapata, 2019) The concept of text summarization with pre-trained encoders, highlighting the
effectiveness of leveraging pre-trained language models for summarization tasks.
● Devlin et al., 2018). Propose BERT, a pre-training technique for deep bidirectional transformers, which
has become a cornerstone in natural language understanding tasks due to its ability to capture
contextual information.
● There are different sizes for BERT, such as BERT-base with 12 encoders and BERT- larger with 24
encoders, but(Abdel-salam et. al., 2022) focus on BERT-base for the purpose of this study.
● (Su et. al., 2020)An extractive model based on the BERT-based summarization model(BERTSUM) is
then constructed to extract the most important sentences from each segment.
PROBLEM DEFINITION
1. Understanding context 2. Limited to English Language
• Some summarization tool available do not • Some text summarization tools are limited
clearly understand the context to give to English language and we want to
accurate and clear summaries like expand our summarization tool so that it
AutoSummarizer, SummarizeBot, and summarization the given input texts from
SummarizeThis other languages like Hindi, Arabic,
Spanish, Scottish and so many more.
3. Big Volume of Data 4. Usability and Accessibility
• Usability and accessibility issues are some
• Considering big volume of information
of the issues we would like to overcome by
available, people do not have enough time
developing a summarization tool that is
to go through all of them to grasp main
easy to use for everyone regardless their
ideas, so they need summarization tool
technical expertise.
BERT Architecture
• BERT is available into sizes such as
BERT base with 12 encoder layers and
BERT large with 24 encoder layers.
• Fine tuning works on pre-trained
dataset to solve the NLP tasks like text
summarization where uses BertSum, to
add summarization layer that can utilize
the context vectors of words(BERT’s
output) and generate the context of
sentences to go into final summary.
Proposed Methodology
Pre-Training
Fine-Tuning
Select the Dataset Get BERT perform extractive
This include Masked Language
Model which let 15% of words in summarization
This involves selecting a dataset
sentence be masked. Next
that contain the corpus or other
Sentence Prediction for order of
textual data
sentence Sentence scoring
Bert based model assigns score
for each sentence in the
document.
Pre-processing
Top ranked sentences are Finally, test the
Clean and preprocess the
chosen for inclusion in performance of our
textual data to remove the
the summary. model and calculate
noise. Tokenize the input.
Current Status
● Sofar, we have been able to understand the basics and all necessary details required for our
project implementation, we have started by understanding our project and what we intend to
implement.
● We have read different research papers and got to know what others have done about this
prominent project and found out that there is a lot yet to be done.
● We have analyzed different implementations done by others for us to know where to start and
improve.
● We have run some sample codes and get to know how BERT works perfectly on this part of text
summarization compared to other models like GPT-2 and XLNet.
Scope of Future
MARCH APRIL MAY FUTURE..
Increase the accuracy of the Working on the research paper for Implement our own powerful Bert Make the text summarization for
summarization model our major project model for the text summarization. other languages other than
English.
Conclusion
● By concluding, we got enough understanding about our problem statement and all basics
skills required to implement our major project.
● we focus on improving text summarization using BERT, a powerful model. We aim to create
short and high-quality summaries of texts by making text summarization more accessible
and efficient, ensuring that valuable insights are easier to discover and use
● Through our team work and continuously working closer with our mentor we will be able to
achieve our goals for this major.
References
1. Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2018). BERT: Pre-training of Deep Bidirectional
Transformers for Language Understanding. ArXiv. /abs/1810.04805
2. Automatic Text Summarization Using Term Frequency, Luhn’s Heuristic, and Cosine Similarity
Approaches. (n.d.). Automatic Text Summarization Using Term Frequency, Luhn’s Heuristic, and
Cosine Similarity Approaches | IEEE Conference Publication | IEEE Xplore.
https://ieeexplore.ieee.org/document/10188527
3. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., &
Polosukhin, I. (2017). Attention Is All You Need. ArXiv. /abs/1706.03762
4. Abdel-Salam, S.; Rafea, A. Performance Study on Extractive Text Summarization Using BERT
Models. Information 2022, 13, 67. https://doi.org/10.3390/info13020067
Thank you!
