Text Generation:Use Technique Like Markov

Models or LSTM Network To Generate Realistic

Text In a Specific Style or Genre.
Engineering
Lovely Proffesional University
Hariprasad Boddepalli Punjab , India
School of Computerscience and [email protected]

Text loading and reading are pre-processing techniques in

the preparation of raw textual data for machine learning
Keywords—machine learning , deep learning ,
algorithms ,LSTM , Markov models , one hot encoding,
Tokenizer.

I. INTRODUCTION tasks. Raw textual data usually lies outside the programming
software within external files and, as such, ought to be
The fast-emerging domain of research in natural language
accessed from these files and loaded into memory for
processing and machine learning is that of text generation:
processing purposes. This can easily be achieved using file
the development of models to generate autonomous, human-
handling methods in any programming language like the
like text from input data. It certainly constitutes one of the
Python built-in file input/output functions. After opening a
most interesting and impactful challenges in artificial
file, you will have all your data read at once as a single large
intelligence. Text generation has applications in many
string or as a list of strings depending on the text structure.
domains, such as automated content creation, chatbots,
The file itself and particularly the encoding type, such as
machine translation, creative writing, and information
UTF-8, for instance, would ensure that special symbols or
retrieval. When machines can understand and produce
non-ASCII characters are read error-free. After reading the
coherent text pertinent to the context, industries like
data, cleaning was also basic: stripping white space,
communication, entertainment, and education will be
removing punctuation, or adjusting inconsistent formatting.
revolutionized in practice.
The cleaned text can then be subjected to further processing
such as tokenization and sequence formation. Hence, correct
At its roots, text generation is based on the capacity to
text loading ensures that the data is represented correctly in
model the structure of natural language and to predict the
the structured format to be used to train the machine
probability of the occurrence of a word or phrase in a
learning models.
particular context. Of course, this is a daunting task owing to
the intrinsically complicated and variable nature of
2. SEQUENCE CREATION
language. The present research paper goes on to discuss
methods and techniques applied in the processes of training
An important step of the pre-processing pipeline for text
text generation models, which include data preprocessing,
generation model training is also sequencing. This step
devising model architectures, to the evaluation of the
transforms raw text data to a form that can be fed to a
generated text. I will also be discussing some of the
machine learning algorithm. By doing this, the model learns
challenges and limitations made by these models, mainly the
patterns that govern word order, sentence structure, and
coherence and contextual validity of the generated text. In
syntactic dependencies in the data. The aim is to divide the
this paper, through a detailed exploration of the body of
text into segments manageable for the model to understand
knowledge in the area, future outlooks of text generation and
and use in predicting subsequent words or phrases. the
their possible implications for industries and society at large
creation process of sequences begins by dividing the cleaned
shall be reflected on.
text into parts such as words or tokens. Each sequence
represents a fixed-size window of words that will be used to
II. METHODOLOGIES predict the next word in the sequence. For instance, if the
text contains the sentence "The quick brown fox jumps,"
A. DATASET SOURCE then a chain may be the first words only to be for example:
I had taken this data from kaggle Customer Churn ["The", "quick", "brown"], such that the learner is trying to
Prediction 2020.[1]this data set is about employee recharge guess the word, "fox."
plan , employee usage ,employee messages.
Sequences are created and encoded and then divided into
pairs of input and output. The input is composed of the
III. PREPROCESSING sequence of words; the output is the next word in the
sequence. Input-output pairs that are used to train the model
1. TEXT LOADING AND READING such that it learns the word relationship in the text and
predicts the next word in a sequence of words before them.

XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE

 4. PADDING

3. TOKENIZATION Padding is one of the significant preprocessing techniques

for natural language processing, which comes to the rescue
Tokenization can be considered one of the most elementary of variable length sequences. Among a wide variety of NLP
steps of NLP. It refers to breaking up a text into smaller applications, such as text classification, sequence labeling,
units called tokens. These are words, subwords, phrases, or or text generation tasks, many models require consistent
even characters- depending on the application and context. sizes; while this is the case with recurrent neural networks
Therefore, the importance of tokenization is that it enables (RNNs) and convolutional neural networks (CNNs).
unstructured text data to take on a structured format which However, text data, by nature, are not of the same length,
goes on to become very important for the analysis and and sentences, paragraphs, or even documents might have
subsequent processing by the machine learning model. In dramatically different numbers of words or tokens. Padding
the case of deep learning, especially for applications such as is used to standardize the length of input sequences, making
text classification, sentiment analysis, and text generation, it easier for machine learning models to process the data.
proper tokenization goes on to have a direct impact on the
performance and accuracy of the model. Padding in the code below is also utilized with tokenization
to get the text data ready for an LSTM model training:.
There is another type of RNN-LSTMs, which work
Inside this code, tokenization is performed with the help of particularly well with sequence data and require sequences
TensorFlow's Keras library namely Tokenizer class. It to have the same lengths. The padding implementation in
transforms text body here used as a poem to machine this code is facilitated by the pad sequences function from
learning model understandable format. The above process of tensorflow.keras.preprocessing.sequence module. This
tokenization used in this implementation steps are broken function plays a critical role in shaping the tokenized
down into several steps. This is important in preparing sequences so that they fit the requirements for the input to
textual data before training any RNN model with LSTM the LSTM model.
layers. It begins with the creation of a Tokenizer object. The
class creates a set of methods to work with tokenization, Padding is a preprocessing operation for the input sequence
such as fitting the tokenizer to a corpus and converting the of various lengths, which is crucial in natural language
text into sequences of integers. By default, the parameters processing. In the given code, padding helps transform
for the initialization of the Tokenizer class are customizable: variable length tokenized sequences into a uniform format
the maximum number of words to keep and filters for suited for training the LSTM model. The code successfully
characters or symbols to ignore. approaches the problems in regard to length variability by
utilization of the pad sequences function for better batch
processing performance for the model. While padding
The vocabulary can be built by the tokenizer reading ensures the imposition of structured input for deep learning
through the text and identifying unique words. It maps a models, there needs to be a consideration of the implications
unique integer index to each word according to its frequency and potential drawbacks regarding padding. This therefore
in the dataset. This indexing is particularly important calls for careful approaches to padding to ensure that the
because the text will now be translated into numbers that the model learns nicely while minimizing the information loss
model can use for training and prediction purposes. The and optimizing the training process.
highest indices correspond to the least frequently appearing
words in the text. This phenomenon aids in getting insight 5. ONE HOT ENCODING
into the data because it ensures common patterns are learned
well by the model. This technique is widely used in NLP as well as in other
machine learning techniques for encoding categorical data in
a binary matrix representation. It can be used more
Tokenization not only prepares the data for modeling but specifically for textual data by applying it to machine
also finds a great role in the generation process of text. The learning models so that they might interpret the categorical
sequences generated by tokenization are the core input to variables in a numerical format. One-hot encoding is notably
train the LSTM model. Capturing the hidden patterns and used in NLP so that words or tokens can be represented in a
structures that the sequences are likely to have allows the way that models might understand and interpret the text
model to generate coherent as well as contextually relevant better.
output once a seed input is provided.
One-hot encoding allows learning words relationship
This process of tokenization helps establish a systematic properly. That is, due to representing each word using the
approach to handling text data, so huge volumes of text data neural network as an unique binary vector, the model
can be effectively and efficiently trained to use deep processes and distinguishes various words without confusing
learning models. Given that the model generates new text at their meaning. Ordinal relationships that may arise from the
each iteration, it will predict the next words of a sequence use of integer encoding are thereby dispelled by one-hot
with regard to the vocabulary and numerical representations encoding. For example, if we use integer encoding, the
set up during tokenization. For this reason, the effective use model will probably misinterpret that the word with index 3
of tokenization strategies represents an important aspect of is "bigger than" or "of a greater importance than" the word
the success of NLP tasks. indexed by 2. One-hot encoding avoids the above
misinterpretation because it considers each word as an dataset after each epoch. This validation set is a separate,
individual case. independent data that was not used for training but rather for
validation purposes-that is, when the model generalizes well
to unseen data. That said, if one has increasing validation
Simply put, the technique of one-hot encoding is the most accuracy, it implies that the model learns and makes better
significant technique in the preprocessing pipeline of natural predictions; if it does not improve or even worsens, then it
language processing, especially in preparing categorical data may be overfitting in nature, which means that the model
for machine learning models. Below is an example code memorizes the training data rather than learning
where one-hot encoding is applied to the target variable and generalizable patterns.
transforms integer-encoded words into binary vectors that
will make the neural network learn effectively. One-hot Other metrics of evaluation include loss and perplexity.
encoding is necessary since it allows the representation of Perplexity, in particular, is another very common metric
categorical data in a manner that avoids ordinal used in language models. It calculates the uncertainty over
misunderstandings, further leaning with requirements of the model's predictions. It is closely related to the cross-
model training. Although the said advantages are enormous, entropy loss. The lower the value of perplexity, the better
it is crucial to beware of the dangers of increased the performance, as it means more confident predictions by
dimensionality and sparsity. Thus one-hot encoding would the model.
prove to be a basic and flexible resource in the whole
scenario of text representation and machine learning. The broader societal implications of text generation are deep
and profound. As these models keep becoming so advanced,
6. CODE IMPLEMENTATION they might eventually replace or augment human roles in
many industries. At the same time, this raises some very
https://colab.research.google.com/drive/ important questions about the future of work, the ethics of
1RVns_kEechYmQHBPq5hBmk_KJgKHb2lH using AI, and potential misuse such as generating fake news
or misleading content. It requires constant discussion
between the scientists, policymakers, and industry over how
IV. TRAINING the latter applies these technologies responsibly for the
betterment of society.
Text generation is that relatively new frontier in the field of
NLP and machine learning, concerning the development of We will discuss here several methods and techniques used
models that can autonomously generate human-like text for training the generation models on the text, preprocessing
given input data, and yet remain perhaps one of the most data, developing model architectures, and finally, some
interesting and influential tasks in artificial intelligence methods and techniques of text evaluation. In the following
today. Major applications include generating automated sections, we will discuss the limitations and weaknesses of
content, creating chatbots, machine translation, creative these models, especially in terms of their ability to produce
writing, and information retrieval. Ultimately, such coherent text in relevant context. Through an in-depth
machines will drive whole industries in terms of how analysis of the current state of the field, we hope to shed
communication, entertainment, and education are some light on future directions of text generation and their
entertained. greater impact on various industries and society at large.

This training is done over several epochs, where every

epoch is defined as one complete pass through the whole
dataset. In each epoch, backpropagation iteratively adjusts
the model's parameters, namely weights and biases, based on
the gradient of the loss function at the set of parameters
given. Then, with the calculated gradients, the weights of the
model are adjusted in the direction where the loss would be
minimized for better performance by the model in making
predictions.

All training is conducted in minibatch mode. This implies

that data is divided into small parts referred to as batches.
The batch size determines how many input-output pairs are
processed in parallel before updating the model's weights.
For instance, if a batch size of 32 is adopted, then it means
that the model at hand will process 32 sequences
simultaneously. mini-batch training helps reduce the usage
of memory and letting the model generalize better as
weights are updated more frequently than in the case when
training is performed on the full data set at once.
V. LITERATURE REVIEW
In training, the performance of the model is controlled in a
way that it is learning properly. This is typically achieved by One understanding of the vanishing gradient problem is that
monitoring the accuracy of the model over a validation the network wants to send all the information through longer
sequences. And with longer sequences, gradients get local responses are a need rather than an extended
diminished during BPTT. To rectify this problem, the coherence, such as in bot responses and automated dialogue
concept of LSTMs has been introduced. It is able to hold systems.[5]
onto long-term information by propagating only the relevant
information back. LSTMs are a kind of RNNs. Here, instead LSTM networks are highly flexible and can generate text
of a single There are several gates regarding the activation based on various tasks by using huge datasets specific to a
function in the hidden layer.[1] There is the input gate, style or genre. For example, Karpathy et al. (2015) trained
forget gate, the output gate, and the cell states. With the several corpora of texts as diverse as poetry and code in an
gates, at every time point, it will determine which of the past LSTM and demonstrated that it could generate syntactically
information to be held and which to discard for being appropriate sequences in each domain, though coherence
forgotten. and depth vary.[14] These indeed account for the wide-scale
applicability of LSTMs in tasks related to creative text
A lot of information from the previous state has to be generation.[11]
forgotten, and the output gate filters what information to
pass to the next layer. Such a nature makes LSTMs preserve Other techniques have been created, including conditional
history in long sentences and have been used extensively in LSTMs, which condition a model onto specific attributes
NLP applications from question-answering systems to such as genre or tone or sentiment. The model leans toward
machine translation.[2] The input gate controls the input those attributes, which have been highlighted during
concerning the current cell information and the forget gate generation to better suit a desired output style. This
concerning how much to keep of the old information. conditional generation has been very useful for creative
applications, such as the generation of text that would
The Hochreiter and Schmidhuber LSTM networks were the maintain the tone of a specific author or that adheres to a
innovation that tackled the "problem of vanishing gradients" pre-defined theme.[15]
in traditional RNNs, unlocking models to incorporate more
information on longer stretches of textual sequences. Recent innovations have taken this type of generating ability
LSTMs can keep track of context over sentences or even further with the advent of LSTM networks models, which
paragraphs, so it is not surprising that they are widely used then moved on to Transformer-based models like GPT-2 and
for tasks that demand semantic and stylistic coherence GPT-3. These models, based on self-attention mechanisms,
preserved over a greater span of text text generation.[4] find long-term dependencies more effectively compared to
LSTMs since the connections are not in a sequential nature;
LSTM networks have been very commonly used in stylized therefore, much better in large-scale applications for text
text generation such as for poetry and song lyrics up to news generation.[16]
articles and dialogue generation. Sutskever, Martens, and
Hinton (2011) already demonstrated that LSTM networks However, there is still an application for LSTMs, especially
can be learned upon complex syntactic structures and for the scenarios of limited resources used in the process of
generate high-quality grammatically correct text if they are training and, finally, in embedded systems where size
trained on large datasets. Other work has addressed the matters. Moreover, they give foundation architectures for
problem of text generation within particular genres and most of the hybrid models, merging recurrent and attention-
demonstrated that with enough training data, LSTMs can based techniques into achieving such a balance between
learn to mimic the stylistic flavor of certain authors or coherence and computational efficiency.
genres, such as Shakespearean English, modern news, and
scientific writing . Markov models and Long Short-Term Memory (LSTM)
The oldest technique used for text generation is the so-called networks are popular applied techniques in natural language
Markov models, traditionally simple, but computationally processing for generating texts in a certain style or genre.
efficient. Markov models also depend on the principle that a Simpler text generation tasks using Markov models have
word in sequence depends only on the immediately often utilized their probabilistic structure that creates
preceding word(s), effectively forming a chain of sequences based on transitions between states or words,
probabilistic predictions based on observed transitions depending on the preceding words in a sequence. These
between states (words or phrases). Studies such as models are better for shorter context and can indeed
Shannon's work in entropy in English text established the generate text with what is considered basic stylistic patterns
Markov assumption for natural language at the very early by adjusting the order of the model to include more or fewer
stages of research and threw more light upon probabilistic preceding words.[11] They utterly fail in longer contextual
generation using n-gram models, or Markov chains with a dependencies and fail to produce the nuance that would be
fixed number of words in a sequence of length n.[13] needed for highly realistic text generation.[12]

Generally, Markov models efficiency in text generation Generalizing from these features, LSTM networks are much
stems from their capability to produce almost short and better for dealing with longer dependencies and thus are
superficial sequences of text. Although n-gram models generally much more useful for capturing and reproducing
effectively model local word dependencies, they fail to the complex language patterns characteristic of specific
model very long contexts and tend to produce repetitive or genres or styles. LSTMs actually use gated memory cells,
inconsistent text with extended passages, with increasing the which enables earlier parts of a word being remembered
size of the n-gram to capture more context to produce when deemed important and filtered out otherwise. As such,
greater complexity and higher memory requirements. they can produce longer passages containing cohesion,
Despite this, Markov models remain relevant where quick, context, and special stylistic features. It has also been
demonstrated that LSTM-based models show impressive neural network, seq length. The padding is essential for the
performance in text generation for purposes, including model to process variable-length sequences in a consistent
literary style imitation, or in generating conversational way. The padded input is then fed into the model and y pred
agents by adjusting hyperparameters such as temperature = np argmax(model predict(encoded), axis=-1) predicts the
that balance creativity with fittingness to the target features next word index, printing the highest probability from the
of genre[11] model's vocabulary. To translate this predicted index back
into a human readable word, the function iterates through
combining LSTM networks with specific preprocessing tokenizer word index items() until the index of the word
techniques—like tokenization, removing noise, and matches y pred, storing the matched word in predicted word.
managing punctuation—enhances their ability to replicate
complex text structures. Comparative analyses indicate that The predicted word is added both to the input text for the
LSTM networks generally outperform Markov models in future prediction and the text list, consisting of the words of
terms of contextual relevance and quality, especially for the current line. Adding to the input text each new word
longer and stylistically nuanced texts. Some studies also enables the function to generate contextually relevant words
explore hybrid approaches, leveraging Markov chains for that continue a coherent sequence. Ending the inner loop
initial state generation and LSTM models for fine-tuning, after the iterations text length, joins together the text-a list of
aiming to optimize both efficiency and coherence in text words-into a string, forming a coherent line, that is,
production appended to general text.

To gain an in-depth understanding of these methods, refer to Once all lines are generated, it outputs the general text in a
the IEEE articles on features and performance of LSTM and structured form suitable for tasks in which generation
the IJRASET studies on text generation models and data requires text lines to be multi-line, such as chatbot
handling in text preprocessing for genre-specific tasks. For dialogues, storytelling, or style-specific text production.
further reading, you can find resources on the IEEE Xplore Iterative word-by-word prediction ensures that each
and IJRASET websites[12] produced line will be coherent by using prior context in each
prediction. Such a structure is particularly well-suited to
neural networks, as these are designed to capture across-
VI. PROCEDURE dependencies within words for the production of fluent and
relevant text output.

The generate text function is the code for a sequence

generation function; this means that the function can output
a specified number of lines in controlled text. It acts as an
iterative predictive loop, where each word inputted becomes
prediction for the next word. Function The function works
on a trained neural network language model; previous
outputs are used to generate coherent and consistent style
text.

The function, generate text, works for neural network

models. It usually uses languages as inputs whereby the aim
is to predict a sentence from the initial word input. The
function updates a phrase in the input and then generates a
specified number of lines of text by predicting the next word
sequentially.

Setting text length to 15 means that this function sets the

number of words per line; this will limit the size of the
generated output since it has to ensure that each line uses a
constant number of words-this is specially useful in
generating paragraphs. Here in the function definition,
general text initializes with setting all the full output of
generated lines. The following loop for i in range(no lines)
executes a certain number of times, once for every line to be
produced. For each line, a temporary list, text, is populated
with words. A final, nested loop with a range of text length
then starts the process of building one line, one word at a
time, by predicting one word at a time.

The first step of each iteration of the inner loop is to encode

the input text using tokenizer texts to sequences that
translate words into numerical values according to the
trained model vocabulary. The encoded sequence is padded
with pad sequences to fit the expected input length of the
 contextually relevant sequences of text. It has increased the
efficiency in composing paragraphs and significantly
reduced the investment time.

REFERENCES

[1] Yoshua Bengio, Patrice Simard, Paolo Frasconi, et al.

Learning long-term dependencies with gradient descent is
difficult. IEEE transactions on neural networks, 5(2):157–
166, 1994.

[2] Mike Schuster and Kuldip K Paliwal. Bidirectional

recurrent neural networks. Signal Processing, IEEE
Transactionson, 45(11):2673–2681, 1997.

[3] Andrew M Dai, Christopher Olah, and Quoc V Le.

Document embedding with paragraph vectors. arXiv
preprint
arXiv:1507.07998, 2015.

[4] Laurens van der Maaten and Geoffrey Hinton.

Visualizing data using t-sne. Journal of machine learning
research,9(Nov):2579–2605, 2008.

[5] Rashid, A. Do-Omri, M. A. Haidar, Q. Liu and M.

Rezagholizadeh.”From Unsupervised Machine Translation
to
Adversarial Text Generation," ICASSP 2020 - 2020 IEEE
International Conference on Acoustics, Speech and Signal
Processing(ICASSP), Barcelona, Spain, 2020, pp. 8194-
8198, doi:10.1109/ICASSP40776.2020.9053236.

[6] L. Xuyuan, T. Lihua and L. Chen, "TCTG:A

Controllable Text Generation Method Using Text to
Control Text Generation," 2021IEEE 6th International
Conference on Signal and Image Processing(ICSIP),
VI1. FUTURE WORK
Nanjing, China, 2021, pp. 1118-1122,
doi:10.1109/ICSIP52628.2021.9688767.
LSTM networks proved to be the best existing type of model
till today for prediction as well as classification over text- [7] R. Ma, Y. Gao, X. Li and L. Yang, "Research on
based data. LSTM can effectively overcome the problem Automatic Generation of Social Short Text Based on
that standard recurrent neural networks have been facing, Backtracking Pattern,"2023 Asia-Pacific Conference on
i.e., the problem of vanishing gradient. It is an efficient Image Processing, Electronics and Computers (IPEC),
model but overall it is quite computationally expensive and Dalian, China, 2023, pp. 336-347,
requires high processing power, hence the use of GPUs in doi:10.1109/IPEC57296.2023.00066.
order to fit and train the model. They are currently employed
in numerous applications ranging from voice assistants, [8] S. V. Hemanth, Saravanan Alagarsamy & T. Dhiliphan
smart virtual keyboar- ds and automated chatbots to Rajkumar," A novel deep learning model for diabetic
sentiment analysis etc. As future research work, the model retinopathy detection in retinal fundus images using pre-
accuracy of current LSTM models may be surpassed by trained CNN and HWBLSTM, Journal of Biomolecular
appending more layers and nodes to the network and Structure and Dynamics, 2024,
applying the notion of transfer learning on the same problem DOI:10.1080/07391102.2024.2314269.
domain.
[9] Y. Wu, H. Yin, D. Liu and Q. Zhou, "Text Semantic
Representation Based on Knowledge Graph Correction,"
VI11. CONCLUSION 2022 International Conference on Computer Engineering
and Artificial Intelligence(ICCEAI), Shijiazhuang, China,
Based on our research, we have been forced to use LSTMs 2022, pp. 404-408, doi:10.1109/ICCEAI55464.2022.00090.
as the ubiquitous challenge of next word prediction is in
such diverse contexts such as emails and WhatsApp [10] Saraswat S, Srivastava G, Shukla S (2018)
messages. The results have turned out excellent when Classification of ECG signals using cross-recurrence
compared to the previous studies with an accuracy rate of quantification analysis and probabilistic neural network
85.50%. LSTMs are found effective in coherent and
classifier for ventricular tachycardia patients. Int J Biomed [17] Keskar, N. S., McCool, M., & Gulrajani, I. (2019).
Eng Technol 26(2):141–156. CTRL: A Conditional Transformer Language Model for
Controllable Generation. arXiv preprintarXiv:1909.05858.
[11]https://colab.research.google.com/drive/
1RVns_kEechYmQHBPq5hBmk_KJgKHb2lH#scrollTo=8
GlQJ4DX6uz9

[12]https://arxiv.org/pdf/2005.00048

[13]https://www.ijraset.com/author.php

[14] Sutskever I, Martens J, Hinton GE (2011) Generating

text with recurrent neural networks. In:
Proceedings of the 28th international conference on machine
learning (ICML-11)

[15] Sundermeyer M et al (2012) LSTM neural networks for

language modeling. In: INTERSPEECH

[16] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel,

V., Thirion, B., Grisel, O., ... & Duchesnay, É. (2011).
Scikit- learn: Machine learning in Python. Journal of
Machine Learning Research, 12, 2825–2830.