LLMs for text

classification and
generation
INTRODUCTION TO LLMS IN PYTHON

Iván Palomares Carrascosa, PhD

Senior Data Science & AI Manager
Loading a pre-trained LLM
Pipelines: pipeline() Auto classes ( AutoModel class)

Simple, high-level interface Flexibility, control and customization

Automatic model and tokenizer selection Complexity: manual set-ups

More abstraction = less control Support very diverse language tasks

Limited task flexibility Enable model fine-tuning

INTRODUCTION TO LLMS IN PYTHON

The AutoModel and AutoTokenizer classes
import torch.nn as nn from_pretrained()
from transformers import AutoModel, AutoTokenizer

Load pre-trained model weights and

model_name = "bert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name) tokenizer as specified in model_name
model = AutoModel.from_pretrained(model_name)
model_name : model checkpoint:
text = "I am an example sequence for text classification."
A unique model version with specific
class SimpleClassifier(nn.Module): architecture, configuration, and weights
def __init__(self, input_size, num_classes):
super(SimpleClassifier, self).__init__() AutoModel does not provide task-specific
self.fc = nn.Linear(input_size, num_classes)
head
def forward(self, x):
return self.fc(x)

INTRODUCTION TO LLMS IN PYTHON

The AutoModel and AutoTokenizer classes
inputs = tokenizer( Tokenize inputs
text, return_tensors="pt", padding=True,
truncation=True, max_length=64) Get model's hidden states in outputs
outputs = model(**inputs)
pooled_output = outputs.pooler_output pooler_output : high-level, aggregated
print("Hidden states size: ", outputs.last_hidden_state.shape)
representation of the sequence
print("Pooled output size: ", pooled_output.shape)

last_hidden_states : raw unaggregated

classifier_head = SimpleClassifier(
pooled_output.size(-1), num_classes=2) hidden states
logits = classifier_head(pooled_output)
probs = torch.softmax(logits, dim=1) Forward pass through classification head
print("Predicted Class Probabilities:", probs)
to obtain class probabilities
Hidden states size: torch.Size([1, 11, 768])
Pooled output size: torch.Size([1, 768])

Predicted Class Probabilities:

tensor([[0.4334, 0.5666]], grad_fn=<SoftmaxBackward0>)

INTRODUCTION TO LLMS IN PYTHON

Auto class for text classification
from transformers import AutoModelForSequenceClassification, AutoModelForSequenceClassification class:
AutoTokenizer

model_name = "nlptown/bert-base-multilingual-uncased-sentiment" Provides pre-configured model with a

tokenizer = AutoTokenizer.from_pretrained(model_name)
classification head
model = AutoModelForSequenceClassification.from_pretrained(
model_name)
No need to manually add model head
text = "The quality of the product was just okay."
inputs = tokenizer(text, return_tensors="pt")
outputs = model(**inputs)
logits = outputs.logits outputs already passed through head's
predicted_class = torch.argmax(logits, dim=1).item() linear layer
print(f"Predicted class index: {predicted_class + 1} star.")
Access raw class logits and return "most
likely" class
Predicted class index: 3 star.

INTRODUCTION TO LLMS IN PYTHON

Auto class for text generation
from transformers import AutoModelForCausalLM, AutoTokenizer AutoModelForCausalLM class:

model_name = "gpt2"
tokenizer = AutoTokenizer.from_pretrained(model_name) Pre-configured model for causal (auto-
model = AutoModelForCausalLM.from_pretrained(model_name)
regressive) language generation, e.g.:
prompt = "This is a simple example for text generation," "gpt2"
inputs = tokenizer.encode(
prompt, return_tensors="pt") Model head for next-word prediction
output = model.generate(inputs, max_length=26)
generate() takes prompt and generates
generated_text = tokenizer.decode(
output[0], skip_special_tokens=True) up to max_length subsequent tokens
print("Generated Text:")
print(generated_text) Raw outputs are decoded before printing
extended prompt with generated text
Generated Text:
This is a simple example for text generation, but it's also
a good way to get a feel for how the text is generated.

INTRODUCTION TO LLMS IN PYTHON

Exploring a dataset for text classification
from datasets import load_dataset load_dataset() : loads a dataset from
from torch.utils.data import DataLoader
Hugging Face hub
dataset = load_dataset("imdb") imdb: review sentiment classification
train_data = dataset["train"]
dataloader = DataLoader(train_data, batch_size=2, shuffle=True)

for batch in dataloader:

for i in range(len(batch["text"])): DataLoader class: simplifies iterating,
print(f"Example {i + 1}:")
print("Text:", batch["text"][i])
batch processing and parallelism
print("Label:", batch["label"][i]) Iterating through review-sentiment
examples
Example 1:
Text: Much worse than the original. It was actually *painf(...)
Label: tensor(0)
Example 2:
Text: I have to agree with Cal-37 it's a great movie, spec(...)
Label: tensor(1)

INTRODUCTION TO LLMS IN PYTHON

Exploring a dataset for text generation
from datasets import load_dataset Using a dataset from standfordnlp
dataset = load_dataset("stanfordnlp/shp", "askculinary")
catalogue
train_data = dataset["train"] Suitable for text generation and
print(train_data[i])
generative QA
for i in range(5):
example = train_data[i] Display some text information in data
print(f"Example {i + 1}:")
print("Title:", example["post_id"])
instances
print("Paragraph:", example["history"])
print()

Example 1:
Title: himc90
Paragraph: In an interview right before receiving the 2013
Nobel prize in physics, Peter Higgs stated that he (...)

Example 2 (...)

INTRODUCTION TO LLMS IN PYTHON

How text generation LLM training works
Input + target (labels) pairs

Input sequences: a segment of the text, e.g. "the cat is" from "the cat is sleeping on the
mat"

INTRODUCTION TO LLMS IN PYTHON

How text generation LLM training works
Input + target (labels) pairs

Input sequences: a segment of the text, e.g. "the cat is" from "the cat is sleeping on the
mat"
Target sequences: tokens shifted one position to the left, e.g. "cat is sleeping"

INTRODUCTION TO LLMS IN PYTHON

 Let's practice!
INTRODUCTION TO LLMS IN PYTHON
 LLMs for text
summarization and
translation
INTRODUCTION TO LLMS IN PYTHON

Iván Palomares Carrascosa, PhD

Senior Data Science & AI Manager
Inside text summarization
Goal: create a summarized version of a
text, preserving important information

Inputs: Original text

Target (labels): summarized text

INTRODUCTION TO LLMS IN PYTHON

Inside text summarization
Goal: create a summarized version of a
text, preserving important information

Inputs: Original text

Target (labels): summarized text

Extractive summarization: select, extract, and

combine parts of the original text

INTRODUCTION TO LLMS IN PYTHON

Inside text summarization
Goal: create a summarized version of a
text, preserving important information

Inputs: Original text

Target (labels): summarized text

Extractive summarization: select, extract, and

combine parts of the original text

Abstractive summarization: generate a

summary word by word

INTRODUCTION TO LLMS IN PYTHON

Exploring a text summarization dataset
from datasets import load_dataset example = dataset["train"][21]
example['Article']
dataset = load_dataset("ILSUM/ILSUM-1.0", "English")
print(f"Features: {dataset['train'].column_names}")
This is how an Apple Watch saved a man's life after detecting
accident. It all started when Gabe Burdett was waiting for his
Features: ['id', 'Article', 'Heading', 'Summary'] father Bob at their pre-designated location for some mountain
biking at the Riverside State Park when he received a text
alert from his dad's Apple Watch, saying it had detected a
Two main text attributes "hard fall".Burdett, from city of Spokane in Washington State
later received another update from the Watch, saying his father
had reached Sacred Heart Medical Center."We drove straight
Long text: input sequence for the LLM there but he was gone when we arrived. I get another (...)

'Article' in the example

example['Summary']
Summarized text: target, training label
'Summary' in the example Dad flipped his bike at the bottom of Doomsday, hit his head
and was knocked out until sometime during the ambulance ride.
The watch had called 911 with his location and EMS had him
scooped up and to the hospital in under a 1/2hr.

INTRODUCTION TO LLMS IN PYTHON

Loading a pre-trained LLM for summarization
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM Import and use AutoModelForSeq2SeqLM
model_name = "t5-small" Load t5-small : versatile for various tasks
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
Add a task-specific prefix to the input text:
input_ids = tokenizer.encode( "summarize:""
"summarize: " + example["Article"],
return_tensors="pt", max_length=512, truncation=True .generate() passes the tokenized input to
)
the model
summary_ids = model.generate(input_ids, max_length=150)
summary = tokenizer.decode( .decode() post-processes output
summary_ids[0], skip_special_tokens=True)
embedding back into text
print("Original Text:")
print(example["Article"])
print("\nGenerated Summary:")
print(summary)

INTRODUCTION TO LLMS IN PYTHON

 Loading a pre-trained LLM for summarization
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM Original Text:
This is how an Apple Watch saved a man's life after detecting
model_name = "t5-small" accident. It all started when Gabe Burdett was waiting for his
tokenizer = AutoTokenizer.from_pretrained(model_name) father Bob at their pre-designated location for some mountain
model = AutoModelForSeq2SeqLM.from_pretrained(model_name) biking at the Riverside State Park when he received a text
alert from his dad's Apple Watch, saying it had detected a
input_ids = tokenizer.encode( "hard fall".Burdett, from city of Spokane in Washington State
"summarize: " + example["Article"], later received another update from the Watch, saying his father
return_tensors="pt", max_length=512, truncation=True had reached Sacred Heart Medical Center."We drove straight
) there but he was gone when we arrived. I get another (...)

summary_ids = model.generate(input_ids, max_length=150)

Generated Summary:
summary = tokenizer.decode(
a man was waiting for his father when he received a text alert
summary_ids[0], skip_special_tokens=True)
from his dad's apple watch. the watch notified 911 with the
location and within 30 minutes, emergency medical services took
print("Original Text:")
the injured Bob to the hospital. the watch notified 911 with
print(example["Article"])
the location and within 30 minutes, emergency medical services
print("\nGenerated Summary:")
took the injured Bob to the hospital.
print(summary)

1 Due to space limitations, only the first 50% of the original input text is shown in the slide

INTRODUCTION TO LLMS IN PYTHON

Inside language translation
Goal: produce translated version of a text,
conveying same meaning and context

Inputs: text in source language

Target (labels): target language translation

INTRODUCTION TO LLMS IN PYTHON

Inside language translation
Goal: produce translated version of a text,
conveying same meaning and context

Inputs: text in source language

Target (labels): target language translation

Encode source language sequence

INTRODUCTION TO LLMS IN PYTHON

Inside language translation
Goal: produce translated version of a text,
conveying same meaning and context

Inputs: text in source language

Target (labels): target language translation

Encode source language sequence

Decode into target language sequence, using

learned language patterns and associations

INTRODUCTION TO LLMS IN PYTHON

Exploring a language translation dataset
from datasets import load_dataset Load English-Welsh bilingual dataset
dataset = load_dataset("techiaith/legislation-gov-uk_en-cy")
Dataset object

sample_data = dataset["train"] Extract a training example

source : English sequences
input_example = sample_data.data['source'][0]
target_example = sample_data.data['target'][0]
target : Welsh sequences
print("Input (English):", input_example)
print("Target (Welsh):", target_example)

Input (English): 2 Regulations under section 1: supplementary

Target (Welsh): 2 Rheoliadau o dan adran 1: atodol

INTRODUCTION TO LLMS IN PYTHON

Loading a pre-trained LLM for translation
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM Import and use AutoModelForSeq2SeqLM
model_name = "Helsinki-NLP/opus-mt-en-cy" Load Helsinki-NLP model for English-
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name) Welsh translation

input_seq = "2 Regulations under section 1: supplementary"

input_ids = tokenizer.encode(input_seq, return_tensors="pt")
translated_ids = model.generate(input_ids)
translated_text = tokenizer.decode(
Tokenize English sequence ( .encode() )
translated_ids[0], skip_special_tokens=True) and pass it to the model ( .generate() )
print("Predicted (Welsh):", translated_text)
Decode and print Welsh translation

Predicted (Welsh):
2 Rheloiad o dan adran 1:aryary " means "i

INTRODUCTION TO LLMS IN PYTHON

 Let's practice!
INTRODUCTION TO LLMS IN PYTHON
 LLMs for question
answering
INTRODUCTION TO LLMS IN PYTHON

Iván Palomares Carrascosa, PhD

Senior Data Science & AI Manager
Types of question answering (QA) tasks
QA task type Architecture Extractive QA: The LLM extracts the answer
Extractive Encoder-only to a question from a provided context

Open Generative Encoder-Decoder

Closed generative Decoder-only

Open Generative QA: The LLM generates the Closed Generative QA: The LLM fully
answer based on a context generates the answer, no context provided

INTRODUCTION TO LLMS IN PYTHON

Exploring a QA dataset
from datasets import load_dataset Load English subset of the xtreme dataset
mlqa = load_dataset( for extractive QA
"xtreme", name="MLQA.en.en")
DatasetDict object.
print(mlqa)
Test and validation Dataset objects.
DatasetDict({
test: Dataset({
features: ['id', 'title', 'context',
'question', 'answers'], Relevant features:
num_rows: 11590 'context'
})
validation: Dataset({ 'question'
features: ['id', 'title', 'context',
'question','answers'], 'answers'
num_rows: 1148
})
})

INTRODUCTION TO LLMS IN PYTHON

Exploring a QA dataset
Example instance in the dataset:

print("Question:" , mlqa["test"]["question"][53])
print("Answer:" , mlqa["test"]["answers"][53])
print("Context:" , mlqa["test"]["context"][53])

Question: what is a kimchi?

Answer: {'answer_start': [271], 'text': ['a fermented, usually spicy vegetable dish']}

Context: Korean cuisine is largely based on rice, noodles, tofu, vegetables, fish and meats. Traditional Korean
meals are noted for the number of side dishes, banchan, which accompany steam-cooked short-grain rice. Every
meal is accompanied by numerous banchan. Kimchi, a fermented, usually spicy vegetable dish is commonly served
at every meal and is one of the best known Korean dishes. Korean cuisine usually involves heavy seasoning with
sesame oil, doenjang, a type of fermented soybean paste, soy sauce, salt, garlic, ginger, and gochujang, a hot
pepper paste. Other well-known dishes are Bulgogi, grilled marinated beef, Gimbap, and Tteokbokki , a spicy
snack consisting of rice cake seasoned with gochujang or a spicy chili paste.

INTRODUCTION TO LLMS IN PYTHON

Extractive QA: framing the problem

Supervised learning: span classification

INTRODUCTION TO LLMS IN PYTHON

Extractive QA: framing the problem

Supervised learning: span classification

INTRODUCTION TO LLMS IN PYTHON

Extractive QA: framing the problem

Supervised learning: span classification

Prediction result: answer span given by: [start position, end position]

Answer span obtained from most likely raw outputs (logits)

INTRODUCTION TO LLMS IN PYTHON

Extractive QA: tokenizing inputs
from transformers import AutoTokenizer Tokenization results:

model_ckp = "deepset/minilm-uncased-squad2"
Tensor Description
tokenizer = AutoTokenizer.from_pretrained(model_ckp)
input_ids Integer
question = "How is the taste of wasabi?"
attention_mask Boolean
context = """Japanese cuisine captures the essence of \
a harmonious fusion between fresh ingredients and \ token_type_ids 0: Question, 1: Context
traditional culinary techniques, all heightened \
by the zesty taste of the aromatic green condiment \
known as wasabi."""
inputs = tokenizer(question, context,
return_tensors="pt")

INTRODUCTION TO LLMS IN PYTHON

Extractive QA: loading and using model
from transformers import AutoModelForQuestionAnswering Custom model class:
AutoModelForQuestionAnswering
model = AutoModelForQuestionAnswering.
from_pretrained(model_ckp) Inference on example input:
with torch.no_grad():
**inputs unpacks and extracts
outputs = model(**inputs) tokenized inputs

start_idx = torch.argmax(outputs.start_logits) Raw outputs post-processing

end_idx = torch.argmax(outputs.end_logits) + 1
start_logits , end_logits answer
answer_span = inputs["input_ids"][0] start/end likelihoods per input token
[start_idx:end_idx]
answer = tokenizer.decode(answer_span)
start_idx , end_idx : positions of input
tokens delimiting answer span

INTRODUCTION TO LLMS IN PYTHON

Managing long context sequences
long_exmp = tokenizer(example_qt, example_ct,
return_overflowing_tokens=True,
max_length=100, stride=25)

for idx, window in enumerate(long_exmp["input_ids"]):

print("Tokens in window ", idx, ": ", len(window))

No. tokens in window 0 : 100

No. tokens in window 1 : 100
[...]
Sliding window parameters:
No. tokens in window 8 : 50

max_length : sliding window size

for window in long_exmp["input_ids"]:
print(tokenizer.decode(window), "\n")
stride : stride size between windows

[CLS] what is a kimchi? [SEP] Korean cuisine is l[...]

[CLS] what is a kimchi? [SEP] steam-cooked short-[...]

INTRODUCTION TO LLMS IN PYTHON

 Let's practice!
INTRODUCTION TO LLMS IN PYTHON
 LLM fine-tuning and
transfer learning
INTRODUCTION TO LLMS IN PYTHON

Iván Palomares Carrascosa, PhD

Senior Data Science & AI Manager
Revisiting the LLM lifecycle

INTRODUCTION TO LLMS IN PYTHON

Revisiting the LLM lifecycle

Full fine-tuning: The entire model weights are updated; more computationally expensive

INTRODUCTION TO LLMS IN PYTHON

Revisiting the LLM lifecycle

Partial fine-tuning: Lower (body) layers fixed; only task-specific layers (head) are updated

INTRODUCTION TO LLMS IN PYTHON

Demystifying transfer learning
Transfer learning: a model trained on one task is adapted for a different but related task

In pre-trained LLMs, fine-tune on a smaller dataset for a specific task

Zero-shot learning: perform tasks never "seen" during training

One-shot, few-shot learning: adapt a model to a new task with one or a few examples only

INTRODUCTION TO LLMS IN PYTHON

Fine-tuning a pre-trained Hugging Face LLM
import torch Load BERT-based model for text
from transformers import AutoModelForSequenceClassification,
AutoTokenizer
classification and associated tokenizer
from datasets import load_dataset

model_name = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
Tokenize dataset used for fine-tuning
model = AutoModelForSequenceClassification.from_pretrained(
model_name, num_labels=2) IMDB reviews dataset
def tokenize_function(examples): truncation=True truncates input
return tokenizer(
examples["text"], padding="max_length", truncation=True) sequences beyond model's max_length

data = load_dataset("imdb") batched=True to process examples in

tokenized_data = data.map(tokenize_function, batched=True)
batches rather than individually

INTRODUCTION TO LLMS IN PYTHON

Fine-tuning a pre-trained Hugging Face LLM
from transformers import Trainer, TrainingArguments TrainingArguments class: customize training

training_args = TrainingArguments(
settings
output_dir="./smaller_bert_finetuned",
per_device_train_batch_size=8,
Output directory, batch size per GPU,
num_train_epochs=3,
evaluation_strategy="steps", epochs, etc.
eval_steps=500,
save_steps=500,
logging_dir="./logs",
)
Trainer class: manage training and
trainer = Trainer(
validation loop
model=model,
args=training_args,
train_dataset=tokenized_datasets["train"], Specify model, training arguments, training
eval_dataset=tokenized_datasets["test"],
)
and validation sets

trainer.train()
trainer.train() : execute training loop

INTRODUCTION TO LLMS IN PYTHON

Inference and saving a fine-tuned LLM
example_input = tokenizer("I am absolutely amazed with this After fine-tuning, inference is performed as
new and revolutionary AI device",
return_tensors="pt")
usual
output = model(**example_input) Tokenize inputs, pass them to the LLM,
predicted_label = torch.argmax(output.logits, dim=1).item()
print("Predicted Label:", predicted_label)
obtain and post-process outputs

Predicted Label: 0

Fine-tuned model and tokenizer can be

model.save_pretrained("./my_bert_finetuned")
tokenizer.save_pretrained("./my_bert_finetuned")
saved using .save_pretrained()

INTRODUCTION TO LLMS IN PYTHON

 Let's practice!
INTRODUCTION TO LLMS IN PYTHON