CS-C1000 – Introduction to Artificial Intelligence

Computer Assignment A

Mohammad Vali

March 11, 2025

 Computer Assignment A: Text Generation
▶ Today’s task: Generate text with GPT-2 model in JupyterHub.
▶ Optional Read: Alex Hern, “New AI fake text generator may be too
dangerous to release, say creators”. The Guardian, February 14, 2018.
▶ Accessible via the link: https://www.theguardian.com/technology/2019/feb/14/
elon-musk-backed-ai-writes-convincing-news-fiction

CS-C1000 · Mohammad Vali · 2/19

 GPT-2: High-level architecture

GPT-2

Embeddings

Prediction
Layer 1

Layer 2

Layer 3

Layer 4
"The quick brown fox" "jumps"

Figure inspiration from https://huggingface.co/docs/transformers/en/llm_tutorial.

Text example: https://en.wikipedia.org/wiki/The_quick_brown_fox_jumps_over_the_lazy_dog.

CS-C1000 · Mohammad Vali · 5/19

 GPT-2: Text embeddings

"The quick brown fox"

"The" 464 [-0.036, 0.016, ..., -0.079]

Embeddings

Layer 1
"quick" 2068 [ 0.007, -0.082, ..., -0.158]
"brown" 7586 [-0.114, -0.019, ..., -0.042]
"fox" 21831 [ 0.043, -0.181, ..., -0.162]

1. Convert words into (unique) index from vocabulary.

2. Convert word index into (unique) word embedding.
3. Pass word embeddings to Layer 1.

CS-C1000 · Mohammad Vali · 6/19

 GPT-2: Next word prediction
output word
probabilities

p("is") = 0.007
pick next word
p("the") = 0.000 based on its

Prediction
...

Layer 4
probability
"The quick brown fox" ... p("jumps") = 0.792 "jumps"
p("jumped") = 0.111
...
p("<eos>") = 0.000
extend sentence
with "jumps"

1. Outputs probabilities for every word in the vocabulary.

2. Pick (sample) the next word based on the probabilities.
3. Extend sentence with the new word and generate the next.
CS-C1000 · Mohammad Vali · 7/19
 GPT-2: Top-5 word probabilities
"The quick brown fox" "The quick brown fox jumps" " "

GPT-2 GPT-2 GPT-2

p("jumps") = 0.792 p("over") = 0.923 p("The") = 0.257

p("jumped") = 0.111 p("the") = 0.058 p("A") = 0.094
p("to") = 0.013 p("out") = 0.011 p("This") = 0.062
p("es") = 0.008 p("on") = 0.002 p("In") = 0.053
p("has") = 0.007 p(",") = 0.002 p("I") = 0.040
.. .. ..
. . .

CS-C1000 · Mohammad Vali · 8/19

 GPT-2: High-level summary GPT-2

Embeddings

Prediction
Layer 1

Layer 2

Layer 3

Layer 4
"The quick
"jumps"
brown fox"
▶ Training by predicting next word in text,
similar to generation.
▶ Transformer layers are the complex
part1,2,3 .
▶ Combine relationships in text sequence to
each word (Attention).
▶ Refine word representations (MLP).

▶ More parameters and more training Source: https://jalammar.github.io/illustrated-gpt2/

typically leads to better generation. Models GPT-2 S GPT-2 M GPT-2 GPT-3 GPT-4
#params 124M 355M 1.5B 175B ?
Source: https://en.wikipedia.org/wiki/Large_language_model

1
Blog post: https://jalammar.github.io/illustrated-gpt2/
2
”An Introduction to Transformers”, Richard E. Turner (2023). https://arxiv.org/abs/2304.10557
3 CS-C1000 · Mohammad Vali · 9/19
Youtube: https://youtu.be/wjZofJX0v4M?si=zjk7DnuEs_egGhWr
 Computer Assignment A

▶ Run code in JupyterHub to generate text with GPT-2 medium (355M params).
▶ Only need a web browser. No coding required.

▶ Answer quiz questions on MyCourses to receive points.

▶ Deadline for quiz: Tuesday March 18 at 14:00.

CS-C1000 · Mohammad Vali · 10/19

 Log into JupyterHub

Log in to:
https://jupyter.cs.aalto.fi

Using your Aalto account.

CS-C1000 · Mohammad Vali · 11/19

Choose the course

CS-C1000 · Mohammad Vali · 12/19

JupyterHub: Find Assignment List

Click on Nbgrader > Assignment List in top panel.

CS-C1000 · Mohammad Vali · 13/19

 Fetch the notebook (A this time)

Click on ’fetch’.
This will create a folder /introtoai2025/ which holds the notebook.

CS-C1000 · Mohammad Vali · 14/19

 Start the notebook (A this time)

Click on Computer-Assignment-A to open the notebook.

Notebook can also be found under folder /notebooks/introtoai2025.

CS-C1000 · Mohammad Vali · 15/19

 Ready to run (one cell at a time)
▶ 1) Select cell, 2) press Shift+Enter on keyboard, or ’Run’ on panel

CS-C1000 · Mohammad Vali · 16/19

 If problems running the notebooks
▶ Try restarting the Kernel in JupyterHub.

▶ Kernel → Restart

CS-C1000 · Mohammad Vali · 17/19

 Tasks (read the notebook for exact instructions)

▶ Task 1: Generate some unconditional fake text.

▶ Task 2: Auto-complete some text.

▶ Task 3: Generate multiple novel text samples.

▶ Task 4: Generate samples of varying length.

▶ Task 5: Modify the model parameters.

▶ Task 6: Compare GPT-2 Small (124M params, 500MB) vs GPT-2 Medium

(355M params, 1.5GB).

▶ Task 7: Play around with the model.

CS-C1000 · Mohammad Vali · 18/19

 How to get points?

▶ Run the coding notebook on JupyterHub.

▶ Answer the questions in MyCourses under Exercise 2 (quiz).

▶ Deadline: Submit quiz answers before Tuesday March 18 at 14:00.

CS-C1000 · Mohammad Vali · 19/19