Exploring Scratch & AI: Possibilities & Pitfalls

In this lesson plan, we will explore the possibilities and pitfalls of AI and face detection, using the Face Sensing blocks in
Scratch Lab to create games, interactive stories, and accessible projects. We’ll also try out adding playful AI-generated images
to Scratch projects, focusing on how to write prompts that produce the desired creative results. Lastly, we’ll reflect on exploring
both the opportunities and the limitations of these new technologies through topics for classroom discourse.

“We should encourage learners to use ChatGPT and other generative AI tools not to
produce the final result, but as a resource throughout their own creative process.”
— Professor Mitch Resnick, co-founder of Scratch

Audience: Classroom Teachers, Instructional Technology Objectives (Learners Will):

Specialists, Library Media Specialists, Informal Learning ● Understand basic definitions of AI (predictive & generative)
Environments ● Gain hands-on experience using predictive AI via Face
Sensing with Scratch Lab and explore generative AI
integration with Scratch
Time: Approx 5 hours total
● Reflect on ways AI can be integrated into activities in
● Part 1: What Is AI? Pre-Reading and Context - 15 min
support of creative learning
● Part 2: Scratch Face Sensing Activities - 150 min
● Remix and/or adapt classroom prompts for discussing the
● Part 3: Using Generative AI with Scratch - 90 min
possibilities and pitfalls of AI
● Part 4: AI Discussion & Debate Topics - 45 min
● Communicate and share their projects with their learning
community
Resources for Learners:
See page 16 for aligned standards.
● Scratch & AI: Face Sensing (Tutorial Video)
● Scratch Lab Face Sensing Coding Cards

Additional resources provided throughout the guide.

Created by the Scratch Foundation (scratchfoundation.org) 1

Materials: In order to get AI-generated image results from Bing Chat, an AI tool we’ll be using as part of this
lesson, you’ll need to sign in to a Microsoft account. If you don’t have one, you can create a free account here:
https://account.microsoft.com.

Special Note on Working with AI Technology

When working with AI on various platforms, please be sure to check the Terms of Service to ensure the students’ age does
not prohibit them from directly interacting with the application. In some cases, the educator/facilitator may need to do the
direct input and discuss output with students if the output is deemed appropriate to share.

Per Scratch’s Terms of Service, students of any age may directly interact with/code projects using the Face Sensing blocks
on our Scratch Lab site. Face Sensing blocks run securely in your browser. Images and data are not stored or shared
anywhere. No personal data is collected or stored when using these blocks. The extension can detect that a face exists, but
cannot identify the person. (You can learn more on the Face Sensing homepage and our blog post.)

See this Digital Promise article for information on “How School Districts are Integrating Generative AI into their Policies.”

Created by the Scratch Foundation (scratchfoundation.org) 2

Part 1: What Is AI? Pre-Reading and Context

Introductory Material (15 minutes)

Context for Educators:

Begin by reading “AI and Creative Learning: Concerns, Opportunities, and Choices,” a blog post by Mitch Resnick,
Professor at MIT and co-founder of Scratch. This post outlines challenges of the technology as well as guiding principles for
how AI can support the creative process.

Context for Learners:

Below are three resources that be used to introduce and provide context for for learners:
● “AI Explained” (2:37 min video) | Common Sense Media
● “What is AI?” (8 min podcast) | Ken on Ed Tech, discussion with 5th grade students
● “What is data literacy?” (1:37 min video) | The Algorithm & Data Literacy Project

Key Concepts

Artificial intelligence (AI) is the umbrella term for computing systems that use data in order to analyze information, make
predictions, and generate creative ideas.

Predictive AI is a branch of AI that uses machine learning to classify data it receives in order to generate predictions. The
more sophisticated data we feed in, the more sophisticated and accurate the system will be. It learns patterns and
relationships in data. Examples could include:
● Facial detection (Scratch Face Sensing)
● Product recommendations (Facebook advertisements)
● Email automation and spam filtering (language classification)

Created by the Scratch Foundation (scratchfoundation.org) 3

Generative AI is a branch of AI that uses machine learning to analyze user prompts and create new content
such as text or images based on its best predictions of what the user wants. Examples could include:
● ChatGPT – an AI chatbot that uses “natural language processing” to create human-like dialogue and can respond to
questions and compose written content
● Dall-E – a system that enables users to create new images with text to graphics prompts, and develops visual
predictions based on data it has been trained on

Part 2: Scratch Face Sensing Activities

Explore the Face Sensing Blocks (10 minutes)

Scratch Lab is a site where you can engage with experimental blocks, like Face Sensing. With Face Sensing, create projects
that respond to your eyes, nose, and other parts of your face.

Step 1: To begin, navigate to https://lab.scratch.mit.edu/face.

Step 2: Click the “Try it out” button at the top of the page.

Step 3: A new project loads using the default Scratch Cat sprite. Just as in the Scratch editor, users can use this default
sprite or choose another from the sprite library.

Step 4: Face Sensing blocks use the device’s camera. If you haven’t enabled the camera in your browser for the Scratch
Lab site, you may be prompted to give permission.

How do Face Sensing blocks work? The AI we are using was trained on millions of photos of people’s faces, called a data
set. It detects things based on what it has already seen, like parts of a face and how they are arranged. The machine
learning model looks for patterns like: a nose is between the eyes and mouth, eyes are typically a certain size in relation to
the face, mouths may show teeth or not, etc. It can detect that a face exists, but cannot identify the person.

Created by the Scratch Foundation (scratchfoundation.org) 4

Experiment 1: Let’s try out the Face Sensing blocks! (10 minutes)

Step 1: Select the “go to nose” block and place it on the script
area.

Step 2: Click the block while your face is visible on the stage
and see what happens. Did the sprite go to your nose?
Try moving and clicking the block again.
Starter script.
Step 3: How can you make it constantly follow your nose?
Encourage them to find a block to help. Try selecting a
forever loop from the Control category and place the
“go to nose” block inside it.

Step 4: Select the Events category and add a “when green flag
clicked” block to the top of the code stack.

Step 5: Click the green flag to see how this works. Did the
sprite stick to your nose and follow it around as you Adding a forever loop.
moved? What happens if you click on the dropdown list
and choose another feature for the sprite to go to?

Step 6: What happens if you add an additional block like “turn

clockwise 15 degrees” from the Motion category? What
if you choose “go to left ear” from the dropdown menu
in the “go to nose” block?

Adding additional motion.

Created by the Scratch Foundation (scratchfoundation.org) 5

Experiment 2: Let’s draw a hat to stick to our head, like a face filter. (10 minutes)

Step 1: In the costume Paint Editor, try drawing a hat using

brush or shape tools.

Step 2: Add code, or adjust the code you already have, so it

sticks to the top of your head.

Step 3: Is the placement of the object just right on your face?

Experiment with the placement on the costume in the
Paint Editor. For instance, can you see the difference if
a hat costume is centered versus when only one edge Creating a sprite in the Scratch Paint Editor.
is on the centerpoint?

Step 4: Add another sprite from the sprite menu or draw

another sprite and stick it to your ear, or between your
eyes… What else can you try?

Step 5: Have you created multiple hat costumes? Add code to

switch costumes.
Starter script.
Take time to make some observations while you experiment:

● How does the object appear on your face when you are
close to the camera? What happens if you are further
from the camera? Do you see any blocks you could use
so it scales to match the size of your face?

● Does the object match the direction of your face tilt? Do

you see blocks to match it?

Adding blocks to control direction and size.

Created by the Scratch Foundation (scratchfoundation.org) 6

Can You Fool the AI? (30 minutes)

On a new tab, navigate to the Face Sensing homepage again and scroll down until you find the Starter Projects section.
Click on the “Hat and Glasses” starter project. What blocks is this program using? What do you think they do?

False Positives

The AI that we’re using is trained to detect human faces. False

positives are things that are not actually a human face
(false) that it detects as a face (positive). What are some false
positives you can find?

Using the “Hat and Glasses” starter project, make sure your
own face is out of the frame and hold up something with a face
(like a smiley face drawing, a picture of an animal face, or a toy).
See if the hat and glasses track smoothly (a false positive!) or
randomly jump around (indicating it does not recognize the
object as a face).

What variables can you change to try to fool it into thinking it

sees a face? Do all elements of a face (eyes, ears, nose, and
mouth) need to be present for the AI to detect a face?

People can easily see that an animal has a face, but why might
some animal faces (like cats) be recognized while others (like
cows or dogs) may not be seen as a face? Think about the
proportions of a human face and how those are different from a Examples of possible false positives: drawings of faces, animal
cow’s face with a longer snout. Those measurements don’t faces (cat may be seen as a face, while cow not), Lego head, a
match the human faces in the photos the AI trained on, so it filled in circle (depending on the shading and direction of the
lines, etc.).
doesn’t see a cow’s face as a face!

Created by the Scratch Foundation (scratchfoundation.org) 7

False Negatives
A false negative when working with the Face Sensing blocks
is when a human face is in the frame, but the AI does not
detect it. What are some false negatives you can find?
Using the “Hat and Glasses” starter project, try to fool the AI
into not seeing a face. See if the hat and glasses track
smoothly, or if they randomly jump around (a false negative!).
Can the AI find the parts of a face if:
● you are in disguise or your face is covered?
● your face is tilted or upside down or far away? (to make
this easier, try holding up a photo of a face)
● the lighting in the room is very bright or very dark?
People can see an upside down face is a face, but AI may not
be able to. Why? Likely, the data set the model was trained on Examples of possible false negatives: too small or upside down.
contained photos that were all rightside up, so it expects that.

Classroom Reflection Prompts (15 minutes)

Face Sensing blocks provide a meaningful opportunity to open up conversations with your learners about AI ethics and how
these concepts will shape our future. Several classroom reflection prompts:
● What does it recognize as a face? What were you surprised it recognized/did not recognize? Does it appear to treat
people of different races, genders, and ages fairly?
● Why was the technology developed? What do you think it was developed for?
● What if you don’t want to be seen?
Recommended Reading:
● Exploring a Creative, Safe Introduction to Machine Learning (Scratch Team) - This blog post introduces the Face
Sensing blocks and explores the design decisions and inspiration for the project. It can be used to generate
classroom discussion on ethical implications of AI models.

Created by the Scratch Foundation (scratchfoundation.org) 8

Create a Face Sensing Project (1 hour)

Look at the three Face Sensing hat blocks. (A hat block is a

block with a rounded top that starts a script.) Can you make an
interactive project, game, or story using one of these?
Options could include:
● using a variable to keep score when a face is detected
● making a sound, changing colors, or displaying
information when you touch a sprite
● change a background or move a sprite when you tilt your
head
● creating a sound board
Examples using “when a face is detected” and “when face tilts.”
● creating a drawing program that allows users to draw
with their nose or eyes
The possibilities are endless! What will you create? See our
coding cards and video tutorial shared below for more detailed
information on these examples.
Example using “when this sprite touches a (nose)” hat block.

Resources:
● Scratch & AI: Face Sensing (Tutorial Video) - Join Eric and Maren as they explore the face sensing blocks, try to fool
the AI, and show some example projects for inspiration.
● Scratch Lab Face Sensing Coding Cards (Student-Facing Cards) - Printable cards students can use to follow along
with the lesson
● Getting Started Guide (Written Guide) - If you are new to Scratch and just getting started, this resource has helpful
information.
● Scratch Ideas Page (Webpage) - Short tutorials and Scratch Coding Cards.

Created by the Scratch Foundation (scratchfoundation.org) 9

Special Note on Saving Projects

Projects created on Scratch Lab cannot be saved to

scratch.mit.edu. They can be saved locally and uploaded to the
Scratch Lab project page anytime if users want to continue
iterating on their work or share their creations.
To save a project: go to the top menu and choose File > Save
to your computer (saved file .sb3).
To load a project: choose File > Load from your computer.

Share: Gallery Walk (15 minutes)

Scratch Lab projects can’t be saved to a Scratch account and

can only be saved locally. One way to share projects as a class
is to host a gallery walk:
● Learners open their project on their computer.
● Participants can walk around a room, or take turns
sharing their screen in a virtual space.
● Another option is to display one project at a time on a
large screen at the front of the room.
Scratch Lab does not include a project page, but learners can
include recorded instructions in their project or with actual
sticky notes next to their screens. The handout below provides
a guide for sharing feedback between participants.

Resources:
● Face Sensing Project Sharing Sheet (Worksheet) - A
place where learners can share instructions as well as
receive feedback and observations.

Created by the Scratch Foundation (scratchfoundation.org) 10

Part 3: Using Generative AI with Scratch

Testing Our Prompt Skills with a Chatbot (30 minutes)

Chatbots can generate new ideas that expand our creative process. But generative AI models are only as reliable as the data
it has access to and the specificity and context provided in a user’s prompt. Let’s practice writing prompts that generate
positive results. Begin by opening Bing Chat (https://www.bing.com/new) or another approved AI platform, and click “Chat
Now.” (See our Special Note on Working with AI Technology above.)

Getting Started
Since we’ve been talking about faces, let’s try entering the prompt: “What is a face?” There are many ways to answer this
question. If you asked a person, they likely would have a specific perspective.
● How is the answer different from how you would answer?
● Discuss the results with a partner or as a class/group.
● What if you use a more specific prompt like, “What are the key elements of a face?”

Prompt for Ideation & Creativity

How could a chatbot or image generator be used to support or begin the creative process? Some learners find a blank
Scratch project or open-ended project ideas difficult to approach. Try a prompt that generates specific ideas, such as:
● “Help me come up with three creative project ideas for using Face Sensing in Scratch Lab”
● “How do I use a variable in a Scratch game?”
● “Generate an image of an (imaginative background image to upload into Scratch)”

Tips for Writing Prompts

The more specific your prompt is, the more likely you’ll get specific responses back. For instance, you might ask the AI to:
● “Write in the voice of a [teenager, five year old, an expert, or critic.]”
● Focus on the goal or purpose: should the response inform, persuade, or entertain?
● Include keywords or phrases related to your topic.
● Stay clear of certain topics or respond within limits: “what are three ideas for…”.
● Include source citations that support the information shared and allow for source citation.

Created by the Scratch Foundation (scratchfoundation.org) 11

Resources:
● Get the best from ChatGPT with these golden prompts (article) | The New York Times.

Generating AI Images to Use in Scratch (45 minutes)

Now, let’s try generating playful AI-generated images that can be added to Scratch projects as sprites or backgrounds. We
want to generate new images, versus searching the internet for an image that has already been created. There are a couple
of pathways to start:

Step 1: Open Bing Chat. Choose “Chat now.” You can generate a text response or an image response. Since you want an
image, specify in your prompt to “generate an image of…”. Note: you must be logged in to a Microsoft account to
access image creation.

Step 2: Alternatively, you can navigate to https://www.bing.com/images/create. Bing uses “Dall-E,” a generative AI system
that can create new images from text prompts.

Step 3: Type in an image prompt like “rainbow cat.” What are the results?

Step 4: What happens if you specify the art style you want (impressionist, cubism, art deco…)?
Adding details increases the chances you’ll get a better result. Try adding adjectives or locations or additional
objects to your prompt, like “rainbow cat wearing sunglasses sitting on a beach, impressionist.”

Step 5: Download one of the generated images.

Step 6: In Scratch, upload the image as a backdrop or as a sprite. (Images may need to be resized in our Paint Editor after
being uploaded, or you may want to remove the background for a sprite image. See our resources below for more
information on that.)

Step 7: Add some code to bring your object to life! Perhaps you want to use blocks from the Looks category to adjust the
color, or Motion blocks to move a sprite around the stage.

Getting the image you want will likely take some iterations. AI image generation doesn’t need to replace the creative process,
but it can become a part of it. The images generated can be a creative starting point. They can be edited inside of Scratch to
add or remove elements. Or, you can combine multiple generated images into one.

Created by the Scratch Foundation (scratchfoundation.org) 12

Resources:
● “Inside Scratch Lab: AI Image Generation” (Blog Post) by Eric Rosenbaum, Director of Scratch Lab and
co-inventor of Makey Makey.
● Bring Your Drawings Into Scratch (Video Tutorial) and Bring Your Drawings Into Scratch (Written Guide) has helpful
hints for uploading images into Scratch.

Reflection (15 minutes)

Reflect on the generative AI experience using the following discussion prompts:

● How can you adjust the words of the prompt to get the results you want?
● How does AI help us express our ideas?
● How does AI limit our ability to express our ideas?
● Do you think you should credit AI art used in a Scratch project? How would you do that?
● Have you observed any bias present in the images that are generated? For instance, if you type in a prompt like
"computer programmer,” “nurse,” or “teacher,” do you see a diverse representation to choose from, or is one race or
gender primarily showcased?
● What impact do you think generative AI might have on schools and the way people learn? What about the impact on
teachers, doctors, and other professionals?

Created by the Scratch Foundation (scratchfoundation.org) 13

Part 4: AI Discussion & Debate Topics

Discussion, Debate, or Writing Activity (45 minutes)

We’ve explored some of the opportunities and the limitations of these new technologies, but it is important to reflect on the
possible pitfalls of this emerging technology as well. You could approach this activity as a classroom discussion, debate, or a
persuasive writing exercise.
For instance, when evaluating AI tools, you might look at these criteria Scratch uses to evaluate:
● Fairness: Only use models which have a published fairness evaluation (e.g. low false negatives and false positives
across geographic regions)
● Safety: Maintain the privacy and safety of children
● Responsibility: Gather feedback on questions of ethics and equity from advocates and domain experts
● Transparency: Publicly acknowledge the social concerns about face sensing technologies
● Accountability: Use feedback from our community to inform any future Face Sensing updates
Possible topics and associated prompts include:

Privacy and ● AI systems make predictions, but what happens if the predictions are wrong? If the AI cannot
Surveillance recognize a face during a Face Sensing project the project may not work correctly, but what if
the AI identifies a face incorrectly in other situations? What could be the consequences?
● What information is used to train AI systems? What happens if people do not want their
information included in training data? What are the privacy concerns?

Ethics, Equity, and ● What types of biases might be present in AI text or image generation? Whose perspectives or
Access voices are included in these models and who is excluded?
● What regulations might we put in place to protect intellectual property?
● How could AI generated work affect the livelihood of artists and writers, etc.?

Created by the Scratch Foundation (scratchfoundation.org) 14

Misinformation ● How can you determine if AI chatbot results are facts or misinformation? For
instance, if you asked the AI to give you ideas for a Scratch Lab Face Sensing
project, could you actually make all of the programs suggested?
● How can you verify the information you receive from a chatbot? Does the chatbot program you
used provide citations?

Extension Activities

Below are two learning resources you can use with students to explore the social, civic, and ethical implications of AI.
● The Ethics of Generative AI in the Classroom | Designed by Facing History & Ourselves for 6-9th grades. This
resource includes 3 lessons that facilitate classroom discourse on the impact generative AI tools such as ChatGPT
and DALL-E have on education.
● The Algorithm & Data Literacy Project | Designed by Digital Moment in partnership with CCUNESCO and UNESCO.
These lessons support a wide range of grades and provide interactive opportunities for students to ask critical
questions and generate their own ideas and opinions about the presence of algorithms in our lives and how they
influence our decisions and experiences. This site includes a “Sensing Faces With Scratch Lab” workshop, which
focuses on the way humans and machines identify faces via an unplugged activity and exploration of Scratch Lab
Face Sensing blocks.

Additional Resources

Activities and Curriculum Guides:

● Exploring Creative Possibilities with Generative AI in Scratch (slide deck) | Designed by University of Girona for the
Scratch Conference 2023.
● The RAISE Playground (site) | A block-based programming platform developed to support hands-on learning about AI
and robotics for learners and beginning programmers.
● AI + Ethics Curriculum for Middle School (site) | Designed by the MIT Personal Robots group.
● Teachable Machine (site) | A web-based tool that makes creating machine learning models fast, easy, and accessible
to everyone.

Created by the Scratch Foundation (scratchfoundation.org) 15

Context, Ethics, Policy Guidance:
● Artificial Intelligence in Education: A Reading Guide Focused on Promoting Equity and Accountability in AI
(reading list) | Center for Integrative Research in Computing and Learning Sciences (CIRCLS)
● Algorithmic Justice League (site) | Includes “Coded Bias: A Documentary” and much more.
● aiEDU (site) | AI Education Project
● A People’s Guide to AI (site) | Allied Media Projects
● Artificial Intelligence in Education (site) | Developed by ISTE, includes Tips for School Leaders
● “Generative AI and Creative Learning: Concerns, Opportunities, and Choices” (article) | Mitch Resnick, Professor at
MIT and co-founder of Scratch
● Does AI Have Creativity and Imagination? (video) | Code.org, middle or high school level
● “Don’t tell anything to a chatbot you want to keep private” (article) | CNN
● “Engaging children with AI ethics” (report) | The Alan Turing Institute
● Hello World Issue 22 Teachine-g & AI (e-magazine) | Raspberry Pi Foundation
● Resources to Explore and Use ChatGPT and AI (site) | Common Sense Media
● “How Artificial Intelligence Can Deepen Racial and Economic Inequities” (article) | ACLU

Standards Aligned

CSTA Standards ISTE Standards CASEL Framework RITEC Indicators

Link to full standards Link to full standards Link to full standards Link to full standards
● 1B-AP-09 - Create programs ● 1.1d Technology Operations ● Self-Awareness ● Autonomy
● 1B-AP-10 - Create programs ● 1.2.c Intellectual Property ● Social Awareness ● Competence
● 1B-AP-12 - Modify, remix, or incorporate ● 1.3.b Evaluate Information ● Responsible ● Creativity
● 1B-AP-14 - Observe property rights ● 1.3.c Curate Information Decision-Making ● Diversity, equity &
● 1B-AP-15 - Test & debug a program ● 1.4.b Design Constraints inclusion
● 1B-IC-18 - Discuss technologies ● 1.6.a Choose Platforms or ● Safety and security
● 1B-IC-19 - Accessibility & usability Tools
● 1B-IC-20 - Seek diverse perspectives ● 1.6.b Original and Remixed
Works

Created by the Scratch Foundation (scratchfoundation.org) 16

This lesson also fulfills all three of the ISB Indicators of Playful Learning (Choice, Delight, Wonder), developed by
the Pedagogy of Play (PoP) research project at Harvard University.

Tip: If you’d like to translate this guide, click here to make a copy of this Google doc.

Created by the Scratch Foundation (scratchfoundation.org) 17