0% found this document useful (0 votes)

9 views178 pages

Snowflake: New Features

Uploaded by

manishankar

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

9 views178 pages

Snowflake: New Features

Uploaded by

manishankar

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 178

3 -

2
- 20
C

Snowflake
E
8-D
-1
om y
w c
. cop
n o ot-
e -n
New Features i c
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

INTRODUCTIONS
● Name
3 -
2
● Company & Role - 20
E C
● Experience with Snowflake 8-D
-1
om y
○ What do you already know? w c
. cop
n o ot-
i ce -n
○ What do you want to learn?
e rv 3-do
@ s 2
● Tell us something about yourself y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
k ar ©
an
i s h
a n
m

© 2023 Snowflake Computing Inc. All Rights Reserved 2

REMOTE ATTENDEE CONTROLS
● Hover over your Zoom interface to -
2 3
display the control panel at the bottom - 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved 3

CONTROL PANEL

Mute/Unmute 3 -
2
Open the Chat window C-20
DE
8-
- 1Reactions and
om y
w c
. cop
n o ot- nonverbal feedback
i e
c o-n
e v
r 3-d
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved 4

CHAT INTERFACE

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma 1. Use drop-down to choose recipient

2. Click to type your message and

press Enter

© 2023 Snowflake Computing Inc. All Rights Reserved 5

REACTIONS AND NONVERBAL FEEDBACK
Use bottom two rows - to raise your hand,
2 3
click Yes/No, etc.- 2 0 (click again to cancel)
E C
8- D
- 1
om y
Top row emojis
w c
. cop
disappear after 5 seconds n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
k ar ©
an
i s h
a n
m

Click Reactions to display panel

© 2023 Snowflake Computing Inc. All Rights Reserved 6

COURSE OBJECTIVES
Take advantage of a full day to immerse yourself in features introduced to the Snowflake Data Cloud over
the last year. In this fast-paced workshop, you will expand your palette and broaden your working
knowledge of Snowflake. -
2 3
- 2 0
Our expert instructor will accompany you on your trek with lectures, demos, E C and labs designed to give you
8 - D
practical, working knowledge of Snowflake’s recent additions, curated - 1 for experienced Snowflake users.
om y
w c
. cop
n o ot-
Get ahead of the pack! Join us to conquer this snow-capped i ce -nmountain and gain experience with the new
e rv -do
features, so you can determine where they fit in your s use
2 3 cases and solutions.
y @ 20
r t h ke
m a fla
s u ow
● Summarize how the latest features augment, . p a Sn enhance, and accelerate existing Snowflake solutions.
k ar ©
● Explain where these features can an be leveraged in an end-to-end, governed solution.
i s h
a n
m
● Articulate Snowflake’s emerging best practices for the new features.
● Describe the key values of the new features and how they can be applied in creating data applications
and building and managing a robust and efficient Data Cloud implementation.

© 2023 Snowflake Computing Inc. All Rights Reserved 7

COURSE AGENDA
● Class Introduction
● Snowpark Python 3 -
2
- 20
E C
● Snowpark Python for Data Science 8-D
-1
● Streaming Data om y
w c
. cop
n o ot-
● Transformation Pipelines i ce -n
e rv 3-do
@ s 2
● Management and Observability y
h ke 2 0
r t
a fla
m
u ow
● Performance Optimization s
a Sn
. p
k ar ©
● an
Programmability i s h
a n
● Conclusion m

© 2023 Snowflake Computing Inc. All Rights Reserved 8

LAB EXERCISE
10 minutes
Introduction - Prepare for Class -
2 3
● Log in to Snowflake Training Portal - 20
E C
8-D
○ https://training.snowflake.com/login.html -1
om y
● Instructor will provide you access to a Snowflake account w c
. cop like the following:
n o ot-
○ https://app.snowflake.com/us-west-2/xxYY1234rvice do-n
s e 23-
○ User assigned for the course (30 days) thy@ 20
a r ake
○ Do NOT upload any of your own/customer u l
m wfdata
a s no or share such data with the account; this is a
shared environment for class usea r.ponly
© S
nk
● Access student materialsnisha
ma
○ Available for 90 days
○ Need access to workbook for labs
● Download lab files
○ SQL ready to go (/users/ directory)

© 2023 Snowflake Computing Inc. All Rights Reserved 9

LOG IN TO YOUR COURSE DASHBOARD
Go to https://training.snowflake.com Log in with your
and click the "Login or Register" Button -
Snowflake Community
3 credentials
0 2
- 2
E C here
Click to log in -D
Type email address
8
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
k ar ©
an
ish
n
ma
Don't have a Snowflake Community login?
Click "Create Account."

© 2023 Snowflake Computing Inc. All Rights Reserved 10

CLICK TO JOIN SESSION

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
Your course name r t
a fla
m
u fromw
a
will be different s n o
a r .p ©S
whatnisk shown here
h a
ni s
ma
Join here each
day of the course
(link active about
30 minutes ahead)

© 2023 Snowflake Computing Inc. All Rights Reserved 11

NAVIGATE TO YOUR COURSE PAGE

3 -
2
- 20
E C
8-D
Click tile to view details -1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
Your course name r t
a fla
m
u fromw
a
will be different s n o
a r .p ©S
whatnisk shown here
h a
ni s
ma
Join here each
day of the course
(link active about
30 minutes ahead)

© 2023 Snowflake Computing Inc. All Rights Reserved 12

ACCESS THE COURSE MATERIALS

3 - Your course title will be

2
- 20 different from what is shown
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
ha Use links to access materials
nis
ma (file names will vary from
what is shown here)

© 2023 Snowflake Computing Inc. All Rights Reserved 13

ACCESSING CLASS LAB FILES
● The "_labfiles_".zip contains a copy of the
labs for every student 3 -
0 2
2
standard_NF…labfiles…
-
● Only copy your labs from the file E C
8- D
- 1
● Open the lab file and scroll to: om y
w c
. cop
○ /standard_NF_labfiles###/USERS n o ot-
i ce -n
e rv 3-do
● Find the animal assigned to you by your @ s 2
y
h ke 2 0
r t
a fla
instructor m
u ow
s
a Sn
. p
● Copy just that folder to a folder k ar on © your
an
i s h
local machine a n
m
○ For example, if your class login name is
"BULLFROG", copy that one folder to a local
folder on your machine.

© 2023 Snowflake Computing Inc. All Rights Reserved 14

LAUNCH YOUR JUPYTER ENVIRONMENT
● Student Jupyter environments take a few minutes to provision on initial login – do this now!
3 -
0 2
● Use the account, username and password you have been using 2
C- in previous lab work
DE
● Jupyter lab URL: https://labs.snowflakeuniversity.com/ - 1 8-
m o py
. c
w t-co
o
n no
e
ic do-
rv
e 23-
s
@ 20
t y
h ke
r
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved 15

LOG IN TO JUPYTERLAB ENVIRONMENT
Go to https://labs.snowflakeuniversity.com to access the Snowflake JupyterLab environment.
Log in with your instructor-provided account credentials.

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
k ar ©
1 ha
n 3
nis
Type Snowflake ma Once logged in, you
will see the directory
account name,
on the left and the
username, and
Launcher tab on the
password here.
right.
The first time you log in, it may take several
2 minutes for the environment to load.

© 2023 Snowflake Computing Inc. All Rights Reserved 16

LOAD NOTEBOOKS INTO JUPYTERLAB
Click the arrow to upload the Right-click the compressed notebook file and
compressed notebook file. select Extract Archive from the menu.
3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

Click Extract Archive from the menu.

The contents will be extracted to a new folder with a name like
© 2023 Snowflake Computing Inc. All Rights Reserved nf_snowpark_python. 17
DO NOT upload any of

LAB EXERCISE: 1 your own/customer data

or share data of that type
with the account.

LAB: Environment Setup This is a shared

environment for class use
10 minutes -
only.
2 3
- 20
E C
● Log in to your Snowflake account with a URL like the following: 8 -D
-1
om y
○ https://app.snowflake.com/us-west-2/abCD1234
w c
. cop
n o ot-
i ce -n
e rv -do
● Create a worksheet from the "01-lab-environment-setup-compulsory.sql"
@ s 23 file
h y e2 0
○ Work through Lab 1 in Snowsight a t
r ak
u m wfl
a s no
a r.p ©S
● Then log into your class Jupyter nk environment
h a
ni s
○ Complete the steps in m a notebook under the folder “lab1_nf_configuration_compulsory”
the

NOTE: this lab is COMPULSORY as objects used throughout the course are created in it,
so it must be completed first.
© 2023 Snowflake Computing Inc. All Rights Reserved 18
3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
e -n
SNOWPARK PYTHON
i c
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

MODULE AGENDA
● Working with Snowpark
● Developing for Snowflake Overview 3 -
2
- 20
● User Defined Functions (Python) E C
8-D
● User Defined Table Functions (Python) -1
om y
w c
. cop
● Snowpark Stored Procedures (Python) n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
k ar ©
an
i s h
a n
m

© 2023 Snowflake Computing Inc. All Rights Reserved 20

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
ce -n
WORKING WITH SNOWPARK
i
rv 3-do
e
s 2
y @ 2 0
r t h ke
m a fla
s u ow
. p a Sn
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

WHAT IS SNOWPARK? Developer Framework for Snowflake

Snowpark currently- has three major

2 3
components -20
E C
8-D
-1
om y
● Functions let you use system-defined
. c
w t-co p
o
n nSQL o functions or bring your code into
i ce -
e rv 3-do
Snowpark @ s 2 Snowflake via User Defined Functions
y
h ke 2 0
r t
a fla
m
u ow ● The Snowpark API lets you write
s
a Sn
. p
kar © language embedded queries to build
n
isha
n powerful data pipelines
ma
● Stored Procedures let you host and
operationalize procedural code in
Snowflake

© 2023 Snowflake Computing Inc. All Rights Reserved 22

THE SNOWPARK API ● Software library providing programmatic
interaction with Snowflake
○ Does not require Spark or additional libraries
3 -
○ Included as-2part02 of Snowflake service
E C
8 -D
● Generally - 1 Available (GA) for all cloud
o m y
w c
. cop
providers and all Snowflake editions
no not -
e
ic do-
rv
e 23●- Languages support are Java, Scala, and
Snowpark s
@ 20
t y
h ke
r
a fla Python (UDFs, UDTFs, and Stored
m
u ow
s
a Sn
.
r ©p Procedures too)
k a
n
a to
Goal? Open Snowflakeni s h ● Example use cases:
m a
users beyond SQL! ○ Data pipelines, ELT, data preparation and
feature engineering, Spark migration, ML
scoring, ML model training, etc.

© 2023 Snowflake Computing Inc. All Rights Reserved 23

● Translates code into SQL and pushes
SNOWPARK API execution to Snowflake’s elastic
DataFrame API Query performance engine over JDBC
df.collect()
3 -
2
● OptimizedC(Cloud -20 Services layer) and
SNOWPARK
DE
CLIENT API Query Translator executed 1 8- like any other SQL (Compute
-
layer)
om y
. c p w t-co
Snowflake Connector for Python / o
n no
JDBC Driver e
ic do-
rv
e 23-● Provides wrapper methods to run many
s
@ 20
t y
h ke
a r
l a of the standard Snowflake functions
SQL Query
u m wf
a s no
p
kar. ©S ● Includes the session.sql(...)
n
is ha
n method so you can execute any arbitrary
SNOWFLAKE ma
SQL Engine SQL statement

Processing Engine ● Searchable API Reference

documentation catalogs objects, classes,
and functions
© 2023 Snowflake Computing Inc. All Rights Reserved 24
SNOWPARK FLOW

1 2 3-
2 3
Snowflake - 0
2Snowflake
SELECT * FROM …
E C Virtual Storage
Cloud
Services18-
D Warehouse Layer
-
Snowpark Connector / (JDBC or Python) om y SQL Engine

w c
. cop
Snowpark
n o ot- JVM
SDK/Secure
SDK
i ce -n Sandbox
rv 3-do
Row 1, 2, 3, 4…
e
s 2
y @ 2 0
Client
r t h ke
m a fla
s u ow
. p a Sn Server
k ar ©
1
an 2 3
i s h
a n
Snowpark SDK m Cloud Services Virtual Warehouse

• Converts language specific code • Compiles queries • Language specific UDF server
to SQL • Generates execution plan • Runs in secure sandbox
• Sends query to Snowflake • Coordinates with virtual warehouse • Executes SQL
• Serializes response to language • Executes UDFs/UDTFs and Stored
specific objects Procedures
• Can include packages from a stage
© 2023 Snowflake Computing Inc. All Rights Reserved 25
SNOWPARK DATAFRAMES
The core abstraction (class) in Snowpark is the DataFrame
● Optimized data structure conceptually like a table (2D matrix of rows/columns of datatypes)
3 -
0 2
○ Tables, views, and result sets - 2
E C
● Commonly used in programmatic paradigms for working with relational 8- D datasets
- 1
om y
○ Spark, Pandas
w c
. cop
n o ot-
i ce -n
● Includes methods (prebuilt routines or functions) toeroperate v -do on its data called transformations
@ s 23
y 2 0
● Enables powerful and flexible manipulation aof rthstructured
ke data
u m wfla
● Snowpark executes the transformation a s no“lazily”. Transformations aren’t sent to the Snowflake account
a r.p ©S
logic
nk
as SQL until the client code encounters
h a an action. Actions might be:
nis
ma
○ Show me some rows (my_dataframe.show(12))
○ Write the transformed data to a table (my_df.save_as_table(...))
○ Create a Snowflake view from the transformed data (my_df.create_or_replace_view(....))
○ Bring the data back to me, the client (my_df.collect(...))

© 2023 Snowflake Computing Inc. All Rights Reserved 26

SNOWPARK DATAFRAMES
Anatomy of a DataFrame

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved 27

SNOWPARK DEVELOPMENT STEPS
A typical Snowpark developer will carry out the following tasks once or repeated times in a
Snowpark client application or Snowpark Stored Procedure
3 -
0 2
1. Create one or more DataFrame objects that represent tables, - 2views, and/or results of
E C
queries. 8- D
- 1
om y
2. Transform the DataFrame into desired results. ow. t-cop c
e n no
rv ic do-
3. Take action such as writing the results to asetable. 3-
y @ 202
r t h ke
m a fla
s u ow
. p a Sn
k ar ©
an
i s h
a n
m

© 2023 Snowflake Computing Inc. All Rights Reserved 28

"LAZY" SNOWPARK CODE
While the code here is Python, the same could be achieved using the Java or Scala APIs.

3 -
from snowflake.snowpark import Session 2
- 20
E C
8-D
from snowflake.snowpark.functions import *
-1
session = Session.builder.configs(my_config_file).create() o m y
w c
. cop
n o o t -
(session.table("MY_DB.MY_SCHEMA.CUSTOMER) i ce -n Create (wait for an action-lazy)
rv
e 23-d o
.filter(lower(col("CUST_NAME")).like("john%")) s
@ 20
t y
h ke
r
a fla
.select( m
u ow
s
a Sn
. p
col("CUSTKEY") k ar © Transformations (wait for an action - lazy)
an
i s h
,col("CUST_NAME") a n
m
,col("ACCT_BALANCE))
Action (SQL created from create through
.sort(col(ACCT_BALANCE").desc())
all the transformations and then sent to
.create_or_replace_temp_view("THE_BOBS") Snowflake account - eager)
)
© 2023 Snowflake Computing Inc. All Rights Reserved 29
SNOWPARK
UDFS AND UDTFS ● Capabilities extended by creating and using
UDFs and UDTFs
● Serialized from the client to the Snowflake
account at runtime 3 -
0 2
- 2
C
E as any other UDF or UDTF
● Executed-Djust
1 8
● Created - with
o m y
w c
. cop
n o○ oSQL
t - CREATE FUNCTION statements (already
i ce -n
e rv 3-do covered)
@ s 2
y
h ke 2 0
r t
a fla
○ programmatic functions (Java, Scala, or Python)
m
u ow on the client side
s
a Sn
. p
kar © ● This visual shows Python but the same
n
isha
n architecture is used for Java & Scala UDFs
ma
and UDTFs
● Python UDFs and UDTFs get the added
advantage of access to many useful library
packages managed by Anaconda

© 2023 Snowflake Computing Inc. All Rights Reserved 30

SNOWPARK + ANACONDA
Client Environment
Easy access
3 -
conda install geopandas -y -c 2 Curated packages pre-installed
https://repo.anaconda.com/pkgs/snowflake - 20 in Snowflake also available for
E C
8-D
local development
-1
Curated Python
om y
packages
w c
. cop
Anaconda Secure n o ot-
i ce -n No dependency frustration
Repository
e rv 3-do Conda package manager
@ s 2
y
h ke 2 0 integrated in Snowflake
r t
a fla secure sandbox
Snowflake Channel m
u ow
s
a SnPython Secure Sandbox
. p
ar ©
Snowflake
k
an
ish
session.add_packages('geopandas')
Scalable and secure
n
ma Process with secure sandbox
Curated Python integrated into Snowflake
packages
processing engine

All of this with no additional

charges beyond warehouse usage!

© 2023 Snowflake Computing Inc. All Rights Reserved 31

SNOWPARK AND JUPYTER
● The Snowpark documentation provides setup instructions for popular IDEs and CLI tools
○ Java/Scala - Visual Studio, IntelliJ, REPLs, Jupyter notebooks, and more3 -
0 2
- 2
○ Python - General IDE instructions and Jupyter notebooks E C
8- D
1
-exercise in the form of Jupyter
● This course has more materials on Snowpark and a lab m y
c o
. cop
notebooks w
o ot-
n
e -n
i c
e rv 3-do
○ Some of the materials repeat what we have covered @ s 2 here while others introduce new information about
y
h ke 2 0
Snowpark. r t
a fla
m
u ow
s
.pa ©Sn
○ Snowflake Snowpark DataFramearProgramming Training
nk
h a
 1-day Snowpark training n s
i course
ma
● The notebooks for this New Features course are written in Python

© 2023 Snowflake Computing Inc. All Rights Reserved 32

3 -
2
- 20
E C
8-D
-1
om y
DEVELOPING FOR SNOWFLAKE e
s 2
i c
n
e -n
rv 3-do
w c
. cop
o ot-

y @ 0
OVERVIEW t h ke 2
r
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

SNOWFLAKE AND PROGRAMMING
USER-DEFINED FUNCTIONS

Run custom Java, Scala, or Python code inside Snowflake bringing compute
- closer to the data.
2 3
- 20
E C
8-D
● User-defined Functions (UDFs) -1
om y
w c
. cop
○ Invoked in SELECT statements and predicate clauses n o ot-
i ce -n
e rv 3-do
○ Developed in SQL, JavaScript, Java, or Python @ s (GA 2 for all accounts regardless of provider or edition)
h y e2 0
a t
r ak account (No SQL, DDL or DML)
○ No access to or interaction with your Snowflake u m wfl
a s no
 SQL UDFs & UDTFs allow and require a r.p a©single
S SELECT statement. Other types of UDFs & UDTFs do not have
k
n database operations.
h
access to an API that can perform a
ni s
a
m tabular (UDTFs)
○ Can be scalar (UDFs) or

© 2023 Snowflake Computing Inc. All Rights Reserved 34

SNOWFLAKE AND PROGRAMMING
STORED PROCEDURES

Run custom Java, Scala, or Python code inside Snowflake bringing compute
- closer to the data.
2 3
- 20
E C
8-D
● Stored Procedures -1
om y
w c
. cop
○ Invoked using a CALL command n o ot-
i ce -n
e rv -do
○ Can be anonymous (invoke with EXECUTE IMMEDIATE s 2 3 in Snowflake Scripting or a CALL command
y @ 20
that includes WITH in the definition for allarother th kelanguages)
u m wfla
a s no
○ Developed in Snowflake Scripting, a r.pJavaScript,
© S Java, Scala, or Python
n k
h a account for SQL, DDL, or DML
○ Full access to your Snowflake
ni s
ma
○ Snowflake Scripting supports both scalar and tabular Stored Procedures (others do not)

© 2023 Snowflake Computing Inc. All Rights Reserved 35

FUNCTIONS AND STORED PROCEDURES
CUSTOM LOGIC RIGHT NEXT TO YOUR DATA

3 -
2
20
1. Build with your tools OR
inside Snowflake C-
E
8-D
-1
om y
public class MyClass {

w c
. cop
o ot-
public static double myCustomFunctions(String

n
s)

ce -n
{
/*
i
rv 3-do
e
* Let it snow!

s 2
*/

y @ 2 0
h ke
return rval;

}
}

r t
a fla
m
u ow
s
a Sn
. p
k ar ©
n
a into Snowflake stage
i s h
2. Deploy .jar/.py JVM /
n
ma Python

.jar / .py
3. Bind and use in Snowflake

© 2023 Snowflake Computing Inc. All Rights Reserved 36

BENEFITS OF FUNCTIONS AND
STORED PROCEDURES
● Example Scenarios
3 -
2
○ Machine learning model training and scoring - 20
E C
○ Applying custom code 8-D
-1
○ Using third-party libraries and packages om y
w c
. cop
n o ot-
i ce -n
● Benefits e rv 3-do
@ s 2
○ Developers can build functionality into Snowflake y
h ke 2 0
r t
a fla
using popular languages, packages, and
libraries m
u ow
s
a Sn
.
r ©p
○ Users can access this functionality k a as if it were built into Snowflake
an
i s h
a
○ Data never leaves Snowflaken during processing - remaining secure and governed
m
○ Extends the capabilities of Snowflake beyond that of built-in functionality

© 2023 Snowflake Computing Inc. All Rights Reserved 37

3 -
2
- 20
E C
8-D
-1
om y
USER DEFINED FUNCTIONS e
s 2
i c
n
e -n
rv 3-do
w c
. cop
o ot-

y @ 0
(PYTHON) t h ke 2
r
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

PYTHON UDFS
● Develop using pre-compiled/existing library files
3 -
● Develop in-line Python code 2
- 20
E C
● General availability (GA) for all Snowflake accounts regardless 8 -D of cloud provider or
-1
om y
Snowflake edition w c
. cop
n o ot-
i ce -n
● Support for Python 3.8 (currently) e rv 3-do
@ s 2
y
h ke 2 0
r t
● Can use libraries whether internally uor maexternally
w f l a staged
a s no
a r.p ©S
● Can use modules and functions nk in the standard Python packages
h a
ni s
m a
● Can read staged files and write to /tmp on VWH

© 2023 Snowflake Computing Inc. All Rights Reserved 39

PYTHON UDF LIMITATIONS
● Develop using pre-compiled/existing library files
3 -
● Cannot be shared directly or indirectly with Consumers 2
- 20
E C
● Your UDF cannot itself act as a client of any Snowflake account 8 -D
-1
om y
● Maximum size of an output row is 16 MB (subject to w c p
. change)
o
n o ot- c
i ce -n
● When cloning a Python UDF, consult the docs e rv for - d oLimitations on Cloning (extensive)
@ s 23
y
h ke 2 0
● Timeout for a single row operation is m60 r t
a seconds (subject to change)
f l a
s u ow
. p a Sn
o Up from 5 seconds in PuPr
k ar ©
an
i s h
● UDFs and modules brought a n in through stages must be platform-independent and must not
m
contain native extensions
● May not start or fork processes, read/write to file systems other than staged files and /tmp
on VWH, access the network directly or indirectly, open a Socket, or open URL connections

© 2023 Snowflake Computing Inc. All Rights Reserved 40

PYTHON UDF LIMITATIONS (CONTINUED)
● Refer to the docs for an extensive list of
mappings for SQL / Python data types for parameters and returns23 -
- 20
E C
8-D 1
● Snowflake uses the Python zipimport module to import -
om y Python code from stages.
w c
. cop
See the zipimport documentation for added limitations. n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
k ar ©
an
i s h
a n
m

© 2023 Snowflake Computing Inc. All Rights Reserved 41

PYTHON UDFs
● Bring custom Python code to Snowflake
SNOWFLAKE as User Defined Functions (UDFs)
3 -
● Code is serialized 0 2 and pushed down to
- 2
E C
Built-in
Anaconda Packages run in1a8-Dsecure sandboxed environment
Python Secure -
om y
● Seamlessly access third-party packages
Sandbox
. c o p
n ow ot-c
i ce -nwith Anaconda integration inside
e rv 3-do
@ s 2 Snowflake (i.e., packages = ('numpy') )
h y e2 0
SQL Engine
a t
r ak
u m wfl ● Create UDFs with in-line Python code or
a s no
& more

Processing Engine
a r.p ©S Python code uploaded to a stage
nk
h a
nis
ma

© 2023 Snowflake Computing Inc. All Rights Reserved 42

IN-LINE PYTHON UDF

CREATE OR REPLACE FUNCTION my_udf() 3 -

0 2
RETURNS STRING - 2
E C
LANGUAGE PYTHON
8 - D
RUNTIME_VERSION = 3.8 - 1
Define language om y
IMPORTS = ('@my_stage/file.txt')
w c
. cop HANDLER defines
specifications
HANDLER = 'compute' n o ot- the Python function
i ce -n
AS e rv 3-do called
@ s 2
$$ y
h ke 2 0
import sys r t
a fla
m
u ow
Import non-code files a s
IMPORT_DIRECTORY_NAME n = "snowflake_import_directory"
p S
r. = ©sys._xoptions[IMPORT_DIRECTORY_NAME]
into a Python UDF k
import_dira
an
from a stage is h
n
ma compute():
def
with open(import_dir + 'file.txt', 'r') as file:
return file.read()
$$;

© 2023 Snowflake Computing Inc. All Rights Reserved 43

3 -
2
- 20
E C
8-D
USER DEFINED n
w c o
. cop
o ot-
-1
m y

ce -n
TABLE FUNCTIONS
i
rv 3-do
e
s 2
y @ 2 0
r t h ke
m a fla
s u ow
(PYTHON) nisha
nk
.
ar ©
p a Sn

© 2023 Snowflake Computing Inc. All Rights Reserved

PYTHON CLASS FOR UDTFs
The primary component of the UDTF is the handler class.
3 -
0 2
- 2
"Callback" or "lifecycle" methods
Handler Class E C
8- D
- 1
om y
w c
. cop
__init__() – initialization routine n o ot-
i ce -n OUTPUT
rv
e 23-d o
@ s TUPLES
h y e2 0
process() method – row processing logic
a t
r ak
u m wfl
a s no
a r.p ©S
nk
end_partition() method –hafinalization routine
nis
ma

© 2023 Snowflake Computing Inc. All Rights Reserved 45

IN-LINE PYTHON UDTF HANDLER defines
the Python class
CREATE FUNCTION return_two_copies(v VARCHAR)
used
RETURNS TABLE (output_value VARCHAR)
3 -
2
20
LANGUAGE PYTHON
RUNTIME_VERSION = '3.8' C-
E
HANDLER='TestClass'
8-D
AS -1
m
$$ co .
w t-co p y
n o o
ce
i do -n
class TestClass:
rv
e 23-
from some_library import some_datatype s
@ 20
t y
h ke
r
a fla In-line Python code
def __init__(self):
m
u ow
pass s
a Sn No IDE needed!
. p
HANDLER class k ar ©
an
def process(self,input_value:str):
i s h
a n
tups = <process and create tuples>
# or m
yield tuples
return tups

def end_partition(self):
pass
$$;

© 2023 Snowflake Computing Inc. All Rights Reserved 46

3 -
2
- 20
E C
8-D
SNOWPARK n
w c o
. cop
o ot-
-1
m y

ce -n
STORED PROCEDURES
i
rv 3-do
e
s 2
y @ 2 0
r t h ke
m a fla
s u ow
(PYTHON) nisha
nk
.
ar ©
p a Sn

© 2023 Snowflake Computing Inc. All Rights Reserved

STORED PROCEDURES USING SNOWPARK
● Build code that can be called repeatedly within Snowflake
● Supported languages include Java, Scala, and Python 3 -
2
- 20
○ Stored Procedures developed in JavaScript or Snowflake Scripting E C not use Snowpark
do
8 -D
-1
● Host Snowpark pipelines within Snowflake c om y
w . cop
n o ot-
● Schedule execution with Snowflake tasks i ce -n
e rv 3-do
@ s 2
● Reduce the number of systems involved y
- 2
keep
h ke 0 everything self-contained in Snowflake
r t
a fla
m
u ow
s
a Sn
. p
k ar ©
an
○ Makesi s h use of the integrated Snowpark library
a n
○ mWhen a stored procedure is called, Snowflake automatically
creates a Session object and passes it to the stored procedure

Snowpark
© 2023 Snowflake Computing Inc. All Rights Reserved 48
● Create an in-line stored procedure or a stored
PYTHON procedure from a .py file or .zip file
STORED PROCEDURES ○ Upload file(s) to a stage
● Requires the latest Snowpark Python library,
CREATE PROCEDURE myProc (from_table STRING,
to_table which installs additional3 - third-party
STRING,
0 2
count INT) dependencies - 2
E C
RETURNS STRING D
8- data in Snowflake, use the Snowpark
LANGUAGE PYTHON ● To access- 1
RUNTIME_VERSION = '3.8'
om y
w c
. copAPIs (session object)
library
PACKAGES = ('snowflake-snowpark-python')
n o ot-
HANDLER = 'run' i ce -○ n No creating new Session objects
e v
r 3-d o
AS
@ s 2
$$ y
h ke 2 0 ● Specify third-party Anaconda packages to install
def run(session, from_table, to_table, count): r t
a fla
m
u ow
(session.table(from_table)
a s n when you create Python stored procedures
.limit(count).
a r.p ©S
nk ● Snowpark API Restrictions:
write
h a
.save_as_table(to_table)
nis
) ma ○ session.file.get(...) and
return "SUCCESS"
$$; session.file.put(...)
only work between /tmp and the stage
call MYPROC ('BIRTH_AND_CENSUS',
'BIRTH_AND_CENSUS_COPY', ○ Cannot use:
2);
session.sql("GET…") or
© 2023 Snowflake Computing Inc. All Rights Reserved
session.sql("PUT...") 49
CALL WITH ANONYMOUS
PROCEDURE (preview) ● Stored Procedure option for JavaScript,
Java, Scala, and Python
WITH MY_ANON_SP AS PROCEDURE(ARG1 STRING, ARG2
STRING) ○ Snowflake Scripting has anonymous blocks
3 -
RETURNS STRING
0 2
● More narrow 2
- scope than TEMP procedures
LANGUAGE PYTHON
E C
-- Java, Scala, Python, or JavaScript D
8-procedures exist for the life of the
PACKAGES = ('snowflake-snowpark-python')
○ TEMP - 1
om y
HANDLER = 'my_handler_func' . c
w t-c o p
Snowflake session
n o o
AS
ce -n
i d●o Scope is the life of a single statement
$$ rv
e 23-
s
@ 20
def my_handler_func(session, arg1,arg2):
t y
h ke ● No requirement for role to have CREATE
user = (session r
a fla
m
u ow
.sql("SELECT CURRENT_USER()") a s n PROCEDURE privilege on the schema
p
r. © S
.collect()[0][0] k a
an ○ The procedure runs with the creator/caller's rights
) is h
n
ma
return f"{arg1} {arg2} {user}"
● Procedure body has access to the Snowflake
$$
CALL MY_ANON_SP('HELLO','WORLD'); session, session variables, and
parameters
MY_ANON_SP gone after
● Same API restrictions as named procedures
statement executes
All one single statement
© 2023 Snowflake Computing Inc. All Rights Reserved 50
LAUNCH YOUR JUPYTER ENVIRONMENT
● Student Jupyter environments take a few minutes to provision on initial login – do this now!
3 -
0 2
● Use the account, username and password you have been using 2
C- in previous lab work
DE
● Jupyter lab URL: https://labs.snowflakeuniversity.com/ - 1 8-
m o py
. c
w t-co
o
n no
e
ic do-
rv
e 23-
s
@ 20
t y
h ke
r
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved 51

LAB EXERCISE: 2
LAB: Snowpark Python
30 minutes -
2 3
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved 52

3 -
2
- 20
E C
8-D
-1
om y
SNOWPARK PYTHON e
s 2
i c
n
e -n
rv 3-do
w c
. cop
o ot-

@ 0
FOR DATA SCIENCE t y
h ke 2
r
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

MODULE AGENDA
● Snowpark Optimized Warehouses
● Reading Files With a Python Function or Procedure 3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved 54

MACHINE LEARNING WITH SNOWPARK
Prepare & Transform Train Inference
3 -
2
20
model

C- predictions
data lake DATAFRAME: training data SPROC(): E UDF(
________ ________ 8-D
-1
Max_batch_size = n
________ ________
om y ________
________ ________
w c
. cop Stage ________
________ ________
n o ot- ________
i ce -n
e rv 3-do ):

@ s 2
y
hMedium 2 0
r t k e Snowpark-
a fla optimized
multi-node
standard
m
u ow warehouse
inference data standard
warehouse s
a Sn warehouse
. p
k ar ©
n
isha
n
ma

ML Training Tip ML Inference Tip

Use UDTFs to train multiple models Speed up inference using Vectorized UDFs
concurrently using M-4XL Snowpark-optimized to process rows in batches & cachetools to
warehouse cache model load from stage

© 2023 Snowflake Computing Inc. All Rights Reserved 55

MACHINE LEARNING WITH SNOWPARK
Rapid deployment with pre-built models

3-
Prepare & Transform Pre-Built Model Inference
2
- 20
E C predictions
8-D
data lake DATAFRAME: UDF(
________
-1 Max_batch_size = n
________
om y ________
________
w c
. cop ________
n o ot-
________
i ce -n ________
Stage OR rv
e - d o
s 23
):
y @ 2 0
r t h ke
m a fla multi-node
standard
s u ow standard
warehouse
. p a Sn
ar ©
warehouse

nk
isha
n inference data
ma

ML Inference Tip
Speed up inference using Vectorized UDFs
to process rows in batches & cachetools to
cache model load from stage

© 2023 Snowflake Computing Inc. All Rights Reserved 56

3 -
2
- 20
E C
8-D
-1
om y
SNOWPARK OPTIMIZED e
s 2
i c
n
e -n
rv 3-do
w c
. cop
o ot-

y @ 0
WAREHOUSES t h ke 2
r
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

SNOWPARK-OPTIMIZED WAREHOUSES

-
16X 2 3
20

memory per
10X
8-D
E C-

- 1
m y cache
. c o p
node w
o ot- co
n
e -n
i c
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
Effortless execution of memory- m
u ow Accelerate subsequent run
s
intensive operations ar.pa ©Sn execution
nk
ha
Bring ML training,isin-memory analytics Provide speedup when cached
a n
m
(e.g. correlations) artifacts (Python packages,
or other memory-intensive operations intermediate results, JARs, etc) are
inside Snowflake’s secure Python/Java reused on subsequent runs
sandbox.

Plus all the benefits of standard virtual warehouses

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
XS S M L s
XL r.pa 2XL n 3XL 4XL 5XL 6XL
a © S
nk
h a
nis
ma
● Snowpark-optimized warehouses use nodes that have 4X the CPUs and 16X the
memory as a node in a standard warehouse

© 2023 Snowflake Computing Inc. All Rights Reserved 59

3 -
2
- 20
E C
8-D
READING FILES n
w c o
. cop
o ot-
-1
m y

ce -n
WITH A PYTHON
i
rv 3-do
e
s 2
y @ 2 0
r t h ke
m a fla
s u ow
FUNCTION OR PROCEDURE nisha
nk
.
ar ©
p a Sn

© 2023 Snowflake Computing Inc. All Rights Reserved

**PRIVATE *PUBLIC GA

SNOWPARK PYTHON DYNAMIC FILE ACCESS

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
Transformed Data

n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r
a at
Scoped URL um wfl
a s no Python Code
Unstructured Data
a r.p ©S (UDF, UDTF, Stored Procedure)
nk
h a
ni s
ma SnowflakeFile
Machine Learning
.open(file_path) Results

# custom code
# ...

© 2023 Snowflake Computing Inc. All Rights Reserved 61

**PRIVATE *PUBLIC GA

SNOWPARK PYTHON DYNAMIC FILE ACCESS

Import the
from snowflake.snowpark.files import SnowflakeFile
3 - SnowflakeFile class
from snowflake.snowpark.functions import udtf 0 2 from the Snowpark API.
- 2
C
from snowflake.snowpark.types import StructField, StructType,
E
- D
StringType, IntegerType
8
- 1
om y
Register code my_schema = StructType([
w . c o p
as a UDTF, StructField("field_a", StringType()),
n o ot- c Include the scoped URL
i e
c o-n
UDF, or Stored StructField("field_b", IntegerType())
e rv - d as a function parameter.
Procedure. ])
@ s 23
y
h ke 2 0
r t
@udtf(name = "read_files",ais_permanent
m f l a = True, stage_location = "my_stage",
su now
replace = True, aoutput_schema = my_schema)
a
class read_files_class: r.p ©S
nk
h a
def parse_file(self, file_path):
n isSnowflakeFile.open(file_path) as f:
ma # your custom code here
with

# ...

Pass file URL into open() function call

along with any other parameters needed
for your code. (SnowflakeFile extends
IOBase.)
© 2023 Snowflake Computing Inc. All Rights Reserved 62
**PRIVATE *PUBLIC GA

SNOWPARK FOR UNSTRUCTURED DATA

Use Cases for Data Engineers and Data Scientists

3 -
Image 2
Optical0Character
- 2
C
E Recognition
Classification D
8-
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

Natural Language Audio

Processing Transcription

© 2023 Snowflake Computing Inc. All Rights Reserved 63

LAB EXERCISE: 3
LAB: Snowpark Python for Data Science
15 minutes -
2 3
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved 64

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
e -n
STREAMING DATA
i c
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

MODULE AGENDA
● Snowflake Data Ingestion Options
● Snowpipe Streaming Overview 3 -
2
- 20
● Snowpipe Streaming Detail E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved 66

3 -
2
- 20
E C
8-D
SNOWFLAKE n
w c o
. cop
o ot-
-1
m y

ce -n
DATA INGESTION
i
rv 3-do
e
s 2
y @ 2 0
r t h ke
m a fla
s u ow
OPTIONS
nisha
nk
.
ar ©
p a Sn

© 2023 Snowflake Computing Inc. All Rights Reserved

SNOWFLAKE DATA LOADING APPROACHES
BULK CONTINUOUS
(using COPY) -
(using
3
Snowpipe)
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
k ar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved 68

SNOWPIPE AUTO INGEST
Applications
Snowpipe Service023 -
- 2
E C Snowflake
8-D Database
-1
om y
w c
. cop
External n o ot-
i ce -n
Cloud notification e rv 3-do
@ s 2
Storage y
h ke 2 0
r t
a fla Server-less
m
u ow
s
a Sn
File.pdata
Loader
k ar ©
an
i s h
a n
m

Firehose

© 2023 Snowflake Computing Inc. All Rights Reserved

69
KAFKA CONNECTOR FOR SNOWFLAKE

CONFIG
3 -
2
- 20
E C
8-D
JSON or -1
om y
Avro
w c
. cop
REST
Records
n o ot-
Snowflake
i ce -n
Connector
e rv 3-do
@ s 2 File Data
y
h ke 2 0
r t
a Internal
m f l a Snowpipe
u
s now Stage
p a
ar. ©S
Service
Kafka Kafkaank Snowflake
i s h
Cluster n
Connect
a
Database
mCluster

© 2023 Snowflake Computing Inc. All Rights Reserved 70

3 -
2
- 20
E C
8-D
-1
om y
SNOWPIPE STREAMING e
s 2
i c
n
e -n
rv 3-do
w c
. cop
o ot-

y @ 0
OVERVIEW t h ke 2
r
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

SNOWPIPE STREAMING API
The SDK makes REST
Rows, not files API calls to Snowflake
– serverless compute
3 -
2
- 20
E C
1 2
8-D 3
-1
om y
w c
. cop { REST API }
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
Create custom m
u ow
JAVA
a s n Insert-only API calls No staged files or
r .
applications p ©S
using the
k a
Snowflake Ingest SDK
provided PIPE needed
an
s h
a ni
m
● Enables automatic, continuous loading of data streams into Snowflake
● Ordered ingestion and high scalability
● Low seconds latency
● CDC, events, IoT, SaaS apps, security and log analytics, etc.
© 2023 Snowflake Computing Inc. All Rights Reserved 72
SNOWPIPE STREAMING API

3 -
2
- 20
E C API }
{ REST
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar © { REST API }
n
isha
n
ma

Snowflake provided
No staged files or
JAR incorporating the
PIPE needed
Ingest SDK

© 2023 Snowflake Computing Inc. All Rights Reserved 73

SNOWFLAKE CONTINUOUS INGESTION LANDSCAPE

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved 74

SNOWFLAKE DATA INGESTION
No More Boundaries between Streaming & Batch Pipelines

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y 2 0
Simplified Workmawith rth akeStreaming Native to Snowflake’s
u w f l
Architecture a s Data
n o Easily Data Cloud
r . p ©S
ka
n
No more separation of isha Serverless ingestion and Expanding ecosystem in the Data
n
streaming and batch data ma automated transformation with Cloud with consistent and strong
pipelines easy joins security & governance

© 2023 Snowflake Computing Inc. All Rights Reserved 75

3 -
2
- 20
E C
8-D
-1
om y
SNOWPIPE STREAMING e
s 2
i c
n
e -n
rv 3-do
w c
. cop
o ot-

y @ 0
DETAIL t h ke 2
r
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

CLIENT CHANNEL TABLE MAPPING
● The API ingests rows via one or more channels
○ a logical, named streaming connection to Snowflake for data loading 23 -
- 20
● A single channel maps to exactly one table in Snowflake -DEC
1 8
○ Multiple channels can point to the same table -
o m y
w c
. cop
● Ordering of rows and offset tokens preserved within, n o obutt - not across, channels
i ce -n
e rv 3-do
● Inactive channels deleted automatically after @ s 30 2 days
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
k ar ©
an
i s h
a n
m

© 2023 Snowflake Computing Inc. All Rights Reserved 77

OFFSET TOKENS
● A STRING included in a client API insert call to track ingestion progress on a per-channel
basis 3 -
2
- 20
○ Initialized to NULL on channel creation E C
8 -D
○ Updated (asynchronously) when rows with provided offset token - 1 are committed to Snowflake
om y
w c p
. cooffset
○ Snowflake provides a method to obtain the latest committed o ot- token
n
e -n
i c
v -do
● Function as a bookmark/placeholder to track e ringestion
3 progress
s
@ 202
t y
h ke
● Difficult exactly-once-delivery can be rmanaged
a fla by leveraging offset tokens
m
u ow
s
a Sn etc., must be coded in your application logic
. p
○ Handling restarts, ensuring de-duplication,
k ar ©
an
i s h
a n
m

InsertValidationResponse response = channel1.insertRow(row, String.valueOf(appRowNumber));

…
offsetTokenFromSnowflake = channel1.getLatestCommittedOffsetToken();

SAMPLE CODE

© 2023 Snowflake Computing Inc. All Rights Reserved 78

SNOWPIPE STREAMING MIXED TABLE FORMATS
● Streamed data written to a table is initially stored in a temporary intermediate file format
○ Metadata extracted and data query-ready
3 -
2
20
● An automated background process migrates data to optimizedC-native files
DE
● A mixed table contains intermediary and native files 1 8-
-
o m y
○ Transparent to Snowflake users – seamless operation w. cop c
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
Temporary
.
r ©p intermediate file
k a
an MIXED TABLE format
i s h
a n
m MIGRATION
FLOW

Snowflake
native FDN file
format

© 2023 Snowflake Computing Inc. All Rights Reserved 79

● Direct Data Streaming
SNOWPIPE STREAMING ○ Ingestion to tables over HTTPs - no files
staging or pipes
● Exactly-Once and
- Per-Channel Ordering
2 3
0
○ Get data Cin-2once and process in order
DE
-
● Low -Latency18
om y
w○ c
. Datao p typically queryable seconds after
n o ot- c
i ce -n commit
e rv 3-do
@ s 2
h y 2 0 ● High Throughput
a t
r ak e
u m wfl ● Low Overhead
a s no
a r.p ©S
nk ○ Minimal configuration and setup needed
h a
ni s
ma ● Cost Effective
○ For both flood and trickle cases
● Integrated Error Handling Capabilities
○ OnErrorOption.CONTINUE (...)

© 2023 Snowflake Computing Inc. All Rights Reserved 80

● Open Source
KAFKA CONNECTOR ○ Using new APIs starting from version 1.9.0
WITH SNOWPIPE STREAMING ● Exactly-Once Semantics
○ ensures the delivery 3 - of Kafka messages
0 2
- 2
without duplication or data loss with two
E C
offsets8- D
- 1
om y
■ Consumer offset (Kafka)
. c
w t-co p
o
n no ■ Offset token (Snowflake)
e
ic do-
rv
e 23-● Error Handling
s
@ 20
t y
h ke
r
a fla ○ Sent to Dead Letter Queue (Kafka topic)
u m
a s now
a r.p ©S ● Lower Cost
nk
a
nish ● Low Latency
ma
● Schematization
○ Support for schema detection and evolution
● Simple Configuration
○ snowflake.ingestion.method=SNOWPIPE_STREAMING

© 2023 Snowflake Computing Inc. All Rights Reserved 81

SNOWPIPE STREAMING COSTS
Snowpipe Streaming costs comprise:
1. Per-second Client Ingestion Time 3 -
0 2
2
○ Snowflake managed compute (cloud services) charged at an hourlyC-rate of 0.01 Snowflake Credits per
DE
client instance 1 8-
-
om y
 Example: 30 days * 24 hours * 1 client * 0.01 credits = 7.2 credits
w c
. cop
n o ot-
○ Detailed in SNOWFLAKE.ACCOUNT_USAGE.SNOWPIPE_STREAMING_CLIENT_HISTORY
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
2. Migration Compute Costs m
u ow
s
a Sn
. p
○ Snowflake managed compute (background
k ar © service) charged in credits per hour for work performed
a n
i
 File migration may not always s h occur or may be pre-empted by clustering or other DML operations
a n
m
○ Detailed in SNOWFLAKE.ACCOUNT_USAGE.SNOWPIPE_STREAMING_FILE_MIGRATION_HISTORY

* It is difficult to estimate Snowpipe Streaming costs due to multiple factors such as the size of records, data
types, etc. Testing to establish baselines is recommended.

© 2023 Snowflake Computing Inc. All Rights Reserved 82

SNOWPIPE STREAMING CLIENT HISTORY
SNOWFLAKE.ACCOUNT_USAGE VIEW

● Displays the amount of data loaded and timestamp of the Snowpipe- Streaming client calls
2 3
● Latency for the view may be up to 120 minutes - 20
C E
8- D
● Note that there can be multiple events per second listed- 1
om y
○ Credits are consumed only by the actual time spent, not w c
.bycthe
o p number of events
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
k ar ©
an
i s h
a n
m

© 2023 Snowflake Computing Inc. All Rights Reserved 83

SNOWPIPE STREAMING FILE MIGRATION HISTORY
SNOWFLAKE.ACCOUNT_USAGE VIEW

● Displays the number of rows and bytes migrated and credits used for - migration from
intermediary file format => Snowflake’s native FDN file format -2023
E C
● Latency for the view may be up to 12 hours 8 -D
1 -
● File migration may not always occur or may be pre-empted om y
w c
. cop by clustering or other DML
n o ot-
operations i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
k ar ©
an
i s h
a n
m

© 2023 Snowflake Computing Inc. All Rights Reserved 84

SNOWPIPE STREAMING EMERGING PRACTICES
COST-EFFECTIVE PERFORMANCE

● Use fewer clients that write more data per second via the Ingest API-
2 3
○ insertRows will be more performant and cost-effective than calling -insertRow 20 multiple times as less
E C
time spent on locks 8 -D
-1
● Anti-pattern: many clients sending small amounts of.cdata om py
o w t-co
e n no
rv ic do-
● Pass values for the TIME, DATE and TIMESTAMP s e 23- columns as a supported type from the
y @ 20
r t h ke
java.time package m a fla
s u ow
. p a Sn
k ar ©
● Use OnErrorOption.CONTINUE an and manually check the return value from insertRows
i s h
a n
for potential ingestion errors
m for better performance

InsertValidationResponse response = channel1.insertRow(row, String.valueOf(appRowNumber));

if (response.hasErrors()) {
// Take some action here, for example
throw response.getInsertErrors().get(0).getException();
}
© 2023 Snowflake Computing Inc. All Rights Reserved SAMPLE CODE 85
LAB EXERCISE: 4
LAB: Streaming Data
30 minutes -
2 3
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved 86

3 -
2
- 20
E C
8-D
-1
om y
TRANSFORMATION e
s 2
i c
n
e -n
rv 3-do
w c
. cop
o ot-

@ 0
PIPELINES t y
h ke 2
r
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

MODULE AGENDA
● Streams on Views
3 -
● Dynamic Tables 2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved 88

DATA ZONES IN SNOWFLAKE
Presentation Delivery

Inputs RAW CONFORMED MODELED Outputs

3 -
0 2
Integration
- 2
E C
8- D
-1
om y Aggregated Data
w c
. cop data
n o ot- scientist
Snowpipe i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
Batch copy
r t
a fla
m
Ingested su ow BI
Snowflake
a n Enriched/ Modeled
r.p ©S
Ingest tool transformation
Stage
k a
data cleansed tables
an data etc.
ish
n
ma
Data
analyst
Modeled
views
(secure)

© 2023 Snowflake Computing Inc. All Rights Reserved 89

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
ce -n
STREAMS ON VIEWS
i
rv 3-do
e
s 2
y @ 2 0
r t h ke
m a fla
s u ow
. p a Sn
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

STREAMS ON VIEWS
● Streams contain no data; they reflect changes in the data

3 -
● Track change data capture (CDC) records for views 2
- 20
E C
○ Includes secure views (not materialized views) - useful for data8sharing -D scenarios
-1
○ Build streams on views with joins across multiple tables com py
w . co
n o ot-
i ce -n
e rv 3-do
● Change tracking must be enabled in the underlying @ s 2 tables
h y e2 0
a t
r ak
○ Creating a stream adds hidden metadata u f
m columnsl to the source table(s)
a s now
a r.p ©S
nk
● Currently, streams are limited h a to views that satisfy the following requirements:
ni s
ma
○ All of the underlying tables must be native tables
○ GROUP BY, QUALIFY, LIMIT not supported
○ The view may only apply the following operations:
 Projections, filters, inner or cross joins and UNION ALL

© 2023 Snowflake Computing Inc. All Rights Reserved

91
STREAMS ON VIEWS EXAMPLE
CREATE VIEW bird_sightings_vw AS
SELECT b.id AS id,
b.common AS common_name, 3 -
2
b.class AS classification, - 20
BIRDS E C
s.d AS date,
8-D
s.loc AS location, -1
om y BIRD_SIGHTINGS_VW
s.c AS count
w c
. cop
n o ot-
FROM birds b
i ce -n BIRD_SIGHTINGS_S
INNER JOIN sightings s ON b.id = s.b_id; e rv 3-do SIGHTINGS STREAM

@ s 2
y
h ke 2 0
r t
CREATE STREAM bird_sightings_s ma fla
s u ow
ON VIEW bird_sightings_vw; . p a Sn
k ar ©
an
i s h
n SIGHTINGS>
<insert row into BIRDS and
a
m
select * from bird_sightings_s;

© 2023 Snowflake Computing Inc. All Rights Reserved 92

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
ce -n
DYNAMIC TABLES
i
rv 3-do
e
s 2
y @ 2 0
r t h ke
m a fla
s u ow
. p a Sn
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

DYNAMIC TABLES
NEW SNOWFLAKE TABLE TYPE
● Dynamic tables simplify data pipeline
3 -
transformation 2 0 2 using SQL declarative
C-
statementsDE
- 8
- 1
o m y
c
. cop
●owAutomatic refresh process with no need
n no t -
e
ic do-for streams and tasks or external
rv
e 23-
s
@ 20
t h y e
scheduler setup
a r ak
u m wfl
a s no
p
kar. ©S ● Eliminates the need for complex merge
n
isha statements, reducing moving parts and
n
ma simplifying ongoing support and
operations

© 2023 Snowflake Computing Inc. All Rights Reserved 94

DYNAMIC TABLES
● Declarative data pipelines ● User-defined freshness
○ Specify the end goal and Snowflake will ○ Controlled by a target - lag for each table, for
2 3
take care of the implementation sake of reduced - 20 cost and improved
○ Dynamic tables manage the scheduling performance E C
8 -D
and orchestration ○ Data is - 1current up to the target lag time
om y
w c
. cop
n o ot-
● SQL Support ●i ceSnapshot
-n isolation within a single account
rv
e 23-○d o
○ Use any core SQL syntax to define s
@ 20 All dynamic tables in a DAG are refreshed
t y
h ke
transformations, including inner joins, r
a fla consistently from aligned snapshots
m
u ow
unions, and aggregations s
a Sn
. p
k ar ©
an
i s h
● Automatic incremental a nrefreshes
m
○ Refresh only what's changed, even for
complex queries, automatically (when
new data arrives)
 INCREMENTAL and FULL REFRESH modes
 The SQL used determines the refresh mode

© 2023 Snowflake Computing Inc. All Rights Reserved 95

DYNAMIC TABLE ADVANTAGES
● Incremental materialization
3 -
○ Immediate results 2
- 20
E C
○ Stores result of query 8-D
-1
o m y
w c
. cop
● Avoid the complexity of managing scheduling and n o t -
oorchestration for complex pipelines
i e
c o-n
e v
r 3-d
@ s 2
○ Pipelines using JOINS can create complex dependencies y
h ke 2 0
r t
a fla
m
u otow create a DAG pipeline of 100s of tables
○ Dynamic tables can be chained together s
a Sn
. p
k ar ©
an
i s
● Balance cost and data freshness h
a n with lag, to meet business objectives
m
○ Expect data freshness within the defined lag period

● Improve the productivity of data engineers

© 2023 Snowflake Computing Inc. All Rights Reserved

96
DYNAMIC TABLE DDL

REFRESH 3 -
2
SCHEDULE - 20
E C
8-D
-1
om y
w c
. cop SQL
n o ot-
COMPUTE
i ce -n DETERMINES
RESOURCE e rv 3-do REFRESH
@ s 2
y
h ke 2 0 MODE
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

DECLARATIVE
SQL
DEFINITION

© 2023 Snowflake Computing Inc. All Rights Reserved

97
MONITOR DYNAMIC TABLES
● SHOW DYNAMIC TABLES
3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
● The REFRESH_MODE is determined by the SQL e rv 3-used
d o in the dynamic table
@ s 2
y
h ke 2 0
○ INCREMENTAL r t
a fla
m
uan oincremental
w
 The automated refresh process performs a s n refresh, if possible
a r.p ©S
○ FULL_REFRESH is required for nkthe following (may change over time)
h a
ni s
ma
 Lateral joins
 Queries with an ORDER BY clause
 Subqueries outside of FROM clauses (e.g., WHERE EXISTS)

© 2023 Snowflake Computing Inc. All Rights Reserved

98
MONITOR DYNAMIC TABLES
● DYNAMIC_TABLE_REFRESH_HISTORY()
3 -
○ INFORMATION_SCHEMA table function 2
- 20
○ Returns information about each refresh (completed and running)-DofECdynamic tables
1 8
-
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
k ar ©
an
i s h
a n
m

© 2023 Snowflake Computing Inc. All Rights Reserved

99
WORKING WITH DYNAMIC TABLES
● Run costs = storage + cloud services compute + virtual warehouse compute

3 -
● Some query constructs are not currently supported in dynamic -tables, e.g. 2
20
E C
○ Most non-deterministic functions 8-D
-1
○ UDFs o m y
w c
. cop
n o o t -
○ Shared tables i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
● Setting the lag time in DAGs r t
a fla
m
u ow
s
a Sn(a target lag)
o Specify a target “freshness” for the . pdata
k ar ©
an
o The automated process sets i s hup refreshes so that the data in the dynamic table is kept up to date within
a n
this target m
o No VWH compute is consumed if there are no changes in upstream objects (tables, DTs)

● To suspend the refresh of a dynamic table, use the ALTER command

○ ALTER DYNAMIC TABLE conformed.product_sales_dt SUSPEND;
© 2023 Snowflake Computing Inc. All Rights Reserved
100
DYNAMIC TABLE OBSERVABILITY

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

101
DYNAMIC TABLE OBSERVABILITY

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot- SCREEN REFRESH
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

102
LAB EXERCISE: 5
LAB: Transformation Pipelines
30 minutes -
2 3
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved 103

3 -
2
- 20
E C
8-D
-1
om y
MANAGEMENT AND e
s 2
i c
n
e -n
rv 3-do
w c
. cop
o ot-

@ 0
OBSERVABILITY t y
h ke 2
r
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

MODULE AGENDA
● Observability on Snowflake
3 -
● Outbound (push) Notifications 2
- 20
E C
● Snowflake Alerts 8-D
-1
● Observability within Snowsight om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
k ar ©
an
i s h
a n
m

© 2023 Snowflake Computing Inc. All Rights Reserved 105

3 -
2
- 20
E C
8-D
-1
om y
OBSERVABILITY ON e
s 2
i c
n
e -n
rv 3-do
w c
. cop
o ot-

y @ 0
SNOWFLAKE t h ke 2
r
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

OBSERVABILITY ON SNOWFLAKE
Industry defines Observability as “the ability to gain insights into the internal state of a system
by analyzing its external outputs, such as logs, metrics, and traces”.
3 -
2
- 20
What features and capabilities does Snowflake currently offer Dto ECsupport Observability?
1 8-
-
● Outbound notifications om y
w c
. cop
n o ot-
○ Snowpipe and Task error notifications i ce -n
e rv 3-do
○ SYSTEM$SEND_EMAIL stored procedure @ s 2
y
h ke 2 0
r t
a fla
○ Resource Monitor NOTIFY_USERS email m option
s u ow
. p a Sn
● Snowflake Alerts k ar ©
an
● No-code pipeline observability i s h
within Snowsight
n
ma
○ Individual Task Observability
○ Task History
○ Copy History dashboard
○ External Table and Directory Table Auto-Refresh Observability

© 2023 Snowflake Computing Inc. All Rights Reserved 107

3 -
2
- 20
E C
8-D
-1
om y
OUTBOUND e
s 2
i c
n
e -n
rv 3-do
w c
. cop
o ot-

y @ 0
NOTIFICATIONS t h ke 2
r
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

OUTBOUND NOTIFICATION INTEGRATIONS

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
● An integration is a Snowflake object that provides an i ce -n
interface between Snowflake and third-party
e rv -d o
s 3
services y @ 202
r t h ke
m a fla
s u channel
o w
● Outbound “push” notification integrations p a Sn Snowflake-generated messages to external public
a r. ©
nk
cloud vendor services (messageaqueues or email) for delivery
i s h
a n
m
● Type = QUEUE and direction = OUTBOUND => (AWS SNS, Google Pub/Sub, Azure Event Grid)
○ Offer more delivery options (SMS, email, and more) but more complex to configure

● Type = EMAIL => (AWS SES only)

○ Straightforward configuration

© 2023 Snowflake Computing Inc. All Rights Reserved 109

SNOWPIPE ● Snowpipe can push error notifications to a
ERROR NOTIFICATIONS cloud messaging service when it
encounters errors while loading data
AWS example
3 -
● Currently in 2General 0 2 Availability (GA)
C-
DE
1 8-
● Supported- on AWS, Azure and GCP
m
o py
. c o
w○ t-Cross-cloud
o
n no c error notifications not supported
e
ic do-
rv
e 23-
s
@ 20 ● Available for
t y
h ke
r
a fla
m
u ow
a s n ○ Snowpipe auto-ingest and REST API calls
a r.p ©S
nk ○ Snowflake Connector for Kafka
h a
ni s
ma
● Delivers error message in JSON payload

● CREATE or ALTER a pipe including an error

integration to bind to the integration
© 2023 Snowflake Computing Inc. All Rights Reserved 110
TASK ● This feature triggers a notification
ERROR NOTIFICATIONS describing the errors encountered by
Snowflake during task execution failures

3 -
● Supported on 0 2AWS, Azure, and GCP
- 2
E C
○ utilizes8 D
- the cloud provider’s messaging service
-1
o m y
w c
. cop
●
n o t -
o Configurable for both serverless tasks and
ce
i do -n
rv
e 23- user-managed tasks
s
@ 20
t y
h ke
r
a fla
m
u ow
s
a Sn
a r . p ©
● Delivers error message in JSON payload
nk
isha
n
ma ● CREATE or ALTER a task including an error
integration to bind to the integration

© 2023 Snowflake Computing Inc. All Rights Reserved 111

NOTIFICATION SYSTEM STORED PROCEDURE
● Only hosted on AWS currently
● Leverages SES (the AWS Simple Email Service) to deliver emails23to- Snowflake-verified
- 2 0
addresses E C
8 -D
● Utilizes a notification integration of type = email m - 1
. c o py
● Maximum of 10 email notification integrations pernoaccount w t-co (multiple may be active)
ce -n o
rv i do
● Maximum of 50 validated email addresses in s 3-
e ALLOWED_RECIPIENTS
y @ 202
r t h ke Recipients
m a fla
s u ow
. p a Sn
k ar ©
an
i s h
a n
m

Email body

© 2023 Snowflake Computing Inc. All Rights Reserved 112

RESOURCE MONITOR EMAIL NOTIFICATIONS
● No notification integration required – a built-in capability
3 -
0 2
● NOTIFY_USERS parameter specifies users to receive email notifications
C-
2
DE
● The user identifier is the value of the name column from- the 8-
1 output of SHOW USERS
o m y
c
. cop
● Available for administrative and non-administrativeowusers
t- e n no
c
○ Non-admin users only receive notifications fromeindividual rvi 3-do- warehouse resource monitors
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
k ar ©
an
i s h
a n
m

© 2023 Snowflake Computing Inc. All Rights Reserved 113

3 -
2
- 20
E C
8-D
-1
om y
SNOWFLAKE e
s 2
i c
n
e -n
rv 3-do
w c
. cop
o ot-

y @ 0
ALERTS t h ke 2
r
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

© 2023 Snowflake Computing Inc. All Rights Reserved

SNOWFLAKE ALERTS
Empowers Snowflake users to monitor and alert themselves to Snowflake actions and
conditions without setting up external software. 3- 2
- 20
E C
A Snowflake alert is a database object that specifies: 8-D
-1
om y
● When and how often the condition should be evaluated w c p
. co(e.g., every 30min)
o t- n no
i e
c o-
● A condition that triggers the alert (e.g., newserows rv -d added to a table)
3
y @ 202
● The action to perform when the condition r t h is
ke met (e.g., send an email notification)
a
m wfla
u
s no
a
a r.p ©S
Create alerts to monitor: nk
h a
ni s
● Credit usage ma
● Resource consumption
● Data access
● Business rule compliance
© 2023 Snowflake Computing Inc. All Rights Reserved 115
SNOWFLAKE ALERT EXAMPLE

3 -
CREATE OR REPLACE ALERT test_alert-202
E C
WAREHOUSE = my_warehouse 8 -D
- 1
SCHEDULE = '1 minute' om y
w c
. cop USE CRON OR MINUTE
IF( EXISTS( n o ot-
i ce -n
SELECT data_value e rv 3-do
@ s 2
CONDITION
FROM data_table y
h ke 2 0
r t
a fla
WHERE data_value m
u ow > 200
a s n
a r .p ©S
) nk
h a
) ani s
m
THEN
CALL SYSTEM$SEND_EMAIL(...);

ACTION

© 2023 Snowflake Computing Inc. All Rights Reserved 116

SNOWFLAKE ALERTS BENEFITS
● Facilitates “separation of duties” for alerting database objects
3 -
2 0
● Includes separate error statuses and query IDs for the condition - 2 and the action
E C
8 -D
-1
● Status includes info about scheduling, execution, triggering, om y cancellation, and failure
w c
. cop
n o ot-
i ce -n
● Can access deterministic evaluation timestamps e rv 3-dto o avoid issues with ingestion latency and
@ s 2
y
h ke 2 0
r t
evaluation latency (precise timing) uma wfla
a s no
a r.p ©S
k
○ SNOWFLAKE.ALERT.LAST_SUCCESSFUL_SCHEDULED_TIME()
n
h a
ni s
m a
○ SNOWFLAKE.ALERT.SCHEDULED_TIME()

© 2023 Snowflake Computing Inc. All Rights Reserved 117

MONITOR SNOWFLAKE ALERTS
Monitor Snowflake alerts
3 -
2
● SHOW alerts - 20
E C
8-D 1
● DESCRIBE alerts -
om y
w c
. cop
n o ot-
● Review alert execution with INFORMATION_SCHEMA.ALERT_HISTORY() i ce -n table function
rv
e 23- d o
s
@ 20
t y
h ase
○ CONDITION - The text of the SQL statement that serves r
a fla k the condition for the alert.
m
u ow
s
a Sn identifier for the SQL statement executed as the condition of the alert.
○ CONDITION_QUERY_ID - Internal/system-generated
. p
k ar ©
an
s h that serves as the action for the alert.
○ ACTION - The text of the SQL statement
i
a n
m
○ ACTION_QUERY_ID - Internal/system-generated identifier for the SQL statement executed as the action of the alert.

○ STATE - Status of the alert. This can be one of the following:

 SCHEDULED, EXECUTING, FAILED, CANCELLED, CONDITION_FALSE, CONDITION_FAILED, ACTION_FAILED,

TRIGGERED

3 -
2
- 20
E C
8-D
-1
om y
OBSERVABILITY e
s 2
i c
n
e -n
rv 3-do
w c
. cop
o ot-

ACCOUNT-LEVEL TASK HISTORY
● Visualize system-wide task history in a single pane
● Filter by task name, date, status, database, or schema
3 -
2
● Embedded search capability - 20
E C
8 -D Profile
● Open the executed task query in a new worksheet or in -Query
1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn Search box
. p
kar ©
n
isha
n
ma

INDIVIDUAL TASK
RUN HISTORY
● Use Task Run History to visualize
schema level task 3 - history - has a task
0 2
been running 2
- successfully, failed, or is it
E C
8-
scheduled D in the future?
- 1
o m y
w c
. cop
n
● o Filter
o t - options include date ranges and
i ce -n
e rv 3-do
@ s 2 task status
h y e2 0
a t
r1 ak
u m wfl
a s no ● (1) Open the task history query itself in a
. p S
kar ©
n new worksheet
isha
n
ma 2
● (2) Open the executed task query in a
new worksheet or in Query Profile

TASK GRAPH VIEWER
● Visualize the relationship and execution order of your complex Snowflake task logic
3 -
2
● Execute the task graph by selecting “Run Task Graph” from this20screen
C-
DE
8-
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

DYNAMIC TABLE GRAPH AND HISTORY VIEWER
● Visualize the relationship and execution order of your dynamic tables (Graph tab)
● Monitor refreshes to check for problems (Refresh History tab)
3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

COPY HISTORY DASHBOARD
● Access under Activity > Copy History in Snowsight
● Displays COPY INTO and PIPE activity
● Hover over elements for additional information -
2 3
● Filter, or click elements for different views (day level, table, level) - 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

STAGE OBSERVABILITY
● View and upload files to your stages

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

LAB EXERCISE: 6
LAB: Management and Observability
30 minutes -
2 3
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

3 -
2
- 20
E C
8-D
-1
om y
PERFORMANCE e
s 2
i c
n
e -n
rv 3-do
w c
. cop
o ot-

MODULE AGENDA
● Search Optimization Service
3 -
● Query Acceleration Service 2
- 20
E C
● Join Elimination Using Constraint Support 8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

3 -
2
- 20
E C
8-D
-1
om y
SEARCH OPTIMIZATION e
s 2
i c
n
e -n
rv 3-do
w c
. cop
o ot-

SEARCH OPTIMIZATION SERVICE
Quickly return a small number of rows on large tables

Enable faster query 3performance

- on
0 2
2
- and
data exploration DE C
1 8-
-
filtering workloads
om y . c p
w
o ot- co
n
e -n
i c
e rv 3-do
Accelerate
@ s 2
y e2 0 point lookup queries for
t h
r ak
a
m wfl columns of supported data types on
u
s no
p a
kar. ©S large tables
n
isha
n
ma
Common use cases: data sharing,
security log analysis, what-if analysis

SEARCH OPTIMIZATION SERVICE FRAMEWORK
● Builds a persistent data structure that stores the optimized search access path
- 3
○ Maintained in a separate structure (storage) 0 2
- 2
E C
○ Updated as changes are made to the base table 8-D
-1
om y
w c
. cop
Expanded Functionality now includes: ceno -not-
rv i do
s e 23-
y @ 20
r t h ke
○ Specify a subset of columns
m a fla
s u ow
○ VARIANT and Geography columns . p a Sn
k ar ©
an
i s h
○ Substring and regular expression searches
a n
m

POINT QUERY EXAMPLE
● 3.2 Billion rows in STORE_SALES table
● Clustering key on the SS_SOLD_DATE_SK and SS_ITEM_SK columns
3- 2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
-- Find the sales ticket, i ce items
-n sold and sale date
rv
e 23- d o
-- for one customery@ s
in20a store
r t h ke
m a fla
s u ow
SELECT ss_ticket_number,
. p a Sn ss_sold_date_sk, ss_item_sk
a r ©
nk
FROM store_sales
h a
WHEREani s Only a small
m number of rows
ss_customer_sk = 3229284
meet the criteria
AND ss_store_sk = 1388;
for result

SEARCH PERFORMANCE BENEFIT
Base Table (no Search Optimization) Search Optimized Table
3 -
60X+ 0 2
- 2
FASTER E C
8- D
- 1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

10 mins+ on xsmall WH 10 secs on xsmall WH = 63X faster

NEW FLEXIBLE SYNTAX
Choose which columns to index for which types of search methods:

● EQUALITY 3 -
2
● SUBSTRING - 20
E C
● GEO 8-D
-1
om y
w c
. cop
Examples: n o ot-
i ce -n
e rv 3-do
@ s 2
y 2 0
hallksupported
● Add EQUALITY search optimization for a r t e columns in t1:
u m wfla
as Sno
ALTER TABLE t1 ADD SEARCHr.pOPTIMIZATION;
k a ©
an
i s h
a n
● m
Add EQUALITY optimization to c1 and a subpath of c5, and a
GEO optimization to c3 and c4:
ALTER TABLE t1 ADD SEARCH OPTIMIZATION ON
EQUALITY(c1, c5:user:uuid), GEO(c3, c4);

MONITORING SEARCH OPTIMIZATION
● Monitor credit consumption with SEARCH_OPTIMIZATION_HISTORY
- 3
○ A view in SNOWFLAKE.ACCOUNT_USAGE schema 0 2
- 2
E C
○ A table function in the INFORMATION_SCHEMA 8-D
-1
om y
w c
. cop
n o ot-
i ce -n
● DESCRIBE SEARCH OPTIMIZATION e rv 3-do
@ s 2
y 2 0
h ke etc.)
○ Shows optimization information (type, columns, r t
a fla
m
u ow
s
a Sn
. p
k ar ©
an
i s h
● SHOW TABLES providesaninformation on search optimizations
m
○ ON/OFF
○ Progress
○ Bytes used

SEARCH OPTIMIZATION COST CONSIDERATIONS
● Estimate table and column-level search optimization costs with the system function:
- 3
○ 0 2
SYSTEM$ESTIMATE_SEARCH_OPTIMIZATION_COSTS( - 2
E C
'<table name>'[,'<search_method_with_target>’]) 8 -D
-1
om y
w . c o p
● Estimate requires 7 days of history n o ot- c
i ce -n
e rv 3-do
@ s 2
● The larger the number of high-cardinality y 2 0
columns,
h ke the higher the cost
r t
a fla
m
u ow
s
a Sn
. p
r © DML operations) will also result in higher cost
● Tables with frequent mutations k a(from
an
i s h
n
○ Batch DML operations if possible
a
m

136
3 -
2
- 20
E C
8-D
-1
om y
QUERY ACCELERATION e
s 2
i c
n
e -n
rv 3-do
w c
. cop
o ot-

PROBLEM Unpredictable workloads make it difficult
to right-size a virtual warehouse
250
Example: 3 -
2
- 20
E C
200 ● 80%- of 8-D
1 the time, a medium virtual
o m y
c
. cop
wwarehouse handles all queries well
- no not
e
ic do-
rv
e 23-
150 s
@ 20 ● 20% of the time, outlier workloads impact
t y
h ke
r
a fla
u m the virtual warehouse as follows:
a s now
100 a r.p ©S ○ Slow down existing workloads
nk
h a
ni s ○ Consume resources causing new queries to
ma queue
50

● Scaling the warehouse up is not always a

0 workable solution
9AM 10AM 11AM 12PM 1PM 2PM 3PM 4PM 5PM

SOLUTION
QUERY ACCELERATION SERVICE ● Shared pool of compute resources that a
VWH can lease to offload qualifying
before after
portions of query3 -workload to
2
- 20
E C
8-D
● Prevents - 1 oversized (outlier) statements
from o m running
y slowly, and slowing down
. c o p
n owother
o t - c
queries
i ce -n
e rv 3-do
@ s 2 ○ Helps maintain ELT SLA consistency with
h y e2 0
a t
r ak variable data volumes
u m wf l
a s no
p
kar. ©S ● Enables expedient exploration and
n
isha processing of large data sets
n
ma
Query Acceleration Service enabled here

QAS QUERY QUALIFYING CHARACTERISTICS
Which types of queries qualify for the Query Acceleration Service?
This is determined by the following factors:
3 -
2
- 20
1. Query Type E C
8-D
○ Standalone SELECT statements OR -1
om y
○ SELECT portions embedded in INSERT or CTAS w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
2. Query Elements y
h ke 2 0
r t
a fla
m
○ Scan large amounts of data AND haveapredicates su now that
a r .p ©S
 Filter out a significant numbernof k rows OR
h a
ni s
ma
 Contain simple aggregations
Virtual Warehouse
size also influences QAS
3. Query Duration usage at runtime
○ Ideal candidates are queries that run for longer than two minutes
○ Shorter queries may also benefit but are not the intended target for this service
© 2023 Snowflake Computing Inc. All Rights Reserved 140
EXAMPLE: LARGE SCAN + SELECTIVE FILTER
Query runs for
3
3-
~6 minutes on
2
- 20 X-Small WH
E C
8-D
-1
USE SCHEMA om y
w c
. cop
snowflake_sample_data.tpcds_sf10tcl; n o ot-
i ce -n
e rv 3-do
@ s 2
1 SELECT y
h ke 2 0
r t
a fla
m
u ow
ss_ticket_number,
28 billion s
a Sn
. p
rows in table, k ar ©
ss_sold_date_sk,
an Predicate filters out
1.3 TB in size i s h
ss_item_sk
a n most of the rows.
mFROM
store_sales
WHERE
ss_customer_sk = 3229284; 2

3 -
2
- 20
E C
8-D
STEP 1: n
w c o
. cop
o ot-
-1
m y

QUERY ACCELERATION ELIGIBLE VIEW
● ACCOUNT_USAGE.QUERY_ACCELERATION_ELIGIBLE identifies queries eligible for QAS
3- 2
○ There is no Information Schema equivalent - 2 0
E C
● Provides a historical view (up to 1 year) across queries executed 8- D in an account
- 1
○ There is no pre-execution option to determine eligibility for omnew y queries
. c
w t-co p
n o o
○ Has a latency of up to 180 minutes ce
i do -n
rv
e 23-
s
@ 20
t y
h ke
r
a fla
m
u warehouses
w
s
Identify
a n o
a r.pthat
© Swould benefit
nk
h a from QAS
ni s
ma Maximum effective
QAS scale factor for
the query

ESTIMATE
ACCELERATION BENEFIT
● Use the system function
SYSTEM$ESTIMATE_QUERY_ACCELERATION($qid)
3 -
0 2
● Validate whether 2
- a previously executed query is
E C
D
eligible- 1to8- benefit from QAS
o m y
w c
. cop
● n o -
oIf eligible,
t the output includes the estimated
ce
i do -n
rv
e 23- query execution time for different query
s
@ 20
t y
h ke acceleration scale factors in seconds
a r
u m wfla
a s no
a r.p ©S ● Example:
nk
h a
ni s ○ Original query without QAS took 337.844 seconds
ma
○ Engaging QAS at a scale factor of 8, the query
could complete in ~47 seconds
 Note that estimated query times are for analysis purposes
only and are not guaranteed

3 -
2
- 20
E C
8-D
-1
om y
STEP 2: e
s 2
i c
n
e -n
rv 3-do
w c
. cop
o ot-

QUERY ACCELERATION MAX SCALE FACTOR
● This parameter limits the total amount of QAS compute resources a VWH can utilize
● Defines the runtime compute allocation target for QAS (bound by the available
3- resources)
0 2
● Scale factor is a multiplier based on VWH size - 2
E C
8 - D
1
○ e.g. Scale factor = 2 leases up to 2 times the size of the VWH of -QAS compute resources
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
k ar ©
an
i s h
n
Value maVirtual warehouse can borrow up to…

0 ALL the resources currently available to the QAS

1 - 100 Original size of the virtual warehouse times the number specified

8 This is the default value for the parameter

© 2023 Snowflake Computing Inc. All Rights Reserved 146
ENABLING THE QUERY ACCELERATION SERVICE
● Enabling QAS provides a VWH access to a shared pool of compute resources that it can
3 -
lease to offload qualifying portions of query workload to 0 2
- 2
E C
8-D 1
● The pool of QAS resources is shared by all accountsom -
on ya cloud provider, in a region
w . c o p
n o ot- c
i ce -n
● QAS is enabled at the warehouse level via sthe -do
erv23following parameter
y @ 2 0
o enable_query_acceleration r t h ke
m a fla
s u orowlater with an ALTER statement
o Can be set at warehouse creationptime a n
a r. ©S
nk
h a
ni s
ma

3 -
2
- 20
E C
8-D
STEP 3: n
w c o
. cop
o ot-
-1
m y

QUERY PROFILE METRICS
EXECUTED ON ACCELERATED WAREHOUSE ● Key Query Profile metrics for QAS
○ Partitions scanned by service
3 -
2 offloaded for scanning to QAS
 number of 0files
- 2
E C
○ Scans-Dselected for acceleration
1 8
-
m
 number of table scans being accelerated
o py
. c o
w○ t-Partitions
o
n no c total
e
ic do-
rv
e 23-  Combined scan workload for the WH and QAS
s
@ 20
t y
h ke
r
a fla
m
u ow ● For the single table scan displayed:
a s n
a r.p ©S
nk ○ Virtual warehouse scanned 5796
h a
ni s micro-partitions (~9% of total)
ma
○ QAS service scanned 57793
micro-partitions (~91% of total)

MONITORING QAS ACTIVITY
● ACCOUNT_USAGE.QUERY_HISTORY view includes acceleration columns for accelerated
3 -
queries 0 2
- 2
E C
8-D
o query_acceleration_bytes_scanned
-1
o query_acceleration_partitions_scanned
om y
w c
. cop
o query_acceleration_upper_limit_scale factor n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
● ACCOUNT_USAGE.QUERY_ACCELERATION_HISTORY s
a Sn
.
r ©p view reports QAS consumption
k a
per hour, by virtual warehouse an
i s h
n
ma
● The Usage page in the Admin area of Snowsight includes QAS consumption details

3 -
2
- 20
E C
8-D
-1
om y
c
. cop
JOIN ELIMINATION y @
e
s 2
0
i c
rv 3-do
n
e -n
w
o ot-

t h ke 2
USING CONSTRAINT SUPPORT . p
s
m
u ow
a Sn
r
a fla

JOIN ELIMINATION ● Queries often include unnecessary join
USING CONSTRAINT SUPPORT conditions

● The Snowflake 3 -
query planner can
2
- 20
leverageDEthe C primary key (PK) / foreign
8 -
key (FK)
- 1 and uniqueness constraints to
o m y
w c
. cop unnecessary joins
identify
o - n not
e
ic do-
rv
e 23-
s
@ 20 ● Set a RELY property on these constraints
t y
h ke
r
a fla
u m w to tell the planner to evaluate for join
a s no
a r.p ©S elimination
nk
isha
n
ma ● Eliminated joins improve performance!

JOIN ELIMINATION
USING CONSTRAINT SUPPORT
Operational Steps:
1. Check data quality (ensure no nulls, duplicates, and 1:1 match on 3 -
PK/FK)
02 - 2
E C
2. Add RELY property to constraint 8-D
-1
om y
ALTER TABLE t1 ADD CONSTRAINT pk_1 PRIMARY w c
. cop KEY (c1) RELY;
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
SELECT r t
a fla
m
uNUM_OF_ROWS,
w
count (O.*) AS a s n o
a r.p ©S
count (DISTINCT nk C.C_CUSTKEY) as UNIQUE_CUSTOMERS,
h a
ni s
sum (O.O_TOTALPRICE)
ma as sales
FROM
PK
orders o
JOIN customer c
ON o.o_custkey = c.c_custkey;
© 2023 Snowflake Computing Inc. All Rights Reserved FK 153
JOIN ELIMINATION EXAMPLE
Without CONSTRAINT…RELY

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma
With CONSTRAINT…RELY
enabled the unnecessary
join is eliminated!

LAB EXERCISE: 7
LAB: Search Optimization Service
15 minutes -
2 3
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

LAB EXERCISE: 8
LAB: Query Acceleration Service
15 minutes -
2 3
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

LAB EXERCISE: 9
LAB: Join Elimination Using Constraint Support
10 minutes -
2 3
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
e -n
PROGRAMMABILITY
i c
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

MODULE AGENDA
● Tools and Utilities
○ Python Worksheets 3 -
2
- 20
○ Visual Studio Code Extension E C
8-D
-1
○ SnowCLI o m y
w c
. cop
n o o t -
○ Get_Query_Operator_Stats i ce -n
e rv 3-do
@ s 2
○ Logging y
h ke 2 0
r t
a fla
● Streamlit m
u ow
s
a Sn
. p
k ar ©
an
i s h
a n
m

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
ce -n
TOOLS AND UTILITIES
i
rv 3-do
e
s 2
y @ 2 0
r t h ke
m a fla
s u ow
. p a Sn
kar ©
n
isha
n
ma

PYTHON WORKSHEETS
● Use Snowpark Python in Snowsight to perform data manipulations and transformations
3-
● Many options similar to SQL worksheets (sharing, version history…)
2
- 20
● Display results and Python process logging output E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

NOTE: when a Python worksheet runs in Snowsight, an anonymous stored procedure runs
the code and generates queries that execute the Snowpark commands in the code.

PYTHON WORKSHEETS
Run the code as a Snowflake
Set handler type, line Import Python libraries from stored procedure
3 -
2
wrapping, linting Anaconda or a Snowflake stage - 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma Boilerplate code

SNOWFLAKE VS-CODE EXTENSION
● Write and execute Snowflake
SQL directly in VS Code 3 -
2
- 20
● Install from the Visual Studio E C
8-D
Marketplace, free -1
om y
w c
. cop
n o ot-
i ce -n
1. Sidebar panel e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
2. Snowflake SQL panel m
u ow
s
a Sn
. p
3. Query Results panel kar ©
n
ha
4. Query History section ofanis
m
Side Bar
5. Current session information

SNOWCLI
● Open-source command line interface for working with Snowflake
● Create, manage, update, and view apps running in Snowflake
3 -
● Build and deploy functions, stored procedures, and Streamlit apps 2
- 20 to Snowflake
E C
● Download from github.com/Snowflake-Labs/snowcli 8 -D
1 -
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

GET_QUERY_OPERATOR_STATS
● System-provided table function to programmatically retrieve query profile information
3 - has completed
● Returns statistics about individual query operators within a query that
2
- 20
● Run for any completed query executed in the past 14 days DEC
1 8-
-
om y
w c
. cop
● Use to diagnose n o ot-
i ce -n
e rv 3-do
○ “Exploding” joins @ s 2
y
h ke 2 0
○ UNION without ALL r t
a fla
m
u ow
s
a Sn
○ Queries too large to fit . p
k ar ©
in memory (spilling) an
i s h
a n
○ Inefficient pruning m

● Log messages (warning or error) from a
LOGGING stored procedure, UDF, or UDTF

● Define LOG_LEVEL - as object or session

2 3
parametersC-20
DE
8- 1
● Access -
o m y logged messages from a
w . c o p
n t - c
o pre-defined account-level Event table
ce -n o
i
rv 3-do
e
s 2
y @ 2 0 ● Support for Python, Java, JavaScript, Scala,
t h
r ak e
a
m wfl
s u o and Snowflake Scripting (not anonymous
. p a Sn
kar © objects)
n
isha
n
ma ● Messages are collected and ingested by
Snowflake-managed resources (serverless)

● Track logging credit usage with

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
ce -n
STREAMLIT
i
rv 3-do
e
s 2
y @ 2 0
r t h ke
m a fla
s u ow
. p a Sn
kar ©
n
isha
n
ma

STREAMLIT
● Python open-source library for app development, supported by the community
-
2 3
- 20
● Quickly and easily write, share, and deploy data apps E C
8-D
● Build web apps for machine learning and data sciencem - 1
. c o py
o w t-co
e n no
rv ic do-
The combination of Streamlit’s ease s -
eof23use, flexibility, and
@
y e2 0
t h
r ak
open-source community s-umalong a f l with Snowflake’s
a o w
r . p ©Sn
scalability, scope of data, ank and governance - could enable
a
i s h
entirely new mclasses
a n of data apps to be built.
Chris Child – Snowflake Senior Director of Product Management

STREAMLIT PRINCIPLE ONE:
Embrace Python scripting

3 -
2
- 20
E C
8-D
-1
om y
Works with all
w c
. cop
your favorite n o ot-
i ce -n
Python libraries e rv 3-do Go straight from
@ s 2 Python script to
(pandas, Altair, y
h ke 2 0
NumPy, plotly, r t
a fla web app
m
u ow
scikit-learn, etc) s
a Sn
. p
kar ©
n
isha
n
ma

STREAMLIT PRINCIPLE TWO:
Treat widgets as variables

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do No need to write
@ s 2 a backend,
y
h ke 2 0
r t
a fla handle HTTP
Adding a widget m
u ow
s
a Sn requests, write
is the same as . p
kar © HTML, CSS,
declaring a n
variable isha JavaScript, ...
n
ma

STREAMLIT PRINCIPLE THREE:
Reuse data and computation

3 -
2
- 20
Instantiate E C
8-D
-1
db/model
connectors only om y
w c
. cop
once
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
Store query kar ©
n
results in cache isha
n
ma

STREAMLIT ROADMAP HIGHLIGHTS
● Editable DataFrames -
2 3
○ st.data_editor - 20
E C
8-D
● New caching primitives -1
om y
w c
. cop
○ st.cache_data and st.cache_resource n o ot-
i ce -n
e rv 3-do
● Connect apps to databases and APIs with@as single 0 2 line of code
t h y e2
a r ak
○ st.connection u m wfl
a s no
a r .p ©S
● Snowflake integration (moreato nk come, soon…)
i s h
a n
m

LAB EXERCISE: 10
LAB: Programmability
15 minutes -
2 3
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
e -n
HONORABLE MENTIONS
i c
e rv 3-do
@ s 2
y
h ke 2 0
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

HONORABLE MENTIONS
● ServiceNow Connector ● ON_ERROR Copy Option Supports All
● SELECT…EXCLUDE…RENAME File Formats -
2 3
● Automatically Suspend Tasks After Failed ● External Table - 2 0 and Directory Table Auto-
E C
Runs Refresh D
8- Observability and Billing
- 1
o
● wLoadm y
● Secondary Roles Support in Snowsight c
. op Files into Stages Using Snowsight
n o ot-c
Worksheets i ce ● -n
e rv 3-do Load Data into Tables using Snowsight
@ s 2
● Password Policies y 2 0 ● Snowsight Worksheet Tabs
rth e
a flak
m
u ow
● New Account Usage View: PROCEDURES s
a Sn ● Memoizable Functions
. p
k ar ©
● New Account Usage View: han ● Schema Detection for JSON and CSV -
nis
LOCK_WAIT_HISTORY ma preview
● Access Control: Database Roles
● Access Control: SNOWFLAKE Database
Roles

LAB EXERCISE
Please remember to complete the Course Survey to access your Certificate

10 minutes (Your course title may differ)

-
2 3
- 20
E C
8-D
1. Navigate to the
course page -1
om y
w c
. cop
2. Click “To-do” in n o ot-
i ce -n
the section at the e rv 3-do
@ s 2
bottom of the page y
h ke 2 0
r t
a fla
m
u ow
3. Click “Play” s
a Sn
. p
k ar ©
n
4. Complete the isha
n
survey to see the ma
Certificate button
and download

3 -
2
- 20
E C
8-D
-1
om y
w c
. cop
n o ot-
i ce -n
e rv 3-do
@ s 2
y 2
h ke YOU 0
THANK
r t
a fla
m
u ow
s
a Sn
. p
kar ©
n
isha
n
ma

Snowflake PPT 22
50% (2)
Snowflake PPT 22
220 pages
Snowflake Free Lab Guide
50% (4)
Snowflake Free Lab Guide
58 pages
SnowFlake SnowPro Core Certification Notes - Fromblogs (1) 1 2
No ratings yet
SnowFlake SnowPro Core Certification Notes - Fromblogs (1) 1 2
172 pages
Snowflake Training
No ratings yet
Snowflake Training
685 pages
CNC USB Controller Mk1 PDF
50% (2)
CNC USB Controller Mk1 PDF
18 pages
Programming+in+Snowflake+ +All+Slides
100% (1)
Programming+in+Snowflake+ +All+Slides
342 pages
Snowflake
No ratings yet
Snowflake
58 pages
Snowflake Training Slide SANMs
67% (6)
Snowflake Training Slide SANMs
218 pages
SOL-C01-SnowProPlatform StudyGuide 020325
No ratings yet
SOL-C01-SnowProPlatform StudyGuide 020325
10 pages
Snowpro Advanced: Data Engineer: Exam Study Guide
No ratings yet
Snowpro Advanced: Data Engineer: Exam Study Guide
14 pages
Snow Pro Core Study Guide
No ratings yet
Snow Pro Core Study Guide
14 pages
Snowflake Data Engineering
No ratings yet
Snowflake Data Engineering
12 pages
Snowpro™ Core: Study Guide
100% (1)
Snowpro™ Core: Study Guide
17 pages
Solutions Partner Technical Onboarding Guide
100% (1)
Solutions Partner Technical Onboarding Guide
27 pages
SnowProCore Exam Study Guide 011425 COF C02
No ratings yet
SnowProCore Exam Study Guide 011425 COF C02
14 pages
Snowflake - Syllubus and DBT
No ratings yet
Snowflake - Syllubus and DBT
11 pages
Snowpro™ Core: Exam Study Guide
100% (1)
Snowpro™ Core: Exam Study Guide
15 pages
Brtools Update-Georg Chlond
No ratings yet
Brtools Update-Georg Chlond
25 pages
Hands-On Lab Guide For: Virtual Zero-To-Snowflake
No ratings yet
Hands-On Lab Guide For: Virtual Zero-To-Snowflake
63 pages
Snowflake Notes
No ratings yet
Snowflake Notes
20 pages
Snowpro™ Core: Exam Study Guide
No ratings yet
Snowpro™ Core: Exam Study Guide
17 pages
Snowpro™ Advanced: Data Engineer: Exam Study Guide
No ratings yet
Snowpro™ Advanced: Data Engineer: Exam Study Guide
16 pages
Administering Snowflake
No ratings yet
Administering Snowflake
4 pages
Snowfalke DBT ADF Course
No ratings yet
Snowfalke DBT ADF Course
5 pages
Hipertension Epidemio 22-08-2018
No ratings yet
Hipertension Epidemio 22-08-2018
979 pages
Where Can Buy Python Business Intelligence Cookbook 1st Edition Roger Dempsey Dempsey Roger Ebook With Cheap Price
100% (4)
Where Can Buy Python Business Intelligence Cookbook 1st Edition Roger Dempsey Dempsey Roger Ebook With Cheap Price
49 pages
Snowpro Core Guide
No ratings yet
Snowpro Core Guide
16 pages
Chapter 4
No ratings yet
Chapter 4
45 pages
Excel Tutorial PDF
No ratings yet
Excel Tutorial PDF
13 pages
Data Engineers Guide To Python On Snowflake
No ratings yet
Data Engineers Guide To Python On Snowflake
15 pages
Snowflake UNIT II
No ratings yet
Snowflake UNIT II
44 pages
Epc L06020W0000001
No ratings yet
Epc L06020W0000001
331 pages
S V Engineering College: Lecture Notes
No ratings yet
S V Engineering College: Lecture Notes
160 pages
SnowProCore Exam Study Guide 050423
No ratings yet
SnowProCore Exam Study Guide 050423
16 pages
Standard NF Workbook 23G31
No ratings yet
Standard NF Workbook 23G31
128 pages
DMV Lab Manual Final 13th April 24 v2
No ratings yet
DMV Lab Manual Final 13th April 24 v2
73 pages
All Snowflake Details Document
No ratings yet
All Snowflake Details Document
105 pages
GCP Snowflake
No ratings yet
GCP Snowflake
83 pages
Snowflake The Game Changer 1728286095
No ratings yet
Snowflake The Game Changer 1728286095
93 pages
Resume Prashant Agarwal Dec 2023 V6
No ratings yet
Resume Prashant Agarwal Dec 2023 V6
2 pages
Enterprise Angular Monorepo Patterns
No ratings yet
Enterprise Angular Monorepo Patterns
76 pages
SAP BW4HANA Migration Options
No ratings yet
SAP BW4HANA Migration Options
6 pages
Fundamentals of Database Systems 7th Edition CHAPTER 1 Instructor Solutions Manual
88% (8)
Fundamentals of Database Systems 7th Edition CHAPTER 1 Instructor Solutions Manual
3 pages
Lecture 5 - Fragments
No ratings yet
Lecture 5 - Fragments
23 pages
Data Engineering Agenda
No ratings yet
Data Engineering Agenda
19 pages
Best Practices For Using Tableau With Snowflake
No ratings yet
Best Practices For Using Tableau With Snowflake
63 pages
AWS Data Eng
No ratings yet
AWS Data Eng
8 pages
Data Brick
No ratings yet
Data Brick
6 pages
Academia Program Overview
No ratings yet
Academia Program Overview
16 pages
Kent Graziano UKDVUG Snowflake-Features-18NOV20
No ratings yet
Kent Graziano UKDVUG Snowflake-Features-18NOV20
40 pages
Tecnical Seminar
No ratings yet
Tecnical Seminar
16 pages
Generative AI in Seconds Making Sense of Unstructured Data Across Industries
No ratings yet
Generative AI in Seconds Making Sense of Unstructured Data Across Industries
16 pages
SnowProCoreStudyGuide 042621
No ratings yet
SnowProCoreStudyGuide 042621
13 pages
The Data Engineers Guide To Python For Snowflake
No ratings yet
The Data Engineers Guide To Python For Snowflake
15 pages
Snowpro™ Advanced: Data Scientist: Exam Study Guide
No ratings yet
Snowpro™ Advanced: Data Scientist: Exam Study Guide
14 pages
S4 Hana
No ratings yet
S4 Hana
9 pages
X IT LAB Practical FILE 2025-26
No ratings yet
X IT LAB Practical FILE 2025-26
4 pages
Snow
No ratings yet
Snow
17 pages
9.98 Web Client Compatibility
No ratings yet
9.98 Web Client Compatibility
13 pages
Kuchh To Hua Hai
No ratings yet
Kuchh To Hua Hai
12 pages
Snowflake - Billing Components
No ratings yet
Snowflake - Billing Components
9 pages
SSC Service Utility For Epson Stylus Printers.: Russian Version
No ratings yet
SSC Service Utility For Epson Stylus Printers.: Russian Version
4 pages
SAMPLE Winter - MBA Data & Program Analytics
No ratings yet
SAMPLE Winter - MBA Data & Program Analytics
7 pages
Snowflakes Beginner To Intermediate Path Updated
No ratings yet
Snowflakes Beginner To Intermediate Path Updated
4 pages
2-Snowflake Track - Curriculum
No ratings yet
2-Snowflake Track - Curriculum
5 pages
Basic Programming
No ratings yet
Basic Programming
20 pages
3rd Sem C Programing
No ratings yet
3rd Sem C Programing
28 pages
Test Automation FX - User Guide
No ratings yet
Test Automation FX - User Guide
11 pages
Information Technology SQP-01 2024
No ratings yet
Information Technology SQP-01 2024
5 pages
MET CS777 Summer2-2022 Big-Data-Analytics
No ratings yet
MET CS777 Summer2-2022 Big-Data-Analytics
9 pages
379-Article Text-920-1-10-20200517
No ratings yet
379-Article Text-920-1-10-20200517
5 pages
Starting Snwoflake
No ratings yet
Starting Snwoflake
5 pages
Transport Management System Word Document
No ratings yet
Transport Management System Word Document
3 pages
OpenText File System Archiving 10.2.0 Release Notes
No ratings yet
OpenText File System Archiving 10.2.0 Release Notes
13 pages
Day 2
No ratings yet
Day 2
2 pages
Profile Links: Websites
No ratings yet
Profile Links: Websites
8 pages
Mock Test Paper - Unsolved
No ratings yet
Mock Test Paper - Unsolved
8 pages
SF Sy
No ratings yet
SF Sy
2 pages
Data Science With Python Workflow: Click The Links For Documentation
No ratings yet
Data Science With Python Workflow: Click The Links For Documentation
2 pages
Boot Process, Init and Shutdown
No ratings yet
Boot Process, Init and Shutdown
6 pages
Creating A Session For Snowpark Python - Snowflake Documentation
No ratings yet
Creating A Session For Snowpark Python - Snowflake Documentation
1 page
Writing Snowpark Code in Python Worksheets - Snowflake Documentation
No ratings yet
Writing Snowpark Code in Python Worksheets - Snowflake Documentation
1 page
Working With DataFrames in Snowpark Python - Snowflake Documentation
No ratings yet
Working With DataFrames in Snowpark Python - Snowflake Documentation
1 page
Perintah Untuk Memberikan Deskripsi Mengenai Interface Router
No ratings yet
Perintah Untuk Memberikan Deskripsi Mengenai Interface Router
5 pages
XCFGF - Google Search
No ratings yet
XCFGF - Google Search
2 pages
Cash Receipt PDF
No ratings yet
Cash Receipt PDF
1 page
New Ticket For Sunday
No ratings yet
New Ticket For Sunday
2 pages
Mobile Robot Chapter 4: C and 8051 (V4.a)
No ratings yet
Mobile Robot Chapter 4: C and 8051 (V4.a)
10 pages
Cisco ASA Remote Access VPN Syslog Messages
No ratings yet
Cisco ASA Remote Access VPN Syslog Messages
2 pages
Collabera AAAAAAAAAAAAAAAAAAAAAAAAAATACT - Ereceipt - Mani PDF
No ratings yet
Collabera AAAAAAAAAAAAAAAAAAAAAAAAAATACT - Ereceipt - Mani PDF
1 page
Techno Score
No ratings yet
Techno Score
2 pages
Chatlog WWW Globustrainings Com 2014-05-19 10 56
No ratings yet
Chatlog WWW Globustrainings Com 2014-05-19 10 56
1 page