Snowflake Best Practices

For

From

Infometry Inc
39111 Paseo Padre Pkwy, Ste 218
Fremont, CA, 94538
www.infometry.net
Tel: (510) 793 5859
 Change Record

The following table makes it easy to come back later and track what changes were made to the
requirements at each point of the project, who made them, and why they were made. This is a
way of implementing change control on the Functional Specification Document.

Date Author Version Summary of Changes

Distribution (for non-approvers)

Name / Alias Team Role

1. Introduction.............................................................................................................................................6
1.1. Document Purpose....................................................................................................................................6

1.2. Audience and Ownership...........................................................................................................................6

2. References...............................................................................................................................................6
1.1. Glossary of Terms.......................................................................................................................................6

1.1. Related Documents....................................................................................................................................6

3. Snowflake Conventions............................................................................................................................7
1.2. Meta Data Table.........................................................................................................................................7

For Example:.........................................................................................................................................................7

QA test Query to verify the records count information........................................................................................7

DDL of the table....................................................................................................................................................8

Snowflake Best Practices

 To Get all the objects from a DB...........................................................................................................................8

1.3. View Conversion........................................................................................................................................8

SQL Server View....................................................................................................................................................8

Snowflake View...................................................................................................................................................12

1.4. Snowflake Procedure:..............................................................................................................................15

Table information for error capture of the procedure execution.......................................................................15

Procedure for Error Log......................................................................................................................................16

1.5. SQL Server Meta Data Query Reference..................................................................................................17

SQL Server Meta Query......................................................................................................................................17

Number of the tables and columns.....................................................................................................................18

Database Size......................................................................................................................................................18

To know more information of the depend objects of the tables........................................................................18

XML Process name from Processor Log..............................................................................................................19

Object Type information.....................................................................................................................................19

Reference URL....................................................................................................................................................19

1.6. Snowflake Reference...............................................................................................................................20

Parameter Setup.................................................................................................................................................20

For Example........................................................................................................................................................20

 Cloning the database...................................................................................................................................21

AP Payment XML Format Query.........................................................................................................................21

XML Format to load the data..............................................................................................................................22

Snowflake query to flatten..................................................................................................................................22

1.7. Initial Snowflake Environment Setup.......................................................................................................23

#Step1: Create Roles...........................................................................................................................................23

#Step2: Define Role Hierarchy............................................................................................................................23

#Step3: Privs to ROLE_DBA.................................................................................................................................23

#Step4: Granting Role to Users...........................................................................................................................23

#Step5: Create Warehouse.................................................................................................................................24

#Step6: grant usage to the warehouse for the various roles..............................................................................25

Snowflake Best Practices

 #Step7: Lock down the modification of Snowflake provided warehouses..........................................................25

#Step8: Set the warehouse default by user........................................................................................................25

#Step9: Create Database....................................................................................................................................25

1.8. Snowflake Equivalent Data Type for SQL Server......................................................................................25

1.9. Snowflake Best Practices..........................................................................................................................26

Recommendation.............................................................................................................................................27
Use dedicated Virtual Warehouses.....................................................................................................................27

Scale up for known large workloads...................................................................................................................27

Scale out for unknown and unexpected workloads............................................................................................27

Design to maximize cache usage........................................................................................................................27

Use Cluster keys to partition large tables...........................................................................................................27

Role  Best Practices...........................................................................................................................................28

AccountAdmin    (atleast two people owns with MFA).......................................................................................28

SecurityAdmin....................................................................................................................................................28

Sysadmin.............................................................................................................................................................28

Public Role..........................................................................................................................................................29

Suggestion..........................................................................................................................................................29

 Reference....................................................................................................................................................29

To execute any snowflake DML command via procedure..................................................................................29

Create the task and resume................................................................................................................................31

1.10. Snowflake  Free Training......................................................................................................................31

Snowflake Best Practices

1. Introduction
1.1. Document Purpose
The purpose of the document is helpful for Snowflake basic command and Reference

1.2. Audience and Ownership

BI and Support Team.
The intended audience for the BI Naming Conventions is the ETL Developers, Tableau Developers, BI
project and Support team. The intended audience should understand from this document how
their development naming requirements will be met.

2. References
1.1. Glossary of Terms

Term / Acronym Definition

EDW Enterprise Data Warehouse
EDL Enterprise Data Lake
IDW Integration Data Warehouse
CDM common data model

1.1. Related Documents

Document Name Description

Size_dateType_Snowflake Size , Object Details
Information of SQL
Server

3. Snowflake Conventions

1.2. Meta Data Table

The below specified tables are available in EDW. AUDIT schema

Snowflake Best Practices

 No
. Meta Data Tables Comments
1 SQLSERVER_OBJECT_INFO SQL Server Object Information
2 SQLSERVER_TABLE_INFO SQL Server Table Count
3 SP_ERROR_LOG Stored Procedure Error Log
4 ETL_CONTROL_PARAM ETL Control Parameter
5 ETL_AUDIT_LOG ETL Audit Log

For Example:
To know number of views and corresponding snowflake database schema
select * from EDW.AUDIT.sqlserver_object_info
where databasename ='EDW'
and schemaname='ORCL'
and object_type='VIEW'

QA test Query to verify the records count information

EDL Salesforce
with b as (select * from edw.audit.sqlserver_table_info
where databasename = 'SFDC'
and schemaname='STG')
select distinct a.table_schema,a.table_name,a.row_count
snowflake_count,b.numberofrecords sqlserver_count, a.row_count-b.numberofrecords
from (select table_schema,table_name,row_count from
"EDL"."INFORMATION_SCHEMA"."TABLES"
where table_schema= 'SALESFORCE') a, b
where trim(b.tablename)=a.table_name
and b.snf_schemaname=a.table_schema
and a.row_count <> b.numberofrecords

IDW Stage
with b as (select * from edw.audit.sqlserver_table_info
where databasename = 'IDH_INTG'
and schemaname='JOBS')
select distinct a.table_schema,a.table_name,a.row_count
snowflake_count,b.numberofrecords sqlserver_count, a.row_count-b.numberofrecords

Snowflake Best Practices

from (select table_schema,table_name,row_count from
"IDW"."INFORMATION_SCHEMA"."TABLES"
where table_schema= 'JOB') a, b
where trim(b.tablename)=a.table_name
and b.snf_schemaname=a.table_schema

DDL of the table

To get the DDL
 select get_ddl('table', '<DB.SCHEMA.TABLE_NAME');

 select get_ddl(view, '<DB.SCHEMA.View_name>');

 select get_ddl(‘Procedure’, '<DB.SCHEMA.PROC_NAME>');

To Get all the objects from a DB

 select get_ddl(‘DATABASE’, '<SCHEMA_NAME>');

1.3. View Conversion

SQL Server View to Snowflake View
1. Replace ‘[‘,’]’ with blank value
2. Replace SQL schema name with snowflake database schema name
3. Check the snowflake function for corresponding SQL server function
4. Replace ‘+’ with ‘||’
For example the function such as
1. ISNULL to IFNULL
2. Getdate () to current_timestamp()

SQL Server View

The highlighted sub query statement does not support in snowflake. It needs to convert accordingly
and check the result.
 Steps to Convert the SQL Server Query to Snowflake Query.
1) In SQL Server While Creating a View you have to give Create View Syntax, While in Snowflake You
can use both Create View or Create or Replace View
2) In SQL Server if it is Null Function you can give it as ISNULL but in Snowflake if it is null function you
have to give it as IFNULL.
3) In SQL Server if it is Adding or Concatenating You have to give it as “+” BUT WHEN IT COMES TO
Snowflake you have to give it as “||”.
4) Replace [ or ] with “ value
5) Replace SQL schema name with SNOWFLAKE database schema name
6) While Creating Table all Column names should be in Upper Case.

Snowflake Best Practices

 7) In SQL Server WITH (NOLOCK) is required but in Snowflake it is not Required.
8) Replace Getdate() to current_timestamp()
9) In addition, in Snowflake, you cannot insert the value of current_timestamp() (there is no getdate()
function like in MS SQL) into a column with datetime datatype. This is because, in Snowflake,
DATETIME is an alias for TIMESTAMP_NTZ but current_timestamp returns TIMESTAMP_LTZ, which is
incompatible. You can specify TIMESTAMP_TYPE_MAPPING parameter to set it to LTZ by doing:
10) ALTER SESSION SET TIMESTAMP_TYPE_MAPPING=TIMESTAMP_LTZ;

CREATE VIEW [Stage].[vw_WorkerConcur] AS

SELECT ROW_NUMBER() OVER (ORDER BY COMBINED.WorkerTypeSort, COMBINED.AuditID) as

RowNumber,
CASE WHEN LoginId_Temp IS NULL THEN NewLoginID
ELSE LOWER(RTRIM(LoginId_Temp)) + '@guidewire.com' END AS LoginId,

COMBINED.*

FROM
( SELECT
EMP_ADT.worker_AuditID As AuditID,
EMP_ADT.IntegrationID,
WINDOW.StartRecord as winstartrecordid,
WINDOW.EndRecord as winendrecordid
, REPLACE(EMP_ADT.FirstName, '''', '') as FirstName
, REPLACE(EMP_ADT.MiddleName, '''', '') as MiddleName
, REPLACE(EMP_ADT.LastName, '''', '') as LastName
, EMP_ADT.WorkerID as WorkerID
--, LOWER(RTRIM(EMP_ADT.WorkEmailName)) + '@guidewire.com' as LoginId
,(SELECT MAX(WorkEmailName)
FROM Stage.Worker_Audit
WHERE worker_AuditID < WINDOW.StartRecord
AND WorkerID = EMP_ADT.WorkerID
AND worker_AuditID IN (SELECT MAX(worker_AuditID)
FROM Stage.Worker_Audit
WHERE WorkerID = EMP_ADT.WorkerID
AND worker_AuditID < WINDOW.StartRecord
AND Concur_ProcessedFlag = 'P')) AS LoginId_Temp
, RTRIM(EMP_ADT.WorkEmailName) + '@guidewire.com' as EmailAddress
, CASE
WHEN EMP_ADT.RegionID = 'CH' and EMP_ADT.BusinessSiteID = 7 THEN 'CH'
ELSE BS.CountryCode
End as CtryCode
, CasE
WHEN BS.CountryCode = 'CA' THEN BS.CountryCode + '-' + BS.StateOrProvinceCode
ELSE NULL
END as CtrySubCode

Snowflake Best Practices

 , UPPER(EMP_ADT.ExpenseCurrencyID) as ReimbursementCrnCode
, CasE EMP_ADT.RecordStatus
WHEN 'Active' THEN 'Y'
WHEN 'On Leave' THEN 'Y'
WHEN 'Inactive' THEN 'N'
WHEN 'Terminated' THEN 'N'
END as Active -- For now, Only Active and On Leave employees are interfaced
(Check where clause). Other Status will be forced to NULL
, EMP_ADT.LegalEntityID as OrgUnit1
, EMP_ADT.DepartmentID as OrgUnit2
, PL.ProductLineCode as OrgUnit3
, EMP_ADT.RegionID as OrgUnit4
, SF.DepartmentParentID as Custom7 --Function Code
, D.DepartmentParentID as Custom8 --Sub Function Code
, RSR.RegionParentID as Custom11 -- GeoName
, RC.RegionParentID as Custom12 -- Sub-Region
, EMP_ADT.WorkerType as Custom19 --WorkerType
, ISNULL(ECA.Allowance,0) as Custom20 -- Employee Cell Phone Allowance
, CASE
WHEN EMP_ADT.RegionID = 'CH' and EMP_ADT.BusinessSiteID = 7 THEN 'CH'
ELSE BS.CountryCode
End as Custom21
, CASE WHEN EMP_ADT.JobTitle = 'Chief Executive Officer' THEN ( SELECT
MAX(WorkerID)
FROM Stage.Worker
WHERE RecordStatus IN ('Active',
'On Leave') AND JobTitle = 'Chief Financial Officer')
WHEN MGR.JobTitle = 'Chief Executive Officer' AND
D.DefaultExpenseApplication = 'Concur' AND
BS.DefaultExpenseApplication = 'Concur'
THEN ( SELECT MAX(WorkerID)
FROM Stage.Worker
WHERE RecordStatus IN ('Active', 'On
Leave') AND JobTitle = 'Chief Financial Officer')
WHEN D.DefaultExpenseApplication = 'Concur' AND
BS.DefaultExpenseApplication = 'Concur' THEN RTRIM(EMP_ADT.ManagerID)
ELSE ''
END as WorkerIDOfTheExpenseReportApprover
, CASE WHEN BS.DefaultExpenseApplication = 'Concur' THEN 'Y'
ELSE 'N'
END as Approver
, CASE WHEN EMP_ADT.HomeCountry = 'DK' THEN 'N' -- added on 01/15/2018 based on
CHG0032418
WHEN BS.DefaultTravelApplication = 'Concur' THEN 'Y'
ELSE 'N'
END as TripUser --all employees for US & CA site are by default are Travel
Users
, CASE WHEN BS.DefaultExpenseApplication = 'Concur' AND
D.DefaultExpenseApplication = 'Concur' THEN 'Y'
WHEN ISNULL ((SELECT

Snowflake Best Practices

 (SELECT COUNT(*)
FROM Stage.Worker DRE
JOIN Master.Department DRD
ON (DRE.DepartmentID = DRD.DepartmentID AND
DRD.DefaultExpenseApplication = 'Concur')
JOIN Master.BusinessSite DRB
ON (DRE.BusinessSiteID = DRB.BusinessSiteID AND
DRB.DefaultExpenseApplication = 'Concur')
WHERE DRE.ManagerID = EMP_ADT.WorkerID
AND DRE.RecordStatus In ('Active', 'On Leave'))
/*+
(SELECT COUNT(*)
FROM dbo.Contractors DRC
JOIN dbo.Department DRCD ON (DRC.DepartmentID = DRCD.DepartmentID
AND DRCD.DefaultExpenseApplication = 'Concur')
WHERE DRC.ManagerID = EMP_ADT.WorkerID
AND DRC.RecordStatus In ('Active', 'On Leave'))*/
), 0) > 0 THEN 'Y'
ELSE 'N' END as ExpenseUser
--, CASE ISNULL(( SELECT WorkEmailName FROM Stage.Worker_Audit
-- WHERE worker_AuditID < WINDOW.StartRecord
-- AND WorkerID = EMP_ADT.WorkerID
-- AND worker_AuditID IN (SELECT MAX(worker_AuditID)
-- FROM Stage.Worker_Audit
-- WHERE WorkerID = EMP_ADT.WorkerID
-- AND worker_AuditID < WINDOW.StartRecord)), '~')
-- WHEN EMP_ADT.WorkEmailName THEN NULL
-- WHEN '~' THEN NULL
-- ELSE EMP_ADT.WorkEmailName END as NewLoginID
, LOWER(RTRIM(EMP_ADT.WorkEmailName)) + '@guidewire.com' as NewLoginID
, 0 as WorkerTypeSort
FROM Stage.Worker_Audit EMP_ADT WITH (NOLOCK)
JOIN (SELECT MIN(Worker_auditID) StartRecord, MAX(Worker_auditID) EndRecord
FROM Stage.Worker_Audit
WHERE Concur_ProcessedFlag IS NULL AND WorkerType = 'Employee') WINDOW
ON (Worker_AuditID BETWEEN WINDOW.StartRecord AND WINDOW.EndRecord)
LEFT JOIN Stage.Worker MGR
ON EMP_ADT.ManagerID = MGR.WorkerID
LEFT JOIN Master.BusinessSite BS
ON (EMP_ADT.BusinessSiteID = BS.BusinessSiteID)
LEFT JOIN Master.ProductLine PL
ON (EMP_ADT.ProductLineID = PL.ProductLineID)
LEFT JOIN Master.Region RC
ON (EMP_ADT.RegionID = RC.RegionID AND RC.RegionLevelName = 'Country')
LEFT JOIN Master.Region RSR
ON (RC.RegionParentID = RSR.RegionID AND RSR.RegionLevelName = 'Sub-
Region')
--LEFT JOIN dbo.DepartmentHierarchy_Temp DH ON (EMP.DepartmentID =
DH.DepartmentID)
LEFT JOIN Master.Department D

Snowflake Best Practices

 ON (EMP_ADT.DepartmentID = D.DepartmentID AND D.DepartmentLevelName =
'Department')
LEFT JOIN Master.Department SF
ON (D.DepartmentParentID = SF.DepartmentID AND SF.DepartmentLevelName =
'Sub-Function')
LEFT JOIN Master.EmployeeCellPhoneAllowance ECA
ON (EMP_ADT.WorkerID = ECA.EmployeeID)
WHERE EMP_ADT.Concur_ProcessedFlag IS NULL
AND EMP_ADT.RecordStatus IS NOT NULL
AND EMP_ADT.WorkerType = 'Employee'
AND EMP_ADT.worker_AuditID IN (SELECT MAX(worker_AuditID)
FROM Stage.Worker_Audit
WHERE Concur_ProcessedFlag IS NULL AND worker_AuditID
<= WINDOW.EndRecord
GROUP BY WorkerID)
)COMBINED

Snowflake View

CREATE OR REPLACE VIEW IDW.Stage.vw_WorkerConcur AS

SELECT ROW_NUMBER() OVER (ORDER BY COMBINED.WorkerTypeSort, COMBINED.AuditID) as
RowNumber,
CASE WHEN LoginId_Temp IS NULL THEN NewLoginID
ELSE LOWER(RTRIM(LoginId_Temp)) || '@guidewire.com' END AS LoginId,

COMBINED.*

FROM
(SELECT
EMP_ADT.worker_AuditID As AuditID,
EMP_ADT.IntegrationID,
WINDOW.StartRecord as winstartrecordid,
WINDOW.EndRecord as winendrecordid
, REPLACE(EMP_ADT.FirstName, '''', '') as FirstName
, REPLACE(EMP_ADT.MiddleName, '''', '') as MiddleName
, REPLACE(EMP_ADT.LastName, '''', '') as LastName
, EMP_ADT.WorkerID as WorkerID
--, LOWER(RTRIM(EMP_ADT.WorkEmailName)) || '@guidewire.com' as LoginId
,(SELECT MAX(WorkEmailName)
FROM IDW.STAGE.Worker_Audit
WHERE worker_AuditID < WINDOW.StartRecord
AND WorkerID = EMP_ADT.WorkerID
AND worker_AuditID IN (SELECT MAX(worker_AuditID)
FROM IDW.STAGE.Worker_Audit
WHERE WorkerID = EMP_ADT.WorkerID
AND worker_AuditID < WINDOW.StartRecord
AND Concur_ProcessedFlag = 'P')) AS LoginId_Temp

Snowflake Best Practices

 , RTRIM(EMP_ADT.WorkEmailName) || '@guidewire.com' as EmailAddress
, CASE
WHEN EMP_ADT.RegionID = 'CH' and EMP_ADT.BusinessSiteID = 7 THEN 'CH'
ELSE BS.CountryCode
End as CtryCode
, CasE
WHEN BS.CountryCode = 'CA' THEN BS.CountryCode || '-' ||
BS.StateOrProvinceCode
ELSE NULL
END as CtrySubCode
, UPPER(EMP_ADT.ExpenseCurrencyID) as ReimbursementCrnCode
, CasE EMP_ADT.RecordStatus
WHEN 'Active' THEN 'Y'
WHEN 'On Leave' THEN 'Y'
WHEN 'Inactive' THEN 'N'
WHEN 'Terminated' THEN 'N'
END as Active -- For now, Only Active and On Leave employees are interfaced
(Check where clause). Other Status will be forced to NULL
, EMP_ADT.LegalEntityID as OrgUnit1
, EMP_ADT.DepartmentID as OrgUnit2
, PL.ProductLineCode as OrgUnit3
, EMP_ADT.RegionID as OrgUnit4
, SF.DepartmentParentID as Custom7 --Function Code
, D.DepartmentParentID as Custom8 --Sub Function Code
, RSR.RegionParentID as Custom11 -- GeoName
, RC.RegionParentID as Custom12 -- Sub-Region
, EMP_ADT.WorkerType as Custom19 --WorkerType
, IFNULL(ECA.Allowance,0) as Custom20 -- Employee Cell Phone Allowance
, CASE
WHEN EMP_ADT.RegionID = 'CH' and EMP_ADT.BusinessSiteID = 7 THEN 'CH'
ELSE BS.CountryCode
End as Custom21
, CASE WHEN EMP_ADT.JobTitle = 'Chief Executive Officer' THEN ( SELECT
MAX(WorkerID)
FROM IDW.STAGE.Worker
WHERE RecordStatus IN ('Active',
'On Leave') AND JobTitle = 'Chief Financial Officer')
WHEN MGR.JobTitle = 'Chief Executive Officer' AND
D.DefaultExpenseApplication = 'Concur' AND
BS.DefaultExpenseApplication = 'Concur'
THEN ( SELECT MAX(WorkerID)
FROM IDW.STAGE.Worker
WHERE RecordStatus IN ('Active', 'On
Leave') AND JobTitle = 'Chief Financial Officer')
WHEN D.DefaultExpenseApplication = 'Concur' AND
BS.DefaultExpenseApplication = 'Concur' THEN RTRIM(EMP_ADT.ManagerID)
ELSE ''
END as WorkerIDOfTheExpenseReportApprover
, CASE WHEN BS.DefaultExpenseApplication = 'Concur' THEN 'Y'
ELSE 'N'

Snowflake Best Practices

 END as Approver
, CASE WHEN EMP_ADT.HomeCountry = 'DK' THEN 'N' -- added on 01/15/2018 based on
CHG0032418
WHEN BS.DefaultTravelApplication = 'Concur' THEN 'Y'
ELSE 'N'
END as TripUser --all employees for US & CA site are by default are Travel
Users
, CASE WHEN BS.DefaultExpenseApplication = 'Concur' AND
D.DefaultExpenseApplication = 'Concur' THEN 'Y'
WHEN IFNULL ((SELECT
(SELECT COUNT(*)
FROM IDW.STAGE.Worker DRE
JOIN IDW.MASTER.Department DRD
ON (DRE.DepartmentID = DRD.DepartmentID AND
DRD.DefaultExpenseApplication = 'Concur')
JOIN IDW.MASTER.BusinessSite DRB
ON (DRE.BusinessSiteID = DRB.BusinessSiteID AND
DRB.DefaultExpenseApplication = 'Concur')
WHERE DRE.ManagerID = EMP_ADT.WorkerID
AND DRE.RecordStatus In ('Active', 'On Leave'))
), 0) > 0 THEN 'Y'
ELSE 'N' END as ExpenseUser
, LOWER(RTRIM(EMP_ADT.WorkEmailName)) || '@guidewire.com' as NewLoginID
, 0 as WorkerTypeSort
FROM IDW.STAGE.Worker_Audit EMP_ADT --WITH (NOLOCK)
JOIN (SELECT MIN(Worker_auditID) StartRecord, MAX(Worker_auditID) EndRecord
FROM IDW.STAGE.Worker_Audit
WHERE Concur_ProcessedFlag IS NULL AND WorkerType = 'Employee') WINDOW
/** ( SELECT (select Worker_auditID from IDW.STAGE.Worker_Audit
WHERE Concur_ProcessedFlag IS NULL AND WorkerType = 'Employee'
order by Worker_auditID limit 1
) StartRecord, (select Worker_auditID from
IDW.STAGE.Worker_Audit
WHERE Concur_ProcessedFlag IS NULL AND WorkerType = 'Employee'
order by Worker_auditID desc limit 1
) EndRecord ) WINDOW **/
ON (Worker_AuditID BETWEEN WINDOW.StartRecord AND WINDOW.EndRecord)
LEFT JOIN IDW.STAGE.Worker MGR
ON EMP_ADT.ManagerID = MGR.WorkerID
LEFT JOIN IDW.MASTER.BusinessSite BS
ON (EMP_ADT.BusinessSiteID = BS.BusinessSiteID)
LEFT JOIN IDW.MASTER.ProductLine PL
ON (EMP_ADT.ProductLineID = PL.ProductLineID)
LEFT JOIN IDW.MASTER.Region RC
ON (EMP_ADT.RegionID = RC.RegionID AND RC.RegionLevelName = 'Country')
LEFT JOIN IDW.MASTER.Region RSR
ON (RC.RegionParentID = RSR.RegionID AND RSR.RegionLevelName = 'Sub-
Region')
--LEFT JOIN dbo.DepartmentHierarchy_Temp DH ON (EMP.DepartmentID =
DH.DepartmentID)

Snowflake Best Practices

 LEFT JOIN IDW.MASTER.Department D
ON (EMP_ADT.DepartmentID = D.DepartmentID AND D.DepartmentLevelName =
'Department')
LEFT JOIN IDW.MASTER.Department SF
ON (D.DepartmentParentID = SF.DepartmentID AND SF.DepartmentLevelName =
'Sub-Function')
LEFT JOIN IDW.MASTER.EmployeeCellPhoneAllowance ECA
ON (EMP_ADT.WorkerID = ECA.EmployeeID)
WHERE EMP_ADT.Concur_ProcessedFlag IS NULL
AND EMP_ADT.RecordStatus IS NOT NULL
AND EMP_ADT.WorkerType = 'Employee'
AND EMP_ADT.worker_AuditID IN (SELECT worker_AuditID
from (select row_number() over (partition by
WorkerID order by worker_auditid desc) as max_work,workerid, worker_auditid
FROM IDW.STAGE.Worker_Audit
WHERE Concur_ProcessedFlag IS NULL AND worker_AuditID
<= (select Worker_auditID from IDW.STAGE.Worker_Audit
WHERE Concur_ProcessedFlag IS NULL AND WorkerType = 'Employee'
order by Worker_auditID desc limit 1 ))
where max_work=1)
) COMBINED
--ORDER BY COMBINED.WorkerTypeSort, COMBINED.AuditID

1.4. Snowflake Procedure:

The procedure details extract from snowflake:
show procedures.
desc procedure edl.coupa.SP_ANALYTICSFILEDETAILSINSERT(VARCHAR, VARCHAR);

Table information for error capture of the procedure execution

-----------------------------------------------------------------
create or replace TABLE SP_ERROR_LOG (
LOG_ID NUMBER(20,0) autoincrement,
ERROR_CODE NUMBER(10,0),
ERROR_STATE VARCHAR(200),
ERROR_MESSAGE VARCHAR(16777216),
STACK_TRACE VARCHAR(16777216),
PROCEDURENAME VARCHAR(200),
CREATED_DATE TIMESTAMP_NTZ(9) DEFAULT CURRENT_TIMESTAMP(),
CREATED_BY VARCHAR(100) DEFAULT CURRENT_USER(),
DATABASE_NAME VARCHAR(100) DEFAULT CURRENT_DATABASE(),
SCHEMA_NAME VARCHAR(100) DEFAULT CURRENT_SCHEMA(),
SOURCE_SYSTEM VARCHAR(100),
COMMENTS VARCHAR(4000)
);

Snowflake Best Practices

Procedure for Error Log
CREATE OR REPLACE PROCEDURE "SP_ERROR_LOG"(PROCEDURENAME VARCHAR, SOURCESYSTEM
VARCHAR, ERRORCODE VARCHAR, ERRORSTATE VARCHAR, ERRORMESSAGE VARCHAR,
ERRORTRACETEXT VARCHAR)
RETURNS VARCHAR
LANGUAGE JAVASCRIPT
EXECUTE AS OWNER
AS
$$

/**
* Stored procedure to capture the error message in the procedure
 * Note that this procedure is required  the 6 different values
 *This help to identify the procedure error
 * PROCEDURENAME   -->  name of the procedure  e.g. SP_ERROR_LOG
* SORUCESYEM   -->  name of the source system  e.g. COUPA , ORACLE , SALESFORCE
* ERRORCODE, ERRORSTATE, ERRORMESSAGE, and ERRORTRACETEXT -- > system Generated
 *  Usage 
 * It is useful in Catch error
* CALL SP_ERROR_LOG('SPTEST','COUPA',err.code, err.state, err.message, err.stackTraceTxt)
* CALL SP_ERROR_LOG('SPTEST','COUPA','100183','P0000','SQL Complilation Error','At
Statement')

*/
   var result =''
//Captures the procedure error log into the EDW.AUDIT.SP_ERROR_LOG table -- 13 July 2020
// Removed the schema name and database name as defined in default values
var v_sql_command = `insert into EDW.AUDIT.SP_ERROR_LOG(
PROCEDURENAME,
SOURCE_SYSTEM,
ERROR_CODE,
ERROR_STATE,
ERROR_MESSAGE,
STACK_TRACE) VALUES (:1,:2,:3,:4,:5,:6);`;
var v_rs = snowflake.createStatement({sqlText: v_sql_command,

Snowflake Best Practices

 binds: [PROCEDURENAME, SOURCESYSTEM,ERRORCODE,
ERRORSTATE, ERRORMESSAGE, ERRORTRACETEXT]}).execute();
result ='Yes'
return result;

$$
;

1.5. SQL Server Meta Data Query Reference

SQL server meta data reference queries that helps to get the meta data and dependency
information

SQL Server Meta Query

SELECT count(a.name) cnt,'No of Tables' Type,b.name FROM sys.objects a,
sys.schemas b
WHERE a.type = 'U' and a.schema_id =b.schema_id
group by b.name

union

SELECT count(a.name) cnt,'No of Views' Type,b.name FROM sys.objects a, sys.schemas

b
WHERE a.type = 'V' and a.schema_id =b.schema_id
group by b.name

union

SELECT count(a.name) cnt ,'No of Procedures' Type,b.name FROM sys.objects a,

sys.schemas b
WHERE a.type = 'P' and a.schema_id =b.schema_id
group by b.name
union
SELECT count(a.name) cnt ,'No of Funcitons' Type,b.name FROM sys.objects a,
sys.schemas b
WHERE a.type = 'FN' and a.schema_id =b.schema_id
group by b.name
order by b.name,type

Number of the tables and columns

use Empire;
sp_helpdb Empire;
sp_spaceused ;

Snowflake Best Practices

select count(distinct a.name), b.name from sys.tables a, sys.schemas b
where a.schema_id =b.schema_id
and a.name not like '%BKP%'
group by b.name
and a.name like '%BKP%'

select t.name,count( col.column_id)

from sys.tables as tab
left join sys.columns as col
on tab.object_id = col.object_id
left join sys.types as t
on col.user_type_id = t.user_type_id

select schema_name(tab.schema_id) + '.' + tab.name as [table],

sum(part.rows) as [rows]
from sys.tables tab
inner join sys.partitions part
on tab.object_id = part.object_id
where part.index_id IN (1, 0) -- 0 - table without PK, 1 table with PK
group by schema_name(tab.schema_id) + '.' + tab.name
order by sum(part.rows) desc

Database Size
use Empire;
sp_helpdb Empire;
sp_spaceused ;

SELECT DB_NAME() AS DbName,

name AS FileName,
size/128.0 AS CurrentSizeMB,
size/128.0 - CAST(FILEPROPERTY(name, 'SpaceUsed') AS INT)/128.0 AS FreeSpaceMB
FROM sys.database_files;

To know more information of the depend objects of the tables

use IDH_INTG
GO
sp_depends 'Stage.sp_WorkdayWorkerTransformer'
GO
-- Method 1:Using sp_depends
sp_depends 'stage.Worker'
GO
-- Method 2:Using information_schema.routines
SELECT *
FROM information_schema.routines ISR
WHERE CHARINDEX('stage.Worker', ISR.ROUTINE_DEFINITION) > 0
GO
-- Method 3:Using DMV sys.dm_sql_referencing_entities

Snowflake Best Practices

SELECT referencing_schema_name, referencing_entity_name,
referencing_id, referencing_class_desc, is_caller_dependent
FROM sys.dm_sql_referencing_entities ('stage.Workday_Worker', 'OBJECT');
GO

XML Process name from Processor Log

SELECT *
FROM [IDH_INTG].[Stage].[ProcessorLog]
where recordid in (select max(recordid) from stage.ProcessorLog
group by ProcessorName )
order by ProcessorName

select * from stage.ProcessorLog

where processorname ='sp_WorkerSyncReportInsert'
and RecordID=177472

Object Type information

use EDW
SELECT b.name schemaname,a.name ,case when a.type ='FN' then 'Function'
when a.type='P' then 'Procedure'
when a.type ='V' then 'View'
when a.type ='TR' then 'Trigger'
when a.type='U' then 'Table'
when a.type='IT' then 'Internal Table '
when a.type='TF' then 'SQL table valued Function'
when a.type='ET' then 'External Table '
when a.type='PK' then 'Primary Key '
else a.type
end
objecttype FROM sys.objects a, sys.schemas b
WHERE a.schema_id =b.schema_id
and a.type <> 'S'
order by b.name,a.type,a.name

Reference URL
SQL SERVER- OBJECT DEPENDENCIES
The metadata objects information
 METADATA_OBJECTS

1.6. Snowflake Reference

The specified url link is useful to get more information
 Create Procedure

Snowflake Best Practices

  Setup the Stream

 Create procedure and task usage

 Task Scheduling

 Stream and Task Info for Trigger

Parameter Setup
 The Above link to check the same queue type scenario to update the parameter instead of
default .

Changing warehouse parameter, It is required for Sysadmin Role

For Example
ALTER WAREHOUSE s_WH_IDW SET STATEMENT_QUEUED_TIMEOUT_IN_SECONDS =120
STATEMENT_QUEUED_TIMEOUT_IN_SECONDS = 60 seconds or 120 seconds instead of 0 to
clear the queue

MAX_CONCURRENCY_LEVEL  it automatically has taken care when it is in

multicluster environment

 STATEMENT_TIMEOUT_IN_SECONDS  172800 (i.e 2 days ).. if it is required to decrease to

cancel the dml statement time in seconds
Statement_Queued_Timeout_In_Seconds
 To  configure the parameter based upon the below document reference  and try to
reprocess 

Snowflake Best Practices

 JDBC_DRIVER

HANDLING_JDBC_DRIVER

 Cloning the database

AP Payment XML Format Query

The sample XML Query has specified below

<ApPayments>
<TransactionId>202006221400123524</TransactionId>
<ApPayment>
<PAYMENT_ID>969245</PAYMENT_ID>
<PaymentKey>81636</PaymentKey>
<VendorId>E-100333</VendorId>
<TotalAmount>30.7</TotalAmount>
<PaymentDate>2020-06-22T00:00:00.000+00:00</PaymentDate>
<PaymentMethod>EFT</PaymentMethod>
<TransactionState>NEGOTIABLE</TransactionState>
<Currency>EUR</Currency>
<LineItems>
<PAYMENT_ID>969245</PAYMENT_ID>
<LineItem>
<BillNo>FF-a1t2T000004OfMYQA0-101526</BillNo>
<LineItemKey>2753089</LineItemKey>
<TransactionAmount>30.7</TransactionAmount>
<SourceSystem>FF</SourceSystem>
</LineItem>
</LineItems>
</ApPayment>

XML Format to load the data

Snowflake Best Practices

 - To get the table script

select * from edl.information_schema.tables

where table_name ='XML_APPAYMENT'

select get_ddl('table', 'EDL.WORKDAY.XML_APPAYMENT');

create or replace TABLE XML_APPAYMENT (

PROCESSEDDATA VARIANT
);

ALTER FILE FORMAT "IDW"."STAGE".XML_FROMAT SET COMPRESSION = 'AUTO' PRESERVE_SPACE = FALSE

STRIP_OUTER_ELEMENT = FALSE DISABLE_SNOWFLAKE_DATA = FALSE DISABLE_AUTO_CONVERT = FALSE
IGNORE_UTF8_ERRORS = FALSE;

Snowflake query to flatten

SELECT xmlget(ProcessedData,'PaymentKey'):"$"::NUMERIC AS PAYMENTKEY,
xmlget(lineitems.value,'PAYMENT_ID'):"$"::NUMERIC AS PAYMENT_ID,
xmlget(ProcessedData,'VendorId'):"$"::String AS VENDORID,
xmlget(lineitem.value,'BillNo'):"$"::String AS BILLNO,
xmlget(lineitem.value,'LineItemKey'):"$"::NUMERIC AS LineItemKey,
xmlget(lineitem.value,'TransactionAmount'):"$"::NUMERIC AS
TransactionAmount,
xmlget(lineitem.value,'BillNo'):"$"::String AS SourceSystem,
CURRENT_USER() AS CREATEDBY,
current_Date() AS CREATEDDATE
FROM XML_APPAYMENT
, lateral FLATTEN(ProcessedData:"$") lineitems
, lateral FLATTEN(lineitems.value:"$") lineitem
where lineitems.value like '<LineItems%'
and lineitem.value like '<LineItem%'
and xmlget(ProcessedData,'PaymentKey'):"$"::NUMERIC = 81594
order by Paymentkey, Payment_id

1.7. Initial Snowflake Environment Setup

Some useful commands to setup the environment

#Step1: Create Roles

USE ROLE SECURITYADMIN; -- USERADMIN

CREATE ROLE ROLE_DBA;

CREATE ROLE ROLE_ETL;
CREATE ROLE ROLE_READONLY;
 

Snowflake Best Practices

#Step2: Define Role Hierarchy
 
USE ROLE SECURITYADMIN;

GRANT ROLE ROLE_ETL TO ROLE ROLE_DBA;

GRANT ROLE ROLE_READONLY TO ROLE ROLE_DBA;
GRANT ROLE ROLE_DBA TO ROLE SYSADMIN ;
 
#revoke role SYSADMIN from role ROLE_DBA;
 
#Step3: Privs to ROLE_DBA
USE ROLE SYSADMIN;

GRANT CREATE DATABASE ON ACCOUNT TO ROLE ROLE_DBA;

GRANT ALL ON ACCOUNT TO ROLE ROLE_ETL;

#ALL [ PRIVILEGES ] Grant all privileges, except OWNERSHIP, on the table.

 
 USE ROLE SECURITYADMIN;
#grant monitor,operate,usage on warehouse MY_WH to role OBJ_MY_DB_READ;
grant insert,select , update , delete ,truncate on future tables in database EDL to role
role_etl;
 
 
 
#Step4: Granting Role to Users
USE ROLE SECURITYADMIN;
GRANT ROLE ROLE_DBA TO USER NDINAMANI;
GRANT ROLE ROLE_ETL TO USER NDINAMANI;
GRANT ROLE ROLE_READONLY TO USER NDINAMANI;
 
 
#default role for these users
 
USE ROLE SECURITYADMIN;
ALTER USER NDINAMANI SET DEFAULT_ROLE = ROLE_DBA;
 

Snowflake Best Practices

#Step5: Create Warehouse
-- Note: Do not grant Warehouse creation to DBA
USE ROLE ROLE_DBA; -- SECURITYADMIN

CREATE WAREHOUSE "s_WH_EDW" WITH

WAREHOUSE_SIZE = 'SMALL'
WAREHOUSE_TYPE = 'STANDARD'
AUTO_SUSPEND = 600
AUTO_RESUME = TRUE
COMMENT = 'Use for all ETL, ELT while loading data to EDW';

CREATE WAREHOUSE "s_WH_SOURCE_EXTRACT" WITH

WAREHOUSE_SIZE = 'SMALL'
WAREHOUSE_TYPE = 'STANDARD'
AUTO_SUSPEND = 600
AUTO_RESUME = TRUE
COMMENT = 'Use for extracting raw data';

CREATE WAREHOUSE "s_WH_DATA_ANALYTICS" WITH

WAREHOUSE_SIZE = 'SMALL'
WAREHOUSE_TYPE = 'STANDARD'
AUTO_SUSPEND = 600
AUTO_RESUME = TRUE
COMMENT = 'Use for BI Tools to run analytics query';

CREATE WAREHOUSE "s_WH_IDW" WITH

WAREHOUSE_SIZE = 'SMALL'
WAREHOUSE_TYPE = 'STANDARD'
AUTO_SUSPEND = 600
AUTO_RESUME = TRUE
COMMENT = 'Use for Integration Tasks';
 
 
 

#Step6: grant usage to the warehouse for the various roles

  
USE ROLE DBA;
GRANT USAGE, OPERATE ON WAREHOUSE "s_WH_EDW" TO ROLE ROLE_ETL;
GRANT USAGE, OPERATE ON WAREHOUSE "s_WH_SOURCE_EXTRACT" TO ROLE ROLE_ETL;
 
#Step7: Lock down the modification of Snowflake provided warehouses.
 
REVOKE MODIFY ON WAREHOUSE DEMO_WH FROM ROLE PUBLIC;
REVOKE MODIFY ON WAREHOUSE LOAD_WH FROM ROLE PUBLIC;

Snowflake Best Practices

 
#Step8: Set the warehouse default by user
 
USE ROLE SECURITYADMIN;

ALTER USER NDINAMANI SET DEFAULT_WAREHOUSE = "s_WH_EDW";

 
#Step9: Create Database
 USE ROLE ROLE_DBA;
CREATE OR REPLACE TRANSIENT DATABASE EDL;
GRANT USAGE,MODIFY, MONITOR ON DATABASE EDL TO ROLE ROLE_ETL;
GRANT USAGE MODIFY, MONITOR ON DATABASE EDL TO ROLE ROLE_READONLY;

1.8. Snowflake Equivalent Data Type for SQL Server

The below highlighted are required to change

SQL Server Data SNOWFLAKE

No Type DataType Comments
best practice, number(20) instead of default
1
Bigint number(20) number
2 Char Varchar it depends upon the existing data size
3 Date Date
snowflake automatically converts
4
Datetime Datetime timestamp_ntz
5 Decimal Decimal snowflake automatically converts number
6 Float number(38,10) best practice , number(32,10) instead of float
best practice, number(10) instead of default
7
Int number(10) number
8 Ntext Varchar
9 Numeric Numeric snowflake automatically converts number
10 Nvarchar Varchar it depends upon the existing data size

Snowflake Best Practices

 11 Varchar Varchar it depends upon the existing data size
12 Smalldatetime Datetime  
13 Text Varchar it depends upon the existing data size
14 Bit number(1,0) it has the value of 1,0 or null
15 Money number(20,4) it depends on the scale and precision
16 Smallmoney number(12,4) it depends on the scale and precision
17 Sysname varchar(1020)
best practice, number(4) instead of default
18
Tinyint number(4,0) number
19 varchar(max) Varchar  
20 Xml Variant  
21 Varbinary vARBINARY  
22 Smallint number(6)  

1.9. Snowflake Best Practices

Microsoft Excel
Worksheet

Recommendation
 Use always Upper in both sides for validations in query

Use dedicated Virtual Warehouses

o Segregate the Virtual warehouses for different workloads

 Identify and categorize workloads on ETL, Data Science , Business Intelligence ,
Adhoc , and etc.
 Identify and categorize the users executing those workloads
 Each workload classification and attached to your users default virtual
warehouse for their queries

Snowflake Best Practices

  This classification of the workloads and users will need to be depend upon the
workload patterns

Scale up for known large workloads

o Scale the size the virtual warehouses according to known patterns

 This can be adhoc events ( e.g. urgent business requirement results in the rush of
complex queries )
 A workload occurs repeatedly occurring ( ETL runs non peak hours  )
 Number of the users performing complex queries increase exponentially

Scale out for unknown and unexpected workloads

o Auto spawn virtual warehouse based on workloads

 Multi Cluster warehouse can be scaled out automatically on periods of demand
and scaled back as the demand decrease

Design to maximize cache usage

o Caching is automatic , but usage can be maximized

 Results are cached for 24 hours ( form the point of last reuse )
 Allocate virtual warehouse to run common queries on same warehouse

Use Cluster keys to partition large tables

o For tables that have the large Data warehouse cluster keys to improve query
performance
 For very large tables ( multi-terabyte size ) , Cluster keys which are frequency
used in WHERE clauses or are frequently used in JOINS or ORDER BY statements

Role  Best Practices

AccountAdmin    (atleast two people owns with MFA)

 View and manage all objects

Snowflake Best Practices

  View billing and credit data
 Stop any SQL statements
 Perform other account level activities like creating reader account

SecurityAdmin 

 Create and Managed Users -- Useradmin

 Create and Managed Roles --  Useradmin
 Assign privileges to the Roles
 Grant , Revoke any privilege
 Modify and Monitor Sessions

Sysadmin

 Create and Manage Warehouses

 Create and Manage Databases , Tables and other objects

 Maintain Role Hierarchy and custom role directly and indirectly given to Sysadmin
 Other the sysadmin role will not able to administer all the objects
 If you explicitly want to limit privilege to only the owning user then do not assign to Sysadmin

It is best practices to create the hierarchy of roles leading upto Sysadmin

Public Role

 A role automatically given to all users

 The least privileged role as every one has this access
 The role can own objects
 Objects that are created using Public role automatically available to everyone in the system

Suggestion

 Developer account needs to have the privilege of create or drop table such as temporary table

Snowflake Best Practices

  User admin is required to create role and create user
 ETL_Admin user will have the ROLE_DBA privilege.
 ETL_Developer user will have the ROLE_ETL privilege part of the Role_DBA.

 Reference

To execute any snowflake DML command via procedure

create or replace procedure RunBatchSQL(sqlCommand String, continueOnError Boolean)
returns string
language JavaScript
as
$$
/**
* Stored procedure to execute multiple SQL statements generated from a SQL query
* Note that this procedure will always use the column named "SQL_COMMAND".
* This overload of the function will continue on errors if "continueOnError" = true.
*
* @param {String} sqlCommand: The SQL query to run to generate one or more SQL commands
 * @param {Boolean} continueOnError: If true, continues on error. If false, stops after first error.
* @return {String}: A string containing all the SQL commands executed, each separated by a
newline.
 */
     

cmd1_dict = {sqlText: SQLCOMMAND};

stmt = snowflake.createStatement(cmd1_dict);
rs = stmt.execute();
var s = '';
while (rs.next()) {
try{
/**cmd2_dict = {sqlText: rs.getColumnValue(1)};
stmtEx = snowflake.createStatement(cmd2_dict);
stmtEx.execute(); **/
s += rs.getColumnValue(1) + " --Succeeded" + "\n";
}
catch(err) {
s += rs.getColumnValue(1) + " --Failed: " + err.message.replace(/\n/g, " ")
+ "\n";
if (!CONTINUEONERROR) return s;

Snowflake Best Practices

 }
}
return s;
$$
;

-- As a convenience, this grabs the last SQL run so that it's easier to insert into the parameter used to
call the stored procedure.
set query_text = (select 'INSERT INTO sqlserver_table_audit
select databasename, schemaname, tablename,numberofrecords,
snf_databasename, snf_schemaname,insert_date,null status ,current_timestamp() modify_date
from sqlserver_check');
 
-- Confirm that the query_text variable has the correct SQL query to generate our SQL commands
(grants in this case) to run.
select $query_text as QUERY_TEXT; 

--Run the stored procedure. Note that to view its output better, double click on the output to see
it in multi-line format,
Call RunBatchSQL($query_text, true);

select QUERY_TEXT
from table(information_schema.query_history(result_limit => 2))
where SESSION_ID = Current_Session() and QUERY_TYPE = 'SELECT' order by
START_TIME desc
call RunBatchSQL();
SHOW FUTURE GRANTS IN DATABASE EDL;

Create the task and resume

CREATE OR REPLACE TASK PUBLIC.TASK_SQL
WAREHOUSE = "s_WH_SOURCE_EXTRACT"
SCHEDULE = "10 MINUTE"
WHEN
SYSTEM$STREAM_HAS_DATA('SQLSERVER_CHECK') = True
AS
CALL EDL.PUBLIC.RunBatchSQL($query_text, true);

ALTER TASK EDL.PUBLIC.TASK_SQL RESUME;

1.10. Snowflake  Free Training

Please Click below URL and Register Snowflake Lodge Login..   It is free and get more information
insight of Snowflake. 

Snowflake Best Practices

 Step 1
https://snowflakeuniversity.mindtickle.com/

Step 2   Register Snowflake Lodge Login

Step 3  :  Below Left side topic , Go through each topic if you have time

Guidewire: Qlik to Tableau Migration:

Microsoft Excel
Worksheet

-> ISSUE FACED IN INFORMATICA CLOUD:

1. No of Parallel task-run depends on the license. This will help in processing no of mappings.
2. heap-memory in Informatica needs to be increased to process huge volume of data.

Snowflake Best Practices

 3. Sequence transformation for identity column. As identity won't work similar to other relational
DB ,
4. It won't re-set automatically if you need to truncate the table.

-> ISSUE FACED IN SNOWFLAKE:

1. Snowflake use batch process, Not record by record process like traditional DB .So in some condition if
you are concatenating datetime field to get primary key,
it won't work as expected as there is a chance that we will get dame datetime for multiple records.

2. Float data-type issue is a known issue. We can't be sure about the number of digits to the right of the
decimal point

3. Concat operate of SQL Server db (+) is equals to " || " in snowflake.

4. Leap year calculation in snowflake : MOD(DATE_PART(YEAR,V_SRC.TERM_START_DATE__C),4) = 0

5. Snowflake is considering 'Manual%' and 'MANUAL%' as two different values.

It means snowflake is case sensitive .

 ISSUE FACED WHILE TESTING:

 While Comparing Data between SQL and SNOWFLAKE there was lot of mismatches due to Data types.
 Source table has more no of rows Compare to Target table.
 Rows count were mismatching in Snowflake.
 Parallel Run Decision.

 Change in datatype.
 Object verification.
 Verifying views and check the rows.
 Checking minimum and maximum of column value.
 Check sum of columns value.
 Checking data sets
 Migration of values depending on other fields/tables present in the source database.
 Data in the source table may be present as a character, but in the target table the mapped column is an
integer.
 Concatenation of two or more columns.
 There were some structure changes between the source and target database tables which increased the
testing challenge. For example, the source table columns were directly mapped to the primary keys from
master tables with common data types such as varchar or int whereas in the target database, along with
common data types, they also had auto generated GUIDs for each row of table.

Snowflake Best Practices

  The next challenge was to compare the raw data from source database against the views in target
database. In the source database, most information can be fetched by putting simple joins between 2 or
3 tables, but in target database, this was not the case. Hence, testers were required to get the required
data from views rather than raw data.
 Any progressions to source DB during the migration of data prompts inconsistency of data in objective
DB. Moreover, changes in objective application during data migration make it incompatible to the
migrated data.
 Data migration may hamper the functionality and security of app and performance of database
 Check whether all the schema changes as per the new system are updated.
 Rollback Testing – Migration failure test scenarios need to be designed as part of negative testing and
rollback mechanism needs to be tested in case of any issues while carrying out the migration or if there
is a migration failure at any point of time during migration.

 Sample Testing Report

Microsoft Excel
Worksheet

 Testing Strategy

 Data Migration (Source and Target Schema locations)

Ex : IDH – moving to IDW; dbo schema moving STAGE
 Checking Record Count / Metric Sum
 Date Format, Null Values, Constraints, Default values are required to test
 Functions, Procedures, Views to be validated and compared between SQL Server and Snowflake
 Data Quality,  Data Loss and Data completeness
 Performance of Data Load

 Risks

 Snowflake doesn’t support Triggers 

 Snowflake has limited support for Procedures
 Stored procedures have complex logic and testing is complex
 Lots of re-write is involved to replicating IDW processing logic
 Frequent code changes in SQL Server Production results in out-of-sync and need apply changes
and re-test

Snowflake Best Practices

  Mitigation

 Work with Snowflake technical advisory board to come up with best alternatives/work-around

for 
 Triggers
 Function
 Complex Stored procedures

 Frequent interaction and design sessions are required with Guidewire team to understand the
IDW process logic

 Avoid table/schema/column name changes so that ETL procedures need not have to re-written

 Guidewire team has to be pro-active in communicating changes made to production

environment 

Snowflake Best Practices