Managing Large Amounts of Data with Salesforce

How to Manage Large Amounts of Data
with Salesforce
1
Data & Integration Considerations for Architects

Paul McCollum
Cloud Architect
pmccollum@sensecorp.com
2
www.sensecorp.com/contact
@SenseCorp
facebook.com/SenseCorpSC/
linkedin.com/company/sense-corp

It’s just Data, right?
• Map my data scale (Migration and Ongoing)
• Will my application fit (long term) in my Target Architecture?
• Do I have any “Design Skews”?
• Where will my pain points be?
• Pain points and solution patterns for:
– Lots of data (LDV*)
– Lots of connections
– Lots of connections to lots of data sources
• In this Session, we will be focusing on identifying data and integration
issues in the design phase, discussing some solution platforms specifically.
3
GOAL: Learn Architect Design Planning Patterns
“A ‘large data volume’ is an imprecise, elastic
term. If your deployment has tens of thousands of
users, tens of millions of records, or hundreds of
gigabytes of total record storage, you have a large
data volume…”
~Salesforce
https://developer.salesforce.com/docs/atlas.en-us.salesforce_large_data_volumes_bp.meta/salesforce_large_data_volumes_bp/ldv_deployments_introduction.htm

The Science of Architecture
• As soon as you build it, (sometimes before you are even done) applications
take on lives of their own.
• Architects need to see the present and predict the future.
• Good Architecture is responsible for the ultimate success of
application/project.
• In relative dollars, it costs:
 $1 to effectively design and architect a solution
 $10 to re-design it due to technical debt if not architected well
 $100 to completely scrap and re-tool
 $____ to repair your reputation after a write-off
4

It’s just Data, right?
• We create:
– Pages
– Buttons
– Flows
– Validation Rules*
• Talking to:
– Objects
– Fields
• Combined with:
– Authentication: Active Directory, SSO
– HR Profile Data, Medical History, Order History
5
We’re all used to building Applications in Salesforce.
*Salesforce has imposed a limit of 100 Validation Rules per Object.
Ever try to create test data on an object with 100 Validation Rules?

Building Applications Based on Data Scale
1:1:1 or 101 : 101 : 101*
• We try to understand our applications’ consumption of
data relative to the complexity/magnitude of
downstream Data. NOT because data is bad but because
handling lots of anything has a cost.
• For our purposes “Design Skew” is defined as having
components out of proportion with each other.
• Skew can cause both System and Operational
degradation.
• The following slides show examples of how to start
documenting your data interactions and Predict and
Mitigate possible Skew complications.
6
*Obviously, there’s more than 3 elements of solutions, the key is balance.
Master Data
Be aware if you are building or
defining relationships for Master
Data, you may also be creating
duplicate “Master Logic”.
Rules, Logic and Relationships to data
that is not your application’s Master
Data should be conferred upstream
to the Master Data owner.
For Example: Product Price and Tax
Rate calculation should not be done
in multiple systems. Differences in
rounding rules can lead to
discrepancies that can be hard to
reconcile.

Common Types of Design Skews
7

How do I map my application?
8
Salesforce
Data
Start with this simple diagram
• Sketch User counts
• User Types
• Functions
• Connectivity
• Firewall
• Domains
• Data Sources
Objects Services
User
Share it with All teams and Stakeholders.
• Attach it to every deploy and code review
• Any changes must be ratified

Architecture Worksheet
9
Salesforce
Data
Objects Services
User
Users Functions Objects Records Connections Dependencies
Data
Sources
Master
Logic
Master
Data

Reference Architectures
Salesforce
Growth Enabled Company Infrastructure
Scalable
Regular Patterns
Repeatable
Mature Support
Any time the Application has a 1:>1:>1
Objects
Kafka
(Streaming)
NoSQL
ESB
DAM
SFTP
ETL
Main
Frame
MuleSoft
Jitterbit
Data Lake
(Object Storage) Data
Data
Data
Data
Snowflake

Step 1: Dream Big
Architects: Visualize Success
• How big will this get?
• How many people will use this?
• If it gets to size X, will I have to rebuild?
• Define Growth Boundaries with recommended changes.
• Create Transactional Warning systems inside triggers and functions.
11
If(DML.size() > 70% of limit){
SendEmail(to=setting.PerfAdmin);
}

Step 2: Dream Bigger
• Does this data have value?
– The answer is always: Yes.
• The larger the volume of data created, the more likely there will be
value in analyzing that data.
– Trends
– Next Best Action
– Inefficiencies
– Agent Performance
– ROI
• As you create data, think of the many ways it could be used.
– Plan accordingly!
12

Too Many Records
Salesforce Data
Schema Overload
• Trying to keep the data model a perfect twin to reality
• Not having a plan to make data “Mature Out” of the system
• Avoiding Data Skew
• Is every application responsible for Forensics?
Objects
Services
Objects
Objects
Objects
Objects
https://developer.salesforce.com/blogs/engineering/2012/04/avoid-account-data-skew-for-peak-performance.html

14
When to use Big Objects
https://trailhead.salesforce.com/content/learn/modules/big-data-strategy/choose-the-right-big-data-solution

Salesforce Object Storage
15 https://developer.salesforce.com/docs/atlas.en-us.bigobjects.meta/bigobjects/big_object.htm
Big Objects (NoSQL) Data Lake/Cloud Data Warehouse
When to use? When data is only recorded for forensics or
broad reporting and not accessed very often.
When data assets from Salesforce need to be mashed-
up with other enterprise data.
• When companies don’t already have a
reporting ODS available.
• Many 3rd party integrations.
• Data can be reused or resold.
• Organizations that have a strong/evolving cloud
analytics strategy.
• When ETL or external data is required for context.
• Do not bring data into Salesforce JUST to
perform calculations on it.
• Do not store data in Big Objects, if you
expect them to grow infinitely.
• Read/Write Optimized
• Read and Write optimizations.
• They reduce security headaches for consumers.
• Once you get to 1+ TB of Enterprise data
• Transact and Read/Write Optimized
What you need to
be successful?
Operational and Administrative diligence
managing size, load, license cost.
Healthy Cloud management presence or engagement.
Licensing? Licensing through Salesforce is ‘extra’ but
reasonable.
Varies but can be much cheaper from a bulk storage
perspective.
Objects  Big Objects  Data Lake/CDW

When to use? • If your application relies on a service or data provider that is also changing, consider
strongly an investment in an API framework like MuleSoft or Jitterbit.
• These frameworks allow for Services and ETL steps to be performed at the
translation layer and can be versioned and tracked.
Vitally important if your application needs to be UP while it is being changed or source
systems are changing.
What you need to be
successful?
Still new and bespoke management required. Dividends clearly paid in uptime and
efficiency.
Licensing? Varies
Leading API Orchestration Technology Platforms
16

Leading Analytics Technology Platforms
• Dream Big: What is the future of your data scale?
• If Consider Architecting for Data offload to an ODS like
Snowflake early.
• Snowflake is great for Near Real Time Process using the
Snowpipe Feature
• Storage and Compute Resources are Separated.
• It can Scale quickly if you need more Compute Power to
process Queries
• Handles both Structured and Semi Structured Data
17
When to use? Sales trend, customer demographics, Area
Performance, Product Returns, Sales,
discounts and bundles. All data that provides
insight once accumulated.
If you are going to run heavy calculation with
near-real-time data.
Ability to Share Information Across different
Organizations without having to transfer files.
Reporting and Dashboarding against a Data Lake
or Cloud Data Warehouse
Your needs are subject to change and
scalability is a concern.
Multiple data sources
What you need to
be successful?
If metrics and data are going to play a
strategic role in your business, invest early.
Competency in modern cloud analytics
architecture.
Data Literacy
BI skillset
Data Visualization and UI/UX skills
Licensing? Cloud consumption-based model Traditional or Consumption based.

On the
Horizon?
• With the acquisition of Tableau and closer dealings with Snowflake, things
could rapidly change in this area.
• Most likely this will tip the scales towards investments in both, if you are
planning to continue/grow your Salesforce investment.
18

Thanks For Joining Us
We hope you enjoyed the presentation.
If you’d like to learn more about how we use Salesforce
to help transform your organization, contact us.
https://sensecorp.com/salesforce-sense-corp/
CONTACT US
www.sensecorp.com | marketing@sensecorp.com

Managing Large Amounts of Data with Salesforce

More Related Content

What's hot (20)

Similar to Managing Large Amounts of Data with Salesforce (20)

More from Sense Corp (8)

Recently uploaded (20)

Managing Large Amounts of Data with Salesforce

Editor's Notes