KF-SELL ARCHITECTURE and DESIGN

Mission Statement
 Provide Integration between IC 2.0 and Dynamics

Goals

 Allows users to register for access between KF-Sell and Dynamics (Possibly
Salesforce in future)
 Allows users to be configured for access to KF-Sell from Dynamics
 Allow users to interact with access Sales Forecast/analytics data. Users should be
able to update/query/create data related to Sales data.
 Allows CRM systems such as Dynamics to access KF-Sell service
 Allows KF-Sell services to publish and receive data from Dynamics
 Allows KF-Sell services to publish and receive data from KF-Data Science platform
 Allows bulk data transfer from Dynamics (Possibly Salesforce in future)
 Allows KF-Sell services to transfer bulk data from Dynamics (Possibly Salesforce in
future)

Use case Review

Sheet Data use Case

 Getting the Blue/Green/Gold sheet data back and forth between IC

2.0 and Dynamics UI
 Push Surface analytics, alerts and ML outcomes from IC 2.0 to
Dynamics
 Store data in IC 2.0 from Dynamics (only non-PII) for data training
User Personas
 Seller – This user will never log into IC 2.0. Will only access via
Dynamics
 Sales Manager – This user will log into IC 2.0 and needs to be
provisioned
 HR Managers – This user will log into IC 2.0 and needs to be
provisioned

1
Application Integration Secure workflow

When a new Dynamics client is set up to use IC 2.0 (KF Sell), secret has to be provisioned
and shared with Dynamics client.

Dynamics uses JWT token to authenticate and get access KF-Sell services. When the user
connects to IC (KF Sell services) using Webhooks API, the JWT is verified and user
provisioning is done first time if required.

KF-Sell Analytics Platform Architecture Overview

Our goal is to accelerate the development of KF-Sell applications by providing the

necessary building blocks for integration with CRM Services such as Dynamics (and
SalesForce) and KF Data Science platform.

2
 Fig 1. Architecture diagram

The above architecture diagram shows IF-Sell services as part of IC 2.0 platform. The
common platform services such as Authentication, Authorization, Configuration, Log
analytics, and Caching are core services provided by IC 2.0 by leveraging Cloud PaaS
services.

KF-Sell will implement some of the integration to Cloud PaaS services such as Event Hub
to publish/subscribe messages.

 Interface between KF-Sell and CRM

o Tenant data from CRM system (Dynamics) – This includes Dynamics
data. This data is collected in bulk whole on-boarding a new client. This
data is continuously updated via KF-Sell and Dynamics interfaces
whenever the data is updated in Dynamics or whenever user access
screens that interact with KF-Sell service.

3
 o Notification/Analytics and Learning data from KF-Sell to CRM – The
Analytics data is fed from KF-Sell platform to CRM.
 KF-Data Science
o Tenant data to KF-Data Science Platform – The Tenant data is
continuously ingested to KF-Data Science platform.
o Recommendation/Analytics data from KF-Data Science Platform – The
Analytics data is fed from KF-Data Science platform to KF-Sell
 IC 2.0
o IC 2.0 Service interface – KF-Sell will interact with IC 2.0 for several
services such as Authentication, Authorization, Caching, Event, Log
management, Localization services.

CRM Tenant data is ingested into KF-Sell application in, where custom source operators
are written to ingest data from these data sources. The data is moved in bulk at the
beginning or if the data gets out of sync and then data is incrementally transferred as
based on changes.

Once the data lands into the application, the data is then cleaned and normalized in
preparation for it to be analyzed by applications.

This architecture allows KF-Sell to receive data from other sources such as KF-Learn (not
shown in the diagram).

The KF-Sell platform design will be discussed below in KF-Sell Design overview section.

All KF-Sell data is ingested into KF-Data Science platform frequently either in batch
mode or via streaming. The KF-Data Science platform turn the data into actionable
insights which will be consumed by KF-Sell. The analytical data and insights are
delivered to CRM (Dynamics) by KF-Sell either on demand or push.

KF-Sell Design Overview

4
 Fig 2. Component Architecture

This diagram shows the KF-Sell Component Architecture. The goal is to leverage Cloud
PaaS services. For example all streaming will be done using Azure Event Hub to
implement Publish/Subscribe model. The data transformation will be built on top of
Azure Data Factory pipelines.

User Authorization

 Dynamics defines user role and not defined in IC 2.0

5
  The following users are provisioned IC
o Seller – Never
o Admin – Auto provisioned first time when the user connects via Dynamics

Client setup/Onboarding on IC 2.0 - TBD

KF-Sell Platform
KF-Sell platform provides integration with CRM systems such as Dynamics by ingesting
data from these systems. It also provides this data to an KF-Data science platform which
analyzes and creates models to find insights or make predictions/recommendations.
These insights, predictions and recommendations can be combined with other data
sources and provided to CRM system users via visualization screens in CRM platform.
The KF-Platform also sends notifications to CRM Platform some of the
recommendations/insights/predictions.

Fig 3. Component Diagram

This diagram shows the major components of the platform:

 Data Ingest – The first problem that we have to handle ingesting data from the
most data sources. This component contains definitions of common data types

6
 and schema, services for data ingest from external sources and services to
transform the incoming data into the common schema. By transforming all
incoming data to a common data format, downstream applications do not have
to worry about where the data is coming from, and how the data is structured. In
KF-Sell we have to ingest data from CRM systems such as Dynamics and other
sources from KF services. We also have to ingest data to KF Data Science
platforms.
 KF-Sell– This component contains services needed for Visualization service. These
services combine data from KF-Data science component and other KF data
sources.
 KF-Data Science Platform – This component contains services and models for
KF-Sell to correlate data from KF-Sell and other KF-Sources. The service provides
the analytics/predictions/recommendations. KF-Sell can simply reuse the base
analytics as-is or build more complex dashboards by combining other data form
more sophisticated rules.
 Visualization – This component helps our clients quickly visualize data as well as
the analytics results. We would like to provide basic dashboard to help clients
visualize their sales data. Clients can also reuse some of the functions to build a
more elaborate dashboard.
 Notification – This component delivers analytical insights to external systems.
Will use a rule based service to define which notification should published to
different external systems.
 Integration – This component integrates with KF-Sell with CRM platforms.

Microservices and Application

Construction
The KF-Sell Analytics Platform is designed using the Microservice Architecture. This
architecture provides the much-needed flexibility in constructing the application.

The following diagram shows an application constructed using this architecture. In this
application, we compose the application by launching five microservices.

7
Fig 4. Micro services

The Webhooks service provides integration between CRM system and KF-Sell services.
The Webhooks service receives messages from CRM system and publishes data to a
Message Hub topic for Ingest service to process these requests. This service can also
publish other topics in future as required.

The Ingest Service ingests data from multiple sources, one among them is CRM system.
Via Event Hub. The Transformation Service transforms the data from Ingest Service
into a common format and persists in database.

KF-Sell micro service fulfills data requests by Visualization service. The data provided
can include recommendations/insights from KF-Data Science service and other analytics
data as required by Visualization components.

KF-Sell also provides tenant data as required by KF-Data science service. KF-Sell
microservice also provides Rules service to define conditions to identify Insights to be
published to CRM system.

The data is published by KF-Sell service using Publish operator, with a specific topic. We
have 2 services consuming the data stream: The KF-Data Science Platform subscribes to
Tenant data publish topics. KF-Data Science service is an external service as far KF-Sell
application is concerned and nor part of KF-Sell micro services. The Insights topic
messages are subscribed by CRM systems.

8
The Notification service interfaces with messaging system such as Azure Event Hub to
publish events to external system.

One of the key design decisions we have made early on, is to mandate that all services
within the KF-Sell Analytics Platform will communicate with each other using JSON.

Schema Flexibility – JSON allows the schema definition in a microservice to be more

flexible and makes it easier to evolve data schema in the future.

Interoperability – Inn KF-Sell Analytics Platform, the decision on which language to use
for writing a service, depends on which language is best for the job. Some services may
be written in Java if a Java library is available. Some services may be better suited to be
implemented in NodeJS.

Notification service acts an interface to Azure event hub to publish/subscribe events to

external platforms such Data Science platform.

KF-Sell Integration with KF-Data Science Platform

Fig 5.1. Updated CRM data from IC (KF Sell) to Data science platform.

The IC Tenant Publisher (KF-Sell) publishes Tenant data updates using Tenant Publish
topic which will be subscribed by KF-Data Science platform Data ingestion service.

The Data Ingestion service of KF-Data Science platform processes these events and
stores the data to a Data lake.

The Data science models process the data to train the models in Azure Databricks
(Spark).

9
Fig 5.2 Request from IC for prediction/recommendation (will include all data in the request)

When the IC KF-Sell platform needs predictions/recommendation from Data Science

platform, the request can be made via Request event through the Event hub with all
required data needed for this request. The Data science platform can stream the
Predictions/Insights/Recommendations through the Event hub.

Data science platforms also will provide REST interface to make synchronous requests.

Integration of IC KF-Sell with CRM system (Dynamics) for

updating data from Dynamics to KF-Sell

The scope of this document is integration of IC KF-Sell with Dynamics. In future, it can
be extended to other CRM systems.

Privacy rules - Only non-PII data is stored in IC 2.0 (KF Sell). PII data is stored in
Dynamics

10
Fig 5. Technical Options for data updates from Dynamics

There are 4 different technical options for integrating Dynamics and KF-Sell platform.

Option 1 – Web Hooks

Webhooks enable developers and ISV’s to integrate Customer Engagement data with
their own custom code hosted on external services. By using the webhook model, you
can secure your endpoint by using authentication header or query string parameter
keys. This is simpler than the SAS authentication model that you may currently use for
Azure Service Bus integration.

When deciding between the webhook model and the Azure Service Bus integration,
here are some items to keep in mind:

 Azure Service Bus works for high scale processing, and provides a full queueing
mechanism if Dynamics 365 is pushing many events.
 Webhooks can only scale to the point at which your hosted web service can
handle the messages.
 Webhooks enables synchronous and asynchronous steps. Azure Service Bus only
allows for asynchronous steps.
 Webhooks send POST requests with JSON payload and can be consumed by any
programming language or web application hosted anywhere.

11
  Both webhooks and Azure Service Bus can be invoked from a plugin or custom
workflow activity.
 Synchronous - if the transaction fails then there is a roll back of the CRM data
and an error to the user
 Async - if the transaction fails then there is log and a failed transaction, there is
no roll back. The failed transactions have to be maintained on KF-Sell and data
needs to be synced automatically when it goes out of sync.

Disadvantage –
 Bulk updates can fire thousands of transactions - can mitigate by filtering 
 Data Duplication (stored both in Dynamics and IC KF-Sell)
 Mitigation
o How do we deal with KF-Sell being down (syncing strategy)
o Performance Considerations and Recovery strategy
 Costing considerations

Option 2 – Sheets UI Data access

Dynamics UI can pull sheets data from KF-Sell using KF-Sell Sheets REST API and also
can post update using REST option.

Cons

 Client does not have data in CRM. We may be able mitigate this by using Virtual
Entity. Dynamic Virtual is a custom entity in Dynamics 365 Customer Engagement
that has fields containing data from an external data source.
 There are limitations on using filters in queries with Virtual Entity and ability to
use charts
 We also may have to allow access from Virtual Entity to KF-Sell Database opening
up security concerns. This might open up Inadvertent Data disclosure.
 This option was discouraged by Microsoft Dynamics expert

12
Pros

 Single source of data no data duplication

Option 3 – Data Lake

Dynamics can be configured via an Export service to move the data including updates to
a Dara Lake. In this option., each client has to create their own Data lake and allow KF
service via a secure option to access the data in the data lake. This adds additional cost
to the customer along with configuration required for KF access to this data.

Option 4 – Export using Feature change tracking using OData provider

KF-Sell service gets access to Dynamics data via OData provider API. There are API limits.

External connection required - Additional setup beyond managed package is required.

13
Fig 6. Choices for data updates from Dynamics

Design Decisions

 Based on the pros and cons of the design considerations, we only can
choose between Option 1 and Option 4. We decided to go forward with
Option 1 (Web Hooks option).
 In order to reduce the large amounts of data, we will use a one-time Batch
update during onboarding to transfer data in bulk from Dynamics to KF-Sell
platform.
 How do we deal with issues that might arise out of sync data?
o A process to verify and sync data at a regular frequency.

The following diagram depicts the flow of data updated from Dynamics to KF-Sell
based on Webhooks.

14
 

Fig 7. Data updates from Dynamics to KF-Sell

Integration of Dynamics and KF-Sell to push alerts/notifications from

KF-Sell to Dynamics

15
Fig 8. Technical options from pushing alerts from KF-Sell to Dynamics

There are 3 technical options to transfer Alerts/Predictions/Insights to Dynamics.

Option 1 – Virtual Entity (Not recommended by Microsoft Dynamics architect).

Disadvantage: only accessible within the UI

Option 2 – KF-Sell can use Data API to push alerts/notifications to Dynamics. This option
is recommended by Dynamics Architect but needs considerable configuration on client
side during onboarding.

The Data API option requires for us to use native APIs for other CRM systems.

Option 3 – Web Hook and REST request/response. This can be scheduled using Power
Automate option within Dynamics.

Design Decision

 We are going to use Option 3 Dynamics will poll for recommendations and other
notifications from IC KF-Sell.
o We are already using Webhooks to push data from Dynamics to kF-Sell.
This will be consistent.
o No need for extra step as needed for Option 2

Integration using Publish/Subscribe pattern – Event Hub

The KF Sell platform, Data science platform and Dynamics are all integrated using Azure
Vent Hub PaaS service.

The diagram below depicts all the integration between different service and platforms.

Webhooks service – Receives Dynamics Updates and publishes these messages. On to

Event Hub

Ingest Service – Subscribes to Dynamics Updates

KF-Sell Service – Publishes Tenant Data updates, Alerts/Notifications based on Rules and
also subscribes to Predictions/Insights

Notification Service – Subscribes Alerts/Notifications to deliver them to Dynamics

16
Data science platform – Subscribes to Teanant Data and publishes Predictions/Insights

Fig 9. Publish/Subscribe using Azure Event Hub

Service Design
Here’s a list of microservices available in this first release:

Webhooks

17
 Service Description

Foundational Toolkit providing basic

Kfsell.webhooks infrastructure for interfacing with Dynamics
(CRM)

Publish data to Event server (e.g. Azure Event

kfsell.webhooks.dynamics.publish
Hub – TRANSFORM.DYNAMICS topic)

Ingest

Service Description

Foundational Toolkit providing basic infrastructure for

Kfsell.ingest
other ingest services

Publish and subscribe data to Event server (e.g. Azure Event

kfsell.ingest.subscribe
Hub)

Transformation

Service Description

Transform service from different sources to

standard format.
Kfsell.transform

Kfsell.crm.transform.dynamic Adapter for converting data from Dynamics to

s standard format.

Kfsell.crm.transform.persist Persistance framework for data storage

18
kfsell

Service Description

Kfsell.sheet
KPI data REST API for Blue/Green/Gold sheets data
s

Rules API for determining notifications to be sent to CRM

Kfsell.rules

notification

Service Description

Publishing service for notifying alerts/notifications

notification.publish

Publishing service for notifying alerts/notifications

notification.publish.dynamic to Dynamics
s

Shareable Components
All components are reusable

 web hooks,
 transform,
 rules,
 ingest
 sheets

19
 The exception is adapters that processes messages from Dynamics or notification
publishers to Dynamics using Data API.

Database
Options for Database

 Postgres
 Cosmos

Replication Needs/Options

 Region

Service Containerization
The following diagram shows how the micro services of KF-SELL platform
will be deployed. The microservices will be packed using Docker and
Kubernetes container and deployed in Azure Kubernetes Service.

20
Ingres controller

Ingress controller provides reverse proxy, configurable traffic routing, and TLS
termination for Kubernetes services. Kubernetes ingress resources are used to configure
the ingress rules and routes for individual Kubernetes services. Using an ingress
controller and ingress rules, a single IP address can be used to route traffic to multiple
services in a Kubernetes cluster.

ALB - Azure Load Balancer

DR Model – Active/Passive (Create scripts and runbooks for DR in PROD and testing
annually)

Availability zones – scale

Region availability - yes

Azure Devops Pipeline – Provides Devops pipeline to deploy code to AKS service

21
Service Isolation Model

 Security considerations
 Web Application Firewall (Azure Application Gateway)
 Follow Hub/Spoke model
 Hub for common services
 Spokes for each Business line
 Separate VNET and subnet for each business line
 VNET peering as needed between Hub and Spoke
 Security
 Default security option is Deny All for inbound and outbound and
only opened based on requests
 Mutual TLS for security - client and the server must provide
digital certificates to prove their identities.

22
  Databases to use TDE encryption for data at rest
 Database connections to use TLS

Questions and comments

23