INTRODUCTION TO PROJECT MANAGEMENT :

Data virtualization project management : development of best practises for building solution in a
productive way
interactions between :
team role , users of the project , upper management

Development process :
BEFORE :
critical need buy in from management and users along with communication with data consumers
The following is necessary to agree on what is to be built ( data models , data access protocols )
During :
Prefer small features and iterate quickly over big monolithic , let business guide growth
focus on : keeping each iteration as small …

Project management development teams : organizing development teams

two basic models can be defined :
1////Distributed Team with multiple development servers : each dev with local full denodo
installation , locaal or remote vdp +design studio
optional : a local virtual data port administration tool

2////Centralized team : single dev VDP server and design studio

*********two options : private databases : for each project theres one database for each dev of
the project
shared database : for each project theres one single database shared by all the dev

Organizing Dev teams :

USING VCS : is the management of changes to elements of the server catalog
this features strongly recommended if several dev teams can be working simultaneously
great feature of vcs it allows the elements to check in and check out while

Different roles : data virt architect , platform administrator , data virt developer , business users
any organizations needs support provided to their users : could be the same dev team or separate
group , needs a set of procedures and tools to manage the relationship with users of the project and
developers of the project
The differents components of denodo platform

Development scnarios :
two different development scenarios :
centralized model : one VDP server for each environment , each dev woks on their own vdp (one data
base per project ) theres central database ( per project ) where changes are consolidated
Distributed model : each dev works on their own vdp server , theres an additional central server
where changes are consolidated

choosing a workflow : each model has its own pro and cons , important to understand these in order
to choose the right strategy for your project
centralized workflow with shared , private data ,, distributed workflow

centralized model :
Best practises : Manage global elements
THE DENODO SOLUTION MANAGER IS THE RECOMMENDED WAY TO PROMOTE CHANGES it
tales into account all the tasks that denodo platform administrators should perform to promote
changes through env

The agile dev : the recommende project lifecycle for denodo is very similar to a software project ( dev /
test / promotion to production )
following an agile method , an iterative app can be apolied , repeating these steps for ervy sprint

key value of data virtualization is agility in delivery “ reduce time to data “ with minimal it staff

agile to denodo : each code lifecycle management involves ( defining revisions to promote between
environments , promoting revision to a cluster with no downtimes , testing , integration with lifecycle )
et encore

DATA VIRT AND DEVOPS :

can be executed through any one ofthe preferred methods ( waterfall or iterative )
agile : the team will discover requirements and develops solutions through the collaborative effort of
self organzining and cross functional teams and their customer / end user

list of actions carried out in a dev life cycle are : discover and analyze the business need , define the
archi of the sol , design the sol …

sprint : 1 //////dev :: ( can create source/views deploy services and use vcs to build virtual data
model)
2////////testing (optional ) : central server where changes are consolidated …
3 /////////// QA : another team can test the model against similar volum of data
4 //////// production : final users and application will use the views / services …

DEnodo testing tool :

easily automate testing of DV :
safety net , during … ( dev of virtu envi , maintenance and update ))
quick facts ( no java needed , continuous integration , completely automated … )

Monitoring : the denodo platform provides two diff way of monitoring server of the platform across
various env ( Using java management extensions jmx agent such as javatm visualVm or
Jconsole ///////: using the denodo monitor app included in the denodo platform )

denodo platform is available as a docker contaner to be downloaded from the support site

Version Control : vcs is the management of changes to elements of the server catalog
vcs enables team dev : used at different points of the software deployment lifecycle or used at different
geographical location

vcs support allows to do th e main tasks involved in version control : check in / chek out elements ,
reverting elements folder and db , discarding , conflicts detection and resolution …
the configuration of vcs in vdp is a tsk done usually bye a denodo admin ( in a distributed team
with local isnta of denodo platform for each dev this config has to be done by the dev ) so THE
DEV IS THE ADMIN OF THE LOCAL SERVER )

VCS c’est administration > vcs management

When vcs is active for the current DB each node in the admin tool element tree is marked with a vcs
status icon having different color
the contextual menu shows different options to perform the corresponding version control tasks
depending on the version control system configured
the following are the avail option for each system :
git : commit pull push
TFS / SVN check in check out diff and discard

VCS IN SOLUTION MANAGER

denodo solution manager uses VCS to store backup of the state of the denodo servers after a
deployment
Backup for the VDP SERVER IS CREATED AS VQL file , scheduler creates

VCS BEST PRACTISES :

setup process to create tags and branches :

Promotion overview :
recommendation apple when promoting between environments : USE VCS INETGRATION DURING
DEV , USE SOLUTION MANGER OR VQL FILES to promote changes to other environments admin
can use export import svripts for promoting code between diff en in case Solution manager is not
configured +++++++++ Deploy the complete database ++++++++ add a tag to the vcs repository

Two scenarios are possible in dev en :

1 // one virtual dataport server for multiple server
2 // one virtual data port server for each dev

Steps to promote to a new environments :

1 // connect to the vdp server and configure the connection to the central version con cyst
2 // When the VCS configuration is ready
check out the data base into a local database , automated test are recommended at this point to
validate inetgrite of the version to promote

Using export /import scripts

export env specific properties separately option is recommended , include user and pri is used to
include permission ++++++ use the export script ++ edit properties file generated to adapt to the target
environment
+++++++ import the vql in the target en

performance optim :
the time to perform the rollback in a virtual dataport server depends on the size of te metadata
…
to reduce time for the rollback operation and the promotion , following command can be used :
enter /exit promotion mode
Promote to other env : in solution manager a promotion takes into account all the tasks taht you
should perform to migrate changes in the servers metadata from one env to anothe

This is done through two man components : REVISION ++== DEPLOYMENT

PROMOTION BETWEEN ENVIRONMENTS: ( solution manager environments ):

Denodo servers letadata evolve in time , dev user diff en
DEV IN DEV EN +++ TESTED IN TESING ENV ++ DEPLOYED IN PRODUCTION ENV

PROMOTION BETWEEN ENVIRONMENTS: ( Revision creation ):

A revision is a collection of VDP elements and Scheduler tasks :
only globale admin , promotion adm ana prom user can create revisions
Promotion administrators ana promotion used can also be fine grained to specific environments

the following types of elements are not included in the revision candidates : Ressource manager
plans and rules ++ widgets and jms listeners

the list of candidate elements and tasks depends on the permissions of the user
if the server is configured with the authenticate with current user cred fro creation revisions option , the
user auth in the sol manager will be uses , instead of the user password defined in the server

the following features will be helpful for creatnh reisiosn with ease :
1// wizard to compar e the metadata between source and target env
2// Rest API option to download the VQL of the revision

Revisions operations : users can view the list of created revision in the revision table , users can
remove their own revisions ++Edit their own revisions ( not allowed to remove or edit deployed
revisions ) read-only

Operations in revisions table : open the revision …

Revision PROPERTIES :
the elements of revision may depend on a set of properties like data source, urls user name and
passwords : the target env should define a value fo every virtual data port property that parm the vql of
the revision
every cluster of the target en with schedumer servers should define a value for every scheduler
property that parametrizes the task of the revisions

BEFORE DEPLOYING REVISIONS validate that the target env defines all the required prop ++ its
possible to chekck the list of env where a revision was validated on

Cluster CONSIDERATIONS : During the deployment , all nodes in the cluster should be updated
Simple depl + deployment without service interruption

There are two different cluster env scenarios available for deployment without service interup
that depends on the topology of the target en
single cluster environment ++ several cluster environment

1:::::::::::::Simple deployment :
This option assumes that the application service will not be available during the whole deployment
process : the revision is deployed in all the servers without explicitly disabling any cluster or server in
the load balancer

2::::::::::::/deployment without service interruption : one by one deployment executed in an

incremental way only one server is disabled ++++ half by half deployment ( half of the servers or
cluster are disabled ) and then enabled at the same time Deployment without service interruption
( shared cache swap the cache ++ executes all scheduler VDP cache tasks )

deployment configuration : the solution manager automizes the deployment process

in order to perform deployments :
the target environment must be configured first : the deployment configuration section of an
environment dialog allows to configure how a deployment will work
only global administrators and promotionadminstrators can change the deployment configuration
dep can vary based on the cluster , ie the environment has more than one cluster or it has juste one
cluster

The section deployment scripts : allow to administer the scripts that the solution manager uses to
enable and disable servers or clusters in the load balancer
ONE CLUSTER : the solution manager will use scripts ( enable / dsiable in the load balancer )
MORE THAN ONE CLUSTER : enable cluster in the load balancer / disable in the load balancer

Each deploy script can declare a list of parameter ; solution manager supports two kinds of types
Literal argument ++ load balancing variable

The solution manager performs some basic validations before starting the deploy :
Deployment must be enabled in the configuration of the target env
the target env must define all the vdp prop that parametrize the vql of the revision
every cluster of the target en with scheduler servers must define all schedumer pro
theres no other deployment in progress
the env config matches the current env topology

DEPLOY PROCESS :
the solution manager executes first all the revisions in one server continuing to the next one
the deploy config of the target envi defines which actions are actually performed during the dep
process

Types of deploy :
STANDARD MODE DEPLO// CLOUD DEPLOY

DEPLOYM PROCESS :
consists in executing all the changes defines in one mor revisions in the denodo server of an env
the solution manager executes

Documentation :
metadata provides information about the elements like owner and the creation and / or modification
time …
Doc is helpful to help track aspects of application
Dev , maintenance , knowledge transfer to other developers

for managing metada

on VDP elements , denodo users can : add metadata to vdp elements , add descriptions in views ,
examine and search metada , browse metadata using the denodo catalog

the metada of denodo element allows devlopers to access its metadata : depending on their prib they
could : view diff info about the view , set the owner of the element …

denodo allows to add descriptions for fields this helps users to better understand fields purpose
( imported from jdbc sources )
fields are propagated to DV
the catalog search is a tool provided by the VDP admin tool available in the menu tools >
catalogSearch ( useful for searching fo elements metadata such as : owner of an element , creation or
modification date , description USING CATALOG SEARCH OR USING STORED PROCEDURES )

DENODO PROVIDES different out of the box stored procedures dev can use for diff purposes .
Some of them are useful for searching for metadata

GET_ELEMENT , GET_views , Catalog_VDP_metadata_view , GET_catalog_metadata_ws

REST WEB SERVICES : the openai specification is specification for describing rest web services

denod rest web services use the openAPI specification for describing the available operations and their
input and output schemas

Using specification for describing REST WEB SERVICES :

Denodo Rest web services use thor openAPI spec for describing the available operations
obtain the specification using URL

DATA CATALOG : is web tool that lets both technical and business users query search and
brows information and meta information stored in a virtual dataport sever

by thistool , they can generate new knowledge and pve the way to take better decisions

The data catlog allows to :

query search anad browse info
browse dependencies
manage metadata info
search for speicific content
DEVELOPMENT BEST PRACTISES :
DIFFERENT PRACTISES ANY DVEELOPER MUST BEAR IN MINDE WHEN WORKING ON DATA
VIRTU PROJECTS , some of them will be covered in this module

1 / CONFIGURING THE DEV ENVIRONMENT :

A typical dev env for denodo platform cen inlude :
one VDP for all dev ++++++ one VDP for each dev

Project Layers :
for simplifying the dev and management of the data virtu projects with denodo , all these ele mst be
organized in diff layer
CONNECTIVITY ? INTEGRATION ? BUISNESS ENTITIES ? REPORT VIEWS ? DATA SERVICES

DATA VIRTUALIZATION PROJECTS :

When dveloper is working in daata virtu its common to manage a lot of diff elements , these elements
can eb related to some categories
CONNECTIONS ( DATA SOURCES ; BASE VIEWS )
COMBINATION AND INTEGRATION ( joins , unions , groupe by .. )
PUBLICATION ( WEB SERVICES °

ALSO THERE ARE SOME IMPORTANT ELEMENTS THAT MUST BE CONSIDERED

canonical elements exposed to external app ++ finale elements to reporting tools

PROJECT STRUCTURE :
project stru comprises od fiff layers such as connectivity , integration ..
based on the project structure the dolders can also be organised for managing the elements in case of
complex projects
one foder by layer , additional for others elements for example ,: associations , jms listner

in complex projetcs additional subfolder can be added to group related elements

prefix elmeets used in denodo PAGE 10
ds_name …

VIEW DEV :
1/////////:: CACHE
THE DBMS USED Aa cache could also be used as data source

general recom … : t

2///// STORED PROCEDURES :

VDP provides different out of the box stores procedure dev can use for creating advanced views
most of the options available using the graphical interface are also available using stored procedure
dev fcan create views using STORE PROCEDURE
Stored Pro can be extruded from external tools

3 /////////// DOCUMENTATION
along with a naming convention views can be de=ocumneted by adding description to them
4 .//////////////// Integration with version control systems
THE Version control system should be the single source of truth about the current status of the projects
+++++ complete

5/// METADATE STORAGE , VERSIONING and sharing :

VIRTUAL SHCEMA DEFINITON can be easily represneted as sequence of VQL sentences taht
deefines users …
when dealing with VQ format , consider two types of metadata :
CONSIDER TWO TYPES OF METADATA
1111111111111111 - ENV DEPENEDT METADATA : JDBC DATA SOURCE FOR THE DEV
INSTANCE OF A DATA BASE
2222222222222222 - NON ENV DEPENDET METADATA : DERIVED VIEW COMBINING FROM
OTHER VIEW SHOULD NOT CHANGE ITS DEFINITION FROM ONE ENV TO ANOTHER

6///:: METADATA STORAGE , VERSONING AND SHARING

VDP should be exported to vql files in order to be stores and included in version control systems
Stores VQL files hould contain non securuty data

Accessing from external client

denodo can manage large amounts of data from one single query
JDBC SHOULD BE PREFERRED WHEN POSSIBLE OVER WEB SERVICE PUBLISHING
SOAP REST USE TEXT BASED COMMUNICATION PROTOCOLS

DATA LINEAGE : ALLOWS DATA ADMINS TO TRACE THE ORIGIN OF ANY COLUMN IN A
DERIVED VIEWS ACROSS ITS TREE OF VIEW COMBINATION

EXECTIONS TRACES

INTERFACES