1

Issue 1

Next-Generation Test
Data Management
How to deliver the right test data, to the right teams, at the right time

2 Welcome
3 Research from Gartner:
Determining the ‘Right’ Number of Nonproduction Environments
7 Delphix Represents the Next Generation of TDM
10 About Delphix
2

Welcome

Applications are • Accelerates test cycles with environment

critical to the success provisioning, refresh, and reset in just minutes.
of modern businesses, • Eliminates the need for subsets or synthetic
enabling them to data, increasing testers’ ability to catch bugs and
optimize internal improving overall code quality.
processes, gain and
retain customers, • Protects sensitive data in nonproduction with
and outperform automatic masking of test data.
competitors. But to • Dramatically reduces storage requirements for
consistently innovate nonproduction environments.
and deliver at the
cutting edge of their As you read Gartner’s report, “Determining the
industries, companies must execute regular ‘Right’ Number of Nonproduction Environments,”
rollouts of fresh application features, security we encourage you to envision an enterprise
updates, and other improvements. Central to strategy that takes advantage of these features.
these efforts is data. In particular, managing Delphix represents the next generation of test data
nonproduction environments and their requisite management solutions, giving customers more
data needs is of paramount importance to better, flexibility in deciding what the “right” number of
more timely application development. nonproduction environments actually is, while also
allowing them to provision those environments with
At Delphix, many of our customers look to both ease and efficiency. All of these capabilities
improve Test Data Management (TDM) to come together to help customers transform TDM
streamline processes across development, test, and, ultimately, outpace the competition.
and operations teams. Too often, antiquated
technologies and complex processes stand Sincerely,
in the way of fast access to high-quality test
environments. But by helping customers better Charles Moore
manage and deliver test data, Delphix: Director of Product Marketing, Delphix
 3

Research from Gartner:

Determining the ‘Right’ Number of Nonproduction

Environments

Production environments must be physically Analysis

segregated from development and testing It’s a wise practice to separate development,
environments. Determining how many additional testing and training activities from the real work
development and testing environments are of production. For example, in many situations
“enough” is critical to quality and agility. where COBIT, the Sarbanes-Oxley Act, or the U.S.
Food and Drug Administration (FDA) regulations
Overview are at issue, there is a legal or audit requirement
Adequate environmental separation helps to limit access to production application instances
development teams move code smoothly and and their data. Testing and other nonoperational
consistently to the operations team. Do your activities in production have regularly led to
environments meet your needs? outages, data losses and other problems. Most
organizations understand the risks of touching
Key Findings production, and are asking: How many additional
• There are distinctions in access, service environments are needed?
levels and use cases that clearly differentiate
development, test and staging environments. The answer is “just enough,” but that raises more
questions than it answers. Here is a framework for
• Additional logical environments may be needed determining the right number of environments, as
to support particular processes. well as some guidance on sizing and policy.

• Risk assessment of production readiness should Minimum Number of Environments Needed

belong to the organization responsible for the At a minimum, your organization should have three
operational service level. additional nonproduction environments for each
major application:
Recommendations
• Establish, at a minimum, distinct development, • Development — owned and managed by a
test and staging environments. designee within the programming team; it is
used to create, debug and test code
• Access of development and test staff to staging
environments should be minimal and closely • System test — owned and managed by the
supervised. application development organization; it is
used to assess functional completeness and
• Incorporate explicit regression test steps to adherence to models and design
deal with differences from one environment to
the next. • Acceptance test (also called assurance, staging
or preproduction) — typically owned by an
What You Need to Know organization outside of development (such
The “right” number of development and test as production control or quality assurance); it
environments differs from organization to is used to assess compliance with end-user
organization, and by project or application within requirements and risk of release, and may, in
organizations. A clear definition, supplemented by some cases, also be used for system integration
the use of a configuration management tool that testing by the production group
automates the choices, effectively reduces code
and process errors introduced in production.
4

The first two environments clearly should be under The acceptance and system integration test
the control of the development team. Practices environments may be separated for organizational
in these environments are intended to produce reasons, but usually are similar in their configuration
a functionally complete product. Development and service-level requirements.
teams are typically measured by the completeness,
internal quality, timeliness and budget compliance Do You Need Additional Environments?
of their work. If the release cycle is slow, then Address these questions to determine whether you
organizations may attempt to do acceptance need to add environments:
testing in the same environment used for system
tests. Many development environments are heavily • Do we need a separate environment to recreate
loaded with profiling and monitoring tools that production problems? Typically, the answer to
deeply affect performance, making them ill-fitted this is yes. Many production issues can’t be
for this dual usage. Often, these encumbrances recreated in a programmer environment, because
and the need to restrict access to the assurance this environment is inherently unstable, and,
region require that separate system testing and even if it isn’t, it potentially has new or revised
acceptance testing environments be established. code that could disrupt an attempt to recreate
production. Programmer access to production
The focus changes to risk assessment with the for emergency or hot fixes creates security risks,
turnover of a release candidate to the acceptance and is difficult to manage. The developer access
or preproduction environment. In the preproduction has to be temporary and revoked immediately
stage, we see activities such as system regression upon completion of the fix. Alternatively,
testing, load and other performance testing, security some organizations grant temporary access
scanning, service desk training, deployment and to a separate fix-test environment or to the
rollback testing, and user acceptance testing. These preproduction environment. Once the patch is
activities all aim to establish confidence that the ready, production control personnel can deploy
release can be deployed without compromising it through standard mechanisms to production.
production service levels. In other words, the If the need to recreate production problems
objective is to establish that the risk of putting happens infrequently (for example, once every
the candidate into production is acceptable. six months), then it may be more appropriate to
Increasingly, ownership of this stage belongs to a have a process in place for a temporary pseudo-
group reporting to whoever is responsible for the production environment to be created with a
production service levels. This might be a quality few hours’ notice. This facility will sometimes
assurance group or a turnover group reporting also serve as a way to test application recovery
to operations. This stage often also provides the procedures. This environment is typically owned
separation of duties required for audit compliance. by the programming staff.

The acceptance environment may also be used • Do we need a separate environment to train
for parts of the ITIL release process, such as users? For major new applications or revisions
system integration testing, checking the ability to established applications, a model office that
of the application to coexist with other enables the business to simulate a day (or other
infrastructures and applications that potentially time period) or to test staff may be needed.
will be sharing resources. Other uses can include Determine whether the need to train or do other
testing of deployment and rollback scripts, and business-driven work is great enough to require a
checking the various processes that will update separate environment. Typically, this environment
configuration data, application monitoring and is owned and operated by the business, and its
other operational artifacts. code mimics production-level code.
 5

• Do we need a separate environment for • Do we need an environment to act as a

each release team? Many organizations have definitive library? An organization’s release
set up multiple release teams that overlap. processes may call for a library of the source
Each team requires its own environment at codes or executables that are actually in
the programmer level, with its code then production. This is called a definitive software
migrating up the release chain to production. library (DSL) or a definitive media library (DML)
If a development organization has three in some of the ITIL processes.
overlapping release teams, each requiring
its own environment, then the programmer • What if I want to do continuous deployment?
environment should include three separate Continuous deployment automates many
versions, with migration between the levels test and promotion processes, but generally
handled as each team moves its release to uses similar sets of environments, as we’ve
the system test environment. In this case, the discussed. Automation speeds the cycle, and
number of added environments would be two, may allow some staging to be enforced by
and the release manager is the owner of his or the processes, rather than by establishing
her team’s environment. physically distinct environments.

• Do we need a separate environment for Virtual or Real?

usability testing? Most organizations aren’t The principle reason for the physical separation of
sophisticated enough to do formal usability activities is to prevent compromises of the goals
testing early in a project. Those that are need to of each environment; therefore, production should
use the appropriate programmer version to do be strictly separated. The test and preproduction
early screen and process flow checks, or they areas can be more closely linked, but usually
need to set up an environment specifically for have different access lists. The more-fragile
usability testing. The major driver for a formal development and test regions are often physically
and distinct environment is the instability of segregated so that their restarts, builds and other
the programmer environment — that is, if the activities don’t interfere with preproduction test
business is scheduled for a usability test, then schedules. The degree of physical separation must
the version needs to be available. be higher when virtualization can’t successfully
limit the interactions among systems. Successful
• Do we need a separate certification virtualization enables more sharing of physical
environment? In some large organizations, hardware. We see limited use of virtualized
testing the interaction between major performance test regions (preproduction), unless
applications and the hardware, software and production is also exploiting virtualization. In
middleware that run them is critical. Best cases where production is virtualized, virtualized
practices demand that infrastructure changes performance testing can be a good predictor of
be managed independently of application production performance, and becomes acceptable.
changes. Therefore, the two classes must have
separate environments or serialize the use of
one environment between them.
6

Other Concerns to Consider Service levels for nonproduction environments.

Environment size. Sizing an environment is If there are substantial differences between
not an exact science. Generally, the programmer the preproduction and development and test
environments won’t exceed 20% to 40% of environments, then appropriate testing to protect
production, and the test environments won’t against the consequences of those differences
exceed 50% of production size. There may be needs to be in the plan. Support levels for
times, however, where performance tests are preproduction approach those that are needed for
critical, and full-size production databases will be production. If the production support interferes
contained at the test levels. It’s a trade-off: disk with the ability to maintain sufficient similarity
space for risk. Simulation tools can reduce size among environments, then a separate support
requirements in many contexts. team will need to be created to focus on support
of the preproduction, test and development
Environment migration. The software change environments.
management process controls migration patterns
(that is, how a given set of code moves from one Best practices for the transition process in the
environment to another). Large organizations, or stages of testing (integration, acceptance and
those with rapid release rates, will need to have preproduction). Approval workflows are typically
an automated solution to control the migration established that determine when a product can
process. Others do this manually. In either case, move from stage to stage. While these can be
it’s critical that the migration path be carefully manual, they are best-implemented in change
defined. In this day and age, with so many moving management or application life cycle management
parts, most organizations should be moving (ALM) tools, like IBM Rational ClearQuest,
toward automation of these migrations. It’s much Microsoft Visual Studio Team System (VSTS),
easier to implement early and make the process CollabNet TeamForge and Atlassian Jira. Criteria
progressively more complex than to delay and for promotion to the next stage should include the
have to mount a major project to bring complex successful completion of some range of tests, and
processes under control. concurrence from the appropriate stakeholders.

Source: Gartner RAS Core Research Note G00210456,

Complexity. The more environments you own, the Jim Duggan, Matthew Hotle, Thomas E. Murphy
more complex the process of making changes is, 6 November 2014
the more organizational challenges there are and
the more difficult it is to be flexible to changes
in the project, because shifting requirements or
defects are discovered.
 7

Delphix Represents the Next Generation of TDM

An exercise in determining the “right” number In contrast to traditional TDM solutions, Delphix
of environments carries with it an unstated provisions datasets that represent full copies
assumption that businesses must balance of the production source rather than subsets
quantity of environments with availability of and contain real data instead of synthetic data.
IT resources. After all, businesses are careful in Furthermore, Delphix can automatically apply
meting out dev/test environments since they masking to virtual datasets to secure sensitive
are constrained by expensive legacy infrastructure information, while still preserving the value of
or the complex processes with which that the data for testing purposes.
infrastructure is marshaled.
Delphix Redefines the Notion of “Right-
But what if those constraints could be removed? Sized” Nonproduction Landscapes
Specifically, what would nonproduction landscapes With Delphix, customers can meet and exceed
look like if project teams could effortlessly deliver the threshold requirements for providing
new environments, while consuming a fraction of sufficient environments for major applications.
the resources? Virtual data copies consume less storage than
even physical subsets while requiring essentially
Delphix is a test data management solution no labor to provision and refresh. This means
that provides exactly that capability, delivering project teams have the slack to scale to more
space-efficient dev/test environments on expansive landscapes, as needed. Moreover,
demand. Delphix software captures and stores a Delphix positions businesses to satisfy future
copy of production application data—including state requirements for Continuous Delivery or
ongoing changes—then delivers virtual copies for DevOps, methodologies that demand elevated
development, testing, reporting, or other purposes, levels of environment availability and fluidity.
without any data duplication or movement. New
environments can provisioned in 1/100th of the
time using 1/10th of the infrastructure.

FIGURE 1 Delphix delivers lightweight, virtual nonproduction environments on demand

Source: Delphix
8

Threshold Requirements for Nonproduction: Example: One F100 customer used Delphix to
Maximum Productivity Within Minimum reduce test environment refresh time from 2 days
Landscapes to 20 minutes, resulting in an overall project
schedule reduction of 30%.
By virtualizing application data derived from
production systems, Delphix allows customers to Objective Requirements for Nonproduction:
efficiently provision environments dedicated to When More Is Better Than Less
development, testing, and validation. While this
trio of environments represents table stakes—the While separate environments for dev, test, and UAT
minimum amount of IT workspace necessary may be sufficient for simple applications, today’s
to build software—Delphix also provides agility complex enterprise software typically demands
benefits enabling project teams to accelerate the additional flexibility. Small landscapes result in
software development life cycle (SDLC) within either serialization of effort that constrains project
even the leanest of nonproduction landscapes. capacity, or resource sharing schemes that yield
unpredictable, collision-prone testbeds.
For example, test environments leveraging virtual
copies of data can be instantly refreshed. Rather Since Delphix removes the constraints that limit
than waiting hours for IT administrators to restore the number of environments that businesses
datasets from backup copies following destructive deploy, it allows them to easily meet more
test runs, QA teams can perform data refreshes or aggressive requirements. For example, customers
resets in minutes, and as a self-service. can move towards parallel development models
in which release teams are given separate,
The key benefit stemming from these efficiency multi-environment project lines. This allows
gains is a “shift left” in testing: application teams organizations to execute multiple projects
execute more productive test cycles, earlier in the concurrently, even for complex applications—ERP
SDLC when bugs are easier and less expensive to chief among them—that prescribe rigid approaches
fix. Contrast this with TDM solutions based on data to development and testing.
subsetting or synthetic data. Legacy approaches
not only force testers to work with limited datasets
that eventually erode application quality, but also
introduce process overhead that slows them down.

FIGURE 2 Delphix enables “shift left” testing

Source: Delphix
 9

FIGURE 3 Parallel project lines allow teams to execute multiple projects simultaneously

Source: Delphix

Project teams can quickly provision environments Delphix helps customers meet these requirements
dedicated to not only application development, by delivering copies of production data as rapidly
but also training, integration testing, reporting, as a containerization solution delivers code or
or break/fix processes. The efficiency with which configurations.
Delphix creates new temporary environments
increases IT responsiveness, lowers barriers to new By filling this “data gap,” Delphix allows teams
pilot programs, and curbs organizational resistance achieve new levels of self-sufficiency, facilitating
to experimentation. non-production landscapes in which every developer
or tester provisions their own individual environment,
Example: One F500 manufacturer used Delphix complete with secure test data. In addition, Delphix
to create 20 parallel dev/test project lines. This enables new models of collaboration, including the
allowed them to increase the number of SAP ability to version control, bookmark, and share test
projects executed per year from 2 to 11, while data with a few mouse-clicks. Application teams
reducing their storage footprint by 90%. already have these tools for managing source code
through the SDLC. Delphix empowers developers
Future State Requirements: to manage data in the same way that they already
Towards DevOps and Continuous Delivery manage source code.

The ability to spin up and tear down data Example: One F100 retailer used Delphix to expand
environments in minutes instead of days lends from 5 shared nonproduction environments to 200,
itself to more flexible, iterative development one for every developer and tester. This facilitated a
practices. Delphix is a natural fit for organizations transition to a Continuous Delivery model involving
embracing DevOps or other methodologies that daily releases instead of monthly releases.
put a premium on fast environment standup and
enhanced environment reproducibility. While businesses must take a measured approach
to architecting their non-production environments,
Tools like Docker, Puppet, and Chef may be they should not be limited by legacy TDM solutions.
able to quickly stand up lightweight application Delphix redefines the notion of the “right-sized”
instances with the right configurations, using nonproduction landscape. With virtualized data,
minimal hardware and no operations time. They Delphix gives customers more flexibility in sizing
do little, however, to mobilize the test data their landscape along with the agility and self-service
needed to provision full environments at the same features to maximize its value.
pace, and with the same level of automation.
Source: Delphix
10

About Delphix

Business is increasingly digital, and applications enable

digital business. The companies that manage their
application portfolios well will be the disruptors and
leaders of their industries. Data is the fuel for application
projects, and Delphix transforms the way that
organizations manage data for their application projects.
Delphix software enables Data as a Service, within a
firm’s on-premise, private, or public cloud infrastructure.
With a unique focus on applications and databases, and
features that exist nowhere else, the Delphix solution
brings enterprise data into the modern age.

Next-Generation Test Data Management is published by Delphix. Editorial content supplied by Delphix is independent of Gartner analysis. All Gartner research is used with
Gartner’s permission, and was originally published as part of Gartner’s syndicated research service available to all entitled Gartner clients. © 2016 Gartner, Inc. and/or its affiliates.
All rights reserved. The use of Gartner research in this publication does not indicate Gartner’s endorsement of Delphix’s products and/or strategies. Reproduction or distribution
of this publication in any form without Gartner’s prior written permission is forbidden. The information contained herein has been obtained from sources believed to be reliable.
Gartner disclaims all warranties as to the accuracy, completeness or adequacy of such information. The opinions expressed herein are subject to change without notice. Although
Gartner research may include a discussion of related legal issues, Gartner does not provide legal advice or services and its research should not be construed or used as such.
Gartner is a public company, and its shareholders may include firms and funds that have financial interests in entities covered in Gartner research. Gartner’s Board of Directors may
include senior managers of these firms or funds. Gartner research is produced independently by its research organization without input or influence from these firms, funds or their
managers. For further information on the independence and integrity of Gartner research, see “Guiding Principles on Independence and Objectivity” on its website.