Day01 - Lesson01 - Test Plan - Sample - v1.0
Day01 - Lesson01 - Test Plan - Sample - v1.0
Date:
Version:
Status:
Author:
Reviewed by:
Approved by:
Revision History
Table of Contents
1 INTRODUCTION.........................................................................................................................................................5
1.1 PURPOSE....................................................................................................................................................5
1.2 BACKGROUND.............................................................................................................................................5
1.3 SCOPE........................................................................................................................................................5
1.4 ASSUMPTIONS.............................................................................................................................................6
1.6 CONSTRAINTS.............................................................................................................................................8
FAULTS FIXED......................................................................................................................................................15
4 TEST STRATEGY.....................................................................................................................................................16
4.5 TOOLS......................................................................................................................................................18
5 RESOURCES...........................................................................................................................................................18
Software Quality Assurance Test Plan
5.1 STAFFING..................................................................................................................................................18
Test Plan
1 Introduction
1.1 Purpose
1.2 Background
VMware is an operating system that provides for the definition of multiple virtual server operating
environments, each running a different O/S potentially, using a single physical server. VMware is
growing in popularity due to the ROI incurred from power and cooling savings along with intangibles such
as reduced administration costs from consolidating multiple physical machines onto a single machine
that is partitioned. Many XXXXXXX partners are shipping hardware with a VMware option and field sales
is reporting an increase in the number of prospects looking for VMware support by their BAR vendor.
Summary benefits of VMware Server:
Provision a new server in minutes without investing in new hardware.
Run Windows and Linux operating systems and applications on the same physical server.
Increase the utilization of a physical server.
Move virtual machines from one physical host to another without re-configuration.
It is the intention of this document to outline the requirements for testing and validation of the XXXXXXX
7.1.2, and 7.4.2 product s within this operating environment (VMWare ESX 3.0) in order to deem the
solution as supported by the Research and Development organization within the scope of requirements
set forth by the ‘XXXXXXX Software Project Scoping Document - XXXXXXX and VMWare Support’
artifact. This Plan does NOT cover testing of the range of guess operating system support environments
nor the Applications support in guest operating system environments. The intention of this testing will be
to validate the backup and restoration on console and virtual disks on an ESX server version 3.0.
1.3 Scope
The testing is limited to specific tests to cover the functionality addressed by the test suites outlined in
table 1.
Software Quality Assurance Test Plan
For specific functionality being tested against the proposed environments and configurations please refer
to the respective test suites mentioned above at DMS.
1.4 Assumptions
All test binaries shall be downloaded from the Binary Management System
Detailed testscript(s) shall be downloaded from the Document Management System unless otherwise
specified
If necessary and for XXXXXXX own benefits, XXXXXXX may translate or convert the existing test
script(s) to XXXXXXX own latest format test script(s) but there won’t be any resource time allocate for
this task.
XXXXXXX may need to ‘specify’ reduced port test sections from the current test script.
The testscript should be updated with testcases for all fault fixed/new features including in BMSes
under test.
The modular architecture of the XXXXXXX product limits the scope of faults to impact on modules
other than the module where the fault occurs.
This test plan is created based on high-level requirements from BKB emails/meetings. Detailed
information should be added during test phases if needed.
Test artifacts are either resident on the BMS or attached to their relevant FMS/Jira.
All test documentation is resident on the DMS or official VMware websites.
XXXXXXX has all necessary equipments and knowledge to setup environment.
If XXXXXXX is unable to use or is unable to implement the testing with the hardware/test cells in
offshore, then XXXXXXX would need onshore to setup the testing environment. Then new
setup/prepare schedule would need to allocate. It is responsibility of XXXXXXX to notify XXXXXXX if
this is the case.
If XXXXXXX is unable to perform both XXXXXXX1.x and XXXXXXX4.x at the same time due to
resource problem, it is possible that testing will be serialized. It is responsibility of XXXXXXX to notify
XXXXXXX if this is the case.
Testing Staff are competent in the operating environments outlined in table 2.0 and they also have a
degree of understanding of VMWare backup and restore methodologies.
VMWare ESX 3.0 Server Software and Licensing (can be found at Wiki)
Software Quality Assurance Test Plan
Technical Contact at VMWare specifically geared towards providing the means for 3 rd Party Software
Validation. If this is not possible technical contacts that can provide thorough and precise support for
technical alliance related issues will require.
Technical Leads responsible for QA Test Effort issue escalation required.
Note: Again this testing does not cover a testing of neither the range of guess operating system support
environments nor the Applications support in guess operating system environments
GENERAL:
No formal meeting within the Development, QA and PM team for scoping the work before starting a
project.
XXXXXXX core software may have (known and unknown) issues that could be impact to the
application module.
There wouldn’t be any new features, or new fault fixes for this release
Testing being performed by XXXXXXX, lack of experiences of XXXXXXX and VMware may produce
lots of clarification and help request. There should be some buffer time, contingency time allocated for
the project.
If XXXXXXX has problems executing the test script, it could affect the delivery schedule.
XXXXXXX will be using hardware/test cells offshore. Remote access to XXXXXXX Lab machines may
be limited and may affect the testing schedule.
If XXXXXXX unable to use or unable to implement the testing with the hardware/test cells in offshore,
then XXXXXXX would need onshore to setup the testing environment. Then new setup/prepare
schedule would need to allocate. It is responsibility of XXXXXXX to notify XXXXXXX if this is the case.
VMWare:
A1 – XXXXXXX Client on Service Console Validation:
Software Quality Assurance Test Plan
Risk1 – XXXXXXX Client software may have issues with ‘/vmfs’ related mount points (Refer to
Article ID 1166 within VMWare Knowledge Base).
Risk2 – XXXXXXX Client software may have issues with installation on Service Console as the
Service Console runs a proprietary version of Linux (Refer to Article ID 193 within VMWare
Knowledge Base).
Contingency1 – Development Support may be required if this issue arises and prevents progress
with test efforts.
Contingency2 – Development Support may be required if this issue arises and prevents progress
with test efforts.
B1 – XXXXXXX Smart Client on Service Console Validation:
Risk1 – XXXXXXX Client software may have issues with ‘/vmfs’ related mount points (Refer to
Article ID 1166 within VMWare Knowledge Base).
Risk2 – XXXXXXX Client software may have issues installation as the Service Console runs a
proprietary version of Linux (Refer to Article ID 193 within VMWare Knowledge Base).
Risk3 – Since the Service Console runs a proprietary version of Linux our SCSI pass-through
driver may not work properly.
Contingency1 – Development Support may be required if this issue arises and prevents progress
with test efforts.
Contingency2 – Development Support may be required if this issue arises and prevents progress
with test efforts.
Contingency3 – Development Support may be required if this issue arises and prevents progress
with test efforts.
C1 – Guest Operating System Virtual Disk File Backup Investigation:
Risk1 – Not all Linux commands and third-party backup software such as XXXXXXX can access
VMFS directly. Also, this method places a heavy load on the service console, since it has to run
the backup agent software and has to pass all the virtual machine data to tape. As the virtual
disk file grows so does the burden on the system and the network.
Risk 2 – This may require integration and test efforts that may yield unsatisfactory results.
Risk3 – If the Service Console Crashes due to issues related to Virtual Disk Backup / Restore
operations then ALL virtual clients will be inaccessible.
Contingency1 – Other methods to backup Virtual Disk Files will be explored if possible.
Contingency2 – Other methods to backup Virtual Disk Files will be explored if possible.
Contingency3 – Other methods to backup Virtual Disk Files will be explored if possible.
1.6 Constraints
Constrain 2.2 – Service Console runs a proprietary version of Linux some standard XXXXXXX
Client Software functions and features may not be available and may require further R&D efforts.
B2 – XXXXXXX Smart Client on Service Console Validation:
Constraint 4.1 – If Virtual Machine is not available devices and media will also not be available
for backup / recovery operations. These Virtual Machines are recommended to be ALWAYS
accessible to the XXXXXXX Backup Server. If they are not, interaction with the VMWare product
will be required.
Constraint 4.2 – Dynamic Drive Sharing is currently not supported. Preliminary investigations
related to drive locking/unlocking between virtual clients seems to indicate reboots are required.
The investigation has been limited by external resources.
C1 – Guest Operating System Virtual Disk File Backup Investigation:
Constraint 5.1 – We are limited by the access level provided by the XXXXXXX Client Software
with relation to the VMFS filesystem. Although there are Pearl API functions that can be
leveraged, the exact result of such integration work is not yet known. This may result in
unsatisfactory results and may require further R&D efforts.
NOTE 1: There are other tools to export the Virtual Disk Files so that they may be backed up and restored but
this is not a area of expertise hence integration efforts may yield substandard solutions. The following are
alternatives to indirectly access to the VMFS filesystem:
o vmkstools (export to NFS mount)
o SAN Image using 3 rd party application
*NOTE 2: It is expected that the XXXXXXX will carry out the test plan. The access to all the necessary test
environments and equipment will be made available where possible, particularly libraries and servers to outside
members of this test team. Although local access for test purposes is not an issue, and remote access can be
made available for coding efforts, limitations by external variables may vary the effectiveness of the test plan
execution.
The completion date of this project may also depend on the number of defects found during testing and
feedback rate from XXXXXXX development team.
Physical tape drives and library may not available in offshore for testing. Offshore shall use virtual drive
/ library or shall remote use the devices in SD office. Offshore shall notify XXXXXXX about the
limitation of testing result, due to the lack of hardware.
The table below identifies the documentation and availability pertaining to the project:
XXXXXXX 7.4.2 (BMS ID 8345) would be the primary version for Full test scenario and XXXXXXX
7.1.2 (BMS ID 6566) would be version for Reduced test scenario.
Various binaries will be used to certify specific configurations for use within target VMWare environments. Table
3 outlines those binaries which will be targeted for test and validation.
The primary goal will be to conduct basic backup and restore testing on the ESX Server 3.0 based on the
VMWare test suite.
Test environments breakdown:
ESX Server 3.0 will have two guess operating systems:
Windows 2003 latest sp
Linux RHELV3U4
The following matrix is reviewed by XXXXXXX. For each of the combinations, XXXXXXX selects the type
of testing needed. Some of the combinations can be skipped. Some others may not be available and not
need to be validated. In the table below, only the ones that offshore shall test are listed. For a complete
list of all combinations, please find it in the spreadsheet attached below.
The table also contains the BMS ID’s used for testing. It shall be updated with the IDs during the
execution phase.
# Tier Priority VMWare Server Xxxx Xxxx Testing Binary Xxxx Duration Start/Finish
Installed on BMS ID Type? (Full available Server Date
VMWare or Reduced) on? BMS ID
Server
1 1 1 ESX Server 3.0 on 7.4.2 8345(GA) Full Now 8368 (GA) 07 days 07/13/2006
x86 based system 07/25/2006
2 1 2 ESX Server 3.0 on 7.1.2 6566(GA) Reduced Now 8345 (GA) 05 days 07/26/2006
x86 based system 08/1/2006
3 2 3 ESX Server 3.0 on 7.4.2 8363(GA) Full Now 8368 (GA) 07 days On Hold
x86-64 based system
4 2 4 ESX Server 3.0 on 7.1.2 6610(GA) Reduced Now 8345 (GA) 05 days On Hold
x86-64 based system
Note: The 64-bit testing project is on hold until further notice from XXXXXXX and the decision whether to support 64-
bit environment will be determined.
Software Quality Assurance Test Plan
2.2.1 Topology
Table 4 outlines the topology and ESX / Virtual Machine relationship which will be targeted for test and
validation. There are various approaches to testing based on hardware availability. Topologies A through C are
based on limited resource availability. Topology D is based on abundant resource availability.
Note: Increases in hardware availability for testing yields more parallelism and allows for adjustments which can
be advantageous to the test schedule.
A B C
Software Quality Assurance Test Plan
The configurations targeted for test and validation include various operating systems and binaries
running as guest operating environments within Virtual Machines hosted by the ESX server. These
environments will be targeted for backup and restore operations via two main Backup Server Domains.
These environments are listed in table 4 and should be reviewed for approval.
Software Quality Assurance Test Plan
Configuration
System Operating System Role
*Total System Memory will depend on minimum guest operating system requirements (highlighted in bold within
table 5) and a formula which outlines ESX server / Console and virtual system requirements. This formula is as
follows:
24MB_192MB+(1.06*N*(M+54MB)) where N=number of virtual machines and M=total target
memory for each virtual machine.
3 Test Requirements
This section states testing scope for VMWare ESX 3.0 backup and restoration.
The listing below identifies those items (use cases, functional requirements, non-functional requirements) that have
been identified as targets for testing. This list represents what will be tested:
Configurations
1. XXXXXXX Server running on Windows 2003 SP1 with XXXXXXX 7.4.2 and 7.1.2
2. XXXXXXX Client running on ESX Server 3.0 with XXXXXXX 7.4.2 and 7.1.2
Operations
1. Filesystem Backup to Primary Servers via locally attached device(s). Data from Virtual Machines
(Refer to table 2 for details) and ESX Service Console.
2. Filesystem Restore from Primary via locally attached device(s). Data to Virtual Machines (Refer
to table 2 for details) and ESX Service Console.
3. Filesystem Backup² to Primary and Secondary Servers via Smart Client with remotely attached
device(s). Data from Virtual Machines (Refer to table 2 for details) and ESX Service Console.
4. Filesystem Restore² from Primary and Secondary Servers via Smart Client with remotely
attached device(s). Data to Virtual Machines (Refer to table 2 for details) and ESX Service
Console.
5. VMFS – Virtual Machine File direct Backup and Recovery.
Major Components to be tested for backup and recovery
Virtual disks
Virtual machine configuration files
The configuration of the ESX Server itself
New features shall be tested
This is strictly a compatibility/validation testing with new version of VMWare ESX server 3.0 for a GA
XXXXXXX server and client products. So no new features validation required for this project.
Faults Fixed
This is strictly a compatibility/validation testing with new version of VMWare ESX server 3.0 for a GA
XXXXXXX server and client products. So no faults validation required for this project.
Software Quality Assurance Test Plan
4 Test Strategy
The Test Strategy presents the recommended testing approaches. The previous section on Work Products Under
Test and Test Requirements described what should be tested; this describes how it shall be tested.
Phase 1: The primary strategy will be to determine if there are any issues with simple backup and restore operations
when conducting said operations on ESX Server 3.0 service console server which also has two guest operating
environments hosted by VMWare Virtual Machines. The primary tool for this will be the VMWare ESX Test Suite.doc,
the file system plugin test script (if applicable), test data generation, and test data validation tools. In addition there
may be a need to run subsequent test cases based on findings resulting from those efforts.
Full Test: All of the test cases in the related test scripts and common plugin test script must be run.
Reduced Test (Port Test): All of the test cases marked as “Port Test” in related testscripts must be run.
(Note: XXXXXXX may need to ‘specify’ or define reduced port test sections from the current test script)
QA cannot skip any test cases in the test scripts without approval from QA lead engineer.
New fault found during regression testing shall be retested again, once XXXXXXX provides new binary that
addresses the issues. However, it is the right of XXXXXXX to determine whether the fixes should be
included in the release or not.
When receiving new binary to address bad fixed FMSes, only FMSes retested not fixed and not started in
previous binaries shall be tested in this new binary.
Testing shall be completed when all tests in the test plan (Testscript execution and regression testing) pass
on the designated platforms and documentation is uploaded to DMS.
The following table outlines the test iterations to be executed per platform targeted for validation (outlined in table 2)
as follows:
Software Quality Assurance Test Plan
No Severity 1 level issues. As defined in the XXXXXXX Customer Support Handbook, Severity 1
issues are defined as: “Customer cannot recover data critical to its business, and there is no
workaround available.”
For test specifications being run pass criteria is stated in those documents.
No compatibility issues with ESX Server 3.0. XXXXXXX should support ESX Server 3.0 constantly
as it does with previous VMWare ESX Server versions being supported (ESX 2.5.x…)
Finish all test scripts and all are marked as Passed for designated platforms in the scope.
Finished all other test cases in the scope and the result is reviewed/accepted/approved by QA
Lead Engineer.
Regression testing coverage shall be determined by consultation with development and the QA
lead engineer.
The trigger for regression testing shall be changed which put critical or high importance
functionality at any real risk
Any limitations or restrictions of the plugin shall be documented in the Engineering Release Notes
(ERN)
Based on SQA process, Application Note or ERN for this project should be created and completed
by test team if needed.
4.5 Tools
5 Resources
This section presents the recommended resources for the testing of VMWare ESX 3.0 project, their main
responsibilities, and their knowledge or skill set.
This table shows the staffing assumptions for the project. More QA resources shall/can be staffed in the future based
on workload and deadline of the release.
5.1 Staffing
The following table sets forth the system resources for the testing project.
System Resources
Software Quality Assurance Test Plan
5.2.2 Software
XXXXXXX will download ESX Server 3.0 ISO from VMWare web site.
xxxxxxxxxxxxxxxxxxxxx
This section states the machine setup model for the execution of this project.
01 ESX Server 3.0 for each testing (71.2 and 7.4.x)
01 xxxxx for each testing (7.1.2 and 7.4.x)
If possible, use real tape drive or tape drive library. Otherwise, use VTL.
6 Communications Channels
All project communications/issues/problems are handling following the channels in the chart above:
Contact information:
xxxxxxxxxxxxxxxxxxxxxxxxxxxx
7 Estimation
The below table presents the estimation for this project. The project cycle is broken down into Plan /
Prepare / Execute / Closure phase with appropriate high-level tasks. Please refer to project schedule
(MPP) file to get more detail information about tasks, progress, and milestones with appropriate start
dates, end date, assigned resources
8 Deliverables
This is the list of deliverable items necessary for this project. It includes deliverable items from XXXXXXX
to XXXXXXX and from XXXXXXX to XXXXXXX. For all test specs run:
A copy of the specification that has been run, renamed to reflect the test build under test.
3 Binaries for Testing XXXXXXX GA/MA Binaries for XXXXXXX and other plugins
if applicable
4 Test Result XXXXXXX N/A Completed testscripts, ERN
5 Test Summary Report, Fault XXXXXXX N/A
Report
8 Project Plan Vu Pham N/A To monitor and report tasks
Don Le
Brandon Pham
Defects are to be recorded in fault records raised or resubmitted in the Jira system. Any fault that is to
be tested shall be included in this test plan, Test Report and/or Engineering Release Notes (ERN) of
related BMS.
Defects are included also in Test Summary Report in Closure phase.
Please remember to refer to the software quality assurance (SQA) process document found at the following
location in DMS, for process guidelines: Documents/Test and Quality Assurance/Process, Procedures and
Guides/SQA Process Document/SQA Process.doc