Manual Testing Tutorial

Chapter 7: Test Execution and Defect Reporting

7.1 What is Test Execution?

Test Execution is the process of executing test cases as defined in the test plan and observing the actual outcomes.
During test execution, testers run the tests, record the results, compare them with the expected outcomes, and
determine whether the system is functioning as expected.

7.2 Test Execution Process

The test execution process typically involves the following steps:

1. Preparation:

o Ensure that the testing environment is set up.

o Confirm that all required tools, applications, and test data are ready for execution.

2. Executing Test Cases:

o Begin executing the test cases according to the test plan.

o Follow the test steps defined in the test case documentation.

o Capture screenshots, logs, or other evidence as necessary to support results.

3. Comparing Actual and Expected Results:

o After executing each test, compare the actual results with the expected results.

o Identify any discrepancies between the two.

4. Logging Defects (if any):

o If the test fails, log a defect or bug, detailing the issue, steps to reproduce, and severity.

o Provide detailed information, such as error messages, screenshots, or logs, to help developers fix the
issue.

5. Reporting Results:

o Update the test case status (Pass/Fail) and provide feedback to stakeholders about the testing
progress.

7.3 Test Execution Phases

Test execution can be broken down into different phases based on the project lifecycle:

 Alpha Testing: Testing conducted by the development team before releasing to the QA team.

 Beta Testing: Testing performed by the QA team or selected end-users before the product is released to the
public.
  Production Testing: The testing phase after the product has been deployed to the live environment to ensure
stability.

7.4 Real-Time Example: Executing Test Cases for an E-commerce Site

Test Case: Add Item to Cart

1. Preconditions:

o User is logged into the application.

o User is on the product listing page.

2. Test Steps:

o Select a product (e.g., "Laptop").

o Click "Add to Cart."

o Navigate to the shopping cart page.

o Verify that the product appears in the cart with the correct details (product name, price, and quantity).

3. Expected Result:

o The product is successfully added to the cart, and the cart reflects the correct product name, price, and
quantity.

4. Actual Result:

o Pass/Fail: Pass

If the test had failed (e.g., the item was not added to the cart), a defect would be logged.

7.5 Defect Reporting

A defect (or bug) is any deviation from the expected result during test execution. Defects are reported so that
developers can fix them. Here are the key components of a defect report:

1. Defect ID: A unique identifier for the defect.

2. Summary: A brief description of the defect.

3. Description: A detailed explanation of the defect, including how it was discovered.

4. Steps to Reproduce: Clear, concise steps to reproduce the defect.

5. Expected Result: What was supposed to happen.

6. Actual Result: What actually happened.

7. Severity/Priority: The impact of the defect (e.g., Critical, Major, Minor).

8. Environment: Details about the environment in which the defect was found (e.g., OS, browser version).

9. Attachments: Screenshots, logs, or other files that provide more details about the defect.

7.6 Real-Time Example of Defect Reporting

Let’s assume during test execution, we encountered an issue while adding an item to the shopping cart.
  Defect ID: D_001

 Summary: Product not added to the cart after clicking "Add to Cart."

 Description: When attempting to add a product to the cart, the cart does not update with the product.

 Steps to Reproduce:

1. Log in to the application.

2. Navigate to the product listing page.

3. Select the product "Laptop."

4. Click the "Add to Cart" button.

5. Navigate to the cart page.

 Expected Result: The product "Laptop" should appear in the cart with the correct price and quantity.

 Actual Result: The product does not appear in the cart.

 Severity/Priority: High (since the shopping cart functionality is a critical feature).

 Environment: Windows 10, Chrome 90.0.

 Attachments: Screenshot showing the cart page with no product.

7.7 Defect Lifecycle

Once a defect is reported, it goes through the following stages in its lifecycle:

1. New: The defect has been identified and reported but not yet assigned for fixing.

2. Assigned: The defect is assigned to a developer or team for investigation and resolution.

3. In Progress: The developer is working on fixing the defect.

4. Fixed: The defect has been fixed and the developer has verified the solution.

5. Retesting: The QA team tests the fix to ensure the defect is resolved and no new issues have been introduced.

6. Closed: If the defect is successfully fixed, it is closed. If the defect is not reproducible or not valid, it may be
closed as "Not a Bug."

7. Rejected: If the defect is not deemed critical or is determined to be working as expected, it may be rejected.

7.8 Key Defect Reporting Tools

There are several tools available for defect tracking and reporting, including:

1. JIRA: One of the most popular bug tracking tools, used for agile project management and issue tracking.

2. Bugzilla: An open-source defect tracking tool, often used in open-source projects.

3. Trello: A simple board tool that can be used for tracking bugs in smaller projects or teams.

4. Redmine: A project management tool with issue tracking capabilities.

5. Mantis: A web-based open-source issue tracking tool.

7.9 Key Considerations in Test Execution and Defect Reporting

 1. Documentation: Ensure all steps, results, and defects are well-documented to provide clear insights for
developers and stakeholders.

2. Reproducibility: Ensure that defects are reproducible by providing clear, actionable steps.

3. Timely Reporting: Report defects as soon as they are found to prevent delays in the development process.

4. Severity vs. Priority: Understand the difference between defect severity (the impact on functionality) and
priority (how soon it should be fixed).

5. Communication: Effective communication between QA and development teams is essential for resolving
defects efficiently.

7.10 Conclusion

Test execution and defect reporting are critical stages in the software testing process. By following a structured
approach to executing tests and logging defects, teams ensure that the software meets its quality standards. Effective
defect reporting and management contribute to a smoother development cycle and higher-quality software.

Chapter 8: Test Reporting and Closure

8.1 What is Test Reporting?

Test Reporting is the process of documenting and communicating the results of the testing phase. It involves
summarizing the outcomes of executed tests, tracking defects, and providing stakeholders with a clear overview of the
quality of the product.

The primary goal of test reporting is to offer transparency about the status of testing and to provide stakeholders with
the necessary information to make informed decisions about the product's readiness.

8.2 Importance of Test Reporting

Test reports are essential for:

 Tracking Progress: They provide a snapshot of test execution and defect statuses, which helps in understanding
the progress of testing activities.

 Informed Decision Making: They assist stakeholders (e.g., product owners, developers) in making decisions on
release readiness or further work needed.

 Quality Assurance: Test reports document whether the software meets the defined acceptance criteria and
quality standards.

 Documentation and Compliance: They serve as official records for audits and quality control.

8.3 Components of a Test Report

A well-structured Test Report includes the following key components:

 1. Test Summary:

o A brief overview of the testing activities, including objectives, scope, and the testing environment.

2. Test Execution Results:

o A summary of test cases executed, passed, failed, or blocked.

o Provides an overall status of test execution.

3. Defect Summary:

o A summary of defects identified during testing, including their severity and status (open, in-progress,
fixed, closed).

4. Test Coverage:

o Indicates the percentage of the total application or functionality tested against the test plan.

5. Test Metrics:

o Metrics such as test case execution time, defect density, and pass/fail ratio, helping in assessing the
efficiency and effectiveness of testing.

6. Risk and Issues:

o Identifies any risks, blockers, or challenges encountered during testing.

7. Conclusion:

o A summary of the testing status and whether the product is ready for release, along with any open
issues that need resolution.

8.4 Test Execution Report Example

Test Report Summary:

- Project: E-commerce Website Testing

- Test Execution Period: January 5, 2024 – January 15, 2024

- Test Manager: John Doe

- Test Environment: Windows 10, Chrome v90, Production Server

Test Execution Results:

- Total Test Cases: 50

- Test Cases Passed: 40 (80%)

- Test Cases Failed: 8 (16%)

- Test Cases Blocked: 2 (4%)

Defect Summary:

- Total Defects Logged: 10

- Critical Defects: 2

- Major Defects: 5
 - Minor Defects: 3

- Defects Closed: 5

- Defects Pending: 5

Test Coverage:

- Login Functionality: 100% tested

- Shopping Cart Functionality: 90% tested

- Checkout Process: 80% tested

- Payment Gateway: Not tested (Out of Scope)

Test Metrics:

- Test Execution Time: 30 hours

- Pass/Fail Ratio: 80% Pass

- Defect Density: 2 defects per 10 test cases

Risk and Issues:

- Limited test data for edge cases in the cart functionality.

- Performance issues observed on the payment page during high traffic.

Conclusion:

- Testing is 90% complete.

- The product is not yet ready for release due to critical defects in the checkout process and
performance issues.

- Pending defects need to be addressed before the release.

8.5 Test Closure

Test Closure is the final phase of the testing process, where the testing activities are formally concluded, and the
testing team prepares for project completion. This phase involves evaluating the entire testing process, ensuring that
all necessary documentation is completed, and providing final reports to stakeholders.

8.6 Key Activities in Test Closure

1. Test Summary Report:

o Prepare a comprehensive report summarizing all test activities, results, and defect status, as shown in
the previous example.

2. Defect Report Finalization:

 o Ensure that all defects are properly logged, tracked, and resolved. Any open defects that may impact
the release should be flagged.

3. Test Artifacts Finalization:

o All test cases, test scripts, defect logs, and other related artifacts should be archived and finalized for
future reference.

4. Lessons Learned:

o A retrospective session to review what went well and what could have been improved during the
testing process. This helps in improving future testing cycles.

5. Stakeholder Sign-off:

o Obtain final approval from stakeholders (e.g., product owners, project managers) to confirm that
testing is complete and the software is ready for release.

6. Release Test Artifacts:

o Provide stakeholders with access to final test deliverables, such as test cases, defect logs, and test
execution reports, for record-keeping or auditing purposes.

8.7 Real-Time Example: Test Closure for E-commerce Website

Test Summary Report:

 Project: E-commerce Website Testing

 Test Execution Period: January 5, 2024 – January 15, 2024

 Test Manager: Jane Smith

 Test Environment: Windows 10, Chrome 90.0, Production Server

Defect Summary:

 Total Defects Logged: 12

 Critical Defects: 3

 Major Defects: 5

 Minor Defects: 4

 Defects Closed: 6

 Defects Pending: 6 (2 critical defects are pending resolution)

Test Completion Status:

Testing has been completed for all planned features except the Payment Gateway, which was out of scope. Most
features have passed, but critical defects need to be addressed before release. The overall pass rate is 80%.

Lessons Learned:

 The testing team faced challenges with the payment integration, as the test environment was not stable,
impacting the ability to perform certain tests.

 Test automation can be improved by adding more automated tests for critical user flows (like login and
checkout).

Test Artifacts Finalization:

 All test cases and defect logs have been reviewed and archived.
  Test execution reports, defect logs, and test summary have been delivered to the stakeholders.

Stakeholder Sign-off:

 Product Manager: Approved (with the understanding that critical defects will be fixed before release).

 QA Lead: Approved.

8.8 Key Considerations for Test Reporting and Closure

1. Accuracy: Test reports should accurately reflect the status of testing, defects, and overall quality.

2. Clarity: Reports should be easy to understand, even for non-technical stakeholders.

3. Comprehensiveness: Ensure that all necessary information (e.g., defect summary, test coverage) is included in
the final report.

4. Timeliness: Test reports and closure activities should be completed promptly at the end of the testing phase to
allow for timely decision-making.

5. Feedback: Use lessons learned to improve future testing cycles and processes.

8.9 Conclusion

Test Reporting and Closure are essential for finalizing the testing phase and ensuring that all testing activities are
properly documented and communicated. A clear test report and formal test closure process help stakeholders assess
the product’s quality and make informed decisions about its release.