Testing Document Team 18

TESTING DOCUMENT

Project:​ Metamorphobot
Task:​ Construct a robot that can decrease in size to fit through a tunnel, widen to go over a
bridge, identify coloured blocks while in a specified search area.

Document Version Number​: ​1.8

Date: ​12/04/18
Author: ​Sarah Swanson

Edit History:
Version Date Author Section Summary
Updated

1.0 27/02/18 Sarah Swanson All Preliminary

Document
1.1 14/03/18 Sarah Swanson, 4.0 First test
Kevin Tchang completed
1.2 20/03/18 Sarah Swanson 5.0,2.0 Future tests to
be
implemented,
Process of
testing
1.3 28/03/18 Sarah Swanson 2.0 Bullets to
sentences on
main ideas
1.4 04/04/18 Sarah Swanson All Reformatting

1.5 06/04/18 Sarah Swanson 5.0 Completed

tests #1,2,3 of
section 5
1.6 08/04/18 Sarah Swanson 2.0 Bullets to
sentences on
testing ideas
and necessities
1.7 09/04/18 Sarah Swanson 5.0 Completed
remaining tests

1.8 12/04/18 Sarah Swanson 6.0 Final Demo

Report

1
Testing Document Team 18

1.0 Table of Contents

1.0 Table of Contents 2

2.0 Process of Testing 3

2.1 Hardware Testing 3

2.2 Software Testing 3

3.0 Testing Template 5

4.0 Test Results for Changing Wheelbase Robot Design 6

5.0 Test Results for the Final Robot 8

5.1 Basic Tests 8

5.2 Unit Tests 13

5.3 Integration Tests 17

6.0 Final Demonstration Report 22

2
Testing Document Team 18

2.0 Process of Testing

The robot has many parts to. Both software and hardware components. Therefore, many
aspects should be tested on the robot. The parts also need to be tested individually and all
together. If the testing just tested the robot all together, it would be harder to identify which part
of the robot is not working. Therefore each sensor is tested individually, each class that needs
to be tested is tested individually and the wheelbase and other hardware components are tested
individually. Once the hardware and software has been tested, the robot as a whole needs to be
tested.

A test is most often defined as a procedure of to follow to test a part or multiple parts to
identify a weakness in the software or hardware. Tests are meant to be run multiple times to
gain a stronger case against the weakness.

If a test is run multiple times and the error is caught, it is important to fix the error before
moving on to another test. Testing shows the amount of error when the robot is run or observed,
therefore it makes it easier to change numbers or other aspects and test again. If the robot is
not fixed, the next test could be affected by the previous test. Once the robot is fixed, the test
should give consistent wanted results.

2.1 Hardware Testing

The components of the hardware need to be separately tested to determine the

accuracy of each component. It is important to observe the degree of how well the sensors work
in ambient lighting. If the sensors do not work well, the alternative is to put black tape around
the bottom of the sensor the light differentiates less in different lightings. The intensity of the
colour sensors must also be tested so that the robot can properly detect the black lines.

The wheels must also be taken into consideration with hardware testing. The wheel slip
must be observed with respect to acceleration and also the amount of dust on the floor. The
constraints on the floor dust is expanded in section 3.0 of the Constraints Document. The wheel
radius and wheelbase length for final robot design must be tested to correspond with the code.

2.2 Software Testing

The software has a lot of testing that needs to be completed to have a successful run for
the robot. The hardware has to be finalized before the software classes can be tested. First of
all, each class must be tested separately to determine the accuracy of each portion of the

3
Testing Document Team 18

robot’s run. Navigation, Odometry, Localization, Searching, Wifi, etc. must all be tested
separately in order for the robot to be able to run all of these methods together. Once they are
all tested separately, test all together. This integration will run smoothly if all the separate tests
are done and fixed beforehand. The main point of the software testing is to make sure the
software works well with the hardware that is built.

4
Testing Document Team 18

3.0 Testing Template

Test #: Name of the Test (The Purpose)

Test Number: ​#
Date: ​day/month/year
Tester: ​team members testing
Author: ​team member who recorded on the document
Software Description: ​updates of the software of the robot for this test
Hardware Description: ​updates of the hardware of the robot for this test

Goal: ​Purpose of the test- why is it needed to be tested?

Procedure: ​Detailed description of the test procedure and what is being recorded
Expected Result: ​Brief explanation of the outcomes predicted from the test

Test Report: ​Provide the data acquired from the test with the number of runs. An analysis of the
testing results is required.
Conclusion: ​State the outcome of the tests and briefly explain if the test was successful and
met the requirements.

Action: ​Provide a list of the following step(s) that need to be accomplished and who needs to
accomplish them. If the Gantt chart needs to be updated, notify the Project Manager.
Distribution: ​Record and notify the following team members that need to be aware of this test
to make any following changes.

5
Testing Document Team 18

4.0 Test Results for Changing Wheelbase Robot Design

The original robot had a changing wheelbase as shown in section 6.0 of the Hardware
Document. Although this model would have helped the robot travel over the bridge because of
the wider wheelbase, there were faults in the design. The robot was too heavy, causing a dip in
the wheels and they would widen out whenever the robot was travelling. The was a major lack
of stability. To show that the robot design needed to be changed, a Navigation Test follows to
show inconsistent navigation.

Test #1- Navigation Test

Test Number: ​#1

Date: ​20/03/2018
Tester: ​Kevin Tchang
Author: ​Sarah Swanson
Software Description: ​Version 1.2 (Refer to Software Document)
Hardware Description: ​Version 1.2 (Refer to Hardware Document)

Goal:
The purpose of this test is to determine the accuracy of the navigation with the changing
wheelbase robot.

Procedure:
1) Turn on EV3 and make sure it has at least 7.0 battery or above.
2) Program the robot to travel to waypoints (2,1), (1,1), (1,2), (2,0).
3) Note down the destination waypoint ( X d , Y d ) in cm.
4) Place the robot on the origin of the grid tile.
5) Ensure there are no obstacles in the robot’s path.
6) Run the program to drive the robot through the waypoints specified in Step 1.
7) Record ( X f , Y f ) which is the final position of the robot.

Expected Result:
It is expected that the robot can go to any point on the grid while staying completely
straight on the grid lines. The robot is expected not to localize and be able to stay straight.

6
Testing Document Team 18

Test Report:

Conclusion:
The robot did not have more than -4.34cm error in the X direction and -3.98cm error in
the Y direction. The mean is -2.56cm and standard deviation is 1.05cm in the X-direction, and
the mean is -2.92cm and standard deviation is 0.83cm in the Y-direction. From these results, we
decided that the robot was never close enough to the final destination. This is not because we
had an incorrect wheelbase or track in the code, but because the hardware was unstable and
the wheels would slide in and out while travelling. The wheels went in and out randomly while
travelling, explaining the sporadic results.

Action:
The final decision was to get rid of the changing wheelbase because it was not stable
enough. A more simple robot was built and is shown in section 7.0 of the Hardware Document.

Distribution:
Hardware, Software

7
Testing Document Team 18

5.0 Test Results for the Final Robot

The final robot is shown in section 7.0 of the Hardware Document. These tests are done
to show that taking out the changing wheelbase increased the accuracy of the robot’s navigation
and localization.

5.1 Basic Tests

Test #1 - Exact Wheel Radius Test

Test Number: ​#1

Date: ​12/03/18
Tester: ​Kevin Tchang
Author: ​Kevin Tchang
Software Description: ​Custom test code
Hardware Description: ​Version 1.3 (Refer to Hardware Document)

Goal:
To determine to a higher accuracy the exact radius of the wheels. The value is important
as it affects the accuracy of the distance travelled; and will be required as an input parameter
whenever the robot moves.

Procedure:
1) Write a program that makes both wheels of the robot turn a total of 7 revolutions at a
speed of 50. Make sure the acceleration and deceleration is gradual.
2) Run the program and measure the distance of the final position of the robot from the
starting position.
3) Repeat step 2 seven more times.
4) Calculate the average distance travelled from the seven trials.
5) Calculate the error from the difference between the expected distance and the measured
distance.
6) Calculate the exact wheel radius using the mean distance obtained in step 4 and the
formula below:

mean distance travelled

radius = 2π * number of rotations

7) Run the program five more times with the newly calculated wheel radius and measure
the distance travelled.

8
Testing Document Team 18

8) Calculate the error from the difference between the expected distance and the measured
distance. Compare this error with the previous error.

Expected Results:
The measured distances should remain fairly consistent with one another (small
standard deviation), as the only major source of error should come from observational error
when measuring the distance using the ruler. The average distance error after using the
calculated wheel radius should be lower than the initial average distance error.

Test Report:

Using old wheel radius value:

Expected distance after 7 revolutions (using wheel radius of 2.1 cm) = 92.36 cm

Trial Distance travelled (cm) Distance error (cm)

1 91.45 0.91

2 90.55 1.81

3 91.55 0.81

4 91.15 1.21

5 91.55 0.81

6 91.45 0.91

7 91.55 0.81

Average distance travelled = 91.321 cm

Average distance error = 1.04 cm (control)

91.321 cm
radius = 2π * 7

radius = 2.076 cm

Using new wheel radius value:

Expected distance after 7 revolutions (using wheel radius of 2.076 cm) = 91.32 cm

9
Testing Document Team 18

Trial Distance travelled (cm) Distance error (cm)

1 91.3 0.02

2 90.25 0.07

3 91.45 0.13

4 91.55 0.23

5 91.45 0.13

Average distance error = 0.12 cm

Conclusion:
The goal was achieved as the result of using the newly calculated wheel radius brought
the average distance error down by almost 1 cm. The wheel radius is determined to be
approximately 2.076 cm; as opposed to the previously assumed (less accurate) value of 2.1 cm.

Action:
Update the current software version by replacing all instances of the old wheel radius
with the new wheel radius. Keep this value for all future software versions as well.

Distribution:
Software team (Kiren, Kevin)

Test #2- Light Sensor Accuracy Test

Test Number: ​#2

Date: ​06/04/2018
Tester: ​Sarah Swanson
Author: ​Sarah Swanson
Software Description: ​Version 1.7 (Refer to Software Document)
Hardware Description: ​Version 1.5 (Refer to Hardware Document)

Goal:
To determine how accurate the colour sensor is while detecting black lines.

Procedure:
1) Turn on EV3 and make sure it has at least 7.0 battery or above.
2) Select the robot to travel to a point where it will cross 3 lines.
3) Place the robot on the playing field so it will move past 3 black lines when run.

10
Testing Document Team 18

4) Run the program.

5) Record if the robot beeps at the exact moment the sensor crosses the black line.
6) Repeat in different lightings.

Expected Result:
The expected result is that the accuracy of the sensor will decrease if the light gets much
darker or much brighter. Also, the sensor is not always 100% accurate and may miss or detect
an extra black line.

Test Report:

Conclusion:
The light sensor was very accurate each time our robot was run to detect black lines.
The sensor detected the lines 3 out of 3 times for both the “DPM light” on and “DPM light off”.
The third test, “DPM light off & poster board above robot to block light”, detected four out of
three black lines in run 4 because the sensor detected a change in colour of the wood as a line.
The playing field is not consistent in colour in some spots, therefore this may cause
inconsistency in the sensor.

Action:
Keep using the light sensor to detect black lines because it is shown consistent.

Distribution:
Hardware

Test #3- Ultrasonic Sensor Accuracy Test

Test Number: ​#3

Date: ​06/04/2018
Tester: ​Sarah Swanson
Author: ​Sarah Swanson

11
Testing Document Team 18

Software Description: ​Version 1.7 (Refer to Software Document)

Hardware Description: ​Version 1.5 (Refer to Hardware Document)

Goal:
The purpose is to determine the accuracy of the ultrasonic sensor.

Procedure:
1) Turn on EV3 and make sure it has at least 7.0 battery or above.
2) To test the ultrasonic sensor, use the test in the Tools section. Select Tools>Test
Sensors>GO>Port #>EV3 Ultrasonic>Distance.
3) Place the robot with the edge of the ultrasonic sensor at the first black line. Have the
robot face the wall as shown in ​Figure 1​.
4) Record the distance from the tile line to the wall and compare with the actual given
value. The actual distance from the wall to the black line is 30.48cm.
5) Repeat step four for distances of 60.96cm (two blocks distance) and 91.44cm (three
blocks distance).
6) Find the error and the standard deviations to view the accuracy of the ultrasonic sensor.

Figure 1: The distance of 30.48cm from the wall to the ultrasonic sensor

Expected Result:
The expected results are that the values are not exactly 30.48, 60.69 and 90.44 because
of any wave interference there may be in the room. The ultrasonic sensor works by sending out

12
Testing Document Team 18

sound waves, therefore if there is a high pitched noise, this could affect the accuracy of the
sensor.

Test Report:

Conclusion:
The error that is shown in the test report is no more than 0.5cm. This result corresponds
to what the expected result says where the use of the ultrasonic sensor will always have errors
because of the inconsistency of sound waves in different environments. Therefore, the results
hold with the expected result.

Action:
This test has showed us that we should not make extreme noises while testing our robot.

Distribution:
Let the entire team know about the wave interference.

5.2 Unit Tests

Test #4- Localization Test

Test Number: ​#4

Date: ​08/04/2018
Tester: ​Sarah Swanson
Author: ​Sarah Swanson
Software Description: ​Version 1.7 (Refer to Software Document)
Hardware Description: ​Version 1.5 (Refer to Hardware Document)

13
Testing Document Team 18

Goal:
The purpose of the localization test is to see if the robot can consistently localize with the
ultrasonic localization. The localization must be done in 30 seconds.

Procedure:
1) Place the robot in the corner block (it is okay if the wheels are sticking out a bit because
the robot is wide) so it does not hit any walls
2) Run the ultrasonic falling edge localization on the EV3 Brick
3) Repeat steps with different orientations
4) Record the ultrasonic angle error, euclidean distance error and final angle error

Expected Result:
The expected result is that the robot will localize perfectly most of the time (from
previous experience in the labs). Since the ultrasonic sensor is not perfect, as shown in Test 3,
there is always a possibility for one of the localizations to not work perfectly.

Test Report:

Sample calculation of the Euclidean Distance Error ϵ:

The Euclidean error distance ϵ between (xf ,xy ) and the (0,0) point is calculated as follows:

Where xf and yf are the components of the robot’s actual position.

14
Testing Document Team 18

Sample calculation of the Mean:

The general formula of the mean is calculated as follows:

Where xn is an element in the data set, ​N​ is the number of elements in the data set and ​n​ is the
number count of the element.

Sample calculation of the Standard Deviation:

The general formula for the standard deviation is calculated as follows:

Where x bar is the mean, xi is the value from the data and ​n ​is the number of runs.

Conclusion:
As seen from the test report, the final angle errors are all around 2 degrees error. This
means that the localization is very consistent. The final angle error mean is 2.48 degrees and
the standard deviation is 0.30 degrees. In the competition, if the person who places the robot
can place it near where it was placed for us, the localize should perform well.

Action:
Let the software lead know that the ultrasonic localization will work for the competition.

Distribution: ​Software

Test #5- Navigation Test

Test Number: ​#5

Date: ​09/04/2018
Tester: ​Sarah Swanson
Author: ​Sarah Swanson
Software Description: ​Version 1.7 (Refer to Software Document)
Hardware Description: ​Version 1.5 (Refer to Hardware Document)

Goal:

15
Testing Document Team 18

The purpose of this test is to determine the accuracy of the navigation with the changing
wheelbase robot.

Procedure:
1) Turn on EV3 and make sure it has at least 7.0 battery or above.
2) Program the robot to travel to waypoints (2,1), (1,1), (1,2), (2,0).
3) Note down the destination waypoint ( X d , Y d ) in cm.
4) Place the robot on the origin of the grid tile.
5) Ensure there are no obstacles in the robot’s path.
6) Run the program to drive the robot through the waypoints specified in Step 1.
7) Record ( X f , Y f ) which is the final position of the robot.

Expected Result:
It is expected that the robot can go to any point on the grid while staying completely
straight on the grid lines. The robot is expected not to localize and be able to stay straight.

Test Report:

Sample calculation of the Euclidean Distance Error ϵ:

The Euclidean error distance ϵ between (xf ,xy ) and the (0,0) point is calculated as follows:

Where ​xf​ and ​yf​ are the components of the robot’s actual position.

16
Testing Document Team 18

Sample calculation of the Mean:

The general formula of the mean is calculated as follows:

Where xn is an element in the data set, ​N​ is the number of elements in the data set and ​n​ is the
number count of the element.

Sample calculation of the Standard Deviation:

The general formula for the standard deviation is calculated as follows:

Where x bar is the mean, xi is the value from the data and ​n ​is the number of runs.

Conclusion:
The navigation for the new robot design without the changing wheelbase is more
accurate. The mean is 0.06cm and the standard deviation is 0.10cm in the X direction. The
mean is -0.13cm and the standard deviation is 0.10cm in the Y-direction.Comparing to the
previous values of the changing wheelbase, the mean is -2.56cm and standard deviation is
1.05cm in the X-direction, and the mean is -2.92cm and standard deviation is 0.83cm in the
Y-direction. The robot without the changing wheelbase is shown to be a lot more accurate to the
desired waypoints.

Action:
Notify software that the navigation is perfected to the amount wanted.

Distribution:
Hardware, Software

5.3 Integration Tests

Test #7- Travel To and Through the Tunnel Test

Test Number: ​#7

17
Testing Document Team 18

Date: ​09/05/2018
Tester: ​Sarah Swanson
Author: ​Sarah Swanson
Software Description: ​Version 1.7 (Refer to Software Document)
Hardware Description: ​Version 1.5 (Refer to Hardware Document)

Goal:
The purpose of this test is to determine if the robot can successfully travel to and through
the tunnel.

Procedure:
1) Turn on EV3 and make sure it has at least 7.0 battery or above.
2) Program the robot to travel to waypoint where the tunnel is located. In this case we will
test with the coordinates (3,5).
3) Run the robot to travel to and cross the tunnel.
4) Record if the robot successfully navigates and passes through the tunnel.

Expected Result:
Since the robot’s width is small enough to travel through the tunnel, the outcome should
be that it can pass through every time as long as the Navigation is correct.

Test Report:

Successful Navigation Successful Tunnel Traversal

Run 1 Yes- perfect positioning Yes- Did not touch the walls

Run 2 Yes- a little off because of Yes- Touched walls but still
wheel slip managed to get through. Need
extra localization at the end.

Run 3 Yes- perfect positioning Yes- Did not touch the walls

Run 4 Yes- perfect positioning Yes- Did not touch the walls

Run 5 Yes- a little off because of Yes- Touched walls but still
wheel slip managed to get through. Need
extra localization at the end.

Conclusion:
As seen in the chart, there is a correlation between the navigation and the
successfulness of tunnel traversal. The floor is very dusty at times which causes a lot of wheel
slip in the robot. When the robot slips, the navigation is affected. This affects the perfect

18
Testing Document Team 18

positioning of the robot in front of the bridge. As long as the wheels and the floor is dusted, the
robot should have a clear and successful navigation and traversal.

Action:
Let the team know that before testing, to dust the wheels and the floor.

Distribution:
The full team.

Figure 2: Tunnel Traversal

Test #8- Travel To and Over the Bridge Test

Test Number: ​#8

Date: ​09/05/2018
Tester: ​Sarah Swanson

19
Testing Document Team 18

Author: ​Sarah Swanson

Software Description: ​Version 1.7 (Refer to Software Document)
Hardware Description: ​Version 1.5 (Refer to Hardware Document)

Goal:
The purpose of this test is to determine if the robot can successfully travel to and over
the bridge.

Procedure:
1) Turn on EV3 and make sure it has at least 7.0 battery or above.
2) Program the robot to travel to waypoint where the bridge is located. In this case we will
test with the coordinates (3,5).
3) Run the robot to travel to and cross the bridge.
4) Record if the robot successfully navigates and passes over the bridge.

Expected Result:
Since the robot’s width is too small to travel over the bridge and the robot has pieces that
may get stuck on the bridge’s edge, the outcome should be that it can not pass over the bridge
unless the hardware is changed.

Test Report:

Successful Navigation Successful Bridge Traversal

Run 1 Yes- perfect positioning No- the robot could not pass
past the first speed bump

Run 2 Yes- perfect positioning No- the robot could not pass
past the first speed bump

Run 3 Yes- perfect positioning No- the robot could not pass
past the first speed bump

Run 4 Yes- a little off because of No- the robot could not pass
wheel slip past the first speed bump

Run 5 Yes- perfect positioning No- the robot could not pass
past the first speed bump

Conclusion:
The hardware of the robot prevents the robot to travel over the bridge. The rolling ball in
the bottom gets stuck on the bridge edge.

Action:

20
Testing Document Team 18

The rolling ball on the hardware of the robot is essential for the localization of the robot,
therefore the decision is to leave the hardware alone and just be able to travel through the
tunnel. The lack of time determined this decision.

Distribution:
The full team.

Figure 3: Bridge Traversal - Unable to Complete

Test #9- Searching for Block Test

Test Number: ​#9

Date: ​08/05/2018

21
Testing Document Team 18

Conclusion:
This portion of the course was not completed because of a time issue. Since the
hardware design was changed right after the beta demo, there was less time to complete every
aspect of the competition course. This was a decision we wanted to take the risk on because
the robot with the changing wheelbase was not stable. The robot could localize but not navigate
well. The new robot design (refer to section 7.0 of the Hardware Document) is able to
successfully localize, navigate, and travel through the tunnel. Refer to tests 4,5,7 for details on
the successes.

6.0 Final Demonstration Report

The expected result for the demo was for the robot to localize properly and navigate to
the obstacle. The robot was only able to travel through the tunnel, therefore if it was sent to a
tunnel, it would navigate through and be sent straight to the final position. Due to lack of time,
the team decided to skip the search area. If the robot was sent to the bridge, the expected result
was for the robot to try to cross but the rolling ball on the bottom would get stuck on the bridge
entrance as shown in Test #8 in section 5.3 of this document.

The final demo competition results are listed below:

Run 1:​ The robot was left on overnight by human error, therefore the localization and navigation
were not correct. The robot still made it to the tunnel but at an angle and ran into the tunnel.

Run 2:​ The robot was now fully charged. The robot localized properly and navigated to the
bridge properly. Since the results in Test #8 show that the robot could not cross the bridge, this
was the result in the actual demo.

Run 3:​ The robot was fully charged. While localizing, the light sensor missed a black line. This
is rare but still possible as shown in Test #2 of section 5.1 of this document that the sensor can
detect or miss a black line. Since the localization was unsuccessful, the robot navigated into the
wall and the robot was removed from the competition field.

Run 4: ​The robot was fully charged, but we came across another human error. Before the
demo, the motors were checked by rotating them. When the robot was ready to demo, the robot
corrected itself for the movement in the motors. This caused the robot to move backwards very
quickly and the robot was removed from the competition field.

22