qt-testrunner: be more strict if bad XML log files are written

If a test returns 0 but writes an XML logfile that contains FAIL or a
corrupted XML file, then qt-testrunner considers it a CRASH and exits
with 3.

Previously any test execution returning 0 (success) was considered
a PASS. Changing this behavior with this patch finds a lot of test
crashes on Android (QTBUG-100470), because the tests are run indirectly
on the emulator and the test wrapper process could not detect the crash,
thus returning 0 to qt-testrunner. But the corrupt XML file is caught
now.

Likewise, if a test returns != 0 but the XML logfile contains only PASS,
qt-testrunner considers it a FAIL. This used to be the case but now
tests are added.

Finally changed logging for such cases from INFO to WARNING.

Task-number: QTBUG-100470
Change-Id: I404c9d2211c7de027bf776d1914519d37f513ca1
Reviewed-by: Assam Boudjelthia <[email protected]>

1 files changed, 9 insertions, 5 deletions

diff --git a/util/testrunner/qt-testrunner.py b/util/testrunner/qt-testrunner.py
index 8a9b4699778..4dcde830aeb 100755
--- a/util/testrunner/qt-testrunner.py
+++ b/util/testrunner/qt-testrunner.py
@@ -344,21 +344,25 @@ def main():
                     run_full_test(args.test_basename, args.testargs, args.log_dir,
                                   args.no_extra_args, args.dry_run, args.timeout,
                                   args.specific_extra_args)
-                if retcode != 0 and results_file:
+                if results_file:
                     failed_functions = parse_log(results_file)
 
             if retcode == 0:
+                if failed_functions:
+                    L.warning("The test executable returned success but the logfile"
+                             f" contains FAIL for function: {failed_functions[0].func}")
+                    continue
                 sys.exit(0)    # PASS
 
             if len(failed_functions) == 0:
-                L.info("No failures listed in the XML test log!"
-                       " Did the test CRASH right after all its testcases PASSed?")
+                L.warning("No failures listed in the XML test log!"
+                          " Did the test CRASH right after all its testcases PASSed?")
                 continue
 
             cant_rerun = [ f.func for f in failed_functions if f.func in NO_RERUN_FUNCTIONS ]
             if cant_rerun:
-                L.info(f"Failure detected in the special test function '{cant_rerun[0]}'"
-                       " which can not be re-run individually")
+                L.warning(f"Failure detected in the special test function '{cant_rerun[0]}'"
+                          " which can not be re-run individually")
                 continue
 
             assert len(failed_functions) > 0  and  retcode != 0