summaryrefslogtreecommitdiffstats
path: root/util/testrunner/qt-testrunner.py
Commit message (Collapse)AuthorAgeFilesLines
* qt-testrunner: exit(1) instead of exit(3) when asserts triggerDimitrios Apostolou47 hours1-1/+7
| | | | | | | | | | | | Asserts triggering signify programming errors in the testrunner. For the same reason, change also a pre-existing AssertionError exception raised with bad XML input file to a dedicated exception. Pick-to: 6.11 6.10 6.8 Change-Id: I5e5cc746df60075e02964efd97d1a37dc401925f Reviewed-by: Axel Spoerl <[email protected]> Reviewed-by: Frederic Lefebvre <[email protected]>
* qt-testrunner: check validity of the XML file on test function re-runsDimitrios Apostolou47 hours1-2/+13
| | | | | | | | | | | | | | | | | | | | | Tests on certain platforms run with specific wrappers, that might have trouble reporting back the proper exit code. For example, on Android each test is transfered to the device/emulator and is executed not as a separate process, but as an activity. The equivalent of "exit code" is caught and returned by a special wrapper script for the platform. It happens sometimes that these wrapper scripts fail to report back correctly, and report zero (0) despite failed tests. For that reason we now parse the test XML log on individual test re-runs too, and reporting inconsistencies as CRASH, like we do with the main test execution. Task-number: QTQAINFRA-7349 Task-number: QTQAINFRA-7378 Pick-to: 6.11 6.10 6.8 Change-Id: I27525f22331d44141be8825786a6f71e89543e92 Reviewed-by: Edward Welbourne <[email protected]>
* Test qt-testrunner as the first test that CTest runsDimitrios Apostolou47 hours1-3/+2
| | | | | | | | | | | | | Only for inside the Coin CI, as it is Coin that uses qt-testrunner by exporting TESTRUNNER=qt-testrunner. Also rename the test executable from tst_testrunner to tst_qt_testrunner to avoid confusion with other testrunners, or generic $TESTRUNNER testing. Pick-to: 6.11 6.10 6.8 Change-Id: I607f8c2affec2ca5dd38b4a333abb3a324d2078c Reviewed-by: Alexandru Croitor <[email protected]>
* qt_testrunner: fix execution of qt_mock_test.py under testing on windowsDimitrios Apostolou47 hours1-0/+6
| | | | | | | | | | | | | | | | | | tst_qt_testrunner.py invokes qt_testrunner.py which invokes qt_mock_test.py. On Windows a python script is not directly executable. It needs to be executed as "python.exe script.py". To achieve that when testing, the tests set the env var QT_TESTRUNNER_TESTING. When qt-testrunner detects that, it prepends the command line with sys.executable which is the currently used python interpreter. Doing so also ensures that all sub-scripts are executed with the same Python version. Pick-to: 6.11 6.10 6.8 Change-Id: I4786e4015ded328077fd9cc7b443f4b4d9d69de7 Reviewed-by: Mårten Nordheim <[email protected]> Reviewed-by: Edward Welbourne <[email protected]>
* qt-testrunner: if a failed test re-run crashes, then exit(3) immediatelyDimitrios Apostolou2 days1-4/+9
| | | | | | | | | Instead of continuing the re-runs of the failed function multiple times, quit immediately with exit code 3 which signifies crash. Pick-to: 6.11 6.10 6.8 Change-Id: I9997d6fefe946f719019dad85852609528d9a30e Reviewed-by: Edward Welbourne <[email protected]>
* qt-testrunner: comment updatesDimitrios Apostolou2 days1-4/+11
| | | | | | Pick-to: 6.11 6.10 6.8 Change-Id: Ia72fa89b47d99a90d2c669f5b889ee8fba461cf2 Reviewed-by: Edward Welbourne <[email protected]>
* qt-testrunner: consider exit codes >=128 as CRASHDimitrios Apostolou2 days1-11/+29
| | | | | | | | | | | | | | | | | | | Previously it was only < 0, which was valid because Python set those on POSIX platforms to represent signal number. By checking >=128 we also accommodate for crashes on windows and other unpredictable situations (since QTest is limiting itself to [0,127]). Also change the exit code of the mock test used for qt_testrunner testing. 130 is what is shown by bash when a process is interrupted by Ctrl-C (SIGINT), while 131 is what is shown for SIGQUIT and does not happen that easily. Completely insignificant change for the python programs like qt-testrunner and its tests, since subprocess.run() returncode is negative in case of a signal. Only visible if qt_mock_test is invoked in the shell for debugging purposes. Pick-to: 6.11 6.10 6.8 Change-Id: I4759bc0e74ca2fb1d1594c71e541feb5363bb7ea Reviewed-by: Mårten Nordheim <[email protected]>
* qt-testrunner: if a test logs a qFatal message, then treat it as crashedDimitrios Apostolou2 days1-19/+48
| | | | | | | | | | | | | | | | | | QTest can, depending on the platform, clean-up properly after a qFatal. In that case the test <Incident> is marked simply as FAIL, no further tests are run, and the XML logfile is closed properly. The only way to figure out that something went bad and the testrun was aborted, is the qFatal message. This is needed primarily for qt-testrunner to *not* consider re-running individual failed test functions/datatags, to avoid missing the ones that did not start at all. Fixes: QTQAINFRA-7349 Fixes: QTQAINFRA-7378 Pick-to: 6.11 6.10 6.8 Change-Id: I77276f37e0e6668048c78563560af0e15101be0e Reviewed-by: Edward Welbourne <[email protected]>
* qt-testrunner: detect test filename for androidtestrunner --aabDimitrios Apostolou2025-10-061-3/+21
| | | | | | | | | | | | | | | | | Tests on android are wrapped by androidtestrunner. Previously the test was passed to it with --apk testname.apk. A new way was recently added: --aab testname.aab. qt-testrunner now supports these two ways to detect the filename. Additionally, it now exits with error if the testname is undetected. This way we'll avoid all tests being recorded as "androidtestrunner" in the future. Fixes: QTQAINFRA-7351 Pick-to: 6.10 6.9 6.8 Change-Id: I571b617b28cdb4ea51d0b61e629d9bc8ec8e1917 Reviewed-by: Edward Welbourne <[email protected]> Reviewed-by: Assam Boudjelthia <[email protected]>
* qt-testrunner: re-run the full test if it crashedDimitrios Apostolou2024-09-191-0/+3
| | | | | | | | | | | Previously it re-ran only the failed tests, if a proper XML logfile was written. But in case of "crash" caused by the watchdog for QTEST_FUNCTION_TIMEOUT, the subsequent tests were not included in the XML logfile, thus those were never run. Fixes: QTQAINFRA-5226 Change-Id: Ib4f0849fa2511bb34365fd901fd53c5a3e3ab293 Reviewed-by: Edward Welbourne <[email protected]>
* qt-testrunner: Do not print warning when a logfile is not expectedDimitrios Apostolou2024-09-191-2/+4
| | | | | | | | | Do not print warning about no errors in logfile, when a logfile is not expected. Task-number: QTQAINFRA-5084 Change-Id: I92f94452418738d31936d47362aa6090090af6de Reviewed-by: Edward Welbourne <[email protected]>
* Inform qt-testrunner about coin_vxworks_qemu_runner.shDimitrios Apostolou2024-09-191-6/+10
| | | | | | | | | Since we now have quite a few "special" test wrappers, I took the opportunity to refactor the code and add a couple of testcases too. Change-Id: I20e1214351d71c1474be32f03d4218ae6bdd2277 Reviewed-by: Axel Spoerl <[email protected]> Reviewed-by: Toni Saario <[email protected]>
* qt-testrunner: Save test output into fileAlexandru Croitor2022-12-081-6/+13
| | | | | | | | | | | | | Aside from the generation of the xml and junit xml files for each test, also save the test output into a tst_{name}-{timestamp}.txt file. This will allow developers to inspect the output of passed tests, by downloading the test results archive, even if we use ctest's --output-on-failure. Change-Id: I0a6c0ee04b4525d3ad9b207b28117d2182d29c28 Reviewed-by: Tor Arne Vestbø <[email protected]>
* testrunner: Use absolute path for JUnit XML result fileTor Arne Vestbø2022-09-131-5/+6
| | | | | | | | Amends 210020412c1d70b70993dfdec5041d486b28a8c4 Change-Id: Ibe3bbe0bab4370936f82a915ba59a7a89c292ea0 Reviewed-by: Qt CI Bot <[email protected]> Reviewed-by: Dimitrios Apostolou <[email protected]>
* testrunner: Generate JUnit result filesTor Arne Vestbø2022-09-091-3/+7
| | | | | | | | The JUnit format is easier to feed to tools that don't support the native Qt Test XML format. Change-Id: Ie9803cc0fb0577b3b7258b05faa78d8fb1aad1d1 Reviewed-by: Dimitrios Apostolou <[email protected]>
* Add an exception in testrunner for QDocLuca Di Sera2022-05-191-1/+3
| | | | | | | | | | | | | | | QDoc recently added some tests that use Catch2 instead of testlib. Similarly to selftests, the normal behavior of `qt-testrunner.py` will fail, trying to pass an argument that is not supported by Catch2 when running the tests in CI. To avoid the issue, an exception for the test executables that QDoc uses was added to `qt-testrunner.py`, using the already available mechanism that is used for selftests. Change-Id: I5eb4b75f239eda0c493fb06ece164545d159372b Reviewed-by: Topi Reiniö <[email protected]> Reviewed-by: Dimitrios Apostolou <[email protected]>
* Use SPDX license identifiersLucie Gérard2022-05-161-29/+2
| | | | | | | | | | | | | Replace the current license disclaimer in files by a SPDX-License-Identifier. Files that have to be modified by hand are modified. License files are organized under LICENSES directory. Task-number: QTBUG-67283 Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1 Reviewed-by: Qt CI Bot <[email protected]> Reviewed-by: Lars Knoll <[email protected]> Reviewed-by: Jörg Bornemann <[email protected]>
* Generalize exclusion listDimitrios Apostolou2022-04-221-4/+8
| | | | | | | | | | It seems there are some tests in the various Qt modules that skip writing the XML testlog file. If there are good reasons for that (for example test is a custom shellscript) then it should be added to the DUMB_TESTS list. Change-Id: Iffb00e1fda42bb8e7338a94c9320e45900db1bc1 Reviewed-by: Daniel Smith <[email protected]>
* qt-testrunner: improve error loggingDimitrios Apostolou2022-04-221-9/+12
| | | | | | | | Do not print full stack traces when a controlled exception occurs, instead print just the exception name and the message. Change-Id: I5669862d2e98c550d88d1d11fb2bf49f197cfc17 Reviewed-by: Alexandru Croitor <[email protected]>
* qt-testrunner: be more strict if bad XML log files are writtenDimitrios Apostolou2022-04-051-5/+9
| | | | | | | | | | | | | | | | | | | | | | | If a test returns 0 but writes an XML logfile that contains FAIL or a corrupted XML file, then qt-testrunner considers it a CRASH and exits with 3. Previously any test execution returning 0 (success) was considered a PASS. Changing this behavior with this patch finds a lot of test crashes on Android (QTBUG-100470), because the tests are run indirectly on the emulator and the test wrapper process could not detect the crash, thus returning 0 to qt-testrunner. But the corrupt XML file is caught now. Likewise, if a test returns != 0 but the XML logfile contains only PASS, qt-testrunner considers it a FAIL. This used to be the case but now tests are added. Finally changed logging for such cases from INFO to WARNING. Task-number: QTBUG-100470 Change-Id: I404c9d2211c7de027bf776d1914519d37f513ca1 Reviewed-by: Assam Boudjelthia <[email protected]>
* Clean up QNX platform configs and improve testrunner scriptsDimitrios Apostolou2022-02-141-0/+7
| | | | | | | | | | | | | | | | | | | | QNX requires its own TESTRUNNER script in order to SSH to the QEMU guest and run the tests. Simplify it a lot and fix the way it passes arguments so that they retain spaces and quotation. Also wrap it with the generic TESTRUNNER script for CI, which is qt-testrunner.py, so that tests can re-run in case of flakiness. Delete prefix.sh as the environment variables are now properly set in the Coin platform config files. Avoid executing extra chmod commands to make scripts executable, but use the right Coin instruction for that. For reference, we use 493 for file mode which equals to 755 in octal, as Coin expects the file mode in decimal. Change-Id: Ife4d1caef606f48b92ba1da1cfb14ec0dea11ef2 Reviewed-by: Pasi Petäjäjärvi <[email protected]>
* Write XML test log files even for the flaky test re-runsDimitrios Apostolou2022-01-301-6/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously the qt-testrunner script was avoiding to write XML log files for the flaky test re-runs, since parsing them always provides limited information about what went wrong (for example unparseable files) and might miss issues completely (for example a crashed test might not even write a log file), or even mislead if multiple different data points about a single testcase end up into the database.. But currently Coin is the system that uploads test information to the testresults database and expects to find such log files in an attempt to send as much information as possible regarding failures. Re-enabling these log files will allow Coin to deduce as much information as possible about the test runs. This is a temporary step at least, until the test-uploading logic is separated from Coin. Then it will be easier to find the full information about what happened with flaky or crashed tests by special logs from qt-testrunner.py, which is the test re-running orchestrator and therefore has more knowledge about what went wrong. Fixes: QTQAINFRA-4754 Change-Id: I69e4469cbe6c0a6ca5b2473eaf8d034632564342 Reviewed-by: Fabian Kosmale <[email protected]> Reviewed-by: Toni Saario <[email protected]> Reviewed-by: Simo Fält <[email protected]>
* qt-testrunner: do not try to re-run initTestCase and othersDimitrios Apostolou2022-01-301-22/+36
| | | | | | | | | | | | | Some function names are special for qtestlib, in the sense that they can not be specified as a command line argument to run individually. In such cases qt-testrunner treats the failure specially and tries once to re-run the full test executable. Fixes: QTBUG-89011 Change-Id: I0cc25f91c57374e5ac65ade10e2e223fe969f211 Reviewed-by: Volker Hilsheimer <[email protected]> Reviewed-by: Daniel Smith <[email protected]>
* qt-testrunner: be more clear that it was the test that crashedDimitrios Apostolou2022-01-181-4/+9
| | | | | | | Task-number: QTBUG-99970 Change-Id: Id2d01b6ab7d428356d9dfc953107014791393d35 Reviewed-by: Fabian Kosmale <[email protected]> Reviewed-by: Jani Heikkinen <[email protected]>
* qt-testrunner: disable JUnit XML test logsDimitrios Apostolou2022-01-121-3/+1
| | | | | | | | It causes crashes because of interleaved execution of Quick tests. Task-number: QTBUG-98350 Change-Id: I61e26f5da5617e06810db8eb7991f17eecdd3acb Reviewed-by: Tor Arne Vestbø <[email protected]>
* Enable verbose logging in the last re-run of a failing testDimitrios Apostolou2022-01-031-8/+19
| | | | | | Fixes: COIN-728 Change-Id: I08802e377e26e5dd7f7d1c44b5efe4280b09f957 Reviewed-by: Daniel Smith <[email protected]>
* Introduce qt-testrunner.pyDimitrios Apostolou2022-01-031-0/+348
Script that wraps Qt test execution in order to iron-out flakiness. Task-number: QTBUG-96353 Change-Id: Ie8ee780e3f4f8d086c080a7784f9f68fd1867be2 Reviewed-by: Daniel Smith <[email protected]>
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-13 21:43:10 +0000