summaryrefslogtreecommitdiffstats
path: root/util/testrunner
Commit message (Collapse)AuthorAgeFilesLines
* qt-testrunner: exit(1) instead of exit(3) when asserts triggerDimitrios Apostolou2 days1-1/+7
| | | | | | | | | | | | Asserts triggering signify programming errors in the testrunner. For the same reason, change also a pre-existing AssertionError exception raised with bad XML input file to a dedicated exception. Pick-to: 6.11 6.10 6.8 Change-Id: I5e5cc746df60075e02964efd97d1a37dc401925f Reviewed-by: Axel Spoerl <[email protected]> Reviewed-by: Frederic Lefebvre <[email protected]>
* qt-testrunner: check validity of the XML file on test function re-runsDimitrios Apostolou2 days1-2/+13
| | | | | | | | | | | | | | | | | | | | | Tests on certain platforms run with specific wrappers, that might have trouble reporting back the proper exit code. For example, on Android each test is transfered to the device/emulator and is executed not as a separate process, but as an activity. The equivalent of "exit code" is caught and returned by a special wrapper script for the platform. It happens sometimes that these wrapper scripts fail to report back correctly, and report zero (0) despite failed tests. For that reason we now parse the test XML log on individual test re-runs too, and reporting inconsistencies as CRASH, like we do with the main test execution. Task-number: QTQAINFRA-7349 Task-number: QTQAINFRA-7378 Pick-to: 6.11 6.10 6.8 Change-Id: I27525f22331d44141be8825786a6f71e89543e92 Reviewed-by: Edward Welbourne <[email protected]>
* Test qt-testrunner as the first test that CTest runsDimitrios Apostolou2 days5-844/+3
| | | | | | | | | | | | | Only for inside the Coin CI, as it is Coin that uses qt-testrunner by exporting TESTRUNNER=qt-testrunner. Also rename the test executable from tst_testrunner to tst_qt_testrunner to avoid confusion with other testrunners, or generic $TESTRUNNER testing. Pick-to: 6.11 6.10 6.8 Change-Id: I607f8c2affec2ca5dd38b4a333abb3a324d2078c Reviewed-by: Alexandru Croitor <[email protected]>
* qt_testrunner: fix execution of qt_mock_test.py under testing on windowsDimitrios Apostolou2 days2-0/+7
| | | | | | | | | | | | | | | | | | tst_qt_testrunner.py invokes qt_testrunner.py which invokes qt_mock_test.py. On Windows a python script is not directly executable. It needs to be executed as "python.exe script.py". To achieve that when testing, the tests set the env var QT_TESTRUNNER_TESTING. When qt-testrunner detects that, it prepends the command line with sys.executable which is the currently used python interpreter. Doing so also ensures that all sub-scripts are executed with the same Python version. Pick-to: 6.11 6.10 6.8 Change-Id: I4786e4015ded328077fd9cc7b443f4b4d9d69de7 Reviewed-by: Mårten Nordheim <[email protected]> Reviewed-by: Edward Welbourne <[email protected]>
* tst_qt_testrunner: make it runnable on windowsDimitrios Apostolou2 days1-7/+10
| | | | | | | | | | | | | | | | | | | By making sure it invokes other python executables using the same python interpreter. Also on the tests that write to an XML file, close the file handle that mkstemp() generates, otherwise Windows refuse to open and write to the filename. Side change: + For some tests, include all the environment instead of selecting specific env vars, in order to reduce verbosity. Pick-to: 6.11 6.10 6.8 Change-Id: If4ba68b5f9bf64824c888b99e9525d4bdcf42037 Reviewed-by: Mårten Nordheim <[email protected]> Reviewed-by: Edward Welbourne <[email protected]>
* tst_qt_testrunner:Avoid parameter root_dir in glob()Dimitrios Apostolou2 days1-27/+20
| | | | | | | | because it's only available since Python 3.10. Pick-to: 6.11 6.10 6.8 Change-Id: I81170b9cbbd7819fbfe3a13c8caa33e21dcae9b3 Reviewed-by: Edward Welbourne <[email protected]>
* tst_testrunner.py: small fix to make it run with python<3.9Dimitrios Apostolou2 days1-1/+1
| | | | | | Pick-to: 6.11 6.10 6.8 Change-Id: I382e8c8b766c0c36e484ecb16e74a0010ea4fe32 Reviewed-by: Edward Welbourne <[email protected]>
* Test qt-testrunner's functionality of repeating a non-QTest testDimitrios Apostolou2 days1-1/+25
| | | | | | | | | | In that case, it re-runs the whole executable once, since the test does not output a QTest XML log and does not accept selective running through QTest command-line arguments. Pick-to: 6.11 6.10 6.8 Change-Id: I35b255aff10a0f2a79851889c8daeca277c605ed Reviewed-by: Edward Welbourne <[email protected]>
* tst_testrunner: split up a long line to stop gerrit from complainingDimitrios Apostolou3 days1-4/+10
| | | | | | | | | | | The line had been copied verbatim from the CI logs, but gerrit refuses to do any syntax highlighting in the file. So break it down. Also remove some old commented-out code. Pick-to: 6.11 6.10 6.8 Change-Id: Ibde01161edb099eb8b1c9d7d6357a7a8b36e6819 Reviewed-by: Edward Welbourne <[email protected]>
* qt-testrunner: if a failed test re-run crashes, then exit(3) immediatelyDimitrios Apostolou3 days4-14/+46
| | | | | | | | | Instead of continuing the re-runs of the failed function multiple times, quit immediately with exit code 3 which signifies crash. Pick-to: 6.11 6.10 6.8 Change-Id: I9997d6fefe946f719019dad85852609528d9a30e Reviewed-by: Edward Welbourne <[email protected]>
* qt-testrunner: comment updatesDimitrios Apostolou3 days2-4/+21
| | | | | | Pick-to: 6.11 6.10 6.8 Change-Id: Ia72fa89b47d99a90d2c669f5b889ee8fba461cf2 Reviewed-by: Edward Welbourne <[email protected]>
* qt-testrunner: add testcase that verifies a crashed test re-runs fullyDimitrios Apostolou3 days2-3/+46
| | | | | | | | | | | | | Even if it has written a proper, non-corrupt XML file, with no trace of crash. Fixed also a small typo in a comment because it referred to the wrong environment variable. (changed QT_MOCK_FAIL_LIST to QT_MOCK_RUN_LIST). Task-number: QTQAINFRA-5226 Pick-to: 6.11 6.10 6.8 Change-Id: I2e0c443edcaeba053927f4e3fbb325a13f34d238 Reviewed-by: Mårten Nordheim <[email protected]>
* qt-testrunner: consider exit codes >=128 as CRASHDimitrios Apostolou3 days2-13/+31
| | | | | | | | | | | | | | | | | | | Previously it was only < 0, which was valid because Python set those on POSIX platforms to represent signal number. By checking >=128 we also accommodate for crashes on windows and other unpredictable situations (since QTest is limiting itself to [0,127]). Also change the exit code of the mock test used for qt_testrunner testing. 130 is what is shown by bash when a process is interrupted by Ctrl-C (SIGINT), while 131 is what is shown for SIGQUIT and does not happen that easily. Completely insignificant change for the python programs like qt-testrunner and its tests, since subprocess.run() returncode is negative in case of a signal. Only visible if qt_mock_test is invoked in the shell for debugging purposes. Pick-to: 6.11 6.10 6.8 Change-Id: I4759bc0e74ca2fb1d1594c71e541feb5363bb7ea Reviewed-by: Mårten Nordheim <[email protected]>
* qt-testrunner: skip some of the tests on non-POSIX platformsDimitrios Apostolou3 days1-0/+4
| | | | | | Pick-to: 6.11 6.10 6.8 Change-Id: Ib48d1a2e98423218097405fe679c6d68d0e208c7 Reviewed-by: Edward Welbourne <[email protected]>
* qt-testrunner: fix two pre-existing testsDimitrios Apostolou3 days1-0/+2
| | | | | | | | | | | Previously the tests were not testing what they were supposed to. Instead of an empty file, qt-testrunner was finding no file. By using the newly introduced hidden knob, qt-testrunner will read the XML file that the test provides. Pick-to: 6.11 6.10 6.8 Change-Id: I8bd356f4f770a478aabaaee8eb449162abed18f1 Reviewed-by: Edward Welbourne <[email protected]>
* qt-testrunner: if a test logs a qFatal message, then treat it as crashedDimitrios Apostolou3 days3-22/+153
| | | | | | | | | | | | | | | | | | QTest can, depending on the platform, clean-up properly after a qFatal. In that case the test <Incident> is marked simply as FAIL, no further tests are run, and the XML logfile is closed properly. The only way to figure out that something went bad and the testrun was aborted, is the qFatal message. This is needed primarily for qt-testrunner to *not* consider re-running individual failed test functions/datatags, to avoid missing the ones that did not start at all. Fixes: QTQAINFRA-7349 Fixes: QTQAINFRA-7378 Pick-to: 6.11 6.10 6.8 Change-Id: I77276f37e0e6668048c78563560af0e15101be0e Reviewed-by: Edward Welbourne <[email protected]>
* qt-testrunner: detect test filename for androidtestrunner --aabDimitrios Apostolou2025-10-062-16/+99
| | | | | | | | | | | | | | | | | Tests on android are wrapped by androidtestrunner. Previously the test was passed to it with --apk testname.apk. A new way was recently added: --aab testname.aab. qt-testrunner now supports these two ways to detect the filename. Additionally, it now exits with error if the testname is undetected. This way we'll avoid all tests being recorded as "androidtestrunner" in the future. Fixes: QTQAINFRA-7351 Pick-to: 6.10 6.9 6.8 Change-Id: I571b617b28cdb4ea51d0b61e629d9bc8ec8e1917 Reviewed-by: Edward Welbourne <[email protected]> Reviewed-by: Assam Boudjelthia <[email protected]>
* qt-testrunner: re-run the full test if it crashedDimitrios Apostolou2024-09-191-0/+3
| | | | | | | | | | | Previously it re-ran only the failed tests, if a proper XML logfile was written. But in case of "crash" caused by the watchdog for QTEST_FUNCTION_TIMEOUT, the subsequent tests were not included in the XML logfile, thus those were never run. Fixes: QTQAINFRA-5226 Change-Id: Ib4f0849fa2511bb34365fd901fd53c5a3e3ab293 Reviewed-by: Edward Welbourne <[email protected]>
* qt-testrunner: Do not print warning when a logfile is not expectedDimitrios Apostolou2024-09-191-2/+4
| | | | | | | | | Do not print warning about no errors in logfile, when a logfile is not expected. Task-number: QTQAINFRA-5084 Change-Id: I92f94452418738d31936d47362aa6090090af6de Reviewed-by: Edward Welbourne <[email protected]>
* Inform qt-testrunner about coin_vxworks_qemu_runner.shDimitrios Apostolou2024-09-192-11/+48
| | | | | | | | | Since we now have quite a few "special" test wrappers, I took the opportunity to refactor the code and add a couple of testcases too. Change-Id: I20e1214351d71c1474be32f03d4218ae6bdd2277 Reviewed-by: Axel Spoerl <[email protected]> Reviewed-by: Toni Saario <[email protected]>
* sanitizer-testrunner.py: be verbose about FAILs being ignoredDimitrios Apostolou2023-10-181-0/+2
| | | | | | | Pick-to: 6.5 6.6 Change-Id: I8753b8793c744d4fe2cae8d7293abf6961c60601 Reviewed-by: Qt CI Bot <[email protected]> Reviewed-by: Tor Arne Vestbø <[email protected]>
* qt-testrunner: Save test output into fileAlexandru Croitor2022-12-081-6/+13
| | | | | | | | | | | | | Aside from the generation of the xml and junit xml files for each test, also save the test output into a tst_{name}-{timestamp}.txt file. This will allow developers to inspect the output of passed tests, by downloading the test results archive, even if we use ctest's --output-on-failure. Change-Id: I0a6c0ee04b4525d3ad9b207b28117d2182d29c28 Reviewed-by: Tor Arne Vestbø <[email protected]>
* testrunner: Use absolute path for JUnit XML result fileTor Arne Vestbø2022-09-131-5/+6
| | | | | | | | Amends 210020412c1d70b70993dfdec5041d486b28a8c4 Change-Id: Ibe3bbe0bab4370936f82a915ba59a7a89c292ea0 Reviewed-by: Qt CI Bot <[email protected]> Reviewed-by: Dimitrios Apostolou <[email protected]>
* testrunner: Generate JUnit result filesTor Arne Vestbø2022-09-091-3/+7
| | | | | | | | The JUnit format is easier to feed to tools that don't support the native Qt Test XML format. Change-Id: Ie9803cc0fb0577b3b7258b05faa78d8fb1aad1d1 Reviewed-by: Dimitrios Apostolou <[email protected]>
* sanitizer-testrunner: enforce line bufferingDimitrios Apostolou2022-06-161-1/+1
| | | | | | | | Under coin-agent, the stdout/err are not a tty, so flushing needs to be forced. Change-Id: I06de43328a4f4d1c17df7188f31b5f7bc63e3335 Reviewed-by: Alexandru Croitor <[email protected]>
* qt-testrunner: Commit forgotten README fileDimitrios Apostolou2022-06-161-0/+24
| | | | | | Pick-to: 6.4 6.3 6.2 Change-Id: Ie307b95947a6910bd65bb22ecf938fae9de22f9d Reviewed-by: Edward Welbourne <[email protected]>
* Add ASAN build for qtbaseDimitrios Apostolou2022-06-151-0/+49
| | | | | | | | | | | | | | | The test run is wrapped with a special TESTRUNNER script that ignores failing tests (there are several tests failing when built with ASAN) and also ignores LSAN errors (memory leaks - but still visible in the output). The test run only fails if a test reports ASAN errors or if it crashes (or times out, which is like a crash caused by qtestlib's watchdog timer). Fixes: QTQAINFRA-5025 Change-Id: I861756ab49388ac4a52409d3a780684244e469b1 Reviewed-by: Alexandru Croitor <[email protected]>
* Add an exception in testrunner for QDocLuca Di Sera2022-05-191-1/+3
| | | | | | | | | | | | | | | QDoc recently added some tests that use Catch2 instead of testlib. Similarly to selftests, the normal behavior of `qt-testrunner.py` will fail, trying to pass an argument that is not supported by Catch2 when running the tests in CI. To avoid the issue, an exception for the test executables that QDoc uses was added to `qt-testrunner.py`, using the already available mechanism that is used for selftests. Change-Id: I5eb4b75f239eda0c493fb06ece164545d159372b Reviewed-by: Topi Reiniö <[email protected]> Reviewed-by: Dimitrios Apostolou <[email protected]>
* Use SPDX license identifiersLucie Gérard2022-05-163-87/+6
| | | | | | | | | | | | | Replace the current license disclaimer in files by a SPDX-License-Identifier. Files that have to be modified by hand are modified. License files are organized under LICENSES directory. Task-number: QTBUG-67283 Change-Id: Id880c92784c40f3bbde861c0d93f58151c18b9f1 Reviewed-by: Qt CI Bot <[email protected]> Reviewed-by: Lars Knoll <[email protected]> Reviewed-by: Jörg Bornemann <[email protected]>
* Generalize exclusion listDimitrios Apostolou2022-04-221-4/+8
| | | | | | | | | | It seems there are some tests in the various Qt modules that skip writing the XML testlog file. If there are good reasons for that (for example test is a custom shellscript) then it should be added to the DUMB_TESTS list. Change-Id: Iffb00e1fda42bb8e7338a94c9320e45900db1bc1 Reviewed-by: Daniel Smith <[email protected]>
* qt-testrunner: improve error loggingDimitrios Apostolou2022-04-221-9/+12
| | | | | | | | Do not print full stack traces when a controlled exception occurs, instead print just the exception name and the message. Change-Id: I5669862d2e98c550d88d1d11fb2bf49f197cfc17 Reviewed-by: Alexandru Croitor <[email protected]>
* qt-testrunner: be more strict if bad XML log files are writtenDimitrios Apostolou2022-04-052-11/+34
| | | | | | | | | | | | | | | | | | | | | | | If a test returns 0 but writes an XML logfile that contains FAIL or a corrupted XML file, then qt-testrunner considers it a CRASH and exits with 3. Previously any test execution returning 0 (success) was considered a PASS. Changing this behavior with this patch finds a lot of test crashes on Android (QTBUG-100470), because the tests are run indirectly on the emulator and the test wrapper process could not detect the crash, thus returning 0 to qt-testrunner. But the corrupt XML file is caught now. Likewise, if a test returns != 0 but the XML logfile contains only PASS, qt-testrunner considers it a FAIL. This used to be the case but now tests are added. Finally changed logging for such cases from INFO to WARNING. Task-number: QTBUG-100470 Change-Id: I404c9d2211c7de027bf776d1914519d37f513ca1 Reviewed-by: Assam Boudjelthia <[email protected]>
* Clean up QNX platform configs and improve testrunner scriptsDimitrios Apostolou2022-02-141-0/+7
| | | | | | | | | | | | | | | | | | | | QNX requires its own TESTRUNNER script in order to SSH to the QEMU guest and run the tests. Simplify it a lot and fix the way it passes arguments so that they retain spaces and quotation. Also wrap it with the generic TESTRUNNER script for CI, which is qt-testrunner.py, so that tests can re-run in case of flakiness. Delete prefix.sh as the environment variables are now properly set in the Coin platform config files. Avoid executing extra chmod commands to make scripts executable, but use the right Coin instruction for that. For reference, we use 493 for file mode which equals to 755 in octal, as Coin expects the file mode in decimal. Change-Id: Ife4d1caef606f48b92ba1da1cfb14ec0dea11ef2 Reviewed-by: Pasi Petäjäjärvi <[email protected]>
* Write XML test log files even for the flaky test re-runsDimitrios Apostolou2022-01-301-6/+20
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously the qt-testrunner script was avoiding to write XML log files for the flaky test re-runs, since parsing them always provides limited information about what went wrong (for example unparseable files) and might miss issues completely (for example a crashed test might not even write a log file), or even mislead if multiple different data points about a single testcase end up into the database.. But currently Coin is the system that uploads test information to the testresults database and expects to find such log files in an attempt to send as much information as possible regarding failures. Re-enabling these log files will allow Coin to deduce as much information as possible about the test runs. This is a temporary step at least, until the test-uploading logic is separated from Coin. Then it will be easier to find the full information about what happened with flaky or crashed tests by special logs from qt-testrunner.py, which is the test re-running orchestrator and therefore has more knowledge about what went wrong. Fixes: QTQAINFRA-4754 Change-Id: I69e4469cbe6c0a6ca5b2473eaf8d034632564342 Reviewed-by: Fabian Kosmale <[email protected]> Reviewed-by: Toni Saario <[email protected]> Reviewed-by: Simo Fält <[email protected]>
* qt-testrunner: do not try to re-run initTestCase and othersDimitrios Apostolou2022-01-304-23/+53
| | | | | | | | | | | | | Some function names are special for qtestlib, in the sense that they can not be specified as a command line argument to run individually. In such cases qt-testrunner treats the failure specially and tries once to re-run the full test executable. Fixes: QTBUG-89011 Change-Id: I0cc25f91c57374e5ac65ade10e2e223fe969f211 Reviewed-by: Volker Hilsheimer <[email protected]> Reviewed-by: Daniel Smith <[email protected]>
* qt-testrunner: be more clear that it was the test that crashedDimitrios Apostolou2022-01-181-4/+9
| | | | | | | Task-number: QTBUG-99970 Change-Id: Id2d01b6ab7d428356d9dfc953107014791393d35 Reviewed-by: Fabian Kosmale <[email protected]> Reviewed-by: Jani Heikkinen <[email protected]>
* qt-testrunner: disable JUnit XML test logsDimitrios Apostolou2022-01-121-3/+1
| | | | | | | | It causes crashes because of interleaved execution of Quick tests. Task-number: QTBUG-98350 Change-Id: I61e26f5da5617e06810db8eb7991f17eecdd3acb Reviewed-by: Tor Arne Vestbø <[email protected]>
* Enable verbose logging in the last re-run of a failing testDimitrios Apostolou2022-01-033-8/+29
| | | | | | Fixes: COIN-728 Change-Id: I08802e377e26e5dd7f7d1c44b5efe4280b09f957 Reviewed-by: Daniel Smith <[email protected]>
* Introduce qt-testrunner.pyDimitrios Apostolou2022-01-034-0/+878
Script that wraps Qt test execution in order to iron-out flakiness. Task-number: QTBUG-96353 Change-Id: Ie8ee780e3f4f8d086c080a7784f9f68fd1867be2 Reviewed-by: Daniel Smith <[email protected]>
generated by cgit v1.2.3 (git 2.25.1) at 2025-12-14 09:17:44 +0000