run-tests: use the EXTENSIONS section for skipping

[MaxSem]

Currently, most skip checks are just for making sure an extension is
available. Even with recent addition of skip caching, this makes tests
needlessly slow:

• Checks for the same extension in its tests can have small differences
  impacting cacheability.
• Even identical skip checks in two tests can still be executed twice if
  they're run by different workers.

To remedy this, I'm repurposing the existing --EXTENSIONS-- section of
.phpt files to specify wjich extensions are required for current test to
run. Current behavior:

1. If the extension is already visible to PHP, all is good
2. If it isn't, assume it's present as a shared module and attempt to add
   it via a command line parameter
3. If that works, all is good
4. If it doesn't, PHP fails with a cryptic error message trying to
   execute the test itself

After this commit:

1. and 2) are kept unchanged
2. Check if shared extension file from 2) is actually present
3. Skip the test if it isn't

Other benefits include clear skip reasons (vs. sometimes none in many
current skip checks) and moving test information from code to metadata,
opening more opportunities for search and analysis.

Since --EXTENSIONS-- is barely ever used, this change poses no risk of
hidden failures.

As a demonstration of the new approach, this commit migrates one
extension to it. If merged, I will migrate other extensions in
subsequent PRs.

[nikic]

tests/run-test/extensions-nonexistent.phpt failed

[MaxSem]

Ignores be damned...

[mvorisek]

I would like to see something more abstract, not only for extensions, like:

ext-sodium
platform-windows
x86 / x64 / arm64

etc.

[MaxSem]

Yep, I have plans for --PLATFORM-- after that :)

[nikic]

Windows:

========DIFF========
001+ 'TEST_PHP_JUNIT' is not recognized as an internal or external command,
001- .*tests="1" failures="0" errors="0" skip="1".*
002+ operable program or batch file.
003+ Test runner finished with exit code 1
========DONE========
FAIL phpt EXTENSIONS directive - extension not found [C:\projects\php-src\tests\run-test\extensions-nonexistent.phpt]

The CI results on azure look completely broken, but not sure if that's due to this PR or you just hit a bad master revision.

[nikic]

?> terminators in test files should be kept, to make tests directly runnable.

[MaxSem]

Done, though there are a lot of existing tests missing these.

[kocsismate]

Yeah, last year I added thousands of such terminators, but it was still not comprehensive.

[MaxSem]

I can make run-tests warn on such tests.

[nikic]

run-tests is also used by extensions, so warnings are problematic. Maybe add this to tidy.php instead.

[cmb69]

Just seen this; very nice! That may also give the opportunity to improve the performance of the tests on Windows, by only loading the required extenstions in advance. The --with-test-ini-ext-exclude config option which is set in build_task.bat might make that trivial.

[carusogabriel]

@MaxSem Can we document this at http://qa.php.net/write-test.php?

[MaxSem]

@carusogabriel php/web-qa#34

[cmb69]

I just realized that this may not work as expected when all tests are supposed to be run, due to:

php-src/run-tests.php

Lines 863 to 865 in 048cb9b

	// load list of enabled extensions
	save_text($info_file,
	'<?php echo str_replace("Zend OPcache", "opcache", implode(",", get_loaded_extensions())); ?>');

Unless we want to change run-tests.php, the test runner itself would have to be run with all (relevant) extensions loaded.

PS: no, it's not the test runner itself, but rather the TEST_PHP_EXECUTABLE, what defeats the purpose of this PR (only for all tests; testing individual extension tests suites works fine).

[nikic]

@cmb69 We should probably adjust that to include both loaded and loadable extensions. Should be fairly simple?

[cmb69]

@nikic, we could do something like:

 run-tests.php | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/run-tests.php b/run-tests.php
index 3c7189b24d..e767598e67 100755
--- a/run-tests.php
+++ b/run-tests.php
@@ -861,8 +861,21 @@ More .INIs  : " , (function_exists(\'php_ini_scanned_files\') ? str_replace("\n"
     @unlink($info_file);
 
     // load list of enabled extensions
-    save_text($info_file,
-        '<?php echo str_replace("Zend OPcache", "opcache", implode(",", get_loaded_extensions())); ?>');
+    save_text($info_file, <<<'PHP'
+        <?php
+        echo str_replace("Zend OPcache", "opcache", implode(",", get_loaded_extensions()));
+        $ext_dir = ini_get("extension_dir");
+        foreach (scandir($ext_dir) as $file) {
+            if (!preg_match('/^(?:php_)?([_a-zA-Z0-9]+)\.(?:so|dll)$/', $file, $matches)) {
+                continue;
+            }
+            $ext = $matches[1];
+            if (!extension_loaded($ext) && @dl($file)) {
+                echo ",", $ext;
+            }
+        }
+        ?>
+    PHP);
     $exts_to_test = explode(',', `$php $pass_options $info_params $no_file_cache "$info_file"`);
     // check for extensions that need special handling and regenerate
     $info_params_ex = [

But for AppVeyor this would mean that tests would be run for snmp,pdo_firebird,ldap,imap,ftp, although these tests would always be skipped. Might still be a performance improvement in the long run. I'll submit a PR ASAP.

[MaxSem]

Yep, I have plans for --PLATFORM-- after that :)

🎆 #7325 🎆