Expectation is not counted correctly when a doubled method is called more often than is expected

[cabbey]

Q	A
PHPUnit version	11.4.3
PHP version	8.3.12
Installation Method	Composer

acmephp/core                                 2.1.0              Raw implementation of the ACME protocol in PHP
acmephp/ssl                                  2.1.0              PHP wrapper around OpenSSL extension providing SSL...
async-aws/core                               1.20.1             Core package to integrate with AWS. This is a ligh...
async-aws/lambda                             1.9.0              Lambda client, part of the AWS SDK provided by Asy...
aws/aws-crt-php                              v1.2.4             AWS Common Runtime for PHP
aws/aws-php-sns-message-validator            1.9.0              Amazon SNS message validation for PHP
aws/aws-sdk-php                              3.288.1            AWS SDK for PHP - Use Amazon Web Services in your ...
bacon/bacon-qr-code                          2.0.8              BaconQrCode is a QR code generator for PHP.
brick/math                                   0.11.0             Arbitrary-precision arithmetic library
clue/stream-filter                           v1.6.0             A simple and modern approach to stream filtering i...
composer/ca-bundle                           1.4.0              Lets you find a path to the system CA bundle, and ...
dasprid/enum                                 1.0.3              PHP 7.1 enum implementation
elasticsearch/elasticsearch                  v7.17.2            PHP Client for Elasticsearch
endroid/qr-code                              4.0.0              Endroid QR Code
ezimuel/guzzlestreams                        3.1.0              Fork of guzzle/streams (abandoned) to be used with...
ezimuel/ringphp                              1.2.2              Fork of guzzle/RingPHP (abandoned) to be used with...
fidry/cpu-core-counter                       0.4.1              Tiny utility to get the number of CPU cores.
firebase/php-jwt                             v6.3.1             A simple library to encode and decode JSON Web Tok...
fpdf/fpdf                                    1.85.0             FPDF Composer Wrapper
geoip2/geoip2                                v2.13.0            MaxMind GeoIP2 PHP API
giggsey/libphonenumber-for-php-lite          8.13.3             A lite version of giggsey/libphonenumber-for-php, ...
google/apiclient                             v2.17.0            Client library for Google APIs
google/apiclient-services                    v0.375.1           Client library for Google APIs
google/auth                                  v1.37.1            Google Auth Library for PHP
guzzlehttp/guzzle                            7.8.1              Guzzle is a PHP HTTP client library
guzzlehttp/promises                          1.5.3              Guzzle promises library
guzzlehttp/psr7                              2.7.0              PSR-7 message implementation that also provides co...
hollodotme/fast-cgi-client                   v3.1.7             A PHP fast CGI client to send requests (a)synchron...
intercom/intercom-php                        4.4.3              Intercom API client built on top of HTTPlug
kelvinmo/simplejwt                           v0.5.1             A simple JSON Web Token library for PHP.
kickbox/kickbox                              2.2.6.2            Official kickbox API library client for PHP
laminas/laminas-diactoros                    2.26.0             PSR HTTP Message implementations
laminas/laminas-httphandlerrunner            2.10.0             Execute PSR-15 RequestHandlerInterface instances a...
launchdarkly/server-sdk                      4.2.4              Official LaunchDarkly SDK for PHP
launchdarkly/server-sdk-dynamodb             1.0.0              LaunchDarkly PHP SDK DynamoDB integration
lcobucci/clock                               2.2.0              Yet another clock abstraction
lcobucci/jwt                                 4.0.4              A simple library to work with JSON Web Token and J...
league/container                             4.2.0              A fast and intuitive dependency injection container.
league/event                                 3.0.2              Event package
league/iso3166                               4.1.0              ISO 3166-1 PHP Library
league/route                                 5.1.2              Fast routing and dispatch component including PSR-...
league/uri                                   6.8.0              URI manipulation library
league/uri-interfaces                        2.3.0              Common interface for URI representation
maennchen/zipstream-php                      2.0.0              ZipStream is a library for dynamically streaming d...
maxmind-db/reader                            v1.11.1            MaxMind DB Reader API
maxmind/web-service-common                   v0.9.0             Internal MaxMind Web Service API
moneyphp/money                               v4.5.0             PHP implementation of Fowler's Money pattern
monolog/monolog                              2.9.3              Sends your logs to files, sockets, inboxes, databa...
mtdowling/jmespath.php                       2.7.0              Declaratively specify how to extract elements from...
myclabs/deep-copy                            1.12.0             Create deep copies (clones) of your objects
myclabs/php-enum                             1.8.4              PHP Enum implementation
nikic/fast-route                             v1.3.0             Fast request router for PHP
nikic/php-parser                             v5.3.1             A PHP parser written in PHP
nyholm/psr7                                  1.8.1              A fast PHP7 implementation of PSR-7
opensearch-project/opensearch-php            1.0.2              PHP Client for OpenSearch
opis/closure                                 3.6.3              A library that can be used to serialize closures (...
paragonie/constant_time_encoding             v2.6.3             Constant-time Implementations of RFC 4648 Encoding...
paragonie/csp-builder                        v2.9.0             Easily add and update Content-Security-Policy head...
paragonie/random_compat                      v9.99.100          PHP 5.x polyfill for random_bytes() and random_int...
phar-io/manifest                             2.0.4              Component for reading phar.io manifest information...
phar-io/version                              3.2.1              Library for handling version information and const...
php-http/client-common                       2.6.0              Common HTTP Client implementations and tools for H...
php-http/discovery                           1.14.3             Finds installed HTTPlug implementations and PSR-7 ...
php-http/guzzle7-adapter                     1.0.0              Guzzle 7 HTTP Adapter
php-http/httplug                             2.3.0              HTTPlug, the HTTP client abstraction for PHP
php-http/message                             1.13.0             HTTP Message related tools
php-http/message-factory                     1.1.0              Factory interfaces for PSR-7 HTTP Message
php-http/promise                             1.1.0              Promise used for asynchronous HTTP requests
php-parallel-lint/php-parallel-lint          v1.3.2             This tool check syntax of PHP files about 20x fast...
phpseclib/phpseclib                          3.0.37             PHP Secure Communications Library - Pure-PHP imple...
phpstan/phpstan                              1.12.7             PHPStan - PHP Static Analysis Tool
phpunit/php-code-coverage                    11.0.7             Library that provides collection, processing, and ...
phpunit/php-file-iterator                    5.1.0              FilterIterator implementation that filters files b...
phpunit/php-invoker                          5.0.1              Invoke callables with a timeout
phpunit/php-text-template                    4.0.1              Simple template engine.
phpunit/php-timer                            7.0.1              Utility class for timing
phpunit/phpunit                              11.4.3             The PHP Unit Testing framework.
pragmarx/google2fa                           v8.0.1             A One Time Password Authentication package, compat...
psalm/phar                                   5.17.0             Composer-based Psalm Phar
psr/cache                                    1.0.1              Common interface for caching libraries
psr/clock                                    1.0.0              Common interface for reading the clock.
psr/container                                1.1.2              Common Container Interface (PHP FIG PSR-11)
psr/event-dispatcher                         1.0.0              Standard interfaces for event handling.
psr/http-client                              1.0.3              Common interface for HTTP clients
psr/http-factory                             1.0.2              Common interfaces for PSR-7 HTTP message factories
psr/http-message                             1.1                Common interface for HTTP messages
psr/http-server-handler                      1.0.2              Common interface for HTTP server-side request handler
psr/http-server-middleware                   1.0.2              Common interface for HTTP server-side middleware
psr/log                                      2.0.0              Common interface for logging libraries
psr/simple-cache                             1.0.1              Common interfaces for simple caching
ralouphie/getallheaders                      3.0.3              A polyfill for getallheaders.
ramsey/collection                            2.0.0              A PHP library for representing and manipulating co...
ramsey/uuid                                  4.7.5              A PHP library for generating and working with univ...
react/promise                                v2.11.0            A lightweight implementation of CommonJS Promises/...
rector/rector                                1.2.8              Instant Upgrade and Automated Refactoring of any P...
riverline/multipart-parser                   2.1.1              One class library to parse multipart content with ...
roave/security-advisories                    dev-latest 01fd41b Prevents installation of composer packages with kn...
sebastian/cli-parser                         3.0.2              Library for parsing CLI options
sebastian/code-unit                          3.0.1              Collection of value objects that represent the PHP...
sebastian/code-unit-reverse-lookup           4.0.1              Looks up which function or method a line of code b...
sebastian/comparator                         6.1.1              Provides the functionality to compare PHP values f...
sebastian/complexity                         4.0.1              Library for calculating the complexity of PHP code...
sebastian/diff                               6.0.2              Diff implementation
sebastian/environment                        7.2.0              Provides functionality to handle HHVM/PHP environm...
sebastian/exporter                           6.1.3              Provides the functionality to export PHP variables...
sebastian/global-state                       7.0.2              Snapshotting of global state
sebastian/lines-of-code                      3.0.1              Library for counting the lines of code in PHP sour...
sebastian/object-enumerator                  6.0.1              Traverses array structures and object graphs to en...
sebastian/object-reflector                   4.0.1              Allows reflection of object attributes, including ...
sebastian/recursion-context                  6.0.2              Provides functionality to recursively process PHP ...
sebastian/type                               5.1.0              Collection of value objects that represent the typ...
sebastian/version                            5.0.2              Library that helps with managing the version numbe...
smartystreets/phpsdk                         4.17.0             The official PHP client library from SmartyStreets.
snowplow/snowplow-tracker                    0.6.1              Snowplow event tracker for PHP. Add analytics into...
spomky-labs/base64url                        v2.0.4             Base 64 URL Safe Encoding/Decoding PHP Library
squizlabs/php_codesniffer                    3.7.2              PHP_CodeSniffer tokenizes PHP, JavaScript and CSS ...
staabm/annotate-pull-request-from-checkstyle 1.8.5             
stella-maris/clock                           0.1.7              A pre-release of the proposed PSR-20 Clock-Interface
stripe/stripe-php                            v15.2.0            Stripe PHP Library
symfony/console                              v5.4.35            Eases the creation of beautiful and testable comma...
symfony/deprecation-contracts                v3.4.0             A generic function and convention to trigger depre...
symfony/filesystem                           v6.4.3             Provides basic utilities for the filesystem
symfony/http-client                          v6.2.13            Provides powerful methods to fetch HTTP resources ...
symfony/http-client-contracts                v3.4.0             Generic abstractions related to HTTP clients
symfony/options-resolver                     v6.2.0             Provides an improved replacement for the array_rep...
symfony/polyfill-ctype                       v1.29.0            Symfony polyfill for ctype functions
symfony/polyfill-intl-grapheme               v1.29.0            Symfony polyfill for intl's grapheme_* functions
symfony/polyfill-intl-normalizer             v1.29.0            Symfony polyfill for intl's Normalizer class and r...
symfony/polyfill-mbstring                    v1.29.0            Symfony polyfill for the Mbstring extension
symfony/polyfill-php73                       v1.29.0            Symfony polyfill backporting some PHP 7.3+ feature...
symfony/polyfill-php80                       v1.29.0            Symfony polyfill backporting some PHP 8.0+ feature...
symfony/process                              v6.4.3             Executes commands in sub-processes
symfony/service-contracts                    v3.4.1             Generic abstractions related to writing services
symfony/string                               v6.4.3             Provides an object-oriented API to strings and dea...
symfony/yaml                                 v5.4.35            Loads and dumps YAML files
theseer/tokenizer                            1.2.3              A small library for converting tokenized PHP sourc...
tideways/ext-tideways-stubs                  v5.6.0.1           IDE stubs for Tideways Profiler PHP extension
ulrichsg/getopt-php                          v4.0.3             Command line arguments parser for PHP 7.1 and above
vube/monolog-splunk-formatter                v2.0               Splunk Line Formatter for Monolog
webmozart/assert                             1.11.0             Assertions to validate method input/output with ni...
wikimedia/less.php                           v3.1.0             PHP port of the Javascript version of LESS http://...

Summary

We have some testcases that rely 100% on configuring mocks with expectations for their assertions. I'll focus on one of them that has 6 variations from it's data provider. With the upgrade to phpunit 11, all 6 variations now report as risky, saying they did not perform any assertions. This is demonstrably false, as I can intentionally break the mocked objects passed into the test from the dataProvider and observe that it does in fact fail the test because the expectations on another mocked object don't align with how the object is called in the code under test, however if the expectation is not violated it doesn't trigger. (For example, if the expectation is that a method will be called once, and it is never called, then the failure will not be seen. But if it is called twice, the failure will be reported.) This only happens with mocks provided via dataProviders, those created in the test function do not exhibit this behavior.

Current behavior

phpunit -c phpunit.xml tests/phpunit/functional/SmugMug/Stats/Image/ZombieAccessProcessorTest.php 
using vendor/bin/phpunit
PHPUnit 11.4.3 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.12
Configuration: /home/cabbey/sandboxes/10/phpunit.xml

RRRRRR                                                              6 / 6 (100%)

Time: 00:00.463, Memory: 62.50 MB

There were 6 risky tests:

1) TestSuite\Functional\SmugMug\Stats\Image\ZombieAccessProcessorTest::testEvaluateStatsForInvalidAccess with data set "allGoodModernUrlNoSignature" (SmugMug\Stats\Image\CollectionForVerification Object (...), MockObject_FlushCoordinator_e18fb1d3 Object (...), MockObject_ImageRepository_4d1241ba Object (...))
This test did not perform any assertions

/home/cabbey/sandboxes/10/tests/phpunit/functional/SmugMug/Stats/Image/ZombieAccessProcessorTest.php:25

... five more instances of the same thing with a different data set ...

OK, but there were issues!
Tests: 6, Assertions: 0, PHPUnit Deprecations: 1, Risky: 6.

How to reproduce

bugDemo.php.gz

I've extracted the issue out of our test cases and attached a simple testcase file which has 9 test functions in a 3x3 grid as follows:

1. the mock object is:
   a. created locally
   b. injected from a dataProvider
   c. injected from a dataProvider and registered with $this->registerMockObject() in the function
2. the number of calls to the demo() function is:
   a. zero (should fail)
   b. one (should pass)
   c. two (should fail)

0	1	2
F	P	F	Expected
F	P	FR	created locally
R	R	FR	Injected
F	P	FR	Registered

Using the internal registerMockObject() function in the test to register the mocks injected from the data provider appears to bring the behavior closer into alignment with expectations, but still unexpectedly reports a risky test on the two call version. This leaves me with very little faith that using this internal function is a proper solution to our issue. (and needing to use an internal function to make the test case whole seems... alarming.)

(phpunit 11.5.2 behaves the same as 11.4.3 with this test; but there are issues with some of our other tests that prevent me from using 11.5 across the whole test suite just yet.)

Phpunit 10.5.40 with this test case is slightly different, but is consistent:

0	1	2
F	P	F	Expected
F	P	FR	created locally
F	P	FR	Injected
F	P	FR	Registered

Expected behavior

Under phpunit 10.5.40 these tests run perfectly with no deprecation warnings:

phpunit -c phpunit.xml tests/phpunit/functional/SmugMug/Stats/Image/ZombieAccessProcessorTest.php 
using vendor/bin/phpunit
PHPUnit 10.5.40 by Sebastian Bergmann and contributors.

Runtime:       PHP 8.3.12
Configuration: /home/cabbey/sandboxes/10/phpunit.xml

......                                                              6 / 6 (100%)

Time: 00:00.037, Memory: 46.50 MB

OK (6 tests, 6 assertions)

The only difference here is that I did composer require --dev phpunit/phpunit:^10.0 -W after running with 11.4.3 above to back down to 10.5.40... all code under test was the same.

[sebastianbergmann]

Under phpunit 10.5.40 these tests run perfectly with no deprecation warnings

I assume by "deprecation warnings" you mean "risky tests", as I do not see any deprecation warnings in your output for PHPUnit 10.5.

[sebastianbergmann]

Do you use TestCase::registerMockObject() in your regular test suite or do you only use it in your reproducing test case? I ask because this method is marked @internal and you are not supposed to use it.

Also note that creating mock objects in a data provider does not work. Your workaround ($mockFactory = new self('factory'); $mock = $mockFactory->createMock(bugExample::class);) does not work.

[sebastianbergmann]

I ran your tests testLocalZero(), testLocalOne(), and testLocalTwo() with PHPUnit 10.5 and PHPUnit 11.5:

testLocalZero()

$mock = $this->createMock(bugExample::class);
$mock->expects($this->once())
    ->method('demo');

✅ PHPUnit 10.5: The test fails and is not considered to be risky
✅ PHPUnit 11.5: The test fails and is not considered to be risky

---

testLocalOne()

$mock = $this->createMock(bugExample::class);
$mock->expects($this->once())
    ->method('demo');
$mock->demo();

✅ PHPUnit 10.5: The test passes and is not considered to be risky
✅ PHPUnit 11.5: The test passes and is not considered to be risky

---

testLocalTwo()

$mock = $this->createMock(bugExample::class);
$mock->expects($this->once())
    ->method('demo');
$mock->demo();
$mock->demo();

⚡ PHPUnit 10.5: The test fails (correct) and is considered to be risky (incorrect)
⚡ PHPUnit 11.5: The test fails (correct) and is considered to be risky (incorrect)

---

As you can see, I am not able to reproduce that PHPUnit 10.5 and PHPUnit 11.5 behave differently when running these tests.

The fact that both PHPUnit 10.5 and PHPUnit 11.5 consider testLocalTwo() to be risky is a bug that should be fixed.

[sebastianbergmann]

For some reason, I do not remember why, the case that a doubled method is called more often than it is expected to be called is handled differently than all other mock object-related expectation failures.

If a doubled method is expected to be called once and is called a second time then the execution of the test is immediately aborted and the test is considered a failure. However, the fact that an expectation was verified is not recorded by incrementing the assertion counter.

A look into the Git history of this part of the code reminded me that I was working on this a while ago:

• ec59ed6
• 14303e3
• dd6d329
• 2f4d105

[cabbey]

Under phpunit 10.5.40 these tests run perfectly with no deprecation warnings

I assume by "deprecation warnings" you mean "risky tests", as I do not see any deprecation warnings in your output for PHPUnit 10.5.

The "deprecation warning" from PHPUnit 11 was due to using a docblock for the @dataProvider declaration. I updated it to an attribute and that went away. I think we can ignore the deprecation warnings.

[cabbey]

Do you use TestCase::registerMockObject() in your regular test suite or do you only use it in your reproducing test case? I ask because this method is marked @internal and you are not supposed to use it.

No we do not currently use it anywhere in our test suite. It was added to the reproduction case as one of our engineers noted the difference between the behavior of the locally created mocks and the injected mocks was that the injected ones were not registered. After seeing that it almost completely resolved the issues with the pattern, I do have a PR started to add it to the actual tests in our suite that are encountering this... but as you say, it's not something we are supposed to be doing, and as I said in the initial report, it feels dirty to have to do it. I would rather not.

Also note that creating mock objects in a data provider does not work. Your workaround ($mockFactory = new self('factory'); $mock = $mockFactory->createMock(bugExample::class);) does not work.

Until now, it has worked just fine... we do this in a number of our tests. With the addition of the registerMockObject() call, it's almost back to fully functional. If it were not for the fact that this particular test didn't have any other assertions in it, we probably wouldn't have noticed this. In looking at other tests of ours I suspect a number of them are now silently broken and will fail to alert if expectations change in a way that isn't triggered in some other fashion.

We also create a number of stubs in this same method, often those are then injected into the mocks. For example a domain object mock will have a repository stub injected which has some domain object stubs injected into it, all of which are configured for the given test variation to represent a given state of the objects in the situation under test.

Can you point to any additional documentation or background on what doesn't work about this? (I've just re-read both the test doubles and data providers section of the PHPUnit manual, and unless I missed something, neither section mentions the other's concepts at all, let alone gives any advice not to mix the two.)

I ran your tests testLocalZero(), testLocalOne(), and testLocalTwo() with PHPUnit 10.5 and PHPUnit 11.5:

testLocalZero()

Sorry, I perhaps was not clear... testLocal*() were provided as a baseline to show "this works without injection". The problem this issue was raised to address is the behavior of the testInject*() methods. The testRegisterInected*() methods were provided to demonstrate the distasteful workaround of doing the registration in the test method.