Fix #37676 : Add E_WARNING when using array-index on non valid container

[bp1222]

This bugfix attempts to resolve 37676. It will make sure that the container is either compiled, or if it's a return-variable make sure it's something that we know is error-able.

Updated tests to account for the E_NOTICE that will go out with this fix.

Other related bugs:
39915
72636

[bp1222]

The failed tests appear to be unrelated to this change

[nikic]

Existing PRs: #1269, #866 -- I think there might even be one more.

The tl;dr is that this should go through the RFC process, because it will likely have fallout (especially dereferencing of null/false). I'm definitely in favor of the change (and think most other people are as well), but I don't think this can be merged directly.

[bp1222]

I suspect you're right. I guess I didn't look close enough to existing PR's, as none were attached to the bug tickets.

[cmb69]

Yes, an RFC would be required, see the rather lengthy and controversial discussion about this issue from last year. Are you interested in authoring such an RFC, David?

[bp1222]

Thanks Chris. This is exactly what I was looking for before I start to
really work on an RFC for this. I checked all the bugs but they didn't
reference this chain.
On Fri, Jul 29, 2016 at 06:25 Christoph M. Becker [email protected]
wrote:

Yes, an RFC would be required, see the rather lengthy and controversial
discussion http://marc.info/?t=143379796900001&r=1&w=2 about this issue
from last year. Are you interested in authoring such an RFC
https://wiki.php.net/rfc/howto, David?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#2031 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAfiLA-gKjVNYLbnWSDW70l2ioLrXMfgks5qadUMgaJpZM4JTuS1
.

[abiusx]

Any updates on this? At least add the behavior to the documentation in the meanwhile! This is really unexpected.

[bp1222]

hi @abiusx I have a draft RFC https://wiki.php.net/rfc/notice-for-non-valid-array-container which is currently under discussion. There hasn't been a whole lot of communication on it, but I do plan to push it into voting within the week or two.

[abiusx]

Hi
I have a bug report that more accurately covers this https://bugs.php.net/bug.php?id=72786&thanks=4
I think the rfc needs further details and explanation before going into voting.
Abbas

On Aug 8, 2016, at 1:42 PM, David Walker [email protected] wrote:

hi @abiusx I have a draft RFC https://wiki.php.net/rfc/notice-for-non-valid-array-container which is currently under discussion. There hasn't been a whole lot of communication on it, but I do plan to push it into voting within the week or two.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

[cmb69]

I think the rfc needs further details and explanation before going into voting.

Maybe, maybe not. :-) Anyhow, there's no need to hurry, because the RFC is way to late for PHP 7.1 anyway, unfortunately.

[bp1222]

I concur. I didn't think about the differences in execution when assigning by reference rather than by value (Sorry I'm pretty green on internals). Your bug does raise some interesting points, and I'm a bit concerned that $null_r =& $null[$index_r] mutates $null. I would have expected that to toss an notice/warning and keep $null as null. The interesting part to me is EXPECTED(Z_TYPE_P(container) <= IS_FALSE), IS_FALSE is defined to 2, IS_TRUE is defined to 3. So had we done $true_r =& $true[$index_r], $true would remain unmodified.

I would be inclined to attempt to standardize the differences when accessing containers for read or for write. Making it so the container does not magically become an array if the variable is undefined/null/false.

[bp1222]

Well, and here's where my lack of experience probably hinders, when doing an assignment of an arrayesque-element by reference and doing a set of an array to a reference we're doing the same opcode.

$float_v =& $float[$index_r]
and
$foo['val'] =& $float

Are both doing a FETCH_DIM_W, then ASSIGN_REF. My inexperience here would be to question why the former is a FETCH_DIM_W? Is it because we're planning on increasing the ref-count on the value in that container? I get the latter, wherein if $foo is a bool/null/undef it should be created as an array to store the reference to $float.

[abiusx]

The FETCH_DIM_W is the parser struct, anything that is accessed using array operator [] is FETCH_DIM_W,
-A

On Aug 8, 2016, at 3:08 PM, David Walker [email protected] wrote:

Well, and here's where my lack of experience probably hinders, when doing an assignment of an arrayesque-element by reference and doing a set of an array to a reference we're doing the same opcode.

$float_v =& $float[$index_r]
and
$foo['val'] =& $float

Are both doing a FETCH_DIM_W, then ASSIGN_REF. My inexperience here would be to question why the former is a FETCH_DIM_W? Is it because we're planning on increasing the ref-count on the value in that container? I get the latter, wherein if $foo is a bool/null/undef it should be created as an array to store the reference to $float.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #2031 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/ABVjWxHcGm0oIgrYLgvOQoq4jsLXICkpks5qd361gaJpZM4JTuS1.

[nikic]

@bp1222 References are bidirectional. $float_v =& $float[$index_r] turns $float_v and $float[$index_r] into references, so both are fetched for write.

Btw, don't forget that as of PHP 7.1 list('key' => $x) = $y is a thing, so the first access is not necessarily on index 0.

[abiusx]

That should explain the weird null behavior.

On Aug 8, 2016, at 4:20 PM, Nikita Popov [email protected] wrote:

@bp1222 https://github.com/bp1222 References are bidirectional. $float_v =& $float[$index_r] turns $float_v and $float[$index_r] into references.

Btw, don't forget that as of PHP 7.1 list('key' => $x) = $y is a thing, so the first access is not necessarily on index 0.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #2031 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/ABVjWyupfyHdzt6FEqOod782mOCCSsE2ks5qd4-HgaJpZM4JTuS1.

[nikic]

Could you please extract this condition into a function? It's pretty hard to understand what is really going on here...

[bp1222]

I was planning on doing that before all said and done. I'm trying to limit the sheer quantity of notices that could be expelled for many cases. Specifically

foreach(list($a, $b) = each($arr)) was notice'ing when attempting to assign the final null value of each.

[nikic]

I'm not sure special-casing each() is a good idea. At least, the way you do it (not specifically tailored to each()) will have side-effects on other cases as well (and tailoring it is even more crazy). This condition looks like a large assembly of various special-cases, which will introduce a bunch of extra inconsistencies resulting in different behavior based on minor differences in code -- like whether something reads a simple variable or an array offset/function call result.

(I'd actually consider it a benefit if uses of each() in that way started to generate notices -- serves as a natural deprecation. This pattern needs to die.)

[bp1222]

I guess that'd be left to debate/argue, but I'm trying to introduce fewest warnings for code that isn't a user accessing an element: like list(), and each iin a foreach/while. Although there may be better error types for seeing that pattern and raising an E_DEPRECEATED or something more targeted to that behavior.

[nikic]

In that case, might it be preferable to exclude list() entirely?

[bp1222]

I wouldn't be opposed to that. Might want to have list() continue to exist as is without modification

[bp1222]

@nikic - I had forgotten list('key' => $val) was a thing. I'm not certain, how one could keep tabs on list-assignment in a sane way, but I'll take a stab at something.

@abiusx - That would explain the null behavior. Not sure if that'd be something worth changing in this RFC, or not.
Thanks for feedback all.

[bp1222]

I removed list from raising warnings, and would want to treat this change as a read-operation warning, rather than read/write. Barring any comments here (@Internals has been quiet on my thread) I'm thinking this change for what it's purpose is, is just about done.

[abiusx]

Can you try the following code snippet and post the results of your version:
`<?php
$int=5;
$bool=true;
$null=null;
$float=5.1;

$index_v=5;
$index_r=6;

echo "Null:\n";
$null_v=$null[$index_v];
$null_r=&$null[$index_r]; //$null is converted into array and referenced to $null_r, which evaluates to null
var_dump($null,$null_v,$null_r);
echo str_repeat("-",80),PHP_EOL;

// die();
echo "Int:\n";
$int_v=$int[$index_v];
$int_r=&$int[$index_r];
var_dump($int,$int_v,$int_r);
echo str_repeat("-",80),PHP_EOL;

echo "Bool:\n";
$bool_v=$bool[$index_v];
$bool_r=&$bool[$index_r];
var_dump($bool,$bool_v,$bool_r);
echo str_repeat("-",80),PHP_EOL;

echo "Float:\n";
$float_v=$float[$index_v];
$float_r=&$float[$index_r];
var_dump($float,$float_v,$float_r);
echo str_repeat("-",80),PHP_EOL;

echo "Null multidim:\n";
$null2=null;
$null2_v=$null2[1][2][3];
$null2_r=&$null2[1][2][3];
var_dump($null2,$null2_v,$null2_r);
`

[bp1222]

Null:

Notice: Variable of type null does not accept array offsets in /mnt/dev/devshare/home/dwalker/php/foo.php on line 11
array(1) {
  [6]=>
  &NULL
}
NULL
NULL
--------------------------------------------------------------------------------
Int:

Notice: Variable of type integer does not accept array offsets in /mnt/dev/devshare/home/dwalker/php/foo.php on line 18

Notice: Undefined variable: int_r in /mnt/dev/devshare/home/dwalker/php/foo.php on line 20
int(5)
NULL
NULL
--------------------------------------------------------------------------------
Bool:

Notice: Variable of type boolean does not accept array offsets in /mnt/dev/devshare/home/dwalker/php/foo.php on line 24

Notice: Undefined variable: bool_r in /mnt/dev/devshare/home/dwalker/php/foo.php on line 26
bool(true)
NULL
NULL
--------------------------------------------------------------------------------
Float:

Notice: Variable of type float does not accept array offsets in /mnt/dev/devshare/home/dwalker/php/foo.php on line 30

Notice: Undefined variable: float_r in /mnt/dev/devshare/home/dwalker/php/foo.php on line 32
float(5.1)
NULL
NULL
--------------------------------------------------------------------------------
Null multidim:

Notice: Variable of type null does not accept array offsets in /mnt/dev/devshare/home/dwalker/php/foo.php on line 37
array(1) {
  [1]=>
  array(1) {
    [2]=>
    array(1) {
      [3]=>
      &NULL
    }
  }
}
NULL
NULL

[bp1222]

My only 'problem' with this, is that null values used in a reference assignment are converted to array, rather than failing as they do in the case of int/bool/float. It might be out of scope for this RFC, but I'd rather see only undef's be mutated.
$undef_v =& $undef[$elem_r]
would make $undef an array, and reference the null. But that's probably for a different thread of discussion.

[abiusx]

Good.
Maybe state in the RFC that this is out of scope, so that future reference can help.
Your output seems reasonable to me, a notice should suffice.
Currently PHP emits Warnings though, you might wanna keep it compatible.
-A

On Aug 10, 2016, at 1:25 PM, David Walker [email protected] wrote:

My only 'problem' with this, is that null values used in a reference assignment are converted to array, rather than failing as they do in the case of int/bool/float. It might be out of scope for this RFC, but I'd rather see only undef's be mutated.
$undef_v =& $undef[$elem_r]
would make $undef an array, and reference the null. But that's probably for a different thread of discussion.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #2031 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/ABVjW_Sq9b2UB4uJ8wpMvSCDBWonBLILks5qegmdgaJpZM4JTuS1.

[dstogov]

why do we need the check for IS_VAR?

[bp1222]

The check here for not IS_VAR is to aide in preventing raising multiple warnings for

$a = null;
$a[1][2][3];

When accessing $a[1], the container op1_type is a CV, but the return of that for access of [2] is VAR. This was the attempt to only show the warning for user based code like this.

Although since you made me double check that I have found that it's not necessarily perfect.

function foo() {
    return null;
}
foo()[1];

Does not properly throw any warning, since the return of a function IS_VAR

I'll change it around to look if IS_VAR the previous op wasn't a function call, so we can properly throw warning on the first access post-call.

[nikic]

@bp1222 Won't that still incorrectly suppress the warning for the case of $a = [null]; $a[0][1];? The [1] is on null.

[bp1222]

It does. Needs to check more previous op's to determine if we ought to raise the error or not when container is null, but a valid null.

[nikic]

Our standard approach to suppressing duplicate warnings is use of ERROR zvals. Doing this will need some care to ensure there is no impact on the fast-path though.

[bp1222]

Yeah. I looked into using ZVAL_ERROR(), however, it's at the end of zend_fetch_dimension_address_read that it sets the return to NULL if we hit error cases. This could have some cascading effects which I notices, for example:

$a = null;
var_dump($a[0]);

would not be NULL, rather it'd be UNDEFINED:0. This causes many many problems in existing tests where it's expecting the result of a failed lookup to be NULL. I assume internally they are functionally similar but I haven't looked much deeper.

My thought to accommodate your example would be to look to the previous opcode and determine if a null in this context is error-able or not. So if we're not looking at a compiled-var as the container, if it's null, but the previous op-code wasn't a ZEND_FETCH_DIM_R we should error. This would limit the erroring on a null -only- to if the container wasn't a result of a fetch already.

[bugreportuser]

These tests fail when opcache is disabled:

• https://github.com/bp1222/php-src/blob/fix-36767/ext/phar/tests/create_new_and_modify.phpt
• https://github.com/bp1222/php-src/blob/fix-36767/ext/phar/tests/delete_in_phar.phpt
• https://github.com/bp1222/php-src/blob/fix-36767/ext/phar/tests/delete_in_phar_confirm.phpt
• https://github.com/bp1222/php-src/blob/fix-36767/ext/phar/tests/tar/create_new_and_modify.phpt
• https://github.com/bp1222/php-src/blob/fix-36767/ext/phar/tests/tar/delete_in_phar.phpt
• https://github.com/bp1222/php-src/blob/fix-36767/ext/phar/tests/tar/delete_in_phar_confirm.phpt
• https://github.com/bp1222/php-src/blob/fix-36767/ext/phar/tests/zip/create_new_and_modify.phpt
• https://github.com/bp1222/php-src/blob/fix-36767/ext/phar/tests/zip/delete_in_phar.phpt
• https://github.com/bp1222/php-src/blob/fix-36767/ext/phar/tests/zip/delete_in_phar_confirm.phpt

opcache_get_status() can return false, which is also undocumented.

[carusogabriel]

opcache_get_status() can return false, which is also undocumented.

Confirmed:

php-src/ext/opcache/zend_accelerator_module.c

Lines 600 to 606 in d87fdd9

	if (!validate_api_restriction()) {
	RETURN_FALSE;
	}
	if (!accel_startup_ok) {
	RETURN_FALSE;
	}

We should document that.

[cmb69]

We should document that.

Thanks! Done, (might need further clarification, though)

[SCIF]

Hey guys! It's really great to see work on bug aged 12+ years. Any chance to deliver it with v7.4?

[bugreportuser]

@SCIF Thanks for the ping. I meant to revisit this.

@bp1222 Are you still interested in working on this? I can take it over if not.

[bp1222]

I would continue working on this, however, the last time it was in a merge-ready state the discussion was about the implementation relying on examining next op-codes in a delicate way. So even resolving the opcache, and current conflict, would still leave the major issue unaddressed. I wouldn't want to see any more effort going into this 2+year old PR without a solid end goal in place.

[bugreportuser]

Good point. @dstogov @nikic Since you were pinged about this before, what do you think about the delicate checking of op-codes? Is there a better way to check? I assume @cmb69 was talking about this line:
https://github.com/php/php-src/pull/2031/files#diff-a5fb5fd8c4f7311c6d0788763e665daaR2041

[cmb69]

I wouldn't want to see any more effort going into this 2+year old PR without a solid end goal in place.

ACK

@bugreportuser Yes. See also #2031 (comment) f

[SCIF]

Hi again :) Is there any news for past 3 months? Thanks!

[nikic]

Thanks for the ping. I believe that at this point, we mainly need to resolve the question of list() handling.

To recap, some people think that writing if (list($foo, $bar) = getFooBar()) should not generate any kind of warning if getFooBar() returns null. Other people don't think that's a legal code pattern, and don't want to lose warnings in other cases (such as list() outside of if) due to it.

I think the possible options we have to resolve this are:

• list() accesses should be treated exactly the same way as normal array accesses. This is what the PR currently implements.
• list() should be entirely excluded from these checks.
• null should be entirely excluded from these checks.
• Only list() on null should be excluded from these checks.

I'm not really sure what to do about this. I hate the idea of special-casing list() here, but if we want to do it we should probably exclude it entirely (the argument for null can just as well be made for false).

If we can't reach a consensus here, this will need to go back to the internals list.

[bugreportuser]

To recap, some people think that writing if (list($foo, $bar) = getFooBar()) should not generate any kind of warning if getFooBar() returns null. Other people don't think that's a legal code pattern, and don't want to lose warnings in other cases (such as list() outside of if) due to it.

A possible middle ground is to rely on strict_types:

• enabled: list() accesses are treated exactly the same way as normal array accesses
• disabled: null is entirely excluded from these checks

[SCIF]

I believe a lot of developers would love approach to treat as a normal array accesses.

However, as a compromise solution that looks not bad:

A possible middle ground is to rely on strict_types:

I mean, there should be a way to treat things strict at least with defined strict_types.

[Majkl578]

Please don't propose to abuse strict_types for stuff it's not intended for - strict_types are about type coercion, not array accesses.

[carusogabriel]

I believe this is also related to #54556.

[Girgias]

Thanks for the ping. I believe that at this point, we mainly need to resolve the question of list() handling.

To recap, some people think that writing if (list($foo, $bar) = getFooBar()) should not generate any kind of warning if getFooBar() returns null. Other people don't think that's a legal code pattern, and don't want to lose warnings in other cases (such as list() outside of if) due to it.

I think the possible options we have to resolve this are:

* list() accesses should be treated exactly the same way as normal array accesses. This is what the PR currently implements.

* list() should be entirely excluded from these checks.

* null should be entirely excluded from these checks.

* Only list() on null should be excluded from these checks.

I'm not really sure what to do about this. I hate the idea of special-casing list() here, but if we want to do it we should probably exclude it entirely (the argument for null can just as well be made for false).

If we can't reach a consensus here, this will need to go back to the internals list.

I don't see why list() accesses should be treated differently than normal array accesses, moreover it is pretty trivial to do a null check within the same condition.
And as you say if we start special casing null why not false?

[SCIF]

Hey guys, another half of year past. Appreciate your work, but if there any chance to make this working and merged? Thanks!

[nikic]

New PR up at #4386.

[SCIF]

Wow! So this should be closed since #4386 is merged already :)

[nikic]

Yeah, sorry this took so long, should have merged a reduced variant long ago.

I think as far as list() is concerned, we might consider "blessing" the if (list() = $x) pattern by actually doing different codegen if list() is used in a conditional context and make it a first-class refutable binding. But that's a bigger discussion, so for now list() is just excluded from this check.