Skip to content

Fix #37676 : Add E_WARNING when using array-index on non valid container #2031

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from

Conversation

bp1222
Copy link
Contributor

@bp1222 bp1222 commented Jul 25, 2016

This bugfix attempts to resolve 37676. It will make sure that the container is either compiled, or if it's a return-variable make sure it's something that we know is error-able.

Updated tests to account for the E_NOTICE that will go out with this fix.

Other related bugs:
39915
72636

@bp1222
Copy link
Contributor Author

bp1222 commented Jul 25, 2016

The failed tests appear to be unrelated to this change

@nikic
Copy link
Member

nikic commented Jul 25, 2016

Existing PRs: #1269, #866 -- I think there might even be one more.

The tl;dr is that this should go through the RFC process, because it will likely have fallout (especially dereferencing of null/false). I'm definitely in favor of the change (and think most other people are as well), but I don't think this can be merged directly.

@bp1222
Copy link
Contributor Author

bp1222 commented Jul 25, 2016

I suspect you're right. I guess I didn't look close enough to existing PR's, as none were attached to the bug tickets.

@cmb69
Copy link
Member

cmb69 commented Jul 29, 2016

Yes, an RFC would be required, see the rather lengthy and controversial discussion about this issue from last year. Are you interested in authoring such an RFC, David?

@bp1222
Copy link
Contributor Author

bp1222 commented Jul 29, 2016

Thanks Chris. This is exactly what I was looking for before I start to
really work on an RFC for this. I checked all the bugs but they didn't
reference this chain.
On Fri, Jul 29, 2016 at 06:25 Christoph M. Becker [email protected]
wrote:

Yes, an RFC would be required, see the rather lengthy and controversial
discussion http://marc.info/?t=143379796900001&r=1&w=2 about this issue
from last year. Are you interested in authoring such an RFC
https://wiki.php.net/rfc/howto, David?


You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#2031 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAfiLA-gKjVNYLbnWSDW70l2ioLrXMfgks5qadUMgaJpZM4JTuS1
.

@abiusx
Copy link

abiusx commented Aug 8, 2016

Any updates on this? At least add the behavior to the documentation in the meanwhile! This is really unexpected.

@bp1222
Copy link
Contributor Author

bp1222 commented Aug 8, 2016

hi @abiusx I have a draft RFC https://wiki.php.net/rfc/notice-for-non-valid-array-container which is currently under discussion. There hasn't been a whole lot of communication on it, but I do plan to push it into voting within the week or two.

@abiusx
Copy link

abiusx commented Aug 8, 2016

Hi
I have a bug report that more accurately covers this https://bugs.php.net/bug.php?id=72786&thanks=4
I think the rfc needs further details and explanation before going into voting.
Abbas

On Aug 8, 2016, at 1:42 PM, David Walker [email protected] wrote:

hi @abiusx I have a draft RFC https://wiki.php.net/rfc/notice-for-non-valid-array-container which is currently under discussion. There hasn't been a whole lot of communication on it, but I do plan to push it into voting within the week or two.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

@cmb69
Copy link
Member

cmb69 commented Aug 8, 2016

I think the rfc needs further details and explanation before going into voting.

Maybe, maybe not. :-) Anyhow, there's no need to hurry, because the RFC is way to late for PHP 7.1 anyway, unfortunately.

@bp1222
Copy link
Contributor Author

bp1222 commented Aug 8, 2016

I concur. I didn't think about the differences in execution when assigning by reference rather than by value (Sorry I'm pretty green on internals). Your bug does raise some interesting points, and I'm a bit concerned that $null_r =& $null[$index_r] mutates $null. I would have expected that to toss an notice/warning and keep $null as null. The interesting part to me is EXPECTED(Z_TYPE_P(container) <= IS_FALSE), IS_FALSE is defined to 2, IS_TRUE is defined to 3. So had we done $true_r =& $true[$index_r], $true would remain unmodified.

I would be inclined to attempt to standardize the differences when accessing containers for read or for write. Making it so the container does not magically become an array if the variable is undefined/null/false.

@bp1222
Copy link
Contributor Author

bp1222 commented Aug 8, 2016

Well, and here's where my lack of experience probably hinders, when doing an assignment of an arrayesque-element by reference and doing a set of an array to a reference we're doing the same opcode.

$float_v =& $float[$index_r]
and
$foo['val'] =& $float

Are both doing a FETCH_DIM_W, then ASSIGN_REF. My inexperience here would be to question why the former is a FETCH_DIM_W? Is it because we're planning on increasing the ref-count on the value in that container? I get the latter, wherein if $foo is a bool/null/undef it should be created as an array to store the reference to $float.

@abiusx
Copy link

abiusx commented Aug 8, 2016

The FETCH_DIM_W is the parser struct, anything that is accessed using array operator [] is FETCH_DIM_W,
-A

On Aug 8, 2016, at 3:08 PM, David Walker [email protected] wrote:

Well, and here's where my lack of experience probably hinders, when doing an assignment of an arrayesque-element by reference and doing a set of an array to a reference we're doing the same opcode.

$float_v =& $float[$index_r]
and
$foo['val'] =& $float

Are both doing a FETCH_DIM_W, then ASSIGN_REF. My inexperience here would be to question why the former is a FETCH_DIM_W? Is it because we're planning on increasing the ref-count on the value in that container? I get the latter, wherein if $foo is a bool/null/undef it should be created as an array to store the reference to $float.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #2031 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/ABVjWxHcGm0oIgrYLgvOQoq4jsLXICkpks5qd361gaJpZM4JTuS1.

@nikic
Copy link
Member

nikic commented Aug 8, 2016

@bp1222 References are bidirectional. $float_v =& $float[$index_r] turns $float_v and $float[$index_r] into references, so both are fetched for write.

Btw, don't forget that as of PHP 7.1 list('key' => $x) = $y is a thing, so the first access is not necessarily on index 0.

@nikic nikic added the RFC label Aug 8, 2016
@abiusx
Copy link

abiusx commented Aug 8, 2016

That should explain the weird null behavior.

On Aug 8, 2016, at 4:20 PM, Nikita Popov [email protected] wrote:

@bp1222 https://github.com/bp1222 References are bidirectional. $float_v =& $float[$index_r] turns $float_v and $float[$index_r] into references.

Btw, don't forget that as of PHP 7.1 list('key' => $x) = $y is a thing, so the first access is not necessarily on index 0.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #2031 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/ABVjWyupfyHdzt6FEqOod782mOCCSsE2ks5qd4-HgaJpZM4JTuS1.

if (type != BP_VAR_IS) {
if (UNEXPECTED(Z_TYPE_P(container) == IS_UNDEF)) {
zval_undefined_cv(EG(current_execute_data)->opline->op1.var, EG(current_execute_data));
} else if (EXPECTED((EG(current_execute_data)->opline->opcode == ZEND_FETCH_LIST && Z_LVAL_P(dim) == 0 && Z_TYPE_P(container) != IS_FALSE && (Z_TYPE_P(container) != IS_STRING || (Z_TYPE_P(container) == IS_STRING && Z_STRLEN_P(container) != 0))) || (EG(current_execute_data)->opline->opcode != ZEND_FETCH_LIST && (EG(current_execute_data)->opline->op1_type != IS_VAR || (Z_TYPE_P(container) != IS_UNDEF && Z_TYPE_P(container) != IS_NULL && (support_strings && Z_TYPE_P(container) != IS_STRING)))))) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please extract this condition into a function? It's pretty hard to understand what is really going on here...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was planning on doing that before all said and done. I'm trying to limit the sheer quantity of notices that could be expelled for many cases. Specifically

foreach(list($a, $b) = each($arr)) was notice'ing when attempting to assign the final null value of each.

Copy link
Member

@nikic nikic Aug 8, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure special-casing each() is a good idea. At least, the way you do it (not specifically tailored to each()) will have side-effects on other cases as well (and tailoring it is even more crazy). This condition looks like a large assembly of various special-cases, which will introduce a bunch of extra inconsistencies resulting in different behavior based on minor differences in code -- like whether something reads a simple variable or an array offset/function call result.

(I'd actually consider it a benefit if uses of each() in that way started to generate notices -- serves as a natural deprecation. This pattern needs to die.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess that'd be left to debate/argue, but I'm trying to introduce fewest warnings for code that isn't a user accessing an element: like list(), and each iin a foreach/while. Although there may be better error types for seeing that pattern and raising an E_DEPRECEATED or something more targeted to that behavior.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In that case, might it be preferable to exclude list() entirely?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't be opposed to that. Might want to have list() continue to exist as is without modification

@bp1222
Copy link
Contributor Author

bp1222 commented Aug 8, 2016

@nikic - I had forgotten list('key' => $val) was a thing. I'm not certain, how one could keep tabs on list-assignment in a sane way, but I'll take a stab at something.

@abiusx - That would explain the null behavior. Not sure if that'd be something worth changing in this RFC, or not.
Thanks for feedback all.

@bp1222
Copy link
Contributor Author

bp1222 commented Aug 10, 2016

I removed list from raising warnings, and would want to treat this change as a read-operation warning, rather than read/write. Barring any comments here (@Internals has been quiet on my thread) I'm thinking this change for what it's purpose is, is just about done.

@abiusx
Copy link

abiusx commented Aug 10, 2016

Can you try the following code snippet and post the results of your version:
`<?php
$int=5;
$bool=true;
$null=null;
$float=5.1;

$index_v=5;
$index_r=6;

echo "Null:\n";
$null_v=$null[$index_v];
$null_r=&$null[$index_r]; //$null is converted into array and referenced to $null_r, which evaluates to null
var_dump($null,$null_v,$null_r);
echo str_repeat("-",80),PHP_EOL;

// die();
echo "Int:\n";
$int_v=$int[$index_v];
$int_r=&$int[$index_r];
var_dump($int,$int_v,$int_r);
echo str_repeat("-",80),PHP_EOL;

echo "Bool:\n";
$bool_v=$bool[$index_v];
$bool_r=&$bool[$index_r];
var_dump($bool,$bool_v,$bool_r);
echo str_repeat("-",80),PHP_EOL;

echo "Float:\n";
$float_v=$float[$index_v];
$float_r=&$float[$index_r];
var_dump($float,$float_v,$float_r);
echo str_repeat("-",80),PHP_EOL;

echo "Null multidim:\n";
$null2=null;
$null2_v=$null2[1][2][3];
$null2_r=&$null2[1][2][3];
var_dump($null2,$null2_v,$null2_r);
`

@bp1222
Copy link
Contributor Author

bp1222 commented Aug 10, 2016

Null:

Notice: Variable of type null does not accept array offsets in /mnt/dev/devshare/home/dwalker/php/foo.php on line 11
array(1) {
  [6]=>
  &NULL
}
NULL
NULL
--------------------------------------------------------------------------------
Int:

Notice: Variable of type integer does not accept array offsets in /mnt/dev/devshare/home/dwalker/php/foo.php on line 18

Notice: Undefined variable: int_r in /mnt/dev/devshare/home/dwalker/php/foo.php on line 20
int(5)
NULL
NULL
--------------------------------------------------------------------------------
Bool:

Notice: Variable of type boolean does not accept array offsets in /mnt/dev/devshare/home/dwalker/php/foo.php on line 24

Notice: Undefined variable: bool_r in /mnt/dev/devshare/home/dwalker/php/foo.php on line 26
bool(true)
NULL
NULL
--------------------------------------------------------------------------------
Float:

Notice: Variable of type float does not accept array offsets in /mnt/dev/devshare/home/dwalker/php/foo.php on line 30

Notice: Undefined variable: float_r in /mnt/dev/devshare/home/dwalker/php/foo.php on line 32
float(5.1)
NULL
NULL
--------------------------------------------------------------------------------
Null multidim:

Notice: Variable of type null does not accept array offsets in /mnt/dev/devshare/home/dwalker/php/foo.php on line 37
array(1) {
  [1]=>
  array(1) {
    [2]=>
    array(1) {
      [3]=>
      &NULL
    }
  }
}
NULL
NULL

@bp1222
Copy link
Contributor Author

bp1222 commented Aug 10, 2016

My only 'problem' with this, is that null values used in a reference assignment are converted to array, rather than failing as they do in the case of int/bool/float. It might be out of scope for this RFC, but I'd rather see only undef's be mutated.
$undef_v =& $undef[$elem_r]
would make $undef an array, and reference the null. But that's probably for a different thread of discussion.

@abiusx
Copy link

abiusx commented Aug 10, 2016

Good.
Maybe state in the RFC that this is out of scope, so that future reference can help.
Your output seems reasonable to me, a notice should suffice.
Currently PHP emits Warnings though, you might wanna keep it compatible.
-A

On Aug 10, 2016, at 1:25 PM, David Walker [email protected] wrote:

My only 'problem' with this, is that null values used in a reference assignment are converted to array, rather than failing as they do in the case of int/bool/float. It might be out of scope for this RFC, but I'd rather see only undef's be mutated.
$undef_v =& $undef[$elem_r]
would make $undef an array, and reference the null. But that's probably for a different thread of discussion.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub #2031 (comment), or mute the thread https://github.com/notifications/unsubscribe-auth/ABVjW_Sq9b2UB4uJ8wpMvSCDBWonBLILks5qegmdgaJpZM4JTuS1.

@bp1222 bp1222 changed the title Fix #37676 : Add E_NOTICE when using array-index on non valid container Fix #37676 : Add E_WARNING when using array-index on non valid container Aug 10, 2016
@bp1222 bp1222 force-pushed the fix-37676 branch 2 times, most recently from 0bf495e to c8d7077 Compare August 10, 2016 23:17
@@ -1795,6 +1795,18 @@ static zend_never_inline void zend_fetch_dimension_address_UNSET(zval *result, z
zend_fetch_dimension_address(result, container_ptr, dim, dim_type, BP_VAR_UNSET);
}

static zend_always_inline int zend_fetch_dimension_address_read_valid_container(zval *container, int support_strings) {
if (EG(current_execute_data)->opline->opcode != ZEND_FETCH_LIST) {
if (EG(current_execute_data)->opline->op1_type != IS_VAR) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need the check for IS_VAR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The check here for not IS_VAR is to aide in preventing raising multiple warnings for

$a = null;
$a[1][2][3];

When accessing $a[1], the container op1_type is a CV, but the return of that for access of [2] is VAR. This was the attempt to only show the warning for user based code like this.

Although since you made me double check that I have found that it's not necessarily perfect.

function foo() {
    return null;
}
foo()[1];

Does not properly throw any warning, since the return of a function IS_VAR

I'll change it around to look if IS_VAR the previous op wasn't a function call, so we can properly throw warning on the first access post-call.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bp1222 Won't that still incorrectly suppress the warning for the case of $a = [null]; $a[0][1];? The [1] is on null.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does. Needs to check more previous op's to determine if we ought to raise the error or not when container is null, but a valid null.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our standard approach to suppressing duplicate warnings is use of ERROR zvals. Doing this will need some care to ensure there is no impact on the fast-path though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. I looked into using ZVAL_ERROR(), however, it's at the end of zend_fetch_dimension_address_read that it sets the return to NULL if we hit error cases. This could have some cascading effects which I notices, for example:

$a = null;
var_dump($a[0]);

would not be NULL, rather it'd be UNDEFINED:0. This causes many many problems in existing tests where it's expecting the result of a failed lookup to be NULL. I assume internally they are functionally similar but I haven't looked much deeper.

My thought to accommodate your example would be to look to the previous opcode and determine if a null in this context is error-able or not. So if we're not looking at a compiled-var as the container, if it's null, but the previous op-code wasn't a ZEND_FETCH_DIM_R we should error. This would limit the erroring on a null -only- to if the container wasn't a result of a fetch already.

@carusogabriel
Copy link
Contributor

opcache_get_status() can return false, which is also undocumented.

Confirmed:

if (!validate_api_restriction()) {
RETURN_FALSE;
}
if (!accel_startup_ok) {
RETURN_FALSE;
}

We should document that.

@cmb69
Copy link
Member

cmb69 commented Oct 6, 2018

We should document that.

Thanks! Done, (might need further clarification, though)

salathe pushed a commit to salathe/phpdoc-en that referenced this pull request Oct 6, 2018
svn2github pushed a commit to svn2github/phpdoc_en that referenced this pull request Oct 7, 2018
@SCIF
Copy link

SCIF commented Nov 6, 2018

Hey guys! It's really great to see work on bug aged 12+ years. Any chance to deliver it with v7.4?

@bugreportuser
Copy link
Contributor

@SCIF Thanks for the ping. I meant to revisit this.

@bp1222 Are you still interested in working on this? I can take it over if not.

@bp1222
Copy link
Contributor Author

bp1222 commented Nov 7, 2018

I would continue working on this, however, the last time it was in a merge-ready state the discussion was about the implementation relying on examining next op-codes in a delicate way. So even resolving the opcache, and current conflict, would still leave the major issue unaddressed. I wouldn't want to see any more effort going into this 2+year old PR without a solid end goal in place.

@bugreportuser
Copy link
Contributor

Good point. @dstogov @nikic Since you were pinged about this before, what do you think about the delicate checking of op-codes? Is there a better way to check? I assume @cmb69 was talking about this line:
https://github.com/php/php-src/pull/2031/files#diff-a5fb5fd8c4f7311c6d0788763e665daaR2041

@cmb69
Copy link
Member

cmb69 commented Nov 7, 2018

I wouldn't want to see any more effort going into this 2+year old PR without a solid end goal in place.

ACK

@bugreportuser Yes. See also #2031 (comment) f

@SCIF
Copy link

SCIF commented Feb 1, 2019

Hi again :) Is there any news for past 3 months? Thanks!

@nikic
Copy link
Member

nikic commented Feb 1, 2019

Thanks for the ping. I believe that at this point, we mainly need to resolve the question of list() handling.

To recap, some people think that writing if (list($foo, $bar) = getFooBar()) should not generate any kind of warning if getFooBar() returns null. Other people don't think that's a legal code pattern, and don't want to lose warnings in other cases (such as list() outside of if) due to it.

I think the possible options we have to resolve this are:

  • list() accesses should be treated exactly the same way as normal array accesses. This is what the PR currently implements.
  • list() should be entirely excluded from these checks.
  • null should be entirely excluded from these checks.
  • Only list() on null should be excluded from these checks.

I'm not really sure what to do about this. I hate the idea of special-casing list() here, but if we want to do it we should probably exclude it entirely (the argument for null can just as well be made for false).

If we can't reach a consensus here, this will need to go back to the internals list.

@bugreportuser
Copy link
Contributor

To recap, some people think that writing if (list($foo, $bar) = getFooBar()) should not generate any kind of warning if getFooBar() returns null. Other people don't think that's a legal code pattern, and don't want to lose warnings in other cases (such as list() outside of if) due to it.

A possible middle ground is to rely on strict_types:

  • enabled: list() accesses are treated exactly the same way as normal array accesses
  • disabled: null is entirely excluded from these checks

@SCIF
Copy link

SCIF commented Feb 2, 2019

I believe a lot of developers would love approach to treat as a normal array accesses.

However, as a compromise solution that looks not bad:

A possible middle ground is to rely on strict_types:

I mean, there should be a way to treat things strict at least with defined strict_types.

@Majkl578
Copy link
Contributor

Majkl578 commented Feb 2, 2019

Please don't propose to abuse strict_types for stuff it's not intended for - strict_types are about type coercion, not array accesses.

@carusogabriel
Copy link
Contributor

I believe this is also related to #54556.

@Girgias
Copy link
Member

Girgias commented Feb 23, 2019

Thanks for the ping. I believe that at this point, we mainly need to resolve the question of list() handling.

To recap, some people think that writing if (list($foo, $bar) = getFooBar()) should not generate any kind of warning if getFooBar() returns null. Other people don't think that's a legal code pattern, and don't want to lose warnings in other cases (such as list() outside of if) due to it.

I think the possible options we have to resolve this are:

* list() accesses should be treated exactly the same way as normal array accesses. This is what the PR currently implements.

* list() should be entirely excluded from these checks.

* null should be entirely excluded from these checks.

* Only list() on null should be excluded from these checks.

I'm not really sure what to do about this. I hate the idea of special-casing list() here, but if we want to do it we should probably exclude it entirely (the argument for null can just as well be made for false).

If we can't reach a consensus here, this will need to go back to the internals list.

I don't see why list() accesses should be treated differently than normal array accesses, moreover it is pretty trivial to do a null check within the same condition.
And as you say if we start special casing null why not false?

@SCIF
Copy link

SCIF commented Jul 10, 2019

Hey guys, another half of year past. Appreciate your work, but if there any chance to make this working and merged? Thanks!

@nikic
Copy link
Member

nikic commented Jul 10, 2019

New PR up at #4386.

@SCIF
Copy link

SCIF commented Jul 10, 2019

Wow! So this should be closed since #4386 is merged already :)

@nikic
Copy link
Member

nikic commented Jul 10, 2019

Yeah, sorry this took so long, should have merged a reduced variant long ago.

I think as far as list() is concerned, we might consider "blessing" the if (list() = $x) pattern by actually doing different codegen if list() is used in a conditional context and make it a first-class refutable binding. But that's a bigger discussion, so for now list() is just excluded from this check.

@nikic nikic closed this Jul 10, 2019
heiglandreas pushed a commit to phpdoctest/en that referenced this pull request Feb 4, 2020
salathe pushed a commit to salathe/phpdoc-en that referenced this pull request Sep 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.