BUG: parse shell escaping in extra_compile_args and extra_link_args #12925
Conversation
78970cf to
2c1fcc3
Compare
|
LGTM, thanks @eric-wieser. Would be good to get confirmation from @thisco-de that it fixes things for him before merging. |
|
Sorry for the late response, but I haven't had the chance to get back on this any earlier. I can confirm, though, that Eric's patches in eric-wieser:distutils-shlex-split indeed allow me to build numpy and scipy with my old .numpy-site.cfg again. However, what is still weird, though, is that my Anyway, the fixes above appear to address the original issue properly on my side. Thank you, guys! |
|
Tests fail on windows |
|
@eric-wieser Ping. The failures look like they may be a problem with the test. |
|
Yep, looks like the test needs to shlex.quote when generating the site.cfg file. |
|
OK, here is a small update on my problem with NPY_DISTUTILS_APPEND_FLAGS=1 and The environment variable I am not sure if this is actually related to the problem or rather worth another issue. However, from my point of view |
2c1fcc3 to
31e74d7
Compare
|
Sorry for the delay, updated with hopefully fixed tests. Note that there is a behavior change here - previously, Is this a desirable change, or should I try and preserve the old behavior (see: #12979) |
dccb635 to
59eba9f
Compare
numpy/distutils/system_info.py
Outdated
| subprocess.list2cmdline([sys.executable, '-c', 'import sys;print(sys.argv)']), cmd | ||
| ) | ||
| ret = subprocess.check_output(full_cmd) | ||
| return ast.literal_eval(ret) |
There was a problem hiding this comment.
Did not mean to commit this...
In #12979 I opted for json instead, since that's more robust to unicode without depending on console encodings.
b86b205 to
b5f8d5d
Compare
|
Updated with better windows behavior that is less likely to cause problems |
Thanks to a change in exec_command, these strings are no longer passed onto the shell.
Since config files do not support list values, our best bet is to perform shell-splitting immediately.
This brings the behavior back in line a little to how it was before.
On windows systems, the behavior has changed. Previously it was treated as a single argument unless it contained quotes, resulting in the following weird behavior:
# passes as one argument, preserving spaces
extra_link_args=-Wl,rpath=A:/path/with spaces
# passes as two arguments, preserving spaces
extra_link_args="-Wl,rpath=A:\path\with spaces" -lgfortran
# passes as one long quoted argument (surprising and undesirable)
extra_link_args=-Wl,rpath=A:\path\without_spaces -lgfortran
Now it behaves as windows escaping via subprocess (but _not_ via cmd) normally would:
# Passed as two separate arguments (probably not as intended, but should be expected)
extra_link_args=-Wl,rpath=A:/path/with spaces
# passes as two arguments, preserving spaces
extra_link_args="-Wl,rpath=A:\path\with spaces" -lgfortran
# passes as two arguments
extra_link_args=-Wl,rpath=A:\path\without_spaces -lgfortran
Fixes numpygh-12659
b5f8d5d to
00ccdfc
Compare
|
Let's give this a shot. Thanks Eric. |
Thanks to a change in exec_command, these strings are no longer passed onto the shell.
Since config files do not support list values, our best bet is to perform shell-splitting immediately.
Likely fixes gh-12659
Backport here
Note that azure CI fails because this doesn't contain either of #12903 or #12909, in the interest of having a common merge base for backporting. Had we taken the merge-base approach in those PRs, it would be possible to merge the commit in directly.
To be clear: this is blocked by a decision on #12979