BUG: Issue with stripping on macOS Monterey + xcode 13.2

[tacaswell]

Describe your issue.

After successfully compiling, during the final steps of installation

pip install --no-build-isolation .

with the current default branch of scipy fails with an error while stripping the files. I think the key line is

  /Library/Developer/CommandLineTools/usr/bin/strip: error: symbols referenced by indirect symbol table entries that can't be stripped in: /Users/tcaswell/source/p/scipy/scipy/.mesonpy-2w0tepol/install/Users/tcaswell/.pybuild/bleeding/lib/python3.10/site-packages/scipy/_lib/_ccallback_c.cpython-310-darwin.so

This is with the tip of the Python 3.10 branch (3.10.5+). Also tried with the main branch of meson-python.

[2/2] Generating scipy/generate-version with a custom command
  Installing scipy/__config__.py to /Users/tcaswell/source/p/scipy/scipy/.mesonpy-2w0tepol/install/Users/tcaswell/.pybuild/bleeding/lib/python3.10/site-packages//scipy
  Installing scipy/version.py to /Users/tcaswell/source/p/scipy/scipy/.mesonpy-2w0tepol/install/Users/tcaswell/.pybuild/bleeding/lib/python3.10/site-packages//scipy
  Installing scipy/_lib/_ccallback_c.cpython-310-darwin.so to /Users/tcaswell/source/p/scipy/scipy/.mesonpy-2w0tepol/install/Users/tcaswell/.pybuild/bleeding/lib/python3.10/site-packages/scipy/_lib
  Stripping target 'scipy/_lib/_ccallback_c.cpython-310-darwin.so'.
  Could not strip file.

  Stdout:


  Stderr:
  /Library/Developer/CommandLineTools/usr/bin/strip: error: symbols referenced by indirect symbol table entries that can't be stripped in: /Users/tcaswell/source/p/scipy/scipy/.mesonpy-2w0tepol/install/Users/tcaswell/.pybuild/bleeding/lib/python3.10/site-packages/scipy/_lib/_ccallback_c.cpython-310-darwin.so
  _PyBytes_AsStringAndSize
  _PyBytes_FromString
  _PyBytes_FromStringAndSize
  _PyCallable_Check
  _PyCapsule_GetContext
  _PyCapsule_GetName
  _PyCapsule_GetPointer
  _PyCapsule_New
  _PyCapsule_SetContext
  _PyCode_NewEmpty
  _PyCode_NewWithPosOnlyArgs
  _PyDict_GetItemString
  _PyDict_New
  _PyDict_Next
  _PyDict_SetItem
  _PyDict_SetItemString
  _PyDict_Size
  _PyErr_Clear
  _PyErr_ExceptionMatches
  _PyErr_Format
  _PyErr_GivenExceptionMatches
  _PyErr_NoMemory
  _PyErr_NormalizeException
  _PyErr_Occurred
  _PyErr_PrintEx
  _PyErr_SetObject
  _PyErr_SetString
  _PyErr_WarnEx
  _PyErr_WarnFormat
  _PyErr_WriteUnraisable
  _PyEval_RestoreThread
  _PyEval_SaveThread
  _PyException_SetTraceback
  _PyFloat_AsDouble
  _PyFloat_FromDouble
  _PyFloat_FromString
  _PyFrame_New
  _PyGILState_Ensure
  _PyGILState_Release
  _PyImport_AddModule
  _PyImport_GetModule
  _PyImport_GetModuleDict
  _PyImport_ImportModule
  _PyImport_ImportModuleLevelObject
  _PyInterpreterState_GetID
  _PyList_Append
  _PyList_New
  _PyLong_AsDouble
  _PyLong_AsLong
  _PyLong_AsSsize_t
  _PyLong_AsVoidPtr
  _PyLong_FromLong
  _PyLong_FromSize_t
  _PyLong_FromSsize_t
  _PyMem_Free
  _PyMem_Malloc
  _PyMem_Realloc
  _PyMethod_New
  _PyModuleDef_Init
  _PyModule_AddObject
  _PyModule_GetDict
  _PyModule_NewObject
  _PyNumber_Add
  _PyNumber_Index
  _PyOS_snprintf
  _PyOS_string_to_double
  _PyObject_Call
  _PyObject_ClearWeakRefs
  _PyObject_Free
  _PyObject_GC_Del
  _PyObject_GC_Track
  _PyObject_GC_UnTrack
  _PyObject_GetAttr
  _PyObject_GetAttrString
  _PyObject_GetItem
  _PyObject_GetIter
  _PyObject_Hash
  _PyObject_IsTrue
  _PyObject_Not
  _PyObject_RichCompare
  _PyObject_SetAttr
  _PyObject_SetAttrString
  _PyThreadState_Get
  _PyTraceBack_Here
  _PyTuple_GetItem
  _PyTuple_GetSlice
  _PyTuple_New
  _PyTuple_Pack
  _PyType_IsSubtype
  _PyType_Ready
  _PyUnicode_AsUTF8
  _PyUnicode_AsUTF8AndSize
  _PyUnicode_Compare
  _PyUnicode_Decode
  _PyUnicode_DecodeASCII
  _PyUnicode_FromFormat
  _PyUnicode_FromString
  _PyUnicode_FromStringAndSize
  _PyUnicode_InternFromString
  _Py_EnterRecursiveCall
  _Py_GetVersion
  _Py_LeaveRecursiveCall
  __PyDict_GetItem_KnownHash
  __PyObject_GC_New
  __PyObject_GenericGetAttrWithDict
  __PyObject_GetDictPtr
  __PyThreadState_UncheckedGet
  __PyUnicode_IsWhitespace
  __PyUnicode_Ready
  __Py_Dealloc
  ___assert_rtn
  ___stack_chk_fail
  _free
  _memcmp
  _sin
  _strcmp
  _strdup
  _strrchr
  _PyBaseObject_Type
  _PyByteArray_Type
  _PyBytes_Type
  _PyCFunction_Type
  _PyCapsule_Type
  _PyExc_AttributeError
  _PyExc_DeprecationWarning
  _PyExc_ImportError
  _PyExc_NameError
  _PyExc_OverflowError
  _PyExc_RuntimeError
  _PyExc_RuntimeWarning
  _PyExc_StopIteration
  _PyExc_SystemError
  _PyExc_TypeError
  _PyExc_ValueError
  _PyFloat_Type
  _PyList_Type
  _PyLong_Type
  _PyMethod_Type
  _PyObject_GenericGetAttr
  _PyTuple_Type
  _PyUnicode_Type
  __PyByteArray_empty_string
  __Py_FalseStruct
  __Py_NoneStruct
  __Py_TrueStruct
  __Py_ascii_whitespace
  ___stack_chk_guard
  _libcd_validate_csblobs: superblob is invalid

  /Library/Developer/CommandLineTools/usr/bin/strip: warning: changes being made to the file will invalidate the code signature in: /Users/tcaswell/source/p/scipy/scipy/.mesonpy-2w0tepol/install/Users/tcaswell/.pybuild/bleeding/lib/python3.10/site-packages/scipy/_lib/_ccallback_c.cpython-310-darwin.so


  + meson setup --native-file=/Users/tcaswell/source/p/scipy/scipy/.mesonpy-native-file.ini -Ddebug=false -Dstrip=true -Doptimization=2 --prefix=/Users/tcaswell/.pybuild/bleeding /Users/tcaswell/source/p/scipy/scipy /Users/tcaswell/source/p/scipy/scipy/.mesonpy-2w0tepol/build
  + meson compile
  + meson install --destdir /Users/tcaswell/source/p/scipy/scipy/.mesonpy-2w0tepol/install
  Traceback (most recent call last):
    File "/Users/tcaswell/.virtualenvs/bleeding/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 156, in prepare_metadata_for_build_wheel
      hook = backend.prepare_metadata_for_build_wheel
  AttributeError: module 'mesonpy' has no attribute 'prepare_metadata_for_build_wheel'

  During handling of the above exception, another exception occurred:

  Traceback (most recent call last):
    File "/Users/tcaswell/.virtualenvs/bleeding/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/Users/tcaswell/.virtualenvs/bleeding/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/Users/tcaswell/.virtualenvs/bleeding/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 160, in prepare_metadata_for_build_wheel
      whl_basename = backend.build_wheel(metadata_directory, config_settings)
    File "/Users/tcaswell/.virtualenvs/bleeding/lib/python3.10/site-packages/mesonpy/__init__.py", line 837, in build_wheel
      return project.wheel(out).name
    File "/Users/tcaswell/.virtualenvs/bleeding/lib/python3.10/site-packages/mesonpy/__init__.py", line 786, in wheel
      wheel = _WheelBuilder(self).build(self._install_plan, self._copy_files, self._build_dir)
    File "/Users/tcaswell/.virtualenvs/bleeding/lib/python3.10/site-packages/mesonpy/__init__.py", line 318, in build
      self._project.build()  # ensure project is built
    File "/Users/tcaswell/.virtualenvs/bleeding/lib/python3.10/site-packages/mesonpy/__init__.py", line 482, in build
      self._meson('install', '--destdir', os.fspath(self._install_dir))
    File "/Users/tcaswell/.virtualenvs/bleeding/lib/python3.10/site-packages/mesonpy/__init__.py", line 430, in _meson
      return self._proc('meson', *args)
    File "/Users/tcaswell/.virtualenvs/bleeding/lib/python3.10/site-packages/mesonpy/__init__.py", line 426, in _proc
      subprocess.check_call(list(args))
    File "/Users/tcaswell/.pybuild/bleeding/lib/python3.10/subprocess.py", line 369, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['meson', 'install', '--destdir', '/Users/tcaswell/source/p/scipy/scipy/.mesonpy-2w0tepol/install']' returned non-zero exit status 1.
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /Users/tcaswell/.virtualenvs/bleeding/bin/python3 /Users/tcaswell/.virtualenvs/bleeding/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py prepare_metadata_for_build_wheel /var/folders/h8/fznvr0sj7hsb027_j45psrvr0000gn/T/tmpw1kosxgz
  cwd: /Users/tcaswell/source/p/scipy/scipy
  Preparing metadata (pyproject.toml) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Reproducing Code Example

pip install -v --no-build-isolation .

Error message

# see above

SciPy/NumPy/Python version information

N/A (can not install scipy)

[rgommers]

Thanks for the report @tacaswell.

This is with the tip of the Python 3.10 branch (3.10.5+). Also tried with the main branch of meson-python.

Is this a debug build of Python? Or do development builds do anything different in build steps compared to releases?

The meson-python build path is relevant here, and could possibly show a bug in Meson. The default we've been using without meson-python is:

  default_options: [
    'buildtype=debugoptimized',

while with meson-python we use '-Ddebug=false, -Dstrip=true, -Doptimization=2.

[rgommers]

See https://mesonbuild.com/Builtin-options.html#core-options

[rgommers]

Can't find any Meson issues, but a similar issue that is macOS-specific in Rust here: rust-lang/rust#93988

[rgommers]

Okay, I can reproduce this on macOS with Python 3.9.6 from conda-forge. So this is a generic issue; stripping .dylib files on macOS requires -x for safety it looks like. I don't think there's anything specific in _ccallback_c, it's simply the first shared library that is being stripped.

I'll look at fixing this in Meson on the weekend. That leaves us with the problem that building from source through pip or build on macOS doesn't work. That's not a hard release blocker perhaps (or maybe it is - up to @tylerjereddy), but it is pretty annoying.

[rgommers]

It looks like Meson uses -dead_strip_dylibs, which may be relevant (see also AnacondaRecipes/intel_repack-feedstock#8)

[rgommers]

A reproducer that is quicker and does not involve meson-python:

% meson setup build --prefix=$PWD/install-build -Dstrip=true
% ninja -C build
% meson install -C build

Result:

arm64-apple-darwin20.0.0-strip: error: symbols referenced by indirect symbol table entries that can't be stripped in: /Users/rgommers/code/scipy/install-build/lib/python3.9/site-packages/scipy/_lib/_ccallback_c.cpython-39-darwin.so
_PyBytes_AsStringAndSize
...
___stack_chk_guard
dyld_stub_binder
arm64-apple-darwin20.0.0-strip: warning: changes being made to the file will invalidate the code signature in: /Users/rgommers/code/scipy/install-build/lib/python3.9/site-packages/scipy/_lib/_ccallback_c.cpython-39-darwin.so

Full output:

ninja: Entering directory `/Users/rgommers/code/scipy/build'
[2/2] Generating scipy/generate-version with a custom command
Installing scipy/__config__.py to /Users/rgommers/code/scipy/install-build/lib/python3.9/site-packages//scipy
Skipping RPATH fixing
Installing scipy/version.py to /Users/rgommers/code/scipy/install-build/lib/python3.9/site-packages//scipy
Skipping RPATH fixing
Installing scipy/_lib/_ccallback_c.cpython-39-darwin.so to /Users/rgommers/code/scipy/install-build/lib/python3.9/site-packages/scipy/_lib
Stripping target 'scipy/_lib/_ccallback_c.cpython-39-darwin.so'.
Could not strip file.

Stdout:


Stderr:
arm64-apple-darwin20.0.0-strip: error: symbols referenced by indirect symbol table entries that can't be stripped in: /Users/rgommers/code/scipy/install-build/lib/python3.9/site-packages/scipy/_lib/_ccallback_c.cpython-39-darwin.so
_PyBytes_AsStringAndSize
_PyBytes_FromString
_PyBytes_FromStringAndSize
_PyCMethod_New
_PyCallable_Check
_PyCapsule_GetContext
_PyCapsule_GetName
_PyCapsule_GetPointer
_PyCapsule_New
_PyCapsule_SetContext
_PyCode_New
_PyDict_GetItemString
_PyDict_New
_PyDict_Next
_PyDict_SetItem
_PyDict_SetItemString
_PyDict_Size
_PyErr_Clear
_PyErr_ExceptionMatches
_PyErr_Format
_PyErr_GivenExceptionMatches
_PyErr_NormalizeException
_PyErr_Occurred
_PyErr_PrintEx
_PyErr_SetObject
_PyErr_SetString
_PyErr_WarnEx
_PyErr_WarnFormat
_PyErr_WriteUnraisable
_PyEval_EvalCodeEx
_PyEval_EvalFrameEx
_PyEval_RestoreThread
_PyEval_SaveThread
_PyException_SetTraceback
_PyFloat_AsDouble
_PyFloat_FromDouble
_PyFloat_FromString
_PyFrame_New
_PyGILState_Ensure
_PyGILState_Release
_PyImport_AddModule
_PyImport_GetModuleDict
_PyImport_ImportModule
_PyImport_ImportModuleLevelObject
_PyInterpreterState_GetID
_PyList_Append
_PyList_New
_PyLong_AsLong
_PyLong_AsSsize_t
_PyLong_AsVoidPtr
_PyLong_FromLong
_PyLong_FromSize_t
_PyLong_FromSsize_t
_PyMem_Malloc
_PyMem_Realloc
_PyModuleDef_Init
_PyModule_AddObject
_PyModule_GetDict
_PyModule_NewObject
_PyNumber_Add
_PyNumber_Index
_PyOS_snprintf
_PyObject_Call
_PyObject_GetAttr
_PyObject_GetAttrString
_PyObject_GetItem
_PyObject_GetIter
_PyObject_Hash
_PyObject_Not
_PyObject_SetAttr
_PyObject_SetAttrString
_PyThreadState_Get
_PyTraceBack_Here
_PyTuple_New
_PyTuple_Pack
_PyType_IsSubtype
_PyUnicode_Compare
_PyUnicode_Decode
_PyUnicode_DecodeASCII
_PyUnicode_FromFormat
_PyUnicode_FromString
_PyUnicode_FromStringAndSize
_PyUnicode_InternFromString
_Py_EnterRecursiveCall
_Py_GetVersion
_Py_LeaveRecursiveCall
__PyDict_GetItem_KnownHash
__PyObject_GetDictPtr
__PyThreadState_UncheckedGet
__Py_Dealloc
___assert_rtn
___stack_chk_fail
_free
_sin
_strcmp
_strdup
_PyBaseObject_Type
_PyByteArray_Type
_PyBytes_Type
_PyCFunction_Type
_PyCapsule_Type
_PyExc_AttributeError
_PyExc_DeprecationWarning
_PyExc_ImportError
_PyExc_NameError
_PyExc_OverflowError
_PyExc_RuntimeError
_PyExc_StopIteration
_PyExc_SystemError
_PyExc_TypeError
_PyExc_ValueError
_PyFloat_Type
_PyFrame_Type
_PyFunction_Type
_PyList_Type
_PyLong_Type
_PyMethod_Type
_PyTuple_Type
_PyUnicode_Type
__PyByteArray_empty_string
__Py_FalseStruct
__Py_NoneStruct
__Py_TrueStruct
___stack_chk_guard
dyld_stub_binder
arm64-apple-darwin20.0.0-strip: warning: changes being made to the file will invalidate the code signature in: /Users/rgommers/code/scipy/install-build/lib/python3.9/site-packages/scipy/_lib/_ccallback_c.cpython-39-darwin.so

[rgommers]

Stripping any other .so (which is actually a .dylib on macOS) reproduces the issue, and the -x flag works around it:

% strip build/scipy/linalg/_flapack.cpython-39-darwin.so
...
arm64-apple-darwin20.0.0-strip: error: symbols referenced by indirect symbol ...

% strip -x build/scipy/linalg/_flapack.cpython-39-darwin.so
strip: warning: changes being made to the file will invalidate the code signature in: /Users/rgommers/code/scipy/build/scipy/linalg/_flapack.cpython-39-darwin.so
[cctools-port]: generating fake signature for '/Users/rgommers/code/scipy/build/scipy/linalg/_flapack.cpython-39-darwin.so.strip'

IIRC we don't have to worry about code signing, that's for the install tool to do to re-sign on install. EDIT: or I could read better and see that it's re-signed by cctools straight after.

[rgommers]

Okay, that's not the problem. Using strip -x strips PyInit__ symbols, so that doesn't work either. What we do on scipy-wheels is to strip only for non-macOS builds during build, and not running strip afterwards. See MacPython/scipy-wheels#25.

Between multi-build/multibuild#140, numpy/numpy#7570 and pypa/manylinux#119 it looks like there are a lot of potential footguns here, like patchelf bugs that may or may not still exist.

tl;dr don't use -Dstrip=true.

[tacaswell]

I can confirm that this patch to meson-python lets scipy build again:

diff --git a/mesonpy/__init__.py b/mesonpy/__init__.py
index 6594812..37efb7b 100644
--- a/mesonpy/__init__.py
+++ b/mesonpy/__init__.py
@@ -487,7 +487,7 @@ class Project():
                 f'--native-file={os.fspath(self._meson_native_file)}',
                 # TODO: Allow configuring these arguments
                 '-Ddebug=false',
-                '-Dstrip=true',
+                # '-Dstrip=true',
                 '-Doptimization=2',
                 *setup_args,
             )

[tylerjereddy]

I guess we could set meson-python lower bound in pyproject.toml to whatever version gets an appropriate patch. Would that cover all the cases? I wonder if it is worth the trouble to have a separate platform_system==Darwin entry for this scenario.

I guess we don't have to block the release, but sounds like we'd want to patch it fairly swiftly when the fix is ready.