MAINT: sparse: add lazy loading for csgraph and linalg

[j-bowhay]

Reference issue

closes #22382

What does this implement/fix?

Adds lazy loading for the two submodules in order to speed up import times.

Additional information

[dschult]

This change looks like what we want.

• csgraph will no longer be imported at the time sparse is imported; only when sparse.csgraph` is accessed.
• linalg will be auto-imported when accessed (it currently is not imported with sparse)

I'm not sure about the details of tach.toml. Why does linalg stay in the "depends_on" for scipy.sparse? Does something in sparse depends on linalg? If so does linalg get imported when importing sparse when using "tach"?

[j-bowhay]

This change looks like what we want.

• csgraph will no longer be imported at the time sparse is imported; only when sparse.csgraph` is accessed.
• linalg will be auto-imported when accessed (it currently is not imported with sparse)

I'm not sure about the details of tach.toml. Why does linalg stay in the "depends_on" for scipy.sparse? Does something in sparse depends on linalg? If so does linalg get imported when importing sparse when using "tach"?

For whatever reason tach didn't like the style of one of the imports. I tweaked it slightly which now allows us to remove both csgraph and linalg from the tach.toml.

[dschult]

That's a nice way to import matrix_power and allow lazy loading. Thanks!

This looks good to me!

[lucascolley]

For whatever reason tach didn't like the style of one of the imports. I tweaked it slightly which now allows us to remove both csgraph and linalg from the tach.toml.

I would suggest reverting that change. It is true that sparse depends on sparse.linalg for spmatrix.matrix_power, even though the dependency is not at import-time.

One could argue that keeping track of import-time dependencies only is more useful than dependencies in general, but I think at this stage it is helpful to be explicit about potential runtime gotchas.

EDIT: tach itself is not just about import-time deps (hence the initial failures until you tweaked the style), but rather modularity more broadly, so I think that change is more of a circumvention of tach's detection than a sort of 'fix'

[j-bowhay]

For whatever reason tach didn't like the style of one of the imports. I tweaked it slightly which now allows us to remove both csgraph and linalg from the tach.toml.

I would suggest reverting that change. It is true that sparse depends on sparse.linalg for spmatrix.matrix_power, even though the dependency is not at import-time.

One could argue that keeping track of import-time dependencies only is more useful than dependencies in general, but I think at this stage it is helpful to be explicit about potential runtime gotchas.

I'm not sure I entirely agree. If we also use also track runtime dependencies then it looses its "sharpness" as I could add sparse.linalg as an import-time dependency without complaint. The original complaint was about import-time so I think we should focus on that.

[lucascolley]

The original complaint was about import-time so I think we should focus on that.

Sure, but I could imagine a similar issue about runtime instead, so I don't think either way is favoured. Hence I think explicit rather than implicit is better.

If we also use also track runtime dependencies then it looses its "sharpness" as I could add sparse.linalg as an import-time dependency without complaint

There should be other ways to achieve this (or better) sharpness. Reading the docs, it looks like we could add something like:

[[interfaces]]
expose = ["matrix_power"]
from = ["scipy.sparse.linalg"]
visibility = ["scipy.sparse"]
exclusive = true

I'm running into some errors when trying to get recent versions of tach to run locally, though

[lucascolley]

I don't mean to block the PR as tach-bikeshedding is insignificant compared to the import performance improvement, but as another data-point, Ralf was against this pattern at #22412 (comment)

[lucascolley]

thanks Jake, and for the review Dan!