Skip to content

MAINT: signal.csd: port away from using _spectral_helper #22460

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
May 15, 2025

Conversation

DietBru
Copy link
Contributor

@DietBru DietBru commented Feb 3, 2025

This allows to mark the internal functions _spectral_helper, fft_helper and _triage_segements as legacy, since they are only used by the legacy functions spectrogram and stft now. Further changes:

  • Some small additions to the unit tests to achieve 100% coverage for csd().
  • Remove csd related functions from file _scipy_spectral_tst_shim.py
  • An explanation of the ShortTimeFFT usage was added to the csd() docstr.

Related Issues and PRs:

#12544: Close as wont fix, since only _triage_segements is affected.
#13577: Close as wont fix, since only _spectral_helper is affected . Furthermore, ShortTimeFFT supports "chunking".
#15593: Close, since only _triage_segements is changed.
#15587: Close as wont fix, since only _triage_segements is affected.
#19001: Edits relevant parts of spectral_py.py

Closes #22674 (check comment).

* This allows to mark the internal functions `_spectral_helper`, `fft_helper` and `_triage_segements` as legacy, since they are only used by the legacy functions `spectrogram` and `stft`.

* Some small additions to the unit tests to achieve 100% coverage for `csd()`.

* Remove csd related functions from file `_scipy_spectral_tst_shim.py`

* An explanation of the `ShortTimeFFT` usage was added to the `csd()` docstr.
@github-actions github-actions bot added scipy.signal maintenance Items related to regular maintenance tasks labels Feb 3, 2025
@lucascolley lucascolley changed the title MAINT: signal.csd(): Port away from using _spectral_helper MAINT: signal.csd: port away from using _spectral_helper Mar 15, 2025
DietBru added 2 commits March 22, 2025 12:02
… relationship between squared magnitude/cross spectrum and PSD/cross PSD.
… between squared magnitude/cross spectrum and PSD/cross PSD.
@DietBru
Copy link
Contributor Author

DietBru commented Mar 23, 2025

Rendered docs for commit 5b00e15:

@ev-br
Copy link
Member

ev-br commented Apr 27, 2025

It must be frustrating to have a ready PR just sit there for more than a month. Sorry about that @DietBru !
To help moving this forward, could you please give a high-level overview of this spectral analysis functionality: what is the current state and what is the direction it needs moving?

@DietBru
Copy link
Contributor Author

DietBru commented May 6, 2025

It must be frustrating to have a ready PR just sit there for more than a month. Sorry about that @DietBru ! To help moving this forward, could you please give a high-level overview of this spectral analysis functionality: what is the current state and what is the direction it needs moving?

Well, my available time for working on SciPy is also quite unevenly distributed (as you probably noted)—so I have the same problem of not being able to review PRs as much as I would like.

First to the rationale of this PR: The class ShortTimeFFT provides a superset of functionality of _spectral_helper. This illustrated by the shims provided in the file _scipy_spectral_test_shim.py. The aim of this PR is port all non-legacy functions away from _spectral_helper, since it has a couple of bugs (as shown above). It would reduce the amount code and make porting to the Array API easier. A next step would be discussing, which legacy functions could be deprecated (i.e, spectrogram, stft and istft).

Concerning the spectral analysis functionality, I'll just discuss calculating (cross-) spectral power density (PSD) aspects here: From an algorithmic view, the signal module is not in bad shape. Using them properly is not straightforward though, since the choice of parameters can change the underlying modelling assumptions significantly. Hence, I think the main challenge lies in providing useful guidance in the documentation.

To elaborate: The classical Welch algorithm view assumes a stationary stochastic process and provides little insights on choosing window and overlap. This PR illustrates (again) that the Welch algorithm is equivalent to determining the frequency marginal of a ShortTimeFFT spectrogram (assuming the window is correctly scaled). The introduction of dual windows for the ShortTimeFFT allows to interpret the STFT as a deterministic signal decomposition into a non-orthogonal over-complete series expansion (aka frame). Hence, the Welch algorithm calculates a statistical property of that series expansion, which, obviously, depends on the choice of (dual) window. This leads to the very unsatisfying conclusion that the PSD inherently depends on the choice of parameters.

For many practical applications, it is sufficient to calculate a reasonable PSD, which allows comparing signals to each other—hence the normal heuristics suffice. On the other hand, when interpreting spectral cross-densities, the shape of the dual window isn't always negligible. Unfortunately, I am not aware of any literature, which discusses such problems for practical applications. A chapter or two in a textbook is probably more suited for discussing these kind of modeling problems than the SciPy documentation. Sorry, for not being able to provide a more useful answer here.

@larsoner
Copy link
Member

larsoner commented May 6, 2025

@ev-br let me know if that's enough for you to look. If not, I can do it (and sorry for not looking/noticing this PR earlier!)

@ev-br
Copy link
Member

ev-br commented May 6, 2025

Thanks @DietBru! This is a really helpful summary. I'm not claiming detailed knowledge or understanding of details here, and I'm not going to ask you for a couple of chapters worth of textbook for the SciPy docs :-). My impression is that the direction makes total sense, the PR is mostly maintenance + doc improvements with no hard backwards compat breaks and a clear path forward.
So from my POV this PR is good to go in as is.

That said, if you want to take a closer look @larsoner, I happily leave it all to you.

@lucascolley lucascolley added this to the 1.16.0 milestone May 6, 2025
Copy link
Contributor

@tylerjereddy tylerjereddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I'm on the same page as Evgeni here--in favor of moving forward with this PR before we branch.

Also not a domain expert, but added a few comments in line.

Rebased version of this branch is still passing full suite locally.


If `noverlap` is 0, this method is equivalent to Bartlett's method
[2]_.
The ratio of the squared magnitude (``scaling='spectrum'``) divided the spectral
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

divided by?

Copy link
Member

@larsoner larsoner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM after addressing @tylerjereddy's comments!

@DietBru
Copy link
Contributor Author

DietBru commented May 15, 2025

Thanks for the input, @tylerjereddy. I addressed your remarks in commit 11c0c45.

@tylerjereddy
Copy link
Contributor

Looks like my docs-related comments were addressed and there are three core devs in favor of a merge so in it goes, thanks.

@tylerjereddy tylerjereddy merged commit 33b422c into scipy:main May 15, 2025
36 checks passed
DietBru added a commit to DietBru/scipy that referenced this pull request May 16, 2025
* Remove phrase "(consult the code snippet above)."
* Use `csd` instead of `csd()` everywhere in docstr.
* Follow up to PR scipy#22460.

[docs only]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance Items related to regular maintenance tasks scipy.signal
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants