Skip to content

DEV: Add nbval to CI. #132

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Jul 16, 2022
Merged

DEV: Add nbval to CI. #132

merged 3 commits into from
Jul 16, 2022

Conversation

rossbar
Copy link
Collaborator

@rossbar rossbar commented May 17, 2022

Closes #51 .

Add a basic nbval workflow to the CI, borrowing heavily from @MridulS 's recipe in nx-guides. Currently only smoketests the notebooks with pytest --nbval-lax. More rigorous testing (i.e. comparison with saved outputs) can always be explored in followup PRs.

A couple of potential improvements:

  1. Find a better way of programmatically excluding certain notebooks from the test suite (see TODO), preferably via conftest.py or similar mechanism
  2. Add data caching to the workflow
  3. Add a pytest and nbval as testing requirements and update the contributor guide with a quick note on how to run locally.

Co-authored-by: Mridul Seth <[email protected]>
strategy:
max-parallel: 12
matrix:
os: [Ubuntu-20.04, macOS-latest]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reasons to not add windows, too?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only because I expected it would require special windows-specific handling to make it work... I just tried it and it seems that's the case, see: https://github.com/numpy/numpy-tutorials/runs/7309090446?check_suite_focus=true

My vote is for incremental adoption: get linux/macos in sooner since they're working now, and open a follow-up issue for anyone who wants to figure out what magic combo of workarounds is necessary to get it to work on windows :)

@bsipocz
Copy link
Member

bsipocz commented Jun 10, 2022

I'm still not 100% happy with my CI solution here: https://github.com/NASA-NAVO/navo-workshop, but I think adding tox helps a lot with point no3 in your list, and it also helps to manage and run the same command through the different CI providers and by contributors.

@rossbar
Copy link
Collaborator Author

rossbar commented Jul 12, 2022

I'm +1 for a tox.ini (or whatever testing config file) as well. I'd also vote that for that to be done in a follow-up PR so that it's self-contained, but if you'd like to add it here please feel free to do so!

@bsipocz
Copy link
Member

bsipocz commented Jul 13, 2022

@rossbar - it's your PR, so whatever works for you sounds good. I can put doing some incremental stuff for this as part of my scipy sprint, but no promises that I actually manage to go through that list.

@rossbar
Copy link
Collaborator Author

rossbar commented Jul 16, 2022

I'm going to go ahead and put this in as-is to get the additional testing in place. I'll open issues for the follow-up tasks that were identified above. Thanks @bsipocz !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

CI: sphinx-build doesn't fail on notebook execution error.
2 participants