Andrew/test coverage by ayjayt · Pull Request #397 · plotly/Kaleido

ayjayt · 2025-08-27T19:06:17Z

This PR implements testing for the entire public API.

It also fixes all the bugs found by that testing.

Bugs Fixed:

Allows users to pass file paths for plotlyjs/mathjax in url and path form
Explicitly handle mathjax=True
Better handle instances where users pass Path instead of str: (casting and isinstance flexibility)
close browser before closing kaleido, prevents hanging in certain instances
use named arguments in Kaleido constructor instead of generic kwargs (smell fix)
create temporary file during .open() given that it requires close() (prevents leak in instances where user doesn't .open(), real corner case)
properly check for an error log before trying to append to it.
deal with corner case where asyncio.return_task() doesn't return at ask

These were all bugs found from doing tests using the public API as its expected to be used.

Also, typing was added, a private attribute was changed to public to aid testing, and tests are added.

emilykl · 2025-08-28T19:11:15Z

src/py/tests/test_page_generator.py

+    if not find_spec("plotly"):
+        pytest.skip("Plotly not available - cannot test force_cdn override")


Shouldn't this test only run when plotly is available for import?

Yes, here it says if it can't find plotly, it skips the test. Or "only run when plotly is available"

Yeah what I mean is that I don't like the usage of if (condition): pytest.skip() because it could lead to essentially silently skipping this test entirely if Plotly is never available when the test is run.

Can't this if check be removed entirely?

Well, no, I'm happy to change it to a fail, I think that makes sense, but w/o plotly present, this test doesn't accomplish its goal which making sure "force_cdn" actually overrides plotly.py. We have another test to make sure that the CDN is also the default.

Yeah I think changing it to a fail would be better (as long as that doesn't break the test)

Okay, but I did it in a later PR

emilykl · 2025-08-28T19:19:25Z

src/py/tests/test_page_generator.py

+
+# Test user overrides
+@settings(suppress_health_check=[HealthCheck.function_scoped_fixture])
+@given(st.data())


How does @given and data.draw() work? Is it testing all possible values returned from st_valid_path, or a random subset?

data.draw() is basically what hypothesis does normally, it just lets you do it during the test instead of during hypothesis setup. So like all hypothesis testing, it does a random sample of default value n=100.

the st.data() + data.draw() strategy allows you to do the sampling at runtime during your test function instead of during test setup.. I was having trouble with windows not recognizing Path(__file__) as a valid file (according to chat GPT, it can happen), and wanted to use the builtin-in tmp_path fixture (pretty robust) which is only available at runtime.

emilykl · 2025-08-28T19:26:07Z

src/py/tests/test_page_generator.py

+    # Test with regular path
+    with pytest.raises(FileNotFoundError):
+        PageGenerator(plotly=str(nonexistent_file_path))


Should test the case where a Path is passed as well, right? For this test and the others.

Suggested change

# Test with regular path

with pytest.raises(FileNotFoundError):

PageGenerator(plotly=str(nonexistent_file_path))

# Test with Path object

with pytest.raises(FileNotFoundError):

PageGenerator(plotly=nonexistent_file_path)

# Test with path as string

with pytest.raises(FileNotFoundError):

PageGenerator(plotly=str(nonexistent_file_path))

this found an error btw

emilykl · 2025-08-28T19:28:21Z

src/py/tests/test_public_api.py

+    with pytest.warns(RuntimeWarning, match="already"):
+        kaleido.start_sync_server(silence_warnings=False)
+
+    kaleido.start_sync_server(silence_warnings=True)


We need to explicitly test that no warning is emitted here, right?

Or have you set the Pytest configuration such that a warning will trigger a test failure?

Yes, all warnings are upgraded to errors via CLI options.

emilykl · 2025-08-28T19:28:47Z

src/py/tests/test_public_api.py

+    with pytest.warns(RuntimeWarning, match="closed"):
+        kaleido.stop_sync_server(silence_warnings=False)
+
+    kaleido.stop_sync_server(silence_warnings=True)


Likewise here

emilykl · 2025-08-28T22:07:36Z

src/py/tests/test_page_generator.py

+
+    _h_url = st.tuples(
+        st.sampled_from(["s", ""]),
+        st.text(


might be misunderstanding here since I'm not familiar with hypothesis, but the number of possible values that could be sampled here (character strings from 1-20 characters) is huge. Could that space dwarf the other sampling being done here, and reduce the variety of the test cases? I'm not sure if that makes sense, I'm also not sure how hypothesis sampling logic works.

This is an extremely interesting question and I don't have a better response than "hypothesis is said to handle these things pretty reasonably, especially ensuring to try a good range of combinations and corner cases" which I take from AI.

I'm under the impression that In this particular case, everytime text is sampled, all the other options are sampled both independently and intentionally stratified to test all corner cases.

I've been going back and forth on this, and no, I don't think the range of values makes it more favored in sampling. I think actually for continuous ranges hypothesis uses at least a statistical representation of the possible values (min, max, etc)

emilykl

🚀

ayjayt added 30 commits August 21, 2025 13:57

Make tabs_ready public

41133f2

Add some types to kaleido.py

d4a5bd4

Merge branch 'master' into andrew/test_coverage

0cbd7c0

Merge branch 'master' into andrew/sometypes

196aca7

Merge branch 'andrew/sometypes' into andrew/test_coverage

4b1ea3c

Upgrade choreographer for typing.

583b8c8

Add types to __init__

d36dbab

Fix some bad typing in __init__.py

9838ac4

Improve page logic.

7d4d297

Merge branch 'andrew/sometypes' into andrew/test_coverage

5719383

Add claude run on init and public api

cbd925b

Move pytest async detection to auto

d99bb88

Iterate with Claude on test_public_api

6d3cf13

Make corrections to public api tests.

e4ba49c

Fix up test_init

1898bd9

Add comments.

4147850

Reorganize a bit.

7a808b4

Make error_log/profiler 100% optional in __init__

abcc184

Iterate with claude on public_api test.

24d0dab

Parameterize public api tests for fig/to_dict()

f5cc483

Change to using args fixtures.

9489b71

Remove regex for html parsing.

824230c

Fix non-existent function

d152afa

Fix whitespace errors.

ec62908

Add types for page_generaetor.

930f338

Merge branch 'andrew/sometypes' into andrew/test_coverage

3ddb876

Add hypothesis tests generated by Claude.

f9175fb

Fix file path ensuring error.

2b20a3e

Add hypothesis.

a71a202

Allow _ensure_path to deal with str/encoding combos

63332ca

ayjayt added 7 commits August 28, 2025 12:06

Be more explicit in path parsing.

cfb4eff

Add missing function.

f9e7f64

Add logging.

69294ed

Add yet more logging.

5407965

Fix bad logic.

33ead6d

Tone down unreasonable 20 processor test.

449282b

Remove hypo deadlines for slow CI runners.

62fcefe

emilykl reviewed Aug 28, 2025

View reviewed changes

ayjayt marked this pull request as ready for review August 28, 2025 21:55

Organize a bit conftest.py

501fb80

emilykl reviewed Aug 28, 2025

View reviewed changes

ayjayt added 3 commits August 29, 2025 13:26

Clear up mathjax logic.

b8f7185

Test Path() as well as str() in filenotfound tests.

970f8b8

Fix to properly validate Path() types

7c70737

gvwilson assigned emilykl and ayjayt Sep 9, 2025

gvwilson added feature something new P2 needed for current cycle testing automated tests P1 needs immediate attention and removed P2 needed for current cycle labels Sep 9, 2025

emilykl approved these changes Sep 9, 2025

View reviewed changes

ayjayt merged commit e0d5907 into master Sep 10, 2025
4 checks passed

ayjayt deleted the andrew/test_coverage branch September 15, 2025 20:14

This was referenced Oct 27, 2025

fix: move pytest-timeout to dev-dependency #415

Open

Unexpected installation of doc and tests with Kaleido ≥1.0 #414

Open

		if not find_spec("plotly"):
		pytest.skip("Plotly not available - cannot test force_cdn override")

Conversation

ayjayt commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ayjayt Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ayjayt Aug 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

emilykl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ayjayt commented Aug 27, 2025 •

edited

Loading

ayjayt Aug 28, 2025 •

edited

Loading

ayjayt Aug 31, 2025 •

edited

Loading