Add a separate mode to parse footnotes the same way GitHub does #654

notriddle · 2023-06-01T21:58:22Z

Resolves #20

Resolves #530

Resolves #623

This change is similar to, but a more limited change than, #544. It changes the syntax, but does not touch the generated HTML or event API.

Motivation

This commit is written with usage in mdBook, rustdoc, and docs.rs in mind.

Having a standard to follow, or at least a public test suite in cmark-gfm ¹, makes it easier to distinguish bugs from features.
It makes sense to commit to following GitHub's behavior specifically, because mdBook chapters and docs.rs README files are often viewed in GitHub preview windows, so any divergence will be very annoying.
If mdBook and docs.rs are going to use this syntax, then rustdoc should, too.
Having both footnote syntaxes use the same API and rendering makes it more feasible for rustdoc to change the syntax over an edition. To introduce a syntax change in a new edition of Rust, we must make rustdoc warn anyone who writes code that will have its meaning change. To do it, run the parser twice in lockstep (with ENABLE_FOOTNOTES on one parser, and ENABLE_GFM_FOOTNOTES on the other), and warn if they diverge.
- Alternatively, run a Crater build with this same code to check if this actually causes widespread breakage.
In particular, using tree rewriting to push the footnotes to the end is not as useful as it sounds, since that's not enough to exactly copy the way GitHub renders footnotes. To do that, you also need to sort the footnotes by the order in which they are referenced, not the order in which they are defined. This type of tree rewriting is also a waste of time if you want "margin note" rendering instead of putting them all at the end.

cmark-gfm is under the MIT license, so incorporating parts of its test suite into pulldown-cmark should be fine. ↩

Resolves pulldown-cmark#20 Resolves pulldown-cmark#530 Resolves pulldown-cmark#623 This change is similar to, but a more limited change than, <pulldown-cmark#544>. It changes the syntax, but does not touch the generated HTML or event API. Motivation ---------- This commit is written with usage in mdBook, rustdoc, and docs.rs in mind. * Having a standard to follow, or at least a public test suite in [cmark-gfm] [^c], makes it easier to distinguish bugs from features. * It makes sense to commit to following GitHub's behavior specifically, because mdBook chapters and docs.rs README files are often viewed in GitHub preview windows, so any divergence will be very annoying. * If mdBook and docs.rs are going to use this syntax, then rustdoc should, too. * Having both footnote syntaxes use the same API and rendering makes it more feasible for rustdoc to change the syntax over an [edition]. To introduce a syntax change in a new edition of Rust, we must make rustdoc warn anyone who writes code that will have its meaning change. To do it, run the parser twice in lockstep (with `ENABLE_FOOTNOTES` on one parser, and `ENABLE_GFM_FOOTNOTES` on the other), and warn if they diverge. * Alternatively, run a Crater build with this same code to check if this actually causes widespread breakage. * In particular, using tree rewriting to push the footnotes to the end is not as useful as it sounds, since that's not enough to exactly copy the way GitHub renders footnotes. To do that, you also need to sort the footnotes by the order in which they are *referenced*, not the order in which they are defined. This type of tree rewriting is also a waste of time if you want "margin note" rendering instead of putting them all at the end. [cmark-gfm]: https://github.com/github/cmark-gfm/blob/1e230827a584ebc9938c3eadc5059c55ef3c9abf/test/extensions.txt#L702 [edition]: https://doc.rust-lang.org/edition-guide/editions/index.html [^c]: cmark-gfm is under the MIT license, so incorporating parts of its test suite into pulldown-cmark should be fine.

Martin1887 · 2023-06-03T09:17:14Z

Thanks for your contribution, I have to carefully review this and the other related pull requests before merging, as well as to check the possible syntaxes.

It could be included in the 0.10 version but maybe it will not.

Martin1887 · 2023-06-04T10:01:31Z

specs/gfm_footnotes.txt

This spec should be placed in third_party/GitHub

The spec didn't actually come from GitHub themselves, though. It was mostly generated by throwing stuff at GitHub to see what it does (plus looking at some of their test cases, though they weren't really exhaustive).

OK, I thought it came from

cmark-gfm is under the MIT license, so incorporating parts of its test suite into pulldown-cmark should be fine.

Am I missing anything or something prevents using that test suite?

Thanks for the changes.

Am I missing anything or something prevents using that test suite?

We can't use it verbatim, because the generated HTML format is a little different.

The cmark-gfm test suite includes about five test cases for footnotes. I incorporated them into our spec, but also wrote a few dozen more tests for cases that theirs didn't cover.

OK, thanks for the clarification.

Martin1887 · 2023-06-04T10:09:41Z

build.rs

Why using the spec filename here instead of using a new type of example as with other extensions (example_metadata_blocks for instance)?

Doing everything in the same way increases consistency and maintenance.

Martin1887 · 2023-06-04T10:10:25Z

specs/gfm_footnotes.txt

Examples should be marked as example_gfm_footnotes as commented in build.rs.

Taking into account the comments of lib.rs, these should be the normal example examples and old-style footnotes should be example_footnotes_old_style or something similar.

Martin1887 · 2023-06-04T10:11:14Z

tests/lib.rs

@@ -14,14 +14,18 @@ use tendril::stream::TendrilSink;
 mod suite;

 #[inline(never)]
-pub fn test_markdown_html(input: &str, output: &str, smart_punct: bool, metadata_blocks: bool) {
+pub fn test_markdown_html(input: &str, output: &str, smart_punct: bool, metadata_blocks: bool, is_gfm: bool) {


There are several features related to GFM, I would name this gfm_footnotes.

Martin1887 · 2023-06-04T10:35:49Z

src/lib.rs

@@ -358,5 +359,13 @@ bitflags::bitflags! {
        /// - `+++` line at start
        /// - `+++` line at end
        const ENABLE_PLUSES_DELIMITED_METADATA_BLOCKS = 1 << 8;
+        /// GitHub-compatible footnote syntax. Mutually-exclusive with `ENABLE_FOOTNOTES`.
+        const ENABLE_GFM_FOOTNOTES = 1 << 9;


I think that nothing prevents here from enabling both options, which is incorrect. This is a special case because there are two different options for the same thing with small differences.

Pulldown-cmark adheres to CommonMark and GFM extensions, so I think the best solution here is maintaining the ENABLE_FOOTNOTES flag enabling the new behaviour and a new FOOTNOTES_OLD_STYLE or something similar to enable the old behaviour not fully compatible with GFM footnotes (note that both flags must be enabled to support old-style footnotes).

This supposes a breaking change, but 0.10 release contains some of them, so I would prioritize GFM adhesion against not breaking changes here.

What do you think, @raphlinus?

Martin1887 · 2023-06-04T10:39:47Z

src/lib.rs

To actually solving #530, the footnotes documentation should be pointed here.

I think the best solution is putting a link in the docstring and adding a file describing the specification of old-style footnotes and the differences with the new fully compatible GFM footnotes, also indicating the differences with GitHub behaviour if any (missing footnotes backlinks, footnotes order, ...).

Martin1887

Changes should also be done in main.rs to use the new footnotes option.

* `ENABLE_GFM_FOOTNOTES` is now `ENABLE_FOOTNOTES`, and the old mode is now `ENABLE_OLD_FOOTNOTES`. * The spec test case mode tweaks are now done using example tags, just like every other test case. * Instead of declaring it not-allowed to enable both modes, the `ENABLE_OLD_FOOTNOTES` mode now *implies* the regular `ENABLE_FOOTNOTES`. * `main.rs` now has an option to `--enable-old-footnotes`.

…-syntax/#footnotes

Martin1887 · 2023-06-05T16:46:55Z

Everything seems right now, I will do some checks tomorrow and then it will be merged. Thanks for your work and fast responses!

chriskrycho · 2023-06-06T15:21:40Z

For whatever it’s worth, I was revisiting my own incomplete implementation (and also mucking with markdown-rs) and concluded the same about the tree rewriting: You should not rewrite the source, but should make sure you capture enough info the drive a correct output from the event sequence (whether it’s emitting an AST for going straight to HTML), and preserving source order rather than rewrite allows whatever mix of styles of note you want. 👍

The code here was introduced in pulldown-cmark#654 and the crash was found using cargo fuzz run parse Fixes pulldown-cmark#666.

notriddle force-pushed the notriddle/gfm-footnotes branch from 33f63c1 to 0f84d7e Compare June 2, 2023 23:28

notriddle force-pushed the notriddle/gfm-footnotes branch from 0f84d7e to 3d184fb Compare June 2, 2023 23:28

Martin1887 requested changes Jun 4, 2023

View reviewed changes

Martin1887 reviewed Jun 4, 2023

View reviewed changes

Martin1887 mentioned this pull request Jun 4, 2023

[WIP]: support “standard” footnotes #544

Closed

3 tasks

notriddle added 3 commits June 4, 2023 16:46

Add more docs to ENABLE(_OLD)_FOOTNOTES

27db86e

Add another test case based on https://www.markdownguide.org/extended…

ce7538a

…-syntax/#footnotes

Martin1887 approved these changes Jun 5, 2023

View reviewed changes

Remove unuseful gfm_footnotes test file

bdc1017

Martin1887 merged commit 250799d into pulldown-cmark:master Jun 6, 2023

Martin1887 mentioned this pull request Jun 6, 2023

Indented footnote definitions confuse the parser #623

Closed

notriddle deleted the notriddle/gfm-footnotes branch June 6, 2023 15:12

mgeisler mentioned this pull request Jun 17, 2023

Index out of bounds at firstpass.rs:1424 #666

Closed

mgeisler added a commit to mgeisler/pulldown-cmark that referenced this pull request Jun 17, 2023

Fix index out of bounds when parsing GFM footnote

01b6446

The code here was introduced in pulldown-cmark#654 and the crash was found using cargo fuzz run parse Fixes pulldown-cmark#666.

mgeisler mentioned this pull request Jun 17, 2023

Fix index out of bounds when parsing GFM footnote #667

Merged

si14 mentioned this pull request Sep 11, 2023

Markdown footnotes enhancement getzola/zola#1285

Closed

DerDrodt mentioned this pull request Oct 27, 2023

GitHub-Style Footnotes getzola/zola#2326

Closed

Martin1887 mentioned this pull request Jan 30, 2024

Set range end to Paragraph if returned early because of footnote definition #831

Merged

notriddle mentioned this pull request Mar 12, 2024

rustdoc: check parsing diffs between pulldown-cmark 0.9.6 and 0.10 rust-lang/rust#121659

Closed

zoni mentioned this pull request Aug 11, 2024

[regression in 0.10] Consecutive footnote refs fail to parse correctly #927

Closed

Add a separate mode to parse footnotes the same way GitHub does #654

Add a separate mode to parse footnotes the same way GitHub does #654

Uh oh!

Conversation

notriddle commented Jun 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Footnotes

Uh oh!

Martin1887 commented Jun 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

notriddle Jun 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Martin1887 Jun 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Martin1887 Jun 4, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Martin1887 left a comment

Choose a reason for hiding this comment

Uh oh!

Martin1887 commented Jun 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chriskrycho commented Jun 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

notriddle commented Jun 1, 2023 •

edited

Loading

Martin1887 commented Jun 3, 2023 •

edited

Loading

notriddle Jun 5, 2023 •

edited

Loading

Martin1887 Jun 5, 2023 •

edited

Loading

Martin1887 Jun 4, 2023 •

edited

Loading

Martin1887 commented Jun 5, 2023 •

edited

Loading

chriskrycho commented Jun 6, 2023 •

edited

Loading