Debugging Docs and llvm.org/docs/

Project Infrastructure LLVM Dev List Archives
1

Hi -- I've recently added some documentation to the LLVM docs subdirectory, and was wondering about a couple of things:

- How should I best debug whether the docs (in RST) are syntactically correct? Are there recommended practices for doing a sanity check on the docs?
- How often do the documents get built and uploaded to the website?

Cheers

-- Dean

2

The HTML generated using sphinx[1].
At the bottom of each page you can see actual version of sphinx used to generate this page.

- How should I best debug whether the docs (in RST) are syntactically correct? Are there recommended practices for doing a sanity check on the docs?

Just run correct version of sphinx locally and check whether everything is correct.

- How often do the documents get built and uploaded to the website?

AFAIK it happens once per hour (cron-task with polling).

CC’ing Renato in case I missed something.

3

The HTML generated using sphinx[1].
At the bottom of each page you can see actual version of sphinx used to generate this page.

Thanks – interesting that the docs haven’t been updated since March 15th:

© Copyright 2003-2017, LLVM Project. Last updated on 2017-03-16. Created using Sphinx 1.4.5.

  • How should I best debug whether the docs (in RST) are syntactically correct? Are there recommended practices for doing a sanity check on the docs?

Just run correct version of sphinx locally and check whether everything is correct.

I see – nothing part of the CMake build, say “check-docs” that would do this?

  • How often do the documents get built and uploaded to the website?

AFAIK it happens once per hour (cron-task with polling).

Cool – is this cron job running on the llvm.org server itself, or somewhere else?

CC’ing Renato in case I missed something.

Thanks, Alex!

4

LLVM_ENABLE_SPHINX will turn it on.
//Use Sphinx to generate llvm documentation.
LLVM_ENABLE_SPHINX:BOOL=OFF

5

Yes, this happens when the server's sphinx installation picked up on
something that neither your installation or the buildbot's did.

I'll have a look if I can untangle that.

cheers,
--renato

PS: We really need a better way of doing this...

6

FYI, what the polling server does is:

$ cmake -DLLVM_ENABLE_DOXYGEN=ON -DLLVM_BUILD_DOCS=ON
-DCLANG_TOOLS_EXTRA_INCLUDE_DOCS=ON $SRC_DIR
$ make doxygen

This builds llvm, clang clang-extra and polly (if checked out) docs.
I'm not sure why the extra needs its own flag, though.

cheers,
--renato

7

Ignore me, that's the wrong docs build. :S

8

Just to add to that the targets you want are

docs-llvm-html
docs-llvm-man

e.g.

ninja docs-llvm-html

as Daniel mentioned you need to set `LLVM_ENABLE_SPHINX` to on.

9

Cool stuff, thanks!

– Dean

10

Cool stuff, thanks!

Oh another CMake option that is relevant is
`SPHINX_WARNINGS_AS_ERRORS`. It is on by default and causes sphinx to
treat warnings as errors.

11

Related question:

Why do we have both a) http://llvm.org/doxygen/ and b) http://llvm.org/docs/doxygen/?

Google only seems to index b), but it's not been re-generated since January 2016. Can it be deleted?

Is there a robots.txt or similar that's preventing search engines from crawling a)?

Tobias

12

Probably a configuration bug. Someone reported it here: http://llvm.org/PR32412

- Matthias

13

Thanks – interesting that the docs haven’t been updated since March 15th:

Yes, this happens when the server’s sphinx installation picked up on
something that neither your installation or the buildbot’s did.

I’ll have a look if I can untangle that.

Thanks Renato!

cheers,
–renato

PS: We really need a better way of doing this…

Has anybody looked into potentially getting the LLVM docs hosted on readthedocs.org and automatically built from the LLVM Git mirror (or through Subversion post-commit hooks)? Since we’re already using Sphinx, this seems like a low-cost, low-interference way of hosting the generated documentation.

– Dean

14

I'll have a look if I can untangle that.

Thanks Renato!

Well, that didn't last long. Now I remember why last time I fixed
hundreds of warnings on the docs directly instead of fixing the
server...

Has anybody looked into potentially getting the LLVM docs hosted on
readthedocs.org and automatically built from the LLVM Git mirror (or through
Subversion post-commit hooks)? Since we're already using Sphinx, this seems
like a low-cost, low-interference way of hosting the generated
documentation.

This is actually a very good idea. I'll let Tanya and Anton further
comment on that from a foundation's point of view.

cheers,
--renato

15

I'll have a look if I can untangle that.

Thanks Renato!

Well, that didn't last long. Now I remember why last time I fixed
hundreds of warnings on the docs directly instead of fixing the
server...

Has anybody looked into potentially getting the LLVM docs hosted on
readthedocs.org and automatically built from the LLVM Git mirror (or through
Subversion post-commit hooks)? Since we're already using Sphinx, this seems
like a low-cost, low-interference way of hosting the generated
documentation.

This is actually a very good idea. I'll let Tanya and Anton further
comment on that from a foundation's point of view.

The docs are built via SVN post-commit hook, but thats not the bigger issue in my opinion. I don’t know enough about readthedocs.org but if someone has a lot of experience.. perhaps they could summarize pros and cons of using it?

I’m not opposed to change just need to understand it. I’m actually going to be handing off most of this stuff as soon as I can write the email to the list (sorry its Spring Break here in the US so my time is limited this week). I have someone who is going to help with sys admin on llvm.org and head things up in that area but ideally its a team of people.

-Tanya

16
17

You beat me to it Brian, thank you for giving this a go!

/me just happy to see https://llvm.readthedocs.io/en/latest/XRayExample.html working out just fine. :slight_smile:

– Dean

18

That was easy! :slight_smile:

The ones I looked are fine, too. Thanks!

I think we could get them under docs.llvm.org or something, by just pointing our dns somewhere?

Cheers,
Renato

19

So, building the docs isn’t the issue I feel is the problem. The script we have works totally fine. The problem is people breaking the docs.

20

I don't know enough about the website, but there are other problems in
our infrastructure:

* We need to manually update Sphinx. People out there can have much
newer versions, which accept newer syntax, and it doesn't break on
their side, but it breaks on the server.
* We have buildbots that validate the docs, but again, it's a
completely separate machine, with a different version still (at least
potentially).
* The buildbot doesn't push its builds to the server, nor it's
guaranteed to have the same version as the actual builder, so
maintenance is hard.
* The server process doesn't warn people when it breaks. At least not
developers.

What does this website fix?

* Can it report failed builds? Or at least show on a public webpage
what's the problem?
* Can we email people when the docs are broken? At least a generic
list like llvm-admin?
* How often Sphinx is updated on the website? Is it always the most
modern version?

I think those three points are true, but I don't know for sure. If
they are, at least some of the problems are fixed.

Another solution is to make the buildbot push the docs somewhere, so
at least we have a consistent process, and whatever happens on the
public bot, happens on the docs.

But that seems more involved and problematic (SSH/FTP keys, etc.),
which may defeat our "move to public infrastructure to avoid costs"
trend.

In that sense, this website is somewhat similar to hosting our code on
GitHub. It's someone else's problem.

cheers,
--renato