Move contents of library reference into individual modules.

[joshayers]

The library reference documentation currently exists in doc/source/libref.rst. The majority of the contents of that file is duplicated in the docstrings of the individual classes. It should be consolidated into one place to ease future maintenance.

Also, the library reference should be split over several pages to make it easier to navigate.

[joshayers]

The numpydoc Attributes and Methods lists don't work because the __str__ method of PyTables/doc/sphinxext/docscrape_sphinx.SphinxDocString has several lines commented out. This is just a note to myself so I can find the offending lines again.

[joshayers]

I've been continuing to work on this, and I wanted to give a status and hopefully solicit some feedback to see if you agree with the path I've gone down.

I spent quite a bit of time attempting to get numpydoc to work with the PyTables documentation, and eventually concluded it's not an ideal solution. It expects the markup to be in a very specific format which is rather different from that used in PyTables. Also, even when I gave it markup in its expected format, I was unable to get it to produce the correct html output. I made some modifications to it, but was never able to get it to do what I wanted.

My next approach was to disable numpydoc, move all the libref.rst documentation into module-level doc strings, and use automodule directives to extract them. This produces good documentation output, but also has drawbacks. If the corresponding docstrings are removed from the classes and functions - and eliminating duplicate documentation was the motivating goal of this exercise - then the context-sensitive help offered by many IDEs will no longer work. I personally rely on that, and I'm sure many other users do as well, so I don't think it's an acceptable solution.

Moving the same markup to the class/function docstrings and using autoclass doesn't work well either. Sphinx chokes when it finds section headings in a class docstring. It's unclear to me why it handles sections correctly in module docstrings, but not class docstrings. I attempted to tweak the markup using a custom extension, but nothing I did allowed sections in class docstrings to be recognized correctly.

As an alternative to section headings, I'm currently using rubric directives in the reST markup. I wrote a simple Sphinx extension (pytables_doc.py) that converts section headings, when underlined by tildes, into rubric directives. To go along with this conversion, I modified the altered_nature CSS file such that rubric directives are rendered in the same font size and color as an h3 section heading. This makes the output look almost the same as the current documentation.

The extension will also convert a Parameters line, underlined by tildes, into a field list. This modification is performed by numpydoc in the current version. It gives a nicely formatted list of function or method input values.

I created several tests in the doc/sphinxext/pytables_doc.py file that show how the markup is converted.

The one downside to this approach is that sub-section headings - for example File class -> File methods - file handling - no longer appear in the table of contents. I'm not sure if that's considered a major problem. Personally, I think splitting the library reference into multiple pages (which is still something I plan to do) will make up for any loss of readibility due to missing subsection links in the table of contents.

If you're interested in seeing how it looks, the most recent version is available in my fork, in the issue148 branch. Look at the "library reference" page, but please only pay attention to the top-level, File, and Node class sections. The other sections are currently in the previous format.

There are currently a lot of errors about bad references when the docs are built, but that is just in sections that need to be converted to the new format, so please ignore them - they will be fixed.

Please let me know if you agree with this approach and have any suggestions or other feedback.

[scopatz]

So it seems to me @joshayers that you are are re-implementing parts of numpydoc that you like / seem to work. Is the problem that the current docstrings are just not conducive to numpydocs? I, like you, am against eliminating the docstrings entirely. However, I think that the ideal solution would be to fix the docstrings to work with numpydoc rather than supporting our own replacement for it. I would be willing to help with this effort. Are you working on your fork? Or is there some other place I could jump in?

Also, I have been thinking about updating the sphinx theme too, but that should probably wait until we figure out this issue.

Also also, thanks a million for jumping into this issue! It is really appreciated.

[joshayers]

@scopatz My pleasure! I hope we're able to come up with a solution that works well.

Basically, I was unable to get numpydoc to work well, even when I rewrote some of the docstrings per NumPy's guide [1]. It worked okay for showing the "Parameters" and some other fields, but I couldn't get the "Methods" and "Attributes" sections to look reasonably good. It also relies on the "autosummary" extension, which I was unable to get working correctly - it gives a large number of errors when building the documentation.

It's possible that I have something set up wrong. I just created another branch off of my fork - numpydoc_tests. In it, I made a new .rst file - doc/source/usersguide/numpydoc_tests.rst. It uses an autoclass directive to create a docstring from a new file tables/dummy.py. I think it will be easier to run tests on a small section of the documentation like this, since Sphinx can get confused if the section headings are inconsistent in a larger file. Maybe you can play with the formatting of it and see if you can come up with something that works well and that you're happy with?

If you're able to get numpydoc working, then I'd be happy to reformat things as needed. Whatever format we come up with, I recommend starting with the current state of my issue148 branch. In addition to the format changes, I've made a lot of other minor improvements to the documentation - fixing broken links, making formatting more consistent, etc.

For reference, I'm using Sphinx 1.1.3.

Also, agreed on updating the theme at some point. I'm particularly not a fan of the red text/pink background in the field lists. :)

[1] https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt

[scopatz]

Great thanks! I'll take a look at your branches sometime this weekend. Tonight if I am lucky, tomorrow if I am not ;)

[scopatz]

Hey @joshayers, fixing to make numpydoc look good is actually pretty minor (though subtle). I have pushed up the changes to the numpydoc_tests branch of my fork (which you can pull from). Basically, you only need to do two things.

1. remove sphinx.ext.autosummary from the extensions list in doc/source/conf.py
2. Comment out lines 191 - 192 as follows in doc/sphinxext/docscrape_sphinx.py:

        out += self._str_references()
        out += self._str_examples()
        #for param_list in ('Attributes', 'Methods'):
        #    out += self._str_member_list(param_list)
        out = self._str_indent(out, indent)
        return '\n'.join(out)

Doing this makes everything look just fine and avoids the sphinx autosummary issue. Alternatively, if you really wanted custom behavior for methods and attributes, you could just re-implement the _str_member_list() method in this file.

I hope this helps! From here, we can make the actual theme look a lot better too.

[joshayers]

Cool...this does look okay and builds without any errors.

Personally, I like the way the current docs delineate well the "Methods", "Special Methods", and other sections within each class, and I was trying to maintain something like that. With this setup, they all sort of run together, and there's no clear separation between attributes and methods, except that one lists arguments and the other doesn't. There's also the "= None" that gets appended to each attribute. That seems to me to be a bug in Sphinx, or else a very odd design choice. I may look into writing a simple extension to strip it off.

Of course, the advantage of this approach is that the method and attribute related documentation can be removed from the class docstrings, which will make them much shorter. So I think this becomes something of a trade-off between easier to read and maintain docstrings, vs slightly less readable (IMO) documentation output.

I'm okay with that trade-off if you are. I'll go back to my other branch and start updating the docstrings accorgingly. That probably won't happen until later this week though.

Possibly in the future, Sphinx will be upgraded to allow more customization of autodoc output and we can have the best of both worlds.

[scopatz]

I am glad it worked for you.

I agree that it is nice to have the "Methods" and "Special Methods" separate. There is probably a way to fake this by manually including each of the members:

.. autoclass:: tables.dummy.Dummy

     *Methods*
    ..  automethod:: method1

    *Special Method*
    .. automethod:: method2 

I am not sure that this would work and would likely be too tedious and not worth the effort. For now, I am OK with the trade off that you posed. Though I agree that more autodoc customization would be nice ;)

Also what is the = None issue? I wasn't seeing that.

[joshayers]

I'm at work and I can't build the docs here, but I'm 90% sure that each of
the attributes added by autoclass had "= None" appended to the name. So it
looked like this (using the dummy.SampleClass in numpydoc_tests).

filename = None
the filename

mode = None
the mode

Here's a Stack Overflow question about the same issue. It contains a link
to some documentation on readthedocs.org that has the same behavior.

http://stackoverflow.com/questions/9153473/sphinx-values-for-attributes-reported-as-none

http://readthedocs.org/docs/domination-game/en/latest/games.html#domination.core.GameStats

On Wed, May 9, 2012 at 9:04 AM, Anthony Scopatz <
reply@reply.github.com

wrote:

I am glad it worked for you.

I agree that it is nice to have the "Methods" and "Special Methods"
separate. There is probably a way to fake this by manually including each
of the members:

.. autoclass:: tables.dummy.Dummy

    *Methods*
   ..  automethod:: method1

   *Special Method*
   .. automethod:: method2

I am not sure that this would work and would likely be too tedious and not
worth the effort. For now, I am OK with the trade off that you posed.
Though I agree that more autodoc customization would be nice ;)

Also what is the = None issue? I wasn't seeing that.

---

Reply to this email directly or view it on GitHub:
#148 (comment)

[scopatz]

Huh, that is weird and super annoying. I wonder if you added the name of the attribute as the first line of the docstring if that would fix it. This has worked for methods for me. What I mean is something like the following:

class A(object):

    def meth1(self):
         """meth1(self, not_in_func_sig_but_in_docs)
         I am method 1.
         """

     a1 = None
     """a1
     I am attr 1
     """

Just a thought.

[joshayers]

No, unfortunately that doesn't seem to work either.

Quick question - there are a few warnings about external links only being supported with HDF5 1.8. Now that v1.6 is no longer supported, do you think it's okay to remove them?

[scopatz]

Yeah, you can get rid of those. They are not relevant anymore.

[joshayers]

I just finished converting the File class. I'm basically copying the current documentation into the docstring of each method, and then reformatting is slightly to try and follow the NumPy conventions.

It's not an insignificant amount of work, so I'd appreciate it if you could take a look at my last commit (on my issue148 branch) and give me a thumbs up before I continue, to make sure I'm not wasting effort.

Thanks!
Josh

[scopatz]

Josh! This looks amazing! Thanks!

Not that this is a big deal since they aren't used all over the place, but is it correct to assume that docscrapper isn't getting the properties of the File object correctly (ie title and rootUEP). It is not a problem to deal with it in the way that you have here since they aren't that common.

I would say that you are definitely on the right track!

[joshayers]

Great...glad you're happy with it.

I just finished updating node.py and group.py the same way. Thankfully, I discovered the emacs command fill-region-as-paragraph. That makes reformatting the docstrings to take up the entire 80 columns a lot faster than doing it manually.

To answer your question, File.rootUEP is set in the __getRootGroup method. I believe autodoc will only find attributes that are set in __init__. The File.fileno method is defined in hdf5Extension.pyx, so Sphinx can't find it either. I deleted title from the class docstring - I left that one there by mistake.