Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1350219 > unrolled thread

Re: Kernel docs: muddying the waters a bit

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-04 14:10 +0100
    Re: Kernel docs: muddying the waters a bit Johannes Stezenbach <js@linuxtv.org> - 2016-03-04 15:10 +0100
      Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-06 03:40 +0100
        Re: Kernel docs: muddying the waters a bit Johannes Stezenbach <js@linuxtv.org> - 2016-03-07 09:50 +0100
          Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-07 13:20 +0100
        Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-07 13:20 +0100

#1350219 — Re: Kernel docs: muddying the waters a bit

Em Fri, 04 Mar 2016 10:29:08 +0200
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Fri, 04 Mar 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
> > Em Thu, 03 Mar 2016 15:23:23 -0800
> > Keith Packard <keithp@keithp.com> escreveu:
> >  
> >> Mauro Carvalho Chehab <mchehab@osg.samsung.com> writes:
> >>   
> >> > On my tests, Sphinix seemed too limited to format tables. Asciidoc
> >> > produced an output that worked better.    
> >> 
> >> Yes, asciidoc has much more flexibility in table formatting, including
> >> the ability to control text layout within cells and full control over
> >> borders.
> >> 
> >> However, I think asciidoc has two serious problems:
> >> 
> >>   1) the python version (asciidoc) appears to have been abandoned in
> >>      favor of the ruby version. 
> >> 
> >>   2) It really is just a docbook pre-processor. Native html/latex output
> >>      is poorly supported at best, and exposes only a small subset of the
> >>      full capabilities of the input language.
> >> 
> >> As such, we would have to commit to using the ruby version and either
> >> committing to fixing the native html output backend or continuing to use
> >> the rest of the docbook toolchain.
> >> 
> >> We could insist on using the python version, of course. I spent a bit of
> >> time hacking that up to add 'real' support for a table-of-contents in
> >> the native HTML backend and it looks like getting those changes
> >> upstreamed would be reasonably straightforward. However, we'd end up
> >> 'owning' the code, and I'm not sure we want to.  
> >
> > I'm a way more concerned about using a tool that fulfill our needs
> > than to look for something that won't use the docbook toolchain or
> > require to install ruby.  
> 
> I think you meant that to be the other way round, or I fail at parsing
> you. ;)

I mean: I'm a way more concerned about using a tool that fulfill our
needs than on toolchain it uses.

> 
> > In the case of Docbook, we know it works and we know already its
> > issues. Please correct me if I'm wrong, but the big problem we
> > have is not due to the DocBook toolchain, but due to the lack of
> > features at the kernel-doc script. Also, xmlto is already installed
> > by the ones that build the kernel docs. So, keeping use it won't
> > require to install a weird toolchain by hand.  
> 
> I think kernel-doc is just a small part of the puzzle. It's a problem,
> but a small one at that, and we've already made it output asciidoc, rst
> and docbook as part of this exercise. For real, as in code, not as in
> talk.
> 
> The reasons I'm involved in this is that I want to make writing
> documentation and rich kernel-doc comments easier (using lightweight
> markup) and I want to make building the documentation easier (using a
> straightforward toolchain with not too many dependencies). I'm hoping
> the former makes writing documentation more attractive and the latter
> keeps the documentation and the toolchain in a better shape through
> having more people actually build the documentation.

I don't think the toolchain is a problem. We don't attract too many
people because developers don't like writing documentation.

Ok, using a markup language can be easier than writing DocBook
tags directly, but people usually don't even add C comments on
the code they submit.

> IMHO docbook is problematic because the toolchain gets too long and
> fragile. You need plenty of tools installed to build the documentation,
> it's fussy to get working, and people just won't. Like code,
> documentation bitrots too when it's not used. 

On most distros, a single command installs all that it is needed.

> The documentation build is broken too often. 

This is indeed a problem, but the way I solved this at the media
subsystem is that I rebuild the documentation every time a media
file either at Documentation/Docbook or included at device-drivers.tmpl
is touched.

If the script produces error, I nack the patch.

So, IMHO, this is not a toolchain fault, but the lack of a process.

The kernel build robot is now producing e-mails when the documentation
has new troubles, so I guess this will help a lot to avoid adding
new documentation breakages.

> Debugging formatting issues through the entire
> pipeline gets hard; I already faced some of this when playing with the
> kernel-doc->asciidoc->docbook->html chain.
> 
> In short, I don't think the docbook toolchain fills all of our needs
> either.
> 
> > So, to be frank, it doesn't scary me to use either pyhton or
> > ruby script + docbook.
> >
> > Of course, having to own the code has a cost that should be evaluated.
> >
> > If, on the other hand, we decide to use RST, we'll very likely need to
> > patch it to fulfill our needs in order to add proper table support.
> > I've no idea how easy/difficult would be to do that, nor if Sphinx
> > upstream would accept such changes.
> >
> > So, at the end of the day, we may end by having to carry on our own
> > version of Sphinx inside our tree, with doesn't sound good, specially
> > since it is not just a script, but a package with hundreds of
> > files.  
> 
> If we end up having to modify Sphinx, it has a powerful extension
> mechanism for this. We wouldn't have to worry about getting it merged to
> Sphinx upstream, and we wouldn't have to carry a local version of all of
> Sphinx. (In fact, the extension mechanism provides a future path for
> doing kernel-doc within Sphinx instead of as a preprocessing step.)

That's indeed a good news.

> 
> I know none of this alleviates your concerns with table supports right
> now. I'll try to have a look at that a bit more.

I created a PoC tree with a few usecases taken from the V4L2 uAPI
documentation:
	https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/

Perhaps you could help to fix the issues there:

1) We want borderless tables, on both PDF and HTML outputs, for
the table at:
	https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/tree/v4l-table-within-table.rst

2) The tables at packed-rgb.rst are not created:
	https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/diff/packed-rgb.rst

It complains that the table there is malformed:                                                                                                   
	packed-rgb.rst:12: ERROR: Malformed table.

3) I tried to use a .. cssclass, as Johannes suggested, but
I was not able to include the CSS file. I suspect that this is
easy to fix, but I want to see if the cssclass will also work for
the pdf output as well.

4) It seems that it can't produce nested tables in pdf:

Markup is unsupported in LaTeX:
v4l-table-within-table:: nested tables are not yet implemented.
Makefile:115: recipe for target 'latexpdf' failed

Can you help solving those issues?

Thanks,
Mauro

[toc] | [next] | [standalone]

#1350249

On Fri, Mar 04, 2016 at 09:59:50AM -0300, Mauro Carvalho Chehab wrote:
> 
> 3) I tried to use a .. cssclass, as Johannes suggested, but
> I was not able to include the CSS file. I suspect that this is
> easy to fix, but I want to see if the cssclass will also work for
> the pdf output as well.

"cssclass" was (I think) a custom role defined in the example,
unless you also have defined a custom role you can use plain "class".
I have not looked deeper into the theming and template stuff.

> 4) It seems that it can't produce nested tables in pdf:
> 
> Markup is unsupported in LaTeX:
> v4l-table-within-table:: nested tables are not yet implemented.
> Makefile:115: recipe for target 'latexpdf' failed

This:
http://www.sphinx-doc.org/en/stable/markup/misc.html#tables

suggests you need to add the tabularcolumns directive
for complex tables.

BTW, as an alternative to the ASCII-art input
there is also support for CSV and list tables:
http://docutils.sourceforge.net/docs/ref/rst/directives.html#table


Johannes

[toc] | [prev] | [next] | [standalone]

#1351029

Em Fri, 04 Mar 2016 15:09:09 +0100
Johannes Stezenbach <js@linuxtv.org> escreveu:

> On Fri, Mar 04, 2016 at 09:59:50AM -0300, Mauro Carvalho Chehab wrote:
> > 
> > 3) I tried to use a .. cssclass, as Johannes suggested, but
> > I was not able to include the CSS file. I suspect that this is
> > easy to fix, but I want to see if the cssclass will also work for
> > the pdf output as well.  
> 
> "cssclass" was (I think) a custom role defined in the example,
> unless you also have defined a custom role you can use plain "class".
> I have not looked deeper into the theming and template stuff.

Well, it accepted cssclass for html (well, it didn't find the
templates - so I guess it is just me failing to understand how to tell
sphinx to get the stylesheet), but it rejects it for latexPDF.

> 
> > 4) It seems that it can't produce nested tables in pdf:
> > 
> > Markup is unsupported in LaTeX:
> > v4l-table-within-table:: nested tables are not yet implemented.
> > Makefile:115: recipe for target 'latexpdf' failed  
> 
> This:
> http://www.sphinx-doc.org/en/stable/markup/misc.html#tables
> 
> suggests you need to add the tabularcolumns directive
> for complex tables.
> 
> BTW, as an alternative to the ASCII-art input
> there is also support for CSV and list tables:
> http://docutils.sourceforge.net/docs/ref/rst/directives.html#table

I converted one of the big tables to CSV. At least now it recognized
it as a table. Yet, the table was very badly formated:
	https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html

This is how this table should look like:
	https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html

Also, as this table has merged cells at the legend. I've no idea how
to tell sphinx to do that on csv format.

The RST files are on this git tree:
	https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/

Regards,
Mauro

[toc] | [prev] | [next] | [standalone]

#1351386

On Mon, Mar 07, 2016 at 12:29:08AM +0100, Johannes Stezenbach wrote:
> On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote:
> > 
> > I converted one of the big tables to CSV. At least now it recognized
> > it as a table. Yet, the table was very badly formated:
> > 	https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html
> > 
> > This is how this table should look like:
> > 	https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html
> > 
> > Also, as this table has merged cells at the legend. I've no idea how
> > to tell sphinx to do that on csv format.
> > 
> > The RST files are on this git tree:
> > 	https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/
> 
> Yeah, seems it can't do merged cells in csv.  Attached patch converts it
> back to grid table format and fixes the table definition.
> The html output looks usable, but clearly it is no fun to
> work with tables in Sphinx.
> 
> Sphinx' latex writer can't handle nested tables, though.
> Python's docutils rst2latex can, but that doesn't help here.
> rst2pdf also supports it.  But I have doubts such a large
> table would render OK in pdf without using landscape orientation.
> I have not tried because I used python3-sphinx but rst2pdf
> is only availble for Python2 in Debian so it does not integrate
> with Sphinx.

Just a quick idea:
Perhaps one alternative would be to use Graphviz to render
the problematic tables, it supports a HTML-like syntax
and can be embedded in Spinx documents:

http://www.sphinx-doc.org/en/stable/ext/graphviz.html
http://www.graphviz.org/content/node-shapes#html
http://stackoverflow.com/questions/13890568/graphviz-html-nested-tables


Johannes

[toc] | [prev] | [next] | [standalone]

#1351564

Em Mon, 7 Mar 2016 09:48:26 +0100
Johannes Stezenbach <js@linuxtv.org> escreveu:

> On Mon, Mar 07, 2016 at 12:29:08AM +0100, Johannes Stezenbach wrote:
> > On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote:  
> > > 
> > > I converted one of the big tables to CSV. At least now it recognized
> > > it as a table. Yet, the table was very badly formated:
> > > 	https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html
> > > 
> > > This is how this table should look like:
> > > 	https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html
> > > 
> > > Also, as this table has merged cells at the legend. I've no idea how
> > > to tell sphinx to do that on csv format.
> > > 
> > > The RST files are on this git tree:
> > > 	https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/  
> > 
> > Yeah, seems it can't do merged cells in csv.  Attached patch converts it
> > back to grid table format and fixes the table definition.
> > The html output looks usable, but clearly it is no fun to
> > work with tables in Sphinx.
> > 
> > Sphinx' latex writer can't handle nested tables, though.
> > Python's docutils rst2latex can, but that doesn't help here.
> > rst2pdf also supports it.  But I have doubts such a large
> > table would render OK in pdf without using landscape orientation.
> > I have not tried because I used python3-sphinx but rst2pdf
> > is only availble for Python2 in Debian so it does not integrate
> > with Sphinx.  
> 
> Just a quick idea:
> Perhaps one alternative would be to use Graphviz to render
> the problematic tables, it supports a HTML-like syntax
> and can be embedded in Spinx documents:
> 
> http://www.sphinx-doc.org/en/stable/ext/graphviz.html
> http://www.graphviz.org/content/node-shapes#html
> http://stackoverflow.com/questions/13890568/graphviz-html-nested-tables

That could work, but it is scary... Graphviz is great to generate
diagrams, but it really sucks when one wants to put a graph element
on a specific place, as it loves to reorder elements putting them on
unexpected places.

Btw, 

I converted all docs from our uAPI docbook to rst using pandoc.
It was a brainless conversion, except for a few fixes.

The output is at:
	https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/

I added it on the top of my PoC tree at:
	https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/  

Besides tables, I noticed some other bad things that needs to be
corrected somehow:

1) Document divisions are not numbered. We need that. It should be
broken into:
	- Document divisions - one per documented API:
		- V4L2
		- Remote Controllers
		- DVB
		- Media Controller

	- Chapters
	- Sessions

Everything should be numbered, as, when discussing API improvements,
it is usual the need of pinpoint to an specific chapter and section.

Tables and images should also be numbered, and we need a way to
use references for table/image numbers.

2) Images

Most images didn't popup. We have images on different file formats:
	- SVG
	- GIF
	- PDF
	- PNG

3) References

It could be a conversion issue, but there are lots of missing 
references at the documentation.

4) We need to have some way to tell sphinx to not put some things
at the lateral ToC bar. For example, at the V4L2 "Changes" section,
we don't want to have one entry per version at the ToC bar.

Giving that, I suspect that we'll have huge headaches to address
if we use sphinx, as it seems too limited to handle complex
documents. We should try to use some other tool that would give
us better results.


Regards,
Mauro

[toc] | [prev] | [next] | [standalone]

#1351569

Em Mon, 7 Mar 2016 00:29:08 +0100
Johannes Stezenbach <js@linuxtv.org> escreveu:

> On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote:
> > 
> > I converted one of the big tables to CSV. At least now it recognized
> > it as a table. Yet, the table was very badly formated:
> > 	https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html
> > 
> > This is how this table should look like:
> > 	https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html
> > 
> > Also, as this table has merged cells at the legend. I've no idea how
> > to tell sphinx to do that on csv format.
> > 
> > The RST files are on this git tree:
> > 	https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/  
> 
> Yeah, seems it can't do merged cells in csv.  Attached patch converts it
> back to grid table format and fixes the table definition.
> The html output looks usable, but clearly it is no fun to
> work with tables in Sphinx.

Yes, the output is OK, but, as you said, working with tables in
Sphinx is hard, and using asciiart for the kind of tables we have
is not nice.

> 
> Sphinx' latex writer can't handle nested tables, though.

Yeah, this is a big trouble that need to be solved if you're
willing to use Sphinx.

Btw, it crashes when trying to generate man pages:

	Exception occurred:
	  File "/usr/lib/python2.7/site-packages/docutils/writers/manpage.py", line 627, in depart_entry
	    self._active_table.append_cell(self.body[start:])
	AttributeError: 'NoneType' object has no attribute 'append_cell'
	The full traceback has been saved in /tmp/sphinx-err-04qRMz.log, if you want to report the issue to the developers.

So, if we're willing to use sphinx, someone should either fix
it to produce latex nexted table and fix it to generate manpages,
or we'll need to stick with just html output.

> Python's docutils rst2latex can, but that doesn't help here.
> rst2pdf also supports it.

At least here, rst2* scripts were unable to identify that the
index.rst had links to other rst documents. 

In the specific case of rst2latex, I got several errors like:

	index.rst:21: (ERROR/3) Unknown interpreted text role "ref".


> But I have doubts such a large
> table would render OK in pdf without using landscape orientation.

Yeah, in the past, when we had pdf enabled for DocBook (e. g. when
media development was using a separate mercurial tree), I guess
we had tags changing the text orientation on a few tables that
would otherwise won't diplay fine, but I can't remember the dirty
details anymore.

> I have not tried because I used python3-sphinx but rst2pdf
> is only availble for Python2 in Debian so it does not integrate
> with Sphinx.
> 
> 
> Johannes


-- 
Thanks,
Mauro

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web