Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1349182 > unrolled thread

Re: Kernel docs: muddying the waters a bit

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Kernel docs: muddying the waters a bit Jani Nikula <jani.nikula@intel.com> - 2016-03-03 15:10 +0100
    Re: Kernel docs: muddying the waters a bit Jonathan Corbet <corbet@lwn.net> - 2016-03-03 15:20 +0100
      Re: Kernel docs: muddying the waters a bit One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-03-03 15:40 +0100
        Re: Kernel docs: muddying the waters a bit Jonathan Corbet <corbet@lwn.net> - 2016-03-03 16:20 +0100
          Re: Kernel docs: muddying the waters a bit Daniel Vetter <daniel.vetter@ffwll.ch> - 2016-03-03 16:50 +0100
      Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-03 20:00 +0100
        Re: Kernel docs: muddying the waters a bit Keith Packard <keithp@keithp.com> - 2016-03-04 00:30 +0100
          Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-04 02:20 +0100
            Re: Kernel docs: muddying the waters a bit Jani Nikula <jani.nikula@intel.com> - 2016-03-04 09:30 +0100
              Re: Kernel docs: muddying the waters a bit Johannes Stezenbach <js@linuxtv.org> - 2016-03-04 10:00 +0100
              Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-04 14:10 +0100
                Re: Kernel docs: muddying the waters a bit Johannes Stezenbach <js@linuxtv.org> - 2016-03-04 15:10 +0100
                  Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-06 03:40 +0100
                    Re: Kernel docs: muddying the waters a bit Johannes Stezenbach <js@linuxtv.org> - 2016-03-07 09:50 +0100
                      Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-07 13:20 +0100
                    Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-07 13:20 +0100
          Re: Kernel docs: muddying the waters a bit Russel Winder <russel@winder.org.uk> - 2016-03-04 08:30 +0100
            Re: Kernel docs: muddying the waters a bit Jani Nikula <jani.nikula@intel.com> - 2016-03-04 08:50 +0100
              Re: Kernel docs: muddying the waters a bit Russel Winder <russel@winder.org.uk> - 2016-03-08 06:00 +0100
            Re: Kernel docs: muddying the waters a bit Jani Nikula <jani.nikula@intel.com> - 2016-03-08 10:50 +0100
              Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-08 12:40 +0100
                Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-08 14:40 +0100
                  Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-08 16:40 +0100
                    Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-09 22:30 +0100
                      Re: Kernel docs: muddying the waters a bit Jani Nikula <jani.nikula@intel.com> - 2016-03-10 11:30 +0100
                        Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-10 16:30 +0100
                          Re: Kernel docs: muddying the waters a bit Markus Heiser <markus.heiser@darmarit.de> - 2016-03-13 16:50 +0100
              Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-08 14:40 +0100
              Re: Kernel docs: muddying the waters a bit Jani Nikula <jani.nikula@intel.com> - 2016-03-08 15:00 +0100
                Re: Kernel docs: muddying the waters a bit Jani Nikula <jani.nikula@intel.com> - 2016-03-09 10:00 +0100
    Re: Kernel docs: muddying the waters a bit Jonathan Corbet <corbet@lwn.net> - 2016-03-07 04:50 +0100

Page 1 of 2  [1] 2  Next page →

#1349182 — Re: Kernel docs: muddying the waters a bit

On Sat, 13 Feb 2016, Jonathan Corbet <corbet@lwn.net> wrote:
> So can we discuss?  I'm not saying we have to use Sphinx, but, should we
> choose not to, we should do so with open eyes and good reasons for the
> course we do take.  What do you all think?

This stalled a bit, but the waters are still muddy...

Is the Sphinx/reStructuredText table support adequate for media/v4l
documentation?

Are the Sphinx output formats adequate in general? Specifically, is the
lack of DocBook support, and the flexibility it provides, a blocker?

Otherwise, I think Sphinx is promising.

Jon, I think we need a roll of dice, err, a well-thought-out decision
from the maintainer to go with one or the other, so we can make some
real progress.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [next] | [standalone]

#1349197

On Thu, 03 Mar 2016 16:03:14 +0200
Jani Nikula <jani.nikula@intel.com> wrote:

> This stalled a bit, but the waters are still muddy...

I've been dealing with real-world obnoxiousness, something which won't
come to an immediate end, unfortunately.  But I have been taking some time
to mess with things, and hope to have some more POC patches to send out
soon.

> Is the Sphinx/reStructuredText table support adequate for media/v4l
> documentation?

That's perhaps the biggest question.  My sense is "yes", but this needs a
bit more assurance than that.

> Are the Sphinx output formats adequate in general? Specifically, is the
> lack of DocBook support, and the flexibility it provides, a blocker?

DocBook is a means to an end; nobody really wants DocBook itself as far
as I can tell. I've been messing with rst2pdf a bit; it's not hard to get
reasonable output, and, with some effort, we could probably get really
nice output. HTML and EPUB are easily covered, still haven't really played
around with man pages yet.  And there's LaTeX if we really need it.  I
kind of think we're covered there, unless I've missed something?

> Otherwise, I think Sphinx is promising.
> 
> Jon, I think we need a roll of dice, err, a well-thought-out decision
> from the maintainer to go with one or the other, so we can make some
> real progress.

My inclination at the moment is very much in the Sphinx direction.  I had
some vague thoughts of pushing a throwaway experimental directory with a
couple of docs for 4.6 that would just let people play with it easily;
then we'd see how many screams we get.  We'll see if the world lets me get
there.

Thanks,

jon

[toc] | [prev] | [next] | [standalone]

#1349227

> DocBook is a means to an end; nobody really wants DocBook itself as far
> as I can tell. 

We only have docbook because it was the tool of choice rather a lot of
years ago to then get useful output formats. It was just inherited when
borrowed the original scripts from Gnome/Gtk. It's still the most
effective way IMHO of building big structured documents out of the kernel.

The Gtk people long ago rewrote the original document script into a real
tool so they have some different and maintained tools that are close to
equivalent and already have some markdown support. Before we go off and
re-invent the wheel it might be worth just borrowing their wheel and
tweaking it as needed ? In particular they can generate help indexes so
that the entire output becomes nicely browsable with an HTML based help
browser.

Alan

[toc] | [prev] | [next] | [standalone]

#1349279

On Thu, 3 Mar 2016 14:34:25 +0000
One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> wrote:

> We only have docbook because it was the tool of choice rather a lot of
> years ago to then get useful output formats. It was just inherited when
> borrowed the original scripts from Gnome/Gtk. It's still the most
> effective way IMHO of building big structured documents out of the kernel.

...except that we haven't used it that way.  Instead, we make a whole
bunch of smaller, partially structured document silos.

> The Gtk people long ago rewrote the original document script into a real
> tool so they have some different and maintained tools that are close to
> equivalent and already have some markdown support. Before we go off and
> re-invent the wheel it might be worth just borrowing their wheel and
> tweaking it as needed ? In particular they can generate help indexes so
> that the entire output becomes nicely browsable with an HTML based help
> browser.

Well, not inventing the wheel was kind of the motivation behind much of
this effort; I got kind of worried watching us trying to cobble more
functionality into our existing house-of-cards documentation system.

Sphinx is a well-established, heavily used, and well supported system;
using it would not be an exercise in wheel reinvention.  As far as I can
tell, it does everything we need (with some open questions about table
support), lets us drop the whole DocBook toolchain dependency, and move to
a much better-supported setup than we have now.  Plus we get much nicer
output, index generation, cross-references between documents, and the
ability to write documents in a lightweight markup language.  Seems like a
win.

I assume you're referring to gtk-doc?  It's web page
(http://www.gtk.org/gtk-doc/) starts by noting that it's "a bit awkward to
setup and use"; they recommend looking at Doxygen instead.  So I guess I'm
not really sure what it offers that merits throwing another option into
the mix now?  What am I missing?

Thanks,

jon

[toc] | [prev] | [next] | [standalone]

#1349305

On Thu, Mar 3, 2016 at 4:17 PM, Jonathan Corbet <corbet@lwn.net> wrote:
> I assume you're referring to gtk-doc?  It's web page
> (http://www.gtk.org/gtk-doc/) starts by noting that it's "a bit awkward to
> setup and use"; they recommend looking at Doxygen instead.  So I guess I'm
> not really sure what it offers that merits throwing another option into
> the mix now?  What am I missing?

We use gtk-doc for the i915 testcase and tooling repo in userspace
(intel-gpu-tools). The setup is somewhat arcane (some build-fu that is
fumbly, and xml files to tie everything together). But it looks pretty
and works well otherwise. It should be at
https://01.org/linuxgraphics/gfx-docs/igt/ but our autobuilder seems
to be screwed up right now.

Of course I considered it as an option, but like doxygen it has it's
own strong opinion about how in-code comments should look like, and
those differ from kerneldoc syntax. Beyond that I don't really see
benefits over any of the solutions proposed here already (either
sphinx or rst or horror! even the hackfest I still carry around in
drm-intel.git branches).

Btw for igt we went with gtkdoc over docygen because a few people on
our team had "doxygen only over my corpse" level kind of strong
opinions. Everyone just loves their own color choice for this bikeshed
;-)
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

[toc] | [prev] | [next] | [standalone]

#1349491

Em Thu, 03 Mar 2016 07:13:05 -0700
Jonathan Corbet <corbet@lwn.net> escreveu:

> On Thu, 03 Mar 2016 16:03:14 +0200
> Jani Nikula <jani.nikula@intel.com> wrote:
> 
> > This stalled a bit, but the waters are still muddy...  
> 
> I've been dealing with real-world obnoxiousness, something which won't
> come to an immediate end, unfortunately.  But I have been taking some time
> to mess with things, and hope to have some more POC patches to send out
> soon.
> 
> > Is the Sphinx/reStructuredText table support adequate for media/v4l
> > documentation?  
> 
> That's perhaps the biggest question.  My sense is "yes", but this needs a
> bit more assurance than that.

On my tests, Sphinix seemed too limited to format tables. Asciidoc
produced an output that worked better.

Please notice that we tried to convert only one type of table. The result
with RST was not beautiful, but worked.

However, we use tables also to show how bits appear at the video formats,
like the tables at:
	https://linuxtv.org/downloads/v4l-dvb-apis/subdev.html#v4l2-mbus-format

For those tables to look nice, we should be able to remove borders and grids
from the table. I was unable to find a way to control the tables format
with RST to do things like grid/border removal.

> > Are the Sphinx output formats adequate in general? Specifically, is the
> > lack of DocBook support, and the flexibility it provides, a blocker?  
> 
> DocBook is a means to an end; nobody really wants DocBook itself as far
> as I can tell. I've been messing with rst2pdf a bit; it's not hard to get
> reasonable output, and, with some effort, we could probably get really
> nice output. HTML and EPUB are easily covered, still haven't really played
> around with man pages yet.  And there's LaTeX if we really need it.  I
> kind of think we're covered there, unless I've missed something?
> 
> > Otherwise, I think Sphinx is promising.
> > 
> > Jon, I think we need a roll of dice, err, a well-thought-out decision
> > from the maintainer to go with one or the other, so we can make some
> > real progress.  
> 
> My inclination at the moment is very much in the Sphinx direction.  I had
> some vague thoughts of pushing a throwaway experimental directory with a
> couple of docs for 4.6 that would just let people play with it easily;
> then we'd see how many screams we get.  We'll see if the world lets me get
> there.

I'm not against having a staging/Documentation for us to play with,
provided, of course, that whatever tool chosen would allow converting
what we have today.

Regards,
Mauro

[toc] | [prev] | [next] | [standalone]

#1349687

[Multipart message — attachments visible in raw view] — view raw

Mauro Carvalho Chehab <mchehab@osg.samsung.com> writes:

> On my tests, Sphinix seemed too limited to format tables. Asciidoc
> produced an output that worked better.

Yes, asciidoc has much more flexibility in table formatting, including
the ability to control text layout within cells and full control over
borders.

However, I think asciidoc has two serious problems:

  1) the python version (asciidoc) appears to have been abandoned in
     favor of the ruby version. 

  2) It really is just a docbook pre-processor. Native html/latex output
     is poorly supported at best, and exposes only a small subset of the
     full capabilities of the input language.

As such, we would have to commit to using the ruby version and either
committing to fixing the native html output backend or continuing to use
the rest of the docbook toolchain.

We could insist on using the python version, of course. I spent a bit of
time hacking that up to add 'real' support for a table-of-contents in
the native HTML backend and it looks like getting those changes
upstreamed would be reasonably straightforward. However, we'd end up
'owning' the code, and I'm not sure we want to.

-- 
-keith

[toc] | [prev] | [next] | [standalone]

#1349753

[Multipart message — attachments visible in raw view] — view raw

Em Thu, 03 Mar 2016 15:23:23 -0800
Keith Packard <keithp@keithp.com> escreveu:

> Mauro Carvalho Chehab <mchehab@osg.samsung.com> writes:
> 
> > On my tests, Sphinix seemed too limited to format tables. Asciidoc
> > produced an output that worked better.  
> 
> Yes, asciidoc has much more flexibility in table formatting, including
> the ability to control text layout within cells and full control over
> borders.
> 
> However, I think asciidoc has two serious problems:
> 
>   1) the python version (asciidoc) appears to have been abandoned in
>      favor of the ruby version. 
> 
>   2) It really is just a docbook pre-processor. Native html/latex output
>      is poorly supported at best, and exposes only a small subset of the
>      full capabilities of the input language.
> 
> As such, we would have to commit to using the ruby version and either
> committing to fixing the native html output backend or continuing to use
> the rest of the docbook toolchain.
> 
> We could insist on using the python version, of course. I spent a bit of
> time hacking that up to add 'real' support for a table-of-contents in
> the native HTML backend and it looks like getting those changes
> upstreamed would be reasonably straightforward. However, we'd end up
> 'owning' the code, and I'm not sure we want to.

I'm a way more concerned about using a tool that fulfill our needs
than to look for something that won't use the docbook toolchain or
require to install ruby.

In the case of Docbook, we know it works and we know already its
issues. Please correct me if I'm wrong, but the big problem we
have is not due to the DocBook toolchain, but due to the lack of
features at the kernel-doc script. Also, xmlto is already installed
by the ones that build the kernel docs. So, keeping use it won't
require to install a weird toolchain by hand.

So, to be frank, it doesn't scary me to use either pyhton or
ruby script + docbook.

Of course, having to own the code has a cost that should be evaluated.

If, on the other hand, we decide to use RST, we'll very likely need to
patch it to fulfill our needs in order to add proper table support.
I've no idea how easy/difficult would be to do that, nor if Sphinx
upstream would accept such changes.

So, at the end of the day, we may end by having to carry on our own
version of Sphinx inside our tree, with doesn't sound good, specially
since it is not just a script, but a package with hundreds of
files.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1349937

On Fri, 04 Mar 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
> Em Thu, 03 Mar 2016 15:23:23 -0800
> Keith Packard <keithp@keithp.com> escreveu:
>
>> Mauro Carvalho Chehab <mchehab@osg.samsung.com> writes:
>> 
>> > On my tests, Sphinix seemed too limited to format tables. Asciidoc
>> > produced an output that worked better.  
>> 
>> Yes, asciidoc has much more flexibility in table formatting, including
>> the ability to control text layout within cells and full control over
>> borders.
>> 
>> However, I think asciidoc has two serious problems:
>> 
>>   1) the python version (asciidoc) appears to have been abandoned in
>>      favor of the ruby version. 
>> 
>>   2) It really is just a docbook pre-processor. Native html/latex output
>>      is poorly supported at best, and exposes only a small subset of the
>>      full capabilities of the input language.
>> 
>> As such, we would have to commit to using the ruby version and either
>> committing to fixing the native html output backend or continuing to use
>> the rest of the docbook toolchain.
>> 
>> We could insist on using the python version, of course. I spent a bit of
>> time hacking that up to add 'real' support for a table-of-contents in
>> the native HTML backend and it looks like getting those changes
>> upstreamed would be reasonably straightforward. However, we'd end up
>> 'owning' the code, and I'm not sure we want to.
>
> I'm a way more concerned about using a tool that fulfill our needs
> than to look for something that won't use the docbook toolchain or
> require to install ruby.

I think you meant that to be the other way round, or I fail at parsing
you. ;)

> In the case of Docbook, we know it works and we know already its
> issues. Please correct me if I'm wrong, but the big problem we
> have is not due to the DocBook toolchain, but due to the lack of
> features at the kernel-doc script. Also, xmlto is already installed
> by the ones that build the kernel docs. So, keeping use it won't
> require to install a weird toolchain by hand.

I think kernel-doc is just a small part of the puzzle. It's a problem,
but a small one at that, and we've already made it output asciidoc, rst
and docbook as part of this exercise. For real, as in code, not as in
talk.

The reasons I'm involved in this is that I want to make writing
documentation and rich kernel-doc comments easier (using lightweight
markup) and I want to make building the documentation easier (using a
straightforward toolchain with not too many dependencies). I'm hoping
the former makes writing documentation more attractive and the latter
keeps the documentation and the toolchain in a better shape through
having more people actually build the documentation.

IMHO docbook is problematic because the toolchain gets too long and
fragile. You need plenty of tools installed to build the documentation,
it's fussy to get working, and people just won't. Like code,
documentation bitrots too when it's not used. The documentation build is
broken too often. Debugging formatting issues through the entire
pipeline gets hard; I already faced some of this when playing with the
kernel-doc->asciidoc->docbook->html chain.

In short, I don't think the docbook toolchain fills all of our needs
either.

> So, to be frank, it doesn't scary me to use either pyhton or
> ruby script + docbook.
>
> Of course, having to own the code has a cost that should be evaluated.
>
> If, on the other hand, we decide to use RST, we'll very likely need to
> patch it to fulfill our needs in order to add proper table support.
> I've no idea how easy/difficult would be to do that, nor if Sphinx
> upstream would accept such changes.
>
> So, at the end of the day, we may end by having to carry on our own
> version of Sphinx inside our tree, with doesn't sound good, specially
> since it is not just a script, but a package with hundreds of
> files.

If we end up having to modify Sphinx, it has a powerful extension
mechanism for this. We wouldn't have to worry about getting it merged to
Sphinx upstream, and we wouldn't have to carry a local version of all of
Sphinx. (In fact, the extension mechanism provides a future path for
doing kernel-doc within Sphinx instead of as a preprocessing step.)

I know none of this alleviates your concerns with table supports right
now. I'll try to have a look at that a bit more.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1349959

On Fri, Mar 04, 2016 at 10:29:08AM +0200, Jani Nikula wrote:
> On Fri, 04 Mar 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
> >
> > If, on the other hand, we decide to use RST, we'll very likely need to
> > patch it to fulfill our needs in order to add proper table support.
> > I've no idea how easy/difficult would be to do that, nor if Sphinx
> > upstream would accept such changes.
> >
> > So, at the end of the day, we may end by having to carry on our own
> > version of Sphinx inside our tree, with doesn't sound good, specially
> > since it is not just a script, but a package with hundreds of
> > files.
> 
> If we end up having to modify Sphinx, it has a powerful extension
> mechanism for this. We wouldn't have to worry about getting it merged to
> Sphinx upstream, and we wouldn't have to carry a local version of all of
> Sphinx. (In fact, the extension mechanism provides a future path for
> doing kernel-doc within Sphinx instead of as a preprocessing step.)
> 
> I know none of this alleviates your concerns with table supports right
> now. I'll try to have a look at that a bit more.

FWIW, I think table formatting in Sphinx works via style sheets.
The mechanism is documented in the Python docutils docs that
Sphinx is built upon.
Basically you use the "class" or "role" directive and define
the corresponding CSS or LaTeX (or rst2pdf) style.

Here is one example (using a custom "cssclass" role):
https://pythonhosted.org/sphinxjp.themes.basicstrap/sample.html

Directives (especially role and class):
http://www.sphinx-doc.org/en/stable/rest.html#directives

LaTeX styling:
http://docutils.readthedocs.org/en/sphinx-docs/user/latex.html#custom-interpreted-text-roles


HTH,
Johannes

[toc] | [prev] | [next] | [standalone]

#1350219

Em Fri, 04 Mar 2016 10:29:08 +0200
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Fri, 04 Mar 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
> > Em Thu, 03 Mar 2016 15:23:23 -0800
> > Keith Packard <keithp@keithp.com> escreveu:
> >  
> >> Mauro Carvalho Chehab <mchehab@osg.samsung.com> writes:
> >>   
> >> > On my tests, Sphinix seemed too limited to format tables. Asciidoc
> >> > produced an output that worked better.    
> >> 
> >> Yes, asciidoc has much more flexibility in table formatting, including
> >> the ability to control text layout within cells and full control over
> >> borders.
> >> 
> >> However, I think asciidoc has two serious problems:
> >> 
> >>   1) the python version (asciidoc) appears to have been abandoned in
> >>      favor of the ruby version. 
> >> 
> >>   2) It really is just a docbook pre-processor. Native html/latex output
> >>      is poorly supported at best, and exposes only a small subset of the
> >>      full capabilities of the input language.
> >> 
> >> As such, we would have to commit to using the ruby version and either
> >> committing to fixing the native html output backend or continuing to use
> >> the rest of the docbook toolchain.
> >> 
> >> We could insist on using the python version, of course. I spent a bit of
> >> time hacking that up to add 'real' support for a table-of-contents in
> >> the native HTML backend and it looks like getting those changes
> >> upstreamed would be reasonably straightforward. However, we'd end up
> >> 'owning' the code, and I'm not sure we want to.  
> >
> > I'm a way more concerned about using a tool that fulfill our needs
> > than to look for something that won't use the docbook toolchain or
> > require to install ruby.  
> 
> I think you meant that to be the other way round, or I fail at parsing
> you. ;)

I mean: I'm a way more concerned about using a tool that fulfill our
needs than on toolchain it uses.

> 
> > In the case of Docbook, we know it works and we know already its
> > issues. Please correct me if I'm wrong, but the big problem we
> > have is not due to the DocBook toolchain, but due to the lack of
> > features at the kernel-doc script. Also, xmlto is already installed
> > by the ones that build the kernel docs. So, keeping use it won't
> > require to install a weird toolchain by hand.  
> 
> I think kernel-doc is just a small part of the puzzle. It's a problem,
> but a small one at that, and we've already made it output asciidoc, rst
> and docbook as part of this exercise. For real, as in code, not as in
> talk.
> 
> The reasons I'm involved in this is that I want to make writing
> documentation and rich kernel-doc comments easier (using lightweight
> markup) and I want to make building the documentation easier (using a
> straightforward toolchain with not too many dependencies). I'm hoping
> the former makes writing documentation more attractive and the latter
> keeps the documentation and the toolchain in a better shape through
> having more people actually build the documentation.

I don't think the toolchain is a problem. We don't attract too many
people because developers don't like writing documentation.

Ok, using a markup language can be easier than writing DocBook
tags directly, but people usually don't even add C comments on
the code they submit.

> IMHO docbook is problematic because the toolchain gets too long and
> fragile. You need plenty of tools installed to build the documentation,
> it's fussy to get working, and people just won't. Like code,
> documentation bitrots too when it's not used. 

On most distros, a single command installs all that it is needed.

> The documentation build is broken too often. 

This is indeed a problem, but the way I solved this at the media
subsystem is that I rebuild the documentation every time a media
file either at Documentation/Docbook or included at device-drivers.tmpl
is touched.

If the script produces error, I nack the patch.

So, IMHO, this is not a toolchain fault, but the lack of a process.

The kernel build robot is now producing e-mails when the documentation
has new troubles, so I guess this will help a lot to avoid adding
new documentation breakages.

> Debugging formatting issues through the entire
> pipeline gets hard; I already faced some of this when playing with the
> kernel-doc->asciidoc->docbook->html chain.
> 
> In short, I don't think the docbook toolchain fills all of our needs
> either.
> 
> > So, to be frank, it doesn't scary me to use either pyhton or
> > ruby script + docbook.
> >
> > Of course, having to own the code has a cost that should be evaluated.
> >
> > If, on the other hand, we decide to use RST, we'll very likely need to
> > patch it to fulfill our needs in order to add proper table support.
> > I've no idea how easy/difficult would be to do that, nor if Sphinx
> > upstream would accept such changes.
> >
> > So, at the end of the day, we may end by having to carry on our own
> > version of Sphinx inside our tree, with doesn't sound good, specially
> > since it is not just a script, but a package with hundreds of
> > files.  
> 
> If we end up having to modify Sphinx, it has a powerful extension
> mechanism for this. We wouldn't have to worry about getting it merged to
> Sphinx upstream, and we wouldn't have to carry a local version of all of
> Sphinx. (In fact, the extension mechanism provides a future path for
> doing kernel-doc within Sphinx instead of as a preprocessing step.)

That's indeed a good news.

> 
> I know none of this alleviates your concerns with table supports right
> now. I'll try to have a look at that a bit more.

I created a PoC tree with a few usecases taken from the V4L2 uAPI
documentation:
	https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/

Perhaps you could help to fix the issues there:

1) We want borderless tables, on both PDF and HTML outputs, for
the table at:
	https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/tree/v4l-table-within-table.rst

2) The tables at packed-rgb.rst are not created:
	https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/diff/packed-rgb.rst

It complains that the table there is malformed:                                                                                                   
	packed-rgb.rst:12: ERROR: Malformed table.

3) I tried to use a .. cssclass, as Johannes suggested, but
I was not able to include the CSS file. I suspect that this is
easy to fix, but I want to see if the cssclass will also work for
the pdf output as well.

4) It seems that it can't produce nested tables in pdf:

Markup is unsupported in LaTeX:
v4l-table-within-table:: nested tables are not yet implemented.
Makefile:115: recipe for target 'latexpdf' failed

Can you help solving those issues?

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1350249

On Fri, Mar 04, 2016 at 09:59:50AM -0300, Mauro Carvalho Chehab wrote:
> 
> 3) I tried to use a .. cssclass, as Johannes suggested, but
> I was not able to include the CSS file. I suspect that this is
> easy to fix, but I want to see if the cssclass will also work for
> the pdf output as well.

"cssclass" was (I think) a custom role defined in the example,
unless you also have defined a custom role you can use plain "class".
I have not looked deeper into the theming and template stuff.

> 4) It seems that it can't produce nested tables in pdf:
> 
> Markup is unsupported in LaTeX:
> v4l-table-within-table:: nested tables are not yet implemented.
> Makefile:115: recipe for target 'latexpdf' failed

This:
http://www.sphinx-doc.org/en/stable/markup/misc.html#tables

suggests you need to add the tabularcolumns directive
for complex tables.

BTW, as an alternative to the ASCII-art input
there is also support for CSV and list tables:
http://docutils.sourceforge.net/docs/ref/rst/directives.html#table


Johannes

[toc] | [prev] | [next] | [standalone]

#1351029

Em Fri, 04 Mar 2016 15:09:09 +0100
Johannes Stezenbach <js@linuxtv.org> escreveu:

> On Fri, Mar 04, 2016 at 09:59:50AM -0300, Mauro Carvalho Chehab wrote:
> > 
> > 3) I tried to use a .. cssclass, as Johannes suggested, but
> > I was not able to include the CSS file. I suspect that this is
> > easy to fix, but I want to see if the cssclass will also work for
> > the pdf output as well.  
> 
> "cssclass" was (I think) a custom role defined in the example,
> unless you also have defined a custom role you can use plain "class".
> I have not looked deeper into the theming and template stuff.

Well, it accepted cssclass for html (well, it didn't find the
templates - so I guess it is just me failing to understand how to tell
sphinx to get the stylesheet), but it rejects it for latexPDF.

> 
> > 4) It seems that it can't produce nested tables in pdf:
> > 
> > Markup is unsupported in LaTeX:
> > v4l-table-within-table:: nested tables are not yet implemented.
> > Makefile:115: recipe for target 'latexpdf' failed  
> 
> This:
> http://www.sphinx-doc.org/en/stable/markup/misc.html#tables
> 
> suggests you need to add the tabularcolumns directive
> for complex tables.
> 
> BTW, as an alternative to the ASCII-art input
> there is also support for CSV and list tables:
> http://docutils.sourceforge.net/docs/ref/rst/directives.html#table

I converted one of the big tables to CSV. At least now it recognized
it as a table. Yet, the table was very badly formated:
	https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html

This is how this table should look like:
	https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html

Also, as this table has merged cells at the legend. I've no idea how
to tell sphinx to do that on csv format.

The RST files are on this git tree:
	https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/

Regards,
Mauro

[toc] | [prev] | [next] | [standalone]

#1351386

On Mon, Mar 07, 2016 at 12:29:08AM +0100, Johannes Stezenbach wrote:
> On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote:
> > 
> > I converted one of the big tables to CSV. At least now it recognized
> > it as a table. Yet, the table was very badly formated:
> > 	https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html
> > 
> > This is how this table should look like:
> > 	https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html
> > 
> > Also, as this table has merged cells at the legend. I've no idea how
> > to tell sphinx to do that on csv format.
> > 
> > The RST files are on this git tree:
> > 	https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/
> 
> Yeah, seems it can't do merged cells in csv.  Attached patch converts it
> back to grid table format and fixes the table definition.
> The html output looks usable, but clearly it is no fun to
> work with tables in Sphinx.
> 
> Sphinx' latex writer can't handle nested tables, though.
> Python's docutils rst2latex can, but that doesn't help here.
> rst2pdf also supports it.  But I have doubts such a large
> table would render OK in pdf without using landscape orientation.
> I have not tried because I used python3-sphinx but rst2pdf
> is only availble for Python2 in Debian so it does not integrate
> with Sphinx.

Just a quick idea:
Perhaps one alternative would be to use Graphviz to render
the problematic tables, it supports a HTML-like syntax
and can be embedded in Spinx documents:

http://www.sphinx-doc.org/en/stable/ext/graphviz.html
http://www.graphviz.org/content/node-shapes#html
http://stackoverflow.com/questions/13890568/graphviz-html-nested-tables


Johannes

[toc] | [prev] | [next] | [standalone]

#1351564

Em Mon, 7 Mar 2016 09:48:26 +0100
Johannes Stezenbach <js@linuxtv.org> escreveu:

> On Mon, Mar 07, 2016 at 12:29:08AM +0100, Johannes Stezenbach wrote:
> > On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote:  
> > > 
> > > I converted one of the big tables to CSV. At least now it recognized
> > > it as a table. Yet, the table was very badly formated:
> > > 	https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html
> > > 
> > > This is how this table should look like:
> > > 	https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html
> > > 
> > > Also, as this table has merged cells at the legend. I've no idea how
> > > to tell sphinx to do that on csv format.
> > > 
> > > The RST files are on this git tree:
> > > 	https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/  
> > 
> > Yeah, seems it can't do merged cells in csv.  Attached patch converts it
> > back to grid table format and fixes the table definition.
> > The html output looks usable, but clearly it is no fun to
> > work with tables in Sphinx.
> > 
> > Sphinx' latex writer can't handle nested tables, though.
> > Python's docutils rst2latex can, but that doesn't help here.
> > rst2pdf also supports it.  But I have doubts such a large
> > table would render OK in pdf without using landscape orientation.
> > I have not tried because I used python3-sphinx but rst2pdf
> > is only availble for Python2 in Debian so it does not integrate
> > with Sphinx.  
> 
> Just a quick idea:
> Perhaps one alternative would be to use Graphviz to render
> the problematic tables, it supports a HTML-like syntax
> and can be embedded in Spinx documents:
> 
> http://www.sphinx-doc.org/en/stable/ext/graphviz.html
> http://www.graphviz.org/content/node-shapes#html
> http://stackoverflow.com/questions/13890568/graphviz-html-nested-tables

That could work, but it is scary... Graphviz is great to generate
diagrams, but it really sucks when one wants to put a graph element
on a specific place, as it loves to reorder elements putting them on
unexpected places.

Btw, 

I converted all docs from our uAPI docbook to rst using pandoc.
It was a brainless conversion, except for a few fixes.

The output is at:
	https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/

I added it on the top of my PoC tree at:
	https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/  

Besides tables, I noticed some other bad things that needs to be
corrected somehow:

1) Document divisions are not numbered. We need that. It should be
broken into:
	- Document divisions - one per documented API:
		- V4L2
		- Remote Controllers
		- DVB
		- Media Controller

	- Chapters
	- Sessions

Everything should be numbered, as, when discussing API improvements,
it is usual the need of pinpoint to an specific chapter and section.

Tables and images should also be numbered, and we need a way to
use references for table/image numbers.

2) Images

Most images didn't popup. We have images on different file formats:
	- SVG
	- GIF
	- PDF
	- PNG

3) References

It could be a conversion issue, but there are lots of missing 
references at the documentation.

4) We need to have some way to tell sphinx to not put some things
at the lateral ToC bar. For example, at the V4L2 "Changes" section,
we don't want to have one entry per version at the ToC bar.

Giving that, I suspect that we'll have huge headaches to address
if we use sphinx, as it seems too limited to handle complex
documents. We should try to use some other tool that would give
us better results.


Regards,
Mauro

[toc] | [prev] | [next] | [standalone]

#1351569

Em Mon, 7 Mar 2016 00:29:08 +0100
Johannes Stezenbach <js@linuxtv.org> escreveu:

> On Sat, Mar 05, 2016 at 11:29:37PM -0300, Mauro Carvalho Chehab wrote:
> > 
> > I converted one of the big tables to CSV. At least now it recognized
> > it as a table. Yet, the table was very badly formated:
> > 	https://mchehab.fedorapeople.org/media-kabi-docs-test/rst_tests/packed-rgb.html
> > 
> > This is how this table should look like:
> > 	https://linuxtv.org/downloads/v4l-dvb-apis/packed-rgb.html
> > 
> > Also, as this table has merged cells at the legend. I've no idea how
> > to tell sphinx to do that on csv format.
> > 
> > The RST files are on this git tree:
> > 	https://git.linuxtv.org/mchehab/v4l2-docs-poc.git/  
> 
> Yeah, seems it can't do merged cells in csv.  Attached patch converts it
> back to grid table format and fixes the table definition.
> The html output looks usable, but clearly it is no fun to
> work with tables in Sphinx.

Yes, the output is OK, but, as you said, working with tables in
Sphinx is hard, and using asciiart for the kind of tables we have
is not nice.

> 
> Sphinx' latex writer can't handle nested tables, though.

Yeah, this is a big trouble that need to be solved if you're
willing to use Sphinx.

Btw, it crashes when trying to generate man pages:

	Exception occurred:
	  File "/usr/lib/python2.7/site-packages/docutils/writers/manpage.py", line 627, in depart_entry
	    self._active_table.append_cell(self.body[start:])
	AttributeError: 'NoneType' object has no attribute 'append_cell'
	The full traceback has been saved in /tmp/sphinx-err-04qRMz.log, if you want to report the issue to the developers.

So, if we're willing to use sphinx, someone should either fix
it to produce latex nexted table and fix it to generate manpages,
or we'll need to stick with just html output.

> Python's docutils rst2latex can, but that doesn't help here.
> rst2pdf also supports it.

At least here, rst2* scripts were unable to identify that the
index.rst had links to other rst documents. 

In the specific case of rst2latex, I got several errors like:

	index.rst:21: (ERROR/3) Unknown interpreted text role "ref".


> But I have doubts such a large
> table would render OK in pdf without using landscape orientation.

Yeah, in the past, when we had pdf enabled for DocBook (e. g. when
media development was using a separate mercurial tree), I guess
we had tags changing the text orientation on a few tables that
would otherwise won't diplay fine, but I can't remember the dirty
details anymore.

> I have not tried because I used python3-sphinx but rst2pdf
> is only availble for Python2 in Debian so it does not integrate
> with Sphinx.
> 
> 
> Johannes


-- 
Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1349911

[Multipart message — attachments visible in raw view] — view raw

On Thu, 2016-03-03 at 15:23 -0800, Keith Packard wrote:
> 
[…]
> However, I think asciidoc has two serious problems:
> 
>   1) the python version (asciidoc) appears to have been abandoned in
>      favor of the ruby version. 

This is I think true, however the Java-based tool chain Asciidoctor is
I believe the standard bearer for ASCIIdoc these days, albeit called
ASCIIdoctor.

>   2) It really is just a docbook pre-processor. Native html/latex
> output
>      is poorly supported at best, and exposes only a small subset of
> the
>      full capabilities of the input language.

This is not true. Yes ASCIIDoc started as a DocBook/XML frontend so as
to use a sane :-) markup language rather than XML (XML is a notation
for consenting computers only), but the current ASCIIDoctor toolchain
deals very well in direct HTML and PDF generation, without needing a
DocBook/XML toolchain. 

> As such, we would have to commit to using the ruby version and either
> committing to fixing the native html output backend or continuing to
> use
> the rest of the docbook toolchain.

Or trial the JVM-based ASCIIdoctor which is what the projects I am
involved with chose to use. Perhaps as an example I can give you http:/
/gpars.website (it's a redirector) all the HTML and PDF is generated
from ASCIIDoc source using ASCIIDoctor driven with a Gradle build
system. This is still very much a work in progress (by Jim Northrop,
not me currently), but I like it.

> We could insist on using the python version, of course. I spent a bit
> of
> time hacking that up to add 'real' support for a table-of-contents in
> the native HTML backend and it looks like getting those changes
> upstreamed would be reasonably straightforward. However, we'd end up
> 'owning' the code, and I'm not sure we want to.

If the Python version is really not being maintained, I would suggest
that unless you want to take over the project and be it's maintainer,
you would be better advised to use a different version.

-- Russel.=============================================================================Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.ukLondon SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

[toc] | [prev] | [next] | [standalone]

#1349919

On Fri, 04 Mar 2016, Russel Winder <russel@winder.org.uk> wrote:
> On Thu, 2016-03-03 at 15:23 -0800, Keith Packard wrote:
>>   1) the python version (asciidoc) appears to have been abandoned in
>>      favor of the ruby version. 
>
> This is I think true, however the Java-based tool chain Asciidoctor is
> I believe the standard bearer for ASCIIdoc these days, albeit called
> ASCIIdoctor.

If we're talking about the same asciidoctor (http://asciidoctor.org/)
it's written in ruby but you can apparently run it in JVM using
JRuby. Calling it Java-based is misleading.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1352626

[Multipart message — attachments visible in raw view] — view raw

On Fri, 2016-03-04 at 09:46 +0200, Jani Nikula wrote:
> […]
> If we're talking about the same asciidoctor (http://asciidoctor.org/)
> it's written in ruby but you can apparently run it in JVM using
> JRuby. Calling it Java-based is misleading.

Indeed, I was somewhat imprecise. Thanks to the work mostly of Charles
Nutter, JRuby is invariably a faster platform for Ruby code than Ruby
is. So yes ASCIIDoctor is JVM-based via JRuby, not Java-based.

The real point here is that in a move from DocBook/XML as a
documentation source, ASCIIDoctor is an excellent choice.

-- Russel.=============================================================================Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.ukLondon SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

[toc] | [prev] | [next] | [standalone]

#1352807

On Tue, 08 Mar 2016, Dan Allen <dan@opendevise.io> wrote:
> One of the key goals of the Asciidoctor project is to be able to directly
> produce a wide variety of outputs from the same source (without DocBook).
> We've added flexibility and best practices into the syntax and matured the
> converter mechanism to bridge this (sometimes very wide) gap.

I think our conclusion so far was that the native AsciiDoc (and
Asciidoctor) outputs fell short of our needs, forcing us to use the
DocBook pipeline. I, for one, was hoping we could eventually simplify
the toolchain. For example, there was no support for chunked, or split
to chapters, HTML, and the single page result was simply way too big.

> Asciidoctor is the future of AsciiDoc. Even the AsciiDoc Python maintainers
> acknowledge that (including the original creator).

Thanks for the input. We've touched the topic of AsciiDoc
vs. Asciidoctor before [1]. So we should be using Asciidoctor instead of
AsciiDoc. That actually makes choosing asciidoc harder, because
requiring another language environment complicates, not simplifies, the
toolchain. I'd really like to lower the bar for building the
documentation, for everyone, so much so that it becomes part of the
normal checks for patch inclusion.


BR,
Jani.


[1] http://mid.gmane.org/86pow31ddj.fsf@hiro.keithp.com


-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.kernel

csiph-web