Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1349182 > unrolled thread

Re: Kernel docs: muddying the waters a bit

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Kernel docs: muddying the waters a bit Jani Nikula <jani.nikula@intel.com> - 2016-03-03 15:10 +0100
    Re: Kernel docs: muddying the waters a bit Jonathan Corbet <corbet@lwn.net> - 2016-03-03 15:20 +0100
      Re: Kernel docs: muddying the waters a bit One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> - 2016-03-03 15:40 +0100
        Re: Kernel docs: muddying the waters a bit Jonathan Corbet <corbet@lwn.net> - 2016-03-03 16:20 +0100
          Re: Kernel docs: muddying the waters a bit Daniel Vetter <daniel.vetter@ffwll.ch> - 2016-03-03 16:50 +0100
      Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-03 20:00 +0100
        Re: Kernel docs: muddying the waters a bit Keith Packard <keithp@keithp.com> - 2016-03-04 00:30 +0100
          Re: Kernel docs: muddying the waters a bit Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-03-04 02:20 +0100
            Re: Kernel docs: muddying the waters a bit Jani Nikula <jani.nikula@intel.com> - 2016-03-04 09:30 +0100
              Re: Kernel docs: muddying the waters a bit Johannes Stezenbach <js@linuxtv.org> - 2016-03-04 10:00 +0100
          Re: Kernel docs: muddying the waters a bit Russel Winder <russel@winder.org.uk> - 2016-03-04 08:30 +0100
            Re: Kernel docs: muddying the waters a bit Jani Nikula <jani.nikula@intel.com> - 2016-03-04 08:50 +0100

#1349182 — Re: Kernel docs: muddying the waters a bit

On Sat, 13 Feb 2016, Jonathan Corbet <corbet@lwn.net> wrote:
> So can we discuss?  I'm not saying we have to use Sphinx, but, should we
> choose not to, we should do so with open eyes and good reasons for the
> course we do take.  What do you all think?

This stalled a bit, but the waters are still muddy...

Is the Sphinx/reStructuredText table support adequate for media/v4l
documentation?

Are the Sphinx output formats adequate in general? Specifically, is the
lack of DocBook support, and the flexibility it provides, a blocker?

Otherwise, I think Sphinx is promising.

Jon, I think we need a roll of dice, err, a well-thought-out decision
from the maintainer to go with one or the other, so we can make some
real progress.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [next] | [standalone]

#1349197

On Thu, 03 Mar 2016 16:03:14 +0200
Jani Nikula <jani.nikula@intel.com> wrote:

> This stalled a bit, but the waters are still muddy...

I've been dealing with real-world obnoxiousness, something which won't
come to an immediate end, unfortunately.  But I have been taking some time
to mess with things, and hope to have some more POC patches to send out
soon.

> Is the Sphinx/reStructuredText table support adequate for media/v4l
> documentation?

That's perhaps the biggest question.  My sense is "yes", but this needs a
bit more assurance than that.

> Are the Sphinx output formats adequate in general? Specifically, is the
> lack of DocBook support, and the flexibility it provides, a blocker?

DocBook is a means to an end; nobody really wants DocBook itself as far
as I can tell. I've been messing with rst2pdf a bit; it's not hard to get
reasonable output, and, with some effort, we could probably get really
nice output. HTML and EPUB are easily covered, still haven't really played
around with man pages yet.  And there's LaTeX if we really need it.  I
kind of think we're covered there, unless I've missed something?

> Otherwise, I think Sphinx is promising.
> 
> Jon, I think we need a roll of dice, err, a well-thought-out decision
> from the maintainer to go with one or the other, so we can make some
> real progress.

My inclination at the moment is very much in the Sphinx direction.  I had
some vague thoughts of pushing a throwaway experimental directory with a
couple of docs for 4.6 that would just let people play with it easily;
then we'd see how many screams we get.  We'll see if the world lets me get
there.

Thanks,

jon

[toc] | [prev] | [next] | [standalone]

#1349227

> DocBook is a means to an end; nobody really wants DocBook itself as far
> as I can tell. 

We only have docbook because it was the tool of choice rather a lot of
years ago to then get useful output formats. It was just inherited when
borrowed the original scripts from Gnome/Gtk. It's still the most
effective way IMHO of building big structured documents out of the kernel.

The Gtk people long ago rewrote the original document script into a real
tool so they have some different and maintained tools that are close to
equivalent and already have some markdown support. Before we go off and
re-invent the wheel it might be worth just borrowing their wheel and
tweaking it as needed ? In particular they can generate help indexes so
that the entire output becomes nicely browsable with an HTML based help
browser.

Alan

[toc] | [prev] | [next] | [standalone]

#1349279

On Thu, 3 Mar 2016 14:34:25 +0000
One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk> wrote:

> We only have docbook because it was the tool of choice rather a lot of
> years ago to then get useful output formats. It was just inherited when
> borrowed the original scripts from Gnome/Gtk. It's still the most
> effective way IMHO of building big structured documents out of the kernel.

...except that we haven't used it that way.  Instead, we make a whole
bunch of smaller, partially structured document silos.

> The Gtk people long ago rewrote the original document script into a real
> tool so they have some different and maintained tools that are close to
> equivalent and already have some markdown support. Before we go off and
> re-invent the wheel it might be worth just borrowing their wheel and
> tweaking it as needed ? In particular they can generate help indexes so
> that the entire output becomes nicely browsable with an HTML based help
> browser.

Well, not inventing the wheel was kind of the motivation behind much of
this effort; I got kind of worried watching us trying to cobble more
functionality into our existing house-of-cards documentation system.

Sphinx is a well-established, heavily used, and well supported system;
using it would not be an exercise in wheel reinvention.  As far as I can
tell, it does everything we need (with some open questions about table
support), lets us drop the whole DocBook toolchain dependency, and move to
a much better-supported setup than we have now.  Plus we get much nicer
output, index generation, cross-references between documents, and the
ability to write documents in a lightweight markup language.  Seems like a
win.

I assume you're referring to gtk-doc?  It's web page
(http://www.gtk.org/gtk-doc/) starts by noting that it's "a bit awkward to
setup and use"; they recommend looking at Doxygen instead.  So I guess I'm
not really sure what it offers that merits throwing another option into
the mix now?  What am I missing?

Thanks,

jon

[toc] | [prev] | [next] | [standalone]

#1349305

On Thu, Mar 3, 2016 at 4:17 PM, Jonathan Corbet <corbet@lwn.net> wrote:
> I assume you're referring to gtk-doc?  It's web page
> (http://www.gtk.org/gtk-doc/) starts by noting that it's "a bit awkward to
> setup and use"; they recommend looking at Doxygen instead.  So I guess I'm
> not really sure what it offers that merits throwing another option into
> the mix now?  What am I missing?

We use gtk-doc for the i915 testcase and tooling repo in userspace
(intel-gpu-tools). The setup is somewhat arcane (some build-fu that is
fumbly, and xml files to tie everything together). But it looks pretty
and works well otherwise. It should be at
https://01.org/linuxgraphics/gfx-docs/igt/ but our autobuilder seems
to be screwed up right now.

Of course I considered it as an option, but like doxygen it has it's
own strong opinion about how in-code comments should look like, and
those differ from kerneldoc syntax. Beyond that I don't really see
benefits over any of the solutions proposed here already (either
sphinx or rst or horror! even the hackfest I still carry around in
drm-intel.git branches).

Btw for igt we went with gtkdoc over docygen because a few people on
our team had "doxygen only over my corpse" level kind of strong
opinions. Everyone just loves their own color choice for this bikeshed
;-)
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

[toc] | [prev] | [next] | [standalone]

#1349491

Em Thu, 03 Mar 2016 07:13:05 -0700
Jonathan Corbet <corbet@lwn.net> escreveu:

> On Thu, 03 Mar 2016 16:03:14 +0200
> Jani Nikula <jani.nikula@intel.com> wrote:
> 
> > This stalled a bit, but the waters are still muddy...  
> 
> I've been dealing with real-world obnoxiousness, something which won't
> come to an immediate end, unfortunately.  But I have been taking some time
> to mess with things, and hope to have some more POC patches to send out
> soon.
> 
> > Is the Sphinx/reStructuredText table support adequate for media/v4l
> > documentation?  
> 
> That's perhaps the biggest question.  My sense is "yes", but this needs a
> bit more assurance than that.

On my tests, Sphinix seemed too limited to format tables. Asciidoc
produced an output that worked better.

Please notice that we tried to convert only one type of table. The result
with RST was not beautiful, but worked.

However, we use tables also to show how bits appear at the video formats,
like the tables at:
	https://linuxtv.org/downloads/v4l-dvb-apis/subdev.html#v4l2-mbus-format

For those tables to look nice, we should be able to remove borders and grids
from the table. I was unable to find a way to control the tables format
with RST to do things like grid/border removal.

> > Are the Sphinx output formats adequate in general? Specifically, is the
> > lack of DocBook support, and the flexibility it provides, a blocker?  
> 
> DocBook is a means to an end; nobody really wants DocBook itself as far
> as I can tell. I've been messing with rst2pdf a bit; it's not hard to get
> reasonable output, and, with some effort, we could probably get really
> nice output. HTML and EPUB are easily covered, still haven't really played
> around with man pages yet.  And there's LaTeX if we really need it.  I
> kind of think we're covered there, unless I've missed something?
> 
> > Otherwise, I think Sphinx is promising.
> > 
> > Jon, I think we need a roll of dice, err, a well-thought-out decision
> > from the maintainer to go with one or the other, so we can make some
> > real progress.  
> 
> My inclination at the moment is very much in the Sphinx direction.  I had
> some vague thoughts of pushing a throwaway experimental directory with a
> couple of docs for 4.6 that would just let people play with it easily;
> then we'd see how many screams we get.  We'll see if the world lets me get
> there.

I'm not against having a staging/Documentation for us to play with,
provided, of course, that whatever tool chosen would allow converting
what we have today.

Regards,
Mauro

[toc] | [prev] | [next] | [standalone]

#1349687

[Multipart message — attachments visible in raw view] — view raw

Mauro Carvalho Chehab <mchehab@osg.samsung.com> writes:

> On my tests, Sphinix seemed too limited to format tables. Asciidoc
> produced an output that worked better.

Yes, asciidoc has much more flexibility in table formatting, including
the ability to control text layout within cells and full control over
borders.

However, I think asciidoc has two serious problems:

  1) the python version (asciidoc) appears to have been abandoned in
     favor of the ruby version. 

  2) It really is just a docbook pre-processor. Native html/latex output
     is poorly supported at best, and exposes only a small subset of the
     full capabilities of the input language.

As such, we would have to commit to using the ruby version and either
committing to fixing the native html output backend or continuing to use
the rest of the docbook toolchain.

We could insist on using the python version, of course. I spent a bit of
time hacking that up to add 'real' support for a table-of-contents in
the native HTML backend and it looks like getting those changes
upstreamed would be reasonably straightforward. However, we'd end up
'owning' the code, and I'm not sure we want to.

-- 
-keith

[toc] | [prev] | [next] | [standalone]

#1349753

[Multipart message — attachments visible in raw view] — view raw

Em Thu, 03 Mar 2016 15:23:23 -0800
Keith Packard <keithp@keithp.com> escreveu:

> Mauro Carvalho Chehab <mchehab@osg.samsung.com> writes:
> 
> > On my tests, Sphinix seemed too limited to format tables. Asciidoc
> > produced an output that worked better.  
> 
> Yes, asciidoc has much more flexibility in table formatting, including
> the ability to control text layout within cells and full control over
> borders.
> 
> However, I think asciidoc has two serious problems:
> 
>   1) the python version (asciidoc) appears to have been abandoned in
>      favor of the ruby version. 
> 
>   2) It really is just a docbook pre-processor. Native html/latex output
>      is poorly supported at best, and exposes only a small subset of the
>      full capabilities of the input language.
> 
> As such, we would have to commit to using the ruby version and either
> committing to fixing the native html output backend or continuing to use
> the rest of the docbook toolchain.
> 
> We could insist on using the python version, of course. I spent a bit of
> time hacking that up to add 'real' support for a table-of-contents in
> the native HTML backend and it looks like getting those changes
> upstreamed would be reasonably straightforward. However, we'd end up
> 'owning' the code, and I'm not sure we want to.

I'm a way more concerned about using a tool that fulfill our needs
than to look for something that won't use the docbook toolchain or
require to install ruby.

In the case of Docbook, we know it works and we know already its
issues. Please correct me if I'm wrong, but the big problem we
have is not due to the DocBook toolchain, but due to the lack of
features at the kernel-doc script. Also, xmlto is already installed
by the ones that build the kernel docs. So, keeping use it won't
require to install a weird toolchain by hand.

So, to be frank, it doesn't scary me to use either pyhton or
ruby script + docbook.

Of course, having to own the code has a cost that should be evaluated.

If, on the other hand, we decide to use RST, we'll very likely need to
patch it to fulfill our needs in order to add proper table support.
I've no idea how easy/difficult would be to do that, nor if Sphinx
upstream would accept such changes.

So, at the end of the day, we may end by having to carry on our own
version of Sphinx inside our tree, with doesn't sound good, specially
since it is not just a script, but a package with hundreds of
files.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1349937

On Fri, 04 Mar 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
> Em Thu, 03 Mar 2016 15:23:23 -0800
> Keith Packard <keithp@keithp.com> escreveu:
>
>> Mauro Carvalho Chehab <mchehab@osg.samsung.com> writes:
>> 
>> > On my tests, Sphinix seemed too limited to format tables. Asciidoc
>> > produced an output that worked better.  
>> 
>> Yes, asciidoc has much more flexibility in table formatting, including
>> the ability to control text layout within cells and full control over
>> borders.
>> 
>> However, I think asciidoc has two serious problems:
>> 
>>   1) the python version (asciidoc) appears to have been abandoned in
>>      favor of the ruby version. 
>> 
>>   2) It really is just a docbook pre-processor. Native html/latex output
>>      is poorly supported at best, and exposes only a small subset of the
>>      full capabilities of the input language.
>> 
>> As such, we would have to commit to using the ruby version and either
>> committing to fixing the native html output backend or continuing to use
>> the rest of the docbook toolchain.
>> 
>> We could insist on using the python version, of course. I spent a bit of
>> time hacking that up to add 'real' support for a table-of-contents in
>> the native HTML backend and it looks like getting those changes
>> upstreamed would be reasonably straightforward. However, we'd end up
>> 'owning' the code, and I'm not sure we want to.
>
> I'm a way more concerned about using a tool that fulfill our needs
> than to look for something that won't use the docbook toolchain or
> require to install ruby.

I think you meant that to be the other way round, or I fail at parsing
you. ;)

> In the case of Docbook, we know it works and we know already its
> issues. Please correct me if I'm wrong, but the big problem we
> have is not due to the DocBook toolchain, but due to the lack of
> features at the kernel-doc script. Also, xmlto is already installed
> by the ones that build the kernel docs. So, keeping use it won't
> require to install a weird toolchain by hand.

I think kernel-doc is just a small part of the puzzle. It's a problem,
but a small one at that, and we've already made it output asciidoc, rst
and docbook as part of this exercise. For real, as in code, not as in
talk.

The reasons I'm involved in this is that I want to make writing
documentation and rich kernel-doc comments easier (using lightweight
markup) and I want to make building the documentation easier (using a
straightforward toolchain with not too many dependencies). I'm hoping
the former makes writing documentation more attractive and the latter
keeps the documentation and the toolchain in a better shape through
having more people actually build the documentation.

IMHO docbook is problematic because the toolchain gets too long and
fragile. You need plenty of tools installed to build the documentation,
it's fussy to get working, and people just won't. Like code,
documentation bitrots too when it's not used. The documentation build is
broken too often. Debugging formatting issues through the entire
pipeline gets hard; I already faced some of this when playing with the
kernel-doc->asciidoc->docbook->html chain.

In short, I don't think the docbook toolchain fills all of our needs
either.

> So, to be frank, it doesn't scary me to use either pyhton or
> ruby script + docbook.
>
> Of course, having to own the code has a cost that should be evaluated.
>
> If, on the other hand, we decide to use RST, we'll very likely need to
> patch it to fulfill our needs in order to add proper table support.
> I've no idea how easy/difficult would be to do that, nor if Sphinx
> upstream would accept such changes.
>
> So, at the end of the day, we may end by having to carry on our own
> version of Sphinx inside our tree, with doesn't sound good, specially
> since it is not just a script, but a package with hundreds of
> files.

If we end up having to modify Sphinx, it has a powerful extension
mechanism for this. We wouldn't have to worry about getting it merged to
Sphinx upstream, and we wouldn't have to carry a local version of all of
Sphinx. (In fact, the extension mechanism provides a future path for
doing kernel-doc within Sphinx instead of as a preprocessing step.)

I know none of this alleviates your concerns with table supports right
now. I'll try to have a look at that a bit more.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1349959

On Fri, Mar 04, 2016 at 10:29:08AM +0200, Jani Nikula wrote:
> On Fri, 04 Mar 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
> >
> > If, on the other hand, we decide to use RST, we'll very likely need to
> > patch it to fulfill our needs in order to add proper table support.
> > I've no idea how easy/difficult would be to do that, nor if Sphinx
> > upstream would accept such changes.
> >
> > So, at the end of the day, we may end by having to carry on our own
> > version of Sphinx inside our tree, with doesn't sound good, specially
> > since it is not just a script, but a package with hundreds of
> > files.
> 
> If we end up having to modify Sphinx, it has a powerful extension
> mechanism for this. We wouldn't have to worry about getting it merged to
> Sphinx upstream, and we wouldn't have to carry a local version of all of
> Sphinx. (In fact, the extension mechanism provides a future path for
> doing kernel-doc within Sphinx instead of as a preprocessing step.)
> 
> I know none of this alleviates your concerns with table supports right
> now. I'll try to have a look at that a bit more.

FWIW, I think table formatting in Sphinx works via style sheets.
The mechanism is documented in the Python docutils docs that
Sphinx is built upon.
Basically you use the "class" or "role" directive and define
the corresponding CSS or LaTeX (or rst2pdf) style.

Here is one example (using a custom "cssclass" role):
https://pythonhosted.org/sphinxjp.themes.basicstrap/sample.html

Directives (especially role and class):
http://www.sphinx-doc.org/en/stable/rest.html#directives

LaTeX styling:
http://docutils.readthedocs.org/en/sphinx-docs/user/latex.html#custom-interpreted-text-roles


HTH,
Johannes

[toc] | [prev] | [next] | [standalone]

#1349911

[Multipart message — attachments visible in raw view] — view raw

On Thu, 2016-03-03 at 15:23 -0800, Keith Packard wrote:
> 
[…]
> However, I think asciidoc has two serious problems:
> 
>   1) the python version (asciidoc) appears to have been abandoned in
>      favor of the ruby version. 

This is I think true, however the Java-based tool chain Asciidoctor is
I believe the standard bearer for ASCIIdoc these days, albeit called
ASCIIdoctor.

>   2) It really is just a docbook pre-processor. Native html/latex
> output
>      is poorly supported at best, and exposes only a small subset of
> the
>      full capabilities of the input language.

This is not true. Yes ASCIIDoc started as a DocBook/XML frontend so as
to use a sane :-) markup language rather than XML (XML is a notation
for consenting computers only), but the current ASCIIDoctor toolchain
deals very well in direct HTML and PDF generation, without needing a
DocBook/XML toolchain. 

> As such, we would have to commit to using the ruby version and either
> committing to fixing the native html output backend or continuing to
> use
> the rest of the docbook toolchain.

Or trial the JVM-based ASCIIdoctor which is what the projects I am
involved with chose to use. Perhaps as an example I can give you http:/
/gpars.website (it's a redirector) all the HTML and PDF is generated
from ASCIIDoc source using ASCIIDoctor driven with a Gradle build
system. This is still very much a work in progress (by Jim Northrop,
not me currently), but I like it.

> We could insist on using the python version, of course. I spent a bit
> of
> time hacking that up to add 'real' support for a table-of-contents in
> the native HTML backend and it looks like getting those changes
> upstreamed would be reasonably straightforward. However, we'd end up
> 'owning' the code, and I'm not sure we want to.

If the Python version is really not being maintained, I would suggest
that unless you want to take over the project and be it's maintainer,
you would be better advised to use a different version.

-- Russel.=============================================================================Dr Russel Winder      t: +44 20 7585 2200   voip: sip:russel.winder@ekiga.net41 Buckmaster Road    m: +44 7770 465 077   xmpp: russel@winder.org.ukLondon SW11 1EN, UK   w: www.russel.org.uk  skype: russel_winder

[toc] | [prev] | [next] | [standalone]

#1349919

On Fri, 04 Mar 2016, Russel Winder <russel@winder.org.uk> wrote:
> On Thu, 2016-03-03 at 15:23 -0800, Keith Packard wrote:
>>   1) the python version (asciidoc) appears to have been abandoned in
>>      favor of the ruby version. 
>
> This is I think true, however the Java-based tool chain Asciidoctor is
> I believe the standard bearer for ASCIIdoc these days, albeit called
> ASCIIdoctor.

If we're talking about the same asciidoctor (http://asciidoctor.org/)
it's written in ruby but you can apparently run it in JVM using
JRuby. Calling it Java-based is misleading.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web