Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1433218 > unrolled thread

Re: [GIT PULL] doc: sphinx-4.8 DocBook to reST movement on Jon's docs-next

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [GIT PULL] doc: sphinx-4.8 DocBook to reST movement on Jon's docs-next Jani Nikula <jani.nikula@intel.com> - 2016-06-28 21:10 +0200
    Re: [GIT PULL] doc: sphinx-4.8 DocBook to reST movement on Jon's docs-next Markus Heiser <markus.heiser@darmarit.de> - 2016-06-29 13:10 +0200
      Re: [GIT PULL] doc: sphinx-4.8 DocBook to reST movement on Jon's docs-next Jani Nikula <jani.nikula@intel.com> - 2016-06-29 15:20 +0200
        Re: [GIT PULL] doc: sphinx-4.8 DocBook to reST movement on Jon's docs-next Markus Heiser <markus.heiser@darmarit.de> - 2016-06-29 18:50 +0200

#1433218 — Re: [GIT PULL] doc: sphinx-4.8 DocBook to reST movement on Jon's docs-next

On Tue, 28 Jun 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
> Hi Jonathan, hi Mauro,
>
> here is my DocBook to reST movement on top of Jon's docs-next branch. It includes:
>
> * kernel-doc parser & directive
> * flat-table directive
> * man page builder 'kernel-doc-man'
> * the kernel-doc-HOWTO
> * guides for starters (reST-nano-HOWTO.rst & template-book)
> * A index-page which bundles the lose Documentation/*.rst files.
> * sub sphinx-projects aka *books* under Documentation/*/conf.py
> * a full DocBook-XML to reST migration (*books* matching
>  Documentation/books_migrated/*/conf.py) described in [2]. The migration is
>  based on the DocBook-XML content at commit 17defc2 from Johnatan's docs-next
>  branch.
>
> I decided to comment out the pdf creation in the Makefile.reST, rst2pdf is to
> fragil see [2] (pdf is WIP).
>
> The output of html and man is included in the common %doc targets, to get
> more information about reST builds use:
>
>   make books-help 
>
> Any comments are welcome.

Perhaps you misunderstood, I don't know. When we ask you to rebase your
work on something, in this case docs-next, it generally means, accept
what is there, and iteratively build and extend upon it, *not* rip out
and rewrite everything from scratch.

Several people have spent a non-insignificant amount of time to review
and polish what is in docs-next currently. We've converted gpu
documentation on top, in drm-next, and set up autobuilders for it. We've
ironed out python2 vs python3 issues. We've fixed kernel-doc comments
here and there. Written documentation for the whole thing. Generally
tested the stuff in various environments. Etc, etc. That is the baseline
now, and it should be improved on iteratively, not destructively. That's
just sane engineering.

On the actual content (and really, this is orthogonal to the above),
I've repeatedly told you that I disagree with your approach to having
several configuration files, having the distinction between "books" and
other files, rewriting kernel-doc in python, having both rst and
"vintage" kernel-doc comments, converting all the docbook files in one
big lump. I won't repeat my rationale here, I've said it all before, but
sadly I don't see any of that addressed.

IMHO the most productive thing you could do right now is to send out
just the "flat table directive" patch. That's not controversial, and
would still have a chance to make it to v4.8.

BR,
Jani.

>
> -- Markus --
>
> [1] http://return42.github.io/sphkerneldoc/articles/dbtools.html
> [2] https://github.com/rst2pdf/rst2pdf/issues/556#issuecomment-228779542
>
>
> The following changes since commit 17defc282fe6e6ac93edbad8873ce89ef86b2490:
>
>  Documentation: add meta-documentation for Sphinx and kernel-doc (2016-06-24 06:55:28 -0600)
>
> are available in the git repository at:
>
>  https://github.com/return42/linux.git docs-next/linux-doc-reST
>
> for you to fetch changes up to 81bd813a599f8582570a735302ab660e20c7f442:
>
>  doc-rst: full DocBook-XML to reST migration (docs-next) (2016-06-28 17:15:38 +0200)
>
> ----------------------------------------------------------------
>
> Markus Heiser (15):
>      python: add scripts/site-packages
>      kernel-doc-HOWTO: add kernel-doc specification
>      doc-rst: add python package linuxdoc
>      doc-rst: kernel-doc parser - inital python implementation
>      doc-rst: kernel-doc directive - initial implementation
>      doc-rst: flat-table directive - initial implementation
>      doc-rst: kernel-doc-man sphinx builder - initial implementation
>      doc-rst: add basic sphinx-build infrastructure
>      doc-rst: add template book "Get started with reST"
>      doc-rst: removed Jani's kernel-documentation.rst and index.rst
>      doc-rst: infrastructure for *loose reST articles*
>      doc-rst: add build-html decription to book-help.
>      doc-rst: kernel-doc parser assert absolute pathname
>      doc-rst: infrastructure for migrated DocBook-XML
>      doc-rst: full DocBook-XML to reST migration (docs-next)
>

-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [next] | [standalone]

#1433675

Am 28.06.2016 um 21:05 schrieb Jani Nikula <jani.nikula@intel.com>:

> On Tue, 28 Jun 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
>> Hi Jonathan, hi Mauro,
>> 
>> here is my DocBook to reST movement on top of Jon's docs-next branch. It includes:
>> 
>> * kernel-doc parser & directive
>> * flat-table directive
>> * man page builder 'kernel-doc-man'
>> * the kernel-doc-HOWTO
>> * guides for starters (reST-nano-HOWTO.rst & template-book)
>> * A index-page which bundles the lose Documentation/*.rst files.
>> * sub sphinx-projects aka *books* under Documentation/*/conf.py
>> * a full DocBook-XML to reST migration (*books* matching
>> Documentation/books_migrated/*/conf.py) described in [2]. The migration is
>> based on the DocBook-XML content at commit 17defc2 from Johnatan's docs-next
>> branch.
>> 
>> I decided to comment out the pdf creation in the Makefile.reST, rst2pdf is to
>> fragil see [2] (pdf is WIP).
>> 
>> The output of html and man is included in the common %doc targets, to get
>> more information about reST builds use:
>> 
>>  make books-help 
>> 
>> Any comments are welcome.
> 
> Perhaps you misunderstood, I don't know. When we ask you to rebase your
> work on something, in this case docs-next, it generally means, accept
> what is there, and iteratively build and extend upon it, *not* rip out
> and rewrite everything from scratch.
> 
> Several people have spent a non-insignificant amount of time to review
> and polish what is in docs-next currently. We've converted gpu
> documentation on top, in drm-next, and set up autobuilders for it. We've
> ironed out python2 vs python3 issues. We've fixed kernel-doc comments
> here and there. Written documentation for the whole thing. Generally
> tested the stuff in various environments. Etc, etc. That is the baseline
> now, and it should be improved on iteratively, not destructively. That's
> just sane engineering.
> 
> On the actual content (and really, this is orthogonal to the above),
> I've repeatedly told you that I disagree with your approach to having
> several configuration files, having the distinction between "books" and
> other files, rewriting kernel-doc in python, having both rst and
> "vintage" kernel-doc comments, converting all the docbook files in one
> big lump. I won't repeat my rationale here, I've said it all before, but
> sadly I don't see any of that addressed.
> 
> IMHO the most productive thing you could do right now is to send out
> just the "flat table directive" patch. That's not controversial, and
> would still have a chance to make it to v4.8.
> 
> BR,
> Jani.


Take it as what it is: 

   a complete replacement of the XML toolchain.

IMHO, most of what you mentioned are assumptions, so my first 
question is:    

   have you pulled and tested?

To be more concrete, *here* is what I tested on drm-next:

  git checkout -b test123 airlied/drm-next 
  git pull  https://github.com/return42/linux.git docs-next/linux-doc-reST

To complete the merge:

  $ mv Documentation/books_migrated/gpu/conf.py Documentation/gpu
  $ rm -rf Documentation/books_migrated/gpu 
  
  In the Documentation/gpu/conf.py drop the line: --> kernel_doc_mode = "kernel-doc"
  In the Documentation/index.rst drop the line:   -->     gpu/index

  $ make books/gpu.html

... perfect ... 

and now some additional new features:

1.) If you want to see all the "Oops" of your document in one point,
    add the ".. todolist::" directive in the gpu/index.rst :


2.) If you want to create manpages, add option ":man-sect: <man sect-no>"
    to these ".. kernel-doc::" directives e.g:

  <reST> ---------
  .. kernel-doc:: drivers/gpu/drm/i915/i915_irq.c
     :functions: intel_irq_init intel_irq_init_hw intel_hpd_init
     :man-sect: 9
  <reST> ---------

  $ make books/gpu.man
  ...
  writing man pages ... intel_irq_init.9 
  build succeeded, 43 warnings.
  $ man -l Documentation/dist/books/gpu/man/intel_irq_init.9.gz 

  ... perfect ...

3.) If you want to suppress headings of "DOC:" sections, add 
    option ":no-header:" to the kernel-doc directives selecting "DOC:",
    do not suppress headings on declarations, this might end in unwanted
    man-pages constructs.

BTW: You have to accept, that reST is a structured markup, the readers,
writers, translators and builders working with these structures. Inline
bold-faces like the ones, the kernel-doc perl script produce:

    print "**Parameters**\n\n";
    ..
    print "**$section**\n\n";

is more "painting a document", where a section structure (heading) is required.

> I've repeatedly told you that I disagree with your approach to having
> several configuration files, having the distinction between "books" and
> other files,

You want to build one great sphinx-project, a monolith where all is in.
I build those monoliths in the past and it was my fail. The roundtrips 
of those monoliths increase, while the user acceptance decrease and 
this is not a assumption, this is my practical experience from the
last 8 years with sphinx and reST. 

> having both rst and
> "vintage" kernel-doc comments,

your scope is reduced on your single use-case, it is a parser option,
the default is "reST", so if you don't need a *vintage* mode simply
ignore it ... (see above and read the kernel-doc-HOWTO).

> converting all the docbook files in one
> big lump.

We don't have to. I shipped them in the "books_migration" folder for
your tests and to see how all current documentation will be build.
I also placed a README.txt in this folder ... read it.
John could drop this folder, but others like Mauro want to base on this
migration.

Now we can test the toolchain against the complete docs, have
you ever done this? ... now you could build your monolith (see below)
and test the roundtrips in practice ...

The py-parser you fear about, leafs the XML toolchain untouched, is more strict, 
then the perl one and gives more warnings about all these buggy source code 
comments ... and with the "NullTranslator" you get a great lint tool for free.

I studied all your's and John's patches on the perl one and checked if
the python one need a correction.

BTW: it parses only once and shows errors only ones (not n-times). 

> I've said it all before, but
> sadly I don't see any of that addressed.

I addressed all your points and tried to evaluate solutions that match
(e.g. loose-text files, reST mode as default etc), but this does not imply, 
that all your answers get a 1:1 implementation. 

I remember about chunking large files, your answers "no, it breaks the
story" and with commit 2fa91d1 you start to chunk ... 

now the discussion about sub-projects ... my recommendations comes from 
a 8 year ongoing practice so please honor them a bit.

I also ask you for a close collaboration, but you didn't want. I'am at
the end of my ideas ... 

> IMHO the most productive thing you could do right now is to send out
> just the "flat table directive" patch. That's not controversial, and
> would still have a chance to make it to v4.8.

Since your solution is not a full replacement (no man-page builder) and 
does not produce structural markup .... 

IMHO pull my solution, if you have any remarks, let me know. E.g if
you think it is better to use ":no-header:" as default on DOC: 
sections, I implement it and test it against the complete docs.

-- Markus --




> 
>> 
>> -- Markus --
>> 
>> [1] http://return42.github.io/sphkerneldoc/articles/dbtools.html
>> [2] https://github.com/rst2pdf/rst2pdf/issues/556#issuecomment-228779542
>> 
>> 
>> The following changes since commit 17defc282fe6e6ac93edbad8873ce89ef86b2490:
>> 
>> Documentation: add meta-documentation for Sphinx and kernel-doc (2016-06-24 06:55:28 -0600)
>> 
>> are available in the git repository at:
>> 
>> https://github.com/return42/linux.git docs-next/linux-doc-reST
>> 
>> for you to fetch changes up to 81bd813a599f8582570a735302ab660e20c7f442:
>> 
>> doc-rst: full DocBook-XML to reST migration (docs-next) (2016-06-28 17:15:38 +0200)
>> 
>> ----------------------------------------------------------------
>> 
>> Markus Heiser (15):
>>     python: add scripts/site-packages
>>     kernel-doc-HOWTO: add kernel-doc specification
>>     doc-rst: add python package linuxdoc
>>     doc-rst: kernel-doc parser - inital python implementation
>>     doc-rst: kernel-doc directive - initial implementation
>>     doc-rst: flat-table directive - initial implementation
>>     doc-rst: kernel-doc-man sphinx builder - initial implementation
>>     doc-rst: add basic sphinx-build infrastructure
>>     doc-rst: add template book "Get started with reST"
>>     doc-rst: removed Jani's kernel-documentation.rst and index.rst
>>     doc-rst: infrastructure for *loose reST articles*
>>     doc-rst: add build-html decription to book-help.
>>     doc-rst: kernel-doc parser assert absolute pathname
>>     doc-rst: infrastructure for migrated DocBook-XML
>>     doc-rst: full DocBook-XML to reST migration (docs-next)
>> 
> 
> -- 
> Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1433722

On Wed, 29 Jun 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
> Am 28.06.2016 um 21:05 schrieb Jani Nikula <jani.nikula@intel.com>:
>> Perhaps you misunderstood, I don't know. When we ask you to rebase your
>> work on something, in this case docs-next, it generally means, accept
>> what is there, and iteratively build and extend upon it, *not* rip out
>> and rewrite everything from scratch.
>> 
>> Several people have spent a non-insignificant amount of time to review
>> and polish what is in docs-next currently. We've converted gpu
>> documentation on top, in drm-next, and set up autobuilders for it. We've
>> ironed out python2 vs python3 issues. We've fixed kernel-doc comments
>> here and there. Written documentation for the whole thing. Generally
>> tested the stuff in various environments. Etc, etc. That is the baseline
>> now, and it should be improved on iteratively, not destructively. That's
>> just sane engineering.

Please just read the above again, and try to let it sink in.

>> On the actual content (and really, this is orthogonal to the above),
>> I've repeatedly told you that I disagree with your approach to having
>> several configuration files, having the distinction between "books" and
>> other files, rewriting kernel-doc in python, having both rst and
>> "vintage" kernel-doc comments, converting all the docbook files in one
>> big lump. I won't repeat my rationale here, I've said it all before, but
>> sadly I don't see any of that addressed.

> Take it as what it is: 
>
>    a complete replacement of the XML toolchain.
>
> IMHO, most of what you mentioned are assumptions, so my first 
> question is:    
>
>    have you pulled and tested?

I have looked at the patches and read the commit messages. If your work
was reasonably based on docs-next, iteratively improving on what is
there, we could have a sensible discussion on the relative merits of
each commit and change. Now, it all depends on being a full rewrite. It
depends on throwing out everything we've done so far. There isn't a
single commit that's a change or an improvement to existing code.

I'm obviously biased because I've done the bulk of the Sphinx work in
docs-next. Your pull request feels like a complicated way to tell me you
think it's all crap. I'll try to let that slide.

> Since your solution is not a full replacement (no man-page builder) and 
> does not produce structural markup .... 
>
> IMHO pull my solution, if you have any remarks, let me know. E.g if
> you think it is better to use ":no-header:" as default on DOC: 
> sections, I implement it and test it against the complete docs.

You have plenty of good stuff in there. The annoying thing is that you
present it in a take-it-all-or-leave-it-all kind of way.

For example, it would be trivial to add the flat table directive to the
existing configuration file, but instead you opt to rewrite the entire
file. You could base your kernel-doc directive extension on the existing
one, but instead you rewrite it. And then add it in the configuration
file rewrite. And so on and so on.

I'll need to focus on other things now, and it's a good time to let
others chime in. It would have been nice to see iterative improvements
from you. Perhaps they could have made it to v4.8, the merge window
being just a few weeks away.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1433909

Am 29.06.2016 um 15:15 schrieb Jani Nikula <jani.nikula@intel.com>:

> On Wed, 29 Jun 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
>> Am 28.06.2016 um 21:05 schrieb Jani Nikula <jani.nikula@intel.com>:
>>> Perhaps you misunderstood, I don't know. When we ask you to rebase your
>>> work on something, in this case docs-next, it generally means, accept
>>> what is there, and iteratively build and extend upon it, *not* rip out
>>> and rewrite everything from scratch.
>>> 
>>> Several people have spent a non-insignificant amount of time to review
>>> and polish what is in docs-next currently. We've converted gpu
>>> documentation on top, in drm-next, and set up autobuilders for it. We've
>>> ironed out python2 vs python3 issues. We've fixed kernel-doc comments
>>> here and there. Written documentation for the whole thing. Generally
>>> tested the stuff in various environments. Etc, etc. That is the baseline
>>> now, and it should be improved on iteratively, not destructively. That's
>>> just sane engineering.
> 
> Please just read the above again, and try to let it sink in.

I know what you mean, read on about this "unlucky situation" ..

> 
>>> On the actual content (and really, this is orthogonal to the above),
>>> I've repeatedly told you that I disagree with your approach to having
>>> several configuration files, having the distinction between "books" and
>>> other files, rewriting kernel-doc in python, having both rst and
>>> "vintage" kernel-doc comments, converting all the docbook files in one
>>> big lump. I won't repeat my rationale here, I've said it all before, but
>>> sadly I don't see any of that addressed.
> 
>> Take it as what it is: 
>> 
>>  a complete replacement of the XML toolchain.
>> 
>> IMHO, most of what you mentioned are assumptions, so my first 
>> question is:    
>> 
>>  have you pulled and tested?
> 
> I have looked at the patches and read the commit messages. If your work
> was reasonably based on docs-next, iteratively improving on what is
> there, we could have a sensible discussion on the relative merits of
> each commit and change. Now, it all depends on being a full rewrite. It
> depends on throwing out everything we've done so far. There isn't a
> single commit that's a change or an improvement to existing code.
> 
> I'm obviously biased because I've done the bulk of the Sphinx work in
> docs-next. Your pull request feels like a complicated way to tell me you
> think it's all crap. I'll try to let that slide.
> 
>> Since your solution is not a full replacement (no man-page builder) and 
>> does not produce structural markup .... 
>> 
>> IMHO pull my solution, if you have any remarks, let me know. E.g if
>> you think it is better to use ":no-header:" as default on DOC: 
>> sections, I implement it and test it against the complete docs.
> 
> You have plenty of good stuff in there. The annoying thing is that you
> present it in a take-it-all-or-leave-it-all kind of way.

Yes, this was my mistake I'am sorry about.

I started 4 or 5 month ago with the migration in my POC [1] I posted
it on the ML, to show those who had doubt in reST & sphinx that it is
the right decision to switch from DocBook to reST. But the only one how
gives feedback about this was Mauro. We discussed some points and he
explained me some of the developer's requirements, this was productive
and the flat-table is one of the results.

My mistake: after I finished the migration, I continued to
develop the sphinx extensions in the POC and not on the docs-next.
Then I switched to base on a 4.7rc tag... yes, complete chaotic 
Great mistake of mine not to base on the docs-next from the beginning.
... This is my first work in the kernel community and I was not familiar
with the organizational structures. I'am very sorry about and I will
improve myself on this.

In the further course you made huge steps forward and I thought
it is the best to let you work to get productive on that what you
have. While you needed to get productive, I needed more time,
it was "unlucky situation" ... 

side node: It is not really a good alternative, but the history is
present in the POC [1]. E.g. here is one patch from you I ported
to the python kernel-doc module: 

 https://github.com/return42/sphkerneldoc/commit/9ac8fc9023400d26a6a0b6e7f741e1bf788d2326

Yes, I read through all your patches, made tests with your perl
script and compared it with the python one ... I got huge
benefit from your patches -- the situation and my strategy
was unlucky, but your investigation is not lost! -- and
to fix kernel-doc comments is always good.

> For example, it would be trivial to add the flat table directive to the
> existing configuration file, but instead you opt to rewrite the entire
> file.

The config file "replacement" is because, that my configuration
with sub-project is different to the config file from the sphinx 
template. I also added more comments, dropped settings who are 
not relevant and I moved settings like project, author etc. to the
top ... but forgot to drop my latex settings .. not really perfect.

> You could base your kernel-doc directive extension on the existing
> one, but instead you rewrite it.

No this is not possible. After I realized, that I'am at the end with
the perl script and I needed a API to the kernel-doc parser I started
the kernel_doc.py rewrite ... so the kernel-doc directive of mine 
uses a kernel_doc.py API instead of pipes and a command line, thats
why the two implementation are so different.

> And then add it in the configuration
> file rewrite. And so on and so on.
> 
> I'll need to focus on other things now, and it's a good time to let
> others chime in. It would have been nice to see iterative improvements
> from you. Perhaps they could have made it to v4.8, the merge window
> being just a few weeks away.

Mauro will start to migrate, I think we need this time to get more
practice ... 

Regards

 -- Markus --


[1] https://github.com/return42/sphkerneldoc


> 
> BR,
> Jani.
> 
> 
> -- 
> Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web