Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1404432 > unrolled thread

[PATCH 00/10] Documentation/Sphinx

Back to article view | Back to linux.kernel

Contents

  [PATCH 00/10] Documentation/Sphinx Jani Nikula <jani.nikula@intel.com> - 2016-05-20 15:50 +0200
    [PATCH 08/10] Documentation: add kernel hacking rst Jani Nikula <jani.nikula@intel.com> - 2016-05-20 15:50 +0200
    [PATCH 03/10] Documentation/sphinx: add basic working Sphinx configuration and build Jani Nikula <jani.nikula@intel.com> - 2016-05-20 15:50 +0200
    [PATCH 09/10] Documentation: add kernel api rst Jani Nikula <jani.nikula@intel.com> - 2016-05-20 15:50 +0200
    [PATCH 04/10] Documentation: add .gitignore Jani Nikula <jani.nikula@intel.com> - 2016-05-20 15:50 +0200
    [PATCH 02/10] kernel-doc: support printing exported and non-exported symbols Jani Nikula <jani.nikula@intel.com> - 2016-05-20 15:50 +0200
    [PATCH 05/10] Documentation/sphinx: add Sphinx kernel-doc directive extension Jani Nikula <jani.nikula@intel.com> - 2016-05-20 15:50 +0200
      Re: [PATCH 05/10] Documentation/sphinx: add Sphinx kernel-doc  directive extension Jonathan Corbet <corbet@lwn.net> - 2016-06-03 22:40 +0200
        Re: [PATCH 05/10] Documentation/sphinx: add Sphinx kernel-doc directive extension Markus Heiser <markus.heiser@darmarit.de> - 2016-06-04 09:00 +0200
    [PATCH 06/10] Documentation/sphinx: configure the kernel-doc extension Jani Nikula <jani.nikula@intel.com> - 2016-05-20 15:50 +0200
    [PATCH 07/10] sphinx: cheesy script to convert .tmpl files Jani Nikula <jani.nikula@intel.com> - 2016-05-20 15:50 +0200
    [PATCH 01/10] kernel-doc: fix use of uninitialized value Jani Nikula <jani.nikula@intel.com> - 2016-05-20 15:50 +0200
    Re: [PATCH 00/10] Documentation/Sphinx Jani Nikula <jani.nikula@intel.com> - 2016-05-29 22:40 +0200
      Re: [PATCH 00/10] Documentation/Sphinx Daniel Vetter <daniel.vetter@ffwll.ch> - 2016-05-30 11:20 +0200
        Re: [PATCH 00/10] Documentation/Sphinx Markus Heiser <markus.heiser@darmarit.de> - 2016-05-30 12:50 +0200
          Re: [PATCH 00/10] Documentation/Sphinx Jani Nikula <jani.nikula@intel.com> - 2016-05-30 16:50 +0200
            Re: [PATCH 00/10] Documentation/Sphinx Daniel Vetter <daniel.vetter@ffwll.ch> - 2016-05-30 17:30 +0200
            Re: [PATCH 00/10] Documentation/Sphinx Markus Heiser <markus.heiser@darmarit.de> - 2016-05-30 18:50 +0200
              Re: [PATCH 00/10] Documentation/Sphinx Jani Nikula <jani.nikula@intel.com> - 2016-05-30 22:10 +0200
                Re: [PATCH 00/10] Documentation/Sphinx Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-05-30 23:30 +0200
                  Re: [PATCH 00/10] Documentation/Sphinx Markus Heiser <markus.heiser@darmarit.de> - 2016-05-31 12:20 +0200
                Re: [PATCH 00/10] Documentation/Sphinx Markus Heiser <markus.heiser@darmarit.de> - 2016-05-31 09:30 +0200
                  Re: [PATCH 00/10] Documentation/Sphinx Daniel Vetter <daniel.vetter@ffwll.ch> - 2016-05-31 10:10 +0200
                    Re: [PATCH 00/10] Documentation/Sphinx Markus Heiser <markus.heiser@darmarit.de> - 2016-05-31 11:50 +0200
                      Re: [PATCH 00/10] Documentation/Sphinx Jani Nikula <jani.nikula@intel.com> - 2016-05-31 12:40 +0200
                        Re: [PATCH 00/10] Documentation/Sphinx Markus Heiser <markus.heiser@darmarit.de> - 2016-05-31 13:20 +0200
                Re: rst2pdf (was [PATCH 00/10] Documentation/Sphinx) Jonathan Corbet <corbet@lwn.net> - 2016-06-03 22:50 +0200
                  Re: rst2pdf (was [PATCH 00/10] Documentation/Sphinx) Markus Heiser <markus.heiser@darmarit.de> - 2016-06-07 08:10 +0200
                    Re: rst2pdf (was [PATCH 00/10] Documentation/Sphinx) Jani Nikula <jani.nikula@intel.com> - 2016-06-07 08:50 +0200
                      Re: rst2pdf (was [PATCH 00/10] Documentation/Sphinx) Markus Heiser <markus.heiser@darmarit.de> - 2016-06-10 19:10 +0200
                Re: [PATCH 00/10] Documentation/Sphinx Jonathan Corbet <corbet@lwn.net> - 2016-06-03 23:10 +0200
                  Re: [PATCH 00/10] Documentation/Sphinx Daniel Vetter <daniel.vetter@ffwll.ch> - 2016-06-04 01:00 +0200
                  Re: [PATCH 00/10] Documentation/Sphinx Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
        Re: [PATCH 00/10] Documentation/Sphinx Jonathan Corbet <corbet@lwn.net> - 2016-06-01 03:10 +0200
          Re: [PATCH 00/10] Documentation/Sphinx Daniel Vetter <daniel.vetter@ffwll.ch> - 2016-06-01 08:50 +0200
    Re: [PATCH 00/10] Documentation/Sphinx Jonathan Corbet <corbet@lwn.net> - 2016-06-03 22:20 +0200
      Re: [PATCH 00/10] Documentation/Sphinx Jonathan Corbet <corbet@lwn.net> - 2016-06-03 22:30 +0200
        Re: [PATCH 00/10] Documentation/Sphinx Jani Nikula <jani.nikula@intel.com> - 2016-06-04 15:10 +0200
      Re: [PATCH 00/10] Documentation/Sphinx Daniel Vetter <daniel.vetter@ffwll.ch> - 2016-06-03 22:30 +0200
      Re: [PATCH 00/10] Documentation/Sphinx Jani Nikula <jani.nikula@intel.com> - 2016-06-04 15:00 +0200

Page 2 of 2 — ← Prev page 1 [2]

#1410076

Am 30.05.2016 um 23:23 schrieb Mauro Carvalho Chehab <mchehab@osg.samsung.com>:

> Em Mon, 30 May 2016 23:05:34 +0300
> Jani Nikula <jani.nikula@intel.com> escreveu:
> 
>>> I worry a little bit in that reST will be only one more toolchain 
>>> beside DocBook .. in the long term, kernel's documentation 
>>> should get rid of all the DocBook artifacts and for this a more
>>> comprehensive solution is needed.  
>> 
>> We agree on the end goal, eradicate DocBook. I must say that in my
>> experiments, apart from the media docs, almost everything converts
>> surprisingly nicely or IMO "good enough" with just the tmplcvt script in
>> this series.
> 
> With regards to media, my plan is to merge create a topic branch based
> on Kernel 4.7-rc1 at:
> 	https://git.linuxtv.org/media_tree.git/
> 
> As none of the Jani's patches seem to affect the media API docs, it
> seems I don't need to merge back from Jon's -next branch.
> 
> There, I intend to add Markus patches with the conversion from the
> DocBook to rst, plus the flat-table extension logic.
> 
> Then, I'll work to manually fix what's needed and I'll add the 
> automation scripting logic that we have at the DocBook Makefile
> to work with the new media rst files.
> 
> Lastly, once the job's done, I'll drop Documentation/DocBook/media.
> 
> Markus,
> 
> With that regards, could you please send the patches to me?

Yes. What is your timeline ... is it OK if I send you a patch in the 
next two weeks? ... first I wan't to finish my other work / I'am just
back from holiday .. a lot of work to do :-o

--Markus--

> 
> Thanks,
> Mauro

[toc] | [prev] | [next] | [standalone]

#1409925

Am 30.05.2016 um 22:05 schrieb Jani Nikula <jani.nikula@intel.com>:

> On Mon, 30 May 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
>> Am 30.05.2016 um 16:46 schrieb Jani Nikula <jani.nikula@intel.com>:
>>> I am not proposing to merge the documents that I've converted mostly as
>>> samples in the branch. I needed something to demonstrate the build is
>>> sane.
>> 
>>> The authors of the DocBook documents should make the conversions as they
>>> see fit, when they see fit, with the tools they see fit, probably with
>>> some manual work on top.
>> 
>> OK
> 
> To be clear, the "sphinx-for-docs-next" branch of [1], [2] is what I
> propose to merge at this time. There's the Sphinx configuration, kernel
> build integration, Sphinx kernel-doc extension, tons of kernel-doc
> updates, etc. There is no DocBook tmpl conversion; all of that is left
> to the authors (owners, maintainers) of the documents, but this enables
> them to focus on that part.
> 
> I was planning on sending out the patches after some feedback here.
> 
> [1] git://people.freedesktop.org/~jani/drm
> [2] https://cgit.freedesktop.org/~jani/drm/log/?h=sphinx-for-docs-next
> 
>> With DocBook, it was hard to separate a file into small chunks (see media for
>> how it is done). With Sphinx, it is common to split a document in small chunks 
>> (along parts, chapters, sections ...) Thats why I recommend chunking documents
>> (from the beginning).
> 
> Agreed, but up to the authors.
> 
>>> One of the goals was to have nice cross-referencing between the
>>> documents (e.g. from GPU to kernel or device driver API). And it works.
>> 
>> For this, Sphinx-doc brings intersphinx: http://www.sphinx-doc.org/en/stable/ext/intersphinx.html
> 
> If the kernel is split to several intersphinx "prefixes", we won't know
> the prefix of the link targets when we're generating the references in
> kernel-doc. Also, can be deferred to follow-up work if someone figures
> out the how.

It stands to reason that each book should be placed in a separate folder.
This simple role simplifies much, e.g. chunking, "prefixes" for the 
intersphinx, a place for images of this book .. etc. It is also the
base to have one sphinx-doc project (individual config set) for each book.

Please, place each (DocBook) reST-book at least into a separate folder.


>> I can't recommend to use rst2pdf (it is less maintained), use default
>> sphinx LaTeX toolchain.
> 
> I think we'll use whatever works, rst2pdf seemed to work for now, but we
> can change if needed.

The discussion in the past was dominated by the fear, that something
on the sphinx-doc could not be maintained in the future ... I don't
share theses fears, if needed, I also have no problem to repair or throw
my damaged toolchain away ;-)

>>> I find it totally unacceptable to require explicitly marking kernel-doc
>>> comments or source files as being reStructuredText.
>>> Note that it's all opt-in already. If you add a .rst file that includes
>>> kernel-doc via the kernel-doc extension, you better make sure the
>>> comments parse as reStructuredText and render nicely. I'm willing to do
>>> much of the job for all the things that I care about.
>> 
>> We have a different POV ... I try to build up a documentation project,
>> which could use all given kernel-doc markups without any change, where
>> reST is an "addition". Your approach is to fix kernel-doc comments 
>> if they are referred by a kernl-doc directive in a .rst document.
>> There is nothing wrong about your approach, but I try to build
>> a whole source code documentation like the one I started here: 
>> http://return42.github.io/sphkerneldoc/linux_src_doc/index.html
> 
> That looks nice, but I'll argue it would not be much worse even if you
> assumed it's all rst.

A superficial look on the HTML output may give the impression. But in
the log you will find tons of errors and warnings. My experience is,
that authors will not consult logs if there are tons of errors from the 
beginning, which carries a decrease in quality. IMO not a good starting 
point.

> The bigger point is, if you expect people to tag each source file or
> kernel-doc comment with "rst", you'll end up with a mess where some
> places have that tag, some not, but it's not conclusive about whether
> they actually *are* rst or not. (And you've had tons of patch churn to
> add those tags to get there.)

At the end, only sources which have been modified to reST need one
line (in the first lines) :

  /* parse-markup: reST */

which announce the reST markup in this file, normaly this needs
no additional patches, except the author forget to announce his
movement to reST ...

> The kernel-doc comments are written by humans who will screw it up
> anyway. (Apologies for the distrust, fellow developers, but I've been
> reading too many of your fine kernel-doc comments lately.) People will
> happily cargo cult rst and current kernel-doc and javadoc and doxygen
> and whatnot in a fruit salad. The only thing that will help in the end
> is keeping the rules simple and consistent and having the feedback from
> the tools.

You are right, I have seen tons of individual markups in the kernel-doc
comments. In the past some authors ignored the description in the 
kernel-doc-nano-howto. The "/* parse-markup: reST */" will be only one 
addition more to the kernel-doc-nano-howto they could ignore ;-)

>> I worry a little bit in that reST will be only one more toolchain 
>> beside DocBook .. in the long term, kernel's documentation 
>> should get rid of all the DocBook artifacts and for this a more
>> comprehensive solution is needed.
> 
> We agree on the end goal, eradicate DocBook. I must say that in my
> experiments, apart from the media docs, almost everything converts
> surprisingly nicely or IMO "good enough" with just the tmplcvt script in
> this series. Do remember that this is a one time conversion. It needs to
> be good enough that there's not too much manual editing involved, but it
> doesn't need to be perfect. Some degree of editing will be required no
> matter what, not least because the DocBook has also been written by
> humans, and the battle against the GIGO principle is a lost one.

and I feel like Don Quichotte :-) 

-- Markus--

[toc] | [prev] | [next] | [standalone]

#1409958

On Tue, May 31, 2016 at 9:27 AM, Markus Heiser
<markus.heiser@darmarit.de> wrote:
>>>> I find it totally unacceptable to require explicitly marking kernel-doc
>>>> comments or source files as being reStructuredText.
>>>> Note that it's all opt-in already. If you add a .rst file that includes
>>>> kernel-doc via the kernel-doc extension, you better make sure the
>>>> comments parse as reStructuredText and render nicely. I'm willing to do
>>>> much of the job for all the things that I care about.
>>>
>>> We have a different POV ... I try to build up a documentation project,
>>> which could use all given kernel-doc markups without any change, where
>>> reST is an "addition". Your approach is to fix kernel-doc comments
>>> if they are referred by a kernl-doc directive in a .rst document.
>>> There is nothing wrong about your approach, but I try to build
>>> a whole source code documentation like the one I started here:
>>> http://return42.github.io/sphkerneldoc/linux_src_doc/index.html
>>
>> That looks nice, but I'll argue it would not be much worse even if you
>> assumed it's all rst.
>
> A superficial look on the HTML output may give the impression. But in
> the log you will find tons of errors and warnings. My experience is,
> that authors will not consult logs if there are tons of errors from the
> beginning, which carries a decrease in quality. IMO not a good starting
> point.

0-day builds all docs, and checks for new warnings. Even in today's
gpu.tmpl build there's a massive pile of warnings, so yes developers
don't look. But 0-day does, and then developers look at the nice mails
from 0-day. It mostly works to keep out new fail I think.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

[toc] | [prev] | [next] | [standalone]

#1410043

Am 31.05.2016 um 10:07 schrieb Daniel Vetter <daniel.vetter@ffwll.ch>:

> On Tue, May 31, 2016 at 9:27 AM, Markus Heiser
> <markus.heiser@darmarit.de> wrote:
>>>>> I find it totally unacceptable to require explicitly marking kernel-doc
>>>>> comments or source files as being reStructuredText.
>>>>> Note that it's all opt-in already. If you add a .rst file that includes
>>>>> kernel-doc via the kernel-doc extension, you better make sure the
>>>>> comments parse as reStructuredText and render nicely. I'm willing to do
>>>>> much of the job for all the things that I care about.
>>>> 
>>>> We have a different POV ... I try to build up a documentation project,
>>>> which could use all given kernel-doc markups without any change, where
>>>> reST is an "addition". Your approach is to fix kernel-doc comments
>>>> if they are referred by a kernl-doc directive in a .rst document.
>>>> There is nothing wrong about your approach, but I try to build
>>>> a whole source code documentation like the one I started here:
>>>> http://return42.github.io/sphkerneldoc/linux_src_doc/index.html
>>> 
>>> That looks nice, but I'll argue it would not be much worse even if you
>>> assumed it's all rst.
>> 
>> A superficial look on the HTML output may give the impression. But in
>> the log you will find tons of errors and warnings. My experience is,
>> that authors will not consult logs if there are tons of errors from the
>> beginning, which carries a decrease in quality. IMO not a good starting
>> point.
> 
> 0-day builds all docs, and checks for new warnings. Even in today's
> gpu.tmpl build there's a massive pile of warnings, so yes developers
> don't look. But 0-day does, and then developers look at the nice mails
> from 0-day. It mostly works to keep out new fail I think.

In general, I'am not very happy with workarounds like this. IMO these
are workarounds are often, rewards bunglers and punish those with more work,
who want make thinks right. There might be situations where 0-day build
is the only/best solution. But *here* we are talking about one additional
comment line the author adds, when he modify his source comments from kernel-doc 
to reST markup .. IMO not very hard.

This one line helps the doc-builder to distinguish between *vintage* kernel-doc
comments and those with reST additions in.

With the announcement of the markup, we can use all existing kernel-doc
markups "as is" for building a complete src-code documentation, with
thousands of errors less in the log (experience from my POC). IMO a great
benefit ... given by just one additional comment line to distinguish between
vintage and reST ... 

BTW it's is not uncommon to announce the markup in projects with mixed 
markups in the source code comments.


     A few words about my point of view / my thought:

I strict separate markup from the doc-building tools. The decision in favor for
reST is not done because sphinx-doc is a great doc-bulding tool, it is done because
reST is a easy to write / read markup with a clear and expressive syntax definition.
The *best* builder for this markup is sphinx-doc, so it is only natural, that
the decision for the builder falls to sphinx-doc.

But kernel documentation is not a project from scratch, there was the *vintage*
kernel-doc markup first. Therefore, reST is only an additional markup!

With this POV you can add reST (or any other) markup to the doc-building process.
The Linux kernel draws on diverse projects, some of them may use different
markups and these maintainers are not interested in changing there whole markup
when they contribute to the kernel ... but may be there is someone how want's 
to add an other additional markup support to the doc-building process.

I'am not interested in supporting additional markups beside reST. But this is a
scalable future-proof solution, which only needs an announcement of the the markup.
The one additional comment line we are talking about.

--Markus--

> -Daniel
> -- 
> Daniel Vetter
> Software Engineer, Intel Corporation
> +41 (0) 79 365 57 48 - http://blog.ffwll.ch

[toc] | [prev] | [next] | [standalone]

#1410095

On Tue, 31 May 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
> Am 31.05.2016 um 10:07 schrieb Daniel Vetter <daniel.vetter@ffwll.ch>:
>> 0-day builds all docs, and checks for new warnings. Even in today's
>> gpu.tmpl build there's a massive pile of warnings, so yes developers
>> don't look. But 0-day does, and then developers look at the nice mails
>> from 0-day. It mostly works to keep out new fail I think.
>
> In general, I'am not very happy with workarounds like this. IMO these
> are workarounds are often, rewards bunglers and punish those with more work,
> who want make thinks right. There might be situations where 0-day build
> is the only/best solution. But *here* we are talking about one additional
> comment line the author adds, when he modify his source comments from kernel-doc 
> to reST markup .. IMO not very hard.

That "one line" translates to nearly 50000 kernel-doc comments in more
than 6000 files. If you expect people to add a tag in each file/comment,
it will never happen. If we assume it's all rst, we can at least start
converting.

I quickly wrote a small "kernel-doc-rst-lint" script (70 lines of
python) based on rst-lint [1] that runs kernel-doc on a file and reports
all the kernel-doc and rst-lint errors in the output. This can be run as
a "checker" in the kernel build with

$ make CHECKER=scripts/kernel-doc-rst-lint C=1

and it can provide better and more direct warnings on kernel-doc/rst
errors than a full Sphinx build does.

BR,
Jani.


[1] https://pypi.python.org/pypi/restructuredtext_lint



-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1410160

Am 31.05.2016 um 12:30 schrieb Jani Nikula <jani.nikula@intel.com>:

> On Tue, 31 May 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
>> Am 31.05.2016 um 10:07 schrieb Daniel Vetter <daniel.vetter@ffwll.ch>:
>>> 0-day builds all docs, and checks for new warnings. Even in today's
>>> gpu.tmpl build there's a massive pile of warnings, so yes developers
>>> don't look. But 0-day does, and then developers look at the nice mails
>>> from 0-day. It mostly works to keep out new fail I think.
>> 
>> In general, I'am not very happy with workarounds like this. IMO these
>> are workarounds are often, rewards bunglers and punish those with more work,
>> who want make thinks right. There might be situations where 0-day build
>> is the only/best solution. But *here* we are talking about one additional
>> comment line the author adds, when he modify his source comments from kernel-doc 
>> to reST markup .. IMO not very hard.
> 
> That "one line" translates to nearly 50000 kernel-doc comments in more
> than 6000 files. If you expect people to add a tag in each file/comment,
> it will never happen. If we assume it's all rst, we can at least start
> converting.

I have the impression that we misunderstand us ...

You will add this line only to these files where you have changed the 
markup from *vintage* kerenel-doc to reST. In my solution, you can
change the markup on every comment, but you don't have to .. it
is enough to add one line at the top of the file.

It's hard to describe something without an example, let my finish my
work and after this I can show it by example. Then you will see,
that the impact is less then you fear.

> I quickly wrote a small "kernel-doc-rst-lint" script (70 lines of
> python) based on rst-lint [1] that runs kernel-doc on a file and reports
> all the kernel-doc and rst-lint errors in the output. This can be run as
> a "checker" in the kernel build with
> 
> $ make CHECKER=scripts/kernel-doc-rst-lint C=1
> 
> and it can provide better and more direct warnings on kernel-doc/rst
> errors than a full Sphinx build does.

I haven't tested [1], but I assume that it covers only docutils-reST not
the Sphinx-doc superset (thats might be the reason why you see less errors)
... anyway it could be convenient tool.


--Markus--

PS: I looked closer to [1], it uses the docutils puplischer ..

  from docutils.core import Publisher

with a /dev/null like stream

  document.reporter.stream = None

The errors you get from this are the same you get from a rst2xxx 
tool ...

|  >>> import restructuredtext_lint
|  >>> errors = restructuredtext_lint.lint("""
|  ... Hello World
|  ... =======
|  ... 
|  ... :ref:`label_name` 
|  ... """)
|  >>> errors[0].astext()
|  u'None:3: (WARNING/2) Title underline too short.\n\nHello World\n======='
|  >>> errors[1].astext()
|  u'None:5: (INFO/1) No role entry for "ref" in module "docutils.parsers.rst.languages.en".\nTrying "ref" as canonical role name.'

> 
> BR,
> Jani.
> 
> 
> [1] https://pypi.python.org/pypi/restructuredtext_lint
> 
> 
> 
> -- 
> Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1413463 — Re: rst2pdf (was [PATCH 00/10] Documentation/Sphinx)

On Mon, 30 May 2016 23:05:34 +0300
Jani Nikula <jani.nikula@intel.com> wrote:

> > I can't recommend to use rst2pdf (it is less maintained), use default
> > sphinx LaTeX toolchain.  
> 
> I think we'll use whatever works, rst2pdf seemed to work for now, but we
> can change if needed.

I really like the idea of using rst2pdf and keeping the huge latex
dependency out of the mix.  I am a bit concerned, though; I've been able
to crash it in my experiments here.  We may want to have the ability to
support either chain eventually; otherwise, we might just end up picking
up maintenance of rst2pdf at some point so that it works properly for us.

jon

[toc] | [prev] | [next] | [standalone]

#1415723 — Re: rst2pdf (was [PATCH 00/10] Documentation/Sphinx)

Am 03.06.2016 um 22:47 schrieb Jonathan Corbet <corbet@lwn.net>:

> On Mon, 30 May 2016 23:05:34 +0300
> Jani Nikula <jani.nikula@intel.com> wrote:
> 
>>> I can't recommend to use rst2pdf (it is less maintained), use default
>>> sphinx LaTeX toolchain.  
>> 
>> I think we'll use whatever works, rst2pdf seemed to work for now, but we
>> can change if needed.
> 
> I really like the idea of using rst2pdf and keeping the huge latex
> dependency out of the mix.  I am a bit concerned, though; I've been able
> to crash it in my experiments here.  We may want to have the ability to
> support either chain eventually; otherwise, we might just end up picking
> up maintenance of rst2pdf at some point so that it works properly for us.
> 
> jon

I looked closer to rst2pdf, it supports only the docutils reST, but
not the sphinx superset ...  

<SNIP rst2pdf>-------------
$ rst2pdf  index.rst
index.rst:15: (ERROR/3) Unknown interpreted text role "ref".
index.rst:15: (ERROR/3) Unknown interpreted text role "ref".
index.rst:27: (ERROR/3) Unknown directive type "toctree".

.. toctree::
    :maxdepth: 1

    kernel-doc-intro
    kernel-doc-syntax
<SNAP>-------------


rules like ":ref:", domains like ":c:type:" and directives like ".. toctree:"
are a part of the (extended) reST syntax from sphinx, thats why 
standard docutils (like rst2*) will not work ...

> Am 18.04.2016 um 10:10 schrieb Markus Heiser <markus.heiser@darmarIT.de>:
> Re: Kernel docs: muddying the waters a bit
>
> BTW a few words about differences between DockBook and reST (Sphinx).
> 
> With DocBook you write *books*, the protocol (the DocBook application) has
> no facility to *chunk* and interconnect several documents. The external ENTITY 
> is a workaround on the SGML layer, not on XML nor on the DB-application layer.
> Thats the reason, why so many XML-tools don't handle this entities and
> many DocBook to (e.g.) reST tools are fail.
> 
> With **standard** reST it is nearly the same, except there is a "include"
> directive on the application layer. But this directive is very simple,
> comparable to the C preprocessor "#include" directive.
> 
> With the **superset** reST-markup of Sphinx-doc you get a the "toctree" directive,
> which lets you control how a document-tree should be build.
> 
> http://www.sphinx-doc.org/en/stable/markup/toctree.html
> 
> @Mauro: you mentioned a docutils (rst2*) experience in your mail 
>      http://marc.info/?l=linux-doc&m=145735316012094&w=2
> 
>      Because the "toctree" directive -- and other directives
>      we use -- are a part of a superset of the **standard** 
>      reST, the standard docutils (like rst2*) will not work.


-- Markus --

[toc] | [prev] | [next] | [standalone]

#1415753 — Re: rst2pdf (was [PATCH 00/10] Documentation/Sphinx)

On Tue, 07 Jun 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
> I looked closer to rst2pdf, it supports only the docutils reST, but
> not the sphinx superset ...  
>
> <SNIP rst2pdf>-------------
> $ rst2pdf  index.rst
> index.rst:15: (ERROR/3) Unknown interpreted text role "ref".
> index.rst:15: (ERROR/3) Unknown interpreted text role "ref".
> index.rst:27: (ERROR/3) Unknown directive type "toctree".

This doesn't actually happen if you run rst2pdf through Sphinx.

> .. toctree::
>     :maxdepth: 1
>
>     kernel-doc-intro
>     kernel-doc-syntax
> <SNAP>-------------
>
>
> rules like ":ref:", domains like ":c:type:" and directives like ".. toctree:"
> are a part of the (extended) reST syntax from sphinx, thats why 
> standard docutils (like rst2*) will not work ...

You will get warnings like

[WARNING] styles.py:548 Using undefined style 'function', aliased to style 'normal'.
[WARNING] styles.py:548 Using undefined style 'type', aliased to style 'normal'.

but it's a matter of defining a custom rst2pdf stylesheet. It's json
with a sort of inheritance model, so it should be easy to just extend
the existing stylesheet.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1419599 — Re: rst2pdf (was [PATCH 00/10] Documentation/Sphinx)

Am 07.06.2016 um 08:44 schrieb Jani Nikula <jani.nikula@intel.com>:

> On Tue, 07 Jun 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
>> I looked closer to rst2pdf, it supports only the docutils reST, but
>> not the sphinx superset ...  
>> 
>> <SNIP rst2pdf>-------------
>> $ rst2pdf  index.rst
>> index.rst:15: (ERROR/3) Unknown interpreted text role "ref".
>> index.rst:15: (ERROR/3) Unknown interpreted text role "ref".
>> index.rst:27: (ERROR/3) Unknown directive type "toctree".
> 
> This doesn't actually happen if you run rst2pdf through Sphinx.
> 

Aaah, thanks for hinting. With this in mind, I think rst2pdf
is a good choice and the minor bugs in could be handled. In the
long term a XeTeX builder as an alternative to produce layouts
with higher quality would be nice.

I (also) added rst2pdf:

https://github.com/return42/linux/commit/d88c4981f86fed85e54ee4d4955b35eb9eaac063

-- M --


>> .. toctree::
>>    :maxdepth: 1
>> 
>>    kernel-doc-intro
>>    kernel-doc-syntax
>> <SNAP>-------------
>> 
>> 
>> rules like ":ref:", domains like ":c:type:" and directives like ".. toctree:"
>> are a part of the (extended) reST syntax from sphinx, thats why 
>> standard docutils (like rst2*) will not work ...
> 
> You will get warnings like
> 
> [WARNING] styles.py:548 Using undefined style 'function', aliased to style 'normal'.
> [WARNING] styles.py:548 Using undefined style 'type', aliased to style 'normal'.
> 
> but it's a matter of defining a custom rst2pdf stylesheet. It's json
> with a sort of inheritance model, so it should be easy to just extend
> the existing stylesheet.
> 
> BR,
> Jani.
> 
> -- 
> Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1413480

On Mon, 30 May 2016 23:05:34 +0300
Jani Nikula <jani.nikula@intel.com> wrote:

> To be clear, the "sphinx-for-docs-next" branch of [1], [2] is what I
> propose to merge at this time. There's the Sphinx configuration, kernel
> build integration, Sphinx kernel-doc extension, tons of kernel-doc
> updates, etc. 

OK, I do believe that I am ready to do that.  Many, many thanks for doing
this work!  Please drop me a note when you think that the branch is ready
to go.

> There is no DocBook tmpl conversion; all of that is left
> to the authors (owners, maintainers) of the documents, but this enables
> them to focus on that part.

I would like to have some actual documents in there in the near future,
so that interested folks have something to look at and start with.  Maybe
I'll do that myself with some of the docbooks without active maintainers,
or even, maybe, some of the .txt files :)

Thanks,

jon

[toc] | [prev] | [next] | [standalone]

#1413568

On Fri, Jun 3, 2016 at 11:04 PM, Jonathan Corbet <corbet@lwn.net> wrote:
> On Mon, 30 May 2016 23:05:34 +0300
> Jani Nikula <jani.nikula@intel.com> wrote:
>
>> To be clear, the "sphinx-for-docs-next" branch of [1], [2] is what I
>> propose to merge at this time. There's the Sphinx configuration, kernel
>> build integration, Sphinx kernel-doc extension, tons of kernel-doc
>> updates, etc.
>
> OK, I do believe that I am ready to do that.  Many, many thanks for doing
> this work!  Please drop me a note when you think that the branch is ready
> to go.

I've just fixed the last bug in the line number support. I guess next
week Jani will double-check that and then send out the patch bomb for
you. We have more ideas and work, but I think this is a very good
baseline to get going.

>> There is no DocBook tmpl conversion; all of that is left
>> to the authors (owners, maintainers) of the documents, but this enables
>> them to focus on that part.
>
> I would like to have some actual documents in there in the near future,
> so that interested folks have something to look at and start with.  Maybe
> I'll do that myself with some of the docbooks without active maintainers,
> or even, maybe, some of the .txt files :)

I definitely want to get gpu.tmpl converted for 4.8, but that's
probably better done in drm-misc instead of doc-next for coordination
with ongoing drm work. I'd also like to unify dma-buffer-sharing.txt
with the kerneldoc we have into one .rst. Jani has done all the
testing and development on a full conversion of all .tmpl files
(except media), so I guess we could convert all the others directly in
doc-next.

One bit to think of is how much we want to split things up. For gpu
and related documentation I think we'll go with Documentation/gpu/ and
then put an overview.rst, drm_kms.rst, drm_kms_helper.rst drm_uapi.rst
and so on in there. gpu.tmpl is way too big imo, the only reason we
had it all merged is to keep cross-referencing working. But with
sphinx that works across source files, and we can split things up
more. Many other docs seem fairly small in comparison, so maybe ok to
keep as-is.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

[toc] | [prev] | [next] | [standalone]

#1413732

On Sat, 04 Jun 2016, Jonathan Corbet <corbet@lwn.net> wrote:
> On Mon, 30 May 2016 23:05:34 +0300
> Jani Nikula <jani.nikula@intel.com> wrote:
>
>> To be clear, the "sphinx-for-docs-next" branch of [1], [2] is what I
>> propose to merge at this time. There's the Sphinx configuration, kernel
>> build integration, Sphinx kernel-doc extension, tons of kernel-doc
>> updates, etc. 
>
> OK, I do believe that I am ready to do that.  Many, many thanks for doing
> this work!  Please drop me a note when you think that the branch is ready
> to go.

In case you missed the patch bomb in your inbox, I think it's ready
now. ;)

>> There is no DocBook tmpl conversion; all of that is left
>> to the authors (owners, maintainers) of the documents, but this enables
>> them to focus on that part.
>
> I would like to have some actual documents in there in the near future,
> so that interested folks have something to look at and start with.  Maybe
> I'll do that myself with some of the docbooks without active maintainers,
> or even, maybe, some of the .txt files :)

So we'll be working on gpu.tmpl next and I'll look at
Documentation/kernel-doc-nano-HOWTO.txt too.

BR,
Jani.



-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1410713

On Mon, 30 May 2016 11:10:26 +0200
Daniel Vetter <daniel.vetter@ffwll.ch> wrote:

> I think next steps is to get this merged into docs-next, with a stable
> tag, so that I can pull it into drm-misc.

So, I want to take another look at this, which probably will need another
day or two before it can happen.  First impression, though, is that this
is great, so I'm expecting that I'll be applying it.

Not sure about the stable tag, though?  It doesn't really seem like
stable material?  

jon

[toc] | [prev] | [next] | [standalone]

#1410857

On Wed, Jun 1, 2016 at 3:07 AM, Jonathan Corbet <corbet@lwn.net> wrote:
> On Mon, 30 May 2016 11:10:26 +0200
> Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
>
>> I think next steps is to get this merged into docs-next, with a stable
>> tag, so that I can pull it into drm-misc.
>
> So, I want to take another look at this, which probably will need another
> day or two before it can happen.  First impression, though, is that this
> is great, so I'm expecting that I'll be applying it.
>
> Not sure about the stable tag, though?  It doesn't really seem like
> stable material?

Oh, I meant a git tag of a stable (non-rebasing) branch that I can
pull into drm-misc so that we can apply the gpu.tmpl->gpu.rst
conversion on top. Not a cc: stable tag. Too many kinds of stable tags
I guess ;-)
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

[toc] | [prev] | [next] | [standalone]

#1413443

[So I'm finally trying to get into this for real, hopefully I won't be
interrupted too many times...expect a few mails as I catch up.]

On Fri, 20 May 2016 16:39:31 +0300
Jani Nikula <jani.nikula@intel.com> wrote:

> There are a few tradeoffs, of course. First, this requires that the
> EXPORT_SYMBOL markers are placed immediately after the function being
> exported, as kernel-doc will only look at one file at a time. This is
> the recommendation anyway. 

As I understand it, the technical reasons that kept some markers in
separate files should no longer be relevant, so this is probably OK.  It
would be nice to have a sense for how many sites need to be fixed.

> Second, we lose support for the !C docproc directive to check
> that all kernel-doc comments in a file are used. This is probably
> something we'd like to have back in the future, but at this time I think
> it's an acceptable tradeoff wrt the gains.

This is maybe a job for a separate tool.  A related issue is the (fairly
frequent) "oh look, none of the comments in $FILE are being used"
realization that seems to happen fairly often.  It would be nice to check
for that, but that's going to be hard to shoehorn into Sphinx.

jon

[toc] | [prev] | [next] | [standalone]

#1413444

On Fri, 3 Jun 2016 22:24:03 +0200
Daniel Vetter <daniel.vetter@ffwll.ch> wrote:

> > This is maybe a job for a separate tool.  A related issue is the (fairly
> > frequent) "oh look, none of the comments in $FILE are being used"
> > realization that seems to happen fairly often.  It would be nice to check
> > for that, but that's going to be hard to shoehorn into Sphinx.  
> 
> I think much more valuable would be a tool that checks whether
> comments are pulled in anywhere (for a given source file), over the
> entire tree. Very often entire subsystems carg-cult kernel-doc, but
> never use it in a .tmpl, which means 0-day won't notice, and neither
> anyone else.

That is kind of what I was trying to get at, yes :)

jon

[toc] | [prev] | [next] | [standalone]

#1413750

On Fri, 03 Jun 2016, Jonathan Corbet <corbet@lwn.net> wrote:
> On Fri, 3 Jun 2016 22:24:03 +0200
> Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
>
>> > This is maybe a job for a separate tool.  A related issue is the (fairly
>> > frequent) "oh look, none of the comments in $FILE are being used"
>> > realization that seems to happen fairly often.  It would be nice to check
>> > for that, but that's going to be hard to shoehorn into Sphinx.  
>> 
>> I think much more valuable would be a tool that checks whether
>> comments are pulled in anywhere (for a given source file), over the
>> entire tree. Very often entire subsystems carg-cult kernel-doc, but
>> never use it in a .tmpl, which means 0-day won't notice, and neither
>> anyone else.
>
> That is kind of what I was trying to get at, yes :)

For the 0-day, I've already got a preliminary kernel-doc-rst-lint
script, which can be used as a checker in the kernel build. This would
catch all kernel-doc comments, whether they're included to documentation
or not, and complain about kernel-doc and rst issues. It won't be as
comprehensive as a full Sphinx build in terms of validation, but it'll
cover more kernel-doc comments. It should catch the silly errors early.

BR,
Jani.



-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1413445

On Fri, Jun 3, 2016 at 10:16 PM, Jonathan Corbet <corbet@lwn.net> wrote:
>
>> Second, we lose support for the !C docproc directive to check
>> that all kernel-doc comments in a file are used. This is probably
>> something we'd like to have back in the future, but at this time I think
>> it's an acceptable tradeoff wrt the gains.
>
> This is maybe a job for a separate tool.  A related issue is the (fairly
> frequent) "oh look, none of the comments in $FILE are being used"
> realization that seems to happen fairly often.  It would be nice to check
> for that, but that's going to be hard to shoehorn into Sphinx.

I think much more valuable would be a tool that checks whether
comments are pulled in anywhere (for a given source file), over the
entire tree. Very often entire subsystems carg-cult kernel-doc, but
never use it in a .tmpl, which means 0-day won't notice, and neither
anyone else.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

[toc] | [prev] | [next] | [standalone]

#1413748

On Fri, 03 Jun 2016, Jonathan Corbet <corbet@lwn.net> wrote:
> [So I'm finally trying to get into this for real, hopefully I won't be
> interrupted too many times...expect a few mails as I catch up.]
>
> On Fri, 20 May 2016 16:39:31 +0300
> Jani Nikula <jani.nikula@intel.com> wrote:
>
>> There are a few tradeoffs, of course. First, this requires that the
>> EXPORT_SYMBOL markers are placed immediately after the function being
>> exported, as kernel-doc will only look at one file at a time. This is
>> the recommendation anyway. 
>
> As I understand it, the technical reasons that kept some markers in
> separate files should no longer be relevant, so this is probably OK.  It
> would be nice to have a sense for how many sites need to be fixed.

Actually, mostly this is not a problem due to EXPORT_SYMBOL placement,
but rather due to kernel-doc comments being placed in header files above
function declarations while EXPORT_SYMBOL is where it should be next to
the function definition.

I don't think we can force people to move the kernel-doc comments for
exported functions from header files next to the function definitions.

The straightforward fix to this is to add an optional filename parameter
to the kernel-doc extension :export: argument, to pass additional files
to kernel-doc where to look for the EXPORT_SYMBOLs. For example:

.. kernel-doc:: include/drm/foo.h
   :export: drivers/gpu/drm/foo/foo.c drivers/gpu/drm/foo/bar.c

This would instruct kernel-doc to extract documentation from
include/drm/foo.h for all functions that have been exported using
EXPORT_SYMBOL (or _GPL) in include/drm/foo.h, drivers/gpu/drm/foo/foo.c,
or drivers/gpu/drm/foo/bar.c.

We have something along these lines in docproc already with the !D
directive, so nothing altogether surprising.

If my quick grep-fu serves me right, there are about a thousand exported
symbols with kernel-doc comments in the headers. It's a ballpark
figure. They come in batches; a small fraction of that many filenames in
a fraction of the :export: statements would cover most of them.

Before this fix, the workaround is to name the functions with
:functions: argument instead of using :export:.

I'm hoping this is not a blocker for merging the series. If the proposed
fix is acceptable, I'll get it done before v4.8.

BR,
Jani.


The ugly greps:

$ git grep "^EXPORT_SYMBOL" | sed 's/^[^(]*(\([a-zA-Z0-9_]*\)).*/\1/' | sort > exports
$ git grep -h -A1 "^/\*\*$" -- *.h | grep -v "^\(/\*\*\|--\)" | sed 's/^ \*[         ]*\([a-zA-Z0-9_][a-zA-Z0-9_]*\).*/\1/;' | sort > comments
$ comm -1 -2 comments exports | wc -l
952




-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.kernel

csiph-web