Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1566094 > unrolled thread

[RFC PATCH v1 0/6] pure python kernel-doc parser and more

Back to article view | Back to linux.kernel

Contents

  [RFC PATCH v1 0/6] pure python kernel-doc parser and more Markus Heiser <markus.heiser@darmarit.de> - 2017-01-24 21:00 +0100
    [RFC PATCH v1 5/6] kernel-doc: add kerneldoc-src2rst command Markus Heiser <markus.heiser@darmarit.de> - 2017-01-24 21:00 +0100
    [RFC PATCH v1 4/6] kernel-doc: insert TODOs on kernel-doc errors Markus Heiser <markus.heiser@darmarit.de> - 2017-01-24 21:00 +0100
    Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser  with a pure python one (WIP) Jonathan Corbet <corbet@lwn.net> - 2017-01-25 01:20 +0100
      Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser  with a pure python one (WIP) Daniel Vetter <daniel@ffwll.ch> - 2017-01-25 07:40 +0100
        Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP) Markus Heiser <markus.heiser@darmarit.de> - 2017-01-25 08:40 +0100
      Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP) Jani Nikula <jani.nikula@intel.com> - 2017-01-25 11:30 +0100
        Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser  with a pure python one (WIP) Daniel Vetter <daniel@ffwll.ch> - 2017-01-25 11:40 +0100
        Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP) Markus Heiser <markus.heiser@darmarit.de> - 2017-01-25 20:10 +0100
          Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP) Jani Nikula <jani.nikula@intel.com> - 2017-01-25 22:00 +0100
            Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP) Markus Heiser <markus.heiser@darmarit.de> - 2017-01-26 11:00 +0100
              Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP) Jani Nikula <jani.nikula@intel.com> - 2017-01-26 11:20 +0100
          Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser  with a pure python one (WIP) Jonathan Corbet <corbet@lwn.net> - 2017-01-26 20:00 +0100
            Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP) Jani Nikula <jani.nikula@intel.com> - 2017-01-26 20:40 +0100
              Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP) Markus Heiser <markus.heiser@darmarit.de> - 2017-01-27 10:50 +0100

#1566094 — [RFC PATCH v1 0/6] pure python kernel-doc parser and more

Hi Jon,

here is my RFC, replacing the kernel-doc parser perl script with a python
implementation. The parser is implemented as module and is used by several
kernel-doc applications:

* kerneldoc         : the parser
* kerneldoc-lint    : liniting
* kerneldoc-src2rst : autodoc source tree
* manKernelDoc.py   : a builder generating man-pages

All this is mainly merged 1:1 from my POC at:

  https://github.com/return42/linuxdoc  commit 3991d3c

Since it is merged 1:1, you will notice it's CodingStyle is (ATM) not kernel
compliant and it lacks a user doc ('Documentation/doc-guide').  Take this as a
starting point to play around and gain some experience with the parser and its
applications. CodingStyle and user documentation will be patched when the
community agreed about functionalities.

Thanks

  -- Markus --


[1] https://www.mail-archive.com/linux-doc@vger.kernel.org/msg09002.html

Markus Heiser (6):
  kernel-doc: pure python kernel-doc parser (preparation)
  kernel-doc: replace kernel-doc perl parser with a pure python one (WIP)
  kernel-doc: add kerneldoc-lint command
  kernel-doc: insert TODOs on kernel-doc errors
  kernel-doc: add kerneldoc-src2rst command
  kernel-doc: add man page builder (target mandocs)

 Documentation/Makefile.sphinx        |    8 +-
 Documentation/admin-guide/conf.py    |    2 +
 Documentation/admin-guide/index.rst  |    2 +
 Documentation/conf.py                |    8 +-
 Documentation/core-api/conf.py       |    2 +
 Documentation/core-api/index.rst     |    2 +
 Documentation/dev-tools/conf.py      |    2 +
 Documentation/dev-tools/index.rst    |    2 +
 Documentation/doc-guide/conf.py      |    2 +
 Documentation/doc-guide/index.rst    |    2 +
 Documentation/driver-api/conf.py     |    2 +
 Documentation/driver-api/index.rst   |    2 +
 Documentation/gpu/conf.py            |    2 +
 Documentation/gpu/index.rst          |    2 +
 Documentation/media/Makefile         |    1 +
 Documentation/media/conf.py          |    2 +
 Documentation/media/index.rst        |    2 +
 Documentation/process/conf.py        |    2 +
 Documentation/process/index.rst      |    2 +
 Documentation/security/conf.py       |    2 +
 Documentation/security/index.rst     |    9 +
 Documentation/sphinx/fspath.py       |  435 +++++
 Documentation/sphinx/kernel_doc.py   | 2908 ++++++++++++++++++++++++++++++++++
 Documentation/sphinx/kerneldoc.py    |  149 --
 Documentation/sphinx/lint.py         |  121 ++
 Documentation/sphinx/manKernelDoc.py |  408 +++++
 Documentation/sphinx/rstKernelDoc.py |  560 +++++++
 Documentation/sphinx/src2rst.py      |  229 +++
 scripts/kerneldoc                    |   11 +
 scripts/kerneldoc-lint               |   11 +
 scripts/kerneldoc-src2rst            |   11 +
 31 files changed, 4748 insertions(+), 155 deletions(-)
 create mode 100644 Documentation/sphinx/fspath.py
 create mode 100755 Documentation/sphinx/kernel_doc.py
 delete mode 100644 Documentation/sphinx/kerneldoc.py
 create mode 100755 Documentation/sphinx/lint.py
 create mode 100755 Documentation/sphinx/manKernelDoc.py
 create mode 100755 Documentation/sphinx/rstKernelDoc.py
 create mode 100755 Documentation/sphinx/src2rst.py
 create mode 100755 scripts/kerneldoc
 create mode 100755 scripts/kerneldoc-lint
 create mode 100755 scripts/kerneldoc-src2rst

-- 
2.7.4

[toc] | [next] | [standalone]

#1566097 — [RFC PATCH v1 5/6] kernel-doc: add kerneldoc-src2rst command

this patch adds a command to auto-generate documentation from kernel's
source tree.::

  scripts/kerneldoc-src2rst --help

E.g. to autodoc the kernel's ./include folder use::

  scripts/kerneldoc-src2rst ./include /tmp/test123

From the resulting reST-doctree you can build HTML rendered output like
the one in [1]. I use it to see, if patches on the kernel_doc.py parser
cause a regression (compare reST before/after the patch). Autodoc the
whole source tree takes a long time. To speed up, the src2rst module
uses multiprocessing from [2]. This means, it consumes all your CPUs. If
you don't want this, use option '--threads=n'.

[1] https://h2626237.stratoserver.net/kernel/linux_src_doc/index.html
[2] https://docs.python.org/3.6/library/multiprocessing.html

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
---
 Documentation/sphinx/src2rst.py | 229 ++++++++++++++++++++++++++++++++++++++++
 scripts/kerneldoc-src2rst       |  11 ++
 2 files changed, 240 insertions(+)
 create mode 100755 Documentation/sphinx/src2rst.py
 create mode 100755 scripts/kerneldoc-src2rst

diff --git a/Documentation/sphinx/src2rst.py b/Documentation/sphinx/src2rst.py
new file mode 100755
index 0000000..d8d6e7b
--- /dev/null
+++ b/Documentation/sphinx/src2rst.py
@@ -0,0 +1,229 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8; mode: python -*-
+# pylint: disable=C0103
+
+u"""
+    src2rst
+    ~~~~~~~
+
+    Implementation of the ``kerneldoc-src2rst`` command.
+
+    :copyright:  Copyright (C) 2016  Markus Heiser
+    :license:    GPL Version 2, June 1991 see Linux/COPYING for details.
+
+    The ``kerneldoc-src2rst`` command extracts documentation from Linux kernel's
+    source code comments, see ``--help``::
+
+        $ kerneldoc-src2rst --help
+
+"""
+
+# ------------------------------------------------------------------------------
+# imports
+# ------------------------------------------------------------------------------
+
+import sys
+import argparse
+import re
+import multiprocessing
+
+import six
+
+from fspath import FSPath, OS_ENV
+import kernel_doc as kerneldoc
+
+# ------------------------------------------------------------------------------
+# config
+# ------------------------------------------------------------------------------
+
+MARKUP = "kernel-doc" # "reST"
+MSG    = lambda msg: sys.__stderr__.write("INFO : %s\n" % msg)
+ERR    = lambda msg: sys.__stderr__.write("ERROR: %s\n" % msg)
+FATAL  = lambda msg: sys.__stderr__.write("FATAL: %s\n" % msg)
+IGNORE = ['kernel-doc.rst']
+
+TEMPLATE_INDEX=u"""\
+.. -*- coding: utf-8; mode: rst -*-
+
+================================================================================
+%(title)s
+================================================================================
+
+.. toctree::
+    :maxdepth: 1
+
+"""
+
+CMD     = None # global used by multiprocessing
+SRCTREE = FSPath(OS_ENV.get("srctree", ""))
+
+# ------------------------------------------------------------------------------
+def main():
+# ------------------------------------------------------------------------------
+
+    global CMD # pylint: disable=W0603, W0621
+
+    CLI = argparse.ArgumentParser(
+        description = ("Parse *kernel-doc* comments from source code")
+        , formatter_class=argparse.ArgumentDefaultsHelpFormatter)
+
+    CLI.add_argument(
+        "srctree"
+        , help    = "Folder of source code."
+        , type    = lambda x: FSPath(x).ABSPATH)
+
+    CLI.add_argument(
+        "doctree"
+        , help    = "Folder to place reST documentation."
+        , type    = lambda x: FSPath(x).ABSPATH)
+
+    CLI.add_argument(
+        "--sloppy"
+        , action  = "store_true"
+        , help    = "Sloppy comment check, reports only severe errors.")
+
+    CLI.add_argument(
+        "--force"
+        , action  = "store_true"
+        , help    = "Don't stop if doctree exists.")
+
+    CLI.add_argument(
+        "--threads"
+        , type    =  int
+        , default = multiprocessing.cpu_count()
+        , help    = "Use up to n threads.")
+
+    CLI.add_argument(
+        "--markup"
+        , choices = ["reST", "kernel-doc", "auto"]
+        , default = "auto"
+        , help    = (
+            "Markup of the comments. Change this option only if you know"
+            " what you do. New comments must be marked up with reST!"))
+
+    CMD = CLI.parse_args()
+
+    if not CMD.srctree.EXISTS:
+        ERR("%s does not exists." % CMD.srctree)
+        sys.exit(42)
+
+    if not CMD.srctree.ISDIR:
+        ERR("%s is not a folder." % CMD.srctree)
+        sys.exit(42)
+
+    if not CMD.force and CMD.doctree.EXISTS:
+        ERR("%s is in the way, remove it first" % CMD.doctree)
+        sys.exit(42)
+
+    CMD.rst_files = set()
+    if CMD.markup == "auto":
+        CMD.rst_files = docgrep(SRCTREE/"Documentation")
+
+    pool = multiprocessing.Pool(CMD.threads)
+    pool.map(autodoc_file, gather_filenames(CMD))
+    pool.close()
+    pool.join()
+
+    insert_index_files(CMD.doctree)
+
+# ------------------------------------------------------------------------------
+def gather_filenames(cmd):
+# ------------------------------------------------------------------------------
+    MSG("gather files ...")
+
+    for fname in cmd.srctree.reMatchFind(r"^.*\.[ch]$"):
+        if fname.startswith(CMD.srctree/"Documentation"):
+            continue
+        yield fname
+
+# ------------------------------------------------------------------------------
+def autodoc_file(fname):
+# ------------------------------------------------------------------------------
+
+    fname  = fname.relpath(CMD.srctree)
+    markup = CMD.markup
+
+    if CMD.markup == "kernel-doc" and fname in CMD.rst_files:
+        markup = "reST"
+
+    opts = kerneldoc.ParseOptions(
+        rel_fname       = fname
+        , src_tree      = CMD.srctree
+        , verbose_warn  = not (CMD.sloppy)
+        , markup        = markup )
+
+    parser = kerneldoc.Parser(opts, kerneldoc.NullTranslator())
+    try:
+        parser.parse()
+    except Exception: # pylint: disable=W0703
+        FATAL("kernel-doc markup of %s seems buggy / can't parse" % opts.fname)
+        return
+
+    if not parser.ctx.dump_storage:
+        # no kernel-doc comments found
+        MSG("parsed: NONE comments: %s" % opts.fname)
+        return
+
+    MSG("parsed: %4d comments: %s" % (len(parser.ctx.dump_storage), opts.fname))
+
+    try:
+        rst = six.StringIO()
+        translator = kerneldoc.ReSTTranslator()
+        opts.out   = rst
+
+        # First try to output reST, this might fail, because the kernel-doc
+        # parser part is to tollerant ("bad lines", "function name and function
+        # declaration are different", etc ...).
+        parser.parse_dump_storage(translator=translator)
+
+        outFile = CMD.doctree / fname.replace(".","_") + ".rst"
+        outFile.DIRNAME.makedirs()
+        with outFile.openTextFile(mode="w") as out:
+            out.write(rst.getvalue())
+
+    except Exception: # pylint: disable=W0703
+        FATAL("kernel-doc markup of %s seems buggy / can't parse" % opts.fname)
+        return
+
+# ------------------------------------------------------------------------------
+def insert_index_files(folder):
+# ------------------------------------------------------------------------------
+
+    for folder, dirnames, filenames in folder.walk():
+        ctx = kerneldoc.Container( title = folder.FILENAME )
+        dirnames.sort()
+        filenames.sort()
+        indexFile = folder / "index.rst"
+        MSG("create index: %s" % indexFile)
+        with indexFile.openTextFile(mode="w") as index:
+            index.write(TEMPLATE_INDEX % ctx)
+            for d in dirnames:
+                index.write("    %s/index\n" % d.FILENAME)
+            for f in filenames:
+                if f.FILENAME == "index":
+                    continue
+                index.write("    %s\n" % f.FILENAME)
+
+# ------------------------------------------------------------------------------
+def docgrep(folder):
+# ------------------------------------------------------------------------------
+
+    # Hackisch script to grep all '.. kernel-doc::' directives. The assumption
+    # is, that comments from the source files used in those directives are
+    # allready migratetd to the reST format. I guess that (ATM) 95-99% of the
+    # comments are not migrated, those will be parsed with the old kernel-doc
+    # comment style introduced by the old DocBook toolchain.
+
+    pat = re.compile(r"^\s*\.\.\s+kernel-doc::\s*([^\s]+)\s*$")
+    out = set()
+    for rstFile in folder.reMatchFind(r".*\.rst"):
+        if rstFile.BASENAME in IGNORE:
+            continue
+        #print(rstFile)
+        with rstFile.openTextFile() as f:
+            for l in f:
+                match = pat.search(l)
+                if match:
+                    #print(match.group(1))
+                    out.add(match.group(1))
+    return sorted(out)
diff --git a/scripts/kerneldoc-src2rst b/scripts/kerneldoc-src2rst
new file mode 100755
index 0000000..6c5e8b1
--- /dev/null
+++ b/scripts/kerneldoc-src2rst
@@ -0,0 +1,11 @@
+#!/usr/bin/python
+
+import sys
+from os import path
+
+linuxdoc = path.abspath(path.join(path.dirname(__file__), '..'))
+linuxdoc = path.join(linuxdoc, 'Documentation', 'sphinx')
+sys.path.insert(0, linuxdoc)
+
+import src2rst
+src2rst.main()
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1566098 — [RFC PATCH v1 4/6] kernel-doc: insert TODOs on kernel-doc errors

The rstKernelDoc.py sphinx-extensions inserts '.. todo::' on
errors. With this patch sphinx.ext.todo extension [1] is activated.
This is similar to what we know from DocBook's *Oops* functionality of
the kernel-doc perl script [2].

I added this functionality (only) to the subproject, to left the main
build untouched. E.g. run::

  make SPHINXDIRS="driver-api" htmldocs

open the HTML output and scroll down to see those *Oops* (TODO) boxes.

The *Oops* in the HTML output helps authors to find rough errors and
increase the quality of the documentation. ATM there are to many
errors (false positives) and it needs some discussion. Take this as a
starting point.

[1] http://www.sphinx-doc.org/en/stable/ext/todo.html
[2] http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/scripts/kernel-doc#n3073

Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>
---
 Documentation/admin-guide/conf.py   | 2 ++
 Documentation/admin-guide/index.rst | 2 ++
 Documentation/conf.py               | 3 ++-
 Documentation/core-api/conf.py      | 2 ++
 Documentation/core-api/index.rst    | 2 ++
 Documentation/dev-tools/conf.py     | 2 ++
 Documentation/dev-tools/index.rst   | 2 ++
 Documentation/doc-guide/conf.py     | 2 ++
 Documentation/doc-guide/index.rst   | 2 ++
 Documentation/driver-api/conf.py    | 2 ++
 Documentation/driver-api/index.rst  | 2 ++
 Documentation/gpu/conf.py           | 2 ++
 Documentation/gpu/index.rst         | 2 ++
 Documentation/media/conf.py         | 2 ++
 Documentation/media/index.rst       | 2 ++
 Documentation/process/conf.py       | 2 ++
 Documentation/process/index.rst     | 2 ++
 Documentation/security/conf.py      | 2 ++
 Documentation/security/index.rst    | 9 +++++++++
 19 files changed, 45 insertions(+), 1 deletion(-)

diff --git a/Documentation/admin-guide/conf.py b/Documentation/admin-guide/conf.py
index 86f7389..28993c4 100644
--- a/Documentation/admin-guide/conf.py
+++ b/Documentation/admin-guide/conf.py
@@ -8,3 +8,5 @@ latex_documents = [
     ('index', 'linux-user.tex', 'Linux Kernel User Documentation',
      'The kernel development community', 'manual'),
 ]
+
+todo_include_todos = True
diff --git a/Documentation/admin-guide/index.rst b/Documentation/admin-guide/index.rst
index 8ddae4e..8c9245a 100644
--- a/Documentation/admin-guide/index.rst
+++ b/Documentation/admin-guide/index.rst
@@ -67,3 +67,5 @@ configure specific aspects of kernel behavior to your liking.
    =======
 
    * :ref:`genindex`
+
+   .. todolist::
diff --git a/Documentation/conf.py b/Documentation/conf.py
index 4843903..013af9a 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -34,7 +34,8 @@ from load_config import loadConfig
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['rstKernelDoc', 'rstFlatTable', 'kernel_include', 'cdomain' ]
+extensions = ['rstKernelDoc', 'rstFlatTable', 'kernel_include', 'cdomain',
+              'sphinx.ext.todo' ]
 
 # The name of the math extension changed on Sphinx 1.4
 if major == 1 and minor > 3:
diff --git a/Documentation/core-api/conf.py b/Documentation/core-api/conf.py
index db1f765..55e1a0e 100644
--- a/Documentation/core-api/conf.py
+++ b/Documentation/core-api/conf.py
@@ -8,3 +8,5 @@ latex_documents = [
     ('index', 'core-api.tex', project,
      'The kernel development community', 'manual'),
 ]
+
+todo_include_todos = True
diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst
index 0d93d80..e958124 100644
--- a/Documentation/core-api/index.rst
+++ b/Documentation/core-api/index.rst
@@ -32,3 +32,5 @@ Interfaces for kernel debugging
    =======
 
    * :ref:`genindex`
+
+   .. todolist::
diff --git a/Documentation/dev-tools/conf.py b/Documentation/dev-tools/conf.py
index 7faafa3..f9d394b 100644
--- a/Documentation/dev-tools/conf.py
+++ b/Documentation/dev-tools/conf.py
@@ -8,3 +8,5 @@ latex_documents = [
     ('index', 'dev-tools.tex', project,
      'The kernel development community', 'manual'),
 ]
+
+todo_include_todos = True
diff --git a/Documentation/dev-tools/index.rst b/Documentation/dev-tools/index.rst
index 07d8811..48ffb70 100644
--- a/Documentation/dev-tools/index.rst
+++ b/Documentation/dev-tools/index.rst
@@ -31,3 +31,5 @@ whole; patches welcome!
    =======
 
    * :ref:`genindex`
+
+   .. todolist::
diff --git a/Documentation/doc-guide/conf.py b/Documentation/doc-guide/conf.py
index fd37311..8f2af2e 100644
--- a/Documentation/doc-guide/conf.py
+++ b/Documentation/doc-guide/conf.py
@@ -8,3 +8,5 @@ latex_documents = [
     ('index', 'kernel-doc-guide.tex', 'Linux Kernel Documentation Guide',
      'The kernel development community', 'manual'),
 ]
+
+todo_include_todos = True
diff --git a/Documentation/doc-guide/index.rst b/Documentation/doc-guide/index.rst
index 6fff402..b177a4e 100644
--- a/Documentation/doc-guide/index.rst
+++ b/Documentation/doc-guide/index.rst
@@ -18,3 +18,5 @@ How to write kernel documentation
    =======
 
    * :ref:`genindex`
+
+   .. todolist::
diff --git a/Documentation/driver-api/conf.py b/Documentation/driver-api/conf.py
index 202726d..b682e5c 100644
--- a/Documentation/driver-api/conf.py
+++ b/Documentation/driver-api/conf.py
@@ -8,3 +8,5 @@ latex_documents = [
     ('index', 'driver-api.tex', project,
      'The kernel development community', 'manual'),
 ]
+
+todo_include_todos = True
diff --git a/Documentation/driver-api/index.rst b/Documentation/driver-api/index.rst
index a2e5db0..ebc86cb 100644
--- a/Documentation/driver-api/index.rst
+++ b/Documentation/driver-api/index.rst
@@ -38,3 +38,5 @@ available subsections can be seen below.
    =======
 
    * :ref:`genindex`
+
+   .. todolist::
diff --git a/Documentation/gpu/conf.py b/Documentation/gpu/conf.py
index 1757b04..44dc0c2 100644
--- a/Documentation/gpu/conf.py
+++ b/Documentation/gpu/conf.py
@@ -8,3 +8,5 @@ latex_documents = [
     ('index', 'gpu.tex', project,
      'The kernel development community', 'manual'),
 ]
+
+todo_include_todos = True
diff --git a/Documentation/gpu/index.rst b/Documentation/gpu/index.rst
index 367d7c3..fee1857 100644
--- a/Documentation/gpu/index.rst
+++ b/Documentation/gpu/index.rst
@@ -20,3 +20,5 @@ Linux GPU Driver Developer's Guide
    =======
 
    * :ref:`genindex`
+
+   .. todolist::
diff --git a/Documentation/media/conf.py b/Documentation/media/conf.py
index bef927b..09755f9 100644
--- a/Documentation/media/conf.py
+++ b/Documentation/media/conf.py
@@ -8,3 +8,5 @@ latex_documents = [
     ('index', 'media.tex', 'Linux Media Subsystem Documentation',
      'The kernel development community', 'manual'),
 ]
+
+todo_include_todos = True
diff --git a/Documentation/media/index.rst b/Documentation/media/index.rst
index 7f8f0af..eb2754d 100644
--- a/Documentation/media/index.rst
+++ b/Documentation/media/index.rst
@@ -17,3 +17,5 @@ Contents:
    =======
 
    * :ref:`genindex`
+
+   .. todolist::
diff --git a/Documentation/process/conf.py b/Documentation/process/conf.py
index 1b01a80..a322314 100644
--- a/Documentation/process/conf.py
+++ b/Documentation/process/conf.py
@@ -8,3 +8,5 @@ latex_documents = [
     ('index', 'process.tex', 'Linux Kernel Development Documentation',
      'The kernel development community', 'manual'),
 ]
+
+todo_include_todos = True
diff --git a/Documentation/process/index.rst b/Documentation/process/index.rst
index 10aa692..acf13bb 100644
--- a/Documentation/process/index.rst
+++ b/Documentation/process/index.rst
@@ -55,3 +55,5 @@ lack of a better place.
    =======
 
    * :ref:`genindex`
+
+   .. todolist::
diff --git a/Documentation/security/conf.py b/Documentation/security/conf.py
index 472fc9a..78b30ab 100644
--- a/Documentation/security/conf.py
+++ b/Documentation/security/conf.py
@@ -6,3 +6,5 @@ latex_documents = [
     ('index', 'security.tex', project,
      'The kernel development community', 'manual'),
 ]
+
+todo_include_todos = True
diff --git a/Documentation/security/index.rst b/Documentation/security/index.rst
index 9bae6bb..b85944c 100644
--- a/Documentation/security/index.rst
+++ b/Documentation/security/index.rst
@@ -5,3 +5,12 @@ Security documentation
 .. toctree::
 
    tpm/index
+
+.. only::  subproject and html
+
+   Indices
+   =======
+
+   * :ref:`genindex`
+
+   .. todolist::
-- 
2.7.4

[toc] | [prev] | [next] | [standalone]

#1566244 — Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP)

On Tue, 24 Jan 2017 20:52:40 +0100
Markus Heiser <markus.heiser@darmarit.de> wrote:

> This patch is the initial merge of a pure python implementation
> to parse kernel-doc comments and generate reST from.
> 
> It consist mainly of to parts, the parser module (kerneldoc.py) and the
> sphinx-doc extension (rstKernelDoc.py). For the command line, there is
> also a 'scripts/kerneldoc' added.::
> 
>    scripts/kerneldoc --help
> 
> The main two parts are merged 1:1 from
> 
>   https://github.com/return42/linuxdoc  commit 3991d3c
> 
> Take this as a starting point, there is a lot of work to do (WIP).
> Since it is merged 1:1, you will also notice it's CodingStyle is (ATM)
> not kernel compliant and it lacks a user doc ('Documentation/doc-guide').
> 
> I will send patches for this when the community agreed about
> functionalities. I guess there are a lot of topics we have to agree
> about. E.g. the py-implementation is more strict the perl one.  When you
> build doc with the py-module you will see a lot of additional errors and
> warnings compared to the sloppy perl one.

Again, quick comments...

 - I would *much* rather evolve our existing Sphinx extension in the
   direction we want it to go than to just replace it wholesale.
   Replacement is the wrong approach for a few reasons, including the need
   to minimize change and preserve credit for Jani's work.  Can we work on
   that basis, please?

   Ideally at the time of merging, we would be able to build the docs with
   *either* kerneldoc.

 - I'll have to try it out to see how noisy it is.  I'm not opposed to
   stricter checks; indeed, they could be a good thing.  But we might want
   to have an option so we can cut back on the noise by default.

Thanks,

jon

[toc] | [prev] | [next] | [standalone]

#1566340 — Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP)

On Tue, Jan 24, 2017 at 05:13:14PM -0700, Jonathan Corbet wrote:
> On Tue, 24 Jan 2017 20:52:40 +0100
> Markus Heiser <markus.heiser@darmarit.de> wrote:
> 
> > This patch is the initial merge of a pure python implementation
> > to parse kernel-doc comments and generate reST from.
> > 
> > It consist mainly of to parts, the parser module (kerneldoc.py) and the
> > sphinx-doc extension (rstKernelDoc.py). For the command line, there is
> > also a 'scripts/kerneldoc' added.::
> > 
> >    scripts/kerneldoc --help
> > 
> > The main two parts are merged 1:1 from
> > 
> >   https://github.com/return42/linuxdoc  commit 3991d3c
> > 
> > Take this as a starting point, there is a lot of work to do (WIP).
> > Since it is merged 1:1, you will also notice it's CodingStyle is (ATM)
> > not kernel compliant and it lacks a user doc ('Documentation/doc-guide').
> > 
> > I will send patches for this when the community agreed about
> > functionalities. I guess there are a lot of topics we have to agree
> > about. E.g. the py-implementation is more strict the perl one.  When you
> > build doc with the py-module you will see a lot of additional errors and
> > warnings compared to the sloppy perl one.
> 
> Again, quick comments...
> 
>  - I would *much* rather evolve our existing Sphinx extension in the
>    direction we want it to go than to just replace it wholesale.
>    Replacement is the wrong approach for a few reasons, including the need
>    to minimize change and preserve credit for Jani's work.  Can we work on
>    that basis, please?
> 
>    Ideally at the time of merging, we would be able to build the docs with
>    *either* kerneldoc.

Seconded, I think renaming the extension string like this is just fairly
pointless busy-work. Kernel-doc isn't interacting perfectly with rst, but
now we already have a sizeable amount of stuff converted and going through
all that once more needs imo som really clear benefits. I think
bug-for-bug compatibility would be much better. Later on we could do
changes, on a change-by-change basis.
-Daniel


>  - I'll have to try it out to see how noisy it is.  I'm not opposed to
>    stricter checks; indeed, they could be a good thing.  But we might want
>    to have an option so we can cut back on the noise by default.


> 
> Thanks,
> 
> jon

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

[toc] | [prev] | [next] | [standalone]

#1566360 — Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP)

Hi Jon, hi Daniel !

Am 25.01.2017 um 07:37 schrieb Daniel Vetter <daniel@ffwll.ch>:

>> Again, quick comments...
>> 
>> - I would *much* rather evolve our existing Sphinx extension in the
>>   direction we want it to go than to just replace it wholesale.
>>   Replacement is the wrong approach for a few reasons, including the need
>>   to minimize change and preserve credit for Jani's work.  Can we work on
>>   that basis, please?

Sure. But I fear I haven't understood you right .... last post was:

> Markus, would you consider sending out a new patch set for review?  What I
> would like to do see is something adding the new script for the Sphinx
> toolchain, while leaving the DocBook build unchanged, using the old
> script.  We could then delete it once the last template file has moved
> over. 

talking about DocBook and now I read ...

>>   Ideally at the time of merging, we would be able to build the docs with
>>   *either* kerneldoc.

Now I'am totally confused ... it's no about you, but I do not understand
you clearly ... can you help a conceptual man?

> Seconded, I think renaming the extension string like this is just fairly
> pointless busy-work.

Hi Daniel, please help me, what did you mean with "renaming" the extension
string and "busy-work"?

There is a renaming of module's name but there should no work outside this
patch ... 

> Kernel-doc isn't interacting perfectly with rst, but
> now we already have a sizeable amount of stuff converted and going through
> all that once more needs imo som really clear benefits.

from authors POV nothing has changed.

> I think bug-for-bug compatibility would be much better. Later on we could do
> changes, on a change-by-change basis.

Both sphinx-extensions (the one we have and the one in the series) are
adapter to a "parser backend". 

1. Documentation/sphinx/kerneldoc.py    <--> scripts/kerneldoc -rst
2. Documentation/sphinx/rstKernelDoc.py <--> import module Documentation/sphinx/kernel_doc.py

Maintain two adapters for the two backends is possible. But one adapter
for two complete different backends .. is this what you mean?

>> - I'll have to try it out to see how noisy it is.  I'm not opposed to
>>   stricter checks; indeed, they could be a good thing.  But we might want
>>   to have an option so we can cut back on the noise by default.

As said, I'am willing to go communities way, it seems just a communication
problem (on my side) to understand what this way would be.

I try to sum what I guess ... e.g. to build output as usual with (1.)

  $ make htmldocs

to build with the py-parser and its sphinx-extension (see 2. above)::

  $ USE_PY_PARSER=1 make htmldocs

this should be easy and I can realize it in v2, but is this what you want?

Please give me some more hints / Thanks a lot!

--Markus--

[toc] | [prev] | [next] | [standalone]

#1566456 — Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP)

On Wed, 25 Jan 2017, Jonathan Corbet <corbet@lwn.net> wrote:
> On Tue, 24 Jan 2017 20:52:40 +0100
> Markus Heiser <markus.heiser@darmarit.de> wrote:
>
>> This patch is the initial merge of a pure python implementation
>> to parse kernel-doc comments and generate reST from.
>> 
>> It consist mainly of to parts, the parser module (kerneldoc.py) and the
>> sphinx-doc extension (rstKernelDoc.py). For the command line, there is
>> also a 'scripts/kerneldoc' added.::
>> 
>>    scripts/kerneldoc --help
>> 
>> The main two parts are merged 1:1 from
>> 
>>   https://github.com/return42/linuxdoc  commit 3991d3c
>> 
>> Take this as a starting point, there is a lot of work to do (WIP).
>> Since it is merged 1:1, you will also notice it's CodingStyle is (ATM)
>> not kernel compliant and it lacks a user doc ('Documentation/doc-guide').
>> 
>> I will send patches for this when the community agreed about
>> functionalities. I guess there are a lot of topics we have to agree
>> about. E.g. the py-implementation is more strict the perl one.  When you
>> build doc with the py-module you will see a lot of additional errors and
>> warnings compared to the sloppy perl one.

Markus, thanks for your work on this.

> Again, quick comments...
>
>  - I would *much* rather evolve our existing Sphinx extension in the
>    direction we want it to go than to just replace it wholesale.
>    Replacement is the wrong approach for a few reasons, including the need
>    to minimize change and preserve credit for Jani's work.  Can we work on
>    that basis, please?

I would grossly downplay the role of preserving credit for what I've
done, and put much more emphasis on the need to create a patch series
that gradually, step by step, evolves the current approach into
something better.

Excuse me for my bluntness, but I think changing everything in a single
commit, or even a few commits, is strictly not acceptable.

When I changed *small* things in scripts/kernel-doc, I would make
htmldocs before and after the change, and recursively diff the produced
output to ensure there were no surprises. We already have enough
documentation that a manual eyeballing of the output is simply not
sufficient to ensure things don't break.

The diff in output between before and after this series? 160k lines of
unified diff without context ('diff -u0 -r old new | wc -l').

Many of the changes are improvements on the result, such as using proper
<div> tags for function parameter lists etc., but clearly changing the
output should be independent of changing the parser, so we have some
chance of validating the parser.

>    Ideally at the time of merging, we would be able to build the docs with
>    *either* kerneldoc.

I'd be fine with switching over in a single commit that doesn't
drastically change the output. A drop-in replacement. But that's not the
case here.

>  - I'll have to try it out to see how noisy it is.  I'm not opposed to
>    stricter checks; indeed, they could be a good thing.  But we might want
>    to have an option so we can cut back on the noise by default.

The increase in 'make htmldocs' build log was from 1521 to 2791 lines in
my tree. Arguably there was useful extra diagnosis, but some of it was
the printouts of long lists of definitions that were not found, one per
line. So it could be condensed without losing info too.

On to performance. With the default build options the new system was
noticeably slower than the current one, with a 50% increase on my
machine. But what really caught me by surprise was that passing
SPHINXOPTS=-j5 to parallelize worked better on the current system,
making the new one a whopping 70% slower. Of course, the argument is
that the proposed parser does more and is better, but due to the
monolithic change it's impossible to pinpoint the culprit or do a proper
cost/benefit analysis on this. Again, this calls for a more broken down
series of patches to make the changes.

Finally, while I'd love to see scripts/kernel-doc go, I do have to ask
if changing roughly 3k lines of Perl to roughly 3k lines of Python (*)
really makes everything better? They both still parse everything using a
large pile of regular expressions and a clunky state machine. When I
look at the code, I'm afraid I do not get that liberating feeling of
throwing out old junk in favor of something small or elegant or even
obviously more maintainable than the old one. The new one offers more
features, but repeatedly we face the problem that it's all lumped in
together with the parser change. We should be able to look at the parser
change and the other improvements separately.

That said, perhaps having an elegant parser (perhaps based on a compiler
plugin) is incompatible with the idea of making it a bug-for-bug drop-in
replacement of the old one, and it's something we need to think about.

All in all I think the message should be clear: this needs to be split
into small, incremental changes. Just like we do everything in the
kernel.


BR,
Jani.


(*) Please do not get hung up on these numbers. The Python version does
    more in some ways, but adds more deps such as fspath that's not
    included in the figures, and the Perl version outputs more
    formats. It's not an apples to apples comparison. Let's just say
    they are somewhere in the same ballpark.

-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1566469 — Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP)

On Wed, Jan 25, 2017 at 12:24:31PM +0200, Jani Nikula wrote:
> Finally, while I'd love to see scripts/kernel-doc go, I do have to ask
> if changing roughly 3k lines of Perl to roughly 3k lines of Python (*)
> really makes everything better? They both still parse everything using a
> large pile of regular expressions and a clunky state machine. When I
> look at the code, I'm afraid I do not get that liberating feeling of
> throwing out old junk in favor of something small or elegant or even
> obviously more maintainable than the old one. The new one offers more
> features, but repeatedly we face the problem that it's all lumped in
> together with the parser change. We should be able to look at the parser
> change and the other improvements separately.

I share this concern a lot. The kernel-doc perl is a horror show, but it's
a horror show that 3-4 people now somewhat understand. Simply translating
the entire script into python leaves us with the same horror show, but in
a different language. And personally I'm not versed at all in either of
them (and I think that applies to many kernel hackers), so seems a wash.

If the new script would implement the state machinery in some
parser-combinator library to make it much easier to maintain, while still
being bug-for-bug compatible, then I'd be much, much more in favour of
doing this. And once we go to that amount of effort, then rewriting it in
python for more consistency with sphinx is definitely a good idea.

> That said, perhaps having an elegant parser (perhaps based on a compiler
> plugin) is incompatible with the idea of making it a bug-for-bug drop-in
> replacement of the old one, and it's something we need to think about.

Yeah, I fear we'll always need our own parser to avoid breaking the world.
But there's definitely better ways out there to write parsers than
cobbling together regexes in a state machine that uses globals :-)
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

[toc] | [prev] | [next] | [standalone]

#1566866 — Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP)

Am 25.01.2017 um 11:24 schrieb Jani Nikula <jani.nikula@intel.com>:

> Markus, thanks for your work on this.

Thanks for your comments!

> Excuse me for my bluntness, but I think changing everything in a single
> commit, or even a few commits, is strictly not acceptable.

OK, I understand.

> When I changed *small* things in scripts/kernel-doc, I would make
> htmldocs before and after the change, and recursively diff the produced
> output to ensure there were no surprises. We already have enough
> documentation that a manual eyeballing of the output is simply not
> sufficient to ensure things don't break.

> The diff in output between before and after this series? 160k lines of
> unified diff without context ('diff -u0 -r old new | wc -l').
> 
> Many of the changes are improvements on the result, such as using proper
> <div> tags for function parameter lists etc., but clearly changing the
> output should be independent of changing the parser, so we have some
> chance of validating the parser.


Hmm ... I try to sort my thoughts on this:

The both parser are generating reST output. We have tested reST output
so it should be enough to compare the reST from the old one with the
new one ... at least theoretical.

But the problem I see here is, that the perl script generates a
reST output which I can't use. As an example we can take a look at
the man-page builder I shipped in the series.

 https://www.mail-archive.com/linux-doc@vger.kernel.org/msg09017.html

In the commit message there is a small example:

  <section docname="basics ...>
    <title>get_sd_load_idx</title>
    <kernel_doc_man manpage="get_sd_load_idx.9"/>
    ...
    <desc desctype="function" domain="c"....>
      <desc_signature ....>
        <desc_type>int</desc_type>
        ...

You see that it has <section> tag with childs <title/>, <kernel_doc_man/> 
and so on. This structured markup is used by builders, they navigate through
the structured tree picking up nodes and spit out some man-page html, or
whatever builder it is.

ATM the perl parser generates a reST output which does not have such
a structured tree, so the builder can't navigate in.

So, what I mean is, the new parser has to generate a complete different reST
output and thats why we can't compare the perl parser with python one on a reST
basis ... and if reST is different, HTML is different :(

So we do not have any chance to track regression when switching from
the old to the new parser.

Thats are my thoughts on this topic, may be you have a solution for this?

> 
>>   Ideally at the time of merging, we would be able to build the docs with
>>   *either* kerneldoc.
> 
> I'd be fine with switching over in a single commit that doesn't
> drastically change the output.

One solution might be to improve the reST output of the perl script
first, so that it produce something which has a structure and we
all can agree on (short: reST output is the reference, ATM the reference
need some improvements)

If this is a way we like to go, I can send a patch for the perl script,
so that we can commit one a reST reference.

> A drop-in replacement. But that's not the
> case here.
> 
>> - I'll have to try it out to see how noisy it is.  I'm not opposed to
>>   stricter checks; indeed, they could be a good thing.  But we might want
>>   to have an option so we can cut back on the noise by default.
> 
> The increase in 'make htmldocs' build log was from 1521 to 2791 lines in
> my tree. Arguably there was useful extra diagnosis, but some of it was
> the printouts of long lists of definitions that were not found, one per
> line. So it could be condensed without losing info too.

Yes, this was just a 1:1 merge from my POC, there are a lot of things
which could be meld down. ATM, for me it is important to get a feedback 
on the functionalities and concepts of kernel-doc apps (RFC).

> On to performance. With the default build options the new system was
> noticeably slower than the current one, with a 50% increase on my
> machine. But what really caught me by surprise was that passing
> SPHINXOPTS=-j5 to parallelize worked better on the current system,
> making the new one a whopping 70% slower. Of course, the argument is
> that the proposed parser does more and is better, but due to the
> monolithic change it's impossible to pinpoint the culprit or do a proper
> cost/benefit analysis on this. Again, this calls for a more broken down
> series of patches to make the changes.

Ups, I have to look closer ... I thought the py-solution is faster 
since it does not for processes and does some caching.

> Finally, while I'd love to see scripts/kernel-doc go, I do have to ask
> if changing roughly 3k lines of Perl to roughly 3k lines of Python (*)
> really makes everything better? They both still parse everything using a
> large pile of regular expressions and a clunky state machine. When I
> look at the code, I'm afraid I do not get that liberating feeling of
> throwing out old junk in favor of something small or elegant or even
> obviously more maintainable than the old one. The new one offers more
> features, but repeatedly we face the problem that it's all lumped in
> together with the parser change. We should be able to look at the parser
> change and the other improvements separately.
> 
> That said, perhaps having an elegant parser (perhaps based on a compiler
> plugin) is incompatible with the idea of making it a bug-for-bug drop-in
> replacement of the old one, and it's something we need to think about.

Before I started implementing the parser I thought about separating
parsing from generating reST. I played a bit with pycparser

  https://github.com/eliben/pycparser

but I realized that the coverage of those parser might be not
enough for the kernel sources. At this time you mentioned sparse.
I haven't had time to at sparse but I guess that this is the
tool.

-- Markus --


> All in all I think the message should be clear: this needs to be split
> into small, incremental changes. Just like we do everything in the
> kernel.
> 
> 
> BR,
> Jani.
> 
> 
> (*) Please do not get hung up on these numbers. The Python version does
>    more in some ways, but adds more deps such as fspath that's not
>    included in the figures, and the Perl version outputs more
>    formats. It's not an apples to apples comparison. Let's just say
>    they are somewhere in the same ballpark.
> 
> -- 
> Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1566944 — Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP)

On Wed, 25 Jan 2017, Markus Heiser <markus.heiser@darmarit.de> wrote:
> Am 25.01.2017 um 11:24 schrieb Jani Nikula <jani.nikula@intel.com>:
>
>> Markus, thanks for your work on this.
>
> Thanks for your comments!
>
>> Excuse me for my bluntness, but I think changing everything in a single
>> commit, or even a few commits, is strictly not acceptable.
>
> OK, I understand.
>
>> When I changed *small* things in scripts/kernel-doc, I would make
>> htmldocs before and after the change, and recursively diff the produced
>> output to ensure there were no surprises. We already have enough
>> documentation that a manual eyeballing of the output is simply not
>> sufficient to ensure things don't break.
>
>> The diff in output between before and after this series? 160k lines of
>> unified diff without context ('diff -u0 -r old new | wc -l').
>> 
>> Many of the changes are improvements on the result, such as using proper
>> <div> tags for function parameter lists etc., but clearly changing the
>> output should be independent of changing the parser, so we have some
>> chance of validating the parser.
>
>
> Hmm ... I try to sort my thoughts on this:
>
> The both parser are generating reST output. We have tested reST output
> so it should be enough to compare the reST from the old one with the
> new one ... at least theoretical.
>
> But the problem I see here is, that the perl script generates a
> reST output which I can't use. As an example we can take a look at
> the man-page builder I shipped in the series.

Sorry, I still don't understand *why* you can't use the same rst. Your
explanation seems to relate to man pages, but man pages come
*afterwards*, and are a separate improvement. I know you talk about lack
of proper structure and all that, but *why* can it strictly not be used,
if the *current* rst clearly can be used?

BR,
Jani.


>
>  https://www.mail-archive.com/linux-doc@vger.kernel.org/msg09017.html
>
> In the commit message there is a small example:
>
>   <section docname="basics ...>
>     <title>get_sd_load_idx</title>
>     <kernel_doc_man manpage="get_sd_load_idx.9"/>
>     ...
>     <desc desctype="function" domain="c"....>
>       <desc_signature ....>
>         <desc_type>int</desc_type>
>         ...
>
> You see that it has <section> tag with childs <title/>, <kernel_doc_man/> 
> and so on. This structured markup is used by builders, they navigate through
> the structured tree picking up nodes and spit out some man-page html, or
> whatever builder it is.
>
> ATM the perl parser generates a reST output which does not have such
> a structured tree, so the builder can't navigate in.
>
> So, what I mean is, the new parser has to generate a complete different reST
> output and thats why we can't compare the perl parser with python one on a reST
> basis ... and if reST is different, HTML is different :(
>
> So we do not have any chance to track regression when switching from
> the old to the new parser.
>
> Thats are my thoughts on this topic, may be you have a solution for this?
>
>> 
>>>   Ideally at the time of merging, we would be able to build the docs with
>>>   *either* kerneldoc.
>> 
>> I'd be fine with switching over in a single commit that doesn't
>> drastically change the output.
>
> One solution might be to improve the reST output of the perl script
> first, so that it produce something which has a structure and we
> all can agree on (short: reST output is the reference, ATM the reference
> need some improvements)
>
> If this is a way we like to go, I can send a patch for the perl script,
> so that we can commit one a reST reference.
>
>> A drop-in replacement. But that's not the
>> case here.
>> 
>>> - I'll have to try it out to see how noisy it is.  I'm not opposed to
>>>   stricter checks; indeed, they could be a good thing.  But we might want
>>>   to have an option so we can cut back on the noise by default.
>> 
>> The increase in 'make htmldocs' build log was from 1521 to 2791 lines in
>> my tree. Arguably there was useful extra diagnosis, but some of it was
>> the printouts of long lists of definitions that were not found, one per
>> line. So it could be condensed without losing info too.
>
> Yes, this was just a 1:1 merge from my POC, there are a lot of things
> which could be meld down. ATM, for me it is important to get a feedback 
> on the functionalities and concepts of kernel-doc apps (RFC).
>
>> On to performance. With the default build options the new system was
>> noticeably slower than the current one, with a 50% increase on my
>> machine. But what really caught me by surprise was that passing
>> SPHINXOPTS=-j5 to parallelize worked better on the current system,
>> making the new one a whopping 70% slower. Of course, the argument is
>> that the proposed parser does more and is better, but due to the
>> monolithic change it's impossible to pinpoint the culprit or do a proper
>> cost/benefit analysis on this. Again, this calls for a more broken down
>> series of patches to make the changes.
>
> Ups, I have to look closer ... I thought the py-solution is faster 
> since it does not for processes and does some caching.
>
>> Finally, while I'd love to see scripts/kernel-doc go, I do have to ask
>> if changing roughly 3k lines of Perl to roughly 3k lines of Python (*)
>> really makes everything better? They both still parse everything using a
>> large pile of regular expressions and a clunky state machine. When I
>> look at the code, I'm afraid I do not get that liberating feeling of
>> throwing out old junk in favor of something small or elegant or even
>> obviously more maintainable than the old one. The new one offers more
>> features, but repeatedly we face the problem that it's all lumped in
>> together with the parser change. We should be able to look at the parser
>> change and the other improvements separately.
>> 
>> That said, perhaps having an elegant parser (perhaps based on a compiler
>> plugin) is incompatible with the idea of making it a bug-for-bug drop-in
>> replacement of the old one, and it's something we need to think about.
>
> Before I started implementing the parser I thought about separating
> parsing from generating reST. I played a bit with pycparser
>
>   https://github.com/eliben/pycparser
>
> but I realized that the coverage of those parser might be not
> enough for the kernel sources. At this time you mentioned sparse.
> I haven't had time to at sparse but I guess that this is the
> tool.
>
> -- Markus --
>
>
>> All in all I think the message should be clear: this needs to be split
>> into small, incremental changes. Just like we do everything in the
>> kernel.
>> 
>> 
>> BR,
>> Jani.
>> 
>> 
>> (*) Please do not get hung up on these numbers. The Python version does
>>    more in some ways, but adds more deps such as fspath that's not
>>    included in the figures, and the Perl version outputs more
>>    formats. It's not an apples to apples comparison. Let's just say
>>    they are somewhere in the same ballpark.
>> 
>> -- 
>> Jani Nikula, Intel Open Source Technology Center
>

-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1567250 — Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP)

Am 25.01.2017 um 21:59 schrieb Jani Nikula <jani.nikula@intel.com>:

>> But the problem I see here is, that the perl script generates a
>> reST output which I can't use. As an example we can take a look at
>> the man-page builder I shipped in the series.
> 
> Sorry, I still don't understand *why* you can't use the same rst. Your
> explanation seems to relate to man pages, but man pages come
> *afterwards*, and are a separate improvement. I know you talk about lack
> of proper structure and all that, but *why* can it strictly not be used,
> if the *current* rst clearly can be used?

"afterwards" is the word, that lets me slowly realize, that I have to
stop solving the world's problems with one patch. Now I guess how my
next patch series has to look like. Thanks! ... for being patient with
me.

Before I start, I want to hear your thoughts about the parsing
aspect ...

>>> That said, perhaps having an elegant parser (perhaps based on a compiler
>>> plugin) is incompatible with the idea of making it a bug-for-bug drop-in
>>> replacement of the old one, and it's something we need to think about.

Did you have any suggestions?

-- Markus --

[toc] | [prev] | [next] | [standalone]

#1567267 — Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP)

On Thu, 26 Jan 2017, Markus Heiser <markus.heiser@darmarit.de> wrote:
> Am 25.01.2017 um 21:59 schrieb Jani Nikula <jani.nikula@intel.com>:
>
>>> But the problem I see here is, that the perl script generates a
>>> reST output which I can't use. As an example we can take a look at
>>> the man-page builder I shipped in the series.
>> 
>> Sorry, I still don't understand *why* you can't use the same rst. Your
>> explanation seems to relate to man pages, but man pages come
>> *afterwards*, and are a separate improvement. I know you talk about lack
>> of proper structure and all that, but *why* can it strictly not be used,
>> if the *current* rst clearly can be used?
>
> "afterwards" is the word, that lets me slowly realize, that I have to
> stop solving the world's problems with one patch. Now I guess how my
> next patch series has to look like. Thanks! ... for being patient with
> me.

Indeed, we change the world, one small incremental patch at a time. ;)

> Before I start, I want to hear your thoughts about the parsing
> aspect ...
>
>>>> That said, perhaps having an elegant parser (perhaps based on a
>>>> compiler plugin) is incompatible with the idea of making it a
>>>> bug-for-bug drop-in replacement of the old one, and it's something
>>>> we need to think about.
>
> Did you have any suggestions?

The perfect is the enemy of the good... If we see that the current Perl
parser just rewritten in Python really is an improvement, we should
consider it. But as I wrote, there are still issues there, like
performance, that we need to understand. I'll mostly defer to Jon on
this.

But before we plunge on with this, I would like to see at least some
research into reusing existing parsers which I would expect are
plentiful. We may end up deciding regexps are the way to go after all,
but I'd like it to be based on a decision rather than a lack of one. And
we might decide to look at this as a later improvement instead as well.

I've looked at python-clang myself, but it's a huge dependency, and it's
not trivial to cover all the things that the current one does with
that. I'd dismiss that.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1567652 — Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP)

On Wed, 25 Jan 2017 20:07:47 +0100
Markus Heiser <markus.heiser@darmarit.de> wrote:

> So, what I mean is, the new parser has to generate a complete different reST
> output and thats why we can't compare the perl parser with python one on a reST
> basis ... and if reST is different, HTML is different :(
> 
> So we do not have any chance to track regression when switching from
> the old to the new parser.
> 
> Thats are my thoughts on this topic, may be you have a solution for this?

The solution, I think, is as has been described by others in the thread.
I'll make a try at it now :)

The objectives in a patch set are something like this:

 - Replace the kernel-doc utility with one that is easier to maintain and
   enhance.

 - Add various enhancements (man pages, linting, better output, better
   parsing) to the docs build system.

What everybody is complaining about here is that all of that stuff is
being thrown in together into a single patch set.  We don't do things that
way because long experience says we'll create a mess that takes a long
time to straighten out again.

As I said before, I'm very much amenable to the idea of replacing
kernel-doc with one that is easier to work with.  I haven't yet had the
time to look closely enough at yours to have an opinion on whether it does
that or not.  But, assuming it does, the proper way to make this change is
to provide a new kerneldoc that behaves as closely to the old one as
possible, with an absolute minimum of output changes.

Doing it that way probably seems like a pretty annoying request.  But it
lets us validate its basic mechanics and be confident that we won't break
the docs build in weird ways.  It also lets us evaluate the question of
whether the replacement has merit in its own right, independent of any
other change we want to make.  Give me a new kerneldoc that passes those
tests, and I'll happily merge it.  (I have some sympathy with the idea
that we should look into other parsers, but I would not hold up a new
kerneldoc that passed those tests on this basis alone.)

*Then* we can start adding the other stuff, which, from a first look,
appears to be stuff that we very much want to have.  Each one of those,
too, should stand alone and pass muster on its own merits.  Changes
presented in this way could be merged in the same development cycle if
they are ready, but we need to be able to evaluate each one separately.

Does this make sense?  We all really appreciate the work you're doing
here, we're just asking that it be done in an evolutionary manner so we
can evaluate it properly.

Thanks,

jon

[toc] | [prev] | [next] | [standalone]

#1567661 — Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP)

On Thu, 26 Jan 2017, Jonathan Corbet <corbet@lwn.net> wrote:
> Give me a new kerneldoc that passes those tests, and I'll happily
> merge it.  (I have some sympathy with the idea that we should look
> into other parsers, but I would not hold up a new kerneldoc that
> passed those tests on this basis alone.)

I'll just note in passing that having another parser that actually works
for our needs might be a pink unicorn pony. It might exist, it might
not, and someone would have to put in the hours to try to find it, tame
it, and bring it to the kernel. But it would be awesome to
have. Switching to a homebrew Python parser first does not preclude a
unicorn hunt later.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1567989 — Re: [RFC PATCH v1 2/6] kernel-doc: replace kernel-doc perl parser with a pure python one (WIP)

Am 26.01.2017 um 20:26 schrieb Jani Nikula <jani.nikula@intel.com>:

> On Thu, 26 Jan 2017, Jonathan Corbet <corbet@lwn.net> wrote:
>> Give me a new kerneldoc that passes those tests, and I'll happily
>> merge it.  (I have some sympathy with the idea that we should look
>> into other parsers, but I would not hold up a new kerneldoc that
>> passed those tests on this basis alone.)
> 
> I'll just note in passing that having another parser that actually works
> for our needs might be a pink unicorn pony. It might exist, it might
> not, and someone would have to put in the hours to try to find it, tame
> it, and bring it to the kernel. But it would be awesome to
> have. Switching to a homebrew Python parser first does not preclude a
> unicorn hunt later.

Here are my experience about parsing C code and kernel-doc comments.

The reg-expressions divide into two parts:

a.) those parsing "C sources", catching up function prototypes, structs etc. and

b.) those parsing "kernel-doc comments", catching up attribute descriptions,
   cross references etc.

When I developed the py-version in my POC I realized that the reg-expressions
parsing C sources (a.) aren't so bad. They have a long history and are well
tested against kernel' sources (As far as I remember, I added only one regexp
more to match function prototypes).

 This was the time where I looked at some other parsing tool and
 after a day I throw away the idea of using a external parser
 tool, first.

Most problems I have had, was parsing the kernel-doc markup itself. E.g. the
ambiguous attribute markup "* @foo: lorem" and its cross-ref "@foo". The latter
syntax is ambiguous, it fails mostly on new-lines and with strings like
"me@foo.bar".

When I looked at the whole sources, I also realized that we have two flavors of
kernel-doc markups.

b.1) Those from traditional DocBook where whitespaces aren't markups and

b.2) those which has been rewritten with reST markup in, where whitespaces are a
    part of the reST.

But this was only the half truth of b.2) : the 'new' markup did not only
consists of pure reST markup. For convince it is a mix of kernel-doc markup and
reST markup (e.g. remember the cross-ref mentioned above).

I suppose that we will never completely get rid off traditional (b.1), since
this means; changing the whole kernel source ;)

At that time I wanted to implement a parser which has the ability to handle both
flavors. A (undocumented) 'vintage' mode an the user-documented 'reST' mode.
But what is the criteria to switch from one mode to the other?  For this I made
a primitive assumption: every C source file which is used in a ".. kernel-doc::"
directive has to be marked up with the modern reST flavor.

ATM, the py-version of kernel-doc implements the same state machine as the perl
one and the modes are implemented in the same state machine (not perfect but it
worked for me first, suppose we can make it better).

I remember about a very early discussion we had about those modes and I know
that it doesn't find friends in the community (at that time).  May be today we
have more experience and new ideas.

  I really like to see (to work on) a parser with we can parse the
  whole kernel source and generate reST from.

What do you think, is it a bloody idea?

Thanks!

-- Markus --

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web