Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1413696 > unrolled thread

[PATCH v2 00/38] Documentation/sphinx

Back to article view | Back to linux.kernel

Contents

  [PATCH v2 00/38] Documentation/sphinx Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:40 +0200
    [PATCH v2 10/38] Documentation/sphinx: nicer referencing of struct in docbook->rst conversion Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:40 +0200
    [PATCH v2 01/38] kernel-doc/rst: fix use of uninitialized value Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:40 +0200
    [PATCH v2 13/38] kernel-doc/rst: do not output DOC: section titles for requested ones Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:40 +0200
    [PATCH v2 14/38] kernel-doc/rst: reference functions according to C domain spec Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:40 +0200
    [PATCH v2 08/38] sphinx: cheesy script to convert .tmpl files Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:40 +0200
    [PATCH v2 02/38] kernel-doc: support printing exported and non-exported symbols Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:40 +0200
    [PATCH v2 12/38] kernel-doc: add names for output selection Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:40 +0200
    [PATCH v2 15/38] kernel-doc/rst: &foo references are more universal than structs Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:40 +0200
    [PATCH v2 05/38] Documentation/sphinx: add Sphinx kernel-doc directive extension Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:40 +0200
    [PATCH v2 03/38] Documentation/sphinx: add basic working Sphinx configuration and build Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:40 +0200
    [PATCH v2 28/38] kernel-doc/rst: remove fixme comment Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 24/38] kernel-doc/rst: change the output layout Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 22/38] kernel-doc/rst: blank lines in output are not needed Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 37/38] scripts/kernel-doc: Add option to inject line numbers Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 07/38] Documentation/sphinx: set version and release properly Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 32/38] Documentation/sphinx: fix kernel-doc extension on python3 Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 06/38] Documentation/sphinx: configure the kernel-doc extension Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 20/38] kernel-doc: do not regard $, %, or & prefixes as special in section names Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 19/38] kernel-doc/rst: highlight function/struct/enum purpose lines too Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 23/38] kernel-doc: strip leading blank lines from inline doc comments Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 26/38] kernel-doc: strip leading whitespace from continued param descs Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 38/38] doc/sphinx: Track line-number of starting blocks Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 17/38] kernel-doc/rst: add support for struct/union/enum member references Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 31/38] kernel-doc: reset contents and section harder Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 30/38] kernel-doc: concatenate contents of colliding sections Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 29/38] kernel-doc: limit the "section header:" detection to a select few Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
      Re: [PATCH v2 29/38] kernel-doc: limit the "section header:"  detection to a select few Jonathan Corbet <corbet@lwn.net> - 2016-06-09 17:10 +0200
        Re: [PATCH v2 29/38] kernel-doc: limit the "section header:" detection to a select few Jani Nikula <jani.nikula@intel.com> - 2016-06-09 18:50 +0200
    [PATCH v2 36/38] scripts/kernel-doc: Also give functions symbolic names Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 21/38] kernel-doc: fix wrong code indentation Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 34/38] scripts/kernel-doc: Remove duplicated DOC: start handling Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 18/38] kernel-doc/rst: drop redundant unescape in highlighting Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 33/38] doc/sphinx: Pass right filename as source Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 16/38] kernel-doc/rst: add support for &union foo and &typedef foo references Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 25/38] kernel-doc: improve handling of whitespace on the first line param description Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 35/38] doc/sphinx: Stop touching state_machine internals Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 27/38] kernel-doc/rst: use *undescribed* instead of _undescribed_ Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    [PATCH v2 09/38] sphinx: update docbook->rst conversion script match C domain spec Jani Nikula <jani.nikula@intel.com> - 2016-06-04 13:50 +0200
    Re: [PATCH v2 00/38] Documentation/sphinx Daniel Vetter <daniel@ffwll.ch> - 2016-06-04 14:20 +0200
    Re: [PATCH v2 00/38] Documentation/sphinx Jonathan Corbet <corbet@lwn.net> - 2016-06-09 22:00 +0200
      Re: [PATCH v2 00/38] Documentation/sphinx Daniel Vetter <daniel.vetter@ffwll.ch> - 2016-06-10 20:20 +0200
        Re: [PATCH v2 00/38] Documentation/sphinx Dave Airlie <airlied@gmail.com> - 2016-06-10 22:50 +0200

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

#1413719 — [PATCH v2 23/38] kernel-doc: strip leading blank lines from inline doc comments

The inline member markup allows whitespace lines before the actual
documentation starts. Strip the leading blank lines. This improves the
rst output.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index a89ff3ca366c..e8ea295567a3 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -2889,6 +2889,10 @@ sub process_file($) {
 	    } elsif (/$doc_content/) {
 		if ($inline_doc_state == STATE_INLINE_TEXT) {
 		    $contents .= $1 . "\n";
+		    # nuke leading blank lines
+		    if ($contents =~ /^\s*$/) {
+			$contents = "";
+		    }
 		} elsif ($inline_doc_state == STATE_INLINE_NAME) {
 		    $inline_doc_state = STATE_INLINE_ERROR;
 		    print STDERR "Warning(${file}:$.): ";
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1413720 — [PATCH v2 26/38] kernel-doc: strip leading whitespace from continued param descs

If a param description spans multiple lines, check any leading
whitespace in the first continuation line, and remove same amount of
whitespace from following lines.

This allows indentation in the multi-line parameter descriptions for
aesthetical reasons while not causing accidentally significant
indentation in the rst output.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 16 +++++++++++++++-
 1 file changed, 15 insertions(+), 1 deletion(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index e93e796b17ce..f6f37e71dc08 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -2701,6 +2701,7 @@ sub process_file($) {
     my $in_purpose = 0;
     my $initial_section_counter = $section_counter;
     my ($orig_file) = @_;
+    my $leading_space;
 
     if (defined($ENV{'SRCTREE'})) {
 	$file = "$ENV{'SRCTREE'}" . "/" . $orig_file;
@@ -2822,6 +2823,7 @@ sub process_file($) {
 		    $contents .= "\n";
 		}
 		$section = $newsection;
+		$leading_space = undef;
 	    } elsif (/$doc_end/) {
 		if (($contents ne "") && ($contents ne "\n")) {
 		    dump_section($file, $section, xml_escape($contents));
@@ -2856,7 +2858,19 @@ sub process_file($) {
 		    $declaration_purpose .= " " . xml_escape($1);
 		    $declaration_purpose =~ s/\s+/ /g;
 		} else {
-		    $contents .= $1 . "\n";
+		    my $cont = $1;
+		    if ($section =~ m/^@/ || $section eq $section_context) {
+			if (!defined $leading_space) {
+			    if ($cont =~ m/^(\s+)/) {
+				$leading_space = $1;
+			    } else {
+				$leading_space = "";
+			    }
+			}
+
+			$cont =~ s/^$leading_space//;
+		    }
+		    $contents .= $cont . "\n";
 		}
 	    } else {
 		# i dont know - bad line?  ignore.
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1413721 — [PATCH v2 38/38] doc/sphinx: Track line-number of starting blocks

From: Daniel Vetter <daniel.vetter@ffwll.ch>

Design is pretty simple: kernel-doc inserts breadcrumbs with line
numbers, and sphinx picks them up. At first I went with a sphinx
comment, but inserting those at random places seriously upsets the
parser, and must be filtered. Hence why this version now uses "#define
LINEO " since one of these ever escape into output it's pretty clear
there is a bug.

It seems to work well, and at least the 2-3 errors where sphinx
complained about something that was not correct in kernel-doc text the
line numbers matched up perfectly.

v2: Instead of noodling around in the parser state machine, create
a ViewList and parse it ourselves. This seems to be the recommended
way, per Jani's suggestion.

v3:
- Split out ViewList pach. Splitting the kernel-doc changes from the
  sphinx ones isn't possible, since emitting the LINENO lines wreaks
  havoc with the rst formatting. We must filter them.

- Improve the regex per Jani's suggestions, and compile it just once
  for speed.

- Now that LINENO lines are eaten, also add them to function parameter
  descriptions. Much less content and offset than for in-line struct
  member descriptions, but still nice to know which exact continuation
  line upsets sphinx.

- Simplify/clarify the line +/-1 business a bit.

v4: Split out the scripts/kernel-doc changes and make line-numbers
opt-in, as suggested by Jani.

Cc: Jani Nikula <jani.nikula@intel.com>
Cc: linux-doc@vger.kernel.org
Cc: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/sphinx/kernel-doc.py | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/Documentation/sphinx/kernel-doc.py b/Documentation/sphinx/kernel-doc.py
index bd422870101e..4adfb0e91ecc 100644
--- a/Documentation/sphinx/kernel-doc.py
+++ b/Documentation/sphinx/kernel-doc.py
@@ -30,6 +30,7 @@
 import os
 import subprocess
 import sys
+import re
 
 from docutils import nodes, statemachine
 from docutils.statemachine import ViewList
@@ -50,7 +51,7 @@ class KernelDocDirective(Directive):
 
     def run(self):
         env = self.state.document.settings.env
-        cmd = [env.config.kerneldoc_bin, '-rst']
+        cmd = [env.config.kerneldoc_bin, '-rst', '-enable-lineno']
 
         filename = env.config.kerneldoc_srctree + '/' + self.arguments[0]
 
@@ -93,7 +94,19 @@ class KernelDocDirective(Directive):
                 sys.stderr.write(err)
 
             lines = statemachine.string2lines(out, tab_width, convert_whitespace=True)
-            result = ViewList(lines, source)
+            result = ViewList()
+
+            lineoffset = 0;
+            line_regex = re.compile("^#define LINENO ([0-9]+)$")
+            for line in lines:
+                match = line_regex.search(line)
+                if match:
+                    # sphinx counts lines from 0
+                    lineoffset = int(match.group(1)) - 1
+                    # we must eat our comments since the upset the markup
+                else:
+                    result.append(line, source, lineoffset)
+                    lineoffset += 1
 
             node = nodes.section()
             node.document = self.state.document
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1413723 — [PATCH v2 17/38] kernel-doc/rst: add support for struct/union/enum member references

Link "&foo->bar", "&foo->bar()", "&foo.bar", and "&foo.bar()" to the
struct/union/enum foo definition. The members themselves do not
currently have anchors to link to, but this is better than nothing, and
promotes a universal notation.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 446c0912395e..e0fd14f6d711 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -214,6 +214,8 @@ my $type_enum_full = '\&(enum)\s*([_\w]+)';
 my $type_struct_full = '\&(struct)\s*([_\w]+)';
 my $type_typedef_full = '\&(typedef)\s*([_\w]+)';
 my $type_union_full = '\&(union)\s*([_\w]+)';
+my $type_member = '\&([_\w]+)((\.|->)[_\w]+)';
+my $type_member_func = $type_member . '\(\)';
 
 # Output conversion substitutions.
 #  One for each output format
@@ -282,6 +284,9 @@ my $blankline_text = "";
 # rst-mode
 my @highlights_rst = (
                        [$type_constant, "``\$1``"],
+                       # Note: need to escape () to avoid func matching later
+                       [$type_member_func, "\\:c\\:type\\:`\$1\$2\\\\(\\\\) <\$1>`"],
+                       [$type_member, "\\:c\\:type\\:`\$1\$2 <\$1>`"],
                        [$type_func, "\\:c\\:func\\:`\$1()`"],
                        [$type_struct_full, "\\:c\\:type\\:`\$1 \$2 <\$2>`"],
                        [$type_enum_full, "\\:c\\:type\\:`\$1 \$2 <\$2>`"],
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1413724 — [PATCH v2 31/38] kernel-doc: reset contents and section harder

If the documentation comment does not have params or sections, the
section heading may leak from the previous documentation comment.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 3ac4b57ed76a..0eb2e7b5bf10 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -2763,6 +2763,8 @@ sub process_file($) {
 		}
 
 		$state = STATE_FIELD;
+		$contents = "";
+		$section = $section_default;
 		if (/-(.*)/) {
 		    # strip leading/trailing/multiple spaces
 		    $descr= $1;
@@ -2960,6 +2962,7 @@ sub process_file($) {
 		elsif (/$doc_end/)
 		{
 			dump_doc_section($file, $section, xml_escape($contents));
+			$section = $section_default;
 			$contents = "";
 			$function = "";
 			%parameterdescs = ();
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1413725 — [PATCH v2 30/38] kernel-doc: concatenate contents of colliding sections

If there are multiple sections with the same section name, the current
implementation results in several sections by the same heading, with the
content duplicated from the last section to all. Even if there's the
error message, a more graceful approach is to combine all the
identically named sections into one, with concatenated contents.

With the supported sections already limited to select few, there are
massively fewer collisions than there used to be, but this is still
useful for e.g. when function parameters are documented in the middle of
a documentation comment, with description spread out above and
below. (This is not a recommended documentation style, but used in the
kernel nonetheless.)

We can now also demote the error to a warning.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 20136564f264..3ac4b57ed76a 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -524,11 +524,13 @@ sub dump_section {
     } else {
 #	print STDERR "other section '$name' = '$contents'\n";
 	if (defined($sections{$name}) && ($sections{$name} ne "")) {
-		print STDERR "${file}:$.: error: duplicate section name '$name'\n";
-		++$errors;
+	    print STDERR "${file}:$.: warning: duplicate section name '$name'\n";
+	    ++$warnings;
+	    $sections{$name} .= $contents;
+	} else {
+	    $sections{$name} = $contents;
+	    push @sectionlist, $name;
 	}
-	$sections{$name} = $contents;
-	push @sectionlist, $name;
     }
 }
 
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1413726 — [PATCH v2 29/38] kernel-doc: limit the "section header:" detection to a select few

kernel-doc currently identifies anything matching "section header:"
(specifically a string of word characters and spaces followed by a
colon) as a new section in the documentation comment, and renders the
section header accordingly.

Unfortunately, this turns all uses of colon into sections, mostly
unintentionally. Considering the output, erroneously creating sections
when not intended is always worse than erroneously not creating sections
when intended. For example, a line with "http://example.com" turns into
a "http" heading followed by "//example.com" in normal text style, which
is quite ugly. OTOH, "WARNING: Beware of the Leopard" is just fine even
if "WARNING" does not turn into a heading.

It is virtually impossible to change all the kernel-doc comments, either
way. The compromise is to pick the most commonly used and depended on
section headers (with variants) and accept them as section headers.

The accepted section headers are, case insensitive:

 * description:
 * context:
 * return:
 * returns:

Additionally, case sensitive:

 * @return:

All of the above are commonly used in the kernel-doc comments, and will
result in worse output if not identified as section headers. Also,
kernel-doc already has some special handling for all of them, so there's
nothing particularly controversial in adding more special treatment for
them.

While at it, improve the whitespace handling surrounding section
names. Do not consider the whitespace as part of the name.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 19 +++++++++++++++++--
 1 file changed, 17 insertions(+), 2 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 425a94be04f6..20136564f264 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -401,7 +401,8 @@ my $doc_end = '\*/';
 my $doc_com = '\s*\*\s*';
 my $doc_com_body = '\s*\* ?';
 my $doc_decl = $doc_com . '(\w+)';
-my $doc_sect = $doc_com . '(\@?[\w\s]+):(.*)';
+# @params and a strictly limited set of supported section names
+my $doc_sect = $doc_com . '\s*(\@\w+|description|context|returns?)\s*:(.*)';
 my $doc_content = $doc_com_body . '(.*)';
 my $doc_block = $doc_com . 'DOC:\s*(.*)?';
 my $doc_inline_start = '^\s*/\*\*\s*$';
@@ -417,6 +418,8 @@ my $sectcheck;
 my $struct_actual;
 
 my $contents = "";
+
+# the canonical section names. see also $doc_sect above.
 my $section_default = "Description";	# default section
 my $section_intro = "Introduction";
 my $section = $section_default;
@@ -2798,10 +2801,22 @@ sub process_file($) {
 		$state = STATE_NORMAL;
 	    }
 	} elsif ($state == STATE_FIELD) {	# look for head: lines, and include content
-	    if (/$doc_sect/o) {
+	    if (/$doc_sect/i) { # case insensitive for supported section names
 		$newsection = $1;
 		$newcontents = $2;
 
+		# map the supported section names to the canonical names
+		if ($newsection =~ m/^description$/i) {
+		    $newsection = $section_default;
+		} elsif ($newsection =~ m/^context$/i) {
+		    $newsection = $section_context;
+		} elsif ($newsection =~ m/^returns?$/i) {
+		    $newsection = $section_return;
+		} elsif ($newsection =~ m/^\@return$/) {
+		    # special: @return is a section, not a param description
+		    $newsection = $section_return;
+		}
+
 		if (($contents ne "") && ($contents ne "\n")) {
 		    if (!$in_doc_sect && $verbose) {
 			print STDERR "${file}:$.: warning: contents before sections\n";
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1418416 — Re: [PATCH v2 29/38] kernel-doc: limit the "section header:" detection to a select few

On Sat,  4 Jun 2016 14:37:30 +0300
Jani Nikula <jani.nikula@intel.com> wrote:

> kernel-doc currently identifies anything matching "section header:"
> (specifically a string of word characters and spaces followed by a
> colon) as a new section in the documentation comment, and renders the
> section header accordingly.
> 
> Unfortunately, this turns all uses of colon into sections, mostly
> unintentionally.

I've been looking at how the patch series changes (traditional) htmldocs
generation, and this one is responsible for a lot of them.  Those changes
are almost all good!  There is a lot of cruft out there.  Just FWIW, I'm
going to add a patch putting "note|examples|" into the list, since those
appear to be intentional.

jon

[toc] | [prev] | [next] | [standalone]

#1418483 — Re: [PATCH v2 29/38] kernel-doc: limit the "section header:" detection to a select few

On Thu, 09 Jun 2016, Jonathan Corbet <corbet@lwn.net> wrote:
> On Sat,  4 Jun 2016 14:37:30 +0300
> Jani Nikula <jani.nikula@intel.com> wrote:
>
>> kernel-doc currently identifies anything matching "section header:"
>> (specifically a string of word characters and spaces followed by a
>> colon) as a new section in the documentation comment, and renders the
>> section header accordingly.
>> 
>> Unfortunately, this turns all uses of colon into sections, mostly
>> unintentionally.
>
> I've been looking at how the patch series changes (traditional) htmldocs
> generation, and this one is responsible for a lot of them.  Those changes
> are almost all good!  There is a lot of cruft out there.  Just FWIW, I'm
> going to add a patch putting "note|examples|" into the list, since those
> appear to be intentional.

Heh, you scared me a bit until the "almost all good" part. :)

I'm fine with adding more to the list, although I intentionally tried to
keep them to a minimum initially. I had this vague idea of turning some
of the "note" and "warning" type things into rst admonitions [1] later
on, but I don't really have a concrete plan yet.

BR,
Jani.

[1] http://docutils.sourceforge.net/docs/ref/rst/directives.html#admonitions

-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1413727 — [PATCH v2 36/38] scripts/kernel-doc: Also give functions symbolic names

From: Daniel Vetter <daniel.vetter@ffwll.ch>

state3 = prototype parsing, so name them accordingly.

Cc: Jani Nikula <jani.nikula@intel.com>
Cc: linux-doc@vger.kernel.org
Cc: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 9fb26d142a56..4da6f952d18b 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -2593,7 +2593,7 @@ sub syscall_munge() {
 	}
 }
 
-sub process_state3_function($$) {
+sub process_proto_function($$) {
     my $x = shift;
     my $file = shift;
 
@@ -2623,7 +2623,7 @@ sub process_state3_function($$) {
     }
 }
 
-sub process_state3_type($$) {
+sub process_proto_type($$) {
     my $x = shift;
     my $file = shift;
 
@@ -2937,9 +2937,9 @@ sub process_file($) {
 		$state = STATE_INLINE;
 		$inline_doc_state = STATE_INLINE_NAME;
 	    } elsif ($decl_type eq 'function') {
-		process_state3_function($_, $file);
+		process_proto_function($_, $file);
 	    } else {
-		process_state3_type($_, $file);
+		process_proto_type($_, $file);
 	    }
 	} elsif ($state == STATE_DOCBLOCK) {
 		if (/$doc_end/)
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1413728 — [PATCH v2 21/38] kernel-doc: fix wrong code indentation

No functional changes.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index f795660dfc7b..c154c3205df1 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -2878,7 +2878,7 @@ sub process_file($) {
 		           substr($contents, 0, 1) eq "\t") {
 			$contents = substr($contents, 1);
 		    }
-		$contents .= "\n";
+		    $contents .= "\n";
 		}
 		$inline_doc_state = STATE_INLINE_TEXT;
 	    # Documentation block end */
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1413729 — [PATCH v2 34/38] scripts/kernel-doc: Remove duplicated DOC: start handling

From: Daniel Vetter <daniel.vetter@ffwll.ch>

Further up in the state machinery we switch from STATE_NAME to
STATE_DOCBLOCK when we match /$doc_block/. Which means this block of
code here is entirely unreachable, unless there are multiple DOC:
sections within a single kernel-doc comment.

Getting a list of all the files with more than one DOC: section using

$ git grep -c " * DOC:" | grep -v ":1$"

and then doing a full audit of them reveals there are no such comment
blocks in the kernel.

Supporting multiple DOC: sections in a single kernel-doc comment does
not seem like a recommended way of doing things anyway, so nuke the code
for simplicity.

Cc: Jani Nikula <jani.nikula@intel.com>
Cc: linux-doc@vger.kernel.org
Cc: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Daniel Vetter <daniel.vetter@ffwll.ch>
[Jani: amended the commit message]
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 19 +------------------
 1 file changed, 1 insertion(+), 18 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 0eb2e7b5bf10..9fb26d142a56 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -2942,24 +2942,7 @@ sub process_file($) {
 		process_state3_type($_, $file);
 	    }
 	} elsif ($state == STATE_DOCBLOCK) {
-		# Documentation block
-		if (/$doc_block/) {
-			dump_doc_section($file, $section, xml_escape($contents));
-			$contents = "";
-			$function = "";
-			%parameterdescs = ();
-			%parametertypes = ();
-			@parameterlist = ();
-			%sections = ();
-			@sectionlist = ();
-			$prototype = "";
-			if ( $1 eq "" ) {
-				$section = $section_intro;
-			} else {
-				$section = $1;
-			}
-		}
-		elsif (/$doc_end/)
+		if (/$doc_end/)
 		{
 			dump_doc_section($file, $section, xml_escape($contents));
 			$section = $section_default;
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1413730 — [PATCH v2 18/38] kernel-doc/rst: drop redundant unescape in highlighting

This bit is already done by xml_unescape() above.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 1 -
 1 file changed, 1 deletion(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index e0fd14f6d711..8f9eac509377 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -1796,7 +1796,6 @@ sub output_highlight_rst {
 	if ($line eq "") {
 	    print $lineprefix, $blankline;
 	} else {
-	    $line =~ s/\\\\\\/\&/g;
 	    print $lineprefix, $line;
 	}
 	print "\n";
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1413731 — [PATCH v2 33/38] doc/sphinx: Pass right filename as source

From: Daniel Vetter <daniel.vetter@ffwll.ch>

With this error output becomes almost readable. The line numbers are
still totally bonghits, but that's a lot harder to pull out of
kerneldoc. We'd essentially have to insert some special markers in the
kernel-doc output, split the output along these markers and then
insert each block separately using

     state_machine.insert_input(block, source, first_line)

Cc: Jani Nikula <jani.nikula@intel.com>
Cc: linux-doc@vger.kernel.org
Cc: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/sphinx/kernel-doc.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/Documentation/sphinx/kernel-doc.py b/Documentation/sphinx/kernel-doc.py
index a6e170872c8a..9fc2c27916a9 100644
--- a/Documentation/sphinx/kernel-doc.py
+++ b/Documentation/sphinx/kernel-doc.py
@@ -57,7 +57,7 @@ class KernelDocDirective(Directive):
         env.note_dependency(os.path.abspath(filename))
 
         tab_width = self.options.get('tab-width', self.state.document.settings.tab_width)
-        source = self.state_machine.input_lines.source(self.lineno - self.state_machine.input_offset - 1)
+        source = filename
 
         # FIXME: make this nicer and more robust against errors
         if 'export' in self.options:
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1413733 — [PATCH v2 16/38] kernel-doc/rst: add support for &union foo and &typedef foo references

Let the user use "&union foo" and "&typedef foo" to reference foo. The
difference to using "union &foo", "typedef &foo", or just "&foo" (which
are valid too) is that "union" and "typedef" become part of the link
text.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index e7aa792e7f1b..446c0912395e 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -212,6 +212,8 @@ my $type_struct_xml = '\\&amp;((struct\s*)*[_\w]+)';
 my $type_env = '(\$\w+)';
 my $type_enum_full = '\&(enum)\s*([_\w]+)';
 my $type_struct_full = '\&(struct)\s*([_\w]+)';
+my $type_typedef_full = '\&(typedef)\s*([_\w]+)';
+my $type_union_full = '\&(union)\s*([_\w]+)';
 
 # Output conversion substitutions.
 #  One for each output format
@@ -283,6 +285,8 @@ my @highlights_rst = (
                        [$type_func, "\\:c\\:func\\:`\$1()`"],
                        [$type_struct_full, "\\:c\\:type\\:`\$1 \$2 <\$2>`"],
                        [$type_enum_full, "\\:c\\:type\\:`\$1 \$2 <\$2>`"],
+                       [$type_typedef_full, "\\:c\\:type\\:`\$1 \$2 <\$2>`"],
+                       [$type_union_full, "\\:c\\:type\\:`\$1 \$2 <\$2>`"],
                        # in rst this can refer to any type
                        [$type_struct, "\\:c\\:type\\:`\$1`"],
                        [$type_param, "**\$1**"]
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1413734 — [PATCH v2 25/38] kernel-doc: improve handling of whitespace on the first line param description

Handle whitespace on the first line of param text as if it was the empty
string. There is no need to add the newline in this case. This improves
the rst output in particular, where blank lines may be problematic in
parameter lists.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 4f559de8b173..e93e796b17ce 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -2814,11 +2814,11 @@ sub process_file($) {
 		$in_doc_sect = 1;
 		$in_purpose = 0;
 		$contents = $newcontents;
+		while ((substr($contents, 0, 1) eq " ") ||
+		       substr($contents, 0, 1) eq "\t") {
+		    $contents = substr($contents, 1);
+		}
 		if ($contents ne "") {
-		    while ((substr($contents, 0, 1) eq " ") ||
-			substr($contents, 0, 1) eq "\t") {
-			    $contents = substr($contents, 1);
-		    }
 		    $contents .= "\n";
 		}
 		$section = $newsection;
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1413735 — [PATCH v2 35/38] doc/sphinx: Stop touching state_machine internals

From: Daniel Vetter <daniel.vetter@ffwll.ch>

Instead of just forcefully inserting our kernel-doc input and letting
the state machine stumble over it the recommended way is to create
ViewList, parse that and then return the list of parsed nodes.

Suggested by Jani.

Cc: Jani Nikula <jani.nikula@intel.com>
Cc: linux-doc@vger.kernel.org
Cc: Jonathan Corbet <corbet@lwn.net>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/sphinx/kernel-doc.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/Documentation/sphinx/kernel-doc.py b/Documentation/sphinx/kernel-doc.py
index 9fc2c27916a9..bd422870101e 100644
--- a/Documentation/sphinx/kernel-doc.py
+++ b/Documentation/sphinx/kernel-doc.py
@@ -32,6 +32,7 @@ import subprocess
 import sys
 
 from docutils import nodes, statemachine
+from docutils.statemachine import ViewList
 from docutils.parsers.rst import directives
 from sphinx.util.compat import Directive
 
@@ -92,8 +93,14 @@ class KernelDocDirective(Directive):
                 sys.stderr.write(err)
 
             lines = statemachine.string2lines(out, tab_width, convert_whitespace=True)
-            self.state_machine.insert_input(lines, source)
-            return []
+            result = ViewList(lines, source)
+
+            node = nodes.section()
+            node.document = self.state.document
+            self.state.nested_parse(result, self.content_offset, node)
+
+            return node.children
+
         except Exception as e:
             env.app.warn('kernel-doc \'%s\' processing failed with: %s' %
                          (" ".join(cmd), str(e)))
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1413736 — [PATCH v2 27/38] kernel-doc/rst: use *undescribed* instead of _undescribed_

The latter isn't special to rst.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index f6f37e71dc08..19cee0cd53a3 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -1839,7 +1839,7 @@ sub output_function_rst(%) {
 	    $args{'parameterdescs'}{$parameter_name} ne $undescribed) {
 	    output_highlight_rst($args{'parameterdescs'}{$parameter_name});
 	} else {
-	    print "  _undescribed_\n";
+	    print "  *undescribed*\n";
 	}
 	print "\n";
     }
@@ -1882,7 +1882,7 @@ sub output_enum_rst(%) {
 	if ($args{'parameterdescs'}{$parameter} ne $undescribed) {
 	    output_highlight_rst($args{'parameterdescs'}{$parameter});
 	} else {
-	    print "  _undescribed_\n";
+	    print "  *undescribed*\n";
 	}
 	print "\n";
     }
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1413737 — [PATCH v2 09/38] sphinx: update docbook->rst conversion script match C domain spec

Function references should include the parens (), struct references
should not include "struct".

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/sphinx/convert_template.sed | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/Documentation/sphinx/convert_template.sed b/Documentation/sphinx/convert_template.sed
index d53bb8220a26..0553eabeac79 100644
--- a/Documentation/sphinx/convert_template.sed
+++ b/Documentation/sphinx/convert_template.sed
@@ -4,10 +4,10 @@
 #
 # Use "$bq" instead of "`" so that pandoc won't mess with it.
 #
-s%<function>\([^<(]\+\)()</function>%:c:func:$bq\1$bq%g
-s%<function>\([^<(]\+\)</function>%:c:func:$bq\1$bq%g
-s%<structname>struct *\([^<]\+\)</structname>%:ref:$bqstruct \1$bq%g
-s%<structname>\([^<]\+\)</structname>%:ref:$bqstruct \1$bq%g
+s%<function>\([^<(]\+\)()</function>%:c:func:$bq\1()$bq%g
+s%<function>\([^<(]\+\)</function>%:c:func:$bq\1()$bq%g
+s%<structname>struct *\([^<]\+\)</structname>%:c:type:$bq\1$bq%g
+s%<structname>\([^<]\+\)</structname>%:c:type:$bq\1$bq%g
 #
 # Wrap docproc directives in para and code blocks.
 #
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1413744

On Sat, Jun 04, 2016 at 02:37:01PM +0300, Jani Nikula wrote:
> Jon, this is v2 of [1] and [2], with a considerable amount of polish and
> fixes added. We started dogfooding this within drm-intel, and Daniel has
> reviewed the lot and contributed a number of fixes, most notably
> accurate file and line number references from Sphinx build
> errors/warnings to the kernel-doc comments in source code.

dogfooding = our autobuilder also already eats it:

https://01.org/linuxgraphics/gfx-docs/drm/

> We believe this is now in good shape for merging for v4.8. It's all in
> my sphinx-for-docs-next branch that you've already looked at; pull
> details below.

I've read through all the patches and things look good. Now my
python/perl/sphinx fu is by far not good enough for a full review, hence
just:

Acked-by: Daniel Vetter <daniel.vetter@ffwll.ch>

It also works really nicely, we already started to land some kernel-doc
markup fixup patches for drm.

Tested-by: Daniel Vetter <daniel.vetter@ffwll.ch>

I really think this is awesome, and it's a good foundation for more polish
in the toolchain on top (like Markus' improved tables, or some of the
ideas floating around about better linting for kernel-doc). But right now
I think the next step is to convert a pile of docs and gain some
real-world experience with the tool-chain.

> When this lands in docs-next and we can backmerge to drm, we'll plunge
> ahead and convert gpu.tmpl to rst, and have that ready for v4.8. We
> think it's best to contribute that via the drm tree, as it'll involve
> splitting up the documentation and likely numerous updates to kernel-doc
> comments.

Please send me a pull request with the tag to merge into drm-misc.

> I plan to update Documentation/kernel-doc-nano-HOWTO.txt for Sphinx and
> rst, obviously converting it to rst while at it.

Cheers, Daniel

> 
> BR,
> Jani.
> 
> 
> 
> [1] http://mid.gmane.org/cover.1463748027.git.jani.nikula@intel.com
> [2] http://mid.gmane.org/877fec7gfm.fsf@intel.com
> 
> The following changes since commit 1a695a905c18548062509178b98bc91e67510864:
> 
>   Linux 4.7-rc1 (2016-05-29 09:29:24 -0700)
> 
> are available in the git repository at:
> 
>   git://people.freedesktop.org/~jani/drm sphinx-for-docs-next
> 
> for you to fetch changes up to d90368f2fa7ded7c56d214aef087e88bba5199e7:
> 
>   doc/sphinx: Track line-number of starting blocks (2016-06-04 11:35:59 +0300)
> 
> ----------------------------------------------------------------
> Daniel Vetter (6):
>       doc/sphinx: Pass right filename as source
>       scripts/kernel-doc: Remove duplicated DOC: start handling
>       doc/sphinx: Stop touching state_machine internals
>       scripts/kernel-doc: Also give functions symbolic names
>       scripts/kernel-doc: Add option to inject line numbers
>       doc/sphinx: Track line-number of starting blocks
> 
> Jani Nikula (31):
>       kernel-doc/rst: fix use of uninitialized value
>       kernel-doc: support printing exported and non-exported symbols
>       Documentation/sphinx: add basic working Sphinx configuration and build
>       Documentation: add .gitignore
>       Documentation/sphinx: add Sphinx kernel-doc directive extension
>       Documentation/sphinx: configure the kernel-doc extension
>       Documentation/sphinx: set version and release properly
>       sphinx: update docbook->rst conversion script match C domain spec
>       Documentation/sphinx: nicer referencing of struct in docbook->rst conversion
>       kernel-doc: add names for states and substates
>       kernel-doc: add names for output selection
>       kernel-doc/rst: do not output DOC: section titles for requested ones
>       kernel-doc/rst: reference functions according to C domain spec
>       kernel-doc/rst: &foo references are more universal than structs
>       kernel-doc/rst: add support for &union foo and &typedef foo references
>       kernel-doc/rst: add support for struct/union/enum member references
>       kernel-doc/rst: drop redundant unescape in highlighting
>       kernel-doc/rst: highlight function/struct/enum purpose lines too
>       kernel-doc: do not regard $, %, or & prefixes as special in section names
>       kernel-doc: fix wrong code indentation
>       kernel-doc/rst: blank lines in output are not needed
>       kernel-doc: strip leading blank lines from inline doc comments
>       kernel-doc/rst: change the output layout
>       kernel-doc: improve handling of whitespace on the first line param description
>       kernel-doc: strip leading whitespace from continued param descs
>       kernel-doc/rst: use *undescribed* instead of _undescribed_
>       kernel-doc/rst: remove fixme comment
>       kernel-doc: limit the "section header:" detection to a select few
>       kernel-doc: concatenate contents of colliding sections
>       kernel-doc: reset contents and section harder
>       Documentation/sphinx: fix kernel-doc extension on python3
> 
> Jonathan Corbet (1):
>       sphinx: cheesy script to convert .tmpl files
> 
>  Documentation/.gitignore                  |   1 +
>  Documentation/DocBook/Makefile            |   7 +-
>  Documentation/Makefile.sphinx             |  63 +++++
>  Documentation/conf.py                     | 414 ++++++++++++++++++++++++++++++
>  Documentation/index.rst                   |  23 ++
>  Documentation/sphinx/convert_template.sed |  18 ++
>  Documentation/sphinx/kernel-doc.py        | 127 +++++++++
>  Documentation/sphinx/post_convert.sed     |  23 ++
>  Documentation/sphinx/tmplcvt              |  19 ++
>  Makefile                                  |   5 +-
>  scripts/kernel-doc                        | 392 ++++++++++++++++++----------
>  11 files changed, 948 insertions(+), 144 deletions(-)
>  create mode 100644 Documentation/.gitignore
>  create mode 100644 Documentation/Makefile.sphinx
>  create mode 100644 Documentation/conf.py
>  create mode 100644 Documentation/index.rst
>  create mode 100644 Documentation/sphinx/convert_template.sed
>  create mode 100644 Documentation/sphinx/kernel-doc.py
>  create mode 100644 Documentation/sphinx/post_convert.sed
>  create mode 100755 Documentation/sphinx/tmplcvt
> 
> -- 
> 2.1.4
> 

-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

[toc] | [prev] | [next] | [standalone]

Page 2 of 3 — ← Prev page 1 [2] 3  Next page →

Back to top | Article view | linux.kernel

csiph-web