Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1317400 > unrolled thread

[RFC] A first shot at asciidoc-based formatted docs

Back to article view | Back to linux.kernel

Contents

  [RFC] A first shot at asciidoc-based formatted docs Jonathan Corbet <corbet@lwn.net> - 2016-01-26 00:30 +0100
    [PATCH 1/4] kernel-doc: add support for asciidoc output Jonathan Corbet <corbet@lwn.net> - 2016-01-26 00:30 +0100
    [PATCH 2/4] docproc: handle asciidoc templates Jonathan Corbet <corbet@lwn.net> - 2016-01-26 00:30 +0100
    [PATCH 3/4] Docs: Makefile tweaks for asciidoc templates Jonathan Corbet <corbet@lwn.net> - 2016-01-26 00:30 +0100
    [PATCH 4/4] Docs: add a sample asciidoc template Jonathan Corbet <corbet@lwn.net> - 2016-01-26 00:30 +0100
    [RFC 10/10] Documentation: build asciidoc documentation Jani Nikula <jani.nikula@intel.com> - 2016-01-26 13:10 +0100
    Re: [RFC] A first shot at asciidoc-based formatted docs Jani Nikula <jani.nikula@intel.com> - 2016-01-26 13:10 +0100
      [RFC 09/10] Documentation: convert gpu.tmpl to gpu.txt Jani Nikula <jani.nikula@intel.com> - 2016-01-26 13:10 +0100
      [RFC 05/10] scripts: add asciidoc-includes to extract includes from asciidoc Jani Nikula <jani.nikula@intel.com> - 2016-01-26 13:10 +0100
      [RFC 06/10] scripts: add a kernel-doc helper for special invocation Jani Nikula <jani.nikula@intel.com> - 2016-01-26 13:10 +0100
      [RFC 04/10] kernel-doc: add support for printing DOC: comments with escaped names Jani Nikula <jani.nikula@intel.com> - 2016-01-26 13:20 +0100
      [RFC 01/10] kernel-doc: rewrite usage description, remove duplicated comments Jani Nikula <jani.nikula@intel.com> - 2016-01-26 13:20 +0100
      Re: [RFC] A first shot at asciidoc-based formatted docs Daniel Vetter <daniel.vetter@ffwll.ch> - 2016-01-26 13:20 +0100
        Re: [RFC] A first shot at asciidoc-based formatted docs Jani Nikula <jani.nikula@intel.com> - 2016-01-26 13:40 +0100
      [RFC 03/10] kernel-doc: support printing exported and non-exported symbols Jani Nikula <jani.nikula@intel.com> - 2016-01-26 13:20 +0100
      [RFC 07/10] scripts: add tool for generating asciidoc dependencies and rules Jani Nikula <jani.nikula@intel.com> - 2016-01-26 13:20 +0100
      [RFC 08/10] scripts: add a crude converter from DocBook tmpl to asciidoc Jani Nikula <jani.nikula@intel.com> - 2016-01-26 13:20 +0100
      [RFC 02/10] kernel-doc: add support for asciidoc output Jani Nikula <jani.nikula@intel.com> - 2016-01-26 13:20 +0100
      Re: [RFC] A first shot at asciidoc-based formatted docs Jonathan Corbet <corbet@lwn.net> - 2016-01-26 15:50 +0100
        Re: [RFC] A first shot at asciidoc-based formatted docs Jonathan Corbet <corbet@lwn.net> - 2016-01-26 16:00 +0100

#1317400 — [RFC] A first shot at asciidoc-based formatted docs

So here is a proof-of-concept series showing how a fully asciidoc-based
toolchain might work.  Lots of hackery here, this isn't meant to be applied
to anything at this point, but it's a good start.  What this series has is:

 - Jani Nikula's patch adding asciidoc output to kernel-doc.  Thanks for
   doing this!  It was the kickstart that was needed to get this process
   going.

 - Tweak docproc to handle asciidoc template files.  If a template ends in
   ".adt", it's an asciidoc template; it's processed pretty much the same
   way, except that kernel-doc gets the -asciidoc argument.

 - Bash on the Makefile to get it to process asciidoc templates into HTML.
   Naturally this was where most of the time got spent.  *Only* HTML output
   works at the moment.

 - Convert tracepoints.html to tpoint.adt as a proof of concept.  It works,
   and the output is much pleasing, IMO.

I'm sure there's a thousand details to deal with, and there is the issue of
the other output formats.  asciidoctor claims to be able to create man
pages, but I've not tried that yet; neither tool will do PDF.  Maybe we
could rely on pandoc to do that.  Otherwise, getting to asciidoc to XML is
straightforward, so it should be possible to use xmlto as is done now.

It's all in the doc/asciidoc branch of git://git.lwn.net/linux.git if
anybody wants to mess with it.

Comments?

jon

[toc] | [next] | [standalone]

#1317403 — [PATCH 1/4] kernel-doc: add support for asciidoc output

From: Jani Nikula <jani.nikula@intel.com>

Add new -asciidoc option to produce asciidoc output from kernel-doc. The
output is formatted internally, with no dependencies on external
tools. Any asciidoc formatting present in kernel-doc will naturally be
present in the resulting asciidoc as well.

Highlighting of functions(), &struct structures, &enum enumerations,
@parameters, etc. will be done by means of asciidoc. Anchors and
cross-references are added as well, providing hyperlinking support in
the result processed by asciidoc(1).

This support is non-conflicting and orthogonal to the patches adding
asciidoc support to the kernel-doc DocBook output [1]. Those patches
bolt on to the current document generation pipeline; there is currently
none for native asciidoc in the kernel (though ideas have been discussed
[2]). At this time, this patch should be considered a worthwhile
standalone improvement to kernel-doc, independent of the rest.

[1] http://mid.gmane.org/1448471279-19748-1-git-send-email-daniel.vetter@ffwll.ch
[2] http://mid.gmane.org/20160114131823.2ff43a0c@lwn.net

Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Daniel Vetter <daniel@ffwll.ch>
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/kernel-doc | 237 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 237 insertions(+)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index c37255b..06f1649 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -201,6 +201,8 @@ my $type_param = '\@(\w+)';
 my $type_struct = '\&((struct\s*)*[_\w]+)';
 my $type_struct_xml = '\\&amp;((struct\s*)*[_\w]+)';
 my $type_env = '(\$\w+)';
+my $type_enum_full = '\&(enum)\s*([_\w]+)';
+my $type_struct_full = '\&(struct)\s*([_\w]+)';
 
 # Output conversion substitutions.
 #  One for each output format
@@ -266,6 +268,17 @@ my @highlights_text = (
 		      );
 my $blankline_text = "";
 
+# asciidoc-mode
+my @highlights_asciidoc = (
+                       [$type_constant, "`\$1`"],
+                       [$type_func, "<<func_\$1,`\$1()`>>"],
+                       [$type_struct_full, "<<\$1_\$2,`\$1 \$2`>>"],
+                       [$type_enum_full, "<<\$1_\$2,`\$1 \$2`>>"],
+                       [$type_struct, "<<struct_\$1,`\$1`>>"],
+                       [$type_param, "*\$1*"]
+		      );
+my $blankline_asciidoc = "\n";
+
 # list mode
 my @highlights_list = (
                        [$type_constant, "\$1"],
@@ -402,6 +415,10 @@ while ($ARGV[0] =~ m/^-(.*)/) {
 	$output_mode = "text";
 	@highlights = @highlights_text;
 	$blankline = $blankline_text;
+    } elsif ($cmd eq "-asciidoc") {
+	$output_mode = "asciidoc";
+	@highlights = @highlights_asciidoc;
+	$blankline = $blankline_asciidoc;
     } elsif ($cmd eq "-docbook") {
 	$output_mode = "xml";
 	@highlights = @highlights_xml;
@@ -1713,6 +1730,214 @@ sub output_blockhead_text(%) {
     }
 }
 
+##
+# output in asciidoc
+sub output_highlight_asciidoc {
+    my $contents = join "\n",@_;
+    my $line;
+
+    # undo the evil effects of xml_escape() earlier
+    $contents = xml_unescape($contents);
+
+    eval $dohighlight;
+    die $@ if $@;
+
+    foreach $line (split "\n", $contents) {
+	if ($line eq "") {
+	    print $lineprefix, $blankline;
+	} else {
+	    $line =~ s/\\\\\\/\&/g;
+	    print $lineprefix, $line;
+	}
+	print "\n";
+    }
+}
+
+sub output_function_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter, $section);
+    my $start;
+
+    print "[[func_$args{'function'}]]\n";
+    print "=== " . $args{'function'} . " ===\n";
+    print $args{'purpose'} . "\n\n";
+
+    print "----------\n";
+    if ($args{'functiontype'} ne "") {
+	$start = $args{'functiontype'} . " " . $args{'function'} . " (";
+    } else {
+	$start = $args{'function'} . " (";
+    }
+    print $start;
+
+    my $count = 0;
+    foreach my $parameter (@{$args{'parameterlist'}}) {
+	$type = $args{'parametertypes'}{$parameter};
+	if ($type =~ m/([^\(]*\(\*)\s*\)\s*\(([^\)]*)\)/) {
+	    # pointer-to-function
+	    print $1 . $parameter . ") (" . $2;
+	} else {
+	    print $type . " " . $parameter;
+	}
+	if ($count != $#{$args{'parameterlist'}}) {
+	    $count++;
+	    print ",\n";
+	    print " " x length($start);
+	} else {
+	    print ");\n\n";
+	}
+    }
+    print "----------\n";
+
+    print ".Parameters\n";
+    print "[horizontal]\n";
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	my $parameter_name = $parameter;
+	$parameter_name =~ s/\[.*//;
+	$type = $args{'parametertypes'}{$parameter};
+
+	print "`$type $parameter`::\n";
+	if ($args{'parameterdescs'}{$parameter_name} ne $undescribed) {
+	    $blankline = "+";
+	    output_highlight_asciidoc($args{'parameterdescs'}{$parameter_name});
+	    $blankline = "\n";
+	} else {
+	    print "_undescribed_\n";
+	}
+	print "\n";
+    }
+    output_section_asciidoc(@_);
+}
+
+sub output_section_asciidoc(%) {
+    my %args = %{$_[0]};
+    my $section;
+
+    foreach $section (@{$args{'sectionlist'}}) {
+	print ".$section\n\n";
+	output_highlight_asciidoc($args{'sections'}{$section});
+	print "\n";
+    }
+    print "\n";
+}
+
+sub output_enum_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter);
+    my $count;
+
+    print "[[enum_$args{'enum'}]]\n";
+    print "=== enum " . $args{'enum'} . " ===\n";
+    print $args{'purpose'} . "\n\n";
+
+    print "----------\n";
+    print "enum " . $args{'enum'} . " {\n";
+    $count = 0;
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	print "\t$parameter";
+	if ($count != $#{$args{'parameterlist'}}) {
+	    $count++;
+	    print ",";
+	}
+	print "\n";
+    }
+    print "};\n";
+    print "----------\n";
+
+    print ".Constants\n";
+    print "[horizontal]\n";
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	print "`$parameter`::\n";
+	if ($args{'parameterdescs'}{$parameter} ne $undescribed) {
+	    $blankline = "+";
+	    output_highlight_asciidoc($args{'parameterdescs'}{$parameter});
+	    $blankline = "\n";
+	} else {
+	    print "_undescribed_\n";
+	}
+	print "\n";
+    }
+
+    output_section_asciidoc(@_);
+}
+
+sub output_typedef_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter);
+    my $count;
+
+    print "[[$args{'typedef'}]]\n";
+    print "=== typedef " . $args{'typedef'} . " ===\n";
+    print $args{'purpose'} . "\n\n";
+
+    output_section_asciidoc(@_);
+}
+
+sub output_struct_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter);
+
+    print "[[$args{'type'}_$args{'struct'}]]\n";
+    print "=== " . $args{'type'} . " " . $args{'struct'} . " ===\n";
+    print $args{'purpose'} . "\n\n";
+
+    print "----------\n";
+    print $args{'type'} . " " . $args{'struct'} . " {\n";
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	if ($parameter =~ /^#/) {
+	    print "$parameter\n";
+	    next;
+	}
+
+	my $parameter_name = $parameter;
+	$parameter_name =~ s/\[.*//;
+
+	($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
+	$type = $args{'parametertypes'}{$parameter};
+	if ($type =~ m/([^\(]*\(\*)\s*\)\s*\(([^\)]*)\)/) {
+	    # pointer-to-function
+	    print "\t$1 $parameter) ($2);\n";
+	} elsif ($type =~ m/^(.*?)\s*(:.*)/) {
+	    # bitfield
+	    print "\t$1 $parameter$2;\n";
+	} else {
+	    print "\t" . $type . " " . $parameter . ";\n";
+	}
+    }
+    print "};\n";
+    print "----------\n";
+
+    print ".Members\n";
+    print "[horizontal]\n";
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	($parameter =~ /^#/) && next;
+
+	my $parameter_name = $parameter;
+	$parameter_name =~ s/\[.*//;
+
+	($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
+	$type = $args{'parametertypes'}{$parameter};
+	print "`$type $parameter`" . "::\n";
+	$blankline = "+";
+	output_highlight_asciidoc($args{'parameterdescs'}{$parameter_name});
+	$blankline = "\n";
+	print "\n";
+    }
+    print "\n";
+    output_section_asciidoc(@_);
+}
+
+sub output_blockhead_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter, $section);
+
+    foreach $section (@{$args{'sectionlist'}}) {
+	print "=== $section ===\n";
+	output_highlight_asciidoc($args{'sections'}{$section});
+	print "\n";
+    }
+}
+
 ## list mode output functions
 
 sub output_function_list(%) {
@@ -2414,6 +2639,18 @@ sub xml_escape($) {
 	return $text;
 }
 
+# xml_unescape: reverse the effects of xml_escape
+sub xml_unescape($) {
+	my $text = shift;
+	if (($output_mode eq "text") || ($output_mode eq "man")) {
+		return $text;
+	}
+	$text =~ s/\\\\\\amp;/\&/g;
+	$text =~ s/\\\\\\lt;/</g;
+	$text =~ s/\\\\\\gt;/>/g;
+	return $text;
+}
+
 # convert local escape strings to html
 # local escape strings look like:  '\\\\menmonic:' (that's 4 backslashes)
 sub local_unescape($) {
-- 
2.7.0

[toc] | [prev] | [next] | [standalone]

#1317404 — [PATCH 2/4] docproc: handle asciidoc templates

There's really nothing different that needs to be done except for invoking
kernel-doc with the -asciidoc argument.  Look at the input file name to
recognize asciidoc templates, so no special command-line flags are needed.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 scripts/docproc.c | 25 ++++++++++++++++++++-----
 1 file changed, 20 insertions(+), 5 deletions(-)

diff --git a/scripts/docproc.c b/scripts/docproc.c
index e267e621..89284b5 100644
--- a/scripts/docproc.c
+++ b/scripts/docproc.c
@@ -68,12 +68,15 @@ FILELINE * docsection;
 #define KERNELDOCPATH "scripts/"
 #define KERNELDOC     "kernel-doc"
 #define DOCBOOK       "-docbook"
+#define ASCIIDOC      "-asciidoc"
 #define LIST          "-list"
 #define FUNCTION      "-function"
 #define NOFUNCTION    "-nofunction"
 #define NODOCSECTIONS "-no-doc-sections"
 #define SHOWNOTFOUND  "-show-not-found"
 
+static char *doc_format = DOCBOOK;
+
 static char *srctree, *kernsrctree;
 
 static char **all_list = NULL;
@@ -242,7 +245,7 @@ static void find_export_symbols(char * filename)
 /*
  * Document all external or internal functions in a file.
  * Call kernel-doc with following parameters:
- * kernel-doc -docbook -nofunction function_name1 filename
+ * kernel-doc [-docbook|-asciidoc] -nofunction function_name1 filename
  * Function names are obtained from all the src files
  * by find_export_symbols.
  * intfunc uses -nofunction
@@ -263,7 +266,7 @@ static void docfunctions(char * filename, char * type)
 		exit(1);
 	}
 	vec[idx++] = KERNELDOC;
-	vec[idx++] = DOCBOOK;
+	vec[idx++] = doc_format;
 	vec[idx++] = NODOCSECTIONS;
 	for (i=0; i < symfilecnt; i++) {
 		struct symfile * sym = &symfilelist[i];
@@ -275,7 +278,7 @@ static void docfunctions(char * filename, char * type)
 	}
 	vec[idx++]     = filename;
 	vec[idx] = NULL;
-	printf("<!-- %s -->\n", filename);
+	/* printf("<!-- %s -->\n", filename); */
 	exec_kernel_doc(vec);
 	fflush(stdout);
 	free(vec);
@@ -294,7 +297,7 @@ static void singfunc(char * filename, char * line)
 	int i, idx = 0;
 	int startofsym = 1;
 	vec[idx++] = KERNELDOC;
-	vec[idx++] = DOCBOOK;
+	vec[idx++] = doc_format;
 	vec[idx++] = SHOWNOTFOUND;
 
 	/* Split line up in individual parameters preceded by FUNCTION */
@@ -343,7 +346,7 @@ static void docsect(char *filename, char *line)
 	free(s);
 
 	vec[0] = KERNELDOC;
-	vec[1] = DOCBOOK;
+	vec[1] = doc_format;
 	vec[2] = SHOWNOTFOUND;
 	vec[3] = FUNCTION;
 	vec[4] = line;
@@ -497,6 +500,16 @@ static void parse_file(FILE *infile)
 	fflush(stdout);
 }
 
+/*
+ * Is this an asciidoc template?  Answer the question by seeing if its
+ * name ends in ".adt".
+ */
+static int is_asciidoc(const char *file)
+{
+	int len = strlen(file);
+
+	return len > 4 && ! strcmp(file + len - 4, ".adt");
+}
 
 int main(int argc, char *argv[])
 {
@@ -520,6 +533,8 @@ int main(int argc, char *argv[])
 		perror(argv[2]);
 		exit(2);
 	}
+	if (is_asciidoc(argv[2]))
+		doc_format = ASCIIDOC;
 
 	if (strcmp("doc", argv[1]) == 0) {
 		/* Need to do this in two passes.
-- 
2.7.0

[toc] | [prev] | [next] | [standalone]

#1317405 — [PATCH 3/4] Docs: Makefile tweaks for asciidoc templates

This is a hatchet job, but it's something to start with.  Generalize some
of the string manipulation to not assume that templates have a ".tmpl"
suffix, and add rules to translate asciidoc templates to HTML.  Nothing for
any other output formats at this point.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 Documentation/DocBook/Makefile | 72 +++++++++++++++++++++++++-----------------
 1 file changed, 43 insertions(+), 29 deletions(-)

diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile
index d70f9b6..f04e8c8 100644
--- a/Documentation/DocBook/Makefile
+++ b/Documentation/DocBook/Makefile
@@ -6,16 +6,16 @@
 # To add a new book the only step required is to add the book to the
 # list of DOCBOOKS.
 
-DOCBOOKS := z8530book.xml device-drivers.xml \
-	    kernel-hacking.xml kernel-locking.xml deviceiobook.xml \
-	    writing_usb_driver.xml networking.xml \
-	    kernel-api.xml filesystems.xml lsm.xml usb.xml kgdb.xml \
-	    gadget.xml libata.xml mtdnand.xml librs.xml rapidio.xml \
-	    genericirq.xml s390-drivers.xml uio-howto.xml scsi.xml \
-	    80211.xml debugobjects.xml sh.xml regulator.xml \
-	    alsa-driver-api.xml writing-an-alsa-driver.xml \
-	    tracepoint.xml gpu.xml media_api.xml w1.xml \
-	    writing_musb_glue_layer.xml crypto-API.xml iio.xml
+DOCBOOKS := z8530book device-drivers \
+	    kernel-hacking kernel-locking deviceiobook \
+	    writing_usb_driver networking \
+	    kernel-api filesystems lsm usb kgdb \
+	    gadget libata mtdnand librs rapidio \
+	    genericirq s390-drivers uio-howto scsi \
+	    80211 debugobjects sh regulator \
+	    alsa-driver-api writing-an-alsa-driver \
+	    tracepoint gpu media_api w1 \
+	    writing_musb_glue_layer crypto-API iio
 
 include Documentation/DocBook/media/Makefile
 
@@ -39,21 +39,21 @@ PHONY += xmldocs sgmldocs psdocs pdfdocs htmldocs mandocs installmandocs cleando
 
 targets += $(DOCBOOKS)
 BOOKS := $(addprefix $(obj)/,$(DOCBOOKS))
-xmldocs: $(BOOKS)
+xmldocs: $(addsuffix .xml, $(BOOKS))
 sgmldocs: xmldocs
 
-PS := $(patsubst %.xml, %.ps, $(BOOKS))
+PS := $(addsuffix .ps, $(BOOKS))
 psdocs: $(PS)
 
-PDF := $(patsubst %.xml, %.pdf, $(BOOKS))
+PDF := $(addsuffix .pdf, $(BOOKS))
 pdfdocs: $(PDF)
 
-HTML := $(sort $(patsubst %.xml, %.html, $(BOOKS)))
+HTML := $(sort $(addsuffix .html, $(BOOKS)))
 htmldocs: $(HTML)
 	$(call cmd,build_main_index)
 	$(call install_media_images)
 
-MAN := $(patsubst %.xml, %.9, $(BOOKS))
+MAN := $(addsuffix .9, $(BOOKS))
 mandocs: $(MAN)
 	find $(obj)/man -name '*.9' | xargs gzip -nf
 
@@ -103,6 +103,19 @@ endef
 # Tell kbuild to always build the programs
 always := $(hostprogs-y)
 
+#
+# asciidoc stuff.
+#
+quiet_cmd_ad2html = ASCIIDOC	$@
+      cmd_ad2html = asciidoc -b html $< > $@
+
+%.ad: %.adt $(KERNELDOC) $(DOCPROC) FORCE
+	$(call cmd,docproc)
+
+%.html: %.ad
+	$(call cmd,ad2html)
+
+
 notfoundtemplate = echo "*** You have to install docbook-utils or xmlto ***"; \
 		   exit 1
 db2xtemplate = db2TYPE -o $(dir $@) $<
@@ -234,22 +247,23 @@ dochelp:
 
 ###
 # Temporary files left by various tools
-clean-files := $(DOCBOOKS) \
-	$(patsubst %.xml, %.dvi,     $(DOCBOOKS)) \
-	$(patsubst %.xml, %.aux,     $(DOCBOOKS)) \
-	$(patsubst %.xml, %.tex,     $(DOCBOOKS)) \
-	$(patsubst %.xml, %.log,     $(DOCBOOKS)) \
-	$(patsubst %.xml, %.out,     $(DOCBOOKS)) \
-	$(patsubst %.xml, %.ps,      $(DOCBOOKS)) \
-	$(patsubst %.xml, %.pdf,     $(DOCBOOKS)) \
-	$(patsubst %.xml, %.html,    $(DOCBOOKS)) \
-	$(patsubst %.xml, %.9,       $(DOCBOOKS)) \
-	$(patsubst %.xml, %.aux.xml, $(DOCBOOKS)) \
-	$(patsubst %.xml, %.xml.db,  $(DOCBOOKS)) \
-	$(patsubst %.xml, %.xml,     $(DOCBOOKS)) \
+clean-files := \
+	$(addsuffix .dvi,     $(DOCBOOKS)) \
+	$(addsuffix .aux,     $(DOCBOOKS)) \
+	$(addsuffix .tex,     $(DOCBOOKS)) \
+	$(addsuffix .log,     $(DOCBOOKS)) \
+	$(addsuffix .out,     $(DOCBOOKS)) \
+	$(addsuffix .ps,      $(DOCBOOKS)) \
+	$(addsuffix .pdf,     $(DOCBOOKS)) \
+	$(addsuffix .html,    $(DOCBOOKS)) \
+	$(addsuffix .9,       $(DOCBOOKS)) \
+	$(addsuffix .aux.xml, $(DOCBOOKS)) \
+	$(addsuffix .xml.db,  $(DOCBOOKS)) \
+	$(addsuffix .xml,     $(DOCBOOKS)) \
+	$(addsuffix .ad,      $(DOCBOOKS)) \
 	$(index)
 
-clean-dirs := $(patsubst %.xml,%,$(DOCBOOKS)) man
+clean-dirs := $(DOCBOOKS) man
 
 cleandocs: cleanmediadocs
 	$(Q)rm -f $(call objectify, $(clean-files))
-- 
2.7.0

[toc] | [prev] | [next] | [standalone]

#1317407 — [PATCH 4/4] Docs: add a sample asciidoc template

This is just a copy of tracepoints.tmpl (because it was short!) converted
into asciidoc.

Signed-off-by: Jonathan Corbet <corbet@lwn.net>
---
 Documentation/DocBook/Makefile   |  2 +-
 Documentation/DocBook/tpoint.adt | 64 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 65 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/DocBook/tpoint.adt

diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile
index f04e8c8..2e5195e 100644
--- a/Documentation/DocBook/Makefile
+++ b/Documentation/DocBook/Makefile
@@ -15,7 +15,7 @@ DOCBOOKS := z8530book device-drivers \
 	    80211 debugobjects sh regulator \
 	    alsa-driver-api writing-an-alsa-driver \
 	    tracepoint gpu media_api w1 \
-	    writing_musb_glue_layer crypto-API iio
+	    writing_musb_glue_layer crypto-API iio tpoint
 
 include Documentation/DocBook/media/Makefile
 
diff --git a/Documentation/DocBook/tpoint.adt b/Documentation/DocBook/tpoint.adt
new file mode 100644
index 0000000..f93863a
--- /dev/null
+++ b/Documentation/DocBook/tpoint.adt
@@ -0,0 +1,64 @@
+The Linux Kernel Tracepoint API
+===============================
+Jason Baron, William Cohen
+
+.This document
+****************************************************************
+This documentation is free software; you can redistribute
+it and/or modify it under the terms of the GNU General Public
+License as published by the Free Software Foundation; either
+version 2 of the License, or (at your option) any later
+version.
+
+This program is distributed in the hope that it will be
+useful, but WITHOUT ANY WARRANTY; without even the implied
+warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+See the GNU General Public License for more details.
+
+You should have received a copy of the GNU General Public
+License along with this program; if not, write to the Free
+Software Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+MA 02111-1307 USA
+
+For more details see the file COPYING in the source
+distribution of Linux.
+****************************************************************
+
+Introduction
+------------
+
+Tracepoints are static probe points that are located in strategic points
+throughout the kernel. 'Probes' register/unregister with tracepoints
+via a callback mechanism. The 'probes' are strictly typed functions that
+are passed a unique set of parameters defined by each tracepoint.
+
+From this simple callback mechanism, 'probes' can be used to profile, debug,
+and understand kernel behavior. There are a number of tools that provide a
+framework for using 'probes'. These tools include Systemtap, ftrace, and
+LTTng.
+
+Tracepoints are defined in a number of header files via various macros. Thus,
+the purpose of this document is to provide a clear accounting of the available
+tracepoints. The intention is to understand not only what tracepoints are
+available but also to understand where future tracepoints might be added.
+
+The API presented has functions of the form:
++trace_tracepointname(function parameters)+. These are the
+tracepoints callbacks that are found throughout the code. Registering and
+unregistering probes with these callback sites is covered in the
+'Documentation/trace/*' directory.
+
+IRQ
+---
+!Iinclude/trace/events/irq.h
+
+SIGNAL
+------
+!Iinclude/trace/events/signal.h
+
+Block IO
+!Iinclude/trace/events/block.h
+
+Workqueue
+---------
+!Iinclude/trace/events/workqueue.h
-- 
2.7.0

[toc] | [prev] | [next] | [standalone]

#1317791 — [RFC 10/10] Documentation: build asciidoc documentation

Finally wrap up all the work in the preceding patches, and generate a
pipeline from asciidoc documentation as part of the DocBook
documentation.

The biggest difference in building DocBook xml from asciidoc compared to
the docproc pipeline is the use of intermediate asciidoc snippets for
the kernel-doc generated documentation, their inclusion into the source
asciidoc using the markup's own directives, with the dependencies set
right.

This is a big improvement in the build process:

First, kernel-doc is only called on the source files that have actually
changed. (Also, asciidoc is only called on the high level documents
where the included documentation has actually changed.)

Second, all of the kernel-doc processing parallelizes.

The xmlto proessing of the result is still slow, but the build time
improvement for the kernel-doc/asciidoc part is huge.

It's not yet all rosy though:

- The intermediate files should be put in a separate subdirectory
  instead of DocBook where it pollutes everything. Sorry, this is rather
  ugly now.

- The clean targets don't work properly.

- Some of the choices made in kernel-doc asciidoc support looked
  beautiful with the default asciidoc html output; however the DocBook
  pipeline is a different beast with different templates, and it's not
  so pretty anymore. (Particularly the numbered "tables" for parameters
  are ugly.)

- The cross-referencing probably needs more looking into. I think I've
  seen some weird stuff there. Also, since the kernel-doc asciidoc turns
  everything into xrefs, the later cross-referencing stage complains
  about dead links.

NOTE: When switching between this and master, be sure to clean the
Documentation/DocBook/*.cmd files. They don't seem to depend right, and
wasted me a few hours debugging my makefiles.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 Documentation/DocBook/Makefile | 37 +++++++++++++++++++++++++++++--------
 Documentation/DocBook/gpu.txt  |  2 +-
 2 files changed, 30 insertions(+), 9 deletions(-)

diff --git a/Documentation/DocBook/Makefile b/Documentation/DocBook/Makefile
index d70f9b68174e..aed990080aba 100644
--- a/Documentation/DocBook/Makefile
+++ b/Documentation/DocBook/Makefile
@@ -4,7 +4,9 @@
 # See Documentation/kernel-doc-nano-HOWTO.txt for instruction in how
 # to document the SRC - and how to read it.
 # To add a new book the only step required is to add the book to the
-# list of DOCBOOKS.
+# list of DOCBOOKS or ASCIIDOC_BOOKS.
+
+ASCIIDOC_BOOKS := gpu.txt
 
 DOCBOOKS := z8530book.xml device-drivers.xml \
 	    kernel-hacking.xml kernel-locking.xml deviceiobook.xml \
@@ -14,18 +16,19 @@ DOCBOOKS := z8530book.xml device-drivers.xml \
 	    genericirq.xml s390-drivers.xml uio-howto.xml scsi.xml \
 	    80211.xml debugobjects.xml sh.xml regulator.xml \
 	    alsa-driver-api.xml writing-an-alsa-driver.xml \
-	    tracepoint.xml gpu.xml media_api.xml w1.xml \
-	    writing_musb_glue_layer.xml crypto-API.xml iio.xml
+	    tracepoint.xml media_api.xml w1.xml \
+	    writing_musb_glue_layer.xml crypto-API.xml iio.xml \
+	    $(ASCIIDOC_BOOKS:%.txt=%.xml)
 
 include Documentation/DocBook/media/Makefile
 
 ###
 # The build process is as follows (targets):
-#              (xmldocs) [by docproc]
-# file.tmpl --> file.xml +--> file.ps   (psdocs)   [by db2ps or xmlto]
-#                        +--> file.pdf  (pdfdocs)  [by db2pdf or xmlto]
-#                        +--> DIR=file  (htmldocs) [by xmlto]
-#                        +--> man/      (mandocs)  [by xmlto]
+#                (xmldocs) [by docproc or asciidoc]
+# file.tmpl -+               +--> file.ps   (psdocs)   [by db2ps or xmlto]
+#            +--> file.xml --+--> file.pdf  (pdfdocs)  [by db2pdf or xmlto]
+# file.txt  -+               +--> DIR=file  (htmldocs) [by xmlto]
+#                            +--> man/      (mandocs)  [by xmlto]
 
 
 # for PDF and PS output you can choose between xmlto and docbook-utils tools
@@ -69,6 +72,9 @@ KERNELDOCXMLREF = $(srctree)/scripts/kernel-doc-xml-ref
 KERNELDOC       = $(srctree)/scripts/kernel-doc
 DOCPROC         = $(objtree)/scripts/docproc
 CHECK_LC_CTYPE = $(objtree)/scripts/check-lc_ctype
+ASCIIDOC_INCLUDES	= $(srctree)/scripts/asciidoc-includes
+KERNELDOC_DEPS		= $(srctree)/scripts/kernel-doc-deps
+KERNELDOC_HELPER	= $(srctree)/scripts/kernel-doc-helper
 
 # Use a fixed encoding - UTF-8 if the C library has support built-in
 # or ASCII if not
@@ -79,6 +85,21 @@ XMLTOFLAGS = -m $(srctree)/$(src)/stylesheet.xsl
 XMLTOFLAGS += --skip-validation
 
 ###
+# Docbook pipeline for asciidoc and asciidoctor (both should work).
+%.xml: %.txt
+	asciidoctor -b docbook -o $@ $<
+
+# Generate dependencies and rules for building intermediate asciidoc files
+# included from the asciidoc high level documents. The intermediate files are
+# generated from kernel sources using kernel-doc, according to the file names
+# and suffixes in the asciidoc high level documents.
+%.deps: %.txt
+	$(ASCIIDOC_INCLUDES) < $< | $(KERNELDOC_DEPS) -k $(KERNELDOC) -t $(KERNELDOC_HELPER) -s $(srctree) -d $(obj) -m $(@:%.deps=%.xml) > $@
+
+# Include (and generate as necessary) the dependencies.
+-include $(addprefix $(obj)/,$(ASCIIDOC_BOOKS:%.txt=%.deps))
+
+###
 # DOCPROC is used for two purposes:
 # 1) To generate a dependency list for a .tmpl file
 # 2) To preprocess a .tmpl file and call kernel-doc with
diff --git a/Documentation/DocBook/gpu.txt b/Documentation/DocBook/gpu.txt
index 71aa1473461e..9ba9247d1c43 100644
--- a/Documentation/DocBook/gpu.txt
+++ b/Documentation/DocBook/gpu.txt
@@ -6,7 +6,7 @@ The Linux DRM layer contains code intended to support the needs of complex graph
 
 A note on versions: this guide covers features found in the DRM tree, including the TTM memory manager, output configuration and mode setting, and the new vblank internals, in addition to all the regular features found in current kernels.
 
-[Insert diagram of typical DRM stack here]
+// Insert diagram of typical DRM stack here
 
 == Style Guidelines
 
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1317794

On Tue, 26 Jan 2016, Jonathan Corbet <corbet@lwn.net> wrote:
> So here is a proof-of-concept series showing how a fully asciidoc-based
> toolchain might work.  Lots of hackery here, this isn't meant to be applied
> to anything at this point, but it's a good start.  What this series has is:
>
>  - Jani Nikula's patch adding asciidoc output to kernel-doc.  Thanks for
>    doing this!  It was the kickstart that was needed to get this process
>    going.

Hooray! :)

>  - Tweak docproc to handle asciidoc template files.  If a template ends in
>    ".adt", it's an asciidoc template; it's processed pretty much the same
>    way, except that kernel-doc gets the -asciidoc argument.
>
>  - Bash on the Makefile to get it to process asciidoc templates into HTML.
>    Naturally this was where most of the time got spent.  *Only* HTML output
>    works at the moment.
>
>  - Convert tracepoints.html to tpoint.adt as a proof of concept.  It works,
>    and the output is much pleasing, IMO.
>
> I'm sure there's a thousand details to deal with, and there is the issue of
> the other output formats.  asciidoctor claims to be able to create man
> pages, but I've not tried that yet; neither tool will do PDF.  Maybe we
> could rely on pandoc to do that.  Otherwise, getting to asciidoc to XML is
> straightforward, so it should be possible to use xmlto as is done now.
>
> It's all in the doc/asciidoc branch of git://git.lwn.net/linux.git if
> anybody wants to mess with it.
>
> Comments?

I'm afraid we've done some overlapping work in the mean time, but I'm
happy we've both looked at the tool chain, and can have a more
meaningful conversation now.

I first took roughly the same approach as you did. I was really
impressed with the speed and the beauty of the produced HTML. The
trouble is, neither asciidoc nor asciidoctor can produce chunked (split
to several pages) HTML directly. This is a showstopper for the gpu
document which turns into 1.3 MB of HTML, which looks pretty but is a
paint to navigate. To do chunked output, you have to output DocBook and
handle that like we do now. So while I would like to have asciidoc
generate HTML directly for speed and beauty, I ended up going the
asciidoc to DocBook path. The upside is all the output formats are
supported.

(We could of course split the source documents, but then I believe we'd
have lots of trouble cross-referencing between the documents. I could be
proven wrong. I'd *like* to be proven wrong.)

One significant difference between our approaches is that I ditched
docproc out of the equation. Instead of having the docproc ! directives
in the asciidoc, I opted for using asciidoc's native include macro, with
specially crafted filename suffixes to specify what parts of the source
to include. Those files are then generated as intermediate asciidoc
files using kernel-doc, with dependencies set right. There's a bunch of
scripting involved, but it's pretty straight forward. 

So you'd have, for example,

include::drivers/gpu/drm/drm_ioctl.c,export[]

to include all the exported functions, or

include::drivers/gpu/drm/i915/i915_irq.c,doc,interrupt_handling[]

to include the DOC: section. I think I'd like some version of this,
regardless of whether asciidoc generates HTML or DocBook. It lets make
parallelize all of kernel-doc processing for free, which is a big win.

Patches follow, also available in kernel-asciidoc branch of
git://people.freedesktop.org/~jani/drm (web interface
http://cgit.freedesktop.org/~jani/drm/log/?h=kernel-asciidoc)

BR,
Jani.


Jani Nikula (10):
  kernel-doc: rewrite usage description, remove duplicated comments
  kernel-doc: add support for asciidoc output
  kernel-doc: support printing exported and non-exported symbols
  kernel-doc: add support for printing DOC: comments with escaped names
  scripts: add asciidoc-includes to extract includes from asciidoc
  scripts: add a kernel-doc helper for special invocation
  scripts: add tool for generating asciidoc dependencies and rules
  scripts: add a crude converter from DocBook tmpl to asciidoc
  Documentation: convert gpu.tmpl to gpu.txt
  Documentation: build asciidoc documentation

 Documentation/DocBook/Makefile |   37 +-
 Documentation/DocBook/gpu.tmpl | 3499 ----------------------------------------
 Documentation/DocBook/gpu.txt  | 1183 ++++++++++++++
 scripts/asciidoc-includes      |    6 +
 scripts/kernel-doc             |  365 ++++-
 scripts/kernel-doc-deps        |   66 +
 scripts/kernel-doc-helper      |   70 +
 scripts/tmpl2asciidoc          |   39 +
 8 files changed, 1709 insertions(+), 3556 deletions(-)
 delete mode 100644 Documentation/DocBook/gpu.tmpl
 create mode 100644 Documentation/DocBook/gpu.txt
 create mode 100755 scripts/asciidoc-includes
 create mode 100755 scripts/kernel-doc-deps
 create mode 100755 scripts/kernel-doc-helper
 create mode 100755 scripts/tmpl2asciidoc

-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1317795 — [RFC 09/10] Documentation: convert gpu.tmpl to gpu.txt

This is the verbatim result of

$ scripts/tmpl2asciidoc < Documentation/DocBook/gpu.tmpl > Documentation/DocBook/gpu.txt
$ rm Documentation/DocBook/gpu.tmpl

It should probably be split up to separate documents for GPU drivers
etc. using include macros within asciidoc, but this is for demonstration
purposes.

Also, the pandoc --no-wrap is ugly, as explained in the commit message
for the conversion tool.

The output is worth looking at for the new style includes for kernel-doc
comments, for example:

include::drivers/gpu/drm/drm_irq.c,export[]
include::include/drm/drmP.h,function,drm_crtc_vblank_waitqueue[]

Signed-off-by: Jani Nikula <jani.nikula@intel.com>


*ATTENTION*: The patch was big and contained long lines which git send-email
refused to mail (damn --no-wrap again). I just dropped it out and left this as
placeholder. The commit is at:

http://cgit.freedesktop.org/~jani/drm/commit/?h=kernel-asciidoc&id=aa4b1d229401e83c98a38c2b2fe1d9f3012bb5b5

---
 Documentation/DocBook/gpu.tmpl | 3499 ----------------------------------------
 Documentation/DocBook/gpu.txt  | 1183 ++++++++++++++
 2 files changed, 1183 insertions(+), 3499 deletions(-)
 delete mode 100644 Documentation/DocBook/gpu.tmpl
 create mode 100644 Documentation/DocBook/gpu.txt

[toc] | [prev] | [next] | [standalone]

#1317797 — [RFC 05/10] scripts: add asciidoc-includes to extract includes from asciidoc

Given an asciidoc input file in standard input, print the files included
using asciidoc include macros to standard output, one per line. This
will be helpful in asciidoc dependency generation.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/asciidoc-includes | 6 ++++++
 1 file changed, 6 insertions(+)
 create mode 100755 scripts/asciidoc-includes

diff --git a/scripts/asciidoc-includes b/scripts/asciidoc-includes
new file mode 100755
index 000000000000..d832035275eb
--- /dev/null
+++ b/scripts/asciidoc-includes
@@ -0,0 +1,6 @@
+#!/bin/sh
+# Given an asciidoc input file in standard input, print the files included using
+# asciidoc include macros to standard output, one per line.
+
+INCLUDE="^include::\([^[]*\)\[.*\]"
+exec sed "/${INCLUDE}/!d; s/${INCLUDE}/\1/"
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1317798 — [RFC 06/10] scripts: add a kernel-doc helper for special invocation

Add a helper for calling kernel-doc, with the -w parameter describing
the parameters to pass on to kernel-doc, in a special format.

[This is easier to explain and will make more sense in the context of
the following patches. The -w parameter is expected to be a filename
with the kernel-doc parameters encoded at the end of the filename. This
could be included in kernel-doc itself, but it feels like polluting the
tool. So here's a helper, at least for now.]

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc-helper | 70 +++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)
 create mode 100755 scripts/kernel-doc-helper

diff --git a/scripts/kernel-doc-helper b/scripts/kernel-doc-helper
new file mode 100755
index 000000000000..1e26266f9feb
--- /dev/null
+++ b/scripts/kernel-doc-helper
@@ -0,0 +1,70 @@
+#!/bin/bash
+
+KERNELDOC=kernel-doc
+OUTPUT_FORMAT=-asciidoc
+
+WHAT=
+
+usage()
+{
+	cat <<EOF
+$ kernel-doc-helper [-k KERNELDOC] -w WHAT SOURCE
+
+Run KERNELDOC (kernel-doc by default) on SOURCE to extract information specified
+by WHAT.
+
+WHAT is an arbitrary string ending in special suffixes that are converted to
+kernel-doc parameters.
+EOF
+	exit 0
+}
+
+while getopts "hw:k:" opt; do
+	case "$opt" in
+		h)
+			usage
+			;;
+		w)
+			WHAT="$OPTARG"
+			;;
+		k)
+			KERNELDOC="$OPTARG"
+			;;
+		*)
+			exit 1
+	esac
+done
+shift `expr $OPTIND - 1`
+
+if [ "$#" = "0" ]; then
+	echo "$0: source file missing" >&2
+	exit 1
+fi
+
+if [ -z "$WHAT" ]; then
+	echo "$0: output selection (-w) missing" >&2
+	exit 1
+fi
+
+SOURCE=$1
+
+case $WHAT in
+	*,export)
+		exec $KERNELDOC $OUTPUT_FORMAT -export $SOURCE
+		;;
+	*,internal)
+		exec $KERNELDOC $OUTPUT_FORMAT -internal $SOURCE
+		;;
+	*,function,*)
+		name=${WHAT##*,function,}
+		exec $KERNELDOC $OUTPUT_FORMAT -function $name $SOURCE
+		;;
+	*,doc,*)
+		name=${WHAT##*,doc,}
+		exec $KERNELDOC $OUTPUT_FORMAT -doc $name $SOURCE
+		;;
+	*)
+		echo "$0: unknown output selection '$WHAT'" >&2
+		exit 1
+		;;
+esac
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1317805 — [RFC 04/10] kernel-doc: add support for printing DOC: comments with escaped names

-function supports printing named DOC: sections, but spaces and braces
and quotes etc. are allowed in section titles. This is tricky to handle
in scripts, let alone Makefiles.

Add a new -doc parameter for dumping doc sections (to not convolute
-function more than it already is), with support for "escaped" names
with everything non-alphanumeric repaced with underscores, in addition
to verbatim names.

For example, all these three now do the same thing:

$ scripts/kernel-doc -function "Panel Self Refresh (PSR/SRD)" drivers/gpu/drm/i915/intel_psr.c
$ scripts/kernel-doc -doc "Panel Self Refresh (PSR/SRD)" drivers/gpu/drm/i915/intel_psr.c
$ scripts/kernel-doc -doc "Panel_Self_Refresh__PSR_SRD_" drivers/gpu/drm/i915/intel_psr.c

Use of -function for extracting DOC: sections should probably be
deprecated, but keep it around for backward compatibility.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index ee2ac9137a43..0e410daa92a9 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -71,6 +71,9 @@ Output selection (mutually exclusive):
   -nofunction NAME	Do NOT output documentation for the given function(s);
 			only output documentation for the other functions and
 			DOC: sections. May be specified multiple times.
+  -doc NAME		Only output documentation for the given DOC: section
+			titles. NAME is matched both as-is and with all
+			non-alphanumeric characters replaced with underscores.
 
 Output selection modifiers:
   -no-doc-sections	Do not output DOC: sections.
@@ -457,6 +460,10 @@ while ($ARGV[0] =~ m/^-(.*)/) {
     } elsif ($cmd eq "-internal") { # only non-exported symbols
 	$function_only = 4;
 	%function_table = ()
+    } elsif ($cmd eq "-doc") { # to only output specific doc sections
+	$function_only = 5;
+	$function = shift @ARGV;
+	$function_table{$function} = 1;
     } elsif ($cmd eq "-v") {
 	$verbose = 1;
     } elsif (($cmd eq "-h") || ($cmd eq "--help")) {
@@ -520,14 +527,20 @@ sub dump_doc_section {
     my $file = shift;
     my $name = shift;
     my $contents = join "\n", @_;
+    my $escaped_name = $name;
 
     if ($no_doc_sections) {
         return;
     }
 
+    $escaped_name =~ s/[^a-zA-Z0-9]/_/g;
+
     if (($function_only == 0) ||
 	( $function_only == 1 && defined($function_table{$name})) ||
-	( $function_only == 2 && !defined($function_table{$name})))
+	( $function_only == 2 && !defined($function_table{$name})) ||
+	( $function_only == 5 &&
+	  (defined($function_table{$name}) ||
+	   defined($function_table{$escaped_name}))))
     {
 	dump_section($file, $name, $contents);
 	output_blockhead({'sectionlist' => \@sectionlist,
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1317809 — [RFC 01/10] kernel-doc: rewrite usage description, remove duplicated comments

Instead of having the kernel-doc usage in both comments and in output to
the user, merge them all to one here document. While at it, imrove the
text and make it pretty. Give shoemaker's children some shoes.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 83 ++++++++++++++++++++++++------------------------------
 1 file changed, 37 insertions(+), 46 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index c37255bb620d..29fd5cabb657 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -39,41 +39,43 @@ use strict;
 # 25/07/2012 - Added support for HTML5
 # -- Dan Luedtke <mail@danrl.de>
 
-#
-# This will read a 'c' file and scan for embedded comments in the
-# style of gnome comments (+minor extensions - see below).
-#
-
-# Note: This only supports 'c'.
-
-# usage:
-# kernel-doc [ -docbook | -html | -html5 | -text | -man | -list ]
-#            [ -no-doc-sections ]
-#            [ -function funcname [ -function funcname ...] ]
-#            c file(s)s > outputfile
-# or
-#            [ -nofunction funcname [ -function funcname ...] ]
-#            c file(s)s > outputfile
-#
-#  Set output format using one of -docbook -html -html5 -text or -man.
-#  Default is man.
-#  The -list format is for internal use by docproc.
-#
-#  -no-doc-sections
-#	Do not output DOC: sections
-#
-#  -function funcname
-#	If set, then only generate documentation for the given function(s) or
-#	DOC: section titles.  All other functions and DOC: sections are ignored.
-#
-#  -nofunction funcname
-#	If set, then only generate documentation for the other function(s)/DOC:
-#	sections. Cannot be used together with -function (yes, that's a bug --
-#	perl hackers can fix it 8))
-#
-#  c files - list of 'c' files to process
-#
-#  All output goes to stdout, with errors to stderr.
+sub usage {
+    my $message = <<"EOF";
+Usage: $0 [OPTION ...] FILE ...
+
+Read C language source or header FILEs, extract embedded documentation comments,
+and print formatted documentation to standard output.
+
+The documentation comments are identified by "/**" opening comment mark. See
+Documentation/kernel-doc-nano-HOWTO.txt for the documentation comment syntax.
+
+Output format selection (mutually exclusive):
+  -docbook		Output DocBook format.
+  -html			Output HTML format.
+  -html5		Output HTML5 format.
+  -list			Output symbol list format. This is for use by docproc.
+  -man			Output troff manual page format. This is the default.
+  -text			Output plain text format.
+
+Output selection (mutually exclusive):
+  -function NAME	Only output documentation for the given function(s)
+			or DOC: section title(s). All other functions and DOC:
+			sections are ignored. May be specified multiple times.
+  -nofunction NAME	Do NOT output documentation for the given function(s);
+			only output documentation for the other functions and
+			DOC: sections. May be specified multiple times.
+
+Output selection modifiers:
+  -no-doc-sections	Do not output DOC: sections.
+
+Other parameters:
+  -v			Verbose output, more warnings and other information.
+  -h			Print this help.
+
+EOF
+    print $message;
+    exit 1;
+}
 
 #
 # format of comments.
@@ -437,17 +439,6 @@ while ($ARGV[0] =~ m/^-(.*)/) {
 
 # continue execution near EOF;
 
-sub usage {
-    print "Usage: $0 [ -docbook | -html | -html5 | -text | -man | -list ]\n";
-    print "         [ -no-doc-sections ]\n";
-    print "         [ -function funcname [ -function funcname ...] ]\n";
-    print "         [ -nofunction funcname [ -nofunction funcname ...] ]\n";
-    print "         [ -v ]\n";
-    print "         c source file(s) > outputfile\n";
-    print "         -v : verbose output, more warnings & other info listed\n";
-    exit 1;
-}
-
 # get kernel version from env
 sub get_kernel_version() {
     my $version = 'unknown kernel version';
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1317813

On Tue, Jan 26, 2016 at 1:08 PM, Jani Nikula <jani.nikula@intel.com> wrote:
> On Tue, 26 Jan 2016, Jonathan Corbet <corbet@lwn.net> wrote:
>> So here is a proof-of-concept series showing how a fully asciidoc-based
>> toolchain might work.  Lots of hackery here, this isn't meant to be applied
>> to anything at this point, but it's a good start.  What this series has is:
>>
>>  - Jani Nikula's patch adding asciidoc output to kernel-doc.  Thanks for
>>    doing this!  It was the kickstart that was needed to get this process
>>    going.
>
> Hooray! :)
>
>>  - Tweak docproc to handle asciidoc template files.  If a template ends in
>>    ".adt", it's an asciidoc template; it's processed pretty much the same
>>    way, except that kernel-doc gets the -asciidoc argument.
>>
>>  - Bash on the Makefile to get it to process asciidoc templates into HTML.
>>    Naturally this was where most of the time got spent.  *Only* HTML output
>>    works at the moment.
>>
>>  - Convert tracepoints.html to tpoint.adt as a proof of concept.  It works,
>>    and the output is much pleasing, IMO.
>>
>> I'm sure there's a thousand details to deal with, and there is the issue of
>> the other output formats.  asciidoctor claims to be able to create man
>> pages, but I've not tried that yet; neither tool will do PDF.  Maybe we
>> could rely on pandoc to do that.  Otherwise, getting to asciidoc to XML is
>> straightforward, so it should be possible to use xmlto as is done now.
>>
>> It's all in the doc/asciidoc branch of git://git.lwn.net/linux.git if
>> anybody wants to mess with it.
>>
>> Comments?
>
> I'm afraid we've done some overlapping work in the mean time, but I'm
> happy we've both looked at the tool chain, and can have a more
> meaningful conversation now.
>
> I first took roughly the same approach as you did. I was really
> impressed with the speed and the beauty of the produced HTML. The
> trouble is, neither asciidoc nor asciidoctor can produce chunked (split
> to several pages) HTML directly. This is a showstopper for the gpu
> document which turns into 1.3 MB of HTML, which looks pretty but is a
> paint to navigate. To do chunked output, you have to output DocBook and
> handle that like we do now. So while I would like to have asciidoc
> generate HTML directly for speed and beauty, I ended up going the
> asciidoc to DocBook path. The upside is all the output formats are
> supported.

This is a big bummer since with the parralized kernel-doc processing
using Makefiles and using asciidoctor even building something big like
the gpu docs is down to 2-3 seconds. From a clean tree, so not even
counting incremental speed-ups. Unfortunately asciidoc doesn't have an
built-in tooling (there's some experimental extensions) to split
things up.
-Daniel

> (We could of course split the source documents, but then I believe we'd
> have lots of trouble cross-referencing between the documents. I could be
> proven wrong. I'd *like* to be proven wrong.)
>
> One significant difference between our approaches is that I ditched
> docproc out of the equation. Instead of having the docproc ! directives
> in the asciidoc, I opted for using asciidoc's native include macro, with
> specially crafted filename suffixes to specify what parts of the source
> to include. Those files are then generated as intermediate asciidoc
> files using kernel-doc, with dependencies set right. There's a bunch of
> scripting involved, but it's pretty straight forward.
>
> So you'd have, for example,
>
> include::drivers/gpu/drm/drm_ioctl.c,export[]
>
> to include all the exported functions, or
>
> include::drivers/gpu/drm/i915/i915_irq.c,doc,interrupt_handling[]
>
> to include the DOC: section. I think I'd like some version of this,
> regardless of whether asciidoc generates HTML or DocBook. It lets make
> parallelize all of kernel-doc processing for free, which is a big win.
>
> Patches follow, also available in kernel-asciidoc branch of
> git://people.freedesktop.org/~jani/drm (web interface
> http://cgit.freedesktop.org/~jani/drm/log/?h=kernel-asciidoc)
>
> BR,
> Jani.
>
>
> Jani Nikula (10):
>   kernel-doc: rewrite usage description, remove duplicated comments
>   kernel-doc: add support for asciidoc output
>   kernel-doc: support printing exported and non-exported symbols
>   kernel-doc: add support for printing DOC: comments with escaped names
>   scripts: add asciidoc-includes to extract includes from asciidoc
>   scripts: add a kernel-doc helper for special invocation
>   scripts: add tool for generating asciidoc dependencies and rules
>   scripts: add a crude converter from DocBook tmpl to asciidoc
>   Documentation: convert gpu.tmpl to gpu.txt
>   Documentation: build asciidoc documentation
>
>  Documentation/DocBook/Makefile |   37 +-
>  Documentation/DocBook/gpu.tmpl | 3499 ----------------------------------------
>  Documentation/DocBook/gpu.txt  | 1183 ++++++++++++++
>  scripts/asciidoc-includes      |    6 +
>  scripts/kernel-doc             |  365 ++++-
>  scripts/kernel-doc-deps        |   66 +
>  scripts/kernel-doc-helper      |   70 +
>  scripts/tmpl2asciidoc          |   39 +
>  8 files changed, 1709 insertions(+), 3556 deletions(-)
>  delete mode 100644 Documentation/DocBook/gpu.tmpl
>  create mode 100644 Documentation/DocBook/gpu.txt
>  create mode 100755 scripts/asciidoc-includes
>  create mode 100755 scripts/kernel-doc-deps
>  create mode 100755 scripts/kernel-doc-helper
>  create mode 100755 scripts/tmpl2asciidoc
>
> --
> 2.1.4
>



-- 
Daniel Vetter
Software Engineer, Intel Corporation
+41 (0) 79 365 57 48 - http://blog.ffwll.ch

[toc] | [prev] | [next] | [standalone]

#1317840

On Tue, 26 Jan 2016, Daniel Vetter <daniel.vetter@ffwll.ch> wrote:
> On Tue, Jan 26, 2016 at 1:08 PM, Jani Nikula <jani.nikula@intel.com> wrote:
>> I first took roughly the same approach as you did. I was really
>> impressed with the speed and the beauty of the produced HTML. The
>> trouble is, neither asciidoc nor asciidoctor can produce chunked (split
>> to several pages) HTML directly. This is a showstopper for the gpu
>> document which turns into 1.3 MB of HTML, which looks pretty but is a
>> paint to navigate. To do chunked output, you have to output DocBook and
>> handle that like we do now. So while I would like to have asciidoc
>> generate HTML directly for speed and beauty, I ended up going the
>> asciidoc to DocBook path. The upside is all the output formats are
>> supported.
>
> This is a big bummer since with the parralized kernel-doc processing
> using Makefiles and using asciidoctor even building something big like
> the gpu docs is down to 2-3 seconds. From a clean tree, so not even
> counting incremental speed-ups. Unfortunately asciidoc doesn't have an
> built-in tooling (there's some experimental extensions) to split
> things up.

Basically asciidoc -> HTML is about as fast as asciidoc -> XML, and with
parallel kernel-doc it really is fast. Sadly the XML -> HTML part still
takes forever.

I just want to emphasize that we can get parallel kernel-doc with either
pipeline. It is also possible to enable both pipelines, i.e. have a fast
path HTML generation with few external dependencies and the Swiss army
knife slow path with XML.

I should also remind us that the original goal was to enable lightweight
markup for documentation. This seems very much achievable now. We don't
have to solve all the existing problems with the XML pipeline right
now. And asciidoc suits this well, as it can also feed to the existing
pipeline.


BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1317814 — [RFC 03/10] kernel-doc: support printing exported and non-exported symbols

Currently we use docproc to figure out which symbols are exported, and
then docproc calls kernel-doc on specific functions, to get
documentation on exported functions. According to git blame and docproc
comments, this is due to historical reasons, as functions and their
corresponding EXPORT_SYMBOL* may have been in different files. However
for more than ten years the recommendation in CodingStyle has been to
place the EXPORT_SYMBOL* immediately after the closing function brace
line.

Additionally, the kernel-doc comments for functions are generally placed
above the function definition in the .c files (i.e. where the
EXPORT_SYMBOL* is) rather than above the declaration in the .h
files. There are some exceptions to this, but AFAICT none of these are
included in DocBook documentation using the "!E" docproc directive.

Therefore, assuming the EXPORT_SYMBOL* and kernel-doc are with the
function definition, kernel-doc can extract the exported vs. not
information by making two passes on the input file. Add support for that
via the new -export and -internal parameters.

[This should cover most of the cases. If it's not enough, it's trivial
to allow specifying additional files to look for EXPORT_SYMBOL* in.]

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 29 +++++++++++++++++++++++++++--
 1 file changed, 27 insertions(+), 2 deletions(-)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 15077f910e81..ee2ac9137a43 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -59,6 +59,12 @@ Output format selection (mutually exclusive):
   -text			Output plain text format.
 
 Output selection (mutually exclusive):
+  -export		Only output documentation for symbols that have been
+			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
+			in the same FILE.
+  -internal		Only output documentation for symbols that have NOT been
+			exported using EXPORT_SYMBOL() or EXPORT_SYMBOL_GPL()
+			in the same FILE.
   -function NAME	Only output documentation for the given function(s)
 			or DOC: section title(s). All other functions and DOC:
 			sections are ignored. May be specified multiple times.
@@ -380,6 +386,7 @@ my $doc_block = $doc_com . 'DOC:\s*(.*)?';
 my $doc_split_start = '^\s*/\*\*\s*$';
 my $doc_split_sect = '\s*\*\s*(@[\w\s]+):(.*)';
 my $doc_split_end = '^\s*\*/\s*$';
+my $export_symbol = '^\s*EXPORT_SYMBOL(_GPL)?\s*\(\s*(\w+)\s*\)\s*;';
 
 my %constants;
 my %parameterdescs;
@@ -444,6 +451,12 @@ while ($ARGV[0] =~ m/^-(.*)/) {
 	$function_only = 2;
 	$function = shift @ARGV;
 	$function_table{$function} = 1;
+    } elsif ($cmd eq "-export") { # only exported symbols
+	$function_only = 3;
+	%function_table = ()
+    } elsif ($cmd eq "-internal") { # only non-exported symbols
+	$function_only = 4;
+	%function_table = ()
     } elsif ($cmd eq "-v") {
 	$verbose = 1;
     } elsif (($cmd eq "-h") || ($cmd eq "--help")) {
@@ -1976,8 +1989,10 @@ sub output_declaration {
     my $functype = shift;
     my $func = "output_${functype}_$output_mode";
     if (($function_only==0) ||
-	( $function_only == 1 && defined($function_table{$name})) ||
-	( $function_only == 2 && !($functype eq "function" && defined($function_table{$name}))))
+	( ($function_only == 1 || $function_only == 3) &&
+	  defined($function_table{$name})) ||
+	( ($function_only == 2 || $function_only == 4) &&
+	  !($functype eq "function" && defined($function_table{$name}))))
     {
 	&$func(@_);
 	$section_counter++;
@@ -2682,6 +2697,16 @@ sub process_file($) {
 
     $. = 1;
 
+    # two passes for -export and -internal
+    if ($function_only == 3 || $function_only == 4) {
+	while (<IN>) {
+	    if (/$export_symbol/o) {
+		$function_table{$2} = 1;
+	    }
+	}
+	seek(IN, 0, 0);
+    }
+
     $section_counter = 0;
     while (<IN>) {
 	while (s/\\\s*$//) {
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1317815 — [RFC 07/10] scripts: add tool for generating asciidoc dependencies and rules

Given a list of special format filenames, generate dependencies and
rules for kernel-doc invocation on those files, to be included in to
Makefiles.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc-deps | 66 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 66 insertions(+)
 create mode 100755 scripts/kernel-doc-deps

diff --git a/scripts/kernel-doc-deps b/scripts/kernel-doc-deps
new file mode 100755
index 000000000000..c348863b9354
--- /dev/null
+++ b/scripts/kernel-doc-deps
@@ -0,0 +1,66 @@
+#!/bin/bash
+
+SRCTREE=
+DSTTREE=
+DEP=
+KERNELDOC=
+TOOL=kernel-doc-helper
+
+while getopts "k:t:s:d:m:" opt; do
+	case "$opt" in
+		k)
+			KERNELDOC="$OPTARG"
+			;;
+		t)
+			TOOL="$OPTARG"
+			;;
+		s)
+			SRCTREE="$OPTARG/"
+			;;
+		d)
+			DSTTREE="$OPTARG/"
+			;;
+		m)
+			DEP="$OPTARG"
+			;;
+		*)
+			echo "fail"
+			exit 1
+	esac
+done
+shift `expr $OPTIND - 1`
+
+ACTION="\tmkdir \$()\n\t${TOOL} -w \$@ \$^ > \$@"
+
+# $1 = filename
+# $2 = for what type
+print_rule()
+{
+	echo "$1: $2"
+	echo -e "\tmkdir -p \$(dir \$@)"
+	echo -e "\t${TOOL} -k ${KERNELDOC} -w \$@ \$^ > \$@"
+	echo "$DEP: $1"
+}
+
+# given a list of filenames with special notation, generate deps
+
+while read file; do
+	case $file in
+		# all of these are C/H source files
+		*,export)
+			print_rule "${DSTTREE}$file" "${SRCTREE}${file%,export}"
+			;;
+		*,internal)
+			print_rule "${DSTTREE}$file" "${SRCTREE}${file%,internal}"
+			;;
+		*,function,*)
+			print_rule "${DSTTREE}$file" "${SRCTREE}${file%,function,*}"
+			;;
+		*,doc,*)
+			print_rule "${DSTTREE}$file" "${SRCTREE}${file%,doc,*}"
+			;;
+		*)
+			print_rule "${DSTTREE}/${file%,txt,} " "${SRCTREE}${file%,F,*}"
+			;;
+	esac
+done
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1317819 — [RFC 08/10] scripts: add a crude converter from DocBook tmpl to asciidoc

Use some pre- and post-processing to handle the "![EIFPCD]" docproc
directives in the DocBook, and let pandoc do the rest. Manual editing
will be required, but this will give a big jump start.

The asciidoc result would be nicer without the pandoc --no-wrap option,
but unfortunately pandoc also wraps the docproc directives within
<pre></pre> tags, breaking their post-processing. There's probably a way
around this, but I couldn't be bothered for this proof of concept.

The post-processing converts the directives to the new format of having
comma separated filename suffixes describe the content to be included.

Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/tmpl2asciidoc | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)
 create mode 100755 scripts/tmpl2asciidoc

diff --git a/scripts/tmpl2asciidoc b/scripts/tmpl2asciidoc
new file mode 100755
index 000000000000..092400cc702c
--- /dev/null
+++ b/scripts/tmpl2asciidoc
@@ -0,0 +1,39 @@
+#!/bin/bash
+# a crude converter from DocBook tmpl in STDIN to AsciiDoc in STDOUT
+
+sed 's/^\(!.*\)/<pre>\1<\/pre>/' |\
+	pandoc --atx-headers --no-wrap -f docbook -t asciidoc |\
+	while read line; do
+		case "$line" in
+			!E*)
+				file=${line#!?}
+				echo "include::$file,export[]"
+				;;
+			!I*)
+				file=${line#!?}
+				echo "include::$file,internal[]"
+				;;
+			!F*)
+				file=${line#!?}
+				file=${file%% *}
+				functions=${line#* }
+				for f in $functions; do
+					echo "include::$file,function,$f[]"
+				done
+				;;
+			!P*)
+				file=${line#!?}
+				file=${file%% *}
+				doc=${line#* }
+				doc=${doc//[^A-Za-z0-9]/_}
+				echo "include::$file,doc,$doc[]"
+				;;
+			!C*|!D*)
+				echo "$0: WARNING: unsupported: $line" >&2
+				echo "// $line"
+				;;
+			*)
+				echo -E "$line"
+				;;
+		esac
+	done
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1317824 — [RFC 02/10] kernel-doc: add support for asciidoc output

Add new -asciidoc option to produce asciidoc output from kernel-doc. The
output is formatted internally, with no dependencies on external
tools. Any asciidoc formatting present in kernel-doc will naturally be
present in the resulting asciidoc as well.

Highlighting of functions(), &struct structures, &enum enumerations,
@parameters, etc. will be done by means of asciidoc. Anchors and
cross-references are added as well, providing hyperlinking support in
the result processed by asciidoc(1).

This support is non-conflicting and orthogonal to the patches adding
asciidoc support to the kernel-doc DocBook output [1]. Those patches
bolt on to the current document generation pipeline; there is currently
none for native asciidoc in the kernel (though ideas have been discussed
[2]). At this time, this patch should be considered a worthwhile
standalone improvement to kernel-doc, independent of the rest.

[1] http://mid.gmane.org/1448471279-19748-1-git-send-email-daniel.vetter@ffwll.ch
[2] http://mid.gmane.org/20160114131823.2ff43a0c@lwn.net

Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Daniel Vetter <daniel@ffwll.ch>
Signed-off-by: Jani Nikula <jani.nikula@intel.com>
---
 scripts/kernel-doc | 238 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 238 insertions(+)

diff --git a/scripts/kernel-doc b/scripts/kernel-doc
index 29fd5cabb657..15077f910e81 100755
--- a/scripts/kernel-doc
+++ b/scripts/kernel-doc
@@ -50,6 +50,7 @@ The documentation comments are identified by "/**" opening comment mark. See
 Documentation/kernel-doc-nano-HOWTO.txt for the documentation comment syntax.
 
 Output format selection (mutually exclusive):
+  -asciidoc		Output AsciiDoc format.
   -docbook		Output DocBook format.
   -html			Output HTML format.
   -html5		Output HTML5 format.
@@ -203,6 +204,8 @@ my $type_param = '\@(\w+)';
 my $type_struct = '\&((struct\s*)*[_\w]+)';
 my $type_struct_xml = '\\&amp;((struct\s*)*[_\w]+)';
 my $type_env = '(\$\w+)';
+my $type_enum_full = '\&(enum)\s*([_\w]+)';
+my $type_struct_full = '\&(struct)\s*([_\w]+)';
 
 # Output conversion substitutions.
 #  One for each output format
@@ -268,6 +271,17 @@ my @highlights_text = (
 		      );
 my $blankline_text = "";
 
+# asciidoc-mode
+my @highlights_asciidoc = (
+                       [$type_constant, "`\$1`"],
+                       [$type_func, "<<func_\$1,`\$1()`>>"],
+                       [$type_struct_full, "<<\$1_\$2,`\$1 \$2`>>"],
+                       [$type_enum_full, "<<\$1_\$2,`\$1 \$2`>>"],
+                       [$type_struct, "<<struct_\$1,`\$1`>>"],
+                       [$type_param, "*\$1*"]
+		      );
+my $blankline_asciidoc = "\n";
+
 # list mode
 my @highlights_list = (
                        [$type_constant, "\$1"],
@@ -404,6 +418,10 @@ while ($ARGV[0] =~ m/^-(.*)/) {
 	$output_mode = "text";
 	@highlights = @highlights_text;
 	$blankline = $blankline_text;
+    } elsif ($cmd eq "-asciidoc") {
+	$output_mode = "asciidoc";
+	@highlights = @highlights_asciidoc;
+	$blankline = $blankline_asciidoc;
     } elsif ($cmd eq "-docbook") {
 	$output_mode = "xml";
 	@highlights = @highlights_xml;
@@ -1704,6 +1722,214 @@ sub output_blockhead_text(%) {
     }
 }
 
+##
+# output in asciidoc
+sub output_highlight_asciidoc {
+    my $contents = join "\n",@_;
+    my $line;
+
+    # undo the evil effects of xml_escape() earlier
+    $contents = xml_unescape($contents);
+
+    eval $dohighlight;
+    die $@ if $@;
+
+    foreach $line (split "\n", $contents) {
+	if ($line eq "") {
+	    print $lineprefix, $blankline;
+	} else {
+	    $line =~ s/\\\\\\/\&/g;
+	    print $lineprefix, $line;
+	}
+	print "\n";
+    }
+}
+
+sub output_function_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter, $section);
+    my $start;
+
+    print "[[func_$args{'function'}]]\n";
+    print "=== " . $args{'function'} . " ===\n";
+    print $args{'purpose'} . "\n\n";
+
+    print "----------\n";
+    if ($args{'functiontype'} ne "") {
+	$start = $args{'functiontype'} . " " . $args{'function'} . " (";
+    } else {
+	$start = $args{'function'} . " (";
+    }
+    print $start;
+
+    my $count = 0;
+    foreach my $parameter (@{$args{'parameterlist'}}) {
+	$type = $args{'parametertypes'}{$parameter};
+	if ($type =~ m/([^\(]*\(\*)\s*\)\s*\(([^\)]*)\)/) {
+	    # pointer-to-function
+	    print $1 . $parameter . ") (" . $2;
+	} else {
+	    print $type . " " . $parameter;
+	}
+	if ($count != $#{$args{'parameterlist'}}) {
+	    $count++;
+	    print ",\n";
+	    print " " x length($start);
+	} else {
+	    print ");\n\n";
+	}
+    }
+    print "----------\n";
+
+    print ".Parameters\n";
+    print "[horizontal]\n";
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	my $parameter_name = $parameter;
+	$parameter_name =~ s/\[.*//;
+	$type = $args{'parametertypes'}{$parameter};
+
+	print "`$type $parameter`::\n";
+	if ($args{'parameterdescs'}{$parameter_name} ne $undescribed) {
+	    $blankline = "+";
+	    output_highlight_asciidoc($args{'parameterdescs'}{$parameter_name});
+	    $blankline = "\n";
+	} else {
+	    print "_undescribed_\n";
+	}
+	print "\n";
+    }
+    output_section_asciidoc(@_);
+}
+
+sub output_section_asciidoc(%) {
+    my %args = %{$_[0]};
+    my $section;
+
+    foreach $section (@{$args{'sectionlist'}}) {
+	print ".$section\n\n";
+	output_highlight_asciidoc($args{'sections'}{$section});
+	print "\n";
+    }
+    print "\n";
+}
+
+sub output_enum_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter);
+    my $count;
+
+    print "[[enum_$args{'enum'}]]\n";
+    print "=== enum " . $args{'enum'} . " ===\n";
+    print $args{'purpose'} . "\n\n";
+
+    print "----------\n";
+    print "enum " . $args{'enum'} . " {\n";
+    $count = 0;
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	print "\t$parameter";
+	if ($count != $#{$args{'parameterlist'}}) {
+	    $count++;
+	    print ",";
+	}
+	print "\n";
+    }
+    print "};\n";
+    print "----------\n";
+
+    print ".Constants\n";
+    print "[horizontal]\n";
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	print "`$parameter`::\n";
+	if ($args{'parameterdescs'}{$parameter} ne $undescribed) {
+	    $blankline = "+";
+	    output_highlight_asciidoc($args{'parameterdescs'}{$parameter});
+	    $blankline = "\n";
+	} else {
+	    print "_undescribed_\n";
+	}
+	print "\n";
+    }
+
+    output_section_asciidoc(@_);
+}
+
+sub output_typedef_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter);
+    my $count;
+
+    print "[[$args{'typedef'}]]\n";
+    print "=== typedef " . $args{'typedef'} . " ===\n";
+    print $args{'purpose'} . "\n\n";
+
+    output_section_asciidoc(@_);
+}
+
+sub output_struct_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter);
+
+    print "[[$args{'type'}_$args{'struct'}]]\n";
+    print "=== " . $args{'type'} . " " . $args{'struct'} . " ===\n";
+    print $args{'purpose'} . "\n\n";
+
+    print "----------\n";
+    print $args{'type'} . " " . $args{'struct'} . " {\n";
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	if ($parameter =~ /^#/) {
+	    print "$parameter\n";
+	    next;
+	}
+
+	my $parameter_name = $parameter;
+	$parameter_name =~ s/\[.*//;
+
+	($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
+	$type = $args{'parametertypes'}{$parameter};
+	if ($type =~ m/([^\(]*\(\*)\s*\)\s*\(([^\)]*)\)/) {
+	    # pointer-to-function
+	    print "\t$1 $parameter) ($2);\n";
+	} elsif ($type =~ m/^(.*?)\s*(:.*)/) {
+	    # bitfield
+	    print "\t$1 $parameter$2;\n";
+	} else {
+	    print "\t" . $type . " " . $parameter . ";\n";
+	}
+    }
+    print "};\n";
+    print "----------\n";
+
+    print ".Members\n";
+    print "[horizontal]\n";
+    foreach $parameter (@{$args{'parameterlist'}}) {
+	($parameter =~ /^#/) && next;
+
+	my $parameter_name = $parameter;
+	$parameter_name =~ s/\[.*//;
+
+	($args{'parameterdescs'}{$parameter_name} ne $undescribed) || next;
+	$type = $args{'parametertypes'}{$parameter};
+	print "`$type $parameter`" . "::\n";
+	$blankline = "+";
+	output_highlight_asciidoc($args{'parameterdescs'}{$parameter_name});
+	$blankline = "\n";
+	print "\n";
+    }
+    print "\n";
+    output_section_asciidoc(@_);
+}
+
+sub output_blockhead_asciidoc(%) {
+    my %args = %{$_[0]};
+    my ($parameter, $section);
+
+    foreach $section (@{$args{'sectionlist'}}) {
+	print "=== $section ===\n";
+	output_highlight_asciidoc($args{'sections'}{$section});
+	print "\n";
+    }
+}
+
 ## list mode output functions
 
 sub output_function_list(%) {
@@ -2405,6 +2631,18 @@ sub xml_escape($) {
 	return $text;
 }
 
+# xml_unescape: reverse the effects of xml_escape
+sub xml_unescape($) {
+	my $text = shift;
+	if (($output_mode eq "text") || ($output_mode eq "man")) {
+		return $text;
+	}
+	$text =~ s/\\\\\\amp;/\&/g;
+	$text =~ s/\\\\\\lt;/</g;
+	$text =~ s/\\\\\\gt;/>/g;
+	return $text;
+}
+
 # convert local escape strings to html
 # local escape strings look like:  '\\\\menmonic:' (that's 4 backslashes)
 sub local_unescape($) {
-- 
2.1.4

[toc] | [prev] | [next] | [standalone]

#1318000

On Tue, 26 Jan 2016 14:08:45 +0200
Jani Nikula <jani.nikula@intel.com> wrote:

> I'm afraid we've done some overlapping work in the mean time, but I'm
> happy we've both looked at the tool chain, and can have a more
> meaningful conversation now.

Overlapping work is just how this kernel thing works :)

> I first took roughly the same approach as you did. I was really
> impressed with the speed and the beauty of the produced HTML. The
> trouble is, neither asciidoc nor asciidoctor can produce chunked (split
> to several pages) HTML directly. This is a showstopper for the gpu
> document which turns into 1.3 MB of HTML, which looks pretty but is a
> paint to navigate. To do chunked output, you have to output DocBook and
> handle that like we do now. So while I would like to have asciidoc
> generate HTML directly for speed and beauty, I ended up going the
> asciidoc to DocBook path. The upside is all the output formats are
> supported.

So I'm not really going to have time to dig too much more into this until
after LCA, so these are really quick impressions for now.

I would *really* like to get the XML step out of the processing path if
possible.  It adds complexity, makes it harder for others to build the
docs, makes things more fragile, and slows it all down.  It seems to me
that it should be possible to do that.

The issues, it seems, are splitting the output files and format support.
The latter isn't really an issue, I don't think; there are tools to do all
kinds of format conversions.  The only one that's even slightly weird is
man, and kernel-doc already has some (unused, I think) provisions for
doing that.  We could generate man pages directly without much pain.

For HTML-page splitting, we can see if the tools can help us, consider
splitting the template files, or do the splitting in a postprocessing
step.  Docproc (or whatever replaces it) could also maybe do that work.
It doesn't seem to me something that should force the inclusion of an
entire XML-based processing step.  



But then, I'm here spouting ideas without any proof to back them up again,
so who knows...:)

Thanks for doing this work.  As you said, it has been demonstrated 

[toc] | [prev] | [next] | [standalone]

#1318004

On Tue, 26 Jan 2016 07:48:21 -0700
Jonathan Corbet <corbet@lwn.net> wrote:

> For HTML-page splitting, we can see if the tools can help us, consider
> splitting the template files, or do the splitting in a postprocessing
> step.  Docproc (or whatever replaces it) could also maybe do that work.
> It doesn't seem to me something that should force the inclusion of an
> entire XML-based processing step.  
> 
> 
> 
> But then, I'm here spouting ideas without any proof to back them up again,
> so who knows...:)
> 
> Thanks for doing this work.  As you said, it has been demonstrated 

So let's try this again without hitting that "send" button prematurely,
sigh.  Claws is annoying sometimes.

You'd said:

> (We could of course split the source documents, but then I believe we'd
> have lots of trouble cross-referencing between the documents. I could be
> proven wrong. I'd *like* to be proven wrong.)

I'd like to solve the cross-reference problem anyway; the documents
shouldn't be silos unto themselves.  Sphinx and ReST can do that; I *bet*
the other formats can too.  Maybe I can dig into some of that in my
upcoming airplane time (LAX->MEL, 15:45, sigh).

> I should also remind us that the original goal was to enable lightweight
> markup for documentation. This seems very much achievable now.

Agreed, nice work.

jon

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web