Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1515963 > unrolled thread

Including images on Sphinx documents

Back to article view | Back to linux.kernel

Contents

  Including images on Sphinx documents Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2016-11-07 11:00 +0100
    Re: Including images on Sphinx documents Jani Nikula <jani.nikula@intel.com> - 2016-11-07 12:00 +0100
      Re: Including images on Sphinx documents Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2016-11-07 12:50 +0100
        Re: [Ksummit-discuss] Including images on Sphinx documents Josh Triplett <josh@joshtriplett.org> - 2016-11-07 18:10 +0100
          Re: [Ksummit-discuss] Including images on Sphinx documents Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2016-11-08 12:00 +0100
      Re: Including images on Sphinx documents Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2016-11-09 13:30 +0100
    Re: [Ksummit-discuss] Including images on Sphinx documents Josh Triplett <josh@joshtriplett.org> - 2016-11-07 18:30 +0100
      Re: [Ksummit-discuss] Including images on Sphinx documents Markus Heiser <markus.heiser@darmarit.de> - 2016-11-09 10:30 +0100
        Re: [Ksummit-discuss] Including images on Sphinx documents Jani Nikula <jani.nikula@linux.intel.com> - 2016-11-09 12:20 +0100
          Re: [Ksummit-discuss] Including images on Sphinx documents Markus Heiser <markus.heiser@darmarit.de> - 2016-11-09 12:30 +0100
            Re: [Ksummit-discuss] Including images on Sphinx documents Jani Nikula <jani.nikula@linux.intel.com> - 2016-11-09 13:00 +0100
              Re: [Ksummit-discuss] Including images on Sphinx documents Markus Heiser <markus.heiser@darmarit.de> - 2016-11-09 23:20 +0100
                Re: [Ksummit-discuss] Including images on Sphinx documents Jani Nikula <jani.nikula@linux.intel.com> - 2016-11-10 11:40 +0100
                  Re: [Ksummit-discuss] Including images on Sphinx documents Jani Nikula <jani.nikula@linux.intel.com> - 2016-11-11 12:30 +0100
                    Re: [Ksummit-discuss] Including images on Sphinx documents Markus Heiser <markus.heiser@darmarit.de> - 2016-11-11 12:50 +0100
              Re: [Ksummit-discuss] Including images on Sphinx documents Mauro Carvalho Chehab <m.chehab@samsung.com> - 2016-11-11 10:40 +0100
          Re: [Ksummit-discuss] Including images on Sphinx documents Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2016-11-09 12:30 +0100
            Re: [Ksummit-discuss] Including images on Sphinx documents Jani Nikula <jani.nikula@linux.intel.com> - 2016-11-09 12:50 +0100

#1515963 — Including images on Sphinx documents

Hi Jon,

I'm trying to sort out the next steps to do after KS, with regards to
images included on RST files.

The issue is that Sphinx image support highly depends on the output
format. Also, despite TexLive support for svg and png images[1], Sphinx
doesn't produce the right LaTeX commands to use svg[2]. On my tests
with PNG on my notebook, it also didn't seem to do the right thing for
PNG either. So, it seems that the only safe way to support images is
to convert all of them to PDF for latex/pdf build.

[1] On Fedora, via texlive-dvipng and texlive-svg
[2] https://github.com/sphinx-doc/sphinx/issues/1907

As far as I understand from KS, two decisions was taken:

- We're not adding a sphinx extension to run generic commands;
- The PDF images should be build in runtime from their source files
  (either svg or bitmap), and not ship anymore the corresponding
  PDF files generated from its source.

As you know, we use several images at the media documentation:
	https://www.kernel.org/doc/html/latest/_images/

Those images are tightly coupled with the explanation texts. So,
maintaining them away from the documentation is not an option.

I was originally thinking that adding a graphviz extension would solve the
issue, but, in fact, most of the images aren't diagrams. Instead, there are 
several ones with images showing the result of passing certain parameters to
the ioctls, explaining things like scale and cropping and how bytes are
packed on some image formats.

Linus proposed to call some image conversion tool like ImageMagick or
inkscape to convert them to PDF when building the pdfdocs or latexdocs
target at Makefile, but there's an issue with that: Sphinx doesn't read
files from Documentation/output, and writing them directly at the
source dir would be against what it is expected when the "O=" argument
is passed to make. 

So, we have a few alternatives:

1) copy (or symlink) all rst files to Documentation/output (or to the
   build dir specified via O= directive) and generate the *.pdf there,
   and produce those converted images via Makefile.;

2) add an Sphinx extension that would internally call ImageMagick and/or
   inkscape to convert the bitmap;

3) if possible, add an extension to trick Sphinx for it to consider the 
   output dir as a source dir too.

Comments?

Regards,
Mauro

[toc] | [next] | [standalone]

#1516001

On Mon, 07 Nov 2016, Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
> Hi Jon,
>
> I'm trying to sort out the next steps to do after KS, with regards to
> images included on RST files.
>
> The issue is that Sphinx image support highly depends on the output
> format. Also, despite TexLive support for svg and png images[1], Sphinx
> doesn't produce the right LaTeX commands to use svg[2]. On my tests
> with PNG on my notebook, it also didn't seem to do the right thing for
> PNG either. So, it seems that the only safe way to support images is
> to convert all of them to PDF for latex/pdf build.
>
> [1] On Fedora, via texlive-dvipng and texlive-svg
> [2] https://github.com/sphinx-doc/sphinx/issues/1907
>
> As far as I understand from KS, two decisions was taken:
>
> - We're not adding a sphinx extension to run generic commands;
> - The PDF images should be build in runtime from their source files
>   (either svg or bitmap), and not ship anymore the corresponding
>   PDF files generated from its source.
>
> As you know, we use several images at the media documentation:
> 	https://www.kernel.org/doc/html/latest/_images/
>
> Those images are tightly coupled with the explanation texts. So,
> maintaining them away from the documentation is not an option.
>
> I was originally thinking that adding a graphviz extension would solve the
> issue, but, in fact, most of the images aren't diagrams. Instead, there are 
> several ones with images showing the result of passing certain parameters to
> the ioctls, explaining things like scale and cropping and how bytes are
> packed on some image formats.
>
> Linus proposed to call some image conversion tool like ImageMagick or
> inkscape to convert them to PDF when building the pdfdocs or latexdocs
> target at Makefile, but there's an issue with that: Sphinx doesn't read
> files from Documentation/output, and writing them directly at the
> source dir would be against what it is expected when the "O=" argument
> is passed to make. 
>
> So, we have a few alternatives:
>
> 1) copy (or symlink) all rst files to Documentation/output (or to the
>    build dir specified via O= directive) and generate the *.pdf there,
>    and produce those converted images via Makefile.;
>
> 2) add an Sphinx extension that would internally call ImageMagick and/or
>    inkscape to convert the bitmap;
>
> 3) if possible, add an extension to trick Sphinx for it to consider the 
>    output dir as a source dir too.

Looking at the available extensions, and the images to be displayed,
seems to me making svg work, somehow, is the right approach. (As opposed
to trying to represent the images in graphviz or whatnot.)

IIUC texlive supports displaying svg directly, but the problem is that
Sphinx produces bad latex for that. Can we make it work by manually
writing the latex? If yes, we wouldn't need to use an external tool to
convert the svg to something else, but rather fix the latex. Thus:

4a) See if this works:

.. only:: html

   .. image:: foo.svg

.. raw:: latex

   <the correct latex commands required to display foo.svg>

4b) Add a directive extension to make the above happen automatically.

Of course, the correct fix is to have this fixed in upstream Sphinx, but
as a workaround an extension doing the above seems plausible, and not
too much effort - provided that we can make the raw latex work.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1516034

Em Mon, 07 Nov 2016 12:53:55 +0200
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Mon, 07 Nov 2016, Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
> > Hi Jon,
> >
> > I'm trying to sort out the next steps to do after KS, with regards to
> > images included on RST files.
> >
> > The issue is that Sphinx image support highly depends on the output
> > format. Also, despite TexLive support for svg and png images[1], Sphinx
> > doesn't produce the right LaTeX commands to use svg[2]. On my tests
> > with PNG on my notebook, it also didn't seem to do the right thing for
> > PNG either. So, it seems that the only safe way to support images is
> > to convert all of them to PDF for latex/pdf build.
> >
> > [1] On Fedora, via texlive-dvipng and texlive-svg
> > [2] https://github.com/sphinx-doc/sphinx/issues/1907
> >
> > As far as I understand from KS, two decisions was taken:
> >
> > - We're not adding a sphinx extension to run generic commands;
> > - The PDF images should be build in runtime from their source files
> >   (either svg or bitmap), and not ship anymore the corresponding
> >   PDF files generated from its source.
> >
> > As you know, we use several images at the media documentation:
> > 	https://www.kernel.org/doc/html/latest/_images/
> >
> > Those images are tightly coupled with the explanation texts. So,
> > maintaining them away from the documentation is not an option.
> >
> > I was originally thinking that adding a graphviz extension would solve the
> > issue, but, in fact, most of the images aren't diagrams. Instead, there are 
> > several ones with images showing the result of passing certain parameters to
> > the ioctls, explaining things like scale and cropping and how bytes are
> > packed on some image formats.
> >
> > Linus proposed to call some image conversion tool like ImageMagick or
> > inkscape to convert them to PDF when building the pdfdocs or latexdocs
> > target at Makefile, but there's an issue with that: Sphinx doesn't read
> > files from Documentation/output, and writing them directly at the
> > source dir would be against what it is expected when the "O=" argument
> > is passed to make. 
> >
> > So, we have a few alternatives:
> >
> > 1) copy (or symlink) all rst files to Documentation/output (or to the
> >    build dir specified via O= directive) and generate the *.pdf there,
> >    and produce those converted images via Makefile.;
> >
> > 2) add an Sphinx extension that would internally call ImageMagick and/or
> >    inkscape to convert the bitmap;
> >
> > 3) if possible, add an extension to trick Sphinx for it to consider the 
> >    output dir as a source dir too.  
> 
> Looking at the available extensions, and the images to be displayed,
> seems to me making svg work, somehow, is the right approach. (As opposed
> to trying to represent the images in graphviz or whatnot.)
> 
> IIUC texlive supports displaying svg directly, but the problem is that
> Sphinx produces bad latex for that. Can we make it work by manually
> writing the latex?

It might be possible, if we write something at the LaTeX preamble
that would replace \includegraphics by something that would, instead,
use \includesvg, if the image is in SVG format. However, I don't know
enough about LaTeX to write such macro.

> If yes, we wouldn't need to use an external tool to
> convert the svg to something else, but rather fix the latex. Thus:
> 
> 4a) See if this works:
> 
> .. only:: html
> 
>    .. image:: foo.svg
> 
> .. raw:: latex
> 
>    <the correct latex commands required to display foo.svg>

This may work, although it would prevent forever the usage of some
extension to auto-numerate images and to cross-reference them.

That's said, PNG also doesn't seem to work fine on Sphinx 1.4.x.

On my tests, I installed *all* texlive extensions on Fedora 24, to
be sure that the issue is not the lack of some extension[1], with:

	# dnf install $(sudo dnf search texlive |grep all|cut -d. -f 1|grep texlive-)

When running LaTeX in interactive mode, building just the media
PDF file with:

	$ cls;make cleandocs; make SPHINXOPTS="-j5" DOCBOOKS="" SPHINXDIRS=media latexdocs 
	$ PDFLATEX=xelatex LATEXOPTS="-interaction=interactive" -C Documentation/output/media/latex

I get this:

	LaTeX Warning: Hyper reference `uapi/v4l/subdev-formats:bayer-patterns' on page
	 153 undefined on input line 21373.

	<use  "bayer.png" > [153]
	! Extra alignment tab has been changed to \cr.
	<template> \endtemplate 
                        
	l.21429 \unskip}\relax \unskip}
	                               \relax \\
	? 

This patch fixes the issue:
	https://git.linuxtv.org/mchehab/experimental.git/commit/?h=dirty-pdf&id=b709de415f34d77cc121cad95bece9c7ef4d12fd

That means that Sphinx is not generating the right LaTeX output even for
(some?) PNG images.

[1] On a side note, installing texlive-texliveonfly didn't made much
difference... I still had to manually install a lot of texlive extensions
for LaTeX build to work on Fedora. It seems that texliveonfly only solved
automatically font dependencies.

> 4b) Add a directive extension to make the above happen automatically.
> 
> Of course, the correct fix is to have this fixed in upstream Sphinx, but
> as a workaround an extension doing the above seems plausible, and not
> too much effort - provided that we can make the raw latex work.

If we're adding an extension, IMHO, it is better to do the format
conversion inside the extension. From what I understood, texlive
will still require inkscape to be installed in order for the SVG
support to work:
	https://tex.stackexchange.com/questions/122871/include-svg-images-with-the-svg-package

And it may even require "--shell-escape" to be passed at the xelatex
call if inkscape is not in the path, with seems to be a strong
indication that SVG support is not native to texlive, but, instead,
just a way to make LaTeX to call inkscape to do the image conversion.

Regards,
Mauro

[toc] | [prev] | [next] | [standalone]

#1516339 — Re: [Ksummit-discuss] Including images on Sphinx documents

On Mon, Nov 07, 2016 at 09:46:48AM -0200, Mauro Carvalho Chehab wrote:
> That's said, PNG also doesn't seem to work fine on Sphinx 1.4.x.
> 
> On my tests, I installed *all* texlive extensions on Fedora 24, to
> be sure that the issue is not the lack of some extension[1], with:
> 
> 	# dnf install $(sudo dnf search texlive |grep all|cut -d. -f 1|grep texlive-)
> 
> When running LaTeX in interactive mode, building just the media
> PDF file with:
> 
> 	$ cls;make cleandocs; make SPHINXOPTS="-j5" DOCBOOKS="" SPHINXDIRS=media latexdocs 
> 	$ PDFLATEX=xelatex LATEXOPTS="-interaction=interactive" -C Documentation/output/media/latex
> 
> I get this:
> 
> 	LaTeX Warning: Hyper reference `uapi/v4l/subdev-formats:bayer-patterns' on page
> 	 153 undefined on input line 21373.
> 
> 	<use  "bayer.png" > [153]
> 	! Extra alignment tab has been changed to \cr.
> 	<template> \endtemplate 
>                         
> 	l.21429 \unskip}\relax \unskip}
> 	                               \relax \\
> 	? 
> 
> This patch fixes the issue:
> 	https://git.linuxtv.org/mchehab/experimental.git/commit/?h=dirty-pdf&id=b709de415f34d77cc121cad95bece9c7ef4d12fd
> 
> That means that Sphinx is not generating the right LaTeX output even for
> (some?) PNG images.

\includegraphics normally works just fine for PNG images in PDF
documents.

[...]
> And it may even require "--shell-escape" to be passed at the xelatex
> call if inkscape is not in the path, with seems to be a strong
> indication that SVG support is not native to texlive, but, instead,
> just a way to make LaTeX to call inkscape to do the image conversion.

Please don't require --shell-escape as part of the TeX workflow.  If
LaTeX can't handle the desired image format natively, it needs
conversion in advance.

- Josh Triplett

[toc] | [prev] | [next] | [standalone]

#1517073 — Re: [Ksummit-discuss] Including images on Sphinx documents

Em Mon, 7 Nov 2016 09:05:05 -0800
Josh Triplett <josh@joshtriplett.org> escreveu:

> On Mon, Nov 07, 2016 at 09:46:48AM -0200, Mauro Carvalho Chehab wrote:
> > That's said, PNG also doesn't seem to work fine on Sphinx 1.4.x.
> > 
> > On my tests, I installed *all* texlive extensions on Fedora 24, to
> > be sure that the issue is not the lack of some extension[1], with:
> > 
> > 	# dnf install $(sudo dnf search texlive |grep all|cut -d. -f 1|grep texlive-)
> > 
> > When running LaTeX in interactive mode, building just the media
> > PDF file with:
> > 
> > 	$ cls;make cleandocs; make SPHINXOPTS="-j5" DOCBOOKS="" SPHINXDIRS=media latexdocs 
> > 	$ PDFLATEX=xelatex LATEXOPTS="-interaction=interactive" -C Documentation/output/media/latex
> > 
> > I get this:
> > 
> > 	LaTeX Warning: Hyper reference `uapi/v4l/subdev-formats:bayer-patterns' on page
> > 	 153 undefined on input line 21373.
> > 
> > 	<use  "bayer.png" > [153]
> > 	! Extra alignment tab has been changed to \cr.
> > 	<template> \endtemplate 
> >                         
> > 	l.21429 \unskip}\relax \unskip}
> > 	                               \relax \\
> > 	? 
> > 
> > This patch fixes the issue:
> > 	https://git.linuxtv.org/mchehab/experimental.git/commit/?h=dirty-pdf&id=b709de415f34d77cc121cad95bece9c7ef4d12fd
> > 
> > That means that Sphinx is not generating the right LaTeX output even for
> > (some?) PNG images.  
> 
> \includegraphics normally works just fine for PNG images in PDF
> documents.

I didn't try to fix the Sphinx output in LaTeX format when a PNG
image is used, but, from the above log, clearly it did something
wrong. Perhaps there's something bad defined at the sphinx.sty file,
or it simply generates the wrong LaTeX code when the image
is not on PDF format.

> [...]
> > And it may even require "--shell-escape" to be passed at the xelatex
> > call if inkscape is not in the path, with seems to be a strong
> > indication that SVG support is not native to texlive, but, instead,
> > just a way to make LaTeX to call inkscape to do the image conversion.  
> 
> Please don't require --shell-escape as part of the TeX workflow.  If
> LaTeX can't handle the desired image format natively, it needs
> conversion in advance.

Agreed. I sent a patch series to linux-doc, doing the conversion in
advance:
	https://marc.info/?l=linux-doc&m=147859902804144&w=2

Not sure why, but the archives don't have all patches yet.
Anyway, the relevant one is this:
	https://git.linuxtv.org/mchehab/experimental.git/commit/?h=pdf-fixes&id=5d41c452c787f6a6c755a3855312435bc439acb8

It basically calls ImageMagick "convert" tool for all png and
pdf files currently at the documentation (they're all at media,
ATM).
	

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1518119

Em Mon, 07 Nov 2016 12:53:55 +0200
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Mon, 07 Nov 2016, Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
> > Hi Jon,
> >
> > I'm trying to sort out the next steps to do after KS, with regards to
> > images included on RST files.
> >
> > The issue is that Sphinx image support highly depends on the output
> > format. Also, despite TexLive support for svg and png images[1], Sphinx
> > doesn't produce the right LaTeX commands to use svg[2]. On my tests
> > with PNG on my notebook, it also didn't seem to do the right thing for
> > PNG either. So, it seems that the only safe way to support images is
> > to convert all of them to PDF for latex/pdf build.
> >
> > [1] On Fedora, via texlive-dvipng and texlive-svg
> > [2] https://github.com/sphinx-doc/sphinx/issues/1907
> >
> > As far as I understand from KS, two decisions was taken:
> >
> > - We're not adding a sphinx extension to run generic commands;
> > - The PDF images should be build in runtime from their source files
> >   (either svg or bitmap), and not ship anymore the corresponding
> >   PDF files generated from its source.
> >
> > As you know, we use several images at the media documentation:
> > 	https://www.kernel.org/doc/html/latest/_images/
> >
> > Those images are tightly coupled with the explanation texts. So,
> > maintaining them away from the documentation is not an option.
> >
> > I was originally thinking that adding a graphviz extension would solve the
> > issue, but, in fact, most of the images aren't diagrams. Instead, there are 
> > several ones with images showing the result of passing certain parameters to
> > the ioctls, explaining things like scale and cropping and how bytes are
> > packed on some image formats.
> >
> > Linus proposed to call some image conversion tool like ImageMagick or
> > inkscape to convert them to PDF when building the pdfdocs or latexdocs
> > target at Makefile, but there's an issue with that: Sphinx doesn't read
> > files from Documentation/output, and writing them directly at the
> > source dir would be against what it is expected when the "O=" argument
> > is passed to make. 
> >
> > So, we have a few alternatives:
> >
> > 1) copy (or symlink) all rst files to Documentation/output (or to the
> >    build dir specified via O= directive) and generate the *.pdf there,
> >    and produce those converted images via Makefile.;
> >
> > 2) add an Sphinx extension that would internally call ImageMagick and/or
> >    inkscape to convert the bitmap;
> >
> > 3) if possible, add an extension to trick Sphinx for it to consider the 
> >    output dir as a source dir too.  
> 
> Looking at the available extensions, and the images to be displayed,
> seems to me making svg work, somehow, is the right approach. (As opposed
> to trying to represent the images in graphviz or whatnot.)

I guess answered this one already, but it got lost somehow...

The problem is not just with svg. Sphinx also do the wrong thing with
PNG, despite apparently generating the right LaTeX image include command.

> IIUC texlive supports displaying svg directly, but the problem is that
> Sphinx produces bad latex for that. Can we make it work by manually
> writing the latex? If yes, we wouldn't need to use an external tool to
> convert the svg to something else, but rather fix the latex. Thus:
> 
> 4a) See if this works:
> 
> .. only:: html
> 
>    .. image:: foo.svg

We're currently using .. figure:: instead, as it allow optional caption
and legend, but I got the idea.

> .. raw:: latex
> 
>    <the correct latex commands required to display foo.svg>

That is a horrible hack, and will lose other attributes at
image:: (or figure::), like :align:

Also, it won't solve, as the images will need to be copied to the
build dir via Makefile, as Spinx only copies the images it recognizes.

So, in practice, the only difference is that Makefile would be calling
"cp" instead of "convert", plus we'll have to hack all ReST sources.

> 4b) Add a directive extension to make the above happen automatically.

If doable, I agree that this is the best solution. Any volunteers to write
such extension?

> Of course, the correct fix is to have this fixed in upstream Sphinx, but
> as a workaround an extension doing the above seems plausible, and not
> too much effort - provided that we can make the raw latex work.

Yeah, fixing it on Sphinx upstream would be the best, but we'll still
need to maintain the workaround for a while for the unpatched versions
of Sphinx.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1516354 — Re: [Ksummit-discuss] Including images on Sphinx documents

On Mon, Nov 07, 2016 at 07:55:24AM -0200, Mauro Carvalho Chehab wrote:
> 2) add an Sphinx extension that would internally call ImageMagick and/or
>    inkscape to convert the bitmap;

This seems sensible; Sphinx should directly handle the source format we
want to use for images/diagrams.

> 3) if possible, add an extension to trick Sphinx for it to consider the 
>    output dir as a source dir too.

Or to provide an additional source path and point that at the output
directory.

[toc] | [prev] | [next] | [standalone]

#1517849 — Re: [Ksummit-discuss] Including images on Sphinx documents

Am 07.11.2016 um 18:01 schrieb Josh Triplett <josh@joshtriplett.org>:

> On Mon, Nov 07, 2016 at 07:55:24AM -0200, Mauro Carvalho Chehab wrote:
>> 2) add an Sphinx extension that would internally call ImageMagick and/or
>>  inkscape to convert the bitmap;
> 
> This seems sensible; Sphinx should directly handle the source format we
> want to use for images/diagrams.
> 
>> 3) if possible, add an extension to trick Sphinx for it to consider the 
>>  output dir as a source dir too.
> 
> Or to provide an additional source path and point that at the output
> directory.

The sphinx-build command excepts only one 'sourcedir' argument. All
reST files in this folder (and below) are parsed.

Most (all?) directives which include content like images or literalinclude
except only relative pathnames. Where *relative* means, relative to the
reST file where the directive is used. For security reasons relative 
pathnames outside 'sourcepath' are not excepted.

So I vote for :

> 1) copy (or symlink) all rst files to Documentation/output (or to the
>  build dir specified via O= directive) and generate the *.pdf there,
>  and produce those converted images via Makefile.;

Placing reST files together with the *autogenerated* (intermediate) 
content from

* image conversions,
* reST content build from MAINTAINERS,
* reST content build for ABI
* etc.

has the nice side effect, that we can get rid of all theses BUILDDIR
quirks in the Makefile.sphinx

Additional, we can write Makefile targets to build the above listed
intermediate content relative to the $PWD, which is what Linux's
Makefiles usual do (instead of quirking with a BUILDDIR).

E.g. with, we can also get rid of the 'kernel-include' directive 
and replace it, with Sphinx's common 'literaliclude' and we do not
need any extensions to include intermediate PDFs or whatever
intermediate content we might want to generate. 

IMO placing 'sourcedir' to O= is more sane since this marries the
Linux Makefile concept (relative to $PWD) with the sphinx concept
(in or below 'sourcedir').


-- Markus --

[toc] | [prev] | [next] | [standalone]

#1518006 — Re: [Ksummit-discuss] Including images on Sphinx documents

On Wed, 09 Nov 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
> Am 07.11.2016 um 18:01 schrieb Josh Triplett <josh@joshtriplett.org>:
>
>> On Mon, Nov 07, 2016 at 07:55:24AM -0200, Mauro Carvalho Chehab wrote:
>>> 2) add an Sphinx extension that would internally call ImageMagick and/or
>>>  inkscape to convert the bitmap;
>> 
>> This seems sensible; Sphinx should directly handle the source format we
>> want to use for images/diagrams.
>> 
>>> 3) if possible, add an extension to trick Sphinx for it to consider the 
>>>  output dir as a source dir too.
>> 
>> Or to provide an additional source path and point that at the output
>> directory.
>
> The sphinx-build command excepts only one 'sourcedir' argument. All
> reST files in this folder (and below) are parsed.
>
> Most (all?) directives which include content like images or literalinclude
> except only relative pathnames. Where *relative* means, relative to the
> reST file where the directive is used. For security reasons relative 
> pathnames outside 'sourcepath' are not excepted.
>
> So I vote for :
>
>> 1) copy (or symlink) all rst files to Documentation/output (or to the
>>  build dir specified via O= directive) and generate the *.pdf there,
>>  and produce those converted images via Makefile.;

We're supposed to solve problems, not create new ones.

> Placing reST files together with the *autogenerated* (intermediate) 
> content from
>
> * image conversions,
> * reST content build from MAINTAINERS,
> * reST content build for ABI
> * etc.
>
> has the nice side effect, that we can get rid of all theses BUILDDIR
> quirks in the Makefile.sphinx
>
> Additional, we can write Makefile targets to build the above listed
> intermediate content relative to the $PWD, which is what Linux's
> Makefiles usual do (instead of quirking with a BUILDDIR).
>
> E.g. with, we can also get rid of the 'kernel-include' directive 
> and replace it, with Sphinx's common 'literaliclude' and we do not
> need any extensions to include intermediate PDFs or whatever
> intermediate content we might want to generate. 

Well, kernel-include is a hack to make parse-headers.pl work, which is
also a hack that IMHO shouldn't exist...

> IMO placing 'sourcedir' to O= is more sane since this marries the
> Linux Makefile concept (relative to $PWD) with the sphinx concept
> (in or below 'sourcedir').
>
>
> -- Markus --
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-doc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1518017 — Re: [Ksummit-discuss] Including images on Sphinx documents

Am 09.11.2016 um 12:16 schrieb Jani Nikula <jani.nikula@linux.intel.com>:
>> So I vote for :
>> 
>>> 1) copy (or symlink) all rst files to Documentation/output (or to the
>>> build dir specified via O= directive) and generate the *.pdf there,
>>> and produce those converted images via Makefile.;
> 
> We're supposed to solve problems, not create new ones.

... new ones? ...

>> IMO placing 'sourcedir' to O= is more sane since this marries the
>> Linux Makefile concept (relative to $PWD) with the sphinx concept
>> (in or below 'sourcedir').

-- Markus --

[toc] | [prev] | [next] | [standalone]

#1518101 — Re: [Ksummit-discuss] Including images on Sphinx documents

On Wed, 09 Nov 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
> Am 09.11.2016 um 12:16 schrieb Jani Nikula <jani.nikula@linux.intel.com>:
>>> So I vote for :
>>> 
>>>> 1) copy (or symlink) all rst files to Documentation/output (or to the
>>>> build dir specified via O= directive) and generate the *.pdf there,
>>>> and produce those converted images via Makefile.;
>> 
>> We're supposed to solve problems, not create new ones.
>
> ... new ones? ...

Handle in-tree builds without copying.

Make dependency analysis with source rst and "intermediate" rst work.

Make sure your copying gets the timestamps right.

Make Sphinx dependency analysis look at the right copies depending on
in-tree vs. out-of-tree. Generally make sure it doesn't confuse Sphinx's
own dependency analysis.

The stuff I didn't think of.

Sure, it's all supposed to be basic Makefile stuff, but don't make the
mistake of thinking just one invocation of 'cp' will solve all the
problems. It all adds to the complexity we were trying to avoid when
dumping DocBook. It adds to the complexity of debugging stuff. (And hey,
there's still the one rebuilding-stuff-for-no-reason issue open.)

If you want to keep the documentation build sane, try to avoid the
Makefile preprocessing. And same old story, if you fix this for real,
even if as a Sphinx extension, *other* people than kernel developers
will be interested, and *we* don't have to do so much ourselves.


BR,
Jani.



>
>>> IMO placing 'sourcedir' to O= is more sane since this marries the
>>> Linux Makefile concept (relative to $PWD) with the sphinx concept
>>> (in or below 'sourcedir').
>
> -- Markus --

-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1518520 — Re: [Ksummit-discuss] Including images on Sphinx documents

On 09.11.2016 12:58, Jani Nikula wrote:
 > On Wed, 09 Nov 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
 >> Am 09.11.2016 um 12:16 schrieb Jani Nikula <jani.nikula@linux.intel.com>:
 >>>> So I vote for :
 >>>>
 >>>>> 1) copy (or symlink) all rst files to Documentation/output (or to the
 >>>>> build dir specified via O= directive) and generate the *.pdf there,
 >>>>> and produce those converted images via Makefile.;
 >>>
 >>> We're supposed to solve problems, not create new ones.
 >>
 >> ... new ones? ...
 >
 > Handle in-tree builds without copying.
 >
 > Make dependency analysis with source rst and "intermediate" rst work.
 >
 > Make sure your copying gets the timestamps right.
 >
 > Make Sphinx dependency analysis look at the right copies depending on
 > in-tree vs. out-of-tree. Generally make sure it doesn't confuse Sphinx's
 > own dependency analysis.
 >
 > The stuff I didn't think of.

It might be easier than you think first.

 > Sure, it's all supposed to be basic Makefile stuff, but don't make the mistake
 > of thinking just one invocation of 'cp' will solve all the problems.

I act naif using 'cp -sa', see patch below.

 > It all adds to the complexity we were trying to avoid when dumping DocBook. It
 > adds to the complexity of debugging stuff. (And hey, there's still the one
 > rebuilding-stuff-for-no-reason issue open.)

And hey ;-) I wrote you [1], this is a bug in Sphinx. Yes, I haven't had time
to send a bugfix to Sphinx, but this won't even help us (bugfixes in Sphinx will
only apply on top).

 > If you want to keep the documentation build sane, try to avoid the Makefile
 > preprocessing.

I'am just the one helping Mauro to be productive, if he needs preprocessing I
implement proposals. I know that you fear preprocessing since it tend to
fall-back, what we had with DocBook's build process.  We discussed this already,
it might better you unify this with Mauro and the other who need preprocessing.

 > And same old story, if you fix this for real, even if as a Sphinx extension,
 > *other* people than kernel developers will be interested, and *we* don't have
 > to do so much ourselves.

I don't think so, this kind of parsing header files we have and the build of
content from MAINTAINERS, ABI, etc. is very kernel specific.

Anyway, back to my point 'copy (or symlink) all rst files'. Please take a look
at my patch below. Take in mind; its just a POC.

Could this POC persuade you, if so, I send a more elaborate RFC,
what do you think about?

[1] https://www.mail-archive.com/linux-doc@vger.kernel.org/msg07302.html

-- Markus --

 > BR,
 > Jani.
 >>
 >>>> IMO placing 'sourcedir' to O= is more sane since this marries the
 >>>> Linux Makefile concept (relative to $PWD) with the sphinx concept
 >>>> (in or below 'sourcedir').
 >>
 >> -- Markus --
 >

diff --git a/Documentation/Makefile.sphinx b/Documentation/Makefile.sphinx
index ec0c77d..8e904c1 100644
--- a/Documentation/Makefile.sphinx
+++ b/Documentation/Makefile.sphinx
@@ -13,6 +13,10 @@ BUILDDIR      = $(obj)/output
  PDFLATEX      = xelatex
  LATEXOPTS     = -interaction=batchmode

+ifdef SPHINXDIRS
+else
+endif
+
  # User-friendly check for sphinx-build
  HAVE_SPHINX := $(shell if which $(SPHINXBUILD) >/dev/null 2>&1; then echo 1; else echo 0; fi)

@@ -50,30 +54,38 @@ loop_cmd = $(echo-cmd) $(cmd_$(1))
  #    * dest folder relative to $(BUILDDIR) and
  #    * cache folder relative to $(BUILDDIR)/.doctrees
  # $4 dest subfolder e.g. "man" for man pages at media/man
-# $5 reST source folder relative to $(srctree)/$(src),
+# $5 reST source folder relative to $(obj),
  #    e.g. "media" for the linux-tv book-set at ./Documentation/media

  quiet_cmd_sphinx = SPHINX  $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
-      cmd_sphinx = $(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/media all;\
-	BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(srctree)/$(src)/$5/$(SPHINX_CONF)) \
+      cmd_sphinx = $(MAKE) BUILDDIR=$(BUILDDIR) $(build)=Documentation/media all;\
+	BUILDDIR=$(BUILDDIR) SPHINX_CONF=$(obj)/$5/$(SPHINX_CONF) \
  	$(SPHINXBUILD) \
  	-b $2 \
-	-c $(abspath $(srctree)/$(src)) \
-	-d $(abspath $(BUILDDIR)/.doctrees/$3) \
+	-c $(obj) \
+	-d $(obj)/.doctrees/$3 \
  	-D version=$(KERNELVERSION) -D release=$(KERNELRELEASE) \
  	$(ALLSPHINXOPTS) \
-	$(abspath $(srctree)/$(src)/$5) \
-	$(abspath $(BUILDDIR)/$3/$4);
-
-htmldocs:
+	$(obj)/$5 \
+	$(BUILDDIR)/$3/$4;
+
+ifdef O
+sync:
+	rm -rf $(objtree)/$(obj)
+	cp -sa $(srctree)/$(obj) $(objtree)
+else
+sync:
+endif
+
+htmldocs: sync
  	@$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))

-latexdocs:
+latexdocs: sync
  	@$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$(var)))

  ifeq ($(HAVE_PDFLATEX),0)

-pdfdocs:
+pdfdocs: sync
  	$(warning The '$(PDFLATEX)' command was not found. Make sure you have it installed and in PATH to produce PDF output.)
  	@echo "  SKIP    Sphinx $@ target."

@@ -84,10 +96,10 @@ pdfdocs: latexdocs

  endif # HAVE_PDFLATEX

-epubdocs:
+epubdocs: sync
  	@$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))

-xmldocs:
+xmldocs: sync
  	@$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var)))

  # no-ops for the Sphinx toolchain
@@ -98,6 +110,7 @@ installmandocs:

  cleandocs:
  	$(Q)rm -rf $(BUILDDIR)
+	$(Q)rm -rf $(obj)/.doctrees

  endif # HAVE_SPHINX

[toc] | [prev] | [next] | [standalone]

#1518841 — Re: [Ksummit-discuss] Including images on Sphinx documents

On Thu, 10 Nov 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
> Could this POC persuade you, if so, I send a more elaborate RFC,
> what do you think about?

Sorry, I do not wish to be part of this.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1519706 — Re: [Ksummit-discuss] Including images on Sphinx documents

On Thu, 10 Nov 2016, Jani Nikula <jani.nikula@linux.intel.com> wrote:
> On Thu, 10 Nov 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
>> Could this POC persuade you, if so, I send a more elaborate RFC,
>> what do you think about?
>
> Sorry, I do not wish to be part of this.

That was uncalled for, apologies.

Like I said, I don't think this is the right approach. Call it an
unsubstantiated gut feel coming from experience. However, I do not have
the time to properly dig into this either, and that frustrates me. I
wish I could be more helpful, but I can't right now.

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1519722 — Re: [Ksummit-discuss] Including images on Sphinx documents

Am 11.11.2016 um 12:22 schrieb Jani Nikula <jani.nikula@linux.intel.com>:

> On Thu, 10 Nov 2016, Jani Nikula <jani.nikula@linux.intel.com> wrote:
>> On Thu, 10 Nov 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
>>> Could this POC persuade you, if so, I send a more elaborate RFC,
>>> what do you think about?
>> 
>> Sorry, I do not wish to be part of this.
> 
> That was uncalled for, apologies.

It's OK, sometimes we are all in a hurry and want shorten things.

> Like I said, I don't think this is the right approach. Call it an
> unsubstantiated gut feel coming from experience.

Yes, building a bunch of symbolic links "smells". Unfortunately,
I currently see no other solution to solve the conflict of 
Linux's "O=/foo" and Sphinx's "sourcedir", so that was my
proposal.

> However, I do not have
> the time to properly dig into this either, and that frustrates me. I
> wish I could be more helpful, but I can't right now.

its a pity

-- Markus --

[toc] | [prev] | [next] | [standalone]

#1519641 — Re: [Ksummit-discuss] Including images on Sphinx documents

Em Wed, 09 Nov 2016 13:58:12 +0200
Jani Nikula <jani.nikula@linux.intel.com> escreveu:

> On Wed, 09 Nov 2016, Markus Heiser <markus.heiser@darmarit.de> wrote:
> > Am 09.11.2016 um 12:16 schrieb Jani Nikula <jani.nikula@linux.intel.com>:  
> >>> So I vote for :
> >>>   
> >>>> 1) copy (or symlink) all rst files to Documentation/output (or to the
> >>>> build dir specified via O= directive) and generate the *.pdf there,
> >>>> and produce those converted images via Makefile.;  
> >> 
> >> We're supposed to solve problems, not create new ones.  
> >
> > ... new ones? ...  
> 
> Handle in-tree builds without copying.
> 
> Make dependency analysis with source rst and "intermediate" rst work.
> 
> Make sure your copying gets the timestamps right.
> 
> Make Sphinx dependency analysis look at the right copies depending on
> in-tree vs. out-of-tree. Generally make sure it doesn't confuse Sphinx's
> own dependency analysis.

I agree with Jani here: copy the files will make Sphinx recompile
the entire documentation every time, with is bad. Ok, Some Makefile
logic could be added to copy only on changes, but that will increase
the Makefile complexity.

So, I prefer not using copy. As I said before, a Sphinx extension that
would make transparent for PDF document generation when a non-PDF image
is included, doing whatever conversion needed, seems to be the right fix 
here, but someone would need to step up and write such extension.

-- 

Cheers,
Mauro

[toc] | [prev] | [next] | [standalone]

#1518024 — Re: [Ksummit-discuss] Including images on Sphinx documents

Em Wed, 09 Nov 2016 13:16:55 +0200
Jani Nikula <jani.nikula@linux.intel.com> escreveu:

> >> 1) copy (or symlink) all rst files to Documentation/output (or to the
> >>  build dir specified via O= directive) and generate the *.pdf there,
> >>  and produce those converted images via Makefile.;  
> 
> We're supposed to solve problems, not create new ones.

So, what's your proposal?

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1518091 — Re: [Ksummit-discuss] Including images on Sphinx documents

On Wed, 09 Nov 2016, Mauro Carvalho Chehab <mchehab@s-opensource.com> wrote:
> Em Wed, 09 Nov 2016 13:16:55 +0200
> Jani Nikula <jani.nikula@linux.intel.com> escreveu:
>
>> >> 1) copy (or symlink) all rst files to Documentation/output (or to the
>> >>  build dir specified via O= directive) and generate the *.pdf there,
>> >>  and produce those converted images via Makefile.;  
>> 
>> We're supposed to solve problems, not create new ones.
>
> So, what's your proposal?

Second message in the thread,
http://lkml.kernel.org/r/87wpgf8ssc.fsf@intel.com

>
> Thanks,
> Mauro
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web