Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1461504 > unrolled thread

Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf  build Jonathan Corbet <corbet@lwn.net> - 2016-08-13 00:50 +0200
    Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build Markus Heiser <markus.heiser@darmarit.de> - 2016-08-13 18:10 +0200
    Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build Jani Nikula <jani.nikula@intel.com> - 2016-08-15 11:50 +0200
      Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf  build Mauro Carvalho Chehab <mchehab@infradead.org> - 2016-08-15 14:20 +0200
        Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf  build Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-08-16 00:30 +0200
          Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build Jani Nikula <jani.nikula@intel.com> - 2016-08-19 15:00 +0200
            Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf  build Mauro Carvalho Chehab <mchehab@osg.samsung.com> - 2016-08-19 15:30 +0200

#1461504 — Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build

On Wed, 10 Aug 2016 18:54:06 +0300
Jani Nikula <jani.nikula@intel.com> wrote:

> With these you should be able to get started with pdf generation. It's a
> quick transition to pdflatex, the patches are not very pretty, but the
> pdf output is. Patch 3/3 works as an example where to add your stuff
> (latex_documents in conf.py) and how.

OK, now I have a bone to pick with you.

I applied this, then decided to install the needed toolchain on the
Tumbleweed system I've been playing with; it wanted to install 1,727
packages to get pdflatex.  Pandoc just doesn't seem so bad anymore.

So I switched to the Fedora system, and found myself in a twisty maze of
missing font files, missing style files, missing babel crap, etc., each
doled out to me one file per run.  But I did eventually get PDFs out of
it.

The output isn't great; among other things, it seems to be about 1/2 blank
pages.  But it's something.

I've applied this so we have something to play with, but it doesn't feel
like a great solution.  This is the sort of installation hell that we
wanted to get away from.  It makes me wonder how hard it can really be to
fix rst2pdf; I wish I could say I'll find some time to figure that out.
Sigh.

jon

[toc] | [next] | [standalone]

#1461635 — Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build

Am 13.08.2016 um 00:40 schrieb Jonathan Corbet <corbet@lwn.net>:

> On Wed, 10 Aug 2016 18:54:06 +0300
> Jani Nikula <jani.nikula@intel.com> wrote:
> 
>> With these you should be able to get started with pdf generation. It's a
>> quick transition to pdflatex, the patches are not very pretty, but the
>> pdf output is. Patch 3/3 works as an example where to add your stuff
>> (latex_documents in conf.py) and how.
> 
> OK, now I have a bone to pick with you.
> 
> I applied this, then decided to install the needed toolchain on the
> Tumbleweed system I've been playing with; it wanted to install 1,727
> packages to get pdflatex.  Pandoc just doesn't seem so bad anymore.

I'am complete disenchanted on this topic. My experience is:

1) You wan't get any reasonable typesetting engine which preserves
your disk space. I don't know how many files or packages are installed,
the only thing I know is, a TeX installation is always >1GB.

2) You wan't get a (pdf, ps,..) book with a perfect layout without
any handcraft or at least a *theming*. TeX has many options to influence
the layout and Sphinx provides it's own LaTeX-document class (sphinxmanual)
which is IMHO awful.

> So I switched to the Fedora system, and found myself in a twisty maze of
> missing font files, missing style files, missing babel crap, etc., each
> doled out to me one file per run.  But I did eventually get PDFs out of
> it.

On debian it should be enough to install *base* and *recommended*

    sudo apt-get install
          texlive-base texlive-latex-recommended

> The output isn't great; among other things, it seems to be about 1/2 blank
> pages.

1/2 ? .. I have only empty pages at the start of parts or chapters, which
is a typical layout setting.

>  But it's something.

This is the sphinxmanual document class.
 
> I've applied this so we have something to play with, but it doesn't feel
> like a great solution.  This is the sort of installation hell that we
> wanted to get away from.  

See above, on debian it should be enough to install the two meta packages.

> It makes me wonder how hard it can really be to
> fix rst2pdf; I wish I could say I'll find some time to figure that out.
> Sigh.

I gave it a try, but as I come closer to the sources I realized that
it is hair-raising. I looked at the issues, added a comment to a related
issue, a few days later the issue was closed without any comment or code
change.

https://github.com/rst2pdf/rst2pdf/issues/556#issuecomment-228779542

My advice, if you don't like to waste your time: forget it.

Some thoughts of mine, wrote in an earlier mail:

> The sphinx-doc build-in LaTeX builder
> 
> * http://www.sphinx-doc.org/en/stable/config.html#options-for-latex-output
> 
> has some drawbacks, e.g. it produce LaTeX for the pdfTeX engine.
> LaTeX is by default ASCII and it needs some "inputenc" to supporta wider
> range of characters. This is not very helpful if you have a toolchain
> in an international community.
> 
> The alternative to LaTeX is to use the XeTeX engine, which supports UTF-8
> encoded input by default and supports TrueType/OpenType fonts directly.
> Thats why I started to write a XeLaTeX builder ...
> 
> * https://github.com/return42/sphkerneldoc/blob/master/scripts/site-python/xelatex_ext/__init__.py#L15 
> 
> ... but I can't predict when this will be finished ...
> 
> However which tool is used, my experience is, that building
> PDF (books) with a minimum of quality is not simple.
> Layout width tables, split table content over pages, switch
> from landscape to portrait and versus, the flow of objects etc.
> .. all this will need some manually interventions.


-- Markus --

[toc] | [prev] | [next] | [standalone]

#1462678 — Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build

On Sat, 13 Aug 2016, Jonathan Corbet <corbet@lwn.net> wrote:
> On Wed, 10 Aug 2016 18:54:06 +0300
> Jani Nikula <jani.nikula@intel.com> wrote:
>
>> With these you should be able to get started with pdf generation. It's a
>> quick transition to pdflatex, the patches are not very pretty, but the
>> pdf output is. Patch 3/3 works as an example where to add your stuff
>> (latex_documents in conf.py) and how.
>
> OK, now I have a bone to pick with you.
>
> I applied this, then decided to install the needed toolchain on the
> Tumbleweed system I've been playing with; it wanted to install 1,727
> packages to get pdflatex.  Pandoc just doesn't seem so bad anymore.

Jon, I sent these to unblock Luis, and as a starting point for a
discussion about rst2pdf vs. pdflatex. I didn't mean I'd want these
merged as-is! I'm sorry if I didn't make myself clear.

I don't mind at all if you want to drop them.

As I played around, it worked for *me* better than rst2pdf. I had
pdflatex already installed, so I didn't pay attention to the deps, and
the numbers you quote come as a suprise.

> I've applied this so we have something to play with, but it doesn't feel
> like a great solution.  This is the sort of installation hell that we
> wanted to get away from.  It makes me wonder how hard it can really be to
> fix rst2pdf; I wish I could say I'll find some time to figure that out.
> Sigh.

I'm afraid pdf generation is not very high on my list of priorities
right now. Surely I'll get back to this eventually, but not any time
soon. :(

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1462744

Em Mon, 15 Aug 2016 12:40:21 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Sat, 13 Aug 2016, Jonathan Corbet <corbet@lwn.net> wrote:
> > On Wed, 10 Aug 2016 18:54:06 +0300
> > Jani Nikula <jani.nikula@intel.com> wrote:
> >  
> >> With these you should be able to get started with pdf generation. It's a
> >> quick transition to pdflatex, the patches are not very pretty, but the
> >> pdf output is. Patch 3/3 works as an example where to add your stuff
> >> (latex_documents in conf.py) and how.  
> >
> > OK, now I have a bone to pick with you.
> >
> > I applied this, then decided to install the needed toolchain on the
> > Tumbleweed system I've been playing with; it wanted to install 1,727
> > packages to get pdflatex.  Pandoc just doesn't seem so bad anymore.  
> 
> Jon, I sent these to unblock Luis, and as a starting point for a
> discussion about rst2pdf vs. pdflatex. I didn't mean I'd want these
> merged as-is! I'm sorry if I didn't make myself clear.
> 
> I don't mind at all if you want to drop them.
> 
> As I played around, it worked for *me* better than rst2pdf. I had
> pdflatex already installed, so I didn't pay attention to the deps, and
> the numbers you quote come as a suprise.
> 
> > I've applied this so we have something to play with, but it doesn't feel
> > like a great solution.  This is the sort of installation hell that we
> > wanted to get away from.  It makes me wonder how hard it can really be to
> > fix rst2pdf; I wish I could say I'll find some time to figure that out.
> > Sigh.  
> 
> I'm afraid pdf generation is not very high on my list of priorities
> right now. Surely I'll get back to this eventually, but not any time
> soon. :(

I'm afraid that, no matter if we'll be using rst2pdf or pdflatex,
we'll need LaTex anyway.

One of the things that we want on media is to better represent the
math formulas there. Just enabling the math extension is enough for
it to require not only LaTex, but also this package (on Fedora):
	texlive-anyfontsize.noarch

So, unfortunately, I'm afraid that we'll end by having a much worse
dependency chain than what we used to have with DocBook.

An (incomplete) patch to let you test it is enclosed (it is actually after
patches 1 and 2 of this series).

Ah, it seems that rst2pdf is not compatible with math extension...

If we change the conf.py to:
	extensions = ['sphinx.ext.imgmath', 'kernel-doc', 'rstFlatTable', 'kernel_include']

	# Gracefully handle missing rst2pdf.
	try:
	    import rst2pdf
	    extensions += ['rst2pdf.pdfbuilder']
	except ImportError:
	    pass

The logic that tries to import rst2pdf produces the following error:

	Extension error:
	Config value 'math_number_all' already present

And, if we don't add the imgmath extension, the math expression
appears broken at the generated html page:

	The transfer function defined for SMPTE 170M is the same as the one defined in Rec. 709.
	L' = -1.099(-L)^{0.45} + 0.099 \text{ for } L \le-0.018 L' = 4.5L \text{ for } -0.018 < L < 0.018 L' = 1.099L^{0.45} - 0.099 \text{ for } L \ge 0.018

Btw, changing to math is a requirement for pdflatex to work on media,
as the font used by Sphinx/pdflatex doesn't have UTF-8 symbols used
on math. So, trying to build the documentation with pdfdocs (after
patches 1 and 2, before patch 3) would give fatal errors like:

	! Package inputenc Error: Unicode char \u8:≥ not set up for use with LaTeX.

	See the inputenc package documentation for explanation.
	Type  H <return>  for immediate help.
	 ...                                              
                                                  
	l.7759 ....099L $^{\text{0.45}}$ - 0.099 for L ≥
							 0.018

The problem is that the "grater or equal than" UTF-8 Math symbol
is not supported on the LaTex font.


Thanks,
Mauro


diff --git a/Documentation/conf.py b/Documentation/conf.py
index 96b7aa66c89c..0b60a55cea89 100644
--- a/Documentation/conf.py
+++ b/Documentation/conf.py
@@ -28,14 +28,14 @@ sys.path.insert(0, os.path.abspath('sphinx'))
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['kernel-doc', 'rstFlatTable', 'kernel_include']
+extensions = ['sphinx.ext.imgmath', 'kernel-doc', 'rstFlatTable', 'kernel_include']
 
 # Gracefully handle missing rst2pdf.
-try:
-    import rst2pdf
-    extensions += ['rst2pdf.pdfbuilder']
-except ImportError:
-    pass
+#try:
+#    import rst2pdf
+#    extensions += ['rst2pdf.pdfbuilder']
+#except ImportError:
+#    pass
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
diff --git a/Documentation/media/uapi/v4l/pixfmt-007.rst b/Documentation/media/uapi/v4l/pixfmt-007.rst
index 8c946b0c63a0..1253c3283b10 100644
--- a/Documentation/media/uapi/v4l/pixfmt-007.rst
+++ b/Documentation/media/uapi/v4l/pixfmt-007.rst
@@ -72,11 +72,13 @@ SMPTE C set, so this colorspace is sometimes called SMPTE C as well.
 The transfer function defined for SMPTE 170M is the same as the one
 defined in Rec. 709.
 
-    L' = -1.099(-L) :sup:`0.45` + 0.099 for L ≤ -0.018
+.. math::
 
-    L' = 4.5L for -0.018 < L < 0.018
+    L' = -1.099(-L)^{0.45} + 0.099 \text{ for } L \le-0.018
 
-    L' = 1.099L :sup:`0.45` - 0.099 for L ≥ 0.018
+    L' = 4.5L \text{ for } -0.018 < L < 0.018
+
+    L' = 1.099L^{0.45} - 0.099 \text{ for } L \ge 0.018
 
 Inverse Transfer function:
 

[toc] | [prev] | [next] | [standalone]

#1463228

Hi Jon,

Em Mon, 15 Aug 2016 09:17:52 -0300
Mauro Carvalho Chehab <mchehab@infradead.org> escreveu:

> Em Mon, 15 Aug 2016 12:40:21 +0300
> Jani Nikula <jani.nikula@intel.com> escreveu:
> 
> > On Sat, 13 Aug 2016, Jonathan Corbet <corbet@lwn.net> wrote:  
> > > On Wed, 10 Aug 2016 18:54:06 +0300
> > > Jani Nikula <jani.nikula@intel.com> wrote:
> > >    
> > >> With these you should be able to get started with pdf generation. It's a
> > >> quick transition to pdflatex, the patches are not very pretty, but the
> > >> pdf output is. Patch 3/3 works as an example where to add your stuff
> > >> (latex_documents in conf.py) and how.    
> > >
> > > OK, now I have a bone to pick with you.
> > >
> > > I applied this, then decided to install the needed toolchain on the
> > > Tumbleweed system I've been playing with; it wanted to install 1,727
> > > packages to get pdflatex.  Pandoc just doesn't seem so bad anymore.    
> > 
> > Jon, I sent these to unblock Luis, and as a starting point for a
> > discussion about rst2pdf vs. pdflatex. I didn't mean I'd want these
> > merged as-is! I'm sorry if I didn't make myself clear.
> > 
> > I don't mind at all if you want to drop them.

I played for a while with rst2pdf and LaTeX-based tools, plus
using Sphinx math extension to improve the media documentation.

Here's my findings:

1) I'm pretty sure there's no way for us to get rid of LaTex.

The problem is that the Sphinx math extension depends on LaTeX amsmath
extension. Ok, someone could write some extension in the future to add math
support without requiring LaTeX and fix rst2pdf (or rewrite it), but this
would take time and efforts.

So, for now, I think we should just assume that anyone wanting to generate
pdf docs will need LaTeX (or, more likely, TeTeX).

2) rst2pdf seems to be incompatible with Sphinx math extension,
at least on Sphinx 1.4.5;

3) pdflatex doesn't handle UTF-8 chars. This is a problem for media,
as we use two UTF-8 symbols that are incompatible with the fonts
used by pdflatex:

	- ≥ U+2265 (GREATER-THAN OR EQUAL TO) utf-8 character
	- ≤ u+2264 (LESS-THAN OR EQUAL TO) utf-8 character

4) pdflatex output is not nice: all black and white; ugly (IMHO) fonts;

5) At media docs, some tables will only print ok in landscape.

After making the media books build, I think that the best way is to
use xelatex instead of pdfdocs. Visually, xelatex output is, IMHO,
nice - and it has colors :-)

It seems that there's yet another option: lualatex. I didn't try to
build with it. So, not sure if its output is better or not, nor if
some extra config for it is needed at conf.py.

I sent a patch series addressing most of the issues above.

If you want to see the output:
	https://mchehab.fedorapeople.org/TheLinuxKernel.pdf

This was built on Fedora 24.

Issues:
-------

Even with xelatex, there are still some issues to be addressed:

- There are two hacks to build media: one removes the *.h.rst files
  and the other one comments out two C code blocks inside tables;

- There are lots of noise at PDF output. I didn't even try to look
  into them;

- Almost all tables at the media books are mangled. Rotating them to
  landscape can fix several of them. I added support for it, however,
  that requires to manually add LaTeX tags before and after the tables,
  like:

    .. raw:: latex
    
        \begin{landscape}
    
    <some table(s)>
    
    .. raw:: latex
    
        \end{landscape}

As the above tags are LaTeX specific, they should not interfere on 
html (or e-pub) output.

One such example is this table:
	"Table 2.18: Packed RGB Image Formats"
	(at page 92 of the PDF file)

I'm seeking for a solution to scale it and rotate (as just rotating it
is not enough).

-- 
Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1466364 — Re: [RFC PATCH 0/3] Documentation: switch to pdflatex and fix pdf build

On Tue, 16 Aug 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
> 5) At media docs, some tables will only print ok in landscape.
>
> After making the media books build, I think that the best way is to
> use xelatex instead of pdfdocs. Visually, xelatex output is, IMHO,
> nice - and it has colors :-)
>
> It seems that there's yet another option: lualatex. I didn't try to
> build with it. So, not sure if its output is better or not, nor if
> some extra config for it is needed at conf.py.
>
> I sent a patch series addressing most of the issues above.

I didn't see the series, but I'm surprised we now have HAVE_PDFLATEX
that actually checks for 'xelatex'. The commit subject doing the change
doesn't even mention xelatex, it's just about UTF-8:

commit a682ec4ba10c88231cdbb8bb9823b2cc749d6364
Author: Mauro Carvalho Chehab <mchehab@s-opensource.com>
Date:   Tue Aug 16 13:25:39 2016 -0300

    docs-rst: Don't mangle with UTF-8 chars on LaTeX/PDF output

</bikeshedding after the fact>

BR,
Jani.


-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1466397

Em Fri, 19 Aug 2016 15:54:37 +0300
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Tue, 16 Aug 2016, Mauro Carvalho Chehab <mchehab@osg.samsung.com> wrote:
> > 5) At media docs, some tables will only print ok in landscape.
> >
> > After making the media books build, I think that the best way is to
> > use xelatex instead of pdfdocs. Visually, xelatex output is, IMHO,
> > nice - and it has colors :-)
> >
> > It seems that there's yet another option: lualatex. I didn't try to
> > build with it. So, not sure if its output is better or not, nor if
> > some extra config for it is needed at conf.py.
> >
> > I sent a patch series addressing most of the issues above.  
> 
> I didn't see the series, but I'm surprised we now have HAVE_PDFLATEX
> that actually checks for 'xelatex'. The commit subject doing the change
> doesn't even mention xelatex, it's just about UTF-8:
> 
> commit a682ec4ba10c88231cdbb8bb9823b2cc749d6364
> Author: Mauro Carvalho Chehab <mchehab@s-opensource.com>
> Date:   Tue Aug 16 13:25:39 2016 -0300
> 
>     docs-rst: Don't mangle with UTF-8 chars on LaTeX/PDF output
> 
> </bikeshedding after the fact>

It was mentioned at patch 0/9:
	https://www.mail-archive.com/linux-media@vger.kernel.org/msg101408.html

I opted to not change the HAVE_PDFLATEX, as the macro that sets the
latex variant on Sphinx latex Makefile is called "PDFLATEX". Also,
we might change it in the future to use some other tool or even add
some logic to allow supporting multiple tools, like we have with
DocBook.

The main goal of XeTeX is to add unicode font support to LaTex,
according to:
	https://en.wikipedia.org/wiki/XeTeX

We need that, as media uses a lot of math Unicode symbols inside
the documentation.

-- 
Thanks,
Mauro

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web