Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1523637 > unrolled thread

Re: [Ksummit-discuss] Including images on Sphinx documents

Back to article view | Back to linux.kernel

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: [Ksummit-discuss] Including images on Sphinx documents Arnd Bergmann <arnd@arndb.de> - 2016-11-16 17:10 +0100
    Re: [Ksummit-discuss] Including images on Sphinx documents Mauro Carvalho Chehab <mchehab@infradead.org> - 2016-11-16 21:30 +0100
      Re: [Ksummit-discuss] Including images on Sphinx documents Arnd Bergmann <arnd@arndb.de> - 2016-11-17 12:10 +0100
        Re: [Ksummit-discuss] Including images on Sphinx documents Jani Nikula <jani.nikula@intel.com> - 2016-11-17 12:30 +0100
          Re: [Ksummit-discuss] Including images on Sphinx documents Mauro Carvalho Chehab <mchehab@infradead.org> - 2016-11-17 18:10 +0100
        Re: [Ksummit-discuss] Including images on Sphinx documents Mauro Carvalho Chehab <mchehab@infradead.org> - 2016-11-17 18:10 +0100
        Re: [Ksummit-discuss] Including images on Sphinx documents Linus Torvalds <torvalds@linux-foundation.org> - 2016-11-17 18:10 +0100
        Re: [Ksummit-discuss] Including images on Sphinx documents James Bottomley <James.Bottomley@HansenPartnership.com> - 2016-11-17 18:10 +0100
        Re: [Ksummit-discuss] Including images on Sphinx documents Mauro Carvalho Chehab <mchehab@infradead.org> - 2016-11-17 18:20 +0100
          Re: [Ksummit-discuss] Including images on Sphinx documents Johannes Berg <johannes@sipsolutions.net> - 2016-11-17 18:30 +0100
        Re: [Ksummit-discuss] Including images on Sphinx documents Theodore Ts'o <tytso@mit.edu> - 2016-11-17 18:30 +0100
        Re: [Ksummit-discuss] Including images on Sphinx documents Linus Torvalds <torvalds@linux-foundation.org> - 2016-11-17 19:10 +0100
          Re: [Ksummit-discuss] Including images on Sphinx documents Jani Nikula <jani.nikula@intel.com> - 2016-11-18 10:20 +0100
            Re: [Ksummit-discuss] Including images on Sphinx documents Daniel Vetter <daniel@ffwll.ch> - 2016-11-18 11:30 +0100
          Re: [Ksummit-discuss] Including images on Sphinx documents Jonathan Corbet <corbet@lwn.net> - 2016-11-19 18:20 +0100
            Re: [Ksummit-discuss] Including images on Sphinx documents Andrew Lunn <andrew@lunn.ch> - 2016-11-19 18:40 +0100
            Re: [Ksummit-discuss] Including images on Sphinx documents Bart Van Assche <Bart.VanAssche@sandisk.com> - 2016-11-19 19:00 +0100
            Re: [Ksummit-discuss] Including images on Sphinx documents David Woodhouse <dwmw2@infradead.org> - 2016-11-19 19:00 +0100
              Re: [Ksummit-discuss] Including images on Sphinx documents Linus Torvalds <torvalds@linux-foundation.org> - 2016-11-19 19:50 +0100
                Re: [Ksummit-discuss] Including images on Sphinx documents "David Woodhouse" <dwmw2@infradead.org> - 2016-11-20 00:30 +0100
                  Re: [Ksummit-discuss] Including images on Sphinx documents Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2016-11-20 15:30 +0100
            Re: [Ksummit-discuss] Including images on Sphinx documents Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2016-11-19 22:00 +0100
              Re: [Ksummit-discuss] Including images on Sphinx documents Linus Torvalds <torvalds@linux-foundation.org> - 2016-11-19 22:10 +0100
            Re: [Ksummit-discuss] Including images on Sphinx documents Johannes Berg <johannes@sipsolutions.net> - 2016-11-21 11:40 +0100
              Re: [Ksummit-discuss] Including images on Sphinx documents Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2016-11-21 15:10 +0100
                Re: [Ksummit-discuss] Including images on Sphinx documents Jani Nikula <jani.nikula@linux.intel.com> - 2016-11-21 16:50 +0100
                Re: [Ksummit-discuss] Including images on Sphinx documents Johannes Berg <johannes@sipsolutions.net> - 2016-11-21 16:50 +0100
                  Re: [Ksummit-discuss] Including images on Sphinx documents Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2016-11-21 20:50 +0100
                Re: [Ksummit-discuss] Including images on Sphinx documents James Bottomley <James.Bottomley@HansenPartnership.com> - 2016-11-21 16:50 +0100

Page 1 of 2  [1] 2  Next page →

#1523637 — Re: [Ksummit-discuss] Including images on Sphinx documents

On Tuesday, November 8, 2016 8:50:36 AM CET Mauro Carvalho Chehab wrote:
> > [...]
> > > And it may even require "--shell-escape" to be passed at the xelatex
> > > call if inkscape is not in the path, with seems to be a strong
> > > indication that SVG support is not native to texlive, but, instead,
> > > just a way to make LaTeX to call inkscape to do the image conversion.  
> > 
> > Please don't require --shell-escape as part of the TeX workflow.  If
> > LaTeX can't handle the desired image format natively, it needs
> > conversion in advance.
> 
> Agreed. I sent a patch series to linux-doc, doing the conversion in
> advance:
>         https://marc.info/?l=linux-doc&m=147859902804144&w=2
> 
> Not sure why, but the archives don't have all patches yet.
> Anyway, the relevant one is this:
>         https://git.linuxtv.org/mchehab/experimental.git/commit/?h=pdf-fixes&id=5d41c452c787f6a6c755a3855312435bc439acb8
> 
> It basically calls ImageMagick "convert" tool for all png and
> pdf files currently at the documentation (they're all at media,
> ATM).

It looks like we still need to find a way to address the .gif files
though, as they have the same problem as the .pdf files.

During the kernel summit, I looked around for any binary files in
the kernel source tree, and except for the penguin logo, they are
all in Documentation/media/uapi/v4l/, but they are not all pdf
files, but also .png and .pdf.

	Arnd

[toc] | [next] | [standalone]

#1523804

Hi Arnd,

Em Wed, 16 Nov 2016 17:03:47 +0100
Arnd Bergmann <arnd@arndb.de> escreveu:

> On Tuesday, November 8, 2016 8:50:36 AM CET Mauro Carvalho Chehab wrote:
> > It basically calls ImageMagick "convert" tool for all png and
> > pdf files currently at the documentation (they're all at media,
> > ATM).  
> 
> It looks like we still need to find a way to address the .gif files
> though, as they have the same problem as the .pdf files.

Actually, my last patch series removed all *.pdf images and converted
all .gif files under Documentation/media to PNG[1]. I also replaced some
images by .svg, but the remaining ones are more complex. I'm even not
sure if it makes sense to convert a few of them to vectorial graphics,
like on this case:
	https://mchehab.fedorapeople.org/kernel_docs/media/_images/selection.png

>
> During the kernel summit, I looked around for any binary files in
> the kernel source tree, and except for the penguin logo, they are
> all in Documentation/media/uapi/v4l/, but they are not all pdf
> files, but also .png and .pdf.

From what I understood from Linus, his problem is to carry on a
non-editable file at the Kernel tree. With that sense, a PNG file
is OK, as it is editable.

I had, in the past, problems with binary contents on either Mercurial
or git (before migrating to git, we used Mercurial for a while).
So, before Kernel 4.8, those .pdf, .png (and .gif) images were uuencoded,
in order to avoid troubles handling patches with them.

Nowadays, I don't see any issue handling binary images via e-mail or via git.

Btw, with that regards, SVG images are a lot worse to handle, as a single
line can easily have more than 998 characters, with makes some email
servers to reject patches with them. So, at the version 3 of my patch 
series, I had to use inkscape to ungroup some images, and to rewrite their
files, as otherwise, two patches were silently rejected by the VGER 
server.

[1] The reason to convert to PNG is that it means one less format to be
concerned with. Also, it doesn't make much sense to use two different
formats for bitmap images at the documentation.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1524368

On Wednesday, November 16, 2016 6:26:33 PM CET Mauro Carvalho Chehab wrote:
> Em Wed, 16 Nov 2016 17:03:47 +0100
> Arnd Bergmann <arnd@arndb.de> escreveu:
> 
> > On Tuesday, November 8, 2016 8:50:36 AM CET Mauro Carvalho Chehab wrote:
> > > It basically calls ImageMagick "convert" tool for all png and
> > > pdf files currently at the documentation (they're all at media,
> > > ATM).  
> > 
> > It looks like we still need to find a way to address the .gif files
> > though, as they have the same problem as the .pdf files.
> 
> Actually, my last patch series removed all *.pdf images and converted
> all .gif files under Documentation/media to PNG[1]. I also replaced some
> images by .svg, but the remaining ones are more complex. I'm even not
> sure if it makes sense to convert a few of them to vectorial graphics,
> like on this case:
> 	https://mchehab.fedorapeople.org/kernel_docs/media/_images/selection.png
> 
> >
> > During the kernel summit, I looked around for any binary files in
> > the kernel source tree, and except for the penguin logo, they are
> > all in Documentation/media/uapi/v4l/, but they are not all pdf
> > files, but also .png and .pdf.
> 
> From what I understood from Linus, his problem is to carry on a
> non-editable file at the Kernel tree. With that sense, a PNG file
> is OK, as it is editable.

[adding Linus for clarification]

I understood the concern as being about binary files that you cannot
modify with classic 'patch', which is a separate issue.

> I had, in the past, problems with binary contents on either Mercurial
> or git (before migrating to git, we used Mercurial for a while).
> So, before Kernel 4.8, those .pdf, .png (and .gif) images were uuencoded,
> in order to avoid troubles handling patches with them.
> 
> Nowadays, I don't see any issue handling binary images via e-mail or via git.



> Btw, with that regards, SVG images are a lot worse to handle, as a single
> line can easily have more than 998 characters, with makes some email
> servers to reject patches with them. So, at the version 3 of my patch 
> series, I had to use inkscape to ungroup some images, and to rewrite their
> files, as otherwise, two patches were silently rejected by the VGER 
> server.

Ok, good to know.

> [1] The reason to convert to PNG is that it means one less format to be
> concerned with. Also, it doesn't make much sense to use two different
> formats for bitmap images at the documentation.

I just tried converting all the .gif and .png files to .pnm. This would
make the files patchable but also add around 25MB to the uncompressed
kernel source tree (118kb compressed, compared to 113kb for the .gif and
.png files). This is certainly worse than the uuencoded files you
had before

	Arnd

[toc] | [prev] | [next] | [standalone]

#1524396

On Thu, 17 Nov 2016, Arnd Bergmann <arnd@arndb.de> wrote:
> On Wednesday, November 16, 2016 6:26:33 PM CET Mauro Carvalho Chehab wrote:
>> Em Wed, 16 Nov 2016 17:03:47 +0100
>> Arnd Bergmann <arnd@arndb.de> escreveu:
>> 
>> > On Tuesday, November 8, 2016 8:50:36 AM CET Mauro Carvalho Chehab wrote:
>> > > It basically calls ImageMagick "convert" tool for all png and
>> > > pdf files currently at the documentation (they're all at media,
>> > > ATM).  
>> > 
>> > It looks like we still need to find a way to address the .gif files
>> > though, as they have the same problem as the .pdf files.
>> 
>> Actually, my last patch series removed all *.pdf images and converted
>> all .gif files under Documentation/media to PNG[1]. I also replaced some
>> images by .svg, but the remaining ones are more complex. I'm even not
>> sure if it makes sense to convert a few of them to vectorial graphics,
>> like on this case:
>> 	https://mchehab.fedorapeople.org/kernel_docs/media/_images/selection.png
>> 
>> >
>> > During the kernel summit, I looked around for any binary files in
>> > the kernel source tree, and except for the penguin logo, they are
>> > all in Documentation/media/uapi/v4l/, but they are not all pdf
>> > files, but also .png and .pdf.
>> 
>> From what I understood from Linus, his problem is to carry on a
>> non-editable file at the Kernel tree. With that sense, a PNG file
>> is OK, as it is editable.
>
> [adding Linus for clarification]
>
> I understood the concern as being about binary files that you cannot
> modify with classic 'patch', which is a separate issue.

Also reported at [1]. So kernel.org has patches that you can't apply
with either classic patch or git apply. They could at least be in git
binary format so you could apply them with *something*. Of course, not
having binaries at all would be clean.

BR,
Jani.


[1] http://lkml.kernel.org/r/02a78907-933d-3f61-572e-28154b16b9e5@redhat.com

>
>> I had, in the past, problems with binary contents on either Mercurial
>> or git (before migrating to git, we used Mercurial for a while).
>> So, before Kernel 4.8, those .pdf, .png (and .gif) images were uuencoded,
>> in order to avoid troubles handling patches with them.
>> 
>> Nowadays, I don't see any issue handling binary images via e-mail or via git.
>
>
>
>> Btw, with that regards, SVG images are a lot worse to handle, as a single
>> line can easily have more than 998 characters, with makes some email
>> servers to reject patches with them. So, at the version 3 of my patch 
>> series, I had to use inkscape to ungroup some images, and to rewrite their
>> files, as otherwise, two patches were silently rejected by the VGER 
>> server.
>
> Ok, good to know.
>
>> [1] The reason to convert to PNG is that it means one less format to be
>> concerned with. Also, it doesn't make much sense to use two different
>> formats for bitmap images at the documentation.
>
> I just tried converting all the .gif and .png files to .pnm. This would
> make the files patchable but also add around 25MB to the uncompressed
> kernel source tree (118kb compressed, compared to 113kb for the .gif and
> .png files). This is certainly worse than the uuencoded files you
> had before
>
> 	Arnd
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1524490

Em Thu, 17 Nov 2016 13:28:29 +0200
Jani Nikula <jani.nikula@intel.com> escreveu:

> On Thu, 17 Nov 2016, Arnd Bergmann <arnd@arndb.de> wrote:
> > On Wednesday, November 16, 2016 6:26:33 PM CET Mauro Carvalho Chehab wrote:  
> >> Em Wed, 16 Nov 2016 17:03:47 +0100
> >> Arnd Bergmann <arnd@arndb.de> escreveu:
> >>   
> >> > On Tuesday, November 8, 2016 8:50:36 AM CET Mauro Carvalho Chehab wrote:  

> >> > During the kernel summit, I looked around for any binary files in
> >> > the kernel source tree, and except for the penguin logo, they are
> >> > all in Documentation/media/uapi/v4l/, but they are not all pdf
> >> > files, but also .png and .pdf.  
> >> 
> >> From what I understood from Linus, his problem is to carry on a
> >> non-editable file at the Kernel tree. With that sense, a PNG file
> >> is OK, as it is editable.  

Btw, Sphinx is indeed supporting PNG without external conversions, for
both html and pdf. For reference, those are the formats it supports
for images:

	http://www.sphinx-doc.org/en/1.4.8/builders.html

There are just two formats supported for the output types we use:
PNG and JPEG. Any other format requires external conversion.

> >
> > [adding Linus for clarification]
> >
> > I understood the concern as being about binary files that you cannot
> > modify with classic 'patch', which is a separate issue.  

I don't think this is a big issue, as nowadays everyone uses git.

> 
> Also reported at [1]. So kernel.org has patches that you can't apply
> with either classic patch or git apply. They could at least be in git
> binary format so you could apply them with *something*. Of course, not
> having binaries at all would be clean.
> 
> [1] http://lkml.kernel.org/r/02a78907-933d-3f61-572e-28154b16b9e5@redhat.com

This could be solved the other way around: someone could send a patch
to "patch" adding support for binary patches in the git format.

The thing is: images usually don't change that much. For example, the
tux logo only had 3 patches in git:
	3d4f16348b77 Revert "linux.conf.au 2009: Tuz"
	8032b526d1a3 linux.conf.au 2009: Tuz
	1da177e4c3f4 (tag: v2.6.12-rc2) Linux-2.6.12-rc2

Some other examples:

$ git log --oneline ./Documentation/blockdev/drbd/DRBD-data-packets.svg
b411b3637fa7 The DRBD driver

$ git log --oneline ./Documentation/RCU/Design/Data-Structures/HugeTreeClassicRCU.svg
5c1458478c49 documentation: Add documentation for RCU's major data structures

The media images changed a little bit more, but due to the recent
documentation efforts. Usually, it takes years for someone to touch
them. If you look on it at Kernel v4.7, for example, even the media
images didn't have any changes, except due to dir renames or when
the files got encoded with base64:

$ git log --oneline -- v4.7 ./Documentation/DocBook/media/v4l/subdev-image-processing-full.svg
59ef29cc86af [media] v4l: Add subdev selections documentation: svg and dia files
$ git checkout v4.7
$ git log --oneline --follow ./Documentation/DocBook/media/v4l/fieldseq_bt.pdf
4266129964b8 [media] DocBook: Move all media docbook stuff into its own directory
8e080c2e6cad V4L/DVB (12761): DocBook: add media API specs
$ git log --oneline --follow ./Documentation/DocBook/media/v4l/*.gif
bd7319dc325a [media] DocBook: Use base64 for gif/png files
4266129964b8 [media] DocBook: Move all media docbook stuff into its own directory
$ git log --oneline --follow ./Documentation/DocBook/media/bayer.png.b64
bd7319dc325a [media] DocBook: Use base64 for gif/png files

> >> I had, in the past, problems with binary contents on either Mercurial
> >> or git (before migrating to git, we used Mercurial for a while).
> >> So, before Kernel 4.8, those .pdf, .png (and .gif) images were uuencoded,
> >> in order to avoid troubles handling patches with them.
> >> 
> >> Nowadays, I don't see any issue handling binary images via e-mail or via git.  
> >
> >
> >  
> >> Btw, with that regards, SVG images are a lot worse to handle, as a single
> >> line can easily have more than 998 characters, with makes some email
> >> servers to reject patches with them. So, at the version 3 of my patch 
> >> series, I had to use inkscape to ungroup some images, and to rewrite their
> >> files, as otherwise, two patches were silently rejected by the VGER 
> >> server.  
> >
> > Ok, good to know.
> >  
> >> [1] The reason to convert to PNG is that it means one less format to be
> >> concerned with. Also, it doesn't make much sense to use two different
> >> formats for bitmap images at the documentation.  
> >
> > I just tried converting all the .gif and .png files to .pnm. This would
> > make the files patchable but also add around 25MB to the uncompressed
> > kernel source tree (118kb compressed, compared to 113kb for the .gif and
> > .png files). This is certainly worse than the uuencoded files you
> > had before

There's also another drawback: PNM doesn't allow transparent background.
Some images have transparent backgrounds. So, such conversion would lose
it. Also, PNM is not supported on Sphinx, so it would require external
conversion, just like svg.

IMHO, if we're willing to make easier to use patch, the best is to use
uuencode (or base64). In that case, It could make sense to use it for svg 
too, in order to solve the warn of patches that contain lines longer than
998 characters (with is a violation to IETF rfc 2821).

The drawback of uuencode/base64 is that it makes harder to edit the files.
So, IMHO, I would keep them in binary format.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1524474

Em Thu, 17 Nov 2016 12:07:15 +0100
Arnd Bergmann <arnd@arndb.de> escreveu:

> On Wednesday, November 16, 2016 6:26:33 PM CET Mauro Carvalho Chehab wrote:
> > Em Wed, 16 Nov 2016 17:03:47 +0100
> > Arnd Bergmann <arnd@arndb.de> escreveu:
> >   
> > > On Tuesday, November 8, 2016 8:50:36 AM CET Mauro Carvalho Chehab wrote:  

> > From what I understood from Linus, his problem is to carry on a
> > non-editable file at the Kernel tree. With that sense, a PNG file
> > is OK, as it is editable.  

Btw, Sphinx is indeed supporting PNG without external conversions, for
both html and pdf. For reference, those are the formats it supports
for images:

	http://www.sphinx-doc.org/en/1.4.8/builders.html

There are just two formats supported for the output types we use:
PNG and JPEG.


> [adding Linus for clarification]
> 
> I understood the concern as being about binary files that you cannot
> modify with classic 'patch', which is a separate issue.

I don't think this is a big issue, as nowadays everyone uses git.

Also, this could be solved the other way around: someone could send a
patch to "patch" adding support for binary patches in the git format.

Also, images usually don't change that much. For example, the
tux logo only had 3 patches in git:
	3d4f16348b77 Revert "linux.conf.au 2009: Tuz"
	8032b526d1a3 linux.conf.au 2009: Tuz
	1da177e4c3f4 (tag: v2.6.12-rc2) Linux-2.6.12-rc2

Some other examples:

$ git log --oneline ./Documentation/blockdev/drbd/DRBD-data-packets.svg
b411b3637fa7 The DRBD driver

$ git log --oneline ./Documentation/RCU/Design/Data-Structures/HugeTreeClassicRCU.svg
5c1458478c49 documentation: Add documentation for RCU's major data structures

The media images changed a little bit more, but due to the recent
documentation efforts. Usually, it takes years for someone to touch
them. If you look on it at Kernel v4.7, for example, even the media
images didn't have any changes, except due to dir renames:

$ git log --oneline ./Documentation/DocBook/media/v4l/subdev-image-processing-full.svg
59ef29cc86af [media] v4l: Add subdev selections documentation: svg and dia files

$ git log --oneline --follow ./Documentation/DocBook/media/v4l/fieldseq_bt.pdf
4266129964b8 [media] DocBook: Move all media docbook stuff into its own directory
8e080c2e6cad V4L/DVB (12761): DocBook: add media API specs

$ git log --oneline --follow ./Documentation/DocBook/media/v4l/*.gif
bd7319dc325a [media] DocBook: Use base64 for gif/png files
4266129964b8 [media] DocBook: Move all media docbook stuff into its own directory

$ git log --oneline --follow Documentation/DocBook/media/bayer.png.b64
bd7319dc325a [media] DocBook: Use base64 for gif/png files

> > I had, in the past, problems with binary contents on either Mercurial
> > or git (before migrating to git, we used Mercurial for a while).
> > So, before Kernel 4.8, those .pdf, .png (and .gif) images were uuencoded,
> > in order to avoid troubles handling patches with them.
> > 
> > Nowadays, I don't see any issue handling binary images via e-mail or via git.  
> 
> > Btw, with that regards, SVG images are a lot worse to handle, as a single
> > line can easily have more than 998 characters, with makes some email
> > servers to reject patches with them. So, at the version 3 of my patch 
> > series, I had to use inkscape to ungroup some images, and to rewrite their
> > files, as otherwise, two patches were silently rejected by the VGER 
> > server.  
> 
> Ok, good to know.
> 
> > [1] The reason to convert to PNG is that it means one less format to be
> > concerned with. Also, it doesn't make much sense to use two different
> > formats for bitmap images at the documentation.  
> 
> I just tried converting all the .gif and .png files to .pnm. This would
> make the files patchable but also add around 25MB to the uncompressed
> kernel source tree (118kb compressed, compared to 113kb for the .gif and
> .png files). This is certainly worse than the uuencoded files you
> had before

There's also another drawback: PNM doesn't allow transparent background.
Some images have transparent backgrounds. So, such conversion would lose
it. Also, PNM is not supported on Sphinx, so it would require external
conversion, just like svg.

IMHO, if we're willing to make easier to use patch, the best is to use
uuencode (or base64). In that case, It could make sense to use it for svg 
too, in order to solve the warn of patches that contain lines longer than
998 characters (with is a violation to IETF rfc 2821).

The drawback of uuencode/base64 is that it makes harder to edit the files.
So, IMHO, I would keep them in binary format.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1524488

On Thu, Nov 17, 2016 at 8:02 AM, Linus Torvalds
<torvalds@linux-foundation.org> wrote:
>
> No. That is how I *noticed* the issue. Those stupid pdf binary files
> have been around forever, I just didn't notice until the Fedora people
> started complaining about the patches.

Side note: my release patches these days enable both "--binary" and
"-M", so they require "git apply" now. So we handle the binaries fine
in patches now, but as mentioned, that was just what made me notice,
it wasn't a fix for the deeper ("source") problem.

                   Linus

[toc] | [prev] | [next] | [standalone]

#1524504

On Thu, 2016-11-17 at 13:16 -0200, Mauro Carvalho Chehab wrote:
> Hi Ted,
> 
> Em Thu, 17 Nov 2016 09:52:44 -0500
> Theodore Ts'o <tytso@mit.edu> escreveu:
> 
> > On Thu, Nov 17, 2016 at 12:07:15PM +0100, Arnd Bergmann wrote:
> > > [adding Linus for clarification]
> > > 
> > > I understood the concern as being about binary files that you
> > > cannot
> > > modify with classic 'patch', which is a separate issue.  
> > 
> > I think the other complaint is that the image files aren't "source"
> > in
> > the proper term, since they are *not* the preferred form for
> > modification --- that's the svg files.  Beyond the license
> > compliance
> > issues (which are satisified because the .svg files are included in
> > the git tree), there is the SCM cleaniless argument of not
> > including
> > generated files in the distribution, since this increases the
> > opportunites for the "real" source file and the generated source
> > file
> > to get out of sync.  (As just one example, if the patch can't
> > represent the change to binary file.)
> > 
> > I do check in generated files on occasion --- usually because I
> > don't
> > trust autoconf to be a stable in terms of generating a correct
> > configure file from a configure.in across different versions of
> > autoconf and different macro libraries that might be installed on
> > the
> > system.  So this isn't a hard and fast rule by any means (although
> > Linus may be more strict than I on that issue).
> > 
> > I don't understand why it's so terrible to have generate the image
> > file from the .svg file in a Makefile rule, and then copy it
> > somewhere
> > else if Sphinx is too dumb to fetch it from the normal location?
> 
> The images whose source are in .svg are now generated via Makefile
> for the PDF output (after my patches, already applied to the docs
> -next
> tree).
> 
> So, the problem that remains is for those images whose source
> is a bitmap. If we want to stick with the Sphinx supported formats,
> we have only two options for bitmaps: jpg or png. We could eventually
> use uuencode or base64 to make sure that the patches won't use
> git binary diff extension, or, as Arnd proposed, use a portable
> bitmap format, in ascii, converting via Makefile, but losing
> the alpha channel with makes the background transparent.

If it can use svg, why not use that?  SVG files can be a simple xml
wrapper around a wide variety of graphic image formats which are
embedded in the svg using the data-uri format, you know ...

Anything that handles SVGs should be able to handle all the embeddable
image formats, which should give you a way around image restrictions
whatever it is would otherwise have.

James

[toc] | [prev] | [next] | [standalone]

#1524524

Hi Ted,

Em Thu, 17 Nov 2016 09:52:44 -0500
Theodore Ts'o <tytso@mit.edu> escreveu:

> On Thu, Nov 17, 2016 at 12:07:15PM +0100, Arnd Bergmann wrote:
> > [adding Linus for clarification]
> > 
> > I understood the concern as being about binary files that you cannot
> > modify with classic 'patch', which is a separate issue.  
> 
> I think the other complaint is that the image files aren't "source" in
> the proper term, since they are *not* the preferred form for
> modification --- that's the svg files.  Beyond the license compliance
> issues (which are satisified because the .svg files are included in
> the git tree), there is the SCM cleaniless argument of not including
> generated files in the distribution, since this increases the
> opportunites for the "real" source file and the generated source file
> to get out of sync.  (As just one example, if the patch can't
> represent the change to binary file.)
> 
> I do check in generated files on occasion --- usually because I don't
> trust autoconf to be a stable in terms of generating a correct
> configure file from a configure.in across different versions of
> autoconf and different macro libraries that might be installed on the
> system.  So this isn't a hard and fast rule by any means (although
> Linus may be more strict than I on that issue).
> 
> I don't understand why it's so terrible to have generate the image
> file from the .svg file in a Makefile rule, and then copy it somewhere
> else if Sphinx is too dumb to fetch it from the normal location?

The images whose source are in .svg are now generated via Makefile
for the PDF output (after my patches, already applied to the docs-next
tree).

So, the problem that remains is for those images whose source
is a bitmap. If we want to stick with the Sphinx supported formats,
we have only two options for bitmaps: jpg or png. We could eventually
use uuencode or base64 to make sure that the patches won't use
git binary diff extension, or, as Arnd proposed, use a portable
bitmap format, in ascii, converting via Makefile, but losing
the alpha channel with makes the background transparent.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1524562

> So, the problem that remains is for those images whose source
> is a bitmap. If we want to stick with the Sphinx supported formats,
> we have only two options for bitmaps: jpg or png. We could eventually
> use uuencode or base64 to make sure that the patches won't use
> git binary diff extension, or, as Arnd proposed, use a portable
> bitmap format, in ascii, converting via Makefile, but losing
> the alpha channel with makes the background transparent.
> 

Or just "rewrite" them in svg? None of the gif files I can see actually
look like they'd have been drawn in gif format anyway. The original
source may be lost, but it doesn't seem all that hard to recreate them
in svg.

johannes

[toc] | [prev] | [next] | [standalone]

#1524553

On Thu, Nov 17, 2016 at 12:07:15PM +0100, Arnd Bergmann wrote:
> [adding Linus for clarification]
> 
> I understood the concern as being about binary files that you cannot
> modify with classic 'patch', which is a separate issue.

I think the other complaint is that the image files aren't "source" in
the proper term, since they are *not* the preferred form for
modification --- that's the svg files.  Beyond the license compliance
issues (which are satisified because the .svg files are included in
the git tree), there is the SCM cleaniless argument of not including
generated files in the distribution, since this increases the
opportunites for the "real" source file and the generated source file
to get out of sync.  (As just one example, if the patch can't
represent the change to binary file.)

I do check in generated files on occasion --- usually because I don't
trust autoconf to be a stable in terms of generating a correct
configure file from a configure.in across different versions of
autoconf and different macro libraries that might be installed on the
system.  So this isn't a hard and fast rule by any means (although
Linus may be more strict than I on that issue).

I don't understand why it's so terrible to have generate the image
file from the .svg file in a Makefile rule, and then copy it somewhere
else if Sphinx is too dumb to fetch it from the normal location?

     	       	      	      	       - Ted

[toc] | [prev] | [next] | [standalone]

#1524642

On Thu, Nov 17, 2016 at 3:07 AM, Arnd Bergmann <arnd@arndb.de> wrote:
>
> [adding Linus for clarification]
>
> I understood the concern as being about binary files that you cannot
> modify with classic 'patch', which is a separate issue.

No. That is how I *noticed* the issue. Those stupid pdf binary files
have been around forever, I just didn't notice until the Fedora people
started complaining about the patches.

My real problem is not "binary" or even "editable", but SOURCE CODE.

It's like including a "vmlinux" image in the git tree: sure, you can
technically "edit" it with a hex editor, but that doesn't change the
basic issue: it's not the original source code.

I don't want to see generated binary crap.

That goes for png, that goes for gif, that goes for pdf - and in fact
that goes for svg *too*, if the actual source of the svg was something
else, and it was generated from some other data.

We have makefiles, but more importantly, few enough people actually
*generate* the documentation, that I think if it's an option to just
fix sphinx, we should do that instead. If it means that you have to
have some development version of sphinx, so be it. Most people read
the documentation either directly in the unprocessed text-files
("source code") or on the web (by searching for pre-formatted docs)
that I really don't think we need to worry too much about the
toolchain.

But what we *should* worry about is having the kernel source tree
contain source.

                 Linus

[toc] | [prev] | [next] | [standalone]

#1525114

On Thu, 17 Nov 2016, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> We have makefiles, but more importantly, few enough people actually
> *generate* the documentation, that I think if it's an option to just
> fix sphinx, we should do that instead. If it means that you have to
> have some development version of sphinx, so be it. Most people read
> the documentation either directly in the unprocessed text-files
> ("source code") or on the web (by searching for pre-formatted docs)
> that I really don't think we need to worry too much about the
> toolchain.

My secret plan was to make building documentation easy, and then trick
more people into actually doing that on a regular basis, to ensure we
keep the build working and the output sensible in a variety of
environments. Sure we have a bunch of people doing this, and we have
0day doing this, but I'd hate it if it became laborous and fiddly to set
up the toolchain to generate documentation.

So I'm not necessarily disagreeing with anything you say, but I think
there's value in having a low bar for entry (from the toolchain POV) for
people interested in working with documentation, whether they're
seasoned kernel developers or newcomers purely interested in
documentation.

BR,
Jani.

-- 
Jani Nikula, Intel Open Source Technology Center

[toc] | [prev] | [next] | [standalone]

#1525161

On Fri, Nov 18, 2016 at 11:15:09AM +0200, Jani Nikula wrote:
> On Thu, 17 Nov 2016, Linus Torvalds <torvalds@linux-foundation.org> wrote:
> > We have makefiles, but more importantly, few enough people actually
> > *generate* the documentation, that I think if it's an option to just
> > fix sphinx, we should do that instead. If it means that you have to
> > have some development version of sphinx, so be it. Most people read
> > the documentation either directly in the unprocessed text-files
> > ("source code") or on the web (by searching for pre-formatted docs)
> > that I really don't think we need to worry too much about the
> > toolchain.
> 
> My secret plan was to make building documentation easy, and then trick
> more people into actually doing that on a regular basis, to ensure we
> keep the build working and the output sensible in a variety of
> environments. Sure we have a bunch of people doing this, and we have
> 0day doing this, but I'd hate it if it became laborous and fiddly to set
> up the toolchain to generate documentation.
> 
> So I'm not necessarily disagreeing with anything you say, but I think
> there's value in having a low bar for entry (from the toolchain POV) for
> people interested in working with documentation, whether they're
> seasoned kernel developers or newcomers purely interested in
> documentation.

Yeah, I want a low bar for doc building too. The initial hack fest we had
to add cross-linking and other dearly needed stuff increased the build
time so much that everyone stopped creating docs. It's of course not as
extreme, but stating that "no one runs the doc toolchain" is imo akin to
"no one runs gcc, developers just read the source and users install rpms".
It's totally true, until you try to change stuff, then you have to be able
to build that pile fast and with an easily-obtained toolchain.
-Daniel
-- 
Daniel Vetter
Software Engineer, Intel Corporation
http://blog.ffwll.ch

[toc] | [prev] | [next] | [standalone]

#1526078

On Thu, 17 Nov 2016 08:02:50 -0800
Linus Torvalds <torvalds@linux-foundation.org> wrote:

> We have makefiles, but more importantly, few enough people actually
> *generate* the documentation, that I think if it's an option to just
> fix sphinx, we should do that instead. If it means that you have to
> have some development version of sphinx, so be it. Most people read
> the documentation either directly in the unprocessed text-files
> ("source code") or on the web (by searching for pre-formatted docs)
> that I really don't think we need to worry too much about the
> toolchain.
> 
> But what we *should* worry about is having the kernel source tree
> contain source.

I would be happy to take a shot at fixing sphinx; we clearly need to
engage more with sphinx upstream in general.  But I guess I still haven't
figured out what "fixing sphinx" means in this case.

I don't know what the ultimate source of these images is (Mauro, perhaps
you could shed some light there?).  Perhaps its SVG for some of the
diagrams, but for the raster images, probably not; it's probably some
weird-ass diagram-editor format.  We could put those in the tree, but
they are likely to be harder to convert to a useful format and will raise
all of the same obnoxious binary patch issues.

Rather than beating our heads against the wall trying to convert between
various image formats, maybe we need to take a step back.  We're trying
to build better documentation, and there is certainly a place for
diagrams and such in that documentation.  Johannes was asking about it
for the 802.11 docs, and I know Paul has run into these issues with the
RCU docs as well.  Might there be a tool or an extension out there that
would allow us to express these diagrams in a text-friendly, editable
form?

With some effort, I bet we could get rid of a number of the images, and
perhaps end up with something that makes sense when read in the .rst
source files as an extra benefit.  But I'm not convinced that we can,
say, sensibly express the differences between different video interlacing
schemes that way.

jon

[toc] | [prev] | [next] | [standalone]

#1526092

> Rather than beating our heads against the wall trying to convert between
> various image formats, maybe we need to take a step back.  We're trying
> to build better documentation, and there is certainly a place for
> diagrams and such in that documentation.  Johannes was asking about it
> for the 802.11 docs, and I know Paul has run into these issues with the
> RCU docs as well.  Might there be a tool or an extension out there that
> would allow us to express these diagrams in a text-friendly, editable
> form?

Hi Jonathan

A lot depends on what the diagram is supposed to show. I've used
graphviz dot in documents which get processes with Sphinx. That works
well for state machine, linked lists, etc. It uses a mainline Sphinx
extension.

It does however increase the size of your documents toolchain, you
need graphviz. But i doubt there is a distribution which does not have
it.

If you are worried about getting all these needed tools installed, i
think tools/perf might be a useful guide. When you compile it, it
gives helpful hints:

No libdw DWARF unwind found, Please install elfutils-devel/libdw-dev >= 0.158
No sys/sdt.h found, no SDT events are defined, please install systemtap-sdt-devel or systemtap-sdt-dev

     Andrew

[toc] | [prev] | [next] | [standalone]

#1526099

On 11/19/16 09:15, Jonathan Corbet wrote:
> Might there be a tool or an extension out there that would allow us
 > to express these diagrams in a text-friendly, editable form?

How about using the graphviz languages for generating diagrams that can 
be described easily in one of the graphviz languages? The graphviz 
programming languages are well suited for version control. And the 
graphviz software includes a tool for converting diagrams described in a 
graphviz language into many formats, including png, svg and pdf. 
Examples of diagrams generated with graphviz are available at 
http://www.graphviz.org/Gallery.php.

Bart.

[toc] | [prev] | [next] | [standalone]

#1526101

[Multipart message — attachments visible in raw view] — view raw

On Sat, 2016-11-19 at 10:15 -0700, Jonathan Corbet wrote:
> Might there be a tool or an extension out there that
> would allow us to express these diagrams in a text-friendly, editable
> form?

I know it's unfashionable these days, but TeX always used to be bloody
good at that kind of thing.

-- 
dwmw2

[toc] | [prev] | [next] | [standalone]

#1526120

On Sat, Nov 19, 2016 at 9:55 AM, David Woodhouse <dwmw2@infradead.org> wrote:
>
> I know it's unfashionable these days, but TeX always used to be bloody
> good at that kind of thing.

You must have used a different TeX than I did.

TeX is a horrible example. The moment you needed to insert anything
that TeX didn't know about, you were screwed.

I think my go-to for TeX was LaTeX, the "epsfig" thing, and then xfig
and eps files (using fig2dev). Christ, I get flashbacks just thinking
about it.

I thought one of the points of Sphinx was to not have to play those games.

I think that graphviz and svg are the reasonable modern formats. Let's
try to avoid bitmaps in today's world, except perhaps as intermediate
generated things for what we can't avoid.

                    Linus

[toc] | [prev] | [next] | [standalone]

#1526151

> On Sat, Nov 19, 2016 at 9:55 AM, David Woodhouse <dwmw2@infradead.org>
> wrote:
>>
>> I know it's unfashionable these days, but TeX always used to be bloody
>> good at that kind of thing.
>
> You must have used a different TeX than I did.
>
> TeX is a horrible example. The moment you needed to insert anything
> that TeX didn't know about, you were screwed.
>
> I think my go-to for TeX was LaTeX, the "epsfig" thing, and then xfig
> and eps files (using fig2dev). Christ, I get flashbacks just thinking
> about it.

You're right. You included Epson,  which was generated from fig.

Now I'm having flashbacks too, and I actually remember.

> I thought one of the points of Sphinx was to not have to play those games.
>
> I think that graphviz and svg are the reasonable modern formats. Let's
> try to avoid bitmaps in today's world, except perhaps as intermediate
> generated things for what we can't avoid.

Sure, SVG makes sense. It's a text-based format (albeit XML) and it *can*
be edited with a text editor and reasonably kept in version control, at
least if the common tools store it in a diff-friendly way (with some line
breaks occasionally, and maybe no indenting). Do they?


-- 
dwmw2

[toc] | [prev] | [next] | [standalone]

Page 1 of 2  [1] 2  Next page →

Back to top | Article view | linux.kernel

csiph-web