Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.kernel > #1687569 > unrolled thread

[PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

Back to article view | Back to linux.kernel

Contents

  [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-07-14 19:00 +0200
    Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check  Sphinx install Markus Heiser <markus.heiser@darmarit.de> - 2017-07-14 19:40 +0200
      Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check  Sphinx install Jim Davis <jim.epost@gmail.com> - 2017-07-15 01:00 +0200
        Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check  Sphinx install Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-07-15 03:30 +0200
      Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check  Sphinx install Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-07-15 04:30 +0200
        Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check  Sphinx install Markus Heiser <markus.heiser@darmarit.de> - 2017-07-15 12:00 +0200
          Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check  Sphinx install Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-07-15 14:50 +0200
            Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check  Sphinx install Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-07-15 15:00 +0200
              Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check  Sphinx install Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-07-15 21:30 +0200
            Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check  Sphinx install Markus Heiser <markus.heiser@darmarit.de> - 2017-07-16 09:30 +0200
              Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check  Sphinx install Mauro Carvalho Chehab <mchehab@s-opensource.com> - 2017-07-16 12:00 +0200

#1687569 — [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

Solving Sphinx dependencies can be painful. Add a script to
check if everything is ok.

Signed-off-by: Mauro Carvalho Chehab <mchehab@s-opensource.com>
---

For now, this is just a RFC for a script that would detect
(and eventually install) the dependencies for Sphinx builds.

I tested it only with Ubuntu 17.04, as I created a lxc container
with it, from scratch, and tested Sphinx builds there for both
htmldocs and pdfdocs.

That's said, I noticed that, currently, on Ubuntu with Sphinx 1.5, this
is needed, in order to build a SVG file used on media.pdf:

+Please notice that, due to the way LaTeX handles SVG images, you may need
+to increase the memory it uses for handing images, using the following
+settings for texmf.cnf[#f1]_::
+
+	main_memory = 12000000
+	extra_mem_bot = 12000000
+	font_mem_size = 12000000
+	pool_size = 12000000
+	buf_size = 12000000
+
+.. note::
+   On Debian/Ubuntu, the texmf.cnf is auto-generated. So, the above
+   settings should be added at ``/etc/texmf/texmf.d/00debian.cnf``
+   file, and updated with ``update-texmf``.
+
+.. [f1] https://gordonlesti.com/debian-increase-latex-main-memory-size/
+

I guess I'll try, instead, to simplify the SVG there, instead of
documenting the need of another hack for Sphinx build instructions.

Yet, it sucks that PDF build is so fragile!

 scripts/sphinx-pre-install | 249 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 249 insertions(+)
 create mode 100755 scripts/sphinx-pre-install

diff --git a/scripts/sphinx-pre-install b/scripts/sphinx-pre-install
new file mode 100755
index 000000000000..305731e0ab9c
--- /dev/null
+++ b/scripts/sphinx-pre-install
@@ -0,0 +1,249 @@
+#!/usr/bin/perl
+use strict;
+
+#############
+# Static vars
+#############
+
+my @missing;
+my @opt_missing;
+my $system_release;
+
+##################################################
+# Subroutines used at the check missing deps logic
+##################################################
+
+sub catcheck($)
+{
+  my $res = "";
+  $res = qx(cat $_[0]) if (-r $_[0]);
+  return $res;
+}
+
+my $install;
+sub check_missing(%)
+{
+	my %map = %{$_[0]};
+
+	foreach my $prog (@missing) {
+		print "ERROR: please install \"$prog\", otherwise, build won't work.\n";
+		if (defined($map{$prog})) {
+			$install .= " " . $map{$prog};
+		} else {
+			$install .= " " . $prog;
+		}
+	}
+	foreach my $prog (@opt_missing) {
+		print "Warning: better to also install \"$prog\".\n";
+		if (defined($map{$prog})) {
+			$install .= " " . $map{$prog};
+		} else {
+			$install .= " " . $prog;
+		}
+	}
+
+	$install =~ s/^\s//;
+}
+
+sub give_ubuntu_hints()
+{
+	my %map = (
+		"sphinx-build"		=> "python3-sphinx python3-sphinx-rtd-theme",
+		"dot"			=> "graphviz",
+		"convert"		=> "imagemagick",
+		"Pod::Usage"		=> "perl-modules",
+		"xelatex"		=> "texlive-xetex",
+	);
+
+	check_missing(\%map);
+	printf("You should run:\n\n\tsudo apt-get install $install\n\n");
+}
+
+sub give_redhat_hints()
+{
+	my %map = (
+		"sphinx-build"		=> "python3-sphinx python3-sphinx_rtd_theme",
+		"dot"			=> "graphviz",
+		"convert"		=> "ImageMagick",
+		"Pod::Usage"		=> "perl-Pod-Usage",
+		"xelatex"		=> "texlive-xetex-bin",
+	);
+
+	check_missing(\%map);
+	printf("You should run:\n\n\tyum install -y $install\n\n");
+}
+
+sub give_opensuse_hints()
+{
+	my %map = (
+		# FIXME: add hints for SUSE here
+	);
+	
+	check_missing(\%map);
+	printf("You should run:\n\n\tsudo zypper install $install\n\n");
+}
+
+sub give_arch_linux_hints()
+{
+	my %map = (
+		# FIXME: add hints for arch-linux here
+	);
+
+	check_missing(\%map);
+	printf("You should install those package(s) (repository/package): $install\n\n");
+}
+
+sub give_gentoo_hints()
+{
+	my %map = (
+		# FIXME: add hints for Gentoo here
+	);
+
+	check_missing(\%map);
+	printf("You should emerge those package(s): $install\n\n");
+}
+
+sub give_hints()
+{
+	# Distro-specific hints
+	if ($system_release =~ /Red Hat Enterprise Linux/) {
+		give_redhat_hints;
+		return;
+	}
+	if ($system_release =~ /Fedora/) {
+		give_redhat_hints;
+		return;
+	}
+	if ($system_release =~ /Ubuntu/) {
+		give_ubuntu_hints;
+		return;
+	}
+	if ($system_release =~ /openSUSE/) {
+		give_opensuse_hints;
+		return;
+	}
+	if ($system_release =~ /Arch Linux/) {
+		give_arch_linux_hints;
+		return;
+	}
+	if ($system_release =~ /Debian/) {
+		give_ubuntu_hints;
+		return;
+	}
+	if ($system_release =~ /Gentoo/) {
+		give_gentoo_hints;
+		return;
+	}
+	# Fall-back to generic hint code
+	my %map = (
+		"sphinx-build" => "sphinx"
+	);
+	check_missing(\%map);
+	print "I don't know distro $system_release. So, I can't provide you a hint with the install procedure.\n";
+}
+
+my $need = 0;
+my $optional = 0;
+
+sub findprog($)
+{
+	foreach(split(/:/, $ENV{PATH})) {
+		return "$_/$_[0]" if(-x "$_/$_[0]");
+	}
+}
+
+sub need_program($)
+{
+	my $prog = shift;
+
+	return if findprog($prog);
+
+	push @missing, $prog;
+
+	$need++;
+}
+
+sub optional_program($)
+{
+	my $prog = shift;
+
+	return if findprog($prog);
+
+	push @opt_missing, $prog;
+
+	$optional++;
+}
+
+sub need_perl_module($)
+{
+	my $prog = shift;
+
+	my $err = system("perl -M$prog -e 1 2>/dev/null /dev/null");
+	return if ($err == 0);
+
+	push @missing, $prog;
+
+	$need++;
+}
+
+sub check_needs()
+{
+	if ($system_release) {
+		print "Checking if the needed tools for $system_release are available\n";
+	} else {
+		print "Checking if the needed tools are present\n";
+	}
+
+	# Check for needed programs/tools
+	need_perl_module "Pod::Usage";
+	need_program "make";
+	need_program "gcc";
+	need_program "sphinx-build";
+	optional_program "dot";
+	optional_program "convert";
+	optional_program "xelatex";
+
+	give_hints if ($need || $optional);
+
+	print "All optional dependenties are met.\n" if (!$optional);
+
+	die "Can't build as $need mandatory dependency is missing" if ($need == 1);
+	die "Can't build as $need mandatory dependencies are missing" if ($need);
+
+	print "Needed package dependencies are met.\n";
+}
+
+sub which($)
+{
+	my $file = shift;
+	my @path = split ":", $ENV{PATH};
+
+	foreach my $dir(@path) {
+		my $name = $dir.'/'.$file;
+		return $name if (-x $name );
+	}
+	return undef;
+}
+
+######
+# Main
+######
+
+# Determine the system type. There's no standard unique way that would
+# work with all distros with a minimal package install. So, several
+# methods are used here.
+#
+# By default, it will use lsb_release function. If not available, it will
+# fail back to reading the known different places where the distro name
+# is stored
+#
+$system_release = qx(lsb_release -d) if which("lsb_release");
+$system_release =~ s/Description:\s*// if ($system_release);
+$system_release = catcheck("/etc/system-release") if !$system_release;
+$system_release = catcheck("/etc/redhat-release") if !$system_release;
+$system_release = catcheck("/etc/lsb-release") if !$system_release;
+$system_release = catcheck("/etc/gentoo-release") if !$system_release;
+$system_release = catcheck("/etc/issue") if !$system_release;
+$system_release =~ s/\s+$//;
+
+check_needs;
-- 
2.13.0

[toc] | [next] | [standalone]

#1687580 — Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

> Am 14.07.2017 um 18:49 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
> 
> Solving Sphinx dependencies can be painful. Add a script to
> check if everything is ok.

just my 5cent:

What we need is a "requirements.txt" file to define a
**reference environment**. E.g. to stick Sphinx 1.4.9 in
such a reference environment::

<snip: requirements.txt> ---
Sphinx==1.4.9
sphinx_rtd_theme
<snap> ---------------------

The rest is similarly to what you wrote in doc-guide/sphinx.rst ...

The ref-environment can be build with virtualenv & pip::

$ virtualenv --python=python3 docenv
(doc-env) $ source ./docenv/bin/activate
(doc-env) $ pip install -r requirements.txt

From now we can start our build as usual. If not already done,
first activate the environment::

$ . ./docenv/bin/activate
(doc-env) $ make htmldocs

This (requirements.txt) is the way python packaging goes.

Ok, this won't solve TeX installation problems of Linux distros,
for this a script like the one here in your RFC is helpful.

-- Markus --

[toc] | [prev] | [next] | [standalone]

#1687710 — Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

On Fri, Jul 14, 2017 at 10:35 AM, Markus Heiser
<markus.heiser@darmarit.de> wrote:
>
>
> Ok, this won't solve TeX installation problems of Linux distros,

Which seems to be the thorniest problem.  It's one thing to identify
which sphinx versions work, and another to figure out which of the
metric boatload of TeX packages are needed.

On a fresh Fedora 26 install I was able to get "make pdfdocs" to build
(hurrah!) after adding

ImageMagick
latex
sphinx-build
texlive-adjustbox
texlive-babel-english
texlive-capt-of
texlive-cm
texlive-cmap
texlive-ec
texlive-eqparbox
texlive-euenc
texlive-fancyhdr
texlive-fncychap
texlive-framed
texlive-hyphen-base
texlive-mdwtools
texlive-multirow
texlive-parskip
texlive-tablefootnote
texlive-tabulary
texlive-threeparttable
texlive-titlesec
texlive-unicode-data
texlive-upquote
texlive-wrapfig
xelatex

which ended up installing about 160 RPMs.  Figuring out the right
texlive parts wasn't much fun; footnote.sty is well-hidden in
texlive-mdwtools, for instance, and without texlive-babel-english the
build failed with very mysterious errors in userspace-api.

Jim

[toc] | [prev] | [next] | [standalone]

#1687753 — Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

Em Fri, 14 Jul 2017 15:51:03 -0700
Jim Davis <jim.epost@gmail.com> escreveu:

> On Fri, Jul 14, 2017 at 10:35 AM, Markus Heiser
> <markus.heiser@darmarit.de> wrote:
> >
> >
> > Ok, this won't solve TeX installation problems of Linux distros,  
> 
> Which seems to be the thorniest problem.  It's one thing to identify
> which sphinx versions work, and another to figure out which of the
> metric boatload of TeX packages are needed.

Yes. That's likely the hardest part.

> On a fresh Fedora 26 install I was able to get "make pdfdocs" to build
> (hurrah!) after adding

Just sent a version 2 of the script, with support for Fedora 26.

I tested with a Fedora lxc container. The only package I had to manually
install there is perl. I'm pretty sure that a minimal Fedora image
(outside a LXC) already comes with perl, so I guess the script is
an improvement over manually checking all needed dependencies.

> ImageMagick
> latex
> sphinx-build
> texlive-adjustbox
> texlive-babel-english
> texlive-capt-of
> texlive-cm
> texlive-cmap
> texlive-ec
> texlive-eqparbox
> texlive-euenc
> texlive-fancyhdr
> texlive-fncychap
> texlive-framed
> texlive-hyphen-base
> texlive-mdwtools
> texlive-multirow
> texlive-parskip
> texlive-tablefootnote
> texlive-tabulary
> texlive-threeparttable
> texlive-titlesec
> texlive-unicode-data
> texlive-upquote
> texlive-wrapfig
> xelatex

There is a package called "python-sphinx-latex" that seems to install
almost everything needed for Fedora 26. This one, plus:
	xelatex
	texlive-adjustbox

It also optionally require DeJavu fonts.

are enough for PDF build. You still need:
	ImageMagick
	graphviz

for both html and pdf targets, if you want to have support for
images.
 
> which ended up installing about 160 RPMs.  Figuring out the right
> texlive parts wasn't much fun; footnote.sty is well-hidden in
> texlive-mdwtools, for instance, and without texlive-babel-english the
> build failed with very mysterious errors in userspace-api.

Yeah, that's why I think the right solution is to have a script that
will identify the distribution and install the missing dependencies.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1687835 — Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

Em Fri, 14 Jul 2017 19:35:59 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> > Am 14.07.2017 um 18:49 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
> > 
> > Solving Sphinx dependencies can be painful. Add a script to
> > check if everything is ok.  
> 
> just my 5cent:
> 
> What we need is a "requirements.txt" file to define a
> **reference environment**. E.g. to stick Sphinx 1.4.9 in
> such a reference environment::
> 
> <snip: requirements.txt> ---
> Sphinx==1.4.9
> sphinx_rtd_theme
> <snap> ---------------------
> 
> The rest is similarly to what you wrote in doc-guide/sphinx.rst ...
> 
> The ref-environment can be build with virtualenv & pip::
> 
> $ virtualenv --python=python3 docenv
> (doc-env) $ source ./docenv/bin/activate
> (doc-env) $ pip install -r requirements.txt
> 
> From now we can start our build as usual. If not already done,
> first activate the environment::
> 
> $ . ./docenv/bin/activate
> (doc-env) $ make htmldocs
> 
> This (requirements.txt) is the way python packaging goes.


The above assumes that the user wants to use virtenv and
have python3, virtualenv3 and pip3 already installed.

I agree that a virtual environment works better than using
distro-specific packaging, as Sphinx toolchain is really
fragile. But we should give an option for the developer to
use whatever he wants.

I'm actually thinking that the final version of this script
is to have a command line parameter to would allow selecting
between virtenv or package install for Sphinx.

IMHO, virtenv should be the default.

> Ok, this won't solve TeX installation problems of Linux distros,
> for this a script like the one here in your RFC is helpful.

Your proposal doesn't solve Python, TeX, GraphViz nor 
ImageMagick dependencies.

The TeX dependencies are the hardest ones, and, whatever
solution we take, this should be taken into account.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1687907 — Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

> Am 15.07.2017 um 04:21 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
> 
> Em Fri, 14 Jul 2017 19:35:59 +0200
> Markus Heiser <markus.heiser@darmarit.de> escreveu:
> 
>>> Am 14.07.2017 um 18:49 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
>>> 
>>> Solving Sphinx dependencies can be painful. Add a script to
>>> check if everything is ok.  
>> 
>> just my 5cent:
>> 
>> What we need is a "requirements.txt" file to define a
>> **reference environment**. E.g. to stick Sphinx 1.4.9 in
>> such a reference environment::
>> 
>> <snip: requirements.txt> ---
>> Sphinx==1.4.9
>> sphinx_rtd_theme
>> <snap> ---------------------
>> 
>> The rest is similarly to what you wrote in doc-guide/sphinx.rst ...
>> 
>> The ref-environment can be build with virtualenv & pip::
>> 
>> $ virtualenv --python=python3 docenv
>> (doc-env) $ source ./docenv/bin/activate
>> (doc-env) $ pip install -r requirements.txt
>> 
>> From now we can start our build as usual. If not already done,
>> first activate the environment::
>> 
>> $ . ./docenv/bin/activate
>> (doc-env) $ make htmldocs
>> 
>> This (requirements.txt) is the way python packaging goes.
> 
> 
> The above assumes that the user wants to use virtenv and
> have python3, virtualenv3 and pip3 already installed.
> 
> I agree that a virtual environment works better than using
> distro-specific packaging, as Sphinx toolchain is really
> fragile. But we should give an option for the developer to
> use whatever he wants.

The developer is free to choose the way he like. But we are talking
about what is "best practice".

I tested sphinx-pre-install and it works fine for me, thats not the
point. The point is: what do we recommend? E.g. for me it advices me
to run:

	sudo apt-get install python3-sphinx python3-sphinx-rtd-theme

We should not assume that the developer (better: the build-user) owns the
privilege to install fine grained OS packages. There is a admin-part and
a user-part:

The admin (sudoer) installs binaries into the OS:

* gcc
* make
* python3, virtualenv
* LiveTex
* ImageMagick
* perl
* graphviz

The user (developer) installs additional requirements anywhere
under $HOME:

* Python 

  best practice here is virtualenv and pip .. and the *recipe*
  to get a *reference environment* is the requirements.txt
 
* LaTeX

  what is best practice here? .. I know there is some package
  managing with "TeX Live Manager" (tlmgr) but I have no experience
  with. Is there a way to define a *requirements* file and
  to install such requirements anywhere under $HOME?

My english is less eloquent, but I hope its clear what I mean.

To have a script is nice, but first lets explore what best practice
is.


-- Markus --

[toc] | [prev] | [next] | [standalone]

#1687936 — Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

Em Sat, 15 Jul 2017 11:51:45 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> > Am 15.07.2017 um 04:21 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
> > 
> > Em Fri, 14 Jul 2017 19:35:59 +0200
> > Markus Heiser <markus.heiser@darmarit.de> escreveu:
> >   
> >>> Am 14.07.2017 um 18:49 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
> >>> 
> >>> Solving Sphinx dependencies can be painful. Add a script to
> >>> check if everything is ok.    
> >> 
> >> just my 5cent:
> >> 
> >> What we need is a "requirements.txt" file to define a
> >> **reference environment**. E.g. to stick Sphinx 1.4.9 in
> >> such a reference environment::
> >> 
> >> <snip: requirements.txt> ---
> >> Sphinx==1.4.9
> >> sphinx_rtd_theme
> >> <snap> ---------------------
> >> 
> >> The rest is similarly to what you wrote in doc-guide/sphinx.rst ...
> >> 
> >> The ref-environment can be build with virtualenv & pip::
> >> 
> >> $ virtualenv --python=python3 docenv
> >> (doc-env) $ source ./docenv/bin/activate
> >> (doc-env) $ pip install -r requirements.txt
> >> 
> >> From now we can start our build as usual. If not already done,
> >> first activate the environment::
> >> 
> >> $ . ./docenv/bin/activate
> >> (doc-env) $ make htmldocs
> >> 
> >> This (requirements.txt) is the way python packaging goes.  
> > 
> > 
> > The above assumes that the user wants to use virtenv and
> > have python3, virtualenv3 and pip3 already installed.
> > 
> > I agree that a virtual environment works better than using
> > distro-specific packaging, as Sphinx toolchain is really
> > fragile. But we should give an option for the developer to
> > use whatever he wants.  
> 
> The developer is free to choose the way he like. But we are talking
> about what is "best practice".

As I said, the idea is to let the user to decide what it wants.

I focused on the packaging approach first because such logic
is required for other packages. Now that it is working, just
sent a version 5 that will use virtualenv for Sphinx by default.

With such change, it will now do the right thing:

Forcing to use distro-packages:

	$ ./scripts/sphinx-pre-install --no-virtualenv
	Checking if the needed tools for Fedora release 26 (Twenty Six) are available
	ERROR: please install "python-sphinx", otherwise, build won't work.
	Warning: better to also install "sphinx_rtd_theme".
	Warning: better to also install "python-sphinx-latex".
	You should run:

		sudo dnf install -y python3-sphinx python3-sphinx_rtd_theme python-sphinx-latex

	Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 335.

Default:

	$ ./scripts/sphinx-pre-install
	Checking if the needed tools for Fedora release 26 (Twenty Six) are available
	Warning: better to also install "python-sphinx-latex".
	You should run:

		sudo dnf install -y python-sphinx-latex
		virtualenv sphinx_1.4
		. sphinx_1.4/bin/activate
		pip install 'docutils==0.12'
		pip install 'Sphinx==1.4.9'
		pip install sphinx_rtd_theme

	Can't build as 1 mandatory dependency is missing at ./scripts/sphinx-pre-install line 335.

There is one problem there on Fedora that I just noticed:
"python-sphinx-latex" actually installs python2-sphinx.

Fixing it is trivial, but will require some time to adjust, as the
script will need to manually check for the packages that are actually
required on Fedora.

Yet, before spending more time on such script, I'd like to have
more feedback if:

	- is this approach acceptable?
	- should it have an optional argument that will make the
	  script to run the needed commands;
	- should it be integrated at the Documentation/Makefile?
	- what's the best name/location for such script?

I guess it could also use kpsewhich to check if the needed
texlive packages are installed. However, the problem with such
approach is that texlive-kpathsea-bin package should be installed
first, in order to provide such command.

So, installing PDF and math dependencies would require two steps.

> I tested sphinx-pre-install and it works fine for me, thats not the
> point. The point is: what do we recommend? E.g. for me it advices me
> to run:
> 
> 	sudo apt-get install python3-sphinx python3-sphinx-rtd-theme
> 
> We should not assume that the developer (better: the build-user) owns the
> privilege to install fine grained OS packages. There is a admin-part and
> a user-part:

That's not relevant. Typically, anyone that is building a Kernel has
admin privileges, otherwise it can't actually test the Kernel that was
built.

Ok, there are exceptions to that, but, on such case, the user should
be able to request the admin to install whatever packages are needed
to build the Kernel.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1687941 — Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

Em Sat, 15 Jul 2017 09:49:40 -0300
Mauro Carvalho Chehab <mchehab@s-opensource.com> escreveu:

> I guess it could also use kpsewhich to check if the needed
> texlive packages are installed. However, the problem with such
> approach is that texlive-kpathsea-bin package should be installed
> first, in order to provide such command.
> 
> So, installing PDF and math dependencies would require two steps.

Hmm... answering myself: if texlive-kpathsea-bin is not installed,
that probably means that texlive is not installed. So, the script
can just give the command to install all texlive packages that are
needed for a given distribution.

Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1687984 — Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

Em Sat, 15 Jul 2017 09:55:09 -0300
Mauro Carvalho Chehab <mchehab@s-opensource.com> escreveu:

> Em Sat, 15 Jul 2017 09:49:40 -0300
> Mauro Carvalho Chehab <mchehab@s-opensource.com> escreveu:
> 
> > I guess it could also use kpsewhich to check if the needed
> > texlive packages are installed. However, the problem with such
> > approach is that texlive-kpathsea-bin package should be installed
> > first, in order to provide such command.
> > 
> > So, installing PDF and math dependencies would require two steps.  
> 
> Hmm... answering myself: if texlive-kpathsea-bin is not installed,
> that probably means that texlive is not installed. So, the script
> can just give the command to install all texlive packages that are
> needed for a given distribution.

While getting the required stuff for OpenSuse Tumbleweed, I noticed
a problem with the current toolchain, that affects both html and
PDF:

/usr/bin/dot -Tpdf /home/mchehab/docs/Documentation/doc-guide/hello.dot
Format: "pdf" not recognized. Use one of: canon cmap cmapx cmapx_np dot eps fig gv imap imap_np ismap pic plain plain-ext pov ps ps2 svg svgz tk vml vmlz xdot xdot1.2 xdot1.4

$ ls -lctra  Documentation/output/doc-guide/latex/hello.pdf
-rw-rw-r-- 1 mchehab mchehab 0 jul 15 16:20 Documentation/output/doc-guide/latex/hello.pdf

Btw, the same error is happening also with Fedora 26. So, I guess kfigure 
should test if pdf is supported, otherwise convert to some other format
(like ps or svg).

E. g. calling it with something like:

	$ dot -T help
	Format: "help" not recognized. Use one of: canon cmap cmapx cmapx_np dot eps fig gv imap imap_np ismap pic plain plain-ext pov ps ps2 svg svgz tk vml vmlz xdot xdot1.2 xdot1.4

then parse its output, before assuming that the requested format is
available.


Thanks,
Mauro

[toc] | [prev] | [next] | [standalone]

#1688147 — Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

Hy Mauro,

thanks a lot for your RFC, your patch consolidate a lot of
knowledge around Sphinx build requirements and brings a huge
value I will no longer miss.

I tested v6 of this patch on ubuntu and there is only some
conceptual bikeshedding I can do. 


> Am 15.07.2017 um 14:49 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
> 
> As I said, the idea is to let the user to decide what it wants.
> 
> I focused on the packaging approach first because such logic
> is required for other packages. Now that it is working, just
> sent a version 5 that will use virtualenv for Sphinx by default.

Thanks! .. I don't know how I can make it better (I'am not a perl
programmer) but it seems, that global

my @missing;
my @opt_missing;

and the "sub add_package" is dominant, while the 'virtaulenv'
is glued in .. may we can find a better structure (later).


> Yet, before spending more time on such script, I'd like to have
> more feedback if:
> 
> 	- is this approach acceptable?

Truly Yes!

I see, there is a value in the the "OS-packaging approach" even
if I prefer the  "native-packaging approach". Last means I like
to use the native packaging tools from python and LiveTeX.

For python, instead of :

		printf "\t$virtualenv sphinx_1.4\n";
		printf "\tpip install 'docutils==0.12'\n";
		printf "\tpip install 'Sphinx==1.4.9'\n";
		printf "\tpip install sphinx_rtd_theme\n";

add Documentation/sphinx/requirements.txt:

<snip: requirements.txt> ---
docutils==0.12
Sphinx==1.4.9
sphinx_rtd_theme
<snap> ---------------------

And the print ...

		printf "\t$virtualenv sphinx_1.4\n";
		printf "\tpip install -r Documentation/sphinx/requirements.txt\n";

For TexLive:

ATM I have no idea how to set up a *requierements* file and
install everything without sudo. But I have seen your 'kpsewhich'
approach  which is very interesting for me. I suppose a solution
for this will end in a combination of 'kpsewhich' and  'tlmgr'.
But for this I have to do more investigations / sorry that
I can't spend more time on this task right now.

> 	- should it have an optional argument that will make the
> 	  script to run the needed commands;

No. We can do this later upstream.

> 	- should it be integrated at the Documentation/Makefile?

No.

> 	- what's the best name/location for such script?

I like to see the script under Documentation/sphinx

> I guess it could also use kpsewhich to check if the needed
> texlive packages are installed. However, the problem with such
> approach is that texlive-kpathsea-bin package should be installed
> first, in order to provide such command.

I see you have solved it in v6 .. Thanks!

> 
> So, installing PDF and math dependencies would require two steps.
> 
>> I tested sphinx-pre-install and it works fine for me, thats not the
>> point. The point is: what do we recommend? E.g. for me it advices me
>> to run:
>> 
>> 	sudo apt-get install python3-sphinx python3-sphinx-rtd-theme
>> 
>> We should not assume that the developer (better: the build-user) owns the
>> privilege to install fine grained OS packages. There is a admin-part and
>> a user-part:
> 
> That's not relevant. Typically, anyone that is building a Kernel has
> admin privileges, otherwise it can't actually test the Kernel that was
> built.

Hmm .. buildbots and Continuous Integration (CI)?

> Ok, there are exceptions to that, but, on such case, the user should
> be able to request the admin to install whatever packages are needed
> to build the Kernel.
> 
> Thanks,
> Mauro

[toc] | [prev] | [next] | [standalone]

#1688179 — Re: [PATCH RFC] scripts/sphinx-pre-install: add a script to check Sphinx install

Em Sun, 16 Jul 2017 09:29:35 +0200
Markus Heiser <markus.heiser@darmarit.de> escreveu:

> Hy Mauro,
> 
> thanks a lot for your RFC, your patch consolidate a lot of
> knowledge around Sphinx build requirements and brings a huge
> value I will no longer miss.
> 
> I tested v6 of this patch on ubuntu and there is only some
> conceptual bikeshedding I can do. 
> 
> 
> > Am 15.07.2017 um 14:49 schrieb Mauro Carvalho Chehab <mchehab@s-opensource.com>:
> > 
> > As I said, the idea is to let the user to decide what it wants.
> > 
> > I focused on the packaging approach first because such logic
> > is required for other packages. Now that it is working, just
> > sent a version 5 that will use virtualenv for Sphinx by default.  
> 
> Thanks! .. I don't know how I can make it better (I'am not a perl
> programmer) but it seems, that global
> 
> my @missing;
> my @opt_missing;

We could use a hash here, instead, placing the "optional" as
an attribute.

> and the "sub add_package" is dominant, while the 'virtaulenv'
> is glued in .. may we can find a better structure (later).

Yeah, for sure some things there could be simplified.

> > Yet, before spending more time on such script, I'd like to have
> > more feedback if:
> > 
> > 	- is this approach acceptable?  
> 
> Truly Yes!

Good! I'll try to add support today for Gentoo. I suspect that
this is the last major distribution that people use.

> I see, there is a value in the the "OS-packaging approach" even
> if I prefer the  "native-packaging approach". Last means I like
> to use the native packaging tools from python and LiveTeX.

I know some Kernel devs that even package the Kernel they
built for their own consumption with rpm or dpkg - even taking a 
lot more time to do that ;)

The thing if using the OS-packaging is that the packaging tool
can be used to check everything installed at the system, with
makes a lot easier when installing a package or upgrading.

For example, when I upgraded my system to Fedora 26, as python
changed from version 3.5 to version 3.6, I had to destroy the
virtual envs and re-create them again. If I had used, instead,
the OS-packaging, that would be transparent. The same happens if
I wanted to upgrade Sphinx packages to a newer version.

> For python, instead of :
> 
> 		printf "\t$virtualenv sphinx_1.4\n";
> 		printf "\tpip install 'docutils==0.12'\n";
> 		printf "\tpip install 'Sphinx==1.4.9'\n";
> 		printf "\tpip install sphinx_rtd_theme\n";
> 
> add Documentation/sphinx/requirements.txt:
> 
> <snip: requirements.txt> ---
> docutils==0.12
> Sphinx==1.4.9
> sphinx_rtd_theme
> <snap> ---------------------
> 
> And the print ...
> 
> 		printf "\t$virtualenv sphinx_1.4\n";
> 		printf "\tpip install -r Documentation/sphinx/requirements.txt\n";

Yeah, I can do that.

> 
> For TexLive:
> 
> ATM I have no idea how to set up a *requierements* file and
> install everything without sudo. But I have seen your 'kpsewhich'
> approach  which is very interesting for me. I suppose a solution
> for this will end in a combination of 'kpsewhich' and  'tlmgr'.

I see two problems with tlmgr:

- tlmgr is not installed by default (at least on Fedora);
- adding packages with other package managers makes admin
  process a way more complex for upgrades and security fixes.

That's why I opted to use only kpsewhich.

> But for this I have to do more investigations / sorry that
> I can't spend more time on this task right now.
> 
> > 	- should it have an optional argument that will make the
> > 	  script to run the needed commands;  
> 
> No. We can do this later upstream.
> 
> > 	- should it be integrated at the Documentation/Makefile?  
> 
> No.
> 
> > 	- what's the best name/location for such script?  
> 
> I like to see the script under Documentation/sphinx

If the idea is to run the scripts standalone, scripts/ is a better
place.

> 
> > I guess it could also use kpsewhich to check if the needed
> > texlive packages are installed. However, the problem with such
> > approach is that texlive-kpathsea-bin package should be installed
> > first, in order to provide such command.  
> 
> I see you have solved it in v6 .. Thanks!

Yes. It works fine for Fedora and OpenSuse. No need for Debian or
Arch Linux, as texlive is lot less granular there.

Didn't check yet how Gentoo packages it.

> > So, installing PDF and math dependencies would require two steps.
> >   
> >> I tested sphinx-pre-install and it works fine for me, thats not the
> >> point. The point is: what do we recommend? E.g. for me it advices me
> >> to run:
> >> 
> >> 	sudo apt-get install python3-sphinx python3-sphinx-rtd-theme
> >> 
> >> We should not assume that the developer (better: the build-user) owns the
> >> privilege to install fine grained OS packages. There is a admin-part and
> >> a user-part:  
> > 
> > That's not relevant. Typically, anyone that is building a Kernel has
> > admin privileges, otherwise it can't actually test the Kernel that was
> > built.  
> 
> Hmm .. buildbots and Continuous Integration (CI)?

I'm pretty sure that buildbots will use OS-packaging: a build bot farm
could easily become an admin nightmare if the packages are not managed
by a single package management system.

Thanks,
Mauro

[toc] | [prev] | [standalone]

Back to top | Article view | linux.kernel

csiph-web