Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #14959 > unrolled thread

#!/usr/bin/python3 vs virtualenv

Back to article view | Back to linux.debian.maint.python

Contents

  #!/usr/bin/python3 vs virtualenv Marc Glisse <marc.glisse@inria.fr> - 2023-03-03 16:30 +0100
    Re: #!/usr/bin/python3 vs virtualenv Danial Behzadi دانیال بهزادی <dani.behzi@ubuntu.com> - 2023-03-03 19:50 +0100
      Re: #!/usr/bin/python3 vs virtualenv Jeremy Stanley <fungi@yuggoth.org> - 2023-03-03 20:20 +0100
        Re: #!/usr/bin/python3 vs virtualenv Jorge Moraleda <jorge.moraleda@gmail.com> - 2023-03-03 21:30 +0100
          Re: #!/usr/bin/python3 vs virtualenv Jeremy Stanley <fungi@yuggoth.org> - 2023-03-03 21:50 +0100
            Re: #!/usr/bin/python3 vs virtualenv Jorge Moraleda <jorge.moraleda@gmail.com> - 2023-03-03 22:30 +0100
              Re: #!/usr/bin/python3 vs virtualenv Jeremy Stanley <fungi@yuggoth.org> - 2023-03-03 22:50 +0100
              Re: #!/usr/bin/python3 vs virtualenv Julian Gilbey <julian@d-and-j.net> - 2023-03-05 11:50 +0100
                Re: #!/usr/bin/python3 vs virtualenv Jorge Moraleda <jorge.moraleda@gmail.com> - 2023-03-05 13:00 +0100
                Re: #!/usr/bin/python3 vs virtualenv Stefano Rivera <stefanor@debian.org> - 2023-03-09 23:40 +0100
      Re: #!/usr/bin/python3 vs virtualenv Marc Glisse <marc.glisse@inria.fr> - 2023-03-05 14:20 +0100
    Re: #!/usr/bin/python3 vs virtualenv Dmitry Shachnev <mitya57@debian.org> - 2023-03-03 21:50 +0100

#14959 — #!/usr/bin/python3 vs virtualenv

Hello,

I recently reported 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1032283 , and Dmitry 
suggested discussing it on this mailing list.

Virtual environments work by adding some version of Python (possibly just 
a symlink to the system one) at the beginning of PATH. Debian policy for 
Python scripts is to use #!/usr/bin/python3 and **not** #!/usr/bin/env 
python3, so they are not affected.

I think there are 2 types of Python scripts. The first type is 
applications for end users, like hg or calibre (it is possible that the 
existence of plugins makes my choice of examples a bad one). If I have 
some strange Python virtual environment active, I am happy that the Debian 
policy prevents the virtual environment from breaking those applications 
(although I may still be able to break them with PYTHONPATH).

The second type is targeted at developers, like sphinx-build. If I 
activate a virtual environment that contains tensorflow, it is because I 
need `import tensorflow` to work inside sphinx-build, but now the Debian 
policy prevents it.

As discussed in the bug report, there are workarounds. The question here 
is whether it would make sense to amend the policy for some types of 
scripts. The answer is probably no, but it seems worth checking what 
people think.

The suggestion to install a copy of sphinx in the virtual environment is a 
step towards the advice I hear around me but don't really want to follow: 
"Developers should use conda, system python packages are only for system 
applications" :-(

(thanks a lot to everyone here working on packaging Python stuff in 
Debian)

-- 
Marc Glisse

[toc] | [next] | [standalone]

#14960

[Multipart message — attachments visible in raw view] — view raw

You just want to install sphinx via pip in the virtual environment too. Each venv should be atomic and isolated which means not depended to system packages.

[toc] | [prev] | [next] | [standalone]

#14961

[Multipart message — attachments visible in raw view] — view raw

On 2023-03-03 18:44:19 +0000 (+0000), Danial Behzadi دانیال بهزادی wrote:
> You just want to install sphinx via pip in the virtual environment
> too. Each venv should be atomic and isolated which means not
> depended to system packages.

However a venv can be made to use system packages if you use the
--system-site-packages option when creating it.
-- 
Jeremy Stanley

[toc] | [prev] | [next] | [standalone]

#14962

[Multipart message — attachments visible in raw view] — view raw

Would it be hard to support both philosophies?

I would like to suggest a couple of configuration options that by default
disallow using pip outside a virtual environment but that users with root
privilege can modify by editing a config file (probably somewhere in /etc)
and that would enable using pip outside a virtual environment, both as root
and as regular user respectively.

I feel this would satisfy the needs of regular users to be protected
against accidentally breaking their system while enabling power users to
have full control of their computer and enjoy the simplicity of a single
environment. Clearly this discussion suggests that debian has both types of
users and we should support them both if we can.

Thank you.

On Fri, Mar 3, 2023 at 2:17 PM Jeremy Stanley <fungi@yuggoth.org> wrote:

> On 2023-03-03 18:44:19 +0000 (+0000), Danial Behzadi دانیال بهزادی wrote:
> > You just want to install sphinx via pip in the virtual environment
> > too. Each venv should be atomic and isolated which means not
> > depended to system packages.
>
> However a venv can be made to use system packages if you use the
> --system-site-packages option when creating it.
> --
> Jeremy Stanley
>

[toc] | [prev] | [next] | [standalone]

#14965

[Multipart message — attachments visible in raw view] — view raw

On 2023-03-03 15:29:09 -0500 (-0500), Jorge Moraleda wrote:
> Would it be hard to support both philosophies?
> 
> I would like to suggest a couple of configuration options that by default
> disallow using pip outside a virtual environment but that users with root
> privilege can modify by editing a config file (probably somewhere in /etc)
> and that would enable using pip outside a virtual environment, both as root
> and as regular user respectively.
> 
> I feel this would satisfy the needs of regular users to be protected
> against accidentally breaking their system while enabling power users to
> have full control of their computer and enjoy the simplicity of a single
> environment. Clearly this discussion suggests that debian has both types of
> users and we should support them both if we can.
[...]

"Power users" who like to break their systems can simply `sudo pip
install --break-system-packages foo` or even just `sudo rm
/usr/lib/python3.11/EXTERNALLY-MANAGED` and then do whatever they
want anyway. It doesn't seem to me like there's much need for a
config option that does that for you, and adding one would imply
that Debian will help you fix things once you've done it. This
feature is simply a guard rail for users who otherwise wouldn't know
where the edge of that cliff is located.

There are already solutions for your power users, but as is often
said in such situations: If it breaks you get to keep the pieces.
Have fun!
-- 
Jeremy Stanley

[toc] | [prev] | [next] | [standalone]

#14967

[Multipart message — attachments visible in raw view] — view raw

Jeremy, Thank you for your quick reply!

I did not know about `sudo pip install --break-system-packages foo` or
`sudo rm /usr/lib/python3.11/EXTERNALLY-MANAGED` (Frankly I only knew about
this issue what I have read on this discussion). This is very helpful and
it really changes my outlook on this topic.

IMHO the existence of these options completely satisfies the requirements
of advanced users. In particular because, if I understand it correctly, the
second of these options would leave the configuration as it used to be in
previous releases of debian for those who want that.

So then, for those that raised the issue, were you aware of these
approaches? Will either of them satisfy your requirements? And if not,
could you elaborate further on why not?

Thank you again.

On Fri, Mar 3, 2023 at 3:43 PM Jeremy Stanley <fungi@yuggoth.org> wrote:

> On 2023-03-03 15:29:09 -0500 (-0500), Jorge Moraleda wrote:
> > Would it be hard to support both philosophies?
> >
> > I would like to suggest a couple of configuration options that by default
> > disallow using pip outside a virtual environment but that users with root
> > privilege can modify by editing a config file (probably somewhere in
> /etc)
> > and that would enable using pip outside a virtual environment, both as
> root
> > and as regular user respectively.
> >
> > I feel this would satisfy the needs of regular users to be protected
> > against accidentally breaking their system while enabling power users to
> > have full control of their computer and enjoy the simplicity of a single
> > environment. Clearly this discussion suggests that debian has both types
> of
> > users and we should support them both if we can.
> [...]
>
> "Power users" who like to break their systems can simply `sudo pip
> install --break-system-packages foo` or even just `sudo rm
> /usr/lib/python3.11/EXTERNALLY-MANAGED` and then do whatever they
> want anyway. It doesn't seem to me like there's much need for a
> config option that does that for you, and adding one would imply
> that Debian will help you fix things once you've done it. This
> feature is simply a guard rail for users who otherwise wouldn't know
> where the edge of that cliff is located.
>
> There are already solutions for your power users, but as is often
> said in such situations: If it breaks you get to keep the pieces.
> Have fun!
> --
> Jeremy Stanley
>

[toc] | [prev] | [next] | [standalone]

#14968

[Multipart message — attachments visible in raw view] — view raw

On 2023-03-03 16:22:11 -0500 (-0500), Jorge Moraleda wrote:
> I did not know about `sudo pip install --break-system-packages
> foo` or `sudo rm /usr/lib/python3.11/EXTERNALLY-MANAGED` (Frankly
> I only knew about this issue what I have read on this discussion).
[...]

They come from a reading of the pip documentation and associated
PEP-668 specification respectively. I do hope power users read
documentation before they choose to break their systems.

Be aware that the second approach I mentioned is removing a file
which gets installed by the libpython3.11-stdlib package, so will
end up getting replaced each time you upgrade that package as it's
not a conffile. It's a reasonable approach for things like container
base images where subsequent image build steps use pip to install
packages into the container, but it would be annoying for a "normal"
system where you upgrade packages without building entirely new
images.

The pip install command-line option is more of a user-facing escape
hatch, since it's clearly worded to let you know that it can quite
easily render existing Python applications inoperable if you choose
to install things into user or system site directories with pip.
-- 
Jeremy Stanley

[toc] | [prev] | [next] | [standalone]

#14969

On Fri, Mar 03, 2023 at 04:22:11PM -0500, Jorge Moraleda wrote:
> Jeremy, Thank you for your quick reply!
> 
> I did not know about `sudo pip install --break-system-packages foo` or  `sudo rm
> /usr/lib/python3.11/EXTERNALLY-MANAGED` (Frankly I only knew about this issue
> what I have read on this discussion). This is very helpful and it really changes
> my outlook on this topic.

The --break-system-packages option is noted in
/usr/share/doc/python3.11/README.venv, and this file is mentioned in
the NEWS file for python3.11.  The
/usr/lib/python3.11/EXTERNALLY-MANAGED file is not mentioned there; I
personally think that having to type --break-system-packages every
time one installs a package via pip globally or on a per-user basis is
safer, as it reminds you that you run risks doing so.

Best wishes,

   Julian

[toc] | [prev] | [next] | [standalone]

#14970

[Multipart message — attachments visible in raw view] — view raw

FWIW, I am one of those power users that develop python applications and
prefer the simplicity of a single environment. In my personal experience,
installing using `sudo pip` (or user pip, which I don't do at all) is not a
frequent operation for users that choose to manage a single environment.
Personally, I use it only after searching in apt and confirming the package
is not there. Having to type --break-system-packages every time when I have
to do it is a good solution for me that I find completely reasonable.

Thank you!

On Sun, Mar 5, 2023 at 5:47 AM Julian Gilbey <julian@d-and-j.net> wrote:

> On Fri, Mar 03, 2023 at 04:22:11PM -0500, Jorge Moraleda wrote:
> > Jeremy, Thank you for your quick reply!
> >
> > I did not know about `sudo pip install --break-system-packages foo` or
> `sudo rm
> > /usr/lib/python3.11/EXTERNALLY-MANAGED` (Frankly I only knew about this
> issue
> > what I have read on this discussion). This is very helpful and it really
> changes
> > my outlook on this topic.
>
> The --break-system-packages option is noted in
> /usr/share/doc/python3.11/README.venv, and this file is mentioned in
> the NEWS file for python3.11.  The
> /usr/lib/python3.11/EXTERNALLY-MANAGED file is not mentioned there; I
> personally think that having to type --break-system-packages every
> time one installs a package via pip globally or on a per-user basis is
> safer, as it reminds you that you run risks doing so.
>
> Best wishes,
>
>    Julian
>

[toc] | [prev] | [next] | [standalone]

#14975

Hi Julian (2023.03.05_10:47:08_+0000)
> the NEWS file for python3.11.  The
> /usr/lib/python3.11/EXTERNALLY-MANAGED file is not mentioned there

It was mentioned, but I've expanded on that in 3.11.2-5, and even more
in the next upload:

https://salsa.debian.org/cpython-team/python3/-/blob/master/debian/README.venv

SR

-- 
Stefano Rivera
  http://tumbleweed.org.za/
  +1 415 683 3272

[toc] | [prev] | [next] | [standalone]

#14971

(sorry for replying to a random message, I am not subscribed to the list)

I don't use `sudo pip`, too risky. I used to use pip to install packages 
with the implicit --user option (for things not available via apt). And I 
am aware of --break-system-packages, I used it the first time pip 
complained about an externally managed environment so I could keep working 
as before.

I decided to follow the recommandations and use a virtual environment. 
And, in order to benefit from the packaging done in Debian, I thought I 
could use --system-site-packages and restrict pip to a few things that 
Debian does not provide.

I now see that this was too optimistic. For now I'll use `python -m 
sphinx.cmd.build` as a workaround, but the next time I hit some issue I'll 
probably end up using conda, like everyone else. It just feels sad to see 
all the effort done in packaging Python things in Debian end with the 
recommendation not to use them :-(

Thank you all for the advice,

-- 
Marc Glisse

[toc] | [prev] | [next] | [standalone]

#14964

[Multipart message — attachments visible in raw view] — view raw

Hi all,

On Fri, Mar 03, 2023 at 04:10:50PM +0100, Marc Glisse wrote:
> Hello,
>
> I recently reported
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1032283 , and Dmitry
> suggested discussing it on this mailing list.
>
> Virtual environments work by adding some version of Python (possibly just a
> symlink to the system one) at the beginning of PATH. Debian policy for
> Python scripts is to use #!/usr/bin/python3 and **not** #!/usr/bin/env
> python3, so they are not affected.
>
> I think there are 2 types of Python scripts. The first type is applications
> for end users, like hg or calibre (it is possible that the existence of
> plugins makes my choice of examples a bad one). If I have some strange
> Python virtual environment active, I am happy that the Debian policy
> prevents the virtual environment from breaking those applications (although
> I may still be able to break them with PYTHONPATH).

I want to add that this concern applies to Sphinx too.

Newer versions of its dependencies (e.g. docutils or jinja2) can easily break
Sphinx. As well as incompatible versions of some Sphinx extensions installed
from PyPI.

--
Dmitry Shachnev

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web