Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.python > #197413 > unrolled thread

Re: Pip installs to unexpected place

Back to article view | Back to comp.lang.python

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Pip installs to unexpected place Mats Wichmann <mats@wichmann.us> - 2025-04-17 15:41 -0600
    Re: Pip installs to unexpected place Greg Ewing <greg.ewing@canterbury.ac.nz> - 2025-04-18 13:24 +1200
      Re: Pip installs to unexpected place (Posting On Python-List Prohibited) Lawrence D'Oliveiro <ldo@nz.invalid> - 2025-04-18 02:20 +0000
      Re: Pip installs to unexpected place "Peter J. Holzer" <hjp-python@hjp.at> - 2025-04-18 17:38 +0200
      Re: Pip installs to unexpected place Oscar Benjamin <oscar.j.benjamin@gmail.com> - 2025-04-18 17:11 +0100
      Re: Pip installs to unexpected place Thomas Passin <list1@tompassin.net> - 2025-04-18 13:08 -0400
      Re: Pip installs to unexpected place "Peter J. Holzer" <hjp-python@hjp.at> - 2025-04-19 10:38 +0200
      Re: Pip installs to unexpected place "Peter J. Holzer" <hjp-python@hjp.at> - 2025-04-19 10:56 +0200
        Re: Pip installs to unexpected place songbird <songbird@anthive.com> - 2025-04-19 07:49 -0400
      Re: Pip installs to unexpected place Thomas Passin <list1@tompassin.net> - 2025-04-19 15:56 -0400
        Re: Pip installs to unexpected place rbowman <bowman@montana.com> - 2025-04-20 04:34 +0000

#197413 — Re: Pip installs to unexpected place

On 4/17/25 15:15, Grant Edwards via Python-list wrote:
> On 2025-04-17, Left Right via Python-list <python-list@python.org> wrote:
>>> Also... when installing stuff with pip --user, it is always a package
>>> that is not installed for the system (usually not even available for
>>> the system). How can that "break system packages"?
>>
>> pip installs dependencies. Dependencies may disagree on the version
>> with the system packages.
> 
> Good point. I don't generally allow pip to install dependencies, so I
> forgot about that source of conflict. But as far as I can see, it
> still can only break stuff for the user, not for the system. The
> system is never going to look in my ~/.local/lib/python* directories.
Sure, and it's "your fault" (sic) if you break yourself. You and I can 
maybe accept that. But it's usually not obvious that you're done 
anything to break stuff - you just asked to install something, according 
to the instructions you read on the project's website, which it's 
reasonable (?) to expect reflects best practices. Usually a simple "pip 
install".

Say you've been using "MLwiz" installed via a system package.  It has 
two dozen dependencies, some of which have further deps. And it works 
great.  Then you install "Chatbridge" via pip, and it brings with it its 
own dependencies. It's not really useful to refuse to let it bring its 
own deps with it since there may be version-specific requirements on its 
deps, which that project has reasons for. So now you've got a few pkgs 
in the system with different versions in your .local, and you can't run 
MLwiz any longer, because a .local pkg (picked up first) conflicts with 
the requirements of MLwiz.

There's just not a really great answer to this.

"Use a virtualenv". Fine, but that's still daunting to many, and not 
necessarily as well described as it could be, and the choice of the word 
"virtual" in the name makes it sound more complex than it actually is. 
Projects can certainly do better at describing installation scenarios - 
at least if they have more than a trivial number of deps, where the 
conflict chances go up rapidly.

[toc] | [next] | [standalone]

#197414

On 18/04/25 9:41 am, Mats Wichmann wrote:
> There's just not a really great answer to this.

Seems to me a system-installed application shouldn't be looking in the 
user's .local packages in the first place. That should only be for 
things the user has installed "for this user".

-- 
Greg

[toc] | [prev] | [next] | [standalone]

#197415 — Re: Pip installs to unexpected place (Posting On Python-List Prohibited)

On Fri, 18 Apr 2025 13:24:28 +1200, Greg Ewing wrote:

> Seems to me a system-installed application shouldn't be looking in the 
> user's .local packages in the first place.

That might depend on what it’s doing. The user may want to override system 
defaults, not just for configuration, but also for certain code to be 
executed.

[toc] | [prev] | [next] | [standalone]

#197416

[Multipart message — attachments visible in raw view] — view raw

On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:
> On 18/04/25 9:41 am, Mats Wichmann wrote:
> > There's just not a really great answer to this.
> 
> Seems to me a system-installed application shouldn't be looking in the
> user's .local packages in the first place. That should only be for things
> the user has installed "for this user".

It's not the application that looks into .local, it's Python. If you say
that a system-installed Python shouldn't look into ~/.local, then you've
just disabled that mechanism completely. If not then Python would
somehow have to distinguish between system-installed and user-installed
scripts. This isn't as easy as checking whether the path starts with
/usr/bin or whether it belongs to root. Tying into the system's package
manager doesn't look appealing to me (not to mention that it might be
unacceptably slow).

        hjp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp@hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

[toc] | [prev] | [next] | [standalone]

#197417

On Fri, 18 Apr 2025 at 16:50, Peter J. Holzer via Python-list
<python-list@python.org> wrote:
>
> On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:
> > On 18/04/25 9:41 am, Mats Wichmann wrote:
> > > There's just not a really great answer to this.
> >
> > Seems to me a system-installed application shouldn't be looking in the
> > user's .local packages in the first place. That should only be for things
> > the user has installed "for this user".
>
> It's not the application that looks into .local, it's Python. If you say
> that a system-installed Python shouldn't look into ~/.local, then you've
> just disabled that mechanism completely. If not then Python would
> somehow have to distinguish between system-installed and user-installed
> scripts. This isn't as easy as checking whether the path starts with
> /usr/bin or whether it belongs to root. Tying into the system's package
> manager doesn't look appealing to me (not to mention that it might be
> unacceptably slow).

Couldn't the system-installed scripts have a shebang like:

#!/usr/bin/python3 -s

--
Oscar

[toc] | [prev] | [next] | [standalone]

#197418

On 4/18/2025 11:38 AM, Peter J. Holzer via Python-list wrote:
> On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:
>> On 18/04/25 9:41 am, Mats Wichmann wrote:
>>> There's just not a really great answer to this.
>>
>> Seems to me a system-installed application shouldn't be looking in the
>> user's .local packages in the first place. That should only be for things
>> the user has installed "for this user".
> 
> It's not the application that looks into .local, it's Python. If you say
> that a system-installed Python shouldn't look into ~/.local, then you've
> just disabled that mechanism completely. If not then Python would
> somehow have to distinguish between system-installed and user-installed
> scripts. This isn't as easy as checking whether the path starts with
> /usr/bin or whether it belongs to root. Tying into the system's package
> manager doesn't look appealing to me (not to mention that it might be
> unacceptably slow).

Let's try a specific example. Let's say that PyQt6 v6.8.3 is installed 
in the system site directory. The OS uses PyQt6 for some system 
purposes. Now the user comes along and forces an install of PyQt6 v6.9.0 
in the user site directory. 6.9.0 has a bug that would crash one of the 
system's application but not the user's programs. (This is not a 
far-fetched scenario).

When the system launches its application the PYTHONPATH will start with 
system site directories; local user site directories will be on the 
PYTHONPATH but since they come later, the python will use PyQt6 v6.8.3 
because that will come first on the path.  No crash here.

If the user has a program that actually does require the use of v6.9.0, 
he's going to have to make sure that the user's local site directories 
come first on the path. One way to do that is to set the PYTHONPATH to 
point to the user's location.

In what scenario is a system application going to load a wrong version 
of a dependency from a user's site location? The only one I can think of 
is for the user, with the help of sudo, or by editing some 
system-enabled script, were to change the global PYTHONPATH.  That seems 
a stretch.

[toc] | [prev] | [next] | [standalone]

#197419

[Multipart message — attachments visible in raw view] — view raw

On 2025-04-18 17:11:33 +0100, Oscar Benjamin via Python-list wrote:
> On Fri, 18 Apr 2025 at 16:50, Peter J. Holzer via Python-list
> <python-list@python.org> wrote:
> >
> > On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:
> > > On 18/04/25 9:41 am, Mats Wichmann wrote:
> > > > There's just not a really great answer to this.
> > >
> > > Seems to me a system-installed application shouldn't be looking in the
> > > user's .local packages in the first place. That should only be for things
> > > the user has installed "for this user".
> >
> > It's not the application that looks into .local, it's Python. If you say
> > that a system-installed Python shouldn't look into ~/.local, then you've
> > just disabled that mechanism completely. If not then Python would
> > somehow have to distinguish between system-installed and user-installed
> > scripts. This isn't as easy as checking whether the path starts with
> > /usr/bin or whether it belongs to root. Tying into the system's package
> > manager doesn't look appealing to me (not to mention that it might be
> > unacceptably slow).
> 
> Couldn't the system-installed scripts have a shebang like:
> 
> #!/usr/bin/python3 -s

Yes, that should work. At least I can't think of any downsides at the
moment.

        hjp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp@hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

[toc] | [prev] | [next] | [standalone]

#197420

[Multipart message — attachments visible in raw view] — view raw

On 2025-04-18 13:08:36 -0400, Thomas Passin via Python-list wrote:
> On 4/18/2025 11:38 AM, Peter J. Holzer via Python-list wrote:
> > On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:
> > > On 18/04/25 9:41 am, Mats Wichmann wrote:
> > > > There's just not a really great answer to this.
> > > 
> > > Seems to me a system-installed application shouldn't be looking in the
> > > user's .local packages in the first place. That should only be for things
> > > the user has installed "for this user".
> > 
> > It's not the application that looks into .local, it's Python. If you say
> > that a system-installed Python shouldn't look into ~/.local, then you've
> > just disabled that mechanism completely. If not then Python would
> > somehow have to distinguish between system-installed and user-installed
> > scripts. This isn't as easy as checking whether the path starts with
> > /usr/bin or whether it belongs to root. Tying into the system's package
> > manager doesn't look appealing to me (not to mention that it might be
> > unacceptably slow).
> 
> Let's try a specific example. Let's say that PyQt6 v6.8.3 is installed in
> the system site directory. The OS uses PyQt6 for some system purposes. Now
> the user comes along and forces an install of PyQt6 v6.9.0 in the user site
> directory. 6.9.0 has a bug that would crash one of the system's application
> but not the user's programs. (This is not a far-fetched scenario).

Agreed. Also since PyQt6 is a GUI framework, I'm going to assume that
those applications are supposed to be invoked by the user on their
desktop, not by a cronjob controlled by the system.


> When the system launches its application the PYTHONPATH will start with
> system site directories; local user site directories will be on the
> PYTHONPATH but since they come later, the python will use PyQt6 v6.8.3
> because that will come first on the path.  No crash here.
> 
> If the user has a program that actually does require the use of v6.9.0, he's
> going to have to make sure that the user's local site directories come first
> on the path. One way to do that is to set the PYTHONPATH to point to the
> user's location.

This is IMHO not practical. The user would have to set PYTHONPATH for
some programs, but not for others. You can't do this with .bashrc (or
similar), the user would have to write a wrapper script for each of
their programs which depend on something in ~/.local. Possible of course
but cumbersome.

I like Oscar's suggestion that Python scripts provided by the
distribution include -s in the shebang much better.

Or - what I tend to do - simply use a virtual environment for each
script that needs a package that the system doesn't provide. But of
course that's basically "disable/don't use .local" and all those venvs
take space and need to be maintained.

> In what scenario is a system application going to load a wrong version of a
> dependency from a user's site location?

When the user sets PYTHONPATH and then accidentally invokes the
system-supplied application without resetting it first.

> The only one I can think of is for the user, with the help of sudo, or
> by editing some system-enabled script, were to change the global
> PYTHONPATH.  That seems a stretch.

No, there doesn't have to be a global (in the sense that it applies to
all users) PYTHONPATH for that to happen. You just need a PYTHONPATH
that is set for all processes of that user - which the user can
certainly set without sudo (usually by editing .bashrc or maybe using
their desktop environment's settings dialog).

        hjp

-- 
   _  | Peter J. Holzer    | Story must make more sense than reality.
|_|_) |                    |
| |   | hjp@hjp.at         |    -- Charles Stross, "Creative writing
__/   | http://www.hjp.at/ |       challenge!"

[toc] | [prev] | [next] | [standalone]

#197421

Peter J. Holzer wrote:
> On 2025-04-18 13:08:36 -0400, Thomas Passin via Python-list wrote:
...
>> When the system launches its application the PYTHONPATH will start with
>> system site directories; local user site directories will be on the
>> PYTHONPATH but since they come later, the python will use PyQt6 v6.8.3
>> because that will come first on the path.  No crash here.
>> 
>> If the user has a program that actually does require the use of v6.9.0, he's
>> going to have to make sure that the user's local site directories come first
>> on the path. One way to do that is to set the PYTHONPATH to point to the
>> user's location.
>
> This is IMHO not practical. The user would have to set PYTHONPATH for
> some programs, but not for others. You can't do this with .bashrc (or
> similar), the user would have to write a wrapper script for each of
> their programs which depend on something in ~/.local. Possible of course
> but cumbersome.

  currently in my .bashrc i have it set up to look for which
directory the terminal is in and then runs the activate script
for that environment.  i like to keep my desktops/projects
open with various numbers of terminals available so this way
they are all ready to go when the system boots up.


> I like Oscar's suggestion that Python scripts provided by the
> distribution include -s in the shebang much better.
>
> Or - what I tend to do - simply use a virtual environment for each
> script that needs a package that the system doesn't provide. But of
> course that's basically "disable/don't use .local" and all those venvs
> take space and need to be maintained.

  i like that they do not change until i specificly ask them to
be changed.


...
>> The only one I can think of is for the user, with the help of sudo, or
>> by editing some system-enabled script, were to change the global
>> PYTHONPATH.  That seems a stretch.
>
> No, there doesn't have to be a global (in the sense that it applies to
> all users) PYTHONPATH for that to happen. You just need a PYTHONPATH
> that is set for all processes of that user - which the user can
> certainly set without sudo (usually by editing .bashrc or maybe using
> their desktop environment's settings dialog).

  yes, that is part of what .bashrc is for, making sure your
environment variables are set how you'd like them.


  songbird

[toc] | [prev] | [next] | [standalone]

#197422

On 4/19/2025 4:56 AM, Peter J. Holzer via Python-list wrote:
> On 2025-04-18 13:08:36 -0400, Thomas Passin via Python-list wrote:
>> On 4/18/2025 11:38 AM, Peter J. Holzer via Python-list wrote:
>>> On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:
>>>> On 18/04/25 9:41 am, Mats Wichmann wrote:
>>>>> There's just not a really great answer to this.
>>>>
>>>> Seems to me a system-installed application shouldn't be looking in the
>>>> user's .local packages in the first place. That should only be for things
>>>> the user has installed "for this user".
>>>
>>> It's not the application that looks into .local, it's Python. If you say
>>> that a system-installed Python shouldn't look into ~/.local, then you've
>>> just disabled that mechanism completely. If not then Python would
>>> somehow have to distinguish between system-installed and user-installed
>>> scripts. This isn't as easy as checking whether the path starts with
>>> /usr/bin or whether it belongs to root. Tying into the system's package
>>> manager doesn't look appealing to me (not to mention that it might be
>>> unacceptably slow).
>>
>> Let's try a specific example. Let's say that PyQt6 v6.8.3 is installed in
>> the system site directory. The OS uses PyQt6 for some system purposes. Now
>> the user comes along and forces an install of PyQt6 v6.9.0 in the user site
>> directory. 6.9.0 has a bug that would crash one of the system's application
>> but not the user's programs. (This is not a far-fetched scenario).
> 
> Agreed. Also since PyQt6 is a GUI framework, I'm going to assume that
> those applications are supposed to be invoked by the user on their
> desktop, not by a cronjob controlled by the system.
> 
> 
>> When the system launches its application the PYTHONPATH will start with
>> system site directories; local user site directories will be on the
>> PYTHONPATH but since they come later, the python will use PyQt6 v6.8.3
>> because that will come first on the path.  No crash here.
>>
>> If the user has a program that actually does require the use of v6.9.0, he's
>> going to have to make sure that the user's local site directories come first
>> on the path. One way to do that is to set the PYTHONPATH to point to the
>> user's location.
> 
> This is IMHO not practical. The user would have to set PYTHONPATH for
> some programs, but not for others. You can't do this with .bashrc (or
> similar), the user would have to write a wrapper script for each of
> their programs which depend on something in ~/.local. Possible of course
> but cumbersome.
> 
> I like Oscar's suggestion that Python scripts provided by the
> distribution include -s in the shebang much better.
> 
> Or - what I tend to do - simply use a virtual environment for each
> script that needs a package that the system doesn't provide. But of
> course that's basically "disable/don't use .local" and all those venvs
> take space and need to be maintained.

My problem with venvs, especially if I have more than one, is that I 
eventually forget what they were for and what is different about each 
one. If there's only one and it's used for all non-system work, that's 
OK but beyond that and they tend to suffer knowledge rot.

> 
>> In what scenario is a system application going to load a wrong version of a
>> dependency from a user's site location?
> 
> When the user sets PYTHONPATH and then accidentally invokes the
> system-supplied application without resetting it first.
> 
>> The only one I can think of is for the user, with the help of sudo, or
>> by editing some system-enabled script, were to change the global
>> PYTHONPATH.  That seems a stretch.
> 
> No, there doesn't have to be a global (in the sense that it applies to
> all users) PYTHONPATH for that to happen. You just need a PYTHONPATH
> that is set for all processes of that user - which the user can
> certainly set without sudo (usually by editing .bashrc or maybe using
> their desktop environment's settings dialog).

Yes, I agree and I wasn't thinking locally enough.

[toc] | [prev] | [next] | [standalone]

#197423

On Sat, 19 Apr 2025 15:56:16 -0400, Thomas Passin wrote:

> My problem with venvs, especially if I have more than one, is that I
> eventually forget what they were for and what is different about each
> one. If there's only one and it's used for all non-system work, that's
> OK but beyond that and they tend to suffer knowledge rot.

My Python directory has

apple/   create/  fastapi/  lunar/  numerical/  pyside6/  weather/
comics/  django/  folium/   ml/  sqlite/  coursera/  impractical/
nn/  pyqt/  torch/

Not all like sqlite are venvs since no additional modules are needed. Even 
if I'm a little hazy about 'apple' after a while a quick look at the 
Python file shows 'https://itunes.apple.com/search' and I remember it is 
to pull down artist/track info from the itunes database.

I also try to remember to run 'pythom -m pip freeze > requirements.txt' 
after I get everything set up. That way if a OS update installs a new 
version of Python  something like

python3 -m venv folium
cd folium
. bin/activate
python -m pip install -r requirements.txt

will refresh the venv and install the appropriate packages for the new 
Python version. That's faster than looking at lib/python3.10/site-packages 
or whatever the previous version was and adding the dependencies  or 
trying to run the py file and getting

Traceback (most recent call last):
  File "/home/rbowman/work/python/impractical/benford.py", line 5, in 
<module>
    import matplotlib.pyplot as plt
ModuleNotFoundError: No module named 'matplotlib'

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.python

csiph-web