Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #15126 > unrolled thread

Uncleaned egg-info directory giving lots of bugs about failing to build after successful build

Back to article view | Back to linux.debian.maint.python

Contents

  Uncleaned egg-info directory giving lots of bugs about failing to  build after successful build Julian Gilbey <julian@d-and-j.net> - 2023-08-18 15:10 +0200
    Re: Uncleaned egg-info directory giving lots of bugs about failing  to build after successful build Andreas Tille <andreas@an3as.eu> - 2023-08-18 15:20 +0200
      Re: Uncleaned egg-info directory giving lots of bugs about failing  to build after successful build Scott Talbert <swt@techie.net> - 2023-08-18 15:30 +0200
        Re: Uncleaned egg-info directory giving lots of bugs about failing  to build after successful build Julian Gilbey <julian@d-and-j.net> - 2023-08-18 15:50 +0200
    Re: Uncleaned egg-info directory giving lots of bugs about failing  to build after successful build Timo Röhling <roehling@debian.org> - 2023-08-18 15:30 +0200
    Re: Uncleaned egg-info directory giving lots of bugs about failing to  build after successful build Thomas Goirand <zigo@debian.org> - 2023-09-05 14:20 +0200
      Re: Uncleaned egg-info directory giving lots of bugs about failing  to build after successful build Jeremy Stanley <fungi@yuggoth.org> - 2023-09-06 15:20 +0200
          Re: Uncleaned egg-info directory giving lots of bugs about failing to build  after successful build Soren Stoutner <soren@stoutner.com> - 2023-09-06 17:50 +0200
          Re: Uncleaned egg-info directory giving lots of bugs about failing to build after successful build Scott Kitterman <debian@kitterman.com> - 2023-09-06 18:00 +0200
          Re: Uncleaned egg-info directory giving lots of bugs about failing  to build after successful build Julian Gilbey <julian@d-and-j.net> - 2023-09-07 11:20 +0200
          Re: Uncleaned egg-info directory giving lots of bugs about failing to  build after successful build Stuart Prescott <stuart@debian.org> - 2023-09-07 11:20 +0200

#15126 — Uncleaned egg-info directory giving lots of bugs about failing to build after successful build

I'm sure I'm not the only one who received a whole bunch of bugs
entitled "Fails to build source after successful build" last weekend.
There was one theme common to most of them: the presence of a
*.egg-info directory which was not cleaned by debian/rules clean.

I know the bug report said that this policy is currently under
discussion, but I did get thinking about it.  I imagine that this
particular directory should be the responsibility of dh-python to
clean up, but it may not be sensible to always delete *.egg-info
directories, as they may be present in the orig.tar.gz file.  One
could handle it by manually adding this directory to debian/clean in
each package, but perhaps this should be the default behaviour of
dh-python?

Any thoughts?

Best wishes,

   Julian

[toc] | [next] | [standalone]

#15127 — Re: Uncleaned egg-info directory giving lots of bugs about failing to build after successful build

Hi,

Am Fri, Aug 18, 2023 at 01:42:53PM +0100 schrieb Julian Gilbey:
> I'm sure I'm not the only one who received a whole bunch of bugs
> entitled "Fails to build source after successful build" last weekend.
> There was one theme common to most of them: the presence of a
> *.egg-info directory which was not cleaned by debian/rules clean.
> 
> I know the bug report said that this policy is currently under
> discussion, but I did get thinking about it.  I imagine that this
> particular directory should be the responsibility of dh-python to
> clean up, but it may not be sensible to always delete *.egg-info
> directories, as they may be present in the orig.tar.gz file.  One
> could handle it by manually adding this directory to debian/clean in
> each package, but perhaps this should be the default behaviour of
> dh-python?
> 
> Any thoughts?

I agree that having this a no-brainer and getting all this *.egg-info
caused bugs solved by a simple dh-python update without changing
packaging code would be extremely convenient.  I could imagine creating
a backup of the affected *.egg-info files if existent and copy these
back in clean target could solve this.

Kind regards
   Andreas.

-- 
http://fam-tille.de

[toc] | [prev] | [next] | [standalone]

#15129 — Re: Uncleaned egg-info directory giving lots of bugs about failing to build after successful build

On Fri, 18 Aug 2023, Andreas Tille wrote:

> Am Fri, Aug 18, 2023 at 01:42:53PM +0100 schrieb Julian Gilbey:
>> I'm sure I'm not the only one who received a whole bunch of bugs
>> entitled "Fails to build source after successful build" last weekend.
>> There was one theme common to most of them: the presence of a
>> *.egg-info directory which was not cleaned by debian/rules clean.
>>
>> I know the bug report said that this policy is currently under
>> discussion, but I did get thinking about it.  I imagine that this
>> particular directory should be the responsibility of dh-python to
>> clean up, but it may not be sensible to always delete *.egg-info
>> directories, as they may be present in the orig.tar.gz file.  One
>> could handle it by manually adding this directory to debian/clean in
>> each package, but perhaps this should be the default behaviour of
>> dh-python?
>>
>> Any thoughts?
>
> I agree that having this a no-brainer and getting all this *.egg-info
> caused bugs solved by a simple dh-python update without changing
> packaging code would be extremely convenient.  I could imagine creating
> a backup of the affected *.egg-info files if existent and copy these
> back in clean target could solve this.

It is being worked on:
https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/46

Regards,
Scott

[toc] | [prev] | [next] | [standalone]

#15131 — Re: Uncleaned egg-info directory giving lots of bugs about failing to build after successful build

On Fri, Aug 18, 2023 at 09:23:17AM -0400, Scott Talbert wrote:
> On Fri, 18 Aug 2023, Andreas Tille wrote:
> 
> > Am Fri, Aug 18, 2023 at 01:42:53PM +0100 schrieb Julian Gilbey:
> > > I'm sure I'm not the only one who received a whole bunch of bugs
> > > entitled "Fails to build source after successful build" last weekend.
> > > There was one theme common to most of them: the presence of a
> > > *.egg-info directory which was not cleaned by debian/rules clean.
> > > [...]
> 
> It is being worked on:
> https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/46

Amazing!

Thanks,

   Julian

[toc] | [prev] | [next] | [standalone]

#15128 — Re: Uncleaned egg-info directory giving lots of bugs about failing to build after successful build

[Multipart message — attachments visible in raw view] — view raw

* Julian Gilbey <julian@d-and-j.net> [2023-08-18 13:42]:
>One could handle it by manually adding this directory to
>debian/clean in each package, but perhaps this should be the
>default behaviour of dh-python?
See https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/46


Cheers
Timo

-- 
⢀⣴⠾⠻⢶⣦⠀   ╭────────────────────────────────────────────────────╮
⣾⠁⢠⠒⠀⣿⡁   │ Timo Röhling                                       │
⢿⡄⠘⠷⠚⠋⠀   │ 9B03 EBB9 8300 DF97 C2B1  23BF CC8C 6BDD 1403 F4CA │
⠈⠳⣄⠀⠀⠀⠀   ╰────────────────────────────────────────────────────╯

[toc] | [prev] | [next] | [standalone]

#15154

On 8/18/23 14:42, Julian Gilbey wrote:
> I'm sure I'm not the only one who received a whole bunch of bugs
> entitled "Fails to build source after successful build" last weekend.
> There was one theme common to most of them: the presence of a
> *.egg-info directory which was not cleaned by debian/rules clean.
> 
> I know the bug report said that this policy is currently under
> discussion

As much as I know, there's no controversy: this really is a bug in 
packages. However, the discussion in d-devel has shown it's a minor 
issue, as everyone has switch to use Git for packaging.

>, but I did get thinking about it.  I imagine that this
> particular directory should be the responsibility of dh-python to
> clean up, but it may not be sensible to always delete *.egg-info
> directories, as they may be present in the orig.tar.gz file.

They should not. If you're packaging from pypi, you should switch to 
using upstream VCS (most Python modules are in github). Otherwise, one 
possibility is to manually (well, through debian/watch) remove the 
egg-info from the orig.tar.gz.

> One
> could handle it by manually adding this directory to debian/clean in
> each package

I used to add this to all of my packages:

$ cat debian/source/options
extend-diff-ignore = "^[^/]*[.]egg-info/"

This works, and actually fixes the above mentioned bugs. However, as I 
didn't want to have a single remaining instance of this bad clean-up, I 
have setup my sbuild to check on it. The config is over here:

https://wiki.debian.org/zigo/mysbuild#A.2BAH4-.2F.sbuildrc

See the $external_commands thingy. This does a diff between the source 
package, and the result after build + clean. It does report files from 
the egg-info. Therefore, I did add rm -rf *.egg-info in all affected 
packages, so I can continue to use the $external_commands checks. I 
would strongly recommend every DD to do the same thing.

Yes, we can have dh-python to do the work, but IMO, the only thing it 
should be doing, is rm -rf *.egg-info, and error out if the egg-info is 
within the orig tarball, as this should not happen, IMO.

Now, I still think this is a minor issue... :)

Cheers,

Thomas Goirand (zigo)

[toc] | [prev] | [next] | [standalone]

#15155 — Re: Uncleaned egg-info directory giving lots of bugs about failing to build after successful build

[Multipart message — attachments visible in raw view] — view raw

On 2023-09-05 14:16:55 +0200 (+0200), Thomas Goirand wrote:
[...]
> Yes, we can have dh-python to do the work, but IMO, the only thing
> it should be doing, is rm -rf *.egg-info, and error out if the
> egg-info is within the orig tarball, as this should not happen,
> IMO.
[...]

See
https://salsa.debian.org/python-team/tools/dh-python/-/commit/31eff8f
which merged last week.
-- 
Jeremy Stanley

[toc] | [prev] | [next] | [standalone]

#15156 — Re: Uncleaned egg-info directory giving lots of bugs about failing to build after successful build

[Multipart message — attachments visible in raw view] — view raw

As a followup question, I have noticed that a lot of packages (including electrum, which I 
have recently started maintaining) ship the egg-info directory.  Looking through /usr/lib/
python3/dist-packages/, this is common but not universal.  Is there any reason to ship this 
directory or should it be removed from the binary packages?

Also, the commit linked to below states:

"if the egg-info directory is included in the upstream source, then
  deleting it causes no harm as dpkg-source warns on missing files in
  the source package but does not exit with error.”

It might not cause any problems for dpkg-source, but won’t it cause issues for double 
builds?  For reference:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1044268[1]

If so, any package that ships the egg-info directory in the tarball will get flagged with a 
FTBFS bug, which might be the desired behavior if these shouldn’t be in the tarball anyway.

On Wednesday, September 6, 2023 5:52:18 AM MST Jeremy Stanley wrote:
> On 2023-09-05 14:16:55 +0200 (+0200), Thomas Goirand wrote:
> [...]
> 
> > Yes, we can have dh-python to do the work, but IMO, the only thing
> > it should be doing, is rm -rf *.egg-info, and error out if the
> > egg-info is within the orig tarball, as this should not happen,
> > IMO.
> 
> [...]
> 
> See
> https://salsa.debian.org/python-team/tools/dh-python/-/commit/31eff8f
> which merged last week.


-- 
Soren Stoutner
soren@stoutner.com

--------
[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1044268

[toc] | [prev] | [next] | [standalone]

#15157 — Re: Uncleaned egg-info directory giving lots of bugs about failing to build after successful build

On September 6, 2023 3:05:45 PM UTC, Soren Stoutner <soren@stoutner.com> wrote:
>As a followup question, I have noticed that a lot of packages (including electrum, which I 
>have recently started maintaining) ship the egg-info directory.  Looking through /usr/lib/
>python3/dist-packages/, this is common but not universal.  Is there any reason to ship this 
>directory or should it be removed from the binary packages?
>
We should ship them.  Debian packages don't use them, but upstream Python dependency resolution systems do.  Including them allows our Python packages to satisfy dependencies for packages installed by our users via pip or other methods.

Scott K

[toc] | [prev] | [next] | [standalone]

#15158 — Re: Uncleaned egg-info directory giving lots of bugs about failing to build after successful build

On Wed, Sep 06, 2023 at 08:05:45AM -0700, Soren Stoutner wrote:
> As a followup question, I have noticed that a lot of packages (including
> electrum, which I have recently started maintaining) ship the egg-info
> directory.  Looking through /usr/lib/python3/dist-packages/, this is common but
> not universal.  Is there any reason to ship this directory or should it be
> removed from the binary packages?

Lots of packages depend on the egg-info directory being present.

$ grep EASY-INSTALL-ENTRY-SCRIPT /usr/bin/*

will give a (probably very long) list of executables that depend on an
egg-info (or equivalent) directory being present.

Best wishes,

   Julian

[toc] | [prev] | [next] | [standalone]

#15159

Hi Soren

On 07/09/2023 01:05, Soren Stoutner wrote:
> Also, the commit linked to below states:
> 
> "if the egg-info directory is included in the upstream source, then
>    deleting it causes no harm as dpkg-source warns on missing files in
>    the source package but does not exit with error.”
> 
> It might not cause any problems for dpkg-source, but won’t it cause 
> issues for double builds?  For reference:
> 
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1044268 
> <https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1044268>
> 
> If so, any package that ships the egg-info directory in the tarball will 
> get flagged with a FTBFS bug, which might be the desired behavior if 
> these shouldn’t be in the tarball anyway.

No, the opposite is true.

They _currently_ FTBFS when built twice in a row because:

* the build modifies the egg-info files
* the change to the egg-info is not undone by the clean step that is
   run between the two builds
* dpkg-source doesn't want modified upstream files and so exits
   with error

With the change you referred to, these packages do not FTBFS when built 
twice in a row because:

* the build modifies the egg-info files
* the egg-info is removed in the clean step between that runs between
   the two build attempts
* dpkg-source doesn't care about removed upstream files and so
   completes its task without error (it warns about them, that's all)
* the build proceeds and regenerates the egg-info

It was pointed out on IRC today that some packages (like poetry) ship 
egg-info files that are part of the test suite and so are not 
regenerated... and they should not be deleted. That's the next task.

regards
Stuart



-- 
Stuart Prescott   http://www.nanonanonano.net/ stuart@nanonanonano.net
Debian Developer  http://www.debian.org/       stuart@debian.org
GPG fingerprint   90E2 D2C1 AD14 6A1B 7EBB 891D BBC1 7EBB 1396 F2F7

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web