Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.python > #8575 > unrolled thread

Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis

Back to article view | Back to linux.debian.maint.python

Contents

  Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for  log checking, log analysis Fiedler Roman <Roman.Fiedler@ait.ac.at> - 2016-06-09 09:50 +0200
    Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for  log checking, log analysis Gianfranco Costamagna <locutusofborg@debian.org> - 2016-06-09 10:40 +0200
      AW: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for  log checking, log analysis Fiedler Roman <Roman.Fiedler@ait.ac.at> - 2016-06-09 11:50 +0200
    Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for  log checking, log analysis Piotr Ożarowski <piotr@debian.org> - 2016-06-09 10:40 +0200
      AW: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for  log checking, log analysis Fiedler Roman <Roman.Fiedler@ait.ac.at> - 2016-06-09 13:30 +0200
        Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for  log checking, log analysis Piotr Ożarowski <piotr@debian.org> - 2016-06-09 14:40 +0200

#8575 — Re: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis

[Multipart message — attachments visible in raw view] — view raw

Hi Gianfranco, hello Python devs,


Introduction for debian-python members: Gianfranco is giving me great 
assistance in the mentoring process to get the logdata-anomaly-miner package 
included to Debian. There were some issues, we are not completely sure, how to 
sort them out, any help on that would be appreciated!

Sorry for drawing your attention to that in the middle of the running 
discussion. You can find the whole discussion in the ITP bug report 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=813096 . Perhaps reading the 
whole issue would not justify the time spent, so from my opinion most 
important 2 topics and related questions are:

* Rationale not using "usr/lib/python*/": See this mail, first inline reply 
section. Q: Are the arguments relevant? Move code or not?

* dh_python2 woes: Beginning of it, see Bugreport " Fri, 3 Jun 2016 18:54:16 
+0000". Idea was to share as much code as possible between python2.6/python2.7 
systems, but somehow this does not work out with packaging/lintian. Q: is 
building a 2.6/2.7 package even a good idea or should it be avoided? Instead 
upstream code and Debian packaging rules/patches could be optimized for 
generating different packages for those versions.

The package in question can be found at 
https://mentors.debian.net/package/logdata-anomaly-miner


> Von: Gianfranco Costamagna [mailto:locutusofborg@debian.org]
> Hi
>
> >I had already added it already to
> >* control
> >Depends: ${python:Depends}, python-tz, ${misc:Depends}
>
> >which was mangled (with warning) to:
>
> maybe you are installing the files in usr/lib/foo, instead of 
> usr/lib/python*/
> and then dh_python2 is not acting correctly?

Moving the code to "/usr/lib/python2.7/dist-packages/aminer" in fact allows 
dh_python2 to extract the version information:

Depends: python:any (<< 2.8), python:any (>= 2.7.5-5~), python-tz, 
init-system-helpers (>= 1.18~)

(Remark: Is there any reason to restrict the versions to >=2.7.5? The tools 
should have compatibility with >=2.6 and I would expect the "Depends" section 
to somehow reflect that reality. Is DEBPYTHON_SUPPORTED and DEBPYTHON_DEFAULT 
intended to be used to fix that?)


But doing the move, lintian will not like the produced package any more:

E: logdata-anomaly-miner: python-script-but-no-python-dep 
usr/lib/aminer/AMiner


> I don't think installing python stuff outside that directory is a good 
> idea...

The rationale behind not putting aminer into dist-packages (and removing 
dist-packages from python-path) was:

a) require the aminer-admin to "link" in each dist-packages module separately, 
thus reminding it, that it is no good idea to include large amounts of third 
party libraries in security-critical code. Code volume is also bugs and 
attack-surface.

b) prepare against possible future risks due to accidental loading (call to 
__init__) of other packages residing in dist-packages, that may give rise to 
privilege escalation (like the GNU-TLS CVE from this/last week)

Of course, it should be possible to move the code to 
/usr/lib/python2.7/dist-packages/aminer and perform the "link" operation, but 
this could make it more "sexy" for an admin to include whole dist-packages in 
python path again.


In that light, should the code be moved?

Apart from that, this will also make it harder to use the same codebase for 
both python2.6/python2.7, but that should be fixable by providing more 
patches.


> >@@ -1,4 +1,4 @@
> >-#!/usr/bin/python2 -BEsSt
> >+#!/usr/bin/python2.7 -BEsSt
>
> this should be also handled by dh_python2 AFAIK

Even with move dh_python2 does not touch the files. The only difference is, 
that lintian does not like the files any more.

> >Everything seems to work without it also and the same package could have
> been
> >used for python2.6 and python2.7 systems but lintian does not understand
> that.
>
> well, installing into usr/lib/python* should fix that issue

So it would be OK to place the files in usr/lib/python2.7/dist-packages and 
add a symlink to usr/lib/python2.6/dist-packages?

> I would prefer you to ask on debian-python about this issue, because I'm
> wondering about something bad that is just hidden and will be spot when the 
> package
> enters the
> archive.
>
> having an ack for a more python-savvy person would be great
> (I could even sponsor the package as-is, but I'm not too confident with 
> that)

I would not like to push for that. Let's take the time and learn. Also this 
package might be the template for other security-tool packages, so better have 
it clean before cloning.

> >I hope this fixed all the review comments for regarding the packaging. I
> would
> >like to keep the deeper restructuring of upstream code + Debian packaging
> >scripts for upstream/Debian release V0.0.3. Is that OK from Debian side?
>
> it is ok to have new uploads, don't worry about that, I just would like to 
> have a
> feedback from another DD before uploading/finish the review.
>
> is it possible for you?
> otherwise let me know and I'll finish the review and do the final
> checks+upload.

Understood. It might be, that this is all not a big issue for the python-dev 
pros, might be sorted out quickly.

Best regards,
Roman

[toc] | [next] | [standalone]

#8576

Hi,

(answering where I can!)
>Moving the code to "/usr/lib/python2.7/dist-packages/aminer" in fact allows 

>dh_python2 to extract the version information:
>
>Depends: python:any (<< 2.8), python:any (>= 2.7.5-5~), python-tz, 
>init-system-helpers (>= 1.18~)
>
>(Remark: Is there any reason to restrict the versions to >=2.7.5? The tools 
>should have compatibility with >=2.6 and I would expect the "Depends" section 
>to somehow reflect that reality. Is DEBPYTHON_SUPPORTED and DEBPYTHON_DEFAULT 
>intended to be used to fix that?)


that stuff is built for unstable/testing, so the variables are filled
with the current python situation
(2.6 is only on old-stable, and Stretch has 2.7.11 already).

Probably if you try to build it with a jessie chroot, you get different values,
and this is correct.
(you can't in general install deb built against stretch into wheezy/jessie, without
breaking stuff, this is why some dependencies are not too relaxed, to avoid people
doing that, but instead upgrading the minimal set of packages to make sure things will
work after the apt-pinning).

I don't foresee any issue here, because even in case of a backport, the package will need to
be rebuilt on top of that.

>But doing the move, lintian will not like the produced package any more:
>
>E: logdata-anomaly-miner: python-script-but-no-python-dep 
>usr/lib/aminer/AMiner


I can't say, I don't see the package installing stuff in usr/lib/python* on mentors

>The rationale behind not putting aminer into dist-packages (and removing 
>dist-packages from python-path) was:
>
>a) require the aminer-admin to "link" in each dist-packages module separately, 
>thus reminding it, that it is no good idea to include large amounts of third 
>party libraries in security-critical code. Code volume is also bugs and 
>attack-surface.


I didn't get completely this, but I leave other people to answer

>b) prepare against possible future risks due to accidental loading (call to 
>__init__) of other packages residing in dist-packages, that may give rise to 
>privilege escalation (like the GNU-TLS CVE from this/last week)


mmm you want to avoid people importing your library?
ok

>Of course, it should be possible to move the code to 
>/usr/lib/python2.7/dist-packages/aminer and perform the "link" operation, but 
>this could make it more "sexy" for an admin to include whole dist-packages in 
>python path again.
>In that light, should the code be moved?


leaving to other folks the answer
>Apart from that, this will also make it harder to use the same codebase for 
>both python2.6/python2.7, but that should be fixable by providing more 
>patches.


you can't use python2.6 on Stretch, so no issue here.

>> >-#!/usr/bin/python2 -BEsSt
>> >+#!/usr/bin/python2.7 -BEsSt
>>
>> this should be also handled by dh_python2 AFAIK
>
>Even with move dh_python2 does not touch the files. The only difference is, 
>that lintian does not like the files any more.


can you please double check with the documentation?
https://wiki.debian.org/Python/LibraryStyleGuide
>So it would be OK to place the files in usr/lib/python2.7/dist-packages and 

>add a symlink to usr/lib/python2.6/dist-packages?

no, python2.6 is dead, no need to add it here.

>I would not like to push for that. Let's take the time and learn. Also this 
>package might be the template for other security-tool packages, so better have 
>it clean before cloning.


thanks for understanding!

>Understood. It might be, that this is all not a big issue for the python-dev >pros, might be sorted out quickly.


I hope so, I understand the package is doing some non-standard things, so trying
to make it python-library style might be even a bad idea in general, this is why
I would prefer to have some double/triple checks here ;)

thanks for the nice email!

G.

[toc] | [prev] | [next] | [standalone]

#8578 — AW: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis

[Multipart message — attachments visible in raw view] — view raw

> Von: Gianfranco Costamagna [mailto:locutusofborg@debian.org]
> Hi,
> 
> (answering where I can!)
>> Moving the code to "/usr/lib/python2.7/dist-packages/aminer" in fact allows
>> dh_python2 to extract the version information:
>>
>> Depends: python:any (<< 2.8), python:any (>= 2.7.5-5~), python-tz,
>> init-system-helpers (>= 1.18~)
>>
>> (Remark: Is there any reason to restrict the versions to >=2.7.5? The tools
>> should have compatibility with >=2.6 and I would expect the "Depends"
>> section
>> to somehow reflect that reality. Is DEBPYTHON_SUPPORTED and
>> DEBPYTHON_DEFAULT intended to be used to fix that?)
> 
> that stuff is built for unstable/testing, so the variables are filled
> with the current python situation
> (2.6 is only on old-stable, and Stretch has 2.7.11 already).
>
> Probably if you try to build it with a jessie chroot, you get different values, and this is correct.
> (you can't in general install deb built against stretch into wheezy/jessie, without
> breaking stuff, this is why some dependencies are not too relaxed, to avoid people
> doing that, but instead upgrading the minimal set of packages to make sure
> things will work after the apt-pinning).
> 
> I don't foresee any issue here, because even in case of a backport, the
> package will need to be rebuilt on top of that.

OK, so I will not attempt to fiddle with that and stick to the stricter versions.

> >But doing the move, lintian will not like the produced package any more:
> >
> >E: logdata-anomaly-miner: python-script-but-no-python-dep usr/lib/aminer/AMiner
> 
> I can't say, I don't see the package installing stuff in usr/lib/python* on
> mentors

Maybe the hints from Piotr Ożarowski's followup mail will fix that, using his hints.

> >The rationale behind not putting aminer into dist-packages (and removing
> >dist-packages from python-path) was:
>> [Snip]
> 
> >b) prepare against possible future risks due to accidental loading (call to
> >__init__) of other packages residing in dist-packages, that may give rise to
> >privilege escalation (like the GNU-TLS CVE from this/last week)
> 
> mmm you want to avoid people importing your library?
>
> ok

Not directly avoid, but make them think more about it before adding. I also remember having issues with some svg-python library, that starts to misbehave as soon as other packages are installed because of "try: import xxx; ..." blocks. Just want to reduce the risks.
 
> >Of course, it should be possible to move the code to
> >/usr/lib/python2.7/dist-packages/aminer and perform the "link" operation,
> but
> >this could make it more "sexy" for an admin to include whole dist-packages
> in
> >python path again.
> >In that light, should the code be moved?
> 
> leaving to other folks the answer
> >Apart from that, this will also make it harder to use the same codebase for
> >both python2.6/python2.7, but that should be fixable by providing more
> >patches.
> 
> you can't use python2.6 on Stretch, so no issue here.

Understood, dropping the 2.6/2.7 compatibility stuff
 
> >> >-#!/usr/bin/python2 -BEsSt
> >> >+#!/usr/bin/python2.7 -BEsSt
> >>
> >> this should be also handled by dh_python2 AFAIK
> >
> >Even with move dh_python2 does not touch the files. The only difference is,
> >that lintian does not like the files any more.
> 
> can you please double check with the documentation?
> https://wiki.debian.org/Python/LibraryStyleGuide

The "python2" selection in the binaries was effect of reading it, but perhaps I got something wrong:

Intro: "It (the LibraryStyleGuide) is not intended to supplant the Debian Python Policy and in fact, if you have not read that document, stop now and go read it"

And from " Debian Python Policy"
https://www.debian.org/doc/packaging-manuals/python-policy/ch-python.html#s-interpreter_name

"Python scripts that require the default Python 2 version should specify python2 as the interpreter name."

So should be OK in my opinion.

> [Snip]

[toc] | [prev] | [next] | [standalone]

#8577

just a quick reply:

* private dir is the right call, do not install into dist-packages
  (only "python-*" binary packages should install there)
* install into /usr/lib/logdata-anomaly-miner/ and dh_python2 will pick
  it up without any overrides, additional options, etc.
* use dh --with python2 or dh_python2 (/usr/bin/dh_python2) but never
  ever depend on internal implementation detail. I will change
  /usr/share/dh-python/dh_python2 file name just to mess with you ;-P
-- 
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [next] | [standalone]

#8580 — AW: Bug#813096: ITP: logdata-anomaly-miner -- lightweight tool for log checking, log analysis

[Multipart message — attachments visible in raw view] — view raw

> Von: Piotr Ożarowski [mailto:piotr@debian.org]
> 
> just a quick reply:
> 
> * private dir is the right call, do not install into dist-packages
>   (only "python-*" binary packages should install there)

OK, done.

> * install into /usr/lib/logdata-anomaly-miner/ and dh_python2 will pick
>   it up without any overrides, additional options, etc.

So I guess, before that (where dh_python2 refused to operate at all), using /usr/lib/aminer as a shorthand - as the package name is quite long - was a bad idea.

As import names cannot contain a dash, is use of "usr/lib/logdata-anomaly-miner/aminer" as module storage directory and "aminer" as shorthand module name in imports OK?

> * use dh --with python2 or dh_python2 (/usr/bin/dh_python2) but never
>   ever depend on internal implementation detail. I will change
>   /usr/share/dh-python/dh_python2 file name just to mess with you ;-P

OK. Understood. Using the "usr/lib/logdata-anomaly-miner" layout for python stuff as recommended above, dh_python2 is more happy with the situation: It would complain about "python:Versions" substitution in

./debian/control:XB-Python-Version: ${python:Versions}

but XB-.. should be deprecated anyway. I dropped also " XS-Python-Version" (which was hardcoded to >=2.6), as substitution would not work here also.

Still when building, lintian is unhappy with the output package:

E: logdata-anomaly-miner: python-script-but-no-python-dep usr/lib/logdata-anomaly-miner/AMiner
E: logdata-anomaly-miner: python-script-but-no-python-dep usr/lib/logdata-anomaly-miner/AMinerRemoteControl


Since the situation became a lot of clearer using your suggestions, I will again try to tackle the lintian issue.

Roman

> --
> Piotr Ożarowski                         Debian GNU/Linux Developer
> www.ozarowski.pl          www.griffith.cc           www.debian.org
> GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [next] | [standalone]

#8581

[Fiedler Roman, 2016-06-09]
> > * install into /usr/lib/logdata-anomaly-miner/ and dh_python2 will pick
> >   it up without any overrides, additional options, etc.
> 
> So I guess, before that (where dh_python2 refused to operate at all),
> using /usr/lib/aminer as a shorthand - as the package name is quite
> long - was a bad idea.

NAME in /usr/lib/NAME is not used in Python¹. It's not module name, you
cannot import it so Python module naming rules do not apply here.

[¹] at least in /usr/lib/NAME/foo scripts

> As import names cannot contain a dash, is use of
> "usr/lib/logdata-anomaly-miner/aminer" as module storage directory and
> "aminer" as shorthand module name in imports OK?

if /usr/lib/logdata-anomaly-miner is the storage and
/usr/lib/logdata-anomaly-miner/aminer is the module dir (with
__init__.py file), then yes

> Still when building, lintian is unhappy with the output package:
> 
> E: logdata-anomaly-miner: python-script-but-no-python-dep usr/lib/logdata-anomaly-miner/AMiner
> E: logdata-anomaly-miner: python-script-but-no-python-dep usr/lib/logdata-anomaly-miner/AMinerRemoteControl

is AMiner and AMinerRemoteControl symlinked in /usr/bin/?
(you can use debian/logdata-anomaly-miner.links to do that)


PS please don't CC me, I read debian-python@lists.debian.org
-- 
Piotr Ożarowski                         Debian GNU/Linux Developer
www.ozarowski.pl          www.griffith.cc           www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.python

csiph-web