Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.java > #8734 > unrolled thread

Help needed for gatk

Back to article view | Back to linux.debian.maint.java

Contents

  Help needed for gatk Andreas Tille <andreas@an3as.eu> - 2016-01-21 09:50 +0100
    Re: Help needed for gatk Thorsten Glaser <t.glaser@tarent.de> - 2016-01-21 10:00 +0100
      Re: Help needed for gatk Andreas Tille <andreas@an3as.eu> - 2016-01-21 10:10 +0100
        Re: Help needed for gatk gregor herrmann <gregoa@debian.org> - 2016-01-21 10:10 +0100
        Re: Help needed for gatk Thorsten Glaser <t.glaser@tarent.de> - 2016-01-21 10:20 +0100
          Re: Help needed for gatk Emmanuel Bourg <ebourg@apache.org> - 2016-01-21 10:30 +0100
            Re: Help needed for gatk Andreas Tille <andreas@an3as.eu> - 2016-01-21 10:50 +0100
              Re: Help needed for gatk Emmanuel Bourg <ebourg@apache.org> - 2016-01-21 11:10 +0100
                Re: Help needed for gatk Andreas Tille <andreas@an3as.eu> - 2016-01-27 15:10 +0100
                  Re: Help needed for gatk Andrew Schurman <arcticwaters@gmail.com> - 2016-01-28 00:00 +0100
                    Re: Help needed for gatk Andreas Tille <andreas@an3as.eu> - 2016-01-28 16:30 +0100
                      Re: Help needed for gatk Andrew Schurman <arcticwaters@gmail.com> - 2016-01-29 19:50 +0100
                        Re: Help needed for gatk Andreas Tille <andreas@an3as.eu> - 2016-01-29 22:40 +0100
                    Packaging maven-scala-plugin (Was: Help needed for gatk) Andreas Tille <tille@debian.org> - 2016-01-28 19:40 +0100
                      Re: Packaging maven-scala-plugin (Was: Help needed for gatk) Andreas Tille <andreas@an3as.eu> - 2016-01-30 15:00 +0100
                        Re: Packaging maven-scala-plugin (Was: Help needed for gatk) Emmanuel Bourg <ebourg@apache.org> - 2016-01-30 15:50 +0100
                          Re: Packaging maven-scala-plugin (Was: Help needed for gatk) Andreas Tille <andreas@an3as.eu> - 2016-01-30 18:10 +0100
                            Re: Packaging maven-scala-plugin (Was: Help needed for gatk) Andrew Schurman <arcticwaters@gmail.com> - 2016-01-30 22:10 +0100
                              Re: Packaging maven-scala-plugin (Was: Help needed for gatk) Andreas Tille <andreas@an3as.eu> - 2016-02-01 10:00 +0100
                                Re: Packaging maven-scala-plugin (Was: Help needed for gatk) Andrew Schurman <arcticwaters@gmail.com> - 2016-02-01 12:10 +0100
                                  Re: Packaging maven-scala-plugin (Was: Help needed for gatk) Andreas Tille <andreas@an3as.eu> - 2016-02-01 12:20 +0100
                                    Re: Packaging maven-scala-plugin (Was: Help needed for gatk) Andrew Schurman <arcticwaters@gmail.com> - 2016-02-01 13:40 +0100
                                      Re: Packaging maven-scala-plugin (Was: Help needed for gatk) Emmanuel Bourg <ebourg@apache.org> - 2016-02-01 14:40 +0100
                                        Re: Packaging maven-scala-plugin (Was: Help needed for gatk) Andreas Tille <andreas@an3as.eu> - 2016-02-01 14:50 +0100
                                          Re: Packaging maven-scala-plugin (Was: Help needed for gatk) Andreas Tille <andreas@an3as.eu> - 2016-02-01 22:00 +0100
                                            Re: Packaging maven-scala-plugin (Was: Help needed for gatk) David Bernard <david.bernard.31@gmail.com> - 2016-02-01 23:10 +0100
                                              Re: Packaging maven-scala-plugin (Was: Help needed for gatk) Andreas Tille <andreas@fam-tille.de> - 2016-02-01 23:10 +0100
                                              Re: Packaging maven-scala-plugin (Was: Help needed for gatk) David Bernard <david.bernard.31@gmail.com> - 2016-02-01 23:20 +0100
                                      Re: Packaging maven-scala-plugin (Was: Help needed for gatk) Andreas Tille <andreas@an3as.eu> - 2016-02-01 14:40 +0100

Page 2 of 2 — ← Prev page 1 [2]

#8791 — Re: Packaging maven-scala-plugin (Was: Help needed for gatk)

Hi Andrew,

On Mon, Feb 01, 2016 at 03:05:32AM -0800, Andrew Schurman wrote:
> On Mon, 2016-02-01 at 09:51 +0100, Andreas Tille wrote:
> > Thanks for "reading aloud" for me - this might be due to my partial
> > blindness for Java problems. :-)
> 
> No worries. Sometimes people overlook the simplest solutions because
> they've been tackling a problem for too long. 

I'm actually not really worried - just explaining my position to ask in
advance for more patience. ;-)

> > So I think I need to go on packaging:
> > 
> >     https://maven.apache.org/plugins/maven-toolchains-plugin/  and
> >     https://maven.apache.org/plugins/maven-repository-plugin/
> > 
> > I'm just asking here to make sure that there is no work in progress
> > for
> > this or that it somehow could be circumvented somehow ...
> 
> Now you've found a more interesting problem. The scala-plugin doesn't
> have a direct dependency on these, yet expects them to be there. Seeing
> how there have been a few releases with this change, it's unlikely to
> be a mishap by the author. Checking `dependency:tree` doesn't list them
> either, so we don't need these to be packaged. What this has to mean is
> the version of one of the dependencies of the project in debian isn't
> correct. A quick search for toolchain in jars of the dependencies show
> it's available in maven-core:3.0.4.
> 
> It looks like you're building with maven-core 2.x instead of 3.x since
> it was added in 3.x. The debian version of maven-core redirects to 2.x
> by default, so you'll need to add a publishedRule to use version 3.x of
> maven-core.
> 
> Hopefully that troubleshooting logic will help you.

I'm not fully sure.  At least I understood that possibly no other
packaging task will end up on my desktop.  I admit I have no idea how to
craft the suggested publishedRule.

However, what makes me really suspicious is:

$ grep maven.*-core-java maven-scala-plugin_3.2.2-1_amd64.build 
Selecting previously unselected package libmaven2-core-java.
Preparing to unpack .../libmaven2-core-java_2.2.1-24_all.deb ...
Unpacking libmaven2-core-java (2.2.1-24) ...
Selecting previously unselected package libmaven3-core-java.
Preparing to unpack .../libmaven3-core-java_3.3.9-3_all.deb ...
Unpacking libmaven3-core-java (3.3.9-3) ...
Setting up libmaven2-core-java (2.2.1-24) ...
Setting up libmaven3-core-java (3.3.9-3) ...


Is there any reason that libmaven2-core-java is implicitly dragged in by
other Build-Depends and may be this is creating the problem?

Thanks for your patience

       Andreas.


-- 
http://fam-tille.de

[toc] | [prev] | [next] | [standalone]

#8792 — Re: Packaging maven-scala-plugin (Was: Help needed for gatk)

[Multipart message — attachments visible in raw view] — view raw

On Mon, 2016-02-01 at 12:18 +0100, Andreas Tille wrote:
>  I admit I have no idea how
> to
> craft the suggested publishedRule.

The maven.publishedRules usually gets generated with comments to help
you if you used mh_make. If not, check another project for reference
such as [1]. Adding the line below should work:

org.apache.maven maven-core jar s/.*/3.x/

> 
> However, what makes me really suspicious is:
> 
> $ grep maven.*-core-java maven-scala-plugin_3.2.2-1_amd64.build 
> Selecting previously unselected package libmaven2-core-java.
> Preparing to unpack .../libmaven2-core-java_2.2.1-24_all.deb ...
> Unpacking libmaven2-core-java (2.2.1-24) ...
> Selecting previously unselected package libmaven3-core-java.
> Preparing to unpack .../libmaven3-core-java_3.3.9-3_all.deb ...
> Unpacking libmaven3-core-java (3.3.9-3) ...
> Setting up libmaven2-core-java (2.2.1-24) ...
> Setting up libmaven3-core-java (3.3.9-3) ...
> 
> 
> Is there any reason that libmaven2-core-java is implicitly dragged in
> by
> other Build-Depends and may be this is creating the problem?

I wouldn't worry about this. It just means another package depends on a
different version of the library. Both libmaven2-core-java and
libmaven3-core-java are able to coexist on the same system. In your
case, it's brought in by libmaven-docck-plugin-java, a direct
dependency in the maven-scala-plugin.

Your problem is that poms get rewritten in the build phase by maven-
debian-helper. They change the version of dependencies to `debian`
because bananas (there really is a reason, but its not important here).
This happens regardless whether libmaven2-core-java gets pulled in by
dpkg. We just need to tell maven-debian-helper to use a specific
version, 3.x, instead of `debian` which redirects to 2.2.1 likely to
maintain compatibility. This is what the maven.publishedRules file is
for.

Andrew

[1] https://github.com/googlei18n/libphonenumber/blob/master/debian/mav
en.publishedRules

[toc] | [prev] | [next] | [standalone]

#8793 — Re: Packaging maven-scala-plugin (Was: Help needed for gatk)

Le 1/02/2016 14:35, Andreas Tille a écrit :

> I wonder whether I could really skip the Zinc dependency.  I'm also
> afraid that if Zinc is not available I need at least xsbti - so just
> another package I need to create.  Is this correct?

You ignored the zinc dependency, but now you need to patch out its use
to avoid the compilation errors.

Emmanuel Bourg

[toc] | [prev] | [next] | [standalone]

#8795 — Re: Packaging maven-scala-plugin (Was: Help needed for gatk)

On Mon, Feb 01, 2016 at 02:38:19PM +0100, Emmanuel Bourg wrote:
> > I wonder whether I could really skip the Zinc dependency.  I'm also
> > afraid that if Zinc is not available I need at least xsbti - so just
> > another package I need to create.  Is this correct?
> 
> You ignored the zinc dependency, but now you need to patch out its use
> to avoid the compilation errors.

Yes, this is what I concluded from the errors and thus I had a look into
the code.  When I did so I had the impression that if I would be able to
get rid of the zinc usage (and I hope upstream could help here to make
this safely) I would need xsbti instead.  This would leave me with
another new software to package - or am I wrong with this assumption?

Kind regards

       Andreas.

-- 
http://fam-tille.de

[toc] | [prev] | [next] | [standalone]

#8799 — Re: Packaging maven-scala-plugin (Was: Help needed for gatk)

Hi David,

On Mon, Feb 01, 2016 at 09:38:44PM +0100, David Bernard wrote:
> 
> TBH, I don't understand why you package a maven plugin ?

I can understand that from a maven plugin author this sounds totally
crazy since maven is designed to download what is missing.  However, the
Debian archive has to be self consistent and needs to be buildable from
source out of itself.  So per Debian policy you need to build a package
while beeing offline and the usual build process guarantees that this is
ensured by creating a chroot environment that can access only the Debian
archive.  The reason why we are packaging maven plugins is that we want
to package Java software for Debian that is using this plugin to build.
The project I want to build is gatk[1] and there is a thread on the
debian-java lists that exposes my motivation to package
macen-scala-plugin[2].

> Because the maven
> regular flow, is to let maven download and install plugins declared in the
> project, on demand, the first time it's used/required.

As I said above from a distribution point of view downloading is not
possible.
 
> Now, about your issue, zinc is optional (personnaly I never use it). and
> first build of a maven project should not be done offline include, first
> build of a maven's plugin. And from your log, you build the plugin offline.
> You could find the *direct* list of dependency at :
> http://davidb.github.io/scala-maven-plugin/dependencies.html

Its an interesting piece of information that you personally do not use
zinc and thus this could be ignored.  However, as you say its downloaded
at some point in time anyway due to maven's plugin mechanism.  Do you
see any chance that this download could be avoided and the source code
checks the existence before trying to access zinc?

I hope I have explained my point clearly enough and you can make some
sense out of it why we exactly need this offline mode and try to be able
to skip those things that are not used completely.

Kind regards

         Andreas.

[1] https://www.broadinstitute.org/gatk/
[2] https://lists.debian.org/debian-java/2016/01/msg00050.html

-- 
http://fam-tille.de

[toc] | [prev] | [next] | [standalone]

#8802 — Re: Packaging maven-scala-plugin (Was: Help needed for gatk)

[Multipart message — attachments visible in raw view] — view raw

zinc is optional at runtime, but required at build time (part of the code
depends of it).

I don't know the package policy of Debian (I use ArchLinux). Is it
mandatarory to build every package (like in a gentoo)? If yes you'll have
to build all the dependency of plugin, jars, ... from source and install
them into a maven repository. It's a crazy task (including that some of
jar, like scala, are not build with maven). And you'll have to create a
package for every dependencies (directs + transitives).

Do you plan to create a debian package for each artifacts listed via
https://maven.apache.org/plugins/maven-dependency-plugin/analyze-mojo.html
of the gatk project and its plugins ? ( I didn't read the full conversation
yet).



On Mon, Feb 1, 2016 at 9:56 PM, Andreas Tille <andreas@an3as.eu> wrote:

> Hi David,
>
> On Mon, Feb 01, 2016 at 09:38:44PM +0100, David Bernard wrote:
> >
> > TBH, I don't understand why you package a maven plugin ?
>
> I can understand that from a maven plugin author this sounds totally
> crazy since maven is designed to download what is missing.  However, the
> Debian archive has to be self consistent and needs to be buildable from
> source out of itself.  So per Debian policy you need to build a package
> while beeing offline and the usual build process guarantees that this is
> ensured by creating a chroot environment that can access only the Debian
> archive.  The reason why we are packaging maven plugins is that we want
> to package Java software for Debian that is using this plugin to build.
> The project I want to build is gatk[1] and there is a thread on the
> debian-java lists that exposes my motivation to package
> macen-scala-plugin[2].
>
> > Because the maven
> > regular flow, is to let maven download and install plugins declared in
> the
> > project, on demand, the first time it's used/required.
>
> As I said above from a distribution point of view downloading is not
> possible.
>
> > Now, about your issue, zinc is optional (personnaly I never use it). and
> > first build of a maven project should not be done offline include, first
> > build of a maven's plugin. And from your log, you build the plugin
> offline.
> > You could find the *direct* list of dependency at :
> > http://davidb.github.io/scala-maven-plugin/dependencies.html
>
> Its an interesting piece of information that you personally do not use
> zinc and thus this could be ignored.  However, as you say its downloaded
> at some point in time anyway due to maven's plugin mechanism.  Do you
> see any chance that this download could be avoided and the source code
> checks the existence before trying to access zinc?
>
> I hope I have explained my point clearly enough and you can make some
> sense out of it why we exactly need this offline mode and try to be able
> to skip those things that are not used completely.
>
> Kind regards
>
>          Andreas.
>
> [1] https://www.broadinstitute.org/gatk/
> [2] https://lists.debian.org/debian-java/2016/01/msg00050.html
>
> --
> http://fam-tille.de
>

[toc] | [prev] | [next] | [standalone]

#8803 — Re: Packaging maven-scala-plugin (Was: Help needed for gatk)

Hi David,

On Mon, Feb 01, 2016 at 10:41:53PM +0100, David Bernard wrote:
> zinc is optional at runtime, but required at build time (part of the code
> depends of it).

That actually was my question:  Do you see any chance to patch the code
to not require zinc at build time.  If I do not intend to use it at
runtime from my naive point of view it should be possible to exclude it
at build time as well.
 
> I don't know the package policy of Debian (I use ArchLinux). Is it
> mandatarory to build every package (like in a gentoo)?

Yes.

> If yes you'll have
> to build all the dependency of plugin, jars, ... from source and install
> them into a maven repository. It's a crazy task (including that some of
> jar, like scala, are not build with maven). And you'll have to create a
> package for every dependencies (directs + transitives).

Well, we have a certain amount of dependencies packaged and we might
need some more.
 
> Do you plan to create a debian package for each artifacts listed via
> https://maven.apache.org/plugins/maven-dependency-plugin/analyze-mojo.html
> of the gatk project and its plugins ? ( I didn't read the full conversation
> yet).

Only for those that are really required to build the project and to run
without restrictions in functionality.
 
Kind regards

     Andreas.

-- 
http://fam-tille.de

[toc] | [prev] | [next] | [standalone]

#8805 — Re: Packaging maven-scala-plugin (Was: Help needed for gatk)

[Multipart message — attachments visible in raw view] — view raw

Andreas,

the latest version of maven-scala-plugin is available at
https://github.com/davidB/scala-maven-plugin/tree/v2.15.2 (both plugin
share the same history, repo,...) this version work for maven 2 and didn't
rely on zinc. and it should match your initial requirement. As the version
3 also change the name, I guess you should not have issue if someone create
a package for scala-maven-plugin later.


On Mon, Feb 1, 2016 at 10:41 PM, David Bernard <david.bernard.31@gmail.com>
wrote:

> zinc is optional at runtime, but required at build time (part of the code
> depends of it).
>
> I don't know the package policy of Debian (I use ArchLinux). Is it
> mandatarory to build every package (like in a gentoo)? If yes you'll have
> to build all the dependency of plugin, jars, ... from source and install
> them into a maven repository. It's a crazy task (including that some of
> jar, like scala, are not build with maven). And you'll have to create a
> package for every dependencies (directs + transitives).
>
> Do you plan to create a debian package for each artifacts listed via
> https://maven.apache.org/plugins/maven-dependency-plugin/analyze-mojo.html
> of the gatk project and its plugins ? ( I didn't read the full conversation
> yet).
>
>
>
> On Mon, Feb 1, 2016 at 9:56 PM, Andreas Tille <andreas@an3as.eu> wrote:
>
>> Hi David,
>>
>> On Mon, Feb 01, 2016 at 09:38:44PM +0100, David Bernard wrote:
>> >
>> > TBH, I don't understand why you package a maven plugin ?
>>
>> I can understand that from a maven plugin author this sounds totally
>> crazy since maven is designed to download what is missing.  However, the
>> Debian archive has to be self consistent and needs to be buildable from
>> source out of itself.  So per Debian policy you need to build a package
>> while beeing offline and the usual build process guarantees that this is
>> ensured by creating a chroot environment that can access only the Debian
>> archive.  The reason why we are packaging maven plugins is that we want
>> to package Java software for Debian that is using this plugin to build.
>> The project I want to build is gatk[1] and there is a thread on the
>> debian-java lists that exposes my motivation to package
>> macen-scala-plugin[2].
>>
>> > Because the maven
>> > regular flow, is to let maven download and install plugins declared in
>> the
>> > project, on demand, the first time it's used/required.
>>
>> As I said above from a distribution point of view downloading is not
>> possible.
>>
>> > Now, about your issue, zinc is optional (personnaly I never use it). and
>> > first build of a maven project should not be done offline include, first
>> > build of a maven's plugin. And from your log, you build the plugin
>> offline.
>> > You could find the *direct* list of dependency at :
>> > http://davidb.github.io/scala-maven-plugin/dependencies.html
>>
>> Its an interesting piece of information that you personally do not use
>> zinc and thus this could be ignored.  However, as you say its downloaded
>> at some point in time anyway due to maven's plugin mechanism.  Do you
>> see any chance that this download could be avoided and the source code
>> checks the existence before trying to access zinc?
>>
>> I hope I have explained my point clearly enough and you can make some
>> sense out of it why we exactly need this offline mode and try to be able
>> to skip those things that are not used completely.
>>
>> Kind regards
>>
>>          Andreas.
>>
>> [1] https://www.broadinstitute.org/gatk/
>> [2] https://lists.debian.org/debian-java/2016/01/msg00050.html
>>
>> --
>> http://fam-tille.de
>>
>
>

[toc] | [prev] | [next] | [standalone]

#8794 — Re: Packaging maven-scala-plugin (Was: Help needed for gatk)

Hi,

[Upstream in CC.  David for better understanding you might like to read
 my mail
  https://lists.debian.org/debian-java/2016/02/msg00000.html
 and the related patch
  https://anonscm.debian.org/cgit/pkg-java/maven-scala-plugin.git/tree/debian/patches/disable_zinc.patch
 If you read below the part when I start discussing the errors might
 be worth a comment from your side.
]

On Mon, Feb 01, 2016 at 04:30:00AM -0800, Andrew Schurman wrote:
> On Mon, 2016-02-01 at 12:18 +0100, Andreas Tille wrote:
> >  I admit I have no idea how
> > to
> > craft the suggested publishedRule.
> 
> The maven.publishedRules usually gets generated with comments to help
> you if you used mh_make. If not, check another project for reference
> such as [1]. Adding the line below should work:
> 
> org.apache.maven maven-core jar s/.*/3.x/

I seeked on codesearch.debian.net for

    maven-core path:debian/maven.publishedRules

but failed finding such string.  By a more relaxed search I've found
that this string needs to go into debian/maven.rules rather than
debian/maven.publishedRules which brought me another step forward.
 
Unfortunately I'm running into another series of Errors:

...
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /build/maven-scala-plugin-3.2.2/src/main/java/scala_maven/ScalaCompilerSupport.java:[4,42] package org.apache.maven.plugin.descriptor does not exist
[ERROR] /build/maven-scala-plugin-3.2.2/src/main/java/sbt_inc/SbtIncrementalCompiler.java:[3,25] package com.typesafe.zinc does not exist
[ERROR] /build/maven-scala-plugin-3.2.2/src/main/java/sbt_inc/SbtIncrementalCompiler.java:[4,1] package com.typesafe.zinc does not exist
[ERROR] /build/maven-scala-plugin-3.2.2/src/main/java/sbt_inc/SbtIncrementalCompiler.java:[6,13] package scala does not exist
[ERROR] /build/maven-scala-plugin-3.2.2/src/main/java/sbt_inc/SbtIncrementalCompiler.java:[27,13] cannot find symbol
  symbol:   class ZincClient
  location: class sbt_inc.SbtIncrementalCompiler
[ERROR] /build/maven-scala-plugin-3.2.2/src/main/java/sbt_inc/SbtIncrementalCompiler.java:[39,18] package xsbti does not exist
[ERROR] /build/maven-scala-plugin-3.2.2/src/main/java/sbt_inc/SbtIncrementalCompiler.java:[69,13] cannot find symbol
...


I wonder whether I could really skip the Zinc dependency.  I'm also
afraid that if Zinc is not available I need at least xsbti - so just
another package I need to create.  Is this correct?

Kind regards

       Andreas.

-- 
http://fam-tille.de

[toc] | [prev] | [standalone]

Page 2 of 2 — ← Prev page 1 [2]

Back to top | Article view | linux.debian.maint.java

csiph-web