Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > linux.debian.maint.java > #11303 > unrolled thread

Re: Status of sbt package

Back to article view | Back to linux.debian.maint.java

This discussion starts older than the indexed window; earlier articles aren't shown. The article labeled Started by below is the oldest one visible, not the original post.

Contents

  Re: Status of sbt package Emmanuel Bourg <ebourg@apache.org> - 2019-08-08 09:40 +0200
    Re: Status of sbt package Emmanuel Bourg <ebourg@apache.org> - 2019-08-08 16:50 +0200
      Re: Status of sbt package Frédéric Bonnard <frediz@debian.org> - 2019-08-14 12:00 +0200
        Re: Status of sbt package Thomas Finneid <tfinneid@fcon.no> - 2019-08-17 07:10 +0200
          Re: Status of sbt package Thomas Finneid <tfinneid@fcon.no> - 2019-09-25 22:20 +0200
            Re: Status of sbt package Thomas Finneid <tfinneid@fcon.no> - 2019-09-26 21:30 +0200
              Re: Status of sbt package Thomas Finneid <tfinneid@fcon.no> - 2019-09-27 01:10 +0200
                Re: Status of sbt package Thomas Finneid <tfinneid@fcon.no> - 2019-09-27 14:30 +0200
                Re: Status of sbt package Thomas Finneid <tfinneid@fcon.no> - 2019-09-28 00:00 +0200

#11303 — Re: Status of sbt package

Hi Andrius,

sbt is in sid but hasn't been bootstrapped properly yet, it still uses
libraries not built from sources. Nobody is actively working on it, feel
free to give it a try.

Emmanuel Bourg

Le 07/08/2019 à 13:32, merkys@debian.org a écrit :
> Hello,
> 
> I am writing regarding the status of sbt Debian package. Version 0.13.13
> is in sid, but it had never entered testing. Version 1.3.0-RC3 has
> already been released, are there any plans to update the package? If
> not, I would be willing to try my luck.
> 
> Best wishes,
> Andrius

[toc] | [next] | [standalone]

#11304

Le 08/08/2019 à 12:37, merkys@debian.org a écrit :

> Thanks, I'll give it a look. By the way, the source package does not
> have VCS-* fields, neither it is on Salsa. Do you know where its
> packaging repository is?

I don't think there is a repository yet. I've just created an empty one
on Salsa if that helps.

Emmanuel Bourg

[toc] | [prev] | [next] | [standalone]

#11306

[Multipart message — attachments visible in raw view] — view raw

Hi,

On Fri, 9 Aug 2019 18:25:19 +0300, merkys@debian.org wrote:
> On 2019-08-08 17:48, Emmanuel Bourg wrote:
> > I don't think there is a repository yet. I've just created an empty one
> > on Salsa if that helps.
> 
> Thanks - I'll push once I have something.

the point is that the current packaging of sbt, uses loads of embedded
binaries to bootstrap sbt, which is not very Vcs friendly.

Emmanuel tricked several sbt runtime dependencies to have them build
without those embedded binaries, which is great with only javac and
scalac because the build process is simple enough.

But for sbt, as far as I remember, you cannot just javac * and scalac *
(as for the other simple sbt- packages : see d/rules of
sbt-launcher-interface, sbt-serialization etc), because pieces of the
source code is generated during the build process and I didn't find
time to fake that source generation without the embedded sbt, with only
javac/scalac.

Currently, sbt in the archive is 0.13.13 and upstream seems to propose
1.3 .
So, is it more easy to try to build 1.3 from scratch or do we need to
pursue the effort on 0.13.13 to be able to then build 1.3 ... only
digging more can tell :)

F.

[toc] | [prev] | [next] | [standalone]

#11314

Den 14.08.2019 11:54, skrev Frédéric Bonnard:

> But for sbt, as far as I remember, you cannot just javac * and scalac *
> (as for the other simple sbt- packages : see d/rules of
> sbt-launcher-interface, sbt-serialization etc), because pieces of the
> source code is generated during the build process and I didn't find
> time to fake that source generation without the embedded sbt, with only
> javac/scalac.
> Currently, sbt in the archive is 0.13.13 and upstream seems to propose
> 1.3 .
> So, is it more easy to try to build 1.3 from scratch or do we need to
> pursue the effort on 0.13.13 to be able to then build 1.3 ... only
> digging more can tell :)

Hi Frederic and Andrius

I think getting 1.3 working in debian is far more valuable to all 
parties, Debian, Scala and the Sbt projects. 0.13 is old and not 
required by the latest version of the projects.

I looked into building Sbt and Scala half a year ago, trying to get to 
grips with how to build on the Debian build process, but found the 
dependencies to be circular. As it is, Sbt requires Scala and Scala 
requires Sbt to compile. In addition it requires the previous version of 
the spesific build project. I.e to build Scala 2.12.8, one needs Scala 
2.12.7 and so on. The same for Sbt.

So as far as I understand, and remember, in Debian you would first need 
a proper debian build of Scala, to get any where with building Sbt.

The good news is that after talking with some of the maintainers of Sbt 
and Scala, they would be very interrested in getting a script that could 
build Sbt or Scala from command line, using just Java and similr tools.

The propper build process for both Scala and Sbt contains a lot of 
project and QA activites that we dont need for a debian build. That is, 
before any PR's are merged, the projects own build process does the 
quality checks required.
So the only quality checks a Debian build would require is, to make sure 
it builds and runs properly after a Debian build and adheres to all 
Debian packaging rules. That allows us to simplify the build process for 
both Scala and Sbt, for Debian purposes.

Thats how far I got, but I would love to help and continue working on this.

Andrius, whats your thoughts or plans on how to build Sbt?


Regards

Thomas

[toc] | [prev] | [next] | [standalone]

#11403

Hi Andrius

Sorry for the delay in answering, I'm on daddy-leave so time is scarce. 
(and sorry for the long mail... :) )

I have some more good news :) After battling with my own and other 
demons I have managed to create a shell script for compiling Scala 
2.12.0-M1. The experiences from that can probably be used to create a 
similar script for Sbt.

The original build file (sbt) for Scala is miles long and has lots of 
stuff not needed for creating a debian package. I suspect the same is 
true for Sbt. The original scripts have all sorts of bootstrapping, 
infrastructure, test and other administrative or maintenance parts, 
which is not needed for our purpose. All that is taken care of upstream, 
we just need to build and ship it.

My strategy was reverse engineering of the build process, using the 
scala version that exists in Debian (2.11.12) and verbose mode in sbt to 
study the output from a compilation or packaging task.
I know a different strategy was used for Sbt, but I think this is the 
more viable solution.

A Scala release build is somewhat different from a Sbt release build. 
Since I was compiling the compiler, I needed to override the 
bootclasspath so that the new project modules replaced the older modules 
on the bootclasspath. Thats not a problem for sbt, but there still might 
be some interresting classpath issues when compiling sbt.

You can have a look at what I have done at

https://bitbucket.org/tfinneid/scala-build-process/src/default/build-2.12.0-M1.sh

(The scripts is a very naive version, but it actually caused me more 
headache trying to creating a more automatic version of the script 
while, at the same time, trying to figure out the correct Scala build 
process and command arguments to get a working scala release)

I think one of the bigger issues for creating a working sbt debian 
package is:
- sbt downloads the projects correct sbt version and other dependencies 
and compiles the compiler interface for the correct scala version before 
making the project available on sbt command line
   - Setting that up correctly for the installed package, might not be 
as easy as one thinks.
   - I experienced some problems with that when testing the existing sbt 
package in debian on a minor test project I created.
   - I found that project dependencies was only retrieved from local 
files installed by the package, not from the network as it should (this 
applies to the completed and installed binary package, not the building 
of the package
- other than that sbt external build dependencies must either exist as 
jar files in debian or be created as separate packages (or included as a 
separate pre-compile step in the sbt build script.

Also, check what version of the jvm Sbt was originally built with. Scala 
2.12 is built using Java 8 and hence only properly compatible with that. 
Scala 2.12.10 (at least) and 2.13 supports jdk11, but its still built 
with jdk8 (not target argument, but actual jdk8 release).

FYI

sbt 0.13 is required for Scala 2.12, while sbt 1.0 is required for Scala 
2.13 builds. I dont know if that is important information yet, it 
depends on the strategy for moving forward.

Hope this helps. And I love to hear what you find and share my experiences.

Thomas



On 10.09.2019 17:56, merkys@debian.org wrote:
> Hi Frédéric and Thomas,
> 
> Sorry for the delay. I must admit I wasn't aware the dogfooding problem
> is so difficult to avoid.
> 
> On 2019-08-17 07:53, Thomas Finneid wrote:
>> The good news is that after talking with some of the maintainers of
>> Sbt and Scala, they would be very interrested in getting a script that
>> could build Sbt or Scala from command line, using just Java and similr
>> tools.
> 
> This solution sounds the best. Do you know whether the upstream would be
> willing to help with writing and maintaining such a script?
> 
> Building sbt from 0.13.13 incrementally would be an option too, should
> the previous one fail. However, to build sbt incrementally one would
> probably need to demote its dependencies to make them compatible with
> this ancient version of source.
> 
> Not sure where to start, though. Thanks for sharing your experience.
> 
> Best,
> Andrius
> 
> 

[toc] | [prev] | [next] | [standalone]

#11407

On 26.09.2019 16:54, merkys@debian.org wrote:
> I have cloned your repo and started playing around. AFAIK, your build
> script requires jars built from earlier version of Scala (akka-actor.jar
> and friends). Thus I've tried building Scala and ran into a FTBFS [1]. I
> will have to figure it out before moving to Scala 2.12.0-M1.
> 
> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=941211

Yes I used 2.11.12 as the build tool to build 2.12.0-M1. This is the way 
the Scala team has asked it be done, to do it properly.
The script was initially created in Ubuntu, so I downloaed 2.11.12 and 
installed it manually into a separate directory, then set the PATH to 
point to that. There is a lot of hardcoding of paths and files in the 
script.

I am currently working on making the script work in debian unstable, 
using system installed packages for jdk, scala 2.11.12, ant, jline etc.
I will probably have something workign by the end of the evening.


> However, this strategy is for the time being tedious due to sbt not
> migrating to testing. Otherwise it would sound really tempting.

I agree.

Thomas

[toc] | [prev] | [next] | [standalone]

#11409

Den 26.09.2019 21:29, skrev Thomas Finneid:
> 
> I am currently working on making the script work in debian unstable, 
> using system installed packages for jdk, scala 2.11.12, ant, jline etc.
> I will probably have something workign by the end of the evening.

Hi again,

I have updated the scala build script so it compiles on Debian unstable 
now. There are a couple of errors, so it does not run to completion yet, 
but most of it compiles.

The two major errors are

1- The scala library depenencies are hardcoded, so it doesnt find the 
correct version of them, Still it compiles, which is wierd, but wont 
run, mostlikely because of an AbstractMethodException or something 
similar. Its a bootclasspath issue again.
2- The external dependencies are not met, in this case jline.

So... getting there...

Thomas

[toc] | [prev] | [next] | [standalone]

#11411

Hi Andrius,

Den 27.09.2019 10:15, skrev merkys@debian.org:
> I have managed to get around Scala FTBFS problem [1] for now, will
> continue looking into your script.
> 
> [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=941211

Are you trying to compile scala 2.11.12 from the source debian package 
or from the scalateams 2.11.12 src tar ball? Are you doing that to use 
it as the base to compile 2.12.0-M1, to be able to compile sbt or just 
to get expereience with build script build-2.12.0-M1.sh?

The bug report mentioned maven? I am nort familiar with the build 
procedure described, are you using maven to start the ant build script? 
Or have I got it all wrong?

> jline v2.14.6 is in Debian as libjline2-java, wouldn't this work? 

I experimented with using the jline from /usr/share/scala/lib, but it 
did not work, I think the problem is not jline in it self, but rather 
that the bootclasspath is different in the debian package than the local 
installation of scala 2.11.12 I used when writing the initial script.

By the
> way, how are the jars under $BUILDMODULEPATH built?

The short answer is its the release library jars that comes with the 
scala release of 2.11.12, so its precompiled. Also some of the jars I 
just copied from the release tarball to be able to test if it I can 
compile a test scala class and run it. Specifically jline, scala-xml, 
config and a few others.

The long answer is, in two parts  (I appologise if this is obvious...)
1- I downloaded from efpl the release of 2.11.12 and used that as the 
compiler for the build. So when setting up the booclasspath I had to 
control which jars were included at the different stages of the 
compilation script. The next answer explains why.
2- Since we are compiling a compiler, we need to incrementally exchange 
the old version of the scala jar files with the newly compiled version, 
otherwise the new scala version wont run (abstract method error). This 
means overriding the complete bootclasspath for the jvm, not just scala 
booclasspath.

So the strategy is simple, but technically confusing at times, by 
manually creating a new and complete javabootclasspath:
- get all jars from the jre, they are put last on the bootCP
- get all jars from the scala libs, they are put second to last on the 
bootCP
- first use all this to build the first module of the new scala version: 
scala-library
- replace old scala-library jar (from 2.11.12) with newly compiled 
library-jar (2.12.0-M1) on the bootCP (put first on bootCP to guarantee 
shadowing of the old version)
- as the modules are built, keep replacing the old with the new, in the 
order of asm, forkjoin, library, reflect, compiler, scalap, repl, 
interactive

The jars the script produces ar not package exactly the same as the efpl 
release jars are, so that might be a bit configusing


Btw, maybe we should start a new thread about the scala compilation 
stuff so its easier for all to see what the discussions are actually about?

[toc] | [prev] | [next] | [standalone]

#11416

Den 27.09.2019 10:15, skrev merkys@debian.org:

> jline v2.14.6 is in Debian as libjline2-java, wouldn't this work?
FYI
I did a test of using jline 2.14.6 when running scala 2.12.0-M1 repl, 
and it failed with NoSuchMethodError. Downloading and compiling jline 
2.14.1, on the other hand, did work.
Interrestingly enough scala 2.11.12 uses jline 2.14.3, while scala 
2.12.0 uses jline 2.14.1. A downgrade!

All this is of course a different error than what I get while compiling, 
so I will investigate further the compile error.

Thomas

[toc] | [prev] | [standalone]

Back to top | Article view | linux.debian.maint.java

csiph-web