Groups | Search | Server Info | Keyboard shortcuts | Login | Register [http] [https] [nntp] [nntps]

Groups > comp.lang.java.programmer > #39302 > unrolled thread

maven and its downloads

Back to article view | Back to comp.lang.java.programmer

Contents

  maven and its downloads Andreas Leitgeb <avl@logic.at> - 2020-02-26 09:16 +0000
    Re: maven and its downloads Eric Douglas <e.d.programmer@gmail.com> - 2020-02-26 07:51 -0800
      Re: maven and its downloads Andreas Leitgeb <avl@logic.at> - 2020-02-26 16:57 +0000
        Re: maven and its downloads Joerg Meier <joergmmeier@arcor.de> - 2020-02-27 09:35 +0100
          Re: maven and its downloads Andreas Leitgeb <avl@logic.at> - 2020-02-27 11:30 +0000
            Re: maven and its downloads Andreas Leitgeb <avl@logic.at> - 2020-02-27 11:59 +0000
              Re: maven and its downloads Daniele Futtorovic <da.futt.news@laposte-dot-net.invalid> - 2020-02-27 18:59 +0100
            Re: maven and its downloads Eric Douglas <e.d.programmer@gmail.com> - 2020-02-27 05:50 -0800

#39302 — maven and its downloads

I've got a single maven project on a particular user at a machine.
After some development stages (with often changed dependencies),
I made a full clean+build (took only a few minutes), then used
   "find ~/.m2/repository -amin +60 -exec rm {} +"
(remove everything not used in the last 60 minutes)
On next build afterwards, it again downloaded a few packages.

Apparently, it downloads packages that it doesn't ever use.
Majority of them is in org.apache.maven group.

Given that the local repo of that user is used only for that
project, is there a way to prune it of no-longer necessary
stuff, while avoiding to prune stuff that will be re-downloaded?

[toc] | [next] | [standalone]

#39303

On Wednesday, February 26, 2020 at 4:16:26 AM UTC-5, Andreas Leitgeb wrote:
> I've got a single maven project on a particular user at a machine.
> After some development stages (with often changed dependencies),
> I made a full clean+build (took only a few minutes), then used
>    "find ~/.m2/repository -amin +60 -exec rm {} +"
> (remove everything not used in the last 60 minutes)
> On next build afterwards, it again downloaded a few packages.
> 
> Apparently, it downloads packages that it doesn't ever use.
> Majority of them is in org.apache.maven group.
> 
> Given that the local repo of that user is used only for that
> project, is there a way to prune it of no-longer necessary
> stuff, while avoiding to prune stuff that will be re-downloaded?

It should download everything in your pom file(s).
Are you seeing projects downloaded which are not referenced for download anywhere?

Define "doesn't ever use".
Are you thinking it doesn't use things it downloaded because you didn't call them directly?  Sometimes your dependencies have dependencies, and sometimes they're only required if you call certain methods.  You might reference a project which pulls down POI because it can read/write a spreadsheet, but it won't care if you're missing that (runtime) dependency if you never do that in your application.

[toc] | [prev] | [next] | [standalone]

#39305

Eric Douglas <e.d.programmer@gmail.com> wrote:
> On Wednesday, February 26, 2020 at 4:16:26 AM UTC-5, Andreas Leitgeb wrote:
>> I've got a single maven project on a particular user at a machine.
>> After some development stages (with often changed dependencies),
>> I made a full clean+build (took only a few minutes), then used
>>    "find ~/.m2/repository -amin +60 -exec rm {} +"
>> (remove everything not used in the last 60 minutes)
>> On next build afterwards, it again downloaded a few packages.
>> 
>> Apparently, it downloads packages that it doesn't ever use.
>> Majority of them is in org.apache.maven group.
>> 
>> Given that the local repo of that user is used only for that
>> project, is there a way to prune it of no-longer necessary
>> stuff, while avoiding to prune stuff that will be re-downloaded?
> It should download everything in your pom file(s).
> Are you seeing projects downloaded which are not referenced for download anywhere?

Yes.  Well, not "anywhere in my pom-files".

> Define "doesn't ever use".
> Are you thinking it doesn't use things it downloaded because you didn't
> call them directly?

As I wrote:  The OS doesn't record any attempt at reading those files
during the time I do a complete build.

> Sometimes your dependencies have dependencies,
> and sometimes they're only required if you call certain methods.
> You might reference a project which pulls down POI because it can
> read/write a spreadsheet, but it won't care if you're missing that
> (runtime) dependency if you never do that in your application.

I did the same methods before the "find" and afterwards.

Apparently, maven sometimes just checks if a file exists in the repo,
and if not then downloads it, despite not accessing it during a build.

Usually it does seem to get the files only on demand, as I could see
when I first did a "mvn clean" and it downloaded some "clean"-helpers.

[toc] | [prev] | [next] | [standalone]

#39306

On Wed, 26 Feb 2020 16:57:29 -0000 (UTC), Andreas Leitgeb wrote:

> Eric Douglas <e.d.programmer@gmail.com> wrote:
>> On Wednesday, February 26, 2020 at 4:16:26 AM UTC-5, Andreas Leitgeb wrote:
>>> I've got a single maven project on a particular user at a machine.
>>> After some development stages (with often changed dependencies),
>>> I made a full clean+build (took only a few minutes), then used
>>>    "find ~/.m2/repository -amin +60 -exec rm {} +"
>>> (remove everything not used in the last 60 minutes)
>>> On next build afterwards, it again downloaded a few packages.

>>> Apparently, it downloads packages that it doesn't ever use.
>>> Majority of them is in org.apache.maven group.

>>> Given that the local repo of that user is used only for that
>>> project, is there a way to prune it of no-longer necessary
>>> stuff, while avoiding to prune stuff that will be re-downloaded?
>> It should download everything in your pom file(s).
>> Are you seeing projects downloaded which are not referenced for download anywhere?
> Yes.  Well, not "anywhere in my pom-files".

Try "mvn dependency:tree -Dverbose"

>> Define "doesn't ever use".
>> Are you thinking it doesn't use things it downloaded because you didn't
>> call them directly?
> As I wrote:  The OS doesn't record any attempt at reading those files
> during the time I do a complete build.

That doesn't mean they aren't "used" in your Maven context.

>> Sometimes your dependencies have dependencies,
>> and sometimes they're only required if you call certain methods.
>> You might reference a project which pulls down POI because it can
>> read/write a spreadsheet, but it won't care if you're missing that
>> (runtime) dependency if you never do that in your application.
> I did the same methods before the "find" and afterwards.

> Apparently, maven sometimes just checks if a file exists in the repo,
> and if not then downloads it, despite not accessing it during a build.

You seem to have misunderstood Eric. Imagine this scenario: in your
pom.xml, you reference a library. That library references Apache POI, which
is a library that can write a spreadsheet. But your code never actually
writes a spreadsheet, so the Apache POI library is never actually used (in
the code you ran).

But your dependency hierarchy DOES reference the library! So when Maven
sees that the file is missing, it redownloads it. And it really has to:
Maven can't know that you don't plan on ever writing a CSV file. Maybe in
the next application run, you want to open a .xslx file - Maven can't just
say "we'll risk it, he probably won't do that". If your dependency graph
includes libraries that you know you will never use, you can explicitly
exclude them in your pom.xml.

Liebe Gruesse,
		Joerg

-- 
Ich lese meine Emails nicht, replies to Email bleiben also leider
ungelesen.

[toc] | [prev] | [next] | [standalone]

#39307

Joerg Meier <joergmmeier@arcor.de> wrote:
> On Wed, 26 Feb 2020 16:57:29 -0000 (UTC), Andreas Leitgeb wrote:
>> Eric Douglas <e.d.programmer@gmail.com> wrote:
>>> On Wednesday, February 26, 2020 at 4:16:26 AM UTC-5, Andreas Leitgeb wrote:
>>>> I've got a single maven project on a particular user at a machine.
>>>> After some development stages (with often changed dependencies),
>>>> I made a full clean+build (took only a few minutes), then used
>>>>    "find ~/.m2/repository -amin +60 -exec rm {} +"
>>>> (remove everything not used in the last 60 minutes)
>>>> On next build afterwards, it again downloaded a few packages.
>>>> Apparently, it downloads packages that it doesn't ever use.
>>>> Majority of them is in org.apache.maven group.
>>>> Given that the local repo of that user is used only for that
>>>> project, is there a way to prune it of no-longer necessary
>>>> stuff, while avoiding to prune stuff that will be re-downloaded?
>>> It should download everything in your pom file(s).
>>> Are you seeing projects downloaded which are not referenced for download anywhere?
>> Yes.  Well, not "anywhere in my pom-files".
>
> Try "mvn dependency:tree -Dverbose"

Thanks! I didn't know that.  It'll hopefully give me the reasons for
presence of some of the packages, although it doesn't seem to mention
(Fwiw, it says: "Verbose not supported since maven-dependency-plugin 3.0")

>>> Define "doesn't ever use".
>> As I wrote:  The OS doesn't record any attempt at reading those files
>> during the time I do a complete build.
> That doesn't mean they aren't "used" in your Maven context.
...
> You seem to have misunderstood Eric. Imagine this scenario: in your
> pom.xml, you reference a library. That library references Apache POI, which
> is a library that can write a spreadsheet. But your code never actually
> writes a spreadsheet, so the Apache POI library is never actually used (in
> the code you ran).

My logic may still be flawed, and I'd appreciate correction:

My project uses spring-boot-starter to build a standalone 
jar-file with everything included (as nested jar-files).

So, if my code had (e.g.) any chance to create a spreadsheet using a
3rd-party lib from the repo, then the actual code for creating the
spreadsheet would need to be included in the standalone jar, and
for that it would *need to be "accessed" during the build*.

[toc] | [prev] | [next] | [standalone]

#39308

Andreas Leitgeb <avl@logic.at> wrote:
> Joerg Meier <joergmmeier@arcor.de> wrote:
>> On Wed, 26 Feb 2020 16:57:29 -0000 (UTC), Andreas Leitgeb wrote:
>>> Eric Douglas <e.d.programmer@gmail.com> wrote:
>>>> On Wednesday, February 26, 2020 at 4:16:26 AM UTC-5, Andreas Leitgeb wrote:
>>>>> I've got a single maven project on a particular user at a machine.
>>>>> After some development stages (with often changed dependencies),
>>>>> I made a full clean+build (took only a few minutes), then used
>>>>>    "find ~/.m2/repository -amin +60 -exec rm {} +"
>>>>> (remove everything not used in the last 60 minutes)
>>>>> On next build afterwards, it again downloaded a few packages.
>>>>> Apparently, it downloads packages that it doesn't ever use.
>>>>> Majority of them is in org.apache.maven group.
>>>>> Given that the local repo of that user is used only for that
>>>>> project, is there a way to prune it of no-longer necessary
>>>>> stuff, while avoiding to prune stuff that will be re-downloaded?
>>>> It should download everything in your pom file(s).
>>>> Are you seeing projects downloaded which are not referenced for download anywhere?
>>> Yes.  Well, not "anywhere in my pom-files".
>>
>> Try "mvn dependency:tree -Dverbose"
>
> Thanks! I didn't know that.  It'll hopefully give me the reasons for
> presence of some of the packages, although it doesn't seem to mention

oups, rest of sentence got lost somehow...

... doesn't seem to mention those packages that were apparently
(by "find ... -amin"-heuristic) not used, but still re-downloaded.

> (Fwiw, it says: "Verbose not supported since maven-dependency-plugin 3.0")
>
>>>> Define "doesn't ever use".
>>> As I wrote:  The OS doesn't record any attempt at reading those files
>>> during the time I do a complete build.
>> That doesn't mean they aren't "used" in your Maven context.
> ...
>> You seem to have misunderstood Eric. Imagine this scenario: in your
>> pom.xml, you reference a library. That library references Apache POI, which
>> is a library that can write a spreadsheet. But your code never actually
>> writes a spreadsheet, so the Apache POI library is never actually used (in
>> the code you ran).
>
> My logic may still be flawed, and I'd appreciate correction:
>
> My project uses spring-boot-starter to build a standalone 
> jar-file with everything included (as nested jar-files).
>
> So, if my code had (e.g.) any chance to create a spreadsheet using a
> 3rd-party lib from the repo, then the actual code for creating the
> spreadsheet would need to be included in the standalone jar, and
> for that it would *need to be "accessed" during the build*.
>

[toc] | [prev] | [next] | [standalone]

#39313

On 2020-02-27 12:59, Andreas Leitgeb wrote:
>> Joerg Meier <joergmmeier@arcor.de> wrote:
>>> Try "mvn dependency:tree -Dverbose"
>>
>> Thanks! I didn't know that.  It'll hopefully give me the reasons for
>> presence of some of the packages, although it doesn't seem to mention
> those packages that were apparently
> (by "find ... -amin"-heuristic) not used, but still re-downloaded.

I think it would help if now you told us which packages those actually are.

Also, when you say you're building, are we talking IDE or command-line?
Might still be a layer of caching in between...

-- 
DF.

[toc] | [prev] | [next] | [standalone]

#39309

On Thursday, February 27, 2020 at 6:31:08 AM UTC-5, Andreas Leitgeb wrote:
> So, if my code had (e.g.) any chance to create a spreadsheet using a
> 3rd-party lib from the repo, then the actual code for creating the
> spreadsheet would need to be included in the standalone jar, and
> for that it would *need to be "accessed" during the build*.

I think the biggest thing is dependencies require dependencies.
If you write an application that need to draw an image on screen and you know you're only going to access jpg images but you add my image library which gives you easy methods for read image from disk and draw image to screen, you're going to pull in the separate jar needed to read a jbig2.
https://github.com/levigo/jbig2-imageio/releases

[toc] | [prev] | [standalone]

Back to top | Article view | comp.lang.java.programmer

csiph-web