• Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Maven chokes on large artifacts

 
Peter Johnson
author
Bartender
Posts: 5852
7
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I am trying to load large artifacts into a Nexus repository. At first, Maven crashed with an OOME if I tried to load anything larger than about 70MB. So I upped Maven's max heap to 1024MB, and now I can upload a file of about 240MB. I actually want to load a much larger artifact. By the way, I am using the Maven Deploy Plugin with the deploy:deploy-file goal.

Nexus has no problems with large artifacts. Through its UI (i.e., via a browser) I uploaded a 3.3GB artifact. Took a few minutes, but it uploaded just fine.

Has anyone else worked with large (>1GB) artifacts in Nexus and Maven? (Going to a 64-bit machine for Maven so that I can set up a 16GB heap is not the solution I am looking for... )
 
Tim Holloway
Saloon Keeper
Pie
Posts: 18212
53
Android Eclipse IDE Linux
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
What on Earth is in a 1GB artifact?

Actually, I'd expect the heap requirements to be more dependent on how things are put together than on how big the finished product is.

For the record, I've 1 or 2 projects that basically contain the entire set of the Apache projects in them, plus various trivia such as Hibernate, Spring, [i]et. al.[i/]. Tends to top out at about 40GB for the product. My development machine is infamously limited ("only" 2GB RAM) and I've never had problems, or for that matter, had to bump up the Maven's JVM memory settings.
 
Peter Johnson
author
Bartender
Posts: 5852
7
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Just to clarify, yes I do mean a single file that is 1GB is size or larger. The 3.3GB artifact was an ISO file of the Visual Studio 2010 installation DVD - I had that file handy and just used it as a test case. Needless to say, these large artifacts are not JARs.

Digging through the Maven source (for the wagon plugin), it appears that Maven attempts to read the entire artifact into memory before uploading it to the repository. Hence the need for the large heap size. It is obvious from the code that the wagon plugin wasn't designed for large artifacts, but Nexus on the other hand appears to handle them.
 
Peter Johnson
author
Bartender
Posts: 5852
7
Android Eclipse IDE Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I found the solution. Configure the deploy plugin to use external scp, as described at http://maven.apache.org/plugins/maven-deploy-plugin/examples/deploy-ssh-external.html.

The above page lacks a description on how to specify the URL. I never understand why people who write tutorials or manuals never consider the fact that the hard-coded values they provide as examples often never tell the reader enough about how to come up with their own values based on their configuration. Hopefully, this text will help:

The example URL is scpexe://repository.mycompany.com/repository
  • The "scpexe:" is of course the protocol that tells the deploy plugin to use scp.
  • The "repository.mycompany.com" is of course the name or IP address of the host that contains the repository.
  • The "repository" is the full path to the base directory for the repository on that system. For example, I have Nexus installed at /opt/nexus/nexus and the storage area is at /opt/nexus/sonatype-work/ with the thirdparty repository being located at /opt/nexus/sonatype-work/nexus/storage/thirdparty.

  • Given the above, the URL I used was scpexe://myhostname/opt/nexus/sonatype-work/nexus/storage/thirdparty/

    I successfully uploaded a 3.8GB file as a test case. I then created a simple Maven project with that dependency and it was successfully downloaded by Maven to my local repository. I monitored the memory usage on Maven during the download and it stayer under 40MB so apparently the code to download an artifact doesn't mistakenly attempt to cache the whole thing in memory before writing it to disk.
     
    • Post Reply
    • Bookmark Topic Watch Topic
    • New Topic