This week's giveaway is in the EJB and other Java EE Technologies forum.
We're giving away four copies of EJB 3 in Action and have Debu Panda, Reza Rahman, Ryan Cuprak, and Michael Remijan on-line!
See this thread for details.
The moose likes Ant, Maven and Other Build Tools and the fly likes Maven chokes on large artifacts Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of EJB 3 in Action this week in the EJB and other Java EE Technologies forum!
JavaRanch » Java Forums » Engineering » Ant, Maven and Other Build Tools
Bookmark "Maven chokes on large artifacts" Watch "Maven chokes on large artifacts" New topic
Author

Maven chokes on large artifacts

Peter Johnson
author
Bartender

Joined: May 14, 2008
Posts: 5772
    
    7

I am trying to load large artifacts into a Nexus repository. At first, Maven crashed with an OOME if I tried to load anything larger than about 70MB. So I upped Maven's max heap to 1024MB, and now I can upload a file of about 240MB. I actually want to load a much larger artifact. By the way, I am using the Maven Deploy Plugin with the deploy:deploy-file goal.

Nexus has no problems with large artifacts. Through its UI (i.e., via a browser) I uploaded a 3.3GB artifact. Took a few minutes, but it uploaded just fine.

Has anyone else worked with large (>1GB) artifacts in Nexus and Maven? (Going to a 64-bit machine for Maven so that I can set up a 16GB heap is not the solution I am looking for... )


JBoss In Action
Tim Holloway
Saloon Keeper

Joined: Jun 25, 2001
Posts: 15624
    
  15

What on Earth is in a 1GB artifact?

Actually, I'd expect the heap requirements to be more dependent on how things are put together than on how big the finished product is.

For the record, I've 1 or 2 projects that basically contain the entire set of the Apache projects in them, plus various trivia such as Hibernate, Spring, [i]et. al.[i/]. Tends to top out at about 40GB for the product. My development machine is infamously limited ("only" 2GB RAM) and I've never had problems, or for that matter, had to bump up the Maven's JVM memory settings.


Customer surveys are for companies who didn't pay proper attention to begin with.
Peter Johnson
author
Bartender

Joined: May 14, 2008
Posts: 5772
    
    7

Just to clarify, yes I do mean a single file that is 1GB is size or larger. The 3.3GB artifact was an ISO file of the Visual Studio 2010 installation DVD - I had that file handy and just used it as a test case. Needless to say, these large artifacts are not JARs.

Digging through the Maven source (for the wagon plugin), it appears that Maven attempts to read the entire artifact into memory before uploading it to the repository. Hence the need for the large heap size. It is obvious from the code that the wagon plugin wasn't designed for large artifacts, but Nexus on the other hand appears to handle them.
Peter Johnson
author
Bartender

Joined: May 14, 2008
Posts: 5772
    
    7

I found the solution. Configure the deploy plugin to use external scp, as described at http://maven.apache.org/plugins/maven-deploy-plugin/examples/deploy-ssh-external.html.

The above page lacks a description on how to specify the URL. I never understand why people who write tutorials or manuals never consider the fact that the hard-coded values they provide as examples often never tell the reader enough about how to come up with their own values based on their configuration. Hopefully, this text will help:

The example URL is scpexe://repository.mycompany.com/repository
  • The "scpexe:" is of course the protocol that tells the deploy plugin to use scp.
  • The "repository.mycompany.com" is of course the name or IP address of the host that contains the repository.
  • The "repository" is the full path to the base directory for the repository on that system. For example, I have Nexus installed at /opt/nexus/nexus and the storage area is at /opt/nexus/sonatype-work/ with the thirdparty repository being located at /opt/nexus/sonatype-work/nexus/storage/thirdparty.

  • Given the above, the URL I used was scpexe://myhostname/opt/nexus/sonatype-work/nexus/storage/thirdparty/

    I successfully uploaded a 3.8GB file as a test case. I then created a simple Maven project with that dependency and it was successfully downloaded by Maven to my local repository. I monitored the memory usage on Maven during the download and it stayer under 40MB so apparently the code to download an artifact doesn't mistakenly attempt to cache the whole thing in memory before writing it to disk.
     
    I agree. Here's the link: http://aspose.com/file-tools
     
    subject: Maven chokes on large artifacts
     
    Similar Threads
    Maven Local repository Vs nexus
    Apache Ant Installation - Build Failed
    m2eclipse could not find dependencies
    Nexus doesn't index artifacts when using scp to deploy
    Could not resolve dependencies for maven project