Dropbox "java.lang.outofmemoryerror java heap space" when trying to upload a large file.

[pankaj kumar t]

HI, i'm new to this form,
i was getting an outofmemoryerror while trying to upload a large file to dropbox via java dropbox sdk.
Here is my scenario :
Server A contains a file (1GB) --> Server B opens a stream and passes that stream to dropbox api in java
Here is my code :

DbxRequestConfig config = new DbxRequestConfig("dropbox/java-tutorial","en_US");
		DbxClientV2 client = new DbxClientV2(config, ACCESS_TOKEN);
		try {
			URL url = new URL(url_d);			
			// Upload Files to Dropbox
			String fileName = url_d.substring( url_d.lastIndexOf('/')+1, url_d.length() );
			DbxFiles.FileMetadata metadata = client.files.uploadBuilder("/"+fileName).run(url.openStream());
			
		} catch (MalformedURLException e) {
			e.printStackTrace();
		} catch (IOException e) {
			e.printStackTrace();
		} catch (UploadException e) {
			e.printStackTrace();
		} catch (DbxException e) {
			e.printStackTrace();
		}

Could anyone solve my issue thank you.

[Dave Tolls]

Can you post the full stack trace?

[pankaj kumar t]

org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
org.springframework.web.servlet.DispatcherServlet.triggerAfterCompletionWithError(DispatcherServlet.java:1259)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:945)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:953)
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:855)
javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:829)
javax.servlet.http.HttpServlet.service(HttpServlet.java:847)
root cause

java.lang.OutOfMemoryError: Java heap space
java.util.Arrays.copyOf(Arrays.java:2271)
java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:52)
com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:63)
com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:34)
com.dropbox.core.v2.DbxUploadStyleBuilder.run(DbxUploadStyleBuilder.java:30)
org.spring.ourchat.home.uploadTODropBox(home.java:66)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:606)
org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:745)
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:686)
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:80)
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:925)
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:856)
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:953)
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:855)
javax.servlet.http.HttpServlet.service(HttpServlet.java:754)
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:829)
javax.servlet.http.HttpServlet.service(HttpServlet.java:847)

[Campbell Ritchie]

Don't know, but welcome to the Ranch

[Paul Clapham]

This part of the stack trace:

java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:52)

tells me that the Dropbox code is copying something to a byte array. Which suggests it's trying to copy your upload file into memory before uploading it to Dropbox. I could be wrong about that because all I see is the stack trace, but my first sentence is fact and the second sentence is a speculative conclusion based on that fact. But since an out-of-memory condition is occurring, that makes my conclusion much more likely.

Clearly that isn't scalable, because your code only has a fixed amount of memory available so it restricts the size of the file you can upload. And it's surely not necessary to do it that way. But it's not unknown for even well-financed corporations to distribute untested software written by people who don't really know what they are doing.

But again, I might be wrong. If this SDK is open-source then you could look at the code yourself and see if it's really making that error. If not, you could perhaps contact the place you got it from and ask what's going on.

[Dave Tolls]

Unless the code has changed between v1 and v2 (and the cached v2 version in Google looks the same for the copy code) then it uses a 16k buffer.
IOUtil.java

Maybe there's a config thing that's set incorrectly?
I notice the DbxClientV2 takes a config object.

[pankaj kumar t]

I really don't know what to do right now , i will have to do trail and error to resolve this issue.

[Tony Docherty]

No don't do trial and error, that approach rarely works and even if you do get your file to upload you may find your code is extremely brittle. Investigate the problem, find out why the exception is occurring and then plan your approach to overcoming the problem.

Dave and Paul have given you some things to look at, take their advice and investigate those areas.

[pankaj kumar t]

Paul Clapham wrote:This part of the stack trace:

java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
sun.net.www.http.PosterOutputStream.write(PosterOutputStream.java:78)
com.dropbox.core.util.IOUtil.copyStreamToStream(IOUtil.java:52)

tells me that the Dropbox code is copying something to a byte array. Which suggests it's trying to copy your upload file into memory before uploading it to Dropbox. I could be wrong about that because all I see is the stack trace, but my first sentence is fact and the second sentence is a speculative conclusion based on that fact. But since an out-of-memory condition is occurring, that makes my conclusion much more likely.

Clearly that isn't scalable, because your code only has a fixed amount of memory available so it restricts the size of the file you can upload. And it's surely not necessary to do it that way. But it's not unknown for even well-financed corporations to distribute untested software written by people who don't really know what they are doing.

But again, I might be wrong. If this SDK is open-source then you could look at the code yourself and see if it's really making that error. If not, you could perhaps contact the place you got it from and ask what's going on.

How can i avoid loading a total file into memory and pass that stream to dropbox and flush after some 'x' bytes.
can please you give me code sample.
Thank you :)

[Paul Clapham]

pankaj kumar t wrote:How can i avoid loading a total file into memory and pass that stream to dropbox and flush after some 'x' bytes.

I don't know. As I said, my answer was speculation and I suggested some things you could do. Am I right in guessing you haven't done any of those things? Okay, let's try the first one together. Is the Dropbox code open source? If so, give me a link to the source code and I'll help you in finding where to look.

[pankaj kumar t]

Paul Clapham wrote:

pankaj kumar t wrote:How can i avoid loading a total file into memory and pass that stream to dropbox and flush after some 'x' bytes.

I don't know. As I said, my answer was speculation and I suggested some things you could do. Am I right in guessing you haven't done any of those things? Okay, let's try the first one together. Is the Dropbox code open source? If so, give me a link to the source code and I'll help you in finding where to look.

Hi paul thanks for the help
here is the github link https://github.com/dropbox/dropbox-sdk-java for dropbox.

[Dave Tolls]

So this is the IOUtil class that does the transferring.
This is called by the DbxUploadStyleBuilder, with an in and an out stream.

Looking at that version of copyStreamToStream, line 31, we see that it uses the default buffer size, which is 16 k.
So that can't be the issue.

However, look at the output stream being used.
That's a ByteArrayOutputStream.

So, it's not writing to an external place, it's writing to an array internal to your app.
So each 16k block of data is being added to the array held in this class. That's what's happening in the grow() method.

So where is this stream coming from?
Looks like it's from the uploader supplied, which is called on line 30 of the DbxUploadStyleBuilder.

This is supplied by that call to uploadBuilder, which is based (I believe) in the config.

So...what is the "dropbox/java-tutorial" client identifier?
Is it, by any chance, a config for simple testing that doesn't actually hit dropbox?

That would be where I would look.

[pankaj kumar t]

Dave Tolls wrote:So this is the IOUtil class that does the transferring.
This is called by the DbxUploadStyleBuilder, with an in and an out stream.

Looking at that version of copyStreamToStream, line 31, we see that it uses the default buffer size, which is 16 k.
So that can't be the issue.

However, look at the output stream being used.
That's a ByteArrayOutputStream.

So, it's not writing to an external place, it's writing to an array internal to your app.
So each 16k block of data is being added to the array held in this class. That's what's happening in the grow() method.

So where is this stream coming from?
Looks like it's from the uploader supplied, which is called on line 30 of the DbxUploadStyleBuilder.

This is supplied by that call to uploadBuilder, which is based (I believe) in the config.

So...what is the "dropbox/java-tutorial" client identifier?
Is it, by any chance, a config for simple testing that doesn't actually hit dropbox?

That would be where I would look.

Hi Dave finally i got it working using new api

here is the code

static final String ACCESS_TOKEN = "<>";
	private static String s = System.getenv("OPENSHIFT_DATA_DIR");
long buffer = 33554432; // 32mb buffer
				DbxRequestConfig config = new DbxRequestConfig("dropbox/java-tutorial",
						"en_US");
				DbxClientV2 client = new DbxClientV2(config, ACCESS_TOKEN);
				try {
					URL url = new URL(url_d);
					String sessionId = "";
					// Upload Files to Dropbox
					String fileName =  java.net.URLDecoder.decode(url_d.substring(url_d.lastIndexOf('/') + 1,url_d.length()), "UTF-8");;
					HttpURLConnection getLength = (HttpURLConnection) url.openConnection();
					getLength.connect();
					long size = getLength.getContentLengthLong();
					// Start
					HttpURLConnection hc = (HttpURLConnection) url.openConnection();
					hc.addRequestProperty("Range", "bytes=0-"+buffer);
					hc.connect();
					DbxFiles.UploadSessionStartUploader result = client.files.uploadSessionStart();
					result.getBody().write(IOUtils.toByteArray(hc.getInputStream()));
					UploadSessionStartResult sa = result.finish();
					sessionId = sa.sessionId;
					hc.disconnect();
					if (size > 0) {
						//System.out.println("large ");
						
						long s2 = buffer;
						long tmp = 0;
						//System.out.println("0"+" "+buffer);
						while (tmp <= size && (buffer + s2) < size) {
							buffer++;
							tmp = buffer + s2;
							if (tmp < size) {
								System.out.println(buffer + "\t" + (tmp)+"\t"+(tmp-buffer));
								// Append
								HttpURLConnection hc1 = (HttpURLConnection) url.openConnection();
								hc1.addRequestProperty("Range", "bytes="+buffer+"-"+tmp);
								hc1.setReadTimeout(10000);
								hc1.connect();
								UploadSessionAppendBuilder re1 = client.files.uploadSessionAppendBuilder(sessionId, buffer);
								re1.run(hc1.getInputStream());
								hc1.disconnect();
								buffer = tmp;
							}
						}
						if((tmp+1)+buffer>size){
						//System.out.println(tmp + 1 + "\t" + size +"\t"+(tmp-size));
						// finish
						HttpURLConnection hc2 = (HttpURLConnection) url.openConnection();
						hc2.addRequestProperty("Range", "bytes="+(tmp+1)+"-"+size);
						hc2.setReadTimeout(10000);
						hc2.connect();
						UploadSessionCursor usc = new UploadSessionCursor(sessionId, (tmp+1));
						FileMetadata nn = client.files.uploadSessionFinishBuilder(
								usc,
								new CommitInfo("/" + fileName, DbxFiles.WriteMode.add,false, new Date(), false))
								.run(hc2.getInputStream());
						// End
						System.out.println("Done");
						redirectAttrs.addFlashAttribute("AttributeName", "Successfully uploaded to dropbox");
						}
					}
					
					
					
				} catch (MalformedURLException e) {
					e.printStackTrace();
				} catch (IOException e) {
					e.printStackTrace();
				} catch (UploadException e) {
					e.printStackTrace();
				} catch (DbxException e) {
					e.printStackTrace();
				}

[Paul Clapham]

Thanks for coming back to let us know about your fix, Pankaj!