aspose file tools*
The moose likes I/O and Streams and the fly likes File Size Issue while delivering file more than 2gb using java.nio.channels.FileChannel.transferFrom Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » I/O and Streams
Bookmark "File Size Issue while delivering file more than 2gb using java.nio.channels.FileChannel.transferFrom" Watch "File Size Issue while delivering file more than 2gb using java.nio.channels.FileChannel.transferFrom" New topic
Author

File Size Issue while delivering file more than 2gb using java.nio.channels.FileChannel.transferFrom

sup rty
Greenhorn

Joined: Mar 04, 2010
Posts: 22
Hi,

I am facing an issue while delivering a file more than 2gb using the API java.nio.channels.FileChannel.transferFrom. The size of the file copied from source is only till 2gb.

Kindly let me know if anyone could help me out with this.
Basically I need to find out about some JAva api which could help in coping files more than 2 Gb.


thanks
Wouter Oet
Saloon Keeper

Joined: Oct 25, 2008
Posts: 2700

Too difficult for Beginning Java. Moved to I/O and Streams.


"Any fool can write code that a computer can understand. Good programmers write code that humans can understand." --- Martin Fowler
Please correct my English.
sup rty
Greenhorn

Joined: Mar 04, 2010
Posts: 22
thanks Wouter..

also to add give more information...The file delivery is done using FTP copy method...in my code I have something like this:

destChannel.transferFrom(srcChannel, 0, srcChannel.size());

This java.nio.channels.FileChannel.transferFrom is basically transferring only 2 gb of the data and hence there is mismatch between the source file and destination file size.
So I am assuming it does not alllow file copy for more tha 2 gb...So if any one has any idea and then please do let me know...


Thanks in advance
Joe Ess
Bartender

Joined: Oct 29, 2001
Posts: 8911
    
    8

I am not aware of a limit on the amount of data that the Java IO API can handle. You are probably running into a file system limit. What's the file system on the destination?


"blabbing like a narcissistic fool with a superiority complex" ~ N.A.
[How To Ask Questions On JavaRanch]
sup rty
Greenhorn

Joined: Mar 04, 2010
Posts: 22
Hi Joe Thanks for the reply.....
I am checking the file size limit with the system. So according to you the there cannot be any issue with the java api in delivering files over 2 gb size. are there any specific ways or any methods or any API by which we could deliver such huge files?
Please let me know.
Rob Spoor
Sheriff

Joined: Oct 27, 2005
Posts: 19697
    
  20

Is the destination file system FAT32? Because that has its own limit of 2GB.


SCJP 1.4 - SCJP 6 - SCWCD 5 - OCEEJBD 6
How To Ask Questions How To Answer Questions
Raghavan Muthu
Ranch Hand

Joined: Apr 20, 2006
Posts: 3344

May be an additional info:

how about the available memory in terms of total RAM and allocated size to JVM? will it be of some help if being fine tuned?

As far as I have experienced, the I/O related stuff behave differently on different platforms. One good example is the way /tmp file works. Have you checked the documentation on the target platform?


Everything has got its own deadline including one's EGO!
[CodeBarn] [Java Concepts-easily] [Corey's articles] [SCJP-SUN] [Servlet Examples] [Java Beginners FAQ] [Sun-Java Tutorials] [Java Coding Guidelines]
Mike Simmons
Ranch Hand

Joined: Mar 05, 2008
Posts: 3015
    
  10
Are you calling transferFrom() in a loop, checking the return value to ensure that all bytes are transferred? The API does indicate it's possible that not all bytes are transferred in one call, though it's vague about why this may occur. I would do something like this:
Raghavan Muthu
Ranch Hand

Joined: Apr 20, 2006
Posts: 3344

Thats a good tip Mike Simmons!
sup rty
Greenhorn

Joined: Mar 04, 2010
Posts: 22
Hi ALL,

There seems to be no size limit in the destination system.

the filesize limit that you mean witth 2 gb files is in FAT 16 Filesystem (like MS DOS 3.x)
the FAT32 Filesystem has a file limit with 4GB, (like WIN95,...)

we are on UNIX (Solaris) there are know filesize limits on UFS or Storenext Filesystems
Joe Ess
Bartender

Joined: Oct 29, 2001
Posts: 8911
    
    8

Raghavan Muthu wrote:how about the available memory in terms of total RAM and allocated size to JVM? will it be of some help if being fine tuned?


If he were running into a RAM issue, I'd expect to see an OutOfMemory exception. Now there is a size limit on a MappedByteBuffer: Integer.MAX_VALUE (which just happens to be 2GB), but that shouldn't affect a plain transfer.
Sup, are you catching Exceptions correctly?
Have you tried using the plain java.io classes to accomplish this copy?
Have you tried a non-Java method to copy the file?
Do you have some code so we can reproduce this problem?
Mike Simmons
Ranch Hand

Joined: Mar 05, 2008
Posts: 3015
    
  10
So, Sup: have you tried calling the method in a loop like I suggested above?
sup rty
Greenhorn

Joined: Mar 04, 2010
Posts: 22
Hi Mike,

I tried the for loop but facing the same issue. Only 2gb of the file is getting copied on the destination. i tried this:

srcChannel = new FileInputStream(sourceFile).getChannel();

destChannel = new FileOutputStream(fileDeliveryTemp).getChannel();

for (long n = 0, s = srcChannel.size(); n < s; ) {
s_logger.debug("inside for loop " + destChannel.size());
n += destChannel.transferFrom(srcChannel, n, s - n);
}
Mike Simmons
Ranch Hand

Joined: Mar 05, 2008
Posts: 3015
    
  10
Interesting.

By the way, please UseCodeTags in the future.

Have you tried closing the stream and/or channel? Either should be sufficient, and I should have asked this earlier. That should be enough - but if it doesn't work, also try FileChannel's force() method. I don't think it matters much whether you use force(true) or force(false).

If it's still not working, what does your new debug statement tell you? Better yet, let's add a few more debugs:

What does the log output tell you? How many times is the loop really executing? What is the total number of bytes that the methods claims have been written (n)?
 
With a little knowledge, a cast iron skillet is non-stick and lasts a lifetime.
 
subject: File Size Issue while delivering file more than 2gb using java.nio.channels.FileChannel.transferFrom