aspose file tools*
The moose likes I/O and Streams and the fly likes Java nio Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Java » I/O and Streams
Bookmark "Java nio" Watch "Java nio" New topic
Author

Java nio

jubin kart
Greenhorn

Joined: Jul 19, 2008
Posts: 8
Hi,
I was trying out java nio and I wanted to see the difference between nio and the normal file stream.
I tried copying over 115MB using both methods and they both got the job done in 3 sec. Am i not supossed to see considerable difference in copying or is this for only very large files in GB.
Should I try something else to see a difference.

Jubin


The only true wisdom is in knowing you know nothing.
jubin kart
Greenhorn

Joined: Jul 19, 2008
Posts: 8
Hi,

I was trying to run the following code for memory mapped copying of files but it kept giving me java.io.IOException: Map failed. The file is about 700MB and does get copied only thing its corrupted and this exception is thrown.


any ideas... ?
Rob Spoor
Sheriff

Joined: Oct 27, 2005
Posts: 19785
    
  20

"being_nice",

Please read your private messages regarding an important announcement.

Thank you,

Rob


SCJP 1.4 - SCJP 6 - SCWCD 5 - OCEEJBD 6
How To Ask Questions How To Answer Questions
jubin kart
Greenhorn

Joined: Jul 19, 2008
Posts: 8
done...
Bear Bibeault
Author and ninkuma
Marshal

Joined: Jan 10, 2002
Posts: 61766
    
  67

Not done. "jubin", please check your private messages again. And read carefully.


[Asking smart questions] [Bear's FrontMan] [About Bear] [Books by Bear]
Carey Evans
Ranch Hand

Joined: May 27, 2008
Posts: 225

The JavaDoc for the map() methods says:
The behavior of this method when the requested region is not completely contained within this channel's file is unspecified.
If the output file doesn't already exist, you should probably make it the right length first.

I don't think this will work very well though. As far as I can tell, the put() method you're calling will copy the file byte by byte. Copying from a FileInputStream to a FileOutputStream using a large enough byte array as a buffer should work better.
Norm Radder
Ranch Hand

Joined: Aug 10, 2005
Posts: 690
    
    1
You are not seeing the TRUE cause of the problem.
Add a printStackTrace() call to all the catch(){} blocks and you'll see!
jubin kart
Greenhorn

Joined: Jul 19, 2008
Posts: 8
thanks guys,
Printing out the stack trace helped. So how would I know how much memory I can copy over for a system , can I can I do it in parts?

jubin kart
Greenhorn

Joined: Jul 19, 2008
Posts: 8
I did this in the end. Although it copied, the file came out corrupt and there isn't much difference between stream copy and nio.

Mike Simmons
Ranch Hand

Joined: Mar 05, 2008
Posts: 3018
    
  10
Unfortunately, the API tells us nothing about whether memory-mapped files can ever be freed up after the mapping occurs. Depending on the operating system, the memory usage may be permanent. There's no particular reason to expect that it should be possible to map part of the file, free it up, and then map another part. Maybe it will work, or maybe not. But I wouldn't count on it.

Generally, if you want to copy files quickly using NIO, don't use map() - use transferTo():



Depending on your operating system and hardware, you may see substantial improvement with this code - or not. In many cases NIO is not significantly faster than regular IO, because many of the improvements of NIO have already been incorporated into IO beneath the covers. In some cases, however, further improvements are still possible. You won't really know until you try.
[ July 22, 2008: Message edited by: Mike Simmons ]
jubin kart
Greenhorn

Joined: Jul 19, 2008
Posts: 8
Thanks mike there is a very good difference in copy.
Norm Radder
Ranch Hand

Joined: Aug 10, 2005
Posts: 690
    
    1
I tried a few different methods and file sizes to see which way was fastest to copy a file. I used 3 files of about: 53M, 280M and 680M.
It appears no one way is fastest. Depended on the size of the file.
I'm on XP with 512M ram.
The program:

Results in next post
Norm Radder
Ranch Hand

Joined: Aug 10, 2005
Posts: 690
    
    1
The system rejected this when I had it tagged as QUOTE.

Fastest way to copy a large file

After reading the nio post I tried some experiments trying various ways to copy a large file.
Using


** copy2 - FileChannel 53M in 1.2 sec / 280M in 17.8-21 sec / 687M in 90-100 sec(79 w/128M)
D:\JavaDevelopment\Testing>java FileCopyTest
copy2 of E:\Downloads\Java\jdk-6-doc.zip size=54898268
copy2 done
copy time=1203, rate=45634
D:\JavaDevelopment\Testing>java FileCopyTest
copy2 of E:\Downloads\Java\jdk-6-doc.zip size=54898268
copy2 done
copy time=1031, rate=53247

D:\JavaDevelopment\Testing>java FileCopyTest
copy2 of E:\Downloads\Java.zip size=281183548
copy2 done
copy time=20969, rate=13409

D:\JavaDevelopment\Testing>java FileCopyTest
copy2 of E:\Downloads\Java.zip size=281183548
copy2 done
copy time=17656, rate=15925

D:\JavaDevelopment\Testing>java FileCopyTest
copy2 of E:\Downloads\Java.zip size=281183548
copy2 done
copy time=18985, rate=14810

D:\JavaDevelopment\Testing>java FileCopyTest
copy2 of E:\Downloads\Miscellaneous\CentOS-5.1-i386-LiveCD.iso size=720766976
copy2 done
copy time=89984, rate=8009

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy2 of E:\Downloads\Miscellaneous\CentOS-5.1-i386-LiveCD.iso size=720766976
copy2 done
copy time=79922, rate=9018

D:\JavaDevelopment\Testing>java FileCopyTest
copy2 of E:\Downloads\Miscellaneous\CentOS-5.1-i386-LiveCD.iso size=720766976
copy2 done
copy time=100703, rate=7157
>>>>>>>>>>>>> System now very sluggish/slow - effected first copy test below

-----------------------------------------------------------------------------
** copy3 - RandomAccessFile w/64M blocks 53M in 1.4-6.5sec / 280M in 23.5 sec / 687M in 65.5
D:\JavaDevelopment\Testing>java FileCopyTest
copy3 of E:\Downloads\Java\jdk-6-doc.zip to C:\Temp\jdk6doc.zip
setLength took=47, length=54898268, copySize=54898268
i/p seek took=0, seekPt=0
output seek took=0
read took=4875
write took=1078
Close took=0
copy time=6484, rate=8466

>>>>>>>>> System response time more normal
D:\JavaDevelopment\Testing>java FileCopyTest
copy3 of E:\Downloads\Java\jdk-6-doc.zip to C:\Temp\jdk6doc.zip
setLength took=0, length=54898268, copySize=54898268
i/p seek took=0, seekPt=0
output seek took=0
read took=219
write took=1047
Close took=0
copy time=1359, rate=40396 <<<<<<<<< 1.4 sec

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest <<<<<<<<<< NB took 128M
copy3 of E:\Downloads\Java.zip to C:\Temp\java.zip
setLength took=15, length=281183548, copySize=67108863
i/p seek took=0, seekPt=214074685
output seek took=0
read took=2172
write took=5531
Close took=0
opens took=0
i/p seek took=0, seekPt=146965822
output seek took=0
read took=1938
write took=2703
Close took=0
opens took=0
i/p seek took=0, seekPt=79856959
output seek took=0
read took=1844
write took=2547
Close took=0
opens took=0
i/p seek took=0, seekPt=12748096
output seek took=0
read took=1797
write took=2672
Close took=0
opens took=0
i/p seek took=0, seekPt=0
output seek took=0
read took=593
write took=1578
Close took=0
copy time=23531, rate=2851

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy3 of E:\Downloads\Java.zip to C:\Temp\java.zip
setLength took=0, length=281183548, copySize=67108863
i/p seek took=0, seekPt=214074685
output seek took=0
read took=1890
write took=5516
Close took=0
opens took=16
i/p seek took=0, seekPt=146965822
output seek took=0
read took=1860
write took=2593
Close took=0
opens took=0
i/p seek took=0, seekPt=79856959
output seek took=0
read took=1828
write took=2672
Close took=0
opens took=0
i/p seek took=0, seekPt=12748096
output seek took=0
read took=1797
write took=2640
Close took=0
opens took=0
i/p seek took=0, seekPt=0
output seek took=0
read took=609
write took=1500
Close took=0
copy time=23093, rate=2906

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy3 of E:\Downloads\Miscellaneous\CentOS-5.1-i386-LiveCD.iso to C:\Temp\Testing.iso
setLength took=0, length=720766976, copySize=67108863
i/p seek took=0, seekPt=653658113
output seek took=0
read took=2078
write took=15672
Close took=0
opens took=0
i/p seek took=0, seekPt=586549250
output seek took=0
read took=2671
write took=2750
Close took=0
opens took=0
i/p seek took=0, seekPt=519440387
output seek took=0
read took=2063
write took=2765
Close took=0
opens took=0
i/p seek took=0, seekPt=452331524
output seek took=0
read took=2063
write took=2656
Close took=0
opens took=0
i/p seek took=0, seekPt=385222661
output seek took=0
read took=1984
write took=2625
Close took=16
opens took=0
i/p seek took=0, seekPt=318113798
output seek took=0
read took=2000
write took=2640
Close took=0
opens took=0
i/p seek took=0, seekPt=251004935
output seek took=0
read took=2000
write took=2625
Close took=0
opens took=0
i/p seek took=0, seekPt=183896072
output seek took=0
read took=2000
write took=2656
Close took=0
opens took=0
i/p seek took=0, seekPt=116787209
output seek took=0
read took=2000
write took=2578
Close took=0
opens took=0
i/p seek took=0, seekPt=49678346
output seek took=0
read took=2015
write took=2563
Close took=0
opens took=0
i/p seek took=0, seekPt=0
output seek took=0
read took=2031
write took=2891
Close took=0
copy time=65531, rate=1024

-----------------------------------------------------------------------------
** copy1 MappedByteBuffer 53M in 0.15-2.3 sec / 280M in 20-24 sec
??? Was first one slow because System sluggish from above??? 2nd&3rd VERY FAST
D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy1 E:\Downloads\Java\jdk-6-doc.zip size is 54898268, dur=484
MaxRec=54898268
ix=0 at 1216742943687, map input=31, map output=78, put=1609
close=0
copy time=2281, rate=24067 <<<<<<<<<<<<<<<???

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy1 E:\Downloads\Java\jdk-6-doc.zip size is 54898268, dur=0
MaxRec=54898268
ix=0 at 1216743046937, map input=0, map output=0, put=141
close=0
copy time=157, rate=349670

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy1 E:\Downloads\Java\jdk-6-doc.zip size is 54898268, dur=16
MaxRec=54898268
ix=0 at 1216743171734, map input=0, map output=0, put=156
close=0
copy time=188, rate=292012

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy1 E:\Downloads\Java.zip size is 281183548, dur=31
MaxRec=67108863
ix=0 at 1216743254265, map input=0, map output=31, put=2000
ix=67108863 at 1216743256296, map input=0, map output=0, put=1688
ix=134217726 at 1216743257984, map input=0, map output=0, put=4875
ix=201326589 at 1216743264140, map input=31, map output=0, put=9922
ix=214074685 at 1216743274109, map input=0, map output=31, put=3594
close=562
copy time=24141, rate=11647

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy1 E:\Downloads\Java.zip size is 281183548, dur=234
MaxRec=67108863
ix=0 at 1216743448265, map input=31, map output=32, put=6172
ix=67108863 at 1216743454500, map input=0, map output=0, put=1046
ix=134217726 at 1216743455546, map input=0, map output=0, put=4344
ix=201326589 at 1216743460578, map input=78, map output=0, put=2625
ix=214074685 at 1216743464437, map input=266, map output=0, put=2547
close=625
copy time=19906, rate=14125

D:\JavaDevelopment\Testing>java -Xmx128M FileCopyTest
copy1 E:\Downloads\Java.zip size is 281183548, dur=235
MaxRec=67108863
ix=0 at 1216743575750, map input=31, map output=31, put=1734
ix=67108863 at 1216743577546, map input=0, map output=32, put=6906
ix=134217726 at 1216743585781, map input=156, map output=16, put=56250
ix=201326589 at 1216743643828, map input=203, map output=234, put=57891
ix=214074685 at 1216743704968, map input=546, map output=266, put=8750
close=500
copy time=139688, rate=2012 <<<<<<<<<<<<< OS went away ???
Carey Evans
Ranch Hand

Joined: May 27, 2008
Posts: 225

I don�t know how it compares to mapping the files, but you can gain quite a bit of performance by using file channels with a direct byte buffer. For example:
Mike Simmons
Ranch Hand

Joined: Mar 05, 2008
Posts: 3018
    
  10
[Carey]: I don�t know how it compares to mapping the files

Or more pertinently, how it compares to using transferTo() or perhaps transferFrom(). The key here is: if you don't need to change or analyze anything about the content - if you only need to copy the bytes and nothing else: use a technique that communicates this basic intent. And trust that it may be implemented by people who aren't total morons, and are capable of being efficient once they realize that we don't care about the content, as long as it's transferred accurately. Most other APIs (really, all other Java APIs I'm aware of) try to transfer the content to some intermediate form - most commonly, a byte[] array. Which would allow our program to read individual bytes, and analyze them. If we cared to do so. But often, we don't. Why waste time transferring bytes to an intermediate readable data structure if you don't need to read the bytes? Just tell the underlying system what you want: move content from foo to bar, and don't waste time making those bytes available to our Java program (located somewhere else entirely). We don't care. Just move the bytes. That's what transferTo() and transferFrom() can accomplish.
[ July 24, 2008: Message edited by: Mike Simmons ]
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Java nio