File APIs for Java Developers
Manipulate DOC, XLS, PPT, PDF and many others from your application.
http://aspose.com/file-tools
The moose likes General Computing and the fly likes Setting up memory and disk writes to avoid using virtual memory. Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Engineering » General Computing
Bookmark "Setting up memory and disk writes to avoid using virtual memory." Watch "Setting up memory and disk writes to avoid using virtual memory." New topic
Author

Setting up memory and disk writes to avoid using virtual memory.

Nate Lockwood
Ranch Hand

Joined: Feb 22, 2010
Posts: 83
My application drives a couple of high tech cameras on gigabit Ethernet and flogs the data and meta data about the image to disk. I'm not getting the throughput I want but the machine I'm working with will not be the one that drives them when the app is finished (new computer not delivered yet). I am trying to set up my app so it won't use virtual memory and will use direct disk access. I'll support at least two cameras. After setup the application image acquisition threads just wait for data to be made available in a circular buffer for each camera, extract some meta-data, serializes the meta data as xml, and writes the image to disk. The image is about 22 MB, the xml files less than 1000 bytes.

The buffers are byte buffers and llocated directly. AFAIK they are not in the heap but are elsewhere. is this correct? If I allocate 20 of them this would be in excess of 440 MB. I expect that I will need to allocate between 5 and 10 for each camera. How can I see if they are being swapped by virtual memory?

I write them to disk like as below and AFAIK this should not result in a copy being made first, is that correct? I want to be as efficient as possible.

TIA
Campbell Ritchie
Sheriff

Joined: Oct 13, 2005
Posts: 38509
    
  23
Too difficult a question for "beginning". Not sure whether to move it to "Java in General" or "General computing", but maybe the latter would be better.
Rob Spoor
Sheriff

Joined: Oct 27, 2005
Posts: 19684
    
  20

Nate Lockwood wrote:I am trying to set up my app so it won't use virtual memory and will use direct disk access.

I doubt that will give you much performance gain, as virtual memory is also stored on the disk. All you are doing is moving the management of where the data is stored from the OS' memory manager to your own location.


SCJP 1.4 - SCJP 6 - SCWCD 5 - OCEEJBD 6
How To Ask Questions How To Answer Questions
Nate Lockwood
Ranch Hand

Joined: Feb 22, 2010
Posts: 83
Too difficult a question for "beginning".

I'm pretty new at Java but not at programming. Started to learn Java on my own just before I registered here.


I doubt that will give you much performance gain, as virtual memory is also stored on the disk. All you are doing is moving the management of where the data is stored from the OS' memory manager to your own location.


The model in my mind is that the vendor's interface moves the data in each packet from the LAN into a buffer, marks it as ready when it's all there and then moves to the next buffer. Then my code detects that the buffer is ready and tells the VM to store it on disk in whatever chunks it uses. What I don't want and what will make this app fail is if the system or VM decides that the buffers should be subject to virtual memory. If that happens the disk will be (or is) thrashing, eating up cpu cycles for nothing. I'm trying to identify what can be tuned for the very fastest performance. The computer I'm using now appears to be CPU bound and I can only get pairs of images about 1.8 seconds apart but I would like about 1.2 seconds apart.

Thanks
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Setting up memory and disk writes to avoid using virtual memory.