aspose file tools*
The moose likes I/O and Streams and the fly likes Saving lots of data to file Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Soft Skills this week in the Jobs Discussion forum!
JavaRanch » Java Forums » Java » I/O and Streams
Bookmark "Saving lots of data to file" Watch "Saving lots of data to file" New topic
Author

Saving lots of data to file

Colin McTaggart
Ranch Hand

Joined: Feb 01, 2011
Posts: 46
Hi all,
I am writing an app that needs to save a lot of data to a file. It stores information that originates in a database, each entry in the database consists of 9 strings and as it stands I have 142 entries. My app does not connect to the database at any point, another program on my server generates a file from the database and my app downloads this file. As it stands I use a plain text file to store my data and this is becoming a pain.

The file has to be stored in the same directory as my app as it has to be self contained so it can run on multiple computers from a pen drive using the same data file, just to clarify my app is not designed to run from the command line so saving data to the install directory does not cause complications.

It would be nice if this file were not easy for someone to read the data from but this is not all that important.

What are you guys recommendations on a format for storing this data?
Rob Spoor
Sheriff

Joined: Oct 27, 2005
Posts: 19783
    
  20

Easiest way: use Java's serialization mechanism. Create your own class that represents the data; make it implement java.io.Serializable. Then just create instances and serialize them using an ObjectOutputStream wrapped around a FileOutputStream. Because the number of elements may not be fixed you should write the number of objects first, using the writeInt method. After that you write exactly that many objects using the writeObject method.

The reading code then simply creates an ObjectInputStream wrapped around a FileInputStream. It first uses the readInt to get the number of objects, then calls the readObject method and casts the result to your class as often as the number indicates:

You could also have used CSV as file format but that's readable for everyone; serialized object files are readable but only if you know how to. Most users will not be able to get the contents.


SCJP 1.4 - SCJP 6 - SCWCD 5 - OCEEJBD 6
How To Ask Questions How To Answer Questions
Colin McTaggart
Ranch Hand

Joined: Feb 01, 2011
Posts: 46
you must be psychic :P, I already use a custom class as a container for the data and I store it in my app in an array-list. The number of entries in this array-list changes on the fly as users can add to it. Each entry in the array-list is added to a table and the file that stores this data is saved regularly. I like the sound of serialization but the problem is that each object in the array-list has a series of default values that change when the file is read in (as each item must be handled slightly differently on each computer).

I'm guessing what I need to do is is have an app on the server that reads my database into the same kind of array list as my app using the same object class which I will run manually when an update is ready. This server app then resets the relevant values to their defaults, then uses myArrayList.size(); to get my count then serialize and this will give me a single file which can be downloaded by the user and read in by my app which again will have to reset the relevant values to their defaults.

Have I understood this process properly? Also I have used output streams before but they still confuse me a little, should I use a buffered output stream and if so how big should I make the buffer?

Thanks for the reply, I'm actually quite excited about trying this as I've never heard of it before
Colin McTaggart
Ranch Hand

Joined: Feb 01, 2011
Posts: 46
I realized I had one more question as well, right now my app is in late alpha stage of development and as it stands I have just over 100 entries in my database. When I release my app it will most likely have 2-3 thousand entries. Right now I save 9 variables from each object in a text file but the object contains 18 variables in all. I was wondering if using serialization is likely to be quick to save and load as I want my app to be fairly efficient as well.
Colin McTaggart
Ranch Hand

Joined: Feb 01, 2011
Posts: 46
Ok, never mind I figured it out

I made the variables that had a default value transient and now amazingly the new file holds all the information the original did but is slightly smaller and seems just as quick to save.

I have a final question however. I tried opening the generated file in notepad++ and what I saw was a combination of junk data and the contents of my array-list. So it is not possible to see the names of variables but it is possible to see the values.

Is there any way to obfuscate this so that when opened in text editors none of it is legible?
Rob Spoor
Sheriff

Joined: Oct 27, 2005
Posts: 19783
    
  20

You can use an intermediate CipherOutputStream / CipherOutputStream. Instead of wrapping the FileOutputStream in an ObjectOutputStream, you wrap the FileOutputStream in a CipherOutputStream and you wrap that CipherOutputStream in an ObjectOutputStream. You can basically chain streams as far as you want; adding a GZIPOutputStream in the loop, not a problem! Just make sure that:

a) the reading and writing code must use the same order. So FileOutputStream in GZIPOutputStream in CipherOutputStream in ObjectOutputStream -> FileInputStream in GZIPInputStream in CipherInputStream in ObjectInputStream.

b) you read from the outside towards the inside. So in the a) example, you write objects (ObjectOutputStream) that will be encrypted (CipherOutputStream), then compressed (GZIPOutputStream), then written to a file (FileOutputStream). If you swap the GZIPOutputStream and CipherOutputStream, you will compress first, then encrypt.
Colin McTaggart
Ranch Hand

Joined: Feb 01, 2011
Posts: 46
That sounds perfect, I'm going to give it a go now, I'll post results if i succeed
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Saving lots of data to file