Need help regarding serializing multiple Objects by difeerent Threads in a single fil

[Akhil Jain]

Hi Guys,

Need help urgently.
I am working on a thing to serialize my user-define object to a file.
I want to use FileChannel in NIO package of JDK 1.4 efficiently, problem i am facing is that how to serialize multiple objects to a single file by multiple threads. e.g there is a user defined class called Events and this event class object needs to be serialized in a file on disk say "events.ser". and there are multiple threads which are having that "Event Object" with different state and want to serialize them in same file.finally, "events.ser" should be structured like this:

eventobject1
eventobject2
eventobject3

So that, when a seperate program wants to inspect the "event.ser" file, it should be able to deserialize all the objects persisted by different threads and should be able to analyse them using reflection or whatever.

here are my sample codes i have written so far. please help me out in this.

Main Testing Class

 
public class TestEventsThroughChannels {
public static void main(String[] args){
Events event1 = new Events("Event1","Topic1",new Date(),new DummyEvent1(01,"dummyEvent1"));
Events event2 = new Events("Event2","Topic2",new Date(),new DummyEvent1(02,"dummyEvent1"));
Events event3 = new Events("Event3","Topic3",new Date(),new DummyEvent1(01,"dummyEvent2"));
Events event4 = new Events("Event4","Topic4",new Date(),new DummyEvent1(02,"dummyEvent2"));
Events event5 = new Events("Event5","Topic5",new Date(),new DummyEvent1(03,"dummyEvent1"));

//Put all event objects in an Array
Object[] eventArray =  new Object[]{event1, event2, event3, event4, event5};

//filename to persist these objects one by one
String filename = "events.ser";
File file  = new File(filename);
FileOutputStream out = null;
FileChannel channel = null;
try {
out = new FileOutputStream(file);
} catch (FileNotFoundException e) {
e.printStackTrace();
}
channel = out.getChannel();

//pass the same event Array to different threads to perfrom read and write to same event log file
Thread thread1 = new Thread(new Thread1(eventArray,channel));
Thread thread2 = new Thread(new Thread1(eventArray,channel));
Thread thread3 = new Thread(new Thread1(eventArray,channel));

thread1.setName("MyThread1");
thread1.start();

thread2.setName("MyThread2");
thread2.start();

thread3.setName("MyThread3");
thread3.start();
}
}[code] 
[B]Thread Class[/B]
[code]public class Thread1 implements Runnable, Serializable {
 
private Object[] objArray = null;
private FileChannel channel = null;

public Thread1(Object[] eventArray, FileChannel channel){
this.objArray = eventArray;
this.channel =channel;
}

public void run() {
// TODO Auto-generated method stub
for(int i=0;i<10;i++){
try {
Thread.sleep(1000);

//take the object array and pick any object randomly and 
//persist it to the file

ByteBuffer src = null;

ByteArrayOutputStream baos = null;
ObjectOutputStream oos = null;
byte[] bytes = null;

Random generator = new Random();
int index = generator.nextInt(objArray.length);
try{
Events eventToSerialize = (Events)objArray[index];

//create a byteArrayOutputStream to get byte[] finally
baos = new ByteArrayOutputStream();
oos = new ObjectOutputStream(baos);
oos.writeObject(eventToSerialize);

bytes = baos.toByteArray();

//wrap those byte[] to ByteBuffer, which will be send over channel to persist
src = ByteBuffer.wrap(bytes);

if(channel.isOpen()){
channel.write(src);
System.out.println("Picking Object at position : "+index
+" and Event Written by: "+Thread.currentThread().getName());
}

}catch(IOException ex){
ex.printStackTrace();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
}
try {
channel.close();
} catch (IOException e) {
e.printStackTrace();
}
}

}

[ October 17, 2007: Message edited by: Jim Yingst ]

[Stan James]

What does it do now that is not correct?

[Akhil Jain]

No the above code is not working as desired.... it does write to a file but when i deserialize it, it gives me StreamCurruptException and fails badly.
the deserialization code i am using is as follows:

package com.coreobjects.events;
 
import java.io.EOFException;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.ObjectInputStream;
import java.io.OptionalDataException;
import java.io.Serializable;
import java.util.ArrayList;
 
public class EventDeserializerThroughChannel implements Serializable{

private ArrayList objList = null;

public ArrayList deserializeEvents(File file) {

FileInputStream in = null;
ObjectInputStream ois = null;

try{
in = new FileInputStream(file);
ois = new ObjectInputStream(in);
int objectsCount = getObjectCount(ois);
System.out.println("number of Objects are : " + objectsCount);
 

}catch(IOException ex){
ex.printStackTrace();
}finally{
try {
ois.close();
in.close();
} catch (IOException e) {
e.printStackTrace();
}
}

return objList;
}
 
private int getObjectCount(ObjectInputStream ois){
int objectsCount = 0;
        boolean doneReadingInput = false;
        objList = new ArrayList();
        while (!doneReadingInput){
       Object result = null;
       try { 
          result = ois.readObject();
          
          
       } catch (EOFException eofe){
         doneReadingInput = true;
       } catch (OptionalDataException ode) {
          doneReadingInput = ode.eof; 
       } catch (IOException e) {
e.printStackTrace();
   } catch (ClassNotFoundException e) {
e.printStackTrace();
   }
       
          if (result != null){
          objectsCount++;
            Events event = (Events)result;
            objList.add(event); 
            //Printing some value from the object
          System.out.println(event.getName());
          }
        }
        return objectsCount;
}
 
public ArrayList getObjList() {
return objList;
}
 
public void setObjList(ArrayList objList) {
this.objList = objList;
}
}

What is the best way to serialize objects (the most efficient, keeping performance in mind), if we have to serialize around 20,000 Objects per sec to a file? and that too....a strategy which doesn't uses much of a synchronized blocks or methods (because they have their own performance bottlenecks) and allow multiple threads to persist objects in the same file?
[ October 17, 2007: Message edited by: Jim Yingst ]

[Nitesh Kant]

Your problem is that you are creating a new ObjectOutputStream in every thread to write an object. ObjectOutputStream writes a header once for every serialization process. ObjectInputStream expects the input stream to adhere by the above rule.
In your case, this header will be present before every object serialized in the stream. So, you get the exception. More details here

Originally posted by Akhil Jain:

What is the best way to serialize objects (the most efficient, keeping performance in mind), if we have to serialize around 20,000 Objects per sec to a file? and that too....a strategy which doesn't uses much of a synchronized blocks or methods (because they have their own performance bottlenecks) and allow multiple threads to persist objects in the same file?

If you have multiple threads writing to the same file then you have to synchronize the writes i.e. one thread writes to the file at a time. Otherwise the data will be corrupted. This is what channel does for you. It only allows one thread to write to a file at a time.

I think you have to write a single class that writes to the file and every other class just queues its events to this class. The writer takes one event at a time and writes to the file. I dont think this would scale very well as the number of event publishers will increase with time.

If you are writing a production level code, then i dont think this is the correct way of doing things. You can have a messaging system that can take these events and store them reliably.
The performance numbers you have put is for writing to the file or processing the events? I think the numbers are very aggressive to start with.
[ October 17, 2007: Message edited by: Nitesh Kant ]

[Akhil Jain]

I dint really got much from the thread link you pasted..
Does it mean that if multiple threads are writting to a same file, then the header should be written on once? or is it the other way round, that every thread's ObjectoutputStream should first write a header and then write its object to a file?

if you can tell me in detail that will be great, in this particular scenario, cause i tried passing the same ObjectOutputStream object to all the threads while instantiating them.
something like
....
ByteArrayOutputStream baos = new ByteArrayOutputStream();
....

Thread thread1 = new Thread(new Thread1(eventArray,baos,channel));

Please comment.

[Nitesh Kant]

Why do you want to use channels for doing this?

[Akhil Jain]

Because channel will automatically take care of Multiple Thread scenario and moreover, using synchronized startegy will have a nice perfromance hit with so many Objects being serialised and written concurrently by different threads in single file.

[Nitesh Kant]

Originally posted by Akhil Jain:
Because channel will automatically take care of Multiple Thread scenario and moreover, using synchronized startegy will have a nice perfromance hit with so many Objects being serialised and written concurrently by different threads in single file.

Channels does nothing but synchronization to stop concurrent updates to the same file. They do not do a synchronization on all the methods but for the methods that involve change in size of the file. This will be true in your case and so write to the file will always be one at a time.
As i told before, if multiple threads are writing to the same file then you *can not* do without synchronization.

[Akhil Jain]

Can you please give me an example of, how i can achieve the following scenario. because i am still getting StreamCurruptedException while deserializing the objects from file.

Strategy for writting multiple objects to same file with different threads (or may be single thread, which gets event objects from different threads to be serialized in a file), can be decided later on.

But the challenge i am facing right now is How efficient will be the use of FileChannels in this case and how can i test the scenario of serializing event objects to a file (approx, 5000 per sec to start with) and deserializing all objects accurately without getting StreamCurruptedException.

Any directions on this will be highly appreciated.


Thanks
Akhil

[Nitesh Kant]

Originally posted by Akhil Jain:
because i am still getting StreamCurruptedException while deserializing the objects from file.

What changes did you do in the code to avoid this exception?

Originally posted by Akhil Jain:

Strategy for writting multiple objects to same file with different threads (or may be single thread, which gets event objects from different threads to be serialized in a file), can be decided later on.

No, your design will depend totally on the above decision. I think you should have a single class that recieves requests to persist event objects from different threads. Using channels does not make sense to me as ObjectOutputStream does not provide a channel on the stream and putting a work around to use channels, just to get a thread-safe file writing is something non-intutive.
Yeah, there is a bug in ObjectOutputStream that does generate the StreamCorruptException if you want to append to an already exisiting file. Here is the bug parade link
As told earlier, the best and reliable way probably will be to use a messaging system. At least i will suggest that.
[ October 18, 2007: Message edited by: Nitesh Kant ]

[Scott Bro]

Strategy for writing multiple objects to same file with different threads: one way

class that contains .write method

ReentrantLock rtl = new ReentrantLock(true);

public boolean write(String fileName) {
rtl.lock();

// code to write to file

rtl.unlock();

}