aspose file tools*
The moose likes Object Relational Mapping and the fly likes Using JDO for processing large amounts of data? Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login
JavaRanch » Java Forums » Databases » Object Relational Mapping
Bookmark "Using JDO for processing large amounts of data?" Watch "Using JDO for processing large amounts of data?" New topic
Author

Using JDO for processing large amounts of data?

Bob Dobalina
Ranch Hand

Joined: May 24, 2001
Posts: 53
I'm currently working on a solution for a problem that entails pulling large amounts of data from numerous very large XML files, then having to be able to search through that data quickly. The amount of data to be stored and searched through will be more than the system will have memory to handle, so I need to work with some kind of persistent storage that will allow me to search reasonably quickly. Your typical btree situation.
But, I wanted to take a stab at this problem using something like JDO, which seems like it would allow me to store the data I need (wrapped in objects) then use the query language to find it again. But, two problems. First, database interaction is not an option, and while JDO seems to support file-system persistence along with DB persistence, every implementation I've seen is totally DB focused. Secondly, I'm concerned about performance, and have no experience with JDO or any of its competing products/standards in that respect.
So, does anyone have some guidance? Does the reference JDO implementation allow for persistence to the file system? Or, should I go with something like Hibernate, and then tie it into a DBMS that I can create dynamically from my java code and that works on the file system? Or, should I scrap the whole idea and just code up a btree or something similar?
Thanks,
-tim
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
I won't say anything about whether you should go for a JDO solution vs writing a btree implementation since I haven't got the experience to back it up.
However, regarding the JDO stuff...
1) Yes, the reference implementation supports file system-based persistence (in fact, that's all it supports)
2) Yes, you could use Hibernate and a 100% Java database such as HSQLDB.


Author of Test Driven (2007) and Effective Unit Testing (2013) [Blog] [HowToAskQuestionsOnJavaRanch]
Bob Dobalina
Ranch Hand

Joined: May 24, 2001
Posts: 53
Thanks Lasse, I'll spend a little time trying to get the reference implementation working and see if I can't do a little testing there. HSQLDB was exactly what I was looking at as an option to use through Hibernate, which I may evaluate as well should the JDO implementation not fly.
But, if anyone else has any suggestions as far as performance or memory usage goes, please comment.
Thanks,
-tim
Erik Bengtson
Ranch Hand

Joined: Dec 06, 2003
Posts: 90
if you need performance and a mature xml to java binding tool, you are looking for jibx.
http://jibx.sourceforge.net/
no hibernate, no jdo.
Bob Dobalina
Ranch Hand

Joined: May 24, 2001
Posts: 53
Thanks Erik, I'll look into it, even though that sounded deceptively like a sales pitch
I'm just learning the reference implementation of Sun's JDO doesn't support querying, which was one of the main things I was looking for, after the ability to persist to a file. So, that may lead me to using a lightweight file-based DBMS, or just writing my own data structure to handle it.
Erik Bengtson
Ranch Hand

Joined: Dec 06, 2003
Posts: 90
If I wanted to sell something, I would tell you to try JPOX, an open source RDBMS JDO full compliant implementation, which I'm developer.
I also don't suggest you to use the JDORI, which has a lot of bugs.
Chris Mathews
Ranch Hand

Joined: Jul 18, 2001
Posts: 2712
I'll add my word of warning to Erik's. Do not use the JDORI for any production application!
Bob Dobalina
Ranch Hand

Joined: May 24, 2001
Posts: 53
Oh I won't, it doesn't have nearly enough functionality. But, it seems the open-source JDO implementations all are built off of the RI, including JPOX. Are they safe?
Erik Bengtson
Ranch Hand

Joined: Dec 06, 2003
Posts: 90
JPOX has already passed the JDO TCK tests (JDORI too), and runs in production environments.
Safe for me, might be not safe for you, and the opposite.
Safe for me is a product from companies like IBM, BEA, SUN, etc which I have support for at least one year.
Medium Safe for me is an open source application, mature and I have the source code.
It's up to you to evaluate. I can only recommend.
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Using JDO for processing large amounts of data?