This week's book giveaway is in the Servlets forum.
We're giving away four copies of Murach's Java Servlets and JSP and have Joel Murach on-line!
See this thread for details.
The moose likes Other Big Data and the fly likes NoSQL queries for collections Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Murach's Java Servlets and JSP this week in the Servlets forum!
JavaRanch » Java Forums » Databases » Other Big Data
Bookmark "NoSQL queries for collections" Watch "NoSQL queries for collections" New topic
Author

NoSQL queries for collections

surlac surlacovich
Ranch Hand

Joined: Mar 12, 2013
Posts: 296

Hello Folks!
I've stumbled upon CQEngine, which offers NoSQL-like queries for Java collections.
That's cool, but why should we use that if vast majority of search queries goes to a Database which performs search for us?

So we don't need to fetch all the data, create indexed collection, perform search in RAM - everything is done for us by DB Engine.
Thus the CQEngine-like libraries looks not memory-efficient in compare with DB.
Maneesh Godbole
Saloon Keeper

Joined: Jul 26, 2007
Posts: 10167
    
    8

That's cool, but why should we use that if vast majority of search queries goes to a Database which performs search for us?

One scenario I can think off the cuff is serialized objects. You have have objects persisted to the disk or even over the wire which you need to search for some specific scenario (like filtering for e.g.)yo
Another one:
Consider a chat application where you have logged in and the UI shows a list of all your contacts who are currently online. Now you want to search for your friend called "Joe" or a friend who is in "Timbuktoo" Both these would be searchable attributes for the contact object.


[How to ask questions] [Donate a pint, save a life!] [Onff-turn it on!]
Winston Gutkowski
Bartender

Joined: Mar 17, 2011
Posts: 7503
    
  18

surlac surlacovich wrote:I've stumbled upon CQEngine, which offers NoSQL-like queries for Java collections.
That's cool, but why should we use that if vast majority of search queries goes to a Database which performs search for us?

You probably don't. But what about data that wouldn't normally be persisted in a database? Or that you don't want to be constantly querying or updating a database for? For example: properties, configuration information, fixed table data (eg, anything based on an ISO standard, like country or currency codes).

Would you want to be querying a database every time you had to retrieve the current language or timezone? Embedded databases are certainly fast these days, but JDBC and ResultSet's are still pretty clunky pieces of kit compared to native collections, and so more prone to error.

My only query is their choice of SQL. I suppose it was inevitable, but personally, I loathe SQL with a passion - probably an odd thing to hear from a bloke who was a DBA for nearly 20 years, but it's true .

Oddly enough, I've worked on an IndexManager myself, so I can certainly appreciate the idea. Sounds like they've taken it to the next level.

Winston

Isn't it funny how there's always time and money enough to do it WRONG?
Articles by Winston can be found here
surlac surlacovich
Ranch Hand

Joined: Mar 12, 2013
Posts: 296

Winston Gutkowski wrote:But what about data that wouldn't normally be persisted in a database?

Could you please give us small example of that kind of data?

Winston Gutkowski wrote:Or that you don't want to be constantly querying or updating a database for? For example: properties, configuration information, fixed table data (eg, anything based on an ISO standard, like country or currency codes).Would you want to be querying a database every time you had to retrieve the current language or timezone?

That's makes sence, because to query a database involves some overhead, but we're talking about the volumes of data when this overhead doesn't matter much. Because to see the performance gain with CQEngine vs Collections you need to query against really big number of objects. So my understanding is to keep all big data in database, and make database work for searching it, which will save us from waste of memory for keeping the big data in RAM.

Winston Gutkowski wrote:Embedded databases are certainly fast these days, but JDBC and ResultSet's are still pretty clunky pieces of kit compared to native collections, and so more prone to error.

You can use RowSet from Java 7 and all it's subclasses, which works quite fast.

Winston Gutkowski wrote:My only query is their choice of SQL.

So if we compare CQEngine with NoSQL database, I think it's make even less sence to use CQEngine, because NoSQL is even faster that RDBMS.
Winston Gutkowski
Bartender

Joined: Mar 17, 2011
Posts: 7503
    
  18

surlac surlacovich wrote:Could you please give us small example of that kind of data?

What? Other than properties or config stuff? I dunno: thread or connection pools, a cache of pretty much any kind, a collection used for a calculation...there could be any number of things.

You can use RowSet from Java 7 and all it's subclasses, which works quite fast.

Ooof. Really? It has 120 methods; and that doesn't even include the ones it inherits from ResultSet.

Give me a good old Java List any day.

Winston Gutkowski wrote:So if we compare CQEngine with NoSQL database, I think it's make even less sence to use CQEngine, because NoSQL is even faster that RDBMS.

In cases tailored to their use, sure. But NoSQL sets don't offer the ACID capabilities of an RDBMS.

The fact is, I simply don't know enough about CQEngine to argue its merits with you. All I can say is that I've worked on precisely some of the things it appears to incorporate for myself (specifically indexing) with Java collections, so I think I understand the reasoning behind it. But as a substitute for a database? Probably not - and I'm not sure that was ever the intent.

Winston
surlac surlacovich
Ranch Hand

Joined: Mar 12, 2013
Posts: 296

OK, now it's getting more clear for me, the CQEngine is like another level of memory which is fast but consumes RAM. The closes analogue is: CPU has about 16 registers, which is very fast but very small, next are levels of cache which is also in CPU, it's bigger but slower, next is RAM and HDD which are even slower but can contain more data.
So in case of CQEngine we have one more level in RAM which is fast for searching but restricted with actual size of RAM.
Winston Gutkowski
Bartender

Joined: Mar 17, 2011
Posts: 7503
    
  18

surlac surlacovich wrote:So in case of CQEngine we have one more level in RAM which is fast for searching but restricted with actual size of RAM.

Perhaps, but more importantly to me, it's a layer of logic that works with native Java collections, which are far more familiar to most programmers than JDBC, and a lot less clunky. In that respect, I'd say it's more akin to something like Joda Time - it enhances something that's already there.

But, as I say, I've never used it. I may well have a look at it now, though.

Winston
surlac surlacovich
Ranch Hand

Joined: Mar 12, 2013
Posts: 296

surlac surlacovich wrote:So in case of CQEngine we have one more level in RAM which is fast for searching but restricted with actual size of RAM.

I rethink my last sentence - more close example is in-memory cache (like Guava Cache or plain WeakHashMap), the CQEngine is something that will 'cache' not memory but 'calculations' - at the start it wastes time to create indexed collection but you can get really fast operations after that.
 
 
subject: NoSQL queries for collections
 
Similar Threads
How to do 'sql like' queries and joins on my java fields
Queries via MapReduce in NoSQL?
NOSQL
NoSQL Distilled: A Brief Guide to the Emerging World of Polyglot Persistence
Need information about databases for career - SQL and NoSQL