comparing records

[S Shehab]

Hi Guys ,

I got nearly 6 million record , and other 1 million records , i need to check with Java if the 6 million records contains the 1 million records or no , what is the fastest from performance perspective to do this in java ?

[Jim Hoglund]

I would sort both records, independently, and then step
through them in parallel to answer your question.
Jim ...

[Rob Spoor]

Sherif Shehab wrote:I got nearly 6 million record , and other 1 million records , i need to check with Java if the 6 million records contains the 1 million records or no , what is the fastest from performance perspective to do this in java ?

If you're talking about database records, then letting the database handle it is probably the best solution. Use an INNER JOIN, LEFT OUTER JOIN, RIGHT OUTER JOIN or FULL OUTER JOIN to combine the two tables.

For instance, to get the number of records that are in the 1 million but not in the 6 million (let's call them tables "one" and "six"):

SELECT COUNT(*)
FROM one LEFT OUTER JOIN six ON one.field1 = six.field1
WHERE six.field1 IS NULL

This will link the two tables on the two fields specified in the JOIN clause. Every record of table "one" without a matching record in table "six" will have NULL values for all columns of table "six". The WHERE clause then selects only these matches, and the COUNT(*) returns the number of records.


If they're not database records, use Maps.

[S Shehab]

if they're not database records, use Maps.

What about Sets vs Maps from performance perspective, because what i'm thinking in is to use HashSet for the two groups of data as you name them one and six to sure there is no duplication , then check if the six HashSet contains what in one HashSet , What do you think ?

[Rob Spoor]

It depends how you wrote your equals() and hashCode() methods. It they are not compatible a HashMap is better, with the common keys as the map's keys and the records themselves as the objects.

If you use a Set or Map, check out the bulk methods containsAll, removeAll and retainAll defined in java.util.Collection. Map has methods keySet(), values() and entrySet() you can use to get a Collection (Set extends Collection).

[Christophe Verré]

You don't plan to hold these millions of keys in a Map, do you ?

[S Shehab]

I think i;ll go for Sets , but what is more faster in iteration on the Set for loop or an iterator ? or both are same ?

[Christophe Verré]

If possible, I'd do what Jim said in his posts. Where are your records ? In a database ? In a file ?

[S Shehab]

Christophe Verré wrote:If possible, I'd do what Jim said in his posts. Where are your records ? In a database ? In a file ?

Actually the records are in DB , but they dont want me to do anything on the DB for major performance issues , this why i need to put them in some Collections to the comparing on them ..

[Rob Spoor]

So they are moving the performance problem from the database server to the application? Great...

[S Shehab]

Rob Prime wrote:So they are moving the performance problem from the database server to the application? Great...

Ya Rob

[Jim Hoglund]

Maybe you can creep up on those pesky DBAs. If they are
nervous about the joins, maybe you can at least get each
output record sorted before you receive it.
Jim ... ...

[Campbell Ritchie]

Rob is right (he usually is), and you will probably get better performance asking the database to do that; database management programs are specially optimised for that sort of query.