Hi Nut.
I am afraid I have difficulties understanding your question. My mother tongue is not english so this happens from time to time.
You say you have gotten data
from the DB, and put
into some type of collection. This means you are through with accessing the DB? If 'What kind of data structure could I use to store and access (large amounts of) data in the following format, in an efficient manner?' is a valid rephrase of your question, a proposed answer follows:
You can tailor a class to your data needs, and store instances of this class in an array (or maybe a Vector Object or some tree structure) What works best, depends of course on how you intend to use the data. Generally, if you will access them more than once, all calculation should be done during creation/ initialization of your structure. To explain: Let's say you have a huge amount of data, say a million records have been extracted from your DB, and you are about to insert them into your to-be-memory-resident data structure. Now, any way you want to do this, if you are planning on accessing all of the records by using your composite key, you will have to generate the key (from the two sub-keys) one million times. This you can not escape. The concern is whether you can get away with one million. If you plan to access (search through) the structure just once after it's been created, you can generate the key during the search, and then throw away the structure. If you plan to do further searches, the good thing to do is to design your class with room for an extra field, the generated key, and create the key while initializing the structure. That way you don't have to re-create the key next time. (Note that String operations are quite costly, as String Objects are immutable objects, and thus a concat() operation for instance, creates a new String Object. Object creation is especially heavy, but no operation should be done more than once if possible). I don't know what you mean by 'setting up a key object using the two sub keys and implementing a comparitor'. If that means creating another Object (a Key Object) for every record, I would not recommend it. Avoid Object creation when not necessary, at least when we speak large numbers.
You should implement your comparison by writing a method accessible to the Object that accesses the data (alternatively, use a static method in your 'Data' class). Note that the method will be used whenever you need to compare, that means a million times (here) if you search through all records once, or 1M EXP 1M (math notation is not my strongest side but that is 1M raised to the power of 1M) if you do a recursive (bubble-)sorting in a worst case scenario. Your computer will be out of date (and so will you) long before you will be able to calculate that number, so I mention it here just to give the big picture on algorithm optimization
The point is really that one million records are more than enough for you to invest some time in clever coding...
An array or some other Object? Again, that depends. You will probably know the number of records when you start initializing your structure. If not, it would probably be an easy task to add an
int primitive to keep track of this while reading from the DB. (Another approach is to spawn another
Thread and have that Thread ask the DB while the 'main' Thread gets the data. That's Object creation all right, but only once. It's the amount that's significant) If you know the record count, there are few situations where you should not use an array. A Vector, for instance, have more functionality than an array, but is more costly to use. For instance, you have to use a method whenever inserting or retrieving data, and it can only hold objects that are or subclass the 'Object' object. (Your tailored class can subclass Object if you decide to use a Vector, that way you won't have to cast it for insertion/deletion) I am not familiar with the 'TreeMap' Object you mention, (I actually don't do much straight
Java these days, more scripting in fact, so I'm a bit rusty, but what I say here are general design guidelines that apply to all coding independent of language) and it could be a good choice, but from the name it seems to be made for working with tree structures, which you probably don't want. Tree structures are fascinating, so fascinating that many programmers tend to forget or ignore that working on tree structures usually involves recursion, which except from a very limited set of areas are extremely ineffective. No matter how well Object Oriented your code may be, you should remember the fact that in all computers on the market today, memory (and disk drives) is accessed sequentially, and in 99% (or 95, 90? Let's just say an overwhelming majority
)of situations, code written to access an array in a sequential way will be the fastest. (There are some exceptions to this, but they apply to complex data scenarios and I don't think you need such approaches, and to say a lot about this would be exhaustive. I will mention as examples database tech, where indexing is used to retrieve data (In ISAM DB's like MS Access, this is in fact (additional) data about the data, that also is accessed sequentially.) and well-implemented (and large) tree structures. DB servers like Oracle and MS SQL Server employs tree structures for fast data access. (They also employ complex arithmetic to keep their trees tidy)
Also, I wonder where you have your data now that you read them from your DB, but still have not put them in your structure? But let's assume you have written a function that reads all the records and stores them (or will, when you have your data structure
).
OK, enough talk, here's a proposed class fer ya:
<PRE>
class DataClass{
String key;
String data1;
String data2;
//You say you need to access data1 and data2.
//So I saved us the extra space for two String variables
//that you would need to access the k1 and k2.
//Not a lot, it may seem, but it adds up to two million
//strings in our scenario.
public DataClass(String k1,String k2,String d1,String d2){
key=k1.concat(k2);
data1=d1;
data2=d2;
}
}
</PRE>
This class can hold what you need, two data Strings and one key (you didn't need the sub-keys, did you? If you do, just add two more string variables) It also performs the key generation upon construction so you are finished with that.
Now lets say you have a method to access the DB something like this:
<PRE>
DataClass[] mydata; //You need a global variable to keep the
// data between methods. (Global is of
//course a relative expression)
public void getData(){
int recordcount=-1;
UnKnownDBObject cursor; //This is to hold the object returned
//from the DB query. What kind of
//object depends. (That wasn't what
//your question really was, was it?)
try{
//Here goes the code to connect to your database
//and perform the query. This code is dependent on your
//local system. It usually returns some database object like
//a cursor, which you wrap in, or convert to, some object
//supported by your language (Java). This object will often
//keep the recordcount. If not, do a seperate query for the
//count while the data query is running. At any rate, we
//assume that the recordcount and the cursor variable is set
//here. (If there are no exceptions);
}
catch(Exception e){
System.out.println("An exception occured: "+e);
}
if(recordcount>0){
//The query returned records
mydata=new DataClass[recordcount];
UnknownDBObject2 record; //for working with one and one rec.
for(int i=recordcount;--i>=0
{
record=UnKnownDBObject.getRecord(i);
mydata[i]=new DataClass(record[0],record[1],record[2],record[3]);
}
}
}
</PRE>
The comparison and stuff is trivial, if you need help on that too, I will check in in a while..
Hope that helps,
Marius
