Let's forget the data for a moment, and look at what the user wants to do with it. With thousands of possibilities, I guess a user is unlikely to want to step manually through all of them to look for something (I certainly wouldn't want to).
Presumably what a user would sometimes want to do is ask questions like: "how many people work for company A", or "which company does person X work for". In any cases like this the database should do the filtering using a "where" or "count" clause. In these cases the amount of data transmitted will be small.
Sometimes, I can imagine wanting to ask "who works for company A", but as with most web searches,
you should probably consider sending back a page full (say the first 20 matches) and an indicator if there are any more to view. This gives the user a chance to back out without having to wait a long time if he enters the wrong company name by mistake or gets bored with lookimg through the list.
In this case I might expect to see a HTTP request something like
<pre>GET
http://server/servlets/people?company=Microsoft&action=first</pre>
followed by optional requests like
<pre>GET
http://server/servlets/people?company=Microsoft&action=next</pre>
to get the next page worth.
Each "first" request causes a "select name,etc from people where company='Microsoft'" to be sent to the database, and the first 20 (or all if there are less than 20) records to be sent to the client.
If you store your ResultSets in a Hashtable keyed by company name, then each "next" request just grabs the next few records from the ResultSet and sends it onward in a Vector (or whatever).
I'm sure your actual implementation will be different from this, but the things to bear in mind are:
- Allow the user to ask sensible, specific questions, don't just show them the whole lot at once.
- Do as much filtering in the database as you can. It's what it's best at.
- Allow for sending partial blocks of results, and leave the servlet in control of how large they are.
- Make the applet flexible so it will show however many the servlet sends, and allow the user to show more if required.