I am currently in the architecture analysis and design phase for a project which may very likely involve pure socket, thread and JDBC programming.
In this project, thousands of client devices may periodically send measurement data that has to be stored into a database system. The communication with the client devices would be over TCP/IP sockets.
These sockets would probably be long-lived because the server may have to collect information from the client devices in particular scenarios.
This data has to be decompressed/transformed into a format for storing into the database.
As bulk amounts of data are expected, this solution has to be highly performant and scalable.
I have a few questions here- 1. For each socket opened between the client and server, would it be advisable to dedicate a thread for reading the socket? Is there any limit to the threads and sockets that can be kept open on the server? If yes, it would probably be better to reuse the threads across various open sockets. How long can these sockets be kept alive?
2. Each request's data has to be processed (probably transformed) and inserted into database tables. Is it better to acquire a connection from a connection pool for each thread and insert the data into the database table or would a better architecture be to dedicate a thread for batch insertion of database tables over a single connection and make the receiver threads store data in a queue wherefrom this data would be picked up by this thread and inserted into the database in bulk. Again, how many threads for database interaction are recommended.
3. If I need to test this out, what would be the right approach/tools to use?
Also, you may be able to use a more common pattern, like that of a web-service, and instead of having "long" lived connections, have the server return to the client a "session token" that can be used to identify further calls as continuations of the conversation.
2. Connection Pool.
Of course, your server could well be running in a container that provides connection pooling so you'd get that "for free". [ December 14, 2005: Message edited by: Jeff Albrechtsen ]
Here's some code from a web server that does a similar mission in life:
For each inbound request it creates a new runnable handler and runs it on a thread from a pool. This particular pool will grow to as many threads as it needs. You could make one that has a fixed size and queues requests up when too many come in at once. Does that fit the problem?
A good question is never answered. It is not a bolt to be tightened into place but a seed to be planted and to bear more seed toward the hope of greening the landscape of the idea. John Ciardi