Win a copy of Spring Boot in Practice this week in the Spring forum!
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
programming forums Java Mobile Certification Databases Caching Books Engineering Micro Controllers OS Languages Paradigms IDEs Build Tools Frameworks Application Servers Open Source This Site Careers Other Pie Elite all forums
this forum made possible by our volunteer staff, including ...
Marshals:
  • Campbell Ritchie
  • Tim Cooke
  • Ron McLeod
  • Jeanne Boyarsky
  • Paul Clapham
Sheriffs:
  • Liutauras Vilda
  • Henry Wong
  • Devaka Cooray
Saloon Keepers:
  • Tim Moores
  • Stephan van Hulst
  • Tim Holloway
  • Al Hobbs
  • Carey Brown
Bartenders:
  • Piet Souris
  • Mikalai Zaikin
  • Himai Minh

Sockets, threads and JDBC - design question

 
Greenhorn
Posts: 3
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Hello,

I am currently in the architecture analysis and design phase for a project which may very likely involve pure socket, thread and JDBC programming.

In this project, thousands of client devices may periodically send measurement data that has to be stored into a database system. The communication with the client devices would be over TCP/IP sockets.

These sockets would probably be long-lived because the server may have to collect information from the client devices in particular scenarios.

This data has to be decompressed/transformed into a format for storing into the database.

As bulk amounts of data are expected, this solution has to be highly performant and scalable.

I have a few questions here-
1. For each socket opened between the client and server, would it be advisable to dedicate a thread for reading the socket? Is there any limit to the threads and sockets that can be kept open on the server? If yes, it would probably be better to reuse the threads across various open sockets. How long can these sockets be kept alive?

2. Each request's data has to be processed (probably transformed) and inserted into database tables. Is it better to acquire a connection from a connection pool for each thread and insert the data into the database table or would a better architecture be to dedicate a thread for batch insertion of database tables over a single connection and make the receiver threads store data in a queue wherefrom this data would be picked up by this thread and inserted into the database in bulk. Again, how many threads for database interaction are recommended.

3. If I need to test this out, what would be the right approach/tools to use?


Thanks and Regards,
Archanaa
 
Ranch Hand
Posts: 1780
  • Likes 1
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
1. Sockets + Server => SocketChannels. http://java.sun.com/developer/JDCTechTips/2003/tt0909.html#1

Also, you may be able to use a more common pattern, like that of a web-service,
and instead of having "long" lived connections, have the server return to
the client a "session token" that can be used to identify further calls
as continuations of the conversation.

2. Connection Pool.

Of course, your server could well be running in a container that provides
connection pooling so you'd get that "for free".
[ December 14, 2005: Message edited by: Jeff Albrechtsen ]
 
(instanceof Sidekick)
Posts: 8791
  • Mark post as helpful
  • send pies
    Number of slices to send:
    Optional 'thank-you' note:
  • Quote
  • Report post to moderator
Here's some code from a web server that does a similar mission in life:

For each inbound request it creates a new runnable handler and runs it on a thread from a pool. This particular pool will grow to as many threads as it needs. You could make one that has a fixed size and queues requests up when too many come in at once. Does that fit the problem?
 
reply
    Bookmark Topic Watch Topic
  • New Topic