Hi Walter,
Question for you : Is this one hour limit is because target system provides window for one hour for accepting the converted data
or is this because of source system which allows picking of data for only one hour.
or you want to do whole of the thing in one hour window ?
If first is the case, you can split the work in two parts
part 1 : create multiple threads to read data (use some logic to split data between threads) and once conversion is done, keep the converted data in DB or some filesystem
part 2 : when the target system one hour window is available then start sending converted data to target system. You can send the data to target system in batches or again by splitting converted data between multiple http threads or the combination of both.
I assume target system is able to cope with multiple http requests.
If second is the case, you can split the work in two parts
part 1 : pick all data from source system using multiple threads. Ofcourse you need split data between the threads. However, this time do not do the data conversion at this time. Just read and dump it in some db table or files.
part 2 : From the dump table, you can read the data using the threads, convert it and then send it target system.
If third is the case, you can still use the first solution, difference will be number of threads will be high.
What kind of webservice you are using? is it xml based or json based ? json based services are bit faster.
is this webservice is for sending the data to target system or reading the data from Source system ? 1.5 seconds is kind of slow in IMHO.
how much data webservice can support in one call ?