aspose file tools*
The moose likes Web Services and the fly likes How to do Bulk data transfer using Web Service Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Spring in Action this week in the Spring forum!
JavaRanch » Java Forums » Java » Web Services
Bookmark "How to do Bulk data transfer using Web Service" Watch "How to do Bulk data transfer using Web Service" New topic
Author

How to do Bulk data transfer using Web Service

Vijit Krishnan
Greenhorn

Joined: Jun 11, 2009
Posts: 2
In my application I have to write various web services but majority of the web service has to
query database and return back bulk data(rows>10K) through web service.

So I would like to ask what is the efficient way of transferring bulk data using web service as presently
Iam returning the dataset as xml String (using StringBuilder) from web service and consuming the same at client end.
Is there a better way to this in web service?

My env:
Front end can be in any other technology ,UI like C#
Back end : Tomcat 6 on Java 6 with Axis2

Thanks in advance
William Brogden
Author and all-around good cowpoke
Rancher

Joined: Mar 22, 2000
Posts: 12803
    
    5
You certainly dont want to send any bulky data as part of the SOAP body, thats what SAAJ - SOAP with attachments for Java is for.

As I recall, Axis 2 provides what you need.

Bill
(historical note - extremely poor performance with large data items in SOAP is what prompted SAAJ development)
Ivan Krizsan
Ranch Hand

Joined: Oct 04, 2006
Posts: 2198
    
    1
Hi!
You may also want to consider to enable MTOM, which is supported in .NET too.
Best wishes!
Vijit Krishnan
Greenhorn

Joined: Jun 11, 2009
Posts: 2
William Brogden wrote:You certainly dont want to send any bulky data as part of the SOAP body, thats what SAAJ - SOAP with attachments for Java is for.

As I recall, Axis 2 provides what you need.

Bill
(historical note - extremely poor performance with large data items in SOAP is what prompted SAAJ development)


So you are recommending me to use attachments in web service to transfer POJO (like Employee object containing
EmpName,EmpID,Dept,DOB,DOJ) , as my data consists of recordset of similar kin like the emp example.

Thanks in advance
Ivan Krizsan
Ranch Hand

Joined: Oct 04, 2006
Posts: 2198
    
    1
Hi again!
After having reread the question, it sounds to me like a REST web service would be appropriate, provided of course that you must use SOAP web services for some reason.
There are some things that you commonly see with REST web services that may be useful in your case:
- By enclosing, for instance, a startRow and a rowCount parameter in the HTTP request string, you can limit the number of rows you retrieve.
For instance, the user is looking at some table, in which each page is 100 rows. In this case it makes no sense to retrieve all rows, but instead use the suggestion above to retrieve one page worth of rows at a time.
- By enclosing, for instance, a depth parameter in the HTTP request string, you can limit the depth of the data returned.
This is like lazy loading with Hibernate, but instead of it happening automatically, you (on the client side) have to be aware of this.
For instance, you want to retrieve a list of Customers. Each Customer may have many orders, but when looking at the customer list, you are not interested in the orders, so when retrieving the customers you set the depth to 1. This way, the orders associated to each customer are not retrieved, but only represented by an URI from which the order in question can be retrieved (in the customer representation, which can be XML, JSON etc).

Of course, the above suggestions can be implemented in a SOAP web service as well - not in the HTTP request string, though.

The drawback with REST web services the lack standardized technologies like MTOM.
It should not be too hard to implement compression of data using a servlet filter etc.
Best wishes!
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: How to do Bulk data transfer using Web Service