aspose file tools*
The moose likes Threads and Synchronization and the fly likes Opening a URLConnection in several threads Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Java 8 in Action this week in the Java 8 forum!
JavaRanch » Java Forums » Java » Threads and Synchronization
Bookmark "Opening a URLConnection in several threads" Watch "Opening a URLConnection in several threads" New topic
Author

Opening a URLConnection in several threads

Tom Dallas
Greenhorn

Joined: Jul 23, 2008
Posts: 14
The Model is from JENA, but thats not the problem I believe. When I let this code run in a single thread, it works, when I'm letting the folowing code run in several threads, it just stops executing. Is this because of some connection problem when I open several connections to a URL? Even if it are different URLs? I'm a bit lost on this one :-/ ... don't have much experience on Streams, Connections and Threads.

Ajay Saxena
Ranch Hand

Joined: Nov 13, 2006
Posts: 154
Can you show us (the code) how you attempt to run the code in a single thread and in multiple threads?
Tom Dallas
Greenhorn

Joined: Jul 23, 2008
Posts: 14
Ajay Saxena wrote:Can you show us (the code) how you attempt to run the code in a single thread and in multiple threads?


This is in the callbackobject and works but takes about 20 seconds because of the non-parallel HTTP Requests without Threading... which is not acceptable:



This is how I try to get it working with threads ... which doesn't work:



Any ideas? I'm running this under Eclipse on Windows XP. I try both different methods in a Test, the non-threaded works and the threaded breaks because the threads all die when they try to get the input streams. I thought maybe I can only open one URLConnection at a time and tried out the Apache httpclient-implementation, but the threads die all the same, therefore I think it might be something about the streams ...
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3967
    
  17

You could possibly be hitting the server real hard. These kinds of multi-threaded connections to a server can bring a server down, working like a DOS attack, and can be blocked by a savvy network admin. Am I sure this is what is happening? No, but as a responsible programmer you should try to prevent such strain on the resource you are using. If you must work in multi-threaded requests I would talk with the network admin for the site you are requesting and check their logs to see why they are blocking your application and to ask permission to get do this.

Or, you can go about things a different way. You are making requests to the server, and it is the processing of the requests that takes some time. Perhaps if you make a single thread that makes new connections to the server, then passes those connections onto a ThreadPool for processing it would fast enough. Example:

Also, the way you handle your exception is incorrect. For example, when you try to make a URL and get a MalformedURLException, you 'handle' it by printing a stack trace. Does it then make sense to try to make a URLConnection to the malformed URL? No, the rest of the code dependent on the URL and the URLConnection should not be executed, but in your code they are, which could lead to a cascade of improper states, and more exceptions. What you should do if you have exceptions which 1) can't be fixed, and 2) has a bunch of code that depends on it for success is wrap the entire block of dependent code into a single try { } block, so when an exception occurs the entire sequence is exited. Example:


Each successive step will definitely fail if the previous one did. So why make them all fail in order? If the first fails, the others shouldn't execute:


Finally, in your multi-threaded example you have a properties object that you are writing to. Make sure that object is synchronized or you could get wierd behavior.


Steve
Tom Dallas
Greenhorn

Joined: Jul 23, 2008
Posts: 14
Than you for your help!

I don't think I hit their server too hard, as its only about seven get-requests that I try in my testsuite. The access to the shared variable is already synchonized. Do you have a hint how to implement connection pooling easily?

I'll try your suggestions and see if they help. If I get the solution, I'll post it here!
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3967
    
  17

You don't need to pool HTTP connections. Java keeps connections alive for you and will re-use them when possible.

I was looking at your code, then ran some tests on my end trying to duplicate your problem. Before I post anything I want you to clarify what you mean.

At one point you say: " when I'm letting the folowing code run in several threads, it just stops executing" and later on you say "the threaded breaks because the threads all die when they try to get the input streams". Can you tell me exactly the behavior you get when the threads die?
Tom Dallas
Greenhorn

Joined: Jul 23, 2008
Posts: 14
The threads were executed but just stopped being executed when they opened the connection and opened the stream of data to the URL. I think I got it working now, but I tried the Apache HTTPClient again. This is the code:



I also made the callback-mechanism a bit cleaner. The callback object was initiated by a servlet in the first place, and that might have finished executing after some time. In that servlet, I now use a


to let it wait for the callback object to collect its data. When the callback object is ready, it notifies the servlet:



This works for me. Thanks Steve, you got me on the right track. I think the problem might have been the servlet, which might just have stopped executing, but I'm not sure about this. It was originally called by an AJAX request, instanciating the callback object, that spawned the threads to get the data from some URLs and extracted data from them, passing them back to the callback object and then back to the servlet to generate a response. Maybe it was something I did to the code out of desparation and that I got wrong at first, but I believe its just the clearer callback with wait() and notify() that got it working :-) ... Thanks!
Steve Luke
Bartender

Joined: Jan 28, 2003
Posts: 3967
    
  17

With Servlets and Ajax and what not involved, things get a lot more complicated, so any tests I had done don't really apply to your situation, I guess.

Still, here is the code I ran to test for posterity's sake, it seemed to work smoothly no matter the threading condition. Note that I don't really know RDF, JENA, or the freebase interface so when it came time to check concurrent access I didn't use a connection of nodes to iterate over (didn't know where the nodes came from) I basically just collected a bunch of links to iterate over. I guess it is the same in the end (except the cases where the URLs coming from the node may be malformed...)





Tom Dallas
Greenhorn

Joined: Jul 23, 2008
Posts: 14
Whoa! I was already glad that you brought me on the right track (my code runs without problems now) but now your code also shows me some nice stuff from java.util that I had no clue of. Thanks man!
 
I agree. Here's the link: http://aspose.com/file-tools
 
subject: Opening a URLConnection in several threads
 
Similar Threads
replaceregexp workaround/bug
Reading hyperlink in excel through Java
problem reading output of a subprocess
No text displayed, JTextArea.append();?
Reading Windows directory path from .properties file.