How to keep web page updated and alive for long loops

[Farakh khan]

Hello,

I have written a piece of Java code to send bulk emails and its working fine but the problem is when the quantity for sending mails is e.g. 35000 then "page in browser" expires and throws error like "Internal server error" or "Page timeout".

How can I keep it live by counting and displaying number of emails e.g. 2300 mails sent out of 35000

Please advise

Best regards

[Unnar Björnsson]

That functionality must be implemented client side i.e with javascript. If you expect a task to take longer than 60-120 secs it should be sent in batches otherwise you might be timed out.
You can use Ajax to send 100 emails -> update a status message in the browser window -> send next batch of 100 emails.

[Farakh khan]

Thanks unnar for your reply

My code is written as servlet. The servlet performing all the tasks and then outputting its result whereas I am looking for to display real time results on the screen. For example there are many utilities of fileupload that shows completion bar. I mean realtime progress bar. I don't need progress bar in my case but similarly require to show realtime that how many emails sent and how many remains

[Steve Luke]

The reason you get the timeout is because you take too long to send any response to the client. When the servlet starts its work it should immediately send some response - and flush the output to the client gets data. Then, as the system progresses it sends more updates and flushes the content. What I used to send as the initial response was a full page with empty fields for the content/counter. Then as data was generated I would push out small chunks of JavaScript with the data in it which would be executed and update fields on the page.

The other approach is to have the task the Servlet runs pushed into another thread, and respond immediately with a page, then closes the connection. The page has JavaScript which periodically makes new connections to a (usually different) Servlet which checks on the progress started in the first.

It has been a long time since I worked on this stuff, though, so I am sure there are libraries to do this stuff now. I think AJAX would be used for the second approach.

[Unnar Björnsson]

The servlet runs code on server side, so you must be careful not to overload it with some large task. If you need to perform a large task that can be cut down into smaller ones you should do so.
I don't know where your list of emails comes from but lets say they are in a database on the server, you should write a function that gets only the first 100-1000 emails, processes them and returns a confirmation to the client.
Then the client can ask the server to process the next 100-1000 emails if there are any left to process.

1. n = 1
2. Client asks the server to process the n-th batch of emails
3. Server returns with confirmation of completion
4. n = n +1
5. repeat 2,3 and 4 until server return with 'all done' or something like that

The key here is that the server finishes it's task in each iteration before it times out.

[Farakh khan]

Unnar Björnsson wrote:
Then the client can ask the server to process the next 100-1000 emails if there are any left to process.

Then if there are 50K emails in database then client will fed up to request again after 100 emails and say in case 1000k?

[Unnar Björnsson]

Farakh khan wrote:

Unnar Björnsson wrote:
Then the client can ask the server to process the next 100-1000 emails if there are any left to process.

Then if there are 50K emails in database then client will fed up to request again after 100 emails and say in case 1000k?

Well if there are 1 million email addresses in the database and each batch is 100 emails then you'd have to iterate 10 thousand times sure but, depending on the server you could probably process more emails at a time.

I also hope you have considered the general guidlines regarding bulk emailing as to not get your domain blacklisted or your messages flagged as spam.

[Farakh khan]

Thanks again for your favorable reply

The problem is not spam or blacklisting but am trying to seek how can I display real time data in my web page?

[Bear Bibeault]

Farakh khan wrote:am trying to seek how can I display real time data in my web page?

You'll have to explain what you mean by that.

If you mean keeping track of, and displaying, how many emails are sent, they you will have to use Ajax. There'a no other way to keep the page loaded while the processing goes on. You'll alos need to send the emails in a worker thread that doesn't block responses.

[Farakh khan]

I explained above that I am trying to display real time data. For example when we attaching file in gmail then its showing progress bar that indicates the percentage of uploaded file parts. All I want is to show how many emails sent out of how many and when done it show a completion message.

I am googling and found http://www.pushlets.com/ near to my scenario. I am trying to understand them. Any more advise will be highly appreciated

[Bear Bibeault]

I already answered with an approach you can take.

Heavy, server-side frameworks like pushlets seem a bit out-of-date in the face of the modern web.

[Unnar Björnsson]

Yes, google, facebook and many other websites that display live data all use Ajax to do so.

[Farakh khan]

Unnar Björnsson wrote:Yes, google, facebook and many other websites that display live data all use Ajax to do so.

Can you please refer to any helpful link?

I am thankful to all for giving me your precious time

[Unnar Björnsson]

Farakh khan wrote:

Unnar Björnsson wrote:Yes, google, facebook and many other websites that display live data all use Ajax to do so.

Can you please refer to any helpful link?

I am thankful to all for giving me your precious time

This one

[Bear Bibeault]

The hard part isn't the Ajax -- jQuery makes that almost trivial.

The hard part is creating the asynchronous process on the server so that responses aren't blocked, and having that process communicate its status to subsequent requests asking for status.

It could be done using threads, but when I had to do something like this in a previous project, I actually had a mail-processing daemon running separately on the server to process the mail messages. The web app would add records to the database asking for the daemon to process a message, and the daemon would update the record with its progress. It was almost trivial for the web app to keep up-to-date with progress.

These days, I'd handle the communication with JavaScript and a RESTful API, but that didn't exist back then.

[Farakh khan]

You are right

By the way I just made some work around with help of This link. If something else in your mind please share with me referred link(s)

Thanks again