Once in a while (happened 3 times in 1 year) our application uses a wrong Date(time).
Overview: Our application uses Spring 3, Hibernate, MySQL 5 and runs in a Tomcat 6 instance. We have configured the MySQL Instance to store Datetime values in UTC (see JDBC Driver config below). Our application uses a TimeService class to generate dates if needed.
From what I can tell, there's no place were we set the Timezone manually - the TimeService uses the defaultTimezone (see Code below). Affected Fields on entites are defined as Hibernate TIMESTAMP.
The production system is a 64-Bit Linux for System z with Suse Linux Enterprise Server 11.2 using IBM Java Development Kit (JDK) SE 6 SR12 (64-Bit Version)
The production system uses NTP to synchronize the Hosttime (GMT+1 with Daylight Savings).
The actual bug manifests in two ways:
1) when persisting a record, we set a Date field (datetime in the DB) using TimeService.newDate(). This should give local time and persist the UTC date (local -1 hour). When the bug is triggered, the persisted date looks like the local date (missing the -1 hour)
2) for a business process, we calculate intervalls. I.e. for a given day there may be 24 hour-intervall. 23/25 respectively when we hit winter/summertime. When the bug is triggered, the calculation produces 23 Intervals instead of the expected 24.
It is assumed that the bug is only triggered under heavy load. But 1) the system is reguarly under heavy load (where heavy means that approx. 20 Threads are processing 100k+ transactions) and 2) trying to reproduce the load did not trigger the bug on our local testing/dev environments so far.
The code genrally uses java.util.Date and only the TimeService class utilizes Joda's DateTimeFormatter. I was not able to reproduce the error on the development/testing environment and I'm out of ideas what may cause such an error and how to trigger that.
Do you have any ideas what I could do to further analyze this one?
Feel free to ask any questions (except "why did you do this'n'that). It's a rather old application and will be replaced by a new version soon. But as long as we don't know what causes this bug we can't be sure that this wont happen with the new version (moved completely from java.util.date to joda).
ParameterString for the JDBC Driver
Portions from the TimeService
1. Make it run - 2. Make it run correctly - 3. Make it pretty OR fast/small - 4. ??? - 5. Profit
have you considered it could be concurrency issue? dates classes are not thread safe if I recollect correctly.
All desirable things in life are either illegal, banned, expensive or married to someone else !!!
Joined: May 05, 2010
have you considered it could be concurrency issue?
Yes. But as I could not reproduce the problem, I've tried to wrap my head around the underlying code. The Date used to populated the datetime field is not shared among threads. Or are the concurrency issues regarding date even more invasive?
I'll reiterate over the code to eliminate that.
Joined: Feb 10, 2001
The Date used to populated the datetime field is not shared among threads. Or are the concurrency issues regarding date even more invasive?
If you are 100% sure then it may be a different issue but this code looks vulnerable for concurrency if used in multi threaded environment unless timeOffset is constant.
may be it worth to write test case to create few threads and try to reproduce it.
I had this issue few years ago and it was due to concurrency in multi threaded batch processing
Another possibility: DST/Summertime. Most of the US just moved to DST, so that was the first thing that jumped to mind when I saw "date is one hour off." I don't know what the schedule is where you are, but if these problems are occurring around the time change, maybe your DB server's clock and Java app's clock are a bit out of sync, so one jumps by an hour before the other? Or it could be DST in combination with a threading issue, which would make it extra fun to try to reproduce.
Joined: Feb 10, 2001
also, is there any pattern for the failed date/time? you mentioned it fails 3 times a year, what is that time?
Joined: May 05, 2010
Praful Thakare wrote:also, is there any pattern for the failed date/time? you mentioned it fails 3 times a year, what is that time?
As far as I can tell, there's no pattern. "3 times a year" should read "it failed 3 times in the last year".