Win a copy of Re-engineering Legacy Software this week in the Refactoring forum
or Docker in Action in the Agile forum!
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic

Heisenbug: Date is one hour off

 
Stephan Mueller
Ranch Hand
Posts: 50
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Hi.

Once in a while (happened 3 times in 1 year) our application uses a wrong Date(time).
Overview: Our application uses Spring 3, Hibernate, MySQL 5 and runs in a Tomcat 6 instance. We have configured the MySQL Instance to store Datetime values in UTC (see JDBC Driver config below). Our application uses a TimeService class to generate dates if needed.
From what I can tell, there's no place were we set the Timezone manually - the TimeService uses the defaultTimezone (see Code below). Affected Fields on entites are defined as Hibernate TIMESTAMP.
The production system is a 64-Bit Linux for System z with Suse Linux Enterprise Server 11.2 using IBM Java Development Kit (JDK) SE 6 SR12 (64-Bit Version)

The production system uses NTP to synchronize the Hosttime (GMT+1 with Daylight Savings).

The actual bug manifests in two ways:
1) when persisting a record, we set a Date field (datetime in the DB) using TimeService.newDate(). This should give local time and persist the UTC date (local -1 hour). When the bug is triggered, the persisted date looks like the local date (missing the -1 hour)
2) for a business process, we calculate intervalls. I.e. for a given day there may be 24 hour-intervall. 23/25 respectively when we hit winter/summertime. When the bug is triggered, the calculation produces 23 Intervals instead of the expected 24.

It is assumed that the bug is only triggered under heavy load. But 1) the system is reguarly under heavy load (where heavy means that approx. 20 Threads are processing 100k+ transactions) and 2) trying to reproduce the load did not trigger the bug on our local testing/dev environments so far.

The code genrally uses java.util.Date and only the TimeService class utilizes Joda's DateTimeFormatter. I was not able to reproduce the error on the development/testing environment and I'm out of ideas what may cause such an error and how to trigger that.
Do you have any ideas what I could do to further analyze this one?
Feel free to ask any questions (except "why did you do this'n'that). It's a rather old application and will be replaced by a new version soon. But as long as we don't know what causes this bug we can't be sure that this wont happen with the new version (moved completely from java.util.date to joda).


ParameterString for the JDBC Driver


Portions from the TimeService
 
Praful Thakare
Ranch Hand
Posts: 642
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
have you considered it could be concurrency issue? dates classes are not thread safe if I recollect correctly.

-P
 
Stephan Mueller
Ranch Hand
Posts: 50
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
have you considered it could be concurrency issue?

Yes. But as I could not reproduce the problem, I've tried to wrap my head around the underlying code. The Date used to populated the datetime field is not shared among threads. Or are the concurrency issues regarding date even more invasive?
I'll reiterate over the code to eliminate that.
 
Praful Thakare
Ranch Hand
Posts: 642
  • Likes 2
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
The Date used to populated the datetime field is not shared among threads. Or are the concurrency issues regarding date even more invasive?


If you are 100% sure then it may be a different issue but this code looks vulnerable for concurrency if used in multi threaded environment unless timeOffset is constant.



may be it worth to write test case to create few threads and try to reproduce it.

I had this issue few years ago and it was due to concurrency in multi threaded batch processing
 
Junilu Lacar
Bartender
Pie
Posts: 7306
45
Android Eclipse IDE IntelliJ IDE Java Linux Mac Scala Spring Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Praful Thakare wrote:...this code looks vulnerable for concurrency if used in multi threaded environment unless timeOffset is constant.

I concur; good catch.
 
Praful Thakare
Ranch Hand
Posts: 642
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Thanks Juniul, do i get a cow for that ?
 
Junilu Lacar
Bartender
Pie
Posts: 7306
45
Android Eclipse IDE IntelliJ IDE Java Linux Mac Scala Spring Ubuntu
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Praful Thakare wrote:Thanks Juniul, do i get a cow for that ?
You could have but then you misspelled my name...
 
Praful Thakare
Ranch Hand
Posts: 642
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
sorry for that..next time i will be careful Junilu
 
Jeff Verdegan
Bartender
Posts: 6109
6
Android IntelliJ IDE Java
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Another possibility: DST/Summertime. Most of the US just moved to DST, so that was the first thing that jumped to mind when I saw "date is one hour off." I don't know what the schedule is where you are, but if these problems are occurring around the time change, maybe your DB server's clock and Java app's clock are a bit out of sync, so one jumps by an hour before the other? Or it could be DST in combination with a threading issue, which would make it extra fun to try to reproduce.
 
Praful Thakare
Ranch Hand
Posts: 642
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
also, is there any pattern for the failed date/time? you mentioned it fails 3 times a year, what is that time?
 
Stephan Mueller
Ranch Hand
Posts: 50
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Praful Thakare wrote:also, is there any pattern for the failed date/time? you mentioned it fails 3 times a year, what is that time?

As far as I can tell, there's no pattern. "3 times a year" should read "it failed 3 times in the last year".

Regarding the timeOffset: I'll look into that.
 
  • Post Reply
  • Bookmark Topic Watch Topic
  • New Topic