NX: Shutting Down the Server

[Javini Javono]

Hi,
My directions do not require me to lock down the complete database in response
to the client sending in a -1 argument to the lock() method. However, my server
needs to shut down elegantly without damaging the file or losing information.
When my server is told to start shutting down, when a client then calls the lock
method, the server raises an exception: "Server shutting down, no lock requests
are being accepted at this time."
Then, my original idea was simply to monitor the number of threads attempting
to complete from the LockManager. However, I noticed that the number of threads
in the LockManager can be zero, and the server still isn't ready to shut
down!
So, what I will do is the following to safely shut down the server:
1. When it is time to shut down, the server will no longer accept lock
requests, and raise an exception as discussed above.
2. Furthermore, the server will monitor disk writes; so, for instance, if
no create, delete, or update method has been called in the last two minutes,
the server will conclude that there are no outstanding writes.
When both conditions above are true:
1. No threads in the lock manager,
2. No disk writes in the last two minutes,
only then will the server assume that it is safe to shut down.
Here are my test runs. I'm again using my version of George's Super Software
Exerciser to drive the tests, but in this particular run, updates, creates,
and deletes are all occuring. Note how the number of waiting threads drops
to zero for a quite a few minutes in the middle of the run! Only to pick up
again later. I suspect that this behavior is related to the Mac OS X operating
system.
Aside: in this test run, I've doubled the usual size of the database file to
contain 62 records. Also, there were 620 instantiations of the tester class
wherein each instantiation represents one Data instance (i.e., it contains one
Data instance).

Program Starts.
StopWatch[LongTest]: in progress (min) = 1.6666666666666667E-5 : Expected Duration (min) = 15.0
   guard.size() = 0
   getNumberOfWaitingThreads() = 0
StopWatch[LongTest]: in progress (min) = 0.5000166666666667 : Expected Duration (min) = 15.0
Guardian.printStatistics():
   guard.size() = 36
   getNumberOfWaitingThreads() = 103
StopWatch[LongTest]: in progress (min) = 1.0000833333333334 : Expected Duration (min) = 15.0
Guardian.printStatistics():
   guard.size() = 61
   getNumberOfWaitingThreads() = 316
Guardian.printStatistics():
   guard.size() = 61
   getNumberOfWaitingThreads() = 420
StopWatch[LongTest]: in progress (min) = 1.5000833333333334 : Expected Duration (min) = 15.0
Guardian.printStatistics():
   guard.size() = 54
   getNumberOfWaitingThreads() = 415
StopWatch[LongTest]: in progress (min) = 2.0000833333333334 : Expected Duration (min) = 15.0
Guardian.printStatistics():
   guard.size() = 0
   getNumberOfWaitingThreads() = 0
Both guard.size() and getNumberOfWaitingThreads() remain at zero or one for
about 13.5 minutes during this interval, while the computer is very, very busy
doing writes to the file.  It is usually unusual for 415 threads to evaporate
from the LockManager in 30 seconds;  I suspect that the operating system queues
them up, and then carries out the writes for the next 13.5 minutes while not
putting any more threads into the LockManager while it is busy writing.
StopWatch[LongTest]: in progress (min) = 13.0002 : Expected Duration (min) = 15.0
Guardian.printStatistics():
   guard.size() = 0
   getNumberOfWaitingThreads() = 0
StopWatch[LongTest]: in progress (min) = 13.500483333333333 : Expected Duration (min) = 15.0
Guardian.printStatistics():
   guard.size() = 1
   getNumberOfWaitingThreads() = 0
StopWatch[LongTest]: in progress (min) = 14.000483333333333 : Expected Duration (min) = 15.0
Guardian.printStatistics():
   guard.size() = 0
   getNumberOfWaitingThreads() = 0
StopWatch[LongTest]: in progress (min) = 15.000483333333333 : Expected Duration (min) = 15.0
Guardian.printStatistics():
   guard.size() = 0
   getNumberOfWaitingThreads() = 0
StopWatch[LongTest]: in progress (min) = 15.5005 : Expected Duration (min) = 15.0
Guardian.printStatistics():
   guard.size() = 26
   getNumberOfWaitingThreads() = 9
StopWatch[LongTest]: in progress (min) = 16.0005 : Expected Duration (min) = 15.0
Guardian.printStatistics():
   guard.size() = 0
   getNumberOfWaitingThreads() = 0
Program Ends.

Thanks,
Javini Javono

[Don Wood]

Hi Javini,
I am just getting started on my project so you are way ahead of me.

1. When it is time to shut down, the server will no longer accept lock
requests, and raise an exception as discussed above.

I like this approach and expect to do this myself when I get that far.

2. Furthermore, the server will monitor disk writes; so, for instance, if
no create, delete, or update method has been called in the last two minutes,
the server will conclude that there are no outstanding writes.

I generally don't like timed waits as they tend to delay things unneccessarily. It also seems like no matter how long you wait, there is no guarantee with this approach.
It seems to me that you could simply lock every record in the database and be done much quicker than waiting for two minutes of complete idle time. Now on a very large database this would not scale at all but if we had a very large database for this assignment, a lot of other things would need to change too. So it seems to me that locking every record is reasonable and faster.

[Javini Javono]

Hi,
Thanks again for reading and responding to my posting.
You have a good point about the waiting. What I think that I
will do is have a preferences item specify how many minutes
maximum the server will wait before shutting down. Also,
a preferences item saying how much time must expire before
it is assumed that no more writes will occur.
Example:
Maximum Wait Time: 15 minutes
No More Writes Can Occur Time Assumed: 2 minutes

So, the shutdown sequence will look something like this.
First, set a flag so that no new clients can obtain any locks.
If the lock manager says no records are locked, then shutdown
is possible. If the lock manager says that at least one record
is locked, then the client holding this lock has the user-specified
time to complete whatever operation was being conducted on
that record.
Finally, as it is possible, as shown on my computer, for the lock
manager to be empty, and yet for 15 minutes of writes to continue,
while I am waiting for the maximum user-specified wait time,
I will periodically check for any writes having occurred.
If the lock manager is empty, and no writes have occurred within
so many minutes, let's say user-specified at 2 minutes, then it is
safe to shutdown prior to the maximum allowable wait time of
15 minutes.
The wait time, if found undesirable by the user, can be set to very
low values.
NO GUARANTEES
--------------
Your point about there being no guarantees is very good. I may
also have a status panel on the server, where it shows the number
of currently held locks, and the last time a write occurred.
The system administrator can use his own judgement as to when
it is safe to shut down the server. This button may be called:
Override and Shut Down Right Now. Because the system administrator
can see what the situation is, there is no need to wait for any number
of minutes.
Thanks,
Javini Javono
[ March 27, 2004: Message edited by: Javini Javono ]