Coral enable/disable difficulties over the weekend ....

John Shott shott at snf.stanford.edu
Mon Feb 25 14:50:25 PST 2002


SNF Lab Members:

Several of you have noticed varying levels of enable/disable "wierdness" that
occurred over the weekend.  Specifically, what Coral thought was enabled was
not always consistent with what the "ON" light said.  Also, it turns out that
what Coral was displaying was not consistent with the underlying data in the
database.

Without boring you with details, while the equipment server appeared to be
functional during the weekend, an "under to covers" portion that updates the
database failed silently.  In over 2 years of operation, we have never seen
this behavior and, at this point, don't know the cause.  We are, however,
adding improved logging to at least detect and warn us of this failure and
hope to get to the bottom of this quickly.

On Sunday morning, I rebooted the machine and restarted the servers in the
process of adding some operating system patches.  Restarting the servers
causes them to reload their memory from the database ... which caused these
inconsistencies to become visible.  (It actually appears as if equipment
database updates had quit ... silently ... on Friday.)

While Coral does not actually disable any hardware in these circumstances ...
some people who actually had equipment enabled were adversely affected because
others found that equipment was "ON" even though Coral showed it not to be
enabled ... and tried to correct the problem by "resynchronizing" the
equipment with Coral by disabling it.

While we have not had frequent problems of this type, remember to let us know
when you see an inconsistency between "ON" lights and what Coral thinks is
enabled ....

We apologize for diffuculties that this has caused and are working hard to
figure out where the bug occurs.

Thanks for your continued support,

John



More information about the labmembers mailing list