Follow up of this weekend's cooling water incident

Mary Tang mtang at
Tue Aug 21 09:41:18 PDT 2012

Dear labmembers --

Thanks for your patience this weekend as the campus struggled with loss 
of cooling water.  Several people have asked what happened. This is what 
we understand.

Just before 3 pm Friday, a 60Kv electrical line was damaged during 
construction work, causing the Cardinal Cogeneration Plant to shut down. 
Those of you in the lab at that time may have experienced the associated 
power glitch which caused a number of tools to shutdown or lockup.  The 
Cogen plant is the large green "steamy" structure across the street from 
us.  It supplies power, cooling water and steam to the entire Stanford 
campus.  Cogen cooling water is used to absorb heat from the Allen 
building process cooling water (PCW) recirculating system.  SNF uses PCW 
to cool pumps, chillers, and other heat-generating systems.  Cogen 
cooling water is also used for controlling building temperature and fab 
humidity and temperature. Without cooling water, our tools shut off and 
the building/lab heat up.  Friday afternoon, campus wide, buildings got 
hot and computer system shut down.

Emergency backup systems kicked in.  In the case of SNF, city water was 
automatically introduced to keep the PCW temperatures down. However, 
with the whole campus also using city water as backup, the pressure was 
not sufficient to keep our tools running.  By Friday evening, SNF staff 
were able to safety shutdown the major heat generating tools and thus 
avoid damage to tools or the PCW system. Stanford emergency crews worked 
through the weekend and by around 9 pm Sunday evening were able to 
restore cooling water campus wide. Although a couple of SNF tools are 
still suffering from the unexpected shutdown, quick action and long 
hours by SNF and FacOps staff prevented serious damage.  Special thanks 
to Maurice, Mahnaz, Ed, Ray, and Elmer for directing traffic and 
shutting tools down; Jim H for cutting his vacation short to come in, 
help and advise; Ted for calling in instructions and advice while home 
sick; and Tony in FacOps for keeping staff informed and putting in long 
hours to make sure the PCW was OK.

While recognizing the heroic efforts of many staff, there are a number 
of things we need to do to improve communications at the infrastructure 
level and with our labmember community.  These will be addressed in 
coming weeks.  If there are any questions about specific tools, please 
consult with Coral and the responsible staff member.  If there are 
questions about this incident, please don't hesitate to ask.

Thanks again,

Your SNF Staff

More information about the labmembers mailing list