Topic:
NOAA-12 Anomaly Notification
Date/Time(UTC) Message Issued:
28 September 2002 2006 UTC
Satellite(s) Involved:
NOAA-12
Instrument(s) Involved:
All
Product(s) Involved/Affected:
All data users
Date/Time(UTC) of Initial Implementation:
28 September 2002 0230 UTC
Details/Specifics of Change:
On September 28, the Satellite Operations Control Center issued the
following notice concerning a NOAA-12 anomaly and safe recovery:
"N12 experienced an anomaly which started on Sat, 28 Sept at approximately
02:30Z and was detected on its only scheduled pass that same day at 09:50Z.
Some of the major symptoms of the anomaly were as follows:
ADACS: an autonomous switch to YGC attitude control mode, excessive reaction
wheel speeds, a Yaw error of over 30 degrees and roll and pitch momentums of
20 in.-lb. sec. (close to half RWA saturation). However, the IMU data was
nominal, so the yaw gyro was not suspect.
C&CS: Numerous CPU errors including CARP and Level 16 errors.
EPS: Batteries discharging (spacecraft is full sun)
Payload: Numerous HIRS temperature limit violations (HIRS is
non-operational on N12 but used for power management).
Subsequent analysis by the engineering team determined that the problem
originated from an overflow in the ephemeris clock on the control CPU which
“expired” at 02:30Z on 28 Sept causing the spacecraft to become
“disoriented”. (Note: the backup CPU ephemeris clock was valid and up to
date.) The reason the clock expired was that the control side ephemeris
load operation on the previous N12 pass (approximately 40 hours earlier on
Thursday, 26 Sept at 20:05Z) inadvertently did not result in the clock
update command executing as it should have. This meant that the new
ephemeris load was never activated and at 02:30 on 28 Sept, the old load
expired causing the clock overflow and explaining the level 16 errors.
To resolve the anomaly, engineers scheduled several call up passes. On the
first call up pass, the ephemeris load from 26 Sept was reloaded and the
clock update command verified. After the ephemeris restart the yaw updates
and momentums returned to nominal values. On the next pass, most of the
error flags were cleared out, a yaw update was observed and validated and
the spacecraft was verified to be operating within nominal limits with
battery voltages returning to normal and attitude control operating within
prescribed limits (although still in YGC mode with the skew gyro in the yaw
loop in place of the X gyro). Engineers determined that the best course of
action was to remain in YGC mode until next week.
Analysis has determined that this anomaly had the potential to become quite
serious (including spacecraft tumbling, and power safing) if it had been
left unattended for another 8-12 hours. Since N12 can sometimes go 36-48
hours or more between scheduled passes, our ability to prevent a cascading
anomaly in this situation can be partly attributed to pure luck.
Engineering analysis is continuing on this incident and further reports will
be forthcoming. Several lessons learned have also been derived and
corrective action will be implemented as soon as possible."
Contact Person(s) Name/Email/Phone Number for Questions:
Emily D. Harrod/Emily.Harrod@noaa.gov/301-457-5247 ext 117
Web site(s) that contain other relevant information (where applicable):
http://www.oso.noaa.gov/daily-news/index.htm - SOCC morning reports
http://www.osdpd.noaa.gov/PSB/PPP/PPP.html - History of level 1B
notices
http://www.ssd.noaa.gov/PS/SATS/messages.html - All user notices