Update #4: Due to Critical Weather, the full outage to PDA OPS scheduled to begin tomorrow, 6/29 at 1100Z has been delayed one hour to 1200z. If Critical Weather is further extended beyond 1200Z, a new schedule will be announced later tomorrow morning. An additional change has been planned due to problems pushing to one critical user. PDA/NDE will fail back to NSOF OPS for J1, SNPP, and GCOM the afternoon of Tuesday 6/29 (tomorrow) at a time expected to be NLT 2000Z. During the fail back to NSOF OPS, all dataflow from the CBU backup site will stop.

Update #3: The planned failover to CBU activity starting today, Monday June 28 at 1400Z, is expected begin as planned. Distribution from CBU will be turned ON at ~1400Z and distribution from NSOF will be turned OFF ~1430Z. The Full outage at NSOF OPS of ~6-8 hours is scheduled to begin tomorrow June 29 at 1100Z with the PDA/NDE OPS environment shutdown.

Update #2: The GOES team has determined that they will shut off distribution to PDA during this event. All GOES-16 and 17 imagery and products will not be recoverable for the ~6-8 hour outage period on June 29 when PDA comes back up.  We can now finetune the critical times for the partial failover affecting J1, SNPP and GCOM data:

Failover to CBU on June 28:

Turn on distribution at CBU 1400Z

Turn off distribution at NSOF 1430Z

 

Failback to NSOF June 30:

Turn on distribution at NSOF 1400Z

Turn off distribution at CBU 1430Z       

 

Update #1: Reminder about the event scheduled later this month. Please note updated times for June 29 outage. See details at the end of the Details/Specifics of Change section about ancillary data availability at CBU.  

Topic: Planned PDA Partial Failover to CBU and Full Outage of PDA OPS at NSOF 

Date/Time Issued:  June 28, 2021 1900Z

Product(s) or Data Impacted:  All products distributed from PDA. Some products will be available at CBU and some will be unavailable for part of the time maintenance window.  

Date/Time of the Initial Impact: Jun 28, 2021 1400Z

Date/Time of Expected End:  Jun 30, 2021 1400Z

Length of the Outage: Failover to CBU ~48 hours; Full outage at NSOF OPS ~6-8 hours -- shutdown to start at 1100Z on Jun 29.  

Details/Specifics of Change:  

On Jun 29, a significant storage maintenance activity will take place during business hours that will require a period of about 6-8 hours of complete PDA downtime.  Delivery of GOES-16/17, Metop, POES and other data provided from PDA will cease distribution during that affected time.  When the PDA system is returned to service, it is  anticipated that users will be able to receive and distribute products that were ingested/created during the outage time.  However, there will be a significant backlog on PDA, and it may take several hours for the PDA OPS at NSOF to receive and distribute all the products produced during the outage period.  During the outage PDA period, latency will be impacted as the backlog of products are processed and distributed.

In order to minimize the outage impact, NESDIS  plans to partially activate the Consolidated Backup (CBU) site at Fairmont, WV where only distribution of data and products from S-NPP, NOAA-20 and GCOM-W1 are available.  This failover is expected to begin with the first orbit received after 1400Z on Jun 28 (the day before the outage in OPS) and end with the next orbit received after 1400Z on Jun 30 -- the day before and after the full PDA OPS outage. The exact times of the failover will be announced closer to the event.   It is noted that while in this data failover configuration, S-NPP, NOAA-20 and GCOM-W1 data and products will ONLY be at CBU for that period of about 48 hours and then will be returned to the nominal PDA OPS system at NSOF 

At this time duplicate data will not be distributed between PDA OPS and CBU; therefore, pull users will be safe to pull data from both sites starting at the time of partial failover and until about 1.5 hours after failback end time while the final NDE products are created at CBU.  ****See update below about ancillary data that will be at both sites.****

PDA staff will be available to help troubleshoot users’ issues with failover from 0930 EDT / 1330 UTC until 1500 EDT / 1900 UTC on Mon Jun 28 and from 1030 EDT / 1430 UTC until 1400 EDT / 1800 UTC on Wed Jun 30 for failback. Staff will be totally focused on the maintenance on Tue Jun 29, so it will be very important for users to identify any problems with CBU access on Mon Jun 28. Email PDA_DHS@noaa.gov and someone will provide the google meet or phone number.  

 

If there is critical weather (CW) on Jun 28, the event will be delayed one day to begin Jun 29. If CW is declared before start of the event to go into Jun 29 or beyond, the event will be postponed until the week of Jul 12. This is due to expected reduced staffing levels for all affected groups the holiday week.  

 

****Important update about ancillary data availability at CBU: Many of our external providers provide ancillary data (not associated with any satellite) to PDA CBU and during a partial failover that data will likely be available. It will be available for the entire ~48 hour period of the partial failover, duplicating much of the data available at PDA OPS. Users should decide if they want this data, despite having many duplicates. Pull users who are subscribed to ancillary data can determine whether to adjust their pull scripts to not pull the data or turn off “Active at Backup” for subs they do not want at CBU.  Push users need to turn off “Active at Backup” for subs they do not want at CBU to prevent it from being pushed.  The data likely to be available at CBU (in addition to all J1, SNPP and GCOM data and products) are:

Tropical files - including NHC forecast files - *latest*, NHC and JWTC files - ATCF*

BUOYSST*, SST_BUOYSHIP, SST_in-situ_iQuam_STAR, SHIPS*diag

Model data - GFS*, CDAS*, GDAS*, NAM*, RAP*, NPOES*, QMORPH*, RTG*, NAAPS*, NAVGEM* 

DCS_DATA*

GPS_PWV

Leap_Second_Updates

Snow maps - NESDIS_IMS*, NPR_IMS*, NSA_SNOW_COVER, NISE_V5

OISST_EMC_WEEKLY

OMB_Seaice*

Ostia_Analysis*

Reynolds_SST_uncompressed_NetCDF_HTTPS

SWP*

USNO*

Weather_Celestrak, Weather_TLE

 

Contact Information for Further  Information: ESPC Operations at ESPCOperations@noaa.gov and 301-817-3880.    

 

Web Site(s) for applicable information: N/A     


This message was sent by ESPC.Notification@noaa.gov. You have been sent this and other notifications because you have opted in to receive
 it. If for any reason, you wish to unsubscribe, please contact ESPC Help Desk at ESPCOperations@noaa.gov or (301) 817-3880. Please note:
 it may take up to two business days to process your unsubscribe request