ENVISAT Prime Memory Faults - Some Lessons Learned

Abstract

Mitigating the effects of Single Event Upsets (SEU) is critical to ensuring spacecraft safety in low earth orbits. The memory of the Central Communications Unit on Envisat, launched in March 2002 for a nominal life of 5 years, is protected by EDAC ensuring thatany single bit flip identified during memory read instructions is corrected prior to usage by the processing unit. The memory is divided into two blocks (prime and redundant), each consisting of three pages. A memory scrubbing task runs in the background to correct the contents in the memory, and reports periodically the number of corrected SEUs. In November 2006, the memory scrubbing task began reporting SEUs at a very high rate which lead to a diagnosis of permanently faulty memory areas. Alas the limitations in the data available in the scrubbing report prevented assessment of the full extent of the problem. Fortunately, the internal prime memory block redundancy could be activated, masking the problem, and provided time for the development of a diagnostic routine. A similar behavior was then observed in another prime memory page in December 2007 while no further internal prime memory redundancy was available. The new diagnostic routine was used to obtain a detailed assessment of the affected memory areas. Furthermore, the memory fault statistics delivered by the scrubbing were no longer usable, the report being flooded by the permanent errors. The permanently corrupted memory areas also increased the risk of double event upset occurring which would have led to a mission interruption. A permanent swapping of the prime and the redundant memory blocks mitigated this risk, but at the cost of a potential loss in memory redundancy in case of an SEU occurring in the corrupted area. An extension to the Envisat operational lifetime was under discussion at the time of the second memory page anomaly and it was clear that failure to recover some memory redundancy could jeopardize approval. This was achieved by 1/modifying the Failure Detection Isolation and Recovery (FDIR) behavior in case of a double bit flip detection in redundant memory and, 2/improving the scrubbing task to make it tolerant to permanent errors. This paper presents the lessons learned from the Envisat prime memory anomalies and provides recommendations to the design of future LEO spacecraft data handling subsystems.