Introduction

This postmortem aims to provide a transparent account of the issues encountered with MET/CAL version 11.0.2 and the steps we have taken to prevent such incidents in the future.

Incident Summary

MET/CAL version 11.0.2 was intended to be a minor release, targeting specific known issues primarily related to instrument communications. However, a process failure led to unintended code changes being included in this release.

What Went Wrong

During development, additional work was undertaken to replace insecure functions with secure counterparts to prevent potential memory corruption issues. These changes were not part of the original scope of version 11.0.2 and were not extensively tested or reviewed due to process failures.

Impact

The inclusion of these untested changes led to unforeseen issues affecting the stability and performance of the software, impacting our users' experience.

Corrective Actions

In developing version 11.0.3, we took the following steps to address the issues:

1. Code Review: Re-evaluated all areas where function replacements were made, ensuring correct implementation.
2. Additional Protections: Developed wrapper functions to add extra layers of protection against failures.
3. Process Improvements: Implemented changes to ensure all code alterations undergo adequate review and testing before release.
4. Additional Hardware Testing: Performed extensive regression testing with a variety of FSCs to confirm proper operation in multiple configurations.

Moving Forward

We apologize for any inconvenience caused by this incident. We are committed to maintaining the highest quality standards and have taken significant steps to improve our processes to prevent similar issues in the future.

Thank you for your understanding and continued support.