Archive: investigating an expired certificate in RancherOS

This is a record of a 2022 RancherOS incident. The Rancher interface became inaccessible, and the logs reported a certificate that had expired. RancherOS, Rancher, Kubernetes, and certificate recovery procedures have changed since this was written. Do not use the commands below as a current runbook.

The evidence from the incident

The relevant log entries included x509: certificate has expired or is not yet valid and a current time later than the certificate's expiry time. That established a certificate-time problem in the affected deployment. It did not identify every possible cause of a TLS failure.

x509: certificate has expired or is not yet valid
current time 2021-12-29T08:27:34Z is after 2021-12-28T06:35:39

What I checked

I connected to the host, checked its system clock, confirmed the Rancher container was running, and inspected the container before changing anything. Time drift matters because a valid certificate can appear expired when a host clock is wrong.

The original intervention removed specific generated certificate material inside that deployment, triggered certificate refresh activity, and restarted the container. Those steps were tied to its Rancher and K3s state. They included an insecure TLS bypass and destructive secret removal, so I have intentionally not republished them as a generic fix.

A safer current response

For a current certificate incident, start with the exact versions, certificate issuer, clock state, trust chain, renewal method, and a backup or recovery plan. Follow the supported runbook for the running Rancher and Kubernetes versions. Confirm the scope of any secret or certificate deletion before executing it, and record the replacement and verification steps.

The useful lesson from this incident is narrower: the error text, system time, and deployment state should point to the recovery procedure. A copied command sequence should not.