Commit 7970da3b authored by Erik Huelsmann's avatar Erik Huelsmann
Browse files


parent a9f80d6a
Pipeline #849 passed with stages
in 6 seconds
......@@ -24,13 +24,35 @@ infrastructure.
Thanks to the Trac team for their continued efforts!
### 2019-04-28 outage
Earlier today, there was an outage for
Further statement being worked on.
Erik Huelsmann
### 2019-04-28
On Sunday, April 28th, the hosting infrastructure experienced
an outage of approximately 5 hours when we encountered a unexpected
difficulty in upgrading our services. As you know, we take our responsiblity
towards the integrity of the data and services associated with
seriously. Part of that responsiblity is owning up to our mistakes, so here’s
our deeper explanation of how we inadvertently impinged on the integrity of the
main host file system.
While work was in progress on changes to set up support for better and faster
restore from backup, if such was required, a small change was required on the
main BTRFS filesystem: moving a snapshot from one place to another on the file
system. This operation couldn't be performed while the system was online, so
the VM was brought down.
After the VM was down, the main filesystem was mounted on the host system to
perform the operation. After unmounting the filesystem and remounting it to
verify the result, the filesystem at first wouldn't be mounted. After
executing the `btrfs rescue zero-log` command, the filesystem did mount,
but only in read-only mode.
Using a combination of the daily backups and the read-only filesystem,
a new filesystem with the same content been created.
I'm very sorry for the incident in general and having your data at risk in
particular. I haven't found any data-loss.
Erik Hulsmann admin
### 2019-02-09 *Foundation to host ASDF fundraiser!
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment