Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: foreman logs are in local UK time, so real deletion time was 15:06 CET

...

Timeline

all times here in CET

1415:06 - foreman triggered vm remove action.
15:35 - Dick Visser contacted us about problems with wordpress and
filesender because he got monitoring alert that sites are not accessible
15:41 - Konstantin Lepikhov confirmed that hosts are missing in vmware
15:44 - Konstantin Lepikhov identified the issue and started investigation on vmware cluster
16:43 - Konstantin Lepikhov contacted Qaiser in Slack to confirm backup existence
16:45 - DevOps confirmed that there are no backups or extra copies on
vmware storage
17:00 - Konstantin Lepikhov called Qaiser Ahmed in Slack, no response.
...
17:00 - Dick Visser confirmed that he has backups on server at
Amsterdam university (those are daily backups taken directly by VMs
itself).

...