What has been going on again?

Hello everyone,
Certainly, you noticed that there have been several severe technical issues in the past days.

Here is a short summary what happened:

First, all started with a routine server reboot for applying the newest security patches to our operating systems. Simultaneously we have done some Updates concerning the backend of our game servers.

And so all the trouble began ………

After the reboot of our root servers, everything seemed fine at first, apart from Teamspeak, which was fixed very fast after that.
Starting our Minecraft servers was no issue at first. But to make sure everything worked fine after the backend changes, we rebuilt some server containers which resulted in totally broken containers which meant Minecraft servers that we could not start and manage via our interface anymore.

Hours into debugging, following wrong traces and encountering problems that did not make any sense anymore we had no more clue what the issue could be.

After some hours of despair, we got in contact with Tactical Fish, the developer of our management interface for the Minecraft servers.

But he didn’t know either what the problem could be… at first.

Finally, after overall 12 hours of research and testing, we found a short line of text in the depths of the daemons code, that was changed by the latest update which broke everything.

As a result, a hotfix was published for the panel immediately.

Then at about 23:00 CET our servers got back up online… Finally!

In the meantime, there were a few problems with the reachability of our forum, but that has been fixed fast, we had to prioritize the rest of our service at first.

Because of all the reboots and tests, some Minecraft worlds were broken but we also fixed that as fast and good as possible. The only server that is still broken is the Craftmine server is still down, the issue here can only be resolved by the owner of that server, though, on whom’s answer we are currently waiting.

The Novus castor was not reachable till the next morning, but that was a simple fix too, just some DNS settings we changed in our tests to get some servers running on a different machine, without success, though.

At last, I will quote the commit message of the bugfix (written by Dane (Tactical Fish)):

8… hours… later… 

Out of memory. We wish to hold the whole sky, But we never will. Thanks to @ET-Bent for not murdering me when we found this.

So yeah, to summarize this: Big shoutout to Tactical Fish and his awesome Pterodactyl panel for helping us in times of trouble! Only with his help this issue could have been resolved and even though it took us long to figure this out, it only improved the panel even further.

Proudly sharing this awesome software