Hello! As some of you may or may not know the entire network suffered a large amount of downtime around 4AM GMT. I am going to give a post explaining what we have discovered so far. I will edit this and post updates via the discord as we find out more, but as of now, this is on-going. At 4 AM the network started its daily backup system (this includes taking a copy of every server and uploading them to an external backup server for security). The machine didn't have enough disc space to load these servers twice. This resulted in our machines storage capacity maxing out. When storage runs out with Minecraft servers it detects it within Milliseconds and shuts down the server to prevent large amounts of data corruption, however, some plugins run file checks and updates a lot faster than the Minecraft checks these files did, in fact, become corrupt. One of the main plugins that this happens in is: ASkyblock and AcidIsland The two plugins we use for SkyConstruct and Lost Survival. We did have to roll back SkyConstruct to a backup we had saved on 01/12/2017 and we had to roll back the Lost Economy and plugin data to the same day save however that world was saved. Luckily this did not affect any of our MYSQL/core data as this is kept in a separate partition on the machine. I am still looking into the machine's logs in case there was something else that caused or boosted the issue to the severity of what happened. Our developers are changing our backup system to a more storage-friendly type so this shouldn't happen again. If you have any issues with what you are doing in the game then please create a ticket on our forums https://AJGaming.net/support and my staff team will try to help you out in the best way we can. Kind Regards, Jack C.
For anyone who gets confused over the date, as the UK does Day/Month/Year. The date was December 1st, 2017 and NOT January 12th, 2017.
So you're saying server capacity had to be increased? How come there isn't a way to see this before it happens? Like they say, always buy your shoes a size too large, since you'll grow into them soon anyway. (Doesn't really fit if your feet have stopped growing lol)
The thing that I suspect to have caused this is that we still had the Old Advanced Wizardry files downloaded on the machine so it attempted to make a backup of 3 AW servers worth of data.
If this is the case, it would be in combination with reaching max capacity that things failed, and so I still am asking, is it a thing that has to do with running out of space to the layman or is it just a code issue? The difference would be, and by the way I'm just a regular e-citizen of this fine server, mind you, a difference of needing to add more hard disk space in the common parlance.
I would just like to point out for people, it can take a lot of time, resources and effort to find issues with mods on a server, especially one that ran fine while being tested and having new mods added, if anyone has seen server logs before for a modded server, there are literally thousands if not in the millions of lines of the log to go through. Add all the disk space that these logs can take up plus the above and disk space is quickly eaten. The point to take away from this, most of the data was saved from corruption thanks to what resouces were available, thanks to the staff putting in time, the server is back up and thanks to the effort of the staff, you've lost maybe 2 full days of building in a game? It's not as bad as loosing the entire server and having to start from scratch, so all in all, it' a good result and a great effort from the staff
