Re: Recent Crashes, Downtime, and Rollback.

Discussion in 'Announcements' started by Daphsquid, Jan 8, 2020.

  1. Daphsquid

    Daphsquid Server Owner
    Server Owner

    Joined:
    Jan 13, 2017
    Messages:
    672
    Likes Received:
    291
    [​IMG]
    Hello Players,

    We find it necessary after the recent Towny rollback to make a public statement regarding the status of the server and what the Sysadmin Team has been up to in the recent weeks to fix it. Initially, we did not find it ideal to draw this much attention to the issue, but with the growing spread of misinformation regarding the situation, it is now our obligation to the community to update you all on what has been happening behind the scenes.

    It started with random shut downs on both Towny and Creative. We were aware of these shut downs when the first one happened on December 14th, 2019 at around 12:30AM EST and searched the server logs for an explanation. Of course, nothing was there to be seen— not even a crash report was generated to provide us with answers. The issue with immediate shut downs (like the ones experienced on Shadow Kingdom during the last few weeks) is that no proper file saving occurs, which can lead to corruption in some instances. Luckily, nothing seemed to be broken yet, but nonetheless, all of us were searching for an answer that would explain these random shut downs.

    Soon thereafter, the issue spread further and began affecting the entire dedicated server that we host our gamemodes on. The first instance of this was on December 15th, when I came online to notice that the entire dedicated server had restarted itself— similar to how the individual Minecraft servers had been doing prior. We immediately began undoing any of the recent updates and uninstalling new software in hopes of repairing this issue, but the only way to tell if it was fixed was to wait and see if another crash occurred. After contacting our server host, we still had none of the answers we hoped for.

    This went on for a few weeks until finally on Sunday, January 5th, 2020 at around 6:30PM EST when our dedicated server restarted itself, the servers did not start correctly— some data had been corrupted due to the improper shut downs. This was the reason for the few hours of downtime experienced on this day. We back up Shadow Kingdom every few hours, so this issue was easily fixed once we got online to address it. However, our backups software did not update itself, so after the server came back online, the old (broken) server was being backed up instead of the live one. That is why when we required the use of another backup on Tuesday, January 7th, 2020 at around 6:50PM EST, the latest stable version of Towny we had was from before the crash on Sunday (the latest not-broken backup). Of course this has been manually corrected, and double— even triple checked now to ensure nothing of the sort happens again.

    During the downtime on Sunday, though, I went ahead with removing individual pieces of software, and hopefully pin-pointed the file that may have been causing these shut downs. Notice how none of the servers shut down randomly since then (unless done by an admin manually or automatic daily restart). However, only time will tell if the issue is fixed. And regardless of that, we have backups of everything, so in the case of what happened on Sunday (and Tuesday), we should never experience a rollback of that length. It only occurred due to the very particular circumstances of certain software becoming mismatched, which we are now on high alert for. At first, it was our belief that this was a case of faulty hardware, but now we are considering otherwise. Please also note that we have plans to move hosts in the future, but this is unrelated and necessary for other reasons.

    In conclusion, we wanted to put this statement out there to allow everyone to know that we have 1) been aware of this issue since day one, 2) been taking measures to repair it, although progress has been slow due to lack of information, and 3) been doing everything in our power to provide the top-notch Minecraft experience that we have successfully done for the last (nearly) 5 years of being online.

    Please feel free to comment below with any questions or concerns you may have.


    Thank you,
    - SK Sysadmin Team
     
    • Informative Informative x 4
    • Like Like x 3
    • Friendly Friendly x 2
  2. jbean45

    jbean45 Member

    Joined:
    May 18, 2019
    Messages:
    23
    Likes Received:
    5
    Thx for doing all you can guys
     
  3. paylode

    paylode Well-Known Member

    Joined:
    Jan 24, 2018
    Messages:
    53
    Likes Received:
    12
    Ya hope u guys can do your best to keep server up. From the looks of it, it looks like u guys r doing your best currently. Hopefully I no longer have to log in and see towny is dead sense It gets really annoying for me at times when I have a time limit to do stuff and I cant do what I wanted too because its down.
     
  4. Daphsquid

    Daphsquid Server Owner
    Server Owner

    Joined:
    Jan 13, 2017
    Messages:
    672
    Likes Received:
    291
    Considering that the random shut downs have ceased since the updates I completed on Sunday, I'm fairly confident that those issues are now resolved. From now on, the servers should be online 24/7 and only shut down or restart if an admin manually does it (or the automatic daily restart). We're keeping a close eye on the workings of the servers to ensure that it continues operating to our standard.
     
  5. paylode

    paylode Well-Known Member

    Joined:
    Jan 24, 2018
    Messages:
    53
    Likes Received:
    12
    thx for all that u guys did I can understand if it was stress full at times. Glad for getting it working u guys did good :D
     
    • Friendly Friendly x 1

Share This Page