Post by TheGreenKnight on Jan 17, 2010 14:04:33 GMT 9.5
Upon reading Tronic's post (again, thanks for keeping us all in the loop Tronic: You're a Godsend.) I rushed over to read [Admin]-Robocop's latest post. I have copied it here, so that you can have confidence again to play on FLU:
----------------------------------------------------------------------------
Well, no surprise to anyone, we've had some rather severe server issues in the last four months. It's been a very trying time for all of us, and for me and [Admin]-Tim personally. Our personal lives have been affected by these travails as much as or more than our server lives. I know I would like to thank my wife (SWMNBN), and Tim's wife as well for their patience and understanding.
Here's the story. In late August we had a motherboard crash requiring some hardware repairs on our old machine. I believe the crash was due to a sudden power outage in our area. We replaced the motherboard and did some other work while we were at it. Now, either just before this, or just after, we started also experiencing server stability issues where the server would just crash on us for no readily apparent reason.
My first assumption was that it was hardware related and I began the classic 'Let's throw parts at it until the problem goes away' approach. When we replaced the mobo, we also replaced the memory so we didn't think that was the problem. During the troubleshooting period I stripped out all non-essential hardware (VGA card, floppy drive, extra optical drive, sound card, etc.) and kept the machine at the bare minimum we needed in order to run a server. We also completely wiped the hard disk and reinstalled everything. The instability continued.
Next I replaced the HDD and the NIC but that didn't fix it either. After that came the router, the modem, and a call to our ISP to see if there were any issues on their end. Of course, there weren't and the instability remained.
Tim and I spent many long hours going through everything and making sure the server was up as much as possible. We owe the continued existence of FLU as much to [Admin]-Tim as we do to Universe, [Admin]-Firewall, and to you, the players who've remained faithful to us even through these trying times.
Ultimately we ended up building a whole new machine to rule out hardware issues with the old machine and yet we still had the same instability with the same particulars. So, we knew it wasn't hardware related. As nothing had changed software wise since April of 2009 at the latest, we were also comfortable that the software was not the problem.
That led us to two inescapable conclusions.
First, that it might be ISP related. Due to the inconsistency in server donations I did not feel comfortable signing a year-long lease with a pro hosted server. I can't afford it and neither can Tim. That was going to be a last resort option.
Second, that we might be the victims of disruptive activity from the outside, either deliberate or accidental. After much discussion with other server admins, we were unable to come up with any specific thing that could cause a server to crash in the manner ours was crashing. It was very discouraging to me to have to believe that the activity was designed to deliberately disrupt our activities. I suspect(ed) either some 'competing' server, or a disgruntled ex-player as being the cause, but could not with any consistency identify what was going on.
Until the other day...
Quite by accident I happened to be looking at a particular aspect of our server when FLServer.exe dumped and I noticed something not normally visible in the server activity. As it looked odd I investigated further and found an interesting phenomenon. After much cross-referencing between logs and reports, I found that over the last month or so, about 95% of the crashes could be specifically pinned to a very limited number of IP addresses (3) and accounts (4). Login attempts were made to these accounts on these IPs and in every case, the server would immediately dump. Now, I don't know if this was by design or not. What I do know is that as far as our logs go back, this is consistent. It is my opinion that if this were indeed accidental due to something messed up on the player's side, or a corrupted player file on our side, the player would not continue to try logging into the server on that account only to never connect. Sooner or later he or she would give up and stop trying, or they would contact me complaining that they could never connect. This did not happen. For now, I will not speculate on these forums as to the motivations of this or these players.
At any rate, we've taken some steps to address these issues and I can now report that we've not had a crash for the last sixty hours. I think this is promising as we've not had a day without crashes for quite some time.
I don't know if FLU will ever come back to the status it enjoyed before these crashes. I would like to believe that it can. That depends on you however. I can't run a server without your participation, both with your presence on the server, and with the occasional donations I receive that help me to convince SWMNBN that we're not ready to shut down yet. I've received a couple of very generous donations that have helped with the replacement of the hardware, but more than that we need players.
So, I'm asking you all to spread the word amongst your Freelancer friends that we're still in business and committed to our efforts in being the best FL server out there. We still think we have something of benefit to offer.
Thank you for sticking with us. I hope for your understanding, patience, and even forgiveness in this matter. I apologize that it has taken this long to finally get to the place where we might have found the problem and solution.
Humbly,
R
----------------------------------------------------------------------------
Well, no surprise to anyone, we've had some rather severe server issues in the last four months. It's been a very trying time for all of us, and for me and [Admin]-Tim personally. Our personal lives have been affected by these travails as much as or more than our server lives. I know I would like to thank my wife (SWMNBN), and Tim's wife as well for their patience and understanding.
Here's the story. In late August we had a motherboard crash requiring some hardware repairs on our old machine. I believe the crash was due to a sudden power outage in our area. We replaced the motherboard and did some other work while we were at it. Now, either just before this, or just after, we started also experiencing server stability issues where the server would just crash on us for no readily apparent reason.
My first assumption was that it was hardware related and I began the classic 'Let's throw parts at it until the problem goes away' approach. When we replaced the mobo, we also replaced the memory so we didn't think that was the problem. During the troubleshooting period I stripped out all non-essential hardware (VGA card, floppy drive, extra optical drive, sound card, etc.) and kept the machine at the bare minimum we needed in order to run a server. We also completely wiped the hard disk and reinstalled everything. The instability continued.
Next I replaced the HDD and the NIC but that didn't fix it either. After that came the router, the modem, and a call to our ISP to see if there were any issues on their end. Of course, there weren't and the instability remained.
Tim and I spent many long hours going through everything and making sure the server was up as much as possible. We owe the continued existence of FLU as much to [Admin]-Tim as we do to Universe, [Admin]-Firewall, and to you, the players who've remained faithful to us even through these trying times.
Ultimately we ended up building a whole new machine to rule out hardware issues with the old machine and yet we still had the same instability with the same particulars. So, we knew it wasn't hardware related. As nothing had changed software wise since April of 2009 at the latest, we were also comfortable that the software was not the problem.
That led us to two inescapable conclusions.
First, that it might be ISP related. Due to the inconsistency in server donations I did not feel comfortable signing a year-long lease with a pro hosted server. I can't afford it and neither can Tim. That was going to be a last resort option.
Second, that we might be the victims of disruptive activity from the outside, either deliberate or accidental. After much discussion with other server admins, we were unable to come up with any specific thing that could cause a server to crash in the manner ours was crashing. It was very discouraging to me to have to believe that the activity was designed to deliberately disrupt our activities. I suspect(ed) either some 'competing' server, or a disgruntled ex-player as being the cause, but could not with any consistency identify what was going on.
Until the other day...
Quite by accident I happened to be looking at a particular aspect of our server when FLServer.exe dumped and I noticed something not normally visible in the server activity. As it looked odd I investigated further and found an interesting phenomenon. After much cross-referencing between logs and reports, I found that over the last month or so, about 95% of the crashes could be specifically pinned to a very limited number of IP addresses (3) and accounts (4). Login attempts were made to these accounts on these IPs and in every case, the server would immediately dump. Now, I don't know if this was by design or not. What I do know is that as far as our logs go back, this is consistent. It is my opinion that if this were indeed accidental due to something messed up on the player's side, or a corrupted player file on our side, the player would not continue to try logging into the server on that account only to never connect. Sooner or later he or she would give up and stop trying, or they would contact me complaining that they could never connect. This did not happen. For now, I will not speculate on these forums as to the motivations of this or these players.
At any rate, we've taken some steps to address these issues and I can now report that we've not had a crash for the last sixty hours. I think this is promising as we've not had a day without crashes for quite some time.
I don't know if FLU will ever come back to the status it enjoyed before these crashes. I would like to believe that it can. That depends on you however. I can't run a server without your participation, both with your presence on the server, and with the occasional donations I receive that help me to convince SWMNBN that we're not ready to shut down yet. I've received a couple of very generous donations that have helped with the replacement of the hardware, but more than that we need players.
So, I'm asking you all to spread the word amongst your Freelancer friends that we're still in business and committed to our efforts in being the best FL server out there. We still think we have something of benefit to offer.
Thank you for sticking with us. I hope for your understanding, patience, and even forgiveness in this matter. I apologize that it has taken this long to finally get to the place where we might have found the problem and solution.
Humbly,
R
