Archive for the 'Maintenance' Category

What had happened two weeks ago to HY?

Hello,

today we like to explain what had happened two weeks ago leading to a one week HY downtime. (The same applies to ht-flagtools and cupmanager, because they are hosted on HY server!)

During the morning hours of thursday the 22th of January, we recognized some problems with our server. We saw two options: 1) killing a lot of processes and thus try to get rid of the problem or 2) simply restart the server. 2) sounds easier, but since we had not restarted the server for a while we were a bit frightened that may problems could occur, which could lead to a shorter downtime (30 to 60mins). Thus we tried to kill a lot of processes first. Pretty fast we come to the conclusion, that this might take longer as restarting the server and may does not fix the problem as well. Therefor we decided to restart the server.
As we still did not get any response from our server ten minutes after the reboot, we were already frightened that we might run into a smaller problem. We waited additional 30mins to be sure, that no file systems checks were running, no response at all. Then we rebooted into a session enabling us to see the screen of the server – you have to know, that our server is located in a datacenter in the area of Nuremberg and thus we do not have physical access to the server. We saw that the boot loader (grub) did not load the operating system, the reason was, that he did not found the filesystem on our hard drives. We rebooted into a rescue system and reinstalled grub – still it did not work. For an unknown reason grub could not found the filesystems or any software raid-devices, but we had loaded all neccessary drivers properly. Since the boot loader of the rescue system found both, we decided to install this bootloader to our system – still no success. We also tried to down- and upgrade to other versions of grub without success.

Since we were running out of ideas, we were working on a plan B. Plan B was to rent a new server and replace the old one. It is no fast and a more expensive resolution, but a working one. After several successless attemps to get grub working, we rent a new server and started installing the operating system and all server software. We took the opportunity and updated all software to the latest versions (which we had in mind to do for our old server soon anyway). By doing this the whole process got slowed down, but we have more stability and security in future. Each day after our regular work (and sometimes during it) we have worked many hours to install and set-up all the stuff resulting in a massive lack of sleep.

At the beginning of the following week our server was configured and we felt that we can be back online soon, but then we have observed some problems with the new server software, which we we able to fix until Wednesday. On Wednesday evening we had done a last test of all software (with some hy users) and it seem to run well. We decided to release on friday evening, because thurdays we were not able to watch out for the server (or even stop it) if a problem occurs. Since we were aware of the fact that a lot of people will login in the first hours when HY is back, I decided to announce that the server will be back online during the weekend and thus lowering the number of users in the first hours on friday to allow us to watch our for problems and fix them, before all people will harass the new server configuration and report bugs to us.

When we put the server online, 50 users were online within minutes. After 1h the first user posted on HT forums that we are back online and some hours later we announced it on twitter as well. Everything worked well but we are absolutely sure, that it was right to not announce when we were back online because in case of an error it might have ended badly 😉 .

 

We are pretty thankful to all kind tweets we got during the downtime! A few words about our backup strategy which might calm down the less kind tweeters a bit: We are running a full backup of HY site and database (same applies for ht-flagtools and cupmanager) each week and an incremental one every day.

We are also thankful to all people who have donated some money to us – thank you!

Special thanks to former HT manager exciler, who have helped us a lot!

/Markus (Mackshot)

Maintenance, Foxtrick cooperation and future projects

Hello,

first of all we like to announce a short hattrick youthclub downtime. We have to update our server software, thus we have to take down hattrick youthclub for a while. This downtime is scheduled for Wednesday, 10th of October from ~10 am till we have finished 😉 . Since Mackshot was on holiday during the Hattrick season break, we were not able to do these updates during the season break and decided to perform then now.

In the last month we started several cooperations with Foxtrick. We have finished our work on these cooperations now, and now we are waiting for Foxtrick to release these new functions which mainly increase usability of hattrick youthclub. In future your coach match reports will be added to hattrick youthclub automatically when you read them and rejected players are added to our database automatically as well. But both of this is still not working and as soon as Foxtrick have released a new version with those functions we will let you know.

Currently development of hattrick youthclub is a bit stucked, because Hattrick does not concentrate on new CHPP Files right now. So we are waiting for new files from them and started cooperations and worked on other tasks. One topic we might will work on in future is to give you an better overview  of your scouting history (and success). We haven’t talked about the details, but I feel it could be a nice small addition to hattrick youthclub. Besides that we are always working on smaller bug fixes and try to improve the page in details continuously.

We also want to take the opportunity to thank you for supporting us by donating or buying some credits. Thank you. This is a pretty good motivation for us!

Markus

2 weeks with our new server

Hey,

our new server is now 2 weeks old and we think it is time for a short feedback how the new server performs and a short summary of what happened.

In christmas time we had the first problems with our old server. We could fix it and have not seen, that it is a more general problem there. In January the problem occured again. First we thought it were a different problem but after many hours / days of working time we figured out, that we do not have a chance to get things running again on the old server, because there is a hardware failure. We had two options there: Asking our server hoster to replace part by part to figure out, which hardware piece is defect or to order a new server.  We decided to order a new server for two reasons: 1. We felt that this will bring hattrick youthclub back online faster and 2. a new server for the same monthly fee is noticeable faster. The bad thing about ordering a new server is, that we have to pay an installation fee of EUR 150, but thanks to your donations last month we already got round about EUR 100 of this from our lovely users 🙂

After some performance problem during the first 2 to 3 days, the new server is running pretty well. I personally had not a single loading time problem since then. Therefore we will cancel the old server today. We hope you are satisfied with our new set-up too.

In the latest release of hattrick youthclub, we have change the behaviour of the player news. The news does only include information up to the latest match you have imported. So please import the match and then you will get pretty soon a news report, if there are any news. Addionally we will show you another news dialog, if there are news for your twins. This will not happen to often, because most twins are older and no longer in the Youth Academy of the owner, therefore they do not have updates ;).

Cheers,

Markus

New Server

Hello,

after spending most of the time to trying to fix the issue with the corrupt tables in our database system till friday morning, we decided to rent a new server.

Since Friday morning we worked on fixing a bug with corrupt database tables. No matter what we tried one table was corrupt at least. We copied all data to another server with similar configurations, and all tables are fine. We are pretty sure, that the we have an hardware failure there. Since we could not find one we decided to move to a new server, because otherwise it would take too much time. Moving to a new server does take some time too, because we have to set-up this server completly. We do also have to pay some installation charge, but we think it is the best way to bring hattrick youthclub back online fast!

We have already ordered our new server and waiting for it to start with the configuration. We will keep you up-to-date (on facebook, twitter and this post here) about the installation progress, but first of all, the server needs to be installed by our hoster. We will not make any prediction, when the site will be available again, it depends too much on the day, when the server is installed in datacenter.

 

16.1 16:13: Our new server has arrived. Now we have to configure the server in the next days, starting tonight!

17.1 16:58: The basic configuration of our new server is done. Now we have to wait until all data is copied during this night and then we can go on with details!

18.1 18:53: We are progressing very well. The wiki is already working. We are looking for bugs right now :). Afterwards we will fix them and then we will check the option to switch hattrick youthclub on!

18.1 20:04: We are back online. But we will keep an close eye on the server and may have to switch it off, if there is a problem, please be warned!

Downtime…

Hi,

currently hattrick youthclub is down, because there is a database problem. Next let me summarize what happened and what will happen:

  • This morning the database server shutdown automatically. The reason: a database table is corrupt
  • After unmounting and mounting the filesystem everythings worked, but when checking the database tables manually we figured out, that six database tables were corrupt.
  • The database management system were able to repair five of this six tables.
  • We tried to drop the last table and to reinsert the data again, but without any success.
  • We had the feeling that the filesystem has a problem in some way. Therefore we  copied the data to a new file system and started with recreating the database again. We are proceeding well and till now, the table seems to be alright.

Since the database tables are huge those checks and fixes need much time. Therefore hattrick youthclub is down for the whole day. But we are confident, that we can bring hattrick youthclub back within the next hours.

You do not have to worry, no data is lost! Even if we are not able to bring this single table back running, we can just create a new empty table and start the webpage, because this table is only a CHPP File Cache. A cache naturally can be deleted, but it would be nicer to have the old one running, therefore we tried it all day long and keep on trying it this evening/night.

One major question is, why got these database tables corrupt. We do think, that we have an explanation for this. If we are wrong, it is most probably an hardware problem. If this is the case, we will move to another server to fix this issue. But hopefully our idea is right, and it is only a software problem, where we do have a workaround.

We will post an update in this post and on twitter / facebook, when we are back!

We are sorry for this downtime, and want to thank you for your support and patience!

Please follow us on facebook or twitter for the latest news …

hattrick youthclub is back online!

We are back online … finally!

For more information also checkout our previous post… here are the updates:

The problem, was not related to the RAM as we thought first. There is a problem with the filesystem, while rebooting (especially with our database partition). This problem is not fixed yet, but we have a workaround running now, which should guarantee, that this problem will not be a problem again. But we have to find a better solution for this in the long run.

Sorry for the downtime 🙁 …

Markus

 

hattrick youthclub is down!

We are very sorry, but hattrick youthclub is down since 4am this morning. Here is what we do know till now:

It started with a problem with the connection to the database server. I tried to figure out what went wrong there, but could not find a problem. Therefore i decided to restart the server, but the server did not start-up. Then i restarted the server in rescue mode to check what went wrong during starting – without success. After some test i found a problem with the memory (RAM). I created a ticket for our server host and asked him to replace the RAM with new RAM. I also asked them to check other components.

Now we are waiting for reply from their side. I have no idea, how long it will take until they have replaced RAM and i also do not know, if this is the only problem we do have… You can be sure, that we will keep you informed (here in the blog and on facebook/twitter) and that we are fighting for bringing the youthclub back soon!

Happy Holidays 🙁

Maintenance and Updates

Hi,

hattrick-youthclub will be offline in a couple of minutes for a short downtime. During this downtime we will

  • install some server-side security updates
  • delete or hide old coachcomments (training reports)
  • remove older no longer used translations
  • and some smaller stuff

The following updates are released:

  • On statistics page you can see your top resales of your own team
  • On player attributes, player skill matrix, lineup recommendation and other sites you will find the amount of matches up to the next skill-up (if you train the player on the best position with best training). Additionally we show the top3 skills on all pages now.

We hope you enjoy our latest updates and wish you all the best for next season!

We are moving to our new server!

In our latest blog post, we asked for your financial support, because we have to move to a new server. We’ve got a lot of donations, thank you.

We already bought a new server and configured it in the last days. Yesterday our wiki and some static content already moved to the new server. This morning the website and database is moving to our new server. Due to this, hattrick youthclub will be offline for 2 hours. We already have to change the DNS settings (DNS tells your browser, where he can find a domain), because the new server has a another ip address. Depending on the renewing speed of DNS Servers of your Internet Provider, you can reach hattrick youthclub after the downtime of 2 hours or later this day. During the switch to the new server, you will get a redirect to the blog.

We will post updates about the move here!

In the last 2 years, we had cost of EUR 25 a month. These costs are increasing to EUR 75 a month. We hope to get enough Donations to pay our bills for hattrick youthclub 😉 .

Update 10:51h: The website moved successfully. If you cannot reach it, please wait until the DNS Servers of your Internet Provider renewed the ip of hattrick youthclub.

Maintenance downtime

On the 26th and the 27th of December will be one or two downtimes of hattrick youthclub. We scheduled the first downtime for the 26th at 17h HT-Time. We will redirect you during this downtime to the blog, where we will keep you up-to-date if there are any problems.

What is done during the downtime?

Our work is devided in two phases:

  1. We will reorganize some comments. There are two issues we will work on. The first one is related to newer research results. These results will find their way into hattrick youthclub. The second one is the category “skillup”. There are some no longer used comments, which need to be removed. When we change something on the comments, we need to change database entries and translations. This needs a lot of time, because there are a lot of database entries we have to change.
  2. In the second phase, we will install some server updates. A new kernel, apache and php version and a lot of other smaller stuff need to be installed.

We hope to need only the 26th for all maintenance work. But we expect to fix some issues, after all this changes, on 27th, too. There is no reason to be panic, when hattrick youthclub is not working on 27th, too :). If you are interested in how we progress, we will update this blog entry during the update a few times.

Update 26.12.2009 17:11h: Maintenance work had started!

Update 26.12.2009 19:45h: The first phase of the update is done. Now we are going to update the server’s operating system and the applications.

Update 26.12.2009 21:32h: All updates seems to went well. We are working on a small bugfix in webserver application.

Update 26.12.2009 22:56: The downtime ended a few minutes ago. The next hours/days will show, if everything went well 🙂