Formal letter sent to GOA.

[elisera]

What? Companies HAVE gone out of business due to database issues. Infact, crashes like this could have well meant the end of any normal business. Imagine this happening in a hospital. A while ago, in a german hospital something like this happened, a woman got told she had aids and she commited suicide. Turns out she had nothing lethal. Or imagine this happening to a bank, suddenly everyone lost their money. Seriously, database issues like this typically can destroy a company completely.

I think a bit of perpective is needed tbh... this is just a game and the comparisons you made have absolutely no bearing on the issue at all.

 

[Ballard]

nerds gettin' wound up about online games and writing 'formal' letters cracks me up big time. Thanks for the laugh.

 

[scorge]

in business money is king, formal letters get stuck on the staff wall. Good effort wont make one damn bit of differance though....

 

[Stinko]

Hmm... 3 years of 12 Euro a month invested in something you think is fun..... and all of the sudden those 3 years of struggle and money just happen to be lost for nothing. Ofcourse you get upset and want answers

 

[elisera]

the entire 3 years is not lost.. please dont exaggerate. They are doing their best to get everything back to you, its not like you have lost the chars themselves.

 

[Hawkwind]

no it wouldnt and you are completely wrong!!!!!

in any DR situation, you have to goto the last known good backup - in pry's case this was a frikken 4 hour roll back - i bet the GOA team are wasting lots of time on the sad gits who havnt lost ANYTHING - and yet are being greedy trying to get summat for nowt.

Actually you are completely wrong! My only lvl 50 on pry was stripped completely. When I asked if they could simply look at a backup from anywhere up to a 3 weeks previous, the reply stated they could not as they were not confident when the corruption started. As such they were having to ask for lists of peoples items and manually check them against transaction logs on the server.

I upgraded one of my comps and reformated old drives before removing them but forgot to copy char info so have no good records of what the char had. But unlike some people I have Excal chars to play and I feel very sorry for those affected. At least I can wait 6 months before rebuying all the crap i need.

 

[Deepflame]

I think a bit of perpective is needed tbh... this is just a game and the comparisons you made have absolutely no bearing on the issue at all.

I believe we were talking about database crashes putting companies out of business. I didn't think you were talking specifically about game companies.

 

[BimboBane]

I think a bit of perpective is needed tbh... this is just a game and the comparisons you made have absolutely no bearing on the issue at all.

No saying "it is only a game" is so wrong, this is a service we pay hard earned money for. We should demand the same level of service from every service we pay for wether it is a mailprovidor or a game is the same to us.

What? Companies HAVE gone out of business due to database issues. Infact, crashes like this could have well meant the end of any normal business. Imagine this happening in a hospital. A while ago, in a german hospital something like this happened, a woman got told she had aids and she commited suicide. Turns out she had nothing lethal. Or imagine this happening to a bank, suddenly everyone lost their money. Seriously, database issues like this typically can destroy a company completely.

This particulare company business _IS_ games, and I guess severall of their customers have serious doubts about the services quality after this disaster.

//Johan

 

[Sharkith]

Simply impossible to assure anyone on this. Servers issue appear, in every company, gaming or not.

of course they do - but surely you can respond with a statement about what your paying cutomers can expect under reasonable circumstances? I mean seriously it is an extremely common thing to do.

The letter is a fair one and has hit what is the most important issue.

 

[OrreBorre]

Please tell me you're fucking joking?

For the record - you're cool.

 

[Sharkith]

That not close enough to what your after?

nope it does not do what an usual SLA would contain.

 

[BimboBane]

the entire 3 years is not lost.. please dont exaggerate. They are doing their best to get everything back to you, its not like you have lost the chars themselves.

I got two of my BG toons deleted, and I know of few others Norrsken alliance members that have got their accounts totaly whiped and had the "select realm screen" when they logged on Prydwen.

//Johan

 

[elisera]

I got two of my BG toons deleted, and I know of few others Norrsken alliance members that have got their accounts totaly whiped and had the "select realm screen" when they logged on Prydwen.

//Johan

Ok thats slightly different, hadnt heard of anyone losing chars, only items.

I still feel people are overreacting though. Everyone knows they are doing thier best (oddly enough they probably dont want to lose customers) and as for guarenteeing it never happens again, no company can do that. If they had had an SLA before this all happened and it still all went the same would you still complain? What difference is the statement of the agreement gonna make?

 

[Kaun_IA]

Just to sort out a historical FACT! It was Qui-Gon Jinn who said it to Ben Kenobi, later known as Obi-Wan Kenobi in Episode 1 - The Phantom Menace >.<

On subject
Real companies take live backup, not backup each 4 hours or whatever the rollback was. Live backup makes sure you dont lose anything when a database gets corrupted.

Its hard to judge where the fault is here, because we dont know a thing about it. It could be GOA, it could be Mythic - noone knows what exactly is written in their agreement.

And I do agree we need some sort of SLA.

now think for asecond what are u saying.
if there was a live backup, then when DB gets corrupetd the backup will be curropted at the same time.

 

[Alan]

now think for asecond what are u saying.
if there was a live backup, then when DB gets corrupetd the backup will be curropted at the same time.

High end databases keep a transaction log too so rolling back is not a problem to remove corruption - but to be honest here, a 4 hour rollback is more than good enough for a game better than a nightly backup.

 

[Bahumat]

you simply cant guarantee nothing will ever happen to the server or your character.

its like calling your isp and saying "ok two weeks ago your server went down, tell me its never gonna happen again"

 

[Gnejs]

Since the last database failure have caused much frustration to people playing on Prydwen we ask you to provide us with you assurances that this situation is never going to happen again.

It says this SITUATION! And what does it mean? Its not like you want GOA to give their word that DB failure will never happen again. You can never promise something like that. Its about how they will prevent this data loss if DB failure happens again. If you think its ok that people loosing items, chars and a lot of time because there is a server problem, then i disagree with you. Of course we, the paying customers, must have assurance that GOA have a working back up system for this kind of problem. Is it not why there is something called back up? To have a copy when the original file get corrupted? Obviously GOA's back up system havent worked well with this kind of problem and what will they do to prevent this scenario from happening again? I do wanna know if there will be any changes made by GOA to prevent this situation again. So its a fair letter, and very strange that some people dont think that. And laughing at it, like some people do, will for sure not make any differences.

/Gnejs, Midgard/Prydwen

 

[Esselinithia]

Possible measures they can take to prvent repeating this problem:

• Checking databases for corrupted data often. If they see any errors react immediately to prevent the problem from spreading.
• If they have to shut down servers due to a such problem, they should have effective means of communication ready.
• In case of extended downtime, they should have services on a temporary server ready with some event ran by the volunteers who are unable to fix the problem anyway. (and their home server down)
• They should save your character data in a format that allows them quickly readding items to their databases.
• They should change your item restoration policy.

etc. as you see using such things can prevent problems as big as the current one.

 

[living]

rofl.. waste of time =)

 

[AngelHeal]

Star Wars Episode 4 is called "A New Hope", and Obi-Wan Kenobi is his given name (it is used in Episode 1), but after the Jedi were made extinct, Obi-Wan went underground on Tatooine and assumed the name "Ben" Kenobi and took to living in the desert as a hermit.

So, about those facts again, you were saying?

pfft, you should sign up for lotto weekend milionairs, u seem to have an answer on anything

 

[BimboBane]

Possible measures they can take to prvent repeating this problem:

• Checking databases for corrupted data often. If they see any errors react immediately to prevent the problem from spreading.
• If they have to shut down servers due to a such problem, they should have effective means of communication ready.
• In case of extended downtime, they should have services on a temporary server ready with some event ran by the volunteers who are unable to fix the problem anyway. (and their home server down)
• They should save your character data in a format that allows them quickly readding items to their databases.
• They should change your item restoration policy.

etc. as you see using such things can prevent problems as big as the current one.

With daily snapshots and transaction logs backup for the last 30 days (which is the common way to do this), this could have been avoided.
We have had rollback before and we will surely have rollbacks again. I can live with rollbacks, what I cant understand is how a db crash can take 9 days to solve and that so many of the customers actually thinks it is ok. (!)
No it is not ok, it is our money and our efforts to have a ball that they just blew into oblivion, and those steps Esselinithia mentions above is std issue 1A in a DBA manual. Databases is not "fire and forget" applications, a busy databases need constant monitoring and analysis.

//Johan

 

[semanon]

Please tell me you're fucking joking?

Please tell me you are joking. This is another GOA lead debacle and I think its right that they shoould be held accountable to it's paying customers. Damn if I ran a business the way they do I wouldn't have any customers.

It's about time the management of GOA stepped up to the plate and be accountable for the cock ups, rather than just taking our cash and screwing us over.

 

[Tesla Monkor]

While I hope that they fix the b0rked stuff fast, I doubt you're going to get anything more than a 'we're sorry you couldn't play'.

GOA's CoC states that they have to provide a service to you to play the game. It nowhere states that it has to be Prydwen that they have to give access to.

 

[Zede]

Mythic does NOT have same rescources as the likes of Blizzard & Sony. Give Mythic Xmillions to spent on what ever, im quite sure they could come up with some better systems.

Lets face it, this is a small mmorpg. Live with the fact we maybe will never get what we want, unless u want to up the subs to £20.00 a month to pay for everything.

Ive ran my own business for 5 years, and like mythic not at the uber rich side of the business sometimes shit happens, and the only thing that can bail you out ? MONEY !

 

[BimboBane]

I have put numerous backup sollutions for databases. If we assume they have a 0.5 TB DB they can for 2000 EURO get a tapestation that can handle that size. and buying 60 tapes at 40 EURO each is nothing compared to what kind of revenue they have from us playing on their servers.

So no it is not the cost..

//Johan

 

[Farbaute2]

blizzard and sony has showed that they cant make a lasting mmorpg with quality. It is not always about money. the way sony behaved against its costumers in swg killed that game, regardless how much dollars they pumped into it. daoc is still the best mmorpg, but mythic has to be careful not to fall in the same traps that sony/blizzard always seem to do. my bugs on my swg account are still not solved that i got after 2 weeks of playing when it started, why would i play that game when i have no character to play?

 

[Ballard]

I have put numerous backup sollutions for databases. If we assume they have a 0.5 TB DB they can for 2000 EURO get a tapestation that can handle that size. and buying 60 tapes at 40 EURO each is nothing compared to what kind of revenue they have from us playing on their servers.

So no it is not the cost..

//Johan

Yes okay Mr techincal expert. Where the fuck do you think the restore point came from? ermm a backup maybe?? There never has been a question of them taking backups...
The question is more based on the quality, existance and or reliability, in the case of corruption, of the recovery process. If the mythic supplied product does not produce adequate transaction logs (highly likely) then they will have to do what they did. Namely restore the database form the last backup and then manualy roll the database forward (i.e. add items and lost ML's etc..)

Everyone a expert arent they :touch:

 

[Esselinithia]

Ballard: Most sane developers don't redo a work that is already done, if you need a database, you don't write your own database solution, but use an existing one. Say, you connect to the databases with ODBC, and your game software can be configured to use different kinds of databases. A good part of making transaction logs, etc. are made by the database server they are using, if it is configured correctly. We have no reasons to assume it isn't the case.

But the problem begins when GOA can't see which records are good in the database and which ones are bad. Using the logs to fix the database is only possible, if they wouldn't just restore a bugous database. When people ended up with weapons in jewelry slot, items for wrong realm after they checked the database integrity for days, and failed to spot such bugous data, I have to think, that transaction logs can't help with a quick solution. The problem is the extent of this problem, and the fact they don't have measures to detect problems in the database.

The 66000 records they deleted isn't the 66000 bugous records, since if they seen SB items on a reaver as a valid record, not deleted it, then and SB misses his items, and the reaver has the same missing items problems. At the moment, I have reasons to think, noone knows how many items and how many characters are affected, the extent of the problem. But it means on far bigger problems: We don't know if there are any problems in database or if there are anybugs that can cause problems.

Losing title data, registering kills to wrong characters, etc. is a bug, that appeared after database corruption, and affects only prydwen, and mostly affects accounts already affected by the database corruption. On database side all they can notice: The record looks ok, the game can interpret it, but how can a hib have 33 kills against hibernian players recorded? Something is wrong. And we don't know what other problems might appear because of it.

 

[Ballard]

The point was if you didnt get it was... you do not know how there system is configured, you do not know what database or even if it is using popular database package, you do not know how or even if their system stores transaction logs. So do not bother providing 'expert' solutions to something you know nothing about... (note: im not referring to your post here, im just referring to the technical solutions that get posted here)

Yes the situation sucks balls, but please leave the technical solution to the poeple that are actually working on it.

I hope your joking about ODBC for all of our sakes.. hmm on the second hand that might explain the lag we have when teh server gets busy

 

[Azathrim]

Actually Mythic some years ago told, that the characters are stored in a flat file format of their own making.