Too much beta testing?

[GReaper]

I've got no idea how I stumbled on this yesterday, but I found a certain community managers ramblings on beta testing (see Too Much Beta Spoiled the Game), which mostly talks about the Gamasutra article on Tabula Rasa's beta testing. He can't comment on it due to various restrictions, but I'm sure we can!

I'm not talking about WAR specifically here, but I'd say it is one of the games which is taking the term beta testing a little bit too far. Instead of using beta testers to test the game, they seem to be designing the game around them. I'd say it's hardly surprising that beta testers get fatigued if you're inviting them far too early.

There's a balance between all things when designing, developing and marketing an MMO game. In the past most developers launched the game as soon as possible, hoping to patch in additional functionality later. After WoW came a different attitude, mostly because Blizzard weren't afraid to say that they'd only launch the game when they were ready. Now we seem to have developers trying to keep the game in development for as long as they feel is necessary, yet they're inviting people quite early on into the development of the game for testing.

Instead of relying on internal testers, developers seem to be increasing beta participation more than ever. Surely it should be the QA department which is hunting for the majority of bugs?

Are developers relying too heavily on testers to design the game?

 

[Ogrelin Blodig]

I'm no expert, but this is my guess.

In the beginning of the mmo history producers could release games with only the basic functions.

Nowdays they can't release that early, none would buy those games. A game must have all the functions the already existing games have and more... and that means ALOT of testing.

 

[Oskorei]

Well, I think it's logical they think and rethink about a game like this. It's a huge investment, and if done right, it will be the cashcow of the company. I don't have the feeling there's too much beta testing on WAR atm, I think they've got it about right.

For instance: at first they wanted to invent their own new silly RvR, but it were the beta testers who said "bahaaha, this lolsucks, gifv daoc rvr!!1", and then Mythic went for a (probably) enhanced version of Daoc's rvr system.

Beta testers help companies not make mistakes in gamedesign, so indeed, it's more than getting the bugs out: it's about making the game better in general.

 

[Aqe]

Only a very tiny % of all that will play war and even of those that want to beta test is in the beta yet so IF they would burn out a few of the beta testers it isn't a problem.

 

[Maasu]

As a software engineer, let me just stop you right there and say "you can never have enough testing".

 

[Night_Stalker]

As a software engineer, let me just stop you right there and say "you can never have enough testing".

The poster actuly points out not that too much testing was the failur but the wrong testers, too many of the buyers where testers wich ments they burnt out on alle the buggs and never wanted to buy the finalized game. Duke Nukem Forever was also a victim of too much testing, if it by any chance would ever have become a good game is a good question.
Bute surely to little testing and too many bug's are the majorety of problems, currently and i don't think it will change all that soon.

On WAR i also don't think they are overdoing it with the testers just take a look at FH and you'll see that far from all are in Beta.

 

[GReaper]

As a software engineer, let me just stop you right there and say "you can never have enough testing".

There's a huge difference between games and applications though.

Take Microsoft testing XP SP3 for example. This is a typical case where as many people testing the updates is beneficial before they release it to the public in the form of an official update. Bugs can be caught, other various bits of software can be tested with it. Stability is more important than rushing the update.

Now take a look at certain games. Some people just don't have that much long term interest in a game. Now if someone with only limited interest in a game plays the beta and decides that they don't have any interest in the final game, you could argue that it's a lost sale.

 

[AngelHeal]

testing? what testing? .. i havent got an invite yet? REQUIEL!!!!

 

[rynnor]

Its good to see they are reacting to beta testers concerns (the rvr re-design for instance) - as long as they still have a strong idea of what they want the game to be like it should work well.

I dont think we can judge from outside how much testing is required but I can tell you from personal experience of large projects that the testers/developers will be under a lot of pressure to get the game into a retail state - they are extremely unlikely to 'over test' it

 

[Chrismorris]

Only thing wrong with BETA is im not in it.

Just got back from Hols and nothing

 

[Tallen]

Its good to see they are reacting to beta testers concerns (the rvr re-design for instance) - as long as they still have a strong idea of what they want the game to be like it should work well.

I dont think we can judge from outside how much testing is required but I can tell you from personal experience of large projects that the testers/developers will be under a lot of pressure to get the game into a retail state - they are extremely unlikely to 'over test' it

Cant argue with that, but it is an interesting point about overtesting and how dev's value player feedback and how much they take this into consideration when making significant changes to the structure of the game.

There is a real pitfall if a dev team listens too much to their beta crowd, changes the game accordingly and then everyone wonders why it's hopelessly crap come release.

Beta selectees should be there to report on bugs, stress test this and that and give feedback on in-game content like quests etc. Thats all.

 

[Night_Stalker]

There is a real pitfall if a dev team listens too much to their beta crowd, changes the game accordingly and then everyone wonders why it's hopelessly crap come release.

What do you meen with by listnign too much to beta testers? I think its very advisable to listen to beta testers, when they say that a certain class is feeling too much like a healer, or just not living up to what they exspected when they looked at it at first sight. Or when they address balance issues, i think a big part of beta testing also involves in testing if the wanted feeling of the game is the right one.
Fx Chaos havin pink hair stile options, or spells named "embrace of healing lovytooty", wich doesn't give the player the dark feel that chaos should have.

 

[Flimgoblin]

I think the problem with the tabula rasa beta is that they invited people in, then didn't change anything until right before release.

Sounds like they didn't sound enough like they were listening, and didn't communicate that they'd changed things well enough.

That said there's a point to be made about not inviting too many people into beta until you're ready for that number of people - if 5000 people can test your game and give you feedback, but having 10000 people just gives you twice as many reports on the same issues - you probably don't need the extra 5k yet


Also on the listening point - there's a difference between:
1. Listening to what your players are saying, understanding their problems and then amending the game accordingly.
and
2. Doing what the players ask you to do.

 

[Tallen]

What do you meen with by listnign too much to beta testers? I think its very advisable to listen to beta testers, when they say that a certain class is feeling too much like a healer, or just not living up to what they exspected when they looked at it at first sight. Or when they address balance issues, i think a big part of beta testing also involves in testing if the wanted feeling of the game is the right one.
Fx Chaos havin pink hair stile options, or spells named "embrace of healing lovytooty", wich doesn't give the player the dark feel that chaos should have.

Listen to them, yes, but devs have to be careful not to put too much stock in what the beta testers say when it comes to major game changes. The game design ultimately should and usually is determined by the development team, changing the way the game is designed according to a few players who have only played a few days of the lower levels in the beta can be a very risky gamble.

Beta testers are usually hardcore players or hardcore fans who have been following the game since inception, they have been waiting for the beta sometimes for several years and so have firm opinions on what they think it SHOULD be like. Their opinions often will not necessarily be anything like those of the wider gaming public who are the ones who will ultimately buy and make this game a success.

 

[Tallen]

Also on the listening point - there's a difference between:
1. Listening to what your players are saying, understanding their problems and then amending the game accordingly.
and
2. Doing what the players ask you to do.

In a nutshell

 

[Kremlik]

Just to flip it a bit with the situation with TR, it's main and biggest problem was the fact of who they invited. They didn't invite fans or players interested in the game design they invited anyone via limited keys on things like fileplanet and eurogamer.

Even during the supposive closed beta we had the now standard 'this isn't WoW' players running about, they had no plans of playing the game for the game on release, it's a f2p game for them, and those are the players the devs listened to, so they turned from the 'pvp' style in the summer (hight of the 'closed beta'), with the CPs planned for player control as well as PVE control to the 'endgame raid' style by the new year (open beta) and ajusted the content throughout to match.

So with all this flip flopping no other content was done, so when the big wall of must release happened those wanting the 'pve raid TR' had dissapeared once money was required leaving the 'pvp TR' that we had in the summer with a game they didn't want and then the biggest crack in the armour happened when this 'blame' popped up and then refusd to change the game back as 'this is what we wanted'.

Point being is that it's all well and good listing to feedback during beta as long as your 100% sure that most of the feedback in from potenal players in the future not just some punks that just want a new toy to play with, hence why I beleave WAR's beta is being handled very well they've been inviting actal fans via the contests (well if they aren't why bother with the writing/arkwork) plus former players of their other product (seeing as it's kinda the same style of product, logical there) and not do what TR did and open the door to gods knows anyone.

PS. Hi I'm a lurker poping his head up and last :touch:

 

[Lexa]

Not a bad post from a lurker, you should lurk less and post more

 

[rynnor]

I think the biggest danger from listening to beta players is a tendency to ask for everything they had in other games regardless of whether its appropriate to the game.

I'm personally happy to see keeps in the rvr zones now but I do hope WAR doesnt just become Daoc 2 - I'd like it to have its own distinctive feel and I dont just mean graphically

 

[GReaper]

Just to flip it a bit with the situation with TR, it's main and biggest problem was the fact of who they invited. They didn't invite fans or players interested in the game design they invited anyone via limited keys on things like fileplanet and eurogamer.

Probably explains why an even larger than usual number of people didn't bother with playing after beta. I'd guess a higher percentage of people signing up for a beta through the games own site are likely to be more loyal than a typical fileplanet user after a demo to keep them entertained.

Fileplanet is great for getting your game demo out to a wide audience, but I'd avoid it for beta testing!

 

[Kremlik]

I'd agree with you there Reaper, theres been many a MMO that have used Fileplanet to obtain closed beta members and it's never panned out well, in fact just looking at my beta key list theres only 2 westen MMOs that still are running well, just further proves that with MMOs are have get the right kind of players involed in the testing otherwise you could design anything.

TBH the way the console/pc game industry has been going these past few years long gameplay games are a rarity, so gamers loyity to MMOs would be effected by this IMO, console games last players what a few weeks (in the case of the 'hardcore' at tops days)? so in the case of MMOS are the primarally console players expecting to 'win' a MMO in a few months now?

I think MMOs really need that hook now to keep the xbox/playstation gen to keep playing it throughout the year, in the case of WoW's hook that greed which is a bad 'hook' but sadly that pays off, I'm really hoping relm pride rubs off onto them within RVR as it's a damn good hook and morely sound as well

 

[Popov]

The guys an idiot IMO - I beta tested Tabula Rasa and it released way too early so clearly had too little beta testing not too much.

 

[Riddler]

Have been betatesting several games including DAoC, WoW and now WAR. It feels like this beta is being handled the same way as the two former games and we all know both those games had/have great success. Wouldn't be worrying for WAR anyhow!

 

[Pandemic]

As a software engineer, let me just stop you right there and say "you can never have enough testing".

As a current senior software test engineer an ex although very brief warhammer online test manager (climax) i agree with this 100%. Mythic stopped beta testing because beta testers had identified issues mainly with the rvr and having now hopefully sorted those issues have just reopened beta testing, so starting this again so this is in no way to much testing

 

[SonicBorg]

One of the biggest problems for the DEV team has to be finding "good beta testers", not just people who simply want to have access to the game, and offer no feedback/pointless feedback.

Quality > Quantity so to speak, will give a much better effect for beta testing and provide a nicer retail game for the masses. A quick stress test before retail will ofc be useful. (free-for-all beta for a short period of time) to check on the core stability.

WAR has to take into account, mmo gaming features that most of us are used to/expect, and the lore of Warhammer that the TT players are used to, and expect from the game. Its a hard mix, and i'm sure you get many whines in beta atm from both angles.
(make it a better mmo game by adding this idea..... / or / that doesnt follow warhammer lore, change it to fit the bill)

 

[wutae]

There's a huge difference between games and applications though.

Take Microsoft testing XP SP3 for example. This is a typical case where as many people testing the updates is beneficial before they release it to the public in the form of an official update. Bugs can be caught, other various bits of software can be tested with it. Stability is more important than rushing the update.

Now take a look at certain games. Some people just don't have that much long term interest in a game. Now if someone with only limited interest in a game plays the beta and decides that they don't have any interest in the final game, you could argue that it's a lost sale.

The basic question is : If beta testing at the current state takes that long, after 5 years to other games , the beta testing will last for 1 year+ ? : O It's really frustating to wait for this long. Still Greaper, i think that the NDA helps for this things to not happen, worst case scenario they are going to lose some sales . which is the minority of the sales, moreover even if NDA exists, i think many people propably has given details to their IRL friends, convincing them to play it at the release.

P.S Saitoh seems unintrested!! make him play it ffs ! Tell him that AoC SUX !

 

[Boze]

I'd rather have a game fully functionally, and go patching and improve from there. It's worth the wait IMO.

<edit>I have another problem which I'll ask about on the beta forums rather than on a public forum because I really shouldn't be asking questions which may break the NDA here.</edit>


Edited by IainC for sanity.