Help Big Company website testing

[WPKenny]

I was wondering if anyone out there worked for a firm reasonably large firm that has a significant web presence.

I work at a place that's got 350+ employees and one of the largest web presence in it's industry sector (i.e. top 3 in google natural rankings + paid for rankings).

It's all Apple Mac based. As in everyone has a mac desktop or laptop machine. It's a historical thing and something that's not likely to change in the next 5 - 10 years at least.

There's an heated discussion going on between the IT department and the Web team about testing our web site on virtual machines.
We run the VM's using Windows Server 2003 making full use of any hypervisor functionality. As far as we can tell this is near as damn it to having a physical box and it's a damn site easier to manage.

The web team maintain that it's not the same as having a physical box and have claimed mistakes have been made in the past due to visual differences on the VM's to actual people's machines but have not been able to provide proof of even one. When pushed they can barely tell us what version of IE it was let alone if it was XP or Vista.

What I'm looking for is something to go to them with and say "Look, Company X with a million employees and a billion hits a day tests their stuff using VM's and it's good enough for them."

Ideally we'd have a bank of machines with every possible configuration on but we just don't physically have the space. This doesn't stop them complaining.

Is anyone out there able to say with a degree of certainty AND PROOF that a VM running with all the hypervisor gubbins is a near perfect rendition of a physical box?

 

[TdC]

I work for such a company, but I must say I don't understand the problem: running in a VM or base hardware shouldn't make a difference to the user's perceived functionality, content or delivery. Ofc, you should test and develop against an OS / webserver that will reflect the production environment exactly, but hardware or not should make no difference. besides that the discussion is ghei imo.

basically I can't help because I can't tell you what you want to hear.

 

[WPKenny]

I work for such a company, but I must say I don't understand the problem: running in a VM or base hardware shouldn't make a difference to the user's perceived functionality, content or delivery. Ofc, you should test and develop against an OS / webserver that will reflect the production environment exactly, but hardware or not should make no difference. besides that the discussion is ghei imo.

basically I can't help because I can't tell you what you want to hear.

pooh. Are you able to tell me how your company tests their website? i.e. physical machines of varying configs or loads of VM's on RDC?

 

[TdC]

well, we're huge, so what basically happens is a project gets set up. We have a bunch of architects mumble about sizing, hardware choice and flows of communication. We buy a shitload of machines and set up a DTAP environment. While all this is happening, I just heard from one of our WebSphere devs, the devs work on their workstations, presumably against VM's or against their private testing servers.

Once DTAP is in place and installed, the devs move their stuff from local testing to D, and perform what they call an integration test which means faffing about making things work. Once they have that they use the D->T->A flow to get stuff production ready for the the move to the holy grail that is P.

If it is my department that does any of that, and it probably is if we're having the interweb touch our machines (through a shitload of firewalls, load balancers, reverse proxies and whathaveyou), it will probably be on Sun stuff. There will be full compartmentalization of systems, eg webservers will have no content and be in an isolated network area, content servers will be elsewhere again isolated, database servers will be isolated, there will probably be WebSphere somewhere, LDAP (maybe), and possibly connections to things like HP Non-Stop or mainframes. But hey, security through complexity and all that

edit: oh, um, while our machines tend to be the same for a given task, we don't particularly test websites against varying hardware configs, because the hardware shouldn't influence the site if the underlying configurations are the same. As we roll out 10's or 100's of machines for a given project, we use a software distribution mechanism which always delivers the same, or a limited difference, outcome. Eg, all our webservers will always be the same, qua OS and services.

 

[Gahn]

I was wondering if anyone out there worked for a firm reasonably large firm that has a significant web presence.

I work at a place that's got 350+ employees and one of the largest web presence in it's industry sector (i.e. top 3 in google natural rankings + paid for rankings).

It's all Apple Mac based. As in everyone has a mac desktop or laptop machine. It's a historical thing and something that's not likely to change in the next 5 - 10 years at least.

There's an heated discussion going on between the IT department and the Web team about testing our web site on virtual machines.
We run the VM's using Windows Server 2003 making full use of any hypervisor functionality. As far as we can tell this is near as damn it to having a physical box and it's a damn site easier to manage.

The web team maintain that it's not the same as having a physical box and have claimed mistakes have been made in the past due to visual differences on the VM's to actual people's machines but have not been able to provide proof of even one. When pushed they can barely tell us what version of IE it was let alone if it was XP or Vista.

What I'm looking for is something to go to them with and say "Look, Company X with a million employees and a billion hits a day tests their stuff using VM's and it's good enough for them."

Ideally we'd have a bank of machines with every possible configuration on but we just don't physically have the space. This doesn't stop them complaining.

Is anyone out there able to say with a degree of certainty AND PROOF that a VM running with all the hypervisor gubbins is a near perfect rendition of a physical box?

Bollox: i run Microsoft Crm 4 on Boxes and VMs without any difference at all.
You can only have problems if the Web App have to access data and the DB Server is virtualized as well.

 

[WPKenny]

Cheers guys.

It seems that as far as most people are concerned there is no difference.

I'm just left with trying to prove it to them.

Apparently they spent 3 days trying to fix an issue with "lines" in a text box when scrolling before they realised it was to do with refresh rate on the RDC connection. So trying to convince them that anything with the word "Virtual" in is anything less than "pretend" is proving quite a task and if they get their way it means a shite load more hassle for us.

 

[TdC]

ah, so they are looking at a browser on another box via RDC that is looking at the website? but then you're introducing a whole extra set of variables!

 

[WPKenny]

ah, so they are looking at a browser on another box via RDC that is looking at the website? but then you're introducing a whole extra set of variables!

Yup. Right now they're using RDC (could just as easily be VNC) to connect to a VM running all the hypervisor doo-dahs. So in theory that VM is "as good as" a physical box. They dispute this and maintain there's a difference although haven't been able to come up with any proof, even so they're demanding physical machines. Not only is this a pain in the bum to organise there's a physical limitation on space for all these machines since you're looking at about 5 machines with monitors, mice and keyboards as a minimum. Space is already stretched, physical networking availability is also stretched. If we can convince them that VM are the way forward it does away with all this fecking about.

We're also working towards PCI compliance so managing X amount of physical windows machine on a network geared towards macs could prove to be a whole other problem.

 

[Vae]

If they only want them for testing websites can't they just have a few laptops on a wireless network conenction from their own desks?

 

[TdC]

while I agree with your VM statements I feel that the website must be tested directly. any extra communication layers may mess up your findings.

also, I must add that humans test only for usability and perceived performance over here. the rest is done by loading the site with stuff like loadrunner or whatever it's called again (I always forget)

 

[Dark Orb Choir]

isnt this what we have test labs for, we used to call them ring fenced environments, a duplicate of everythnig in a production environment just outside of the normal lan/wan........


imho virtualization isnt like the real thing, its like sex with a condom on

 

[Kryten]

I've heard some bollocks in my time but that's a funny one. Virtualisation is good enough that it's used more commonly than powerful single servers. As long as the VM's are based in the same place as the physical machines they would otherwise be replacing for these testing purposes (a side of it I have no experience in unfortunately) there would be *no conceivable difference* in performance, reliability or validity of the data gained or recorded.
Meaning, it sounds to me as if they're "afraid", so to speak, of the VM's. Would be interesting to get the input of some other companies on that though.

 

[WPKenny]

A heavy side to their testing is the appearance of the website. Buttons positions, does this text line up with that? etc. They maintain that the VM's skew pages to make them look "not as intended" even if functionality remains. Again, they cant provide proof of this.

 

[TdC]

in teh light of what Kryt said, I can find myself somewhat: I knew that visualization would make my job more of a hassle, and denied myself a lot of the good things that come along with it, preferring to focus more on -in my case- how many people I had to call to arrange a server with 20 VM's to be patched :/

 

[TdC]

A heavy side to their testing is the appearance of the website. Buttons positions, does this text line up with that? etc. They maintain that the VM's skew pages to make them look "not as intended" even if functionality remains. Again, they cant provide proof of this.

they're full of shit. ask them when they last had a good poo would you?

 

[Kryten]

VM's can run in a variety of guises - pretty much in the same way as terminal servers, all the way down to what you and I might use at home on VirtualBox or VMWare Workstation. The VM does what it's told, it sits at a particular resolution, it's no different to having different users using different size screens, resolutions etc etc etc. In fact it makes it more ideal, as it's easier to change things like resolution and screen size without physically doing anything.

 

[MYstIC G]

tell them to take screenshots of their shit with browsershots.org

 

[Chilly]

kenny: I work for probably one of the biggest uk online businesses and we do a load of testing via VMs and real boxes. Your dev team are spouting bullshit, I can assure you.

 

[DaGaffer]

If they only want them for testing websites can't they just have a few laptops on a wireless network conenction from their own desks?

That's what we do. Less hassle than getting IT (who by definition smell of wee, YOU ALL DO) involved.

 

[Ctuchik]

That's what we do. Less hassle than getting IT (who by definition smell of wee, YOU ALL DO) involved.

next you point them to this thread and tell em to read it

encourage them to register and post here aswell....

 

[TdC]

That's what we do. Less hassle than getting IT (who by definition smell of wee, YOU ALL DO) involved.

that's six kinds of wrong. a PROPER it dept would never allow such a thing. besides, a PROPER it dept would be clued into your wishes and have standard lappys available for you to break.

 

[DaGaffer]

that's six kinds of wrong. a PROPER it dept would never allow such a thing. besides, a PROPER it dept would be clued into your wishes and have standard lappys available for you to break.

heh. Satire.

 

[Dark Orb Choir]

pleae do not use the words Proper and IT in the same sentance, it makes baby Jesus cry, its like military intelligence etc IT DOES NOT COMPUTE

 

[TdC]

it pains me to think what manner of idiots you people are forced to consort with

 

[rynnor]

Interesting thread - if I may take a dissenting view I'd say that small and medium companies would use virtualisation but large companies would test against a host of different physical machines with different OS/browsers etc.

Testing using virtual machines is cheap and pretty effective but if you really want high quality testing you would use it hand in hand with real machines.

 

[Chilly]

if its just interface testing then there really is no case for physical boxes these days. I can imagine scenarios where the network has been poorly configured for the VMs and you might have problems loading css from a CDN or some such nonsense but that'snothing to do with the actual VM.

 

[yaruar]

vm's are the greatest thing ever invented for testing, being able to snapshot and clone and tinker and kill and rebuild as much as youo want to alter the testing environment is a godsend. I tend to throw up a vm every time i want to test an individual email application because i cba to install it and remove it to re set it up from scratch, much rather start from a virgin os every time.