[HELP] Big Company website testing

[WPKenny]

I was wondering if anyone out there worked for a firm reasonably large firm that has a significant web presence.

I work at a place that's got 350+ employees and one of the largest web presence in it's industry sector (i.e. top 3 in google natural rankings + paid for rankings).

It's all Apple Mac based. As in everyone has a mac desktop or laptop machine. It's a historical thing and something that's not likely to change in the next 5 - 10 years at least.

There's an heated discussion going on between the IT department and the Web team about testing our web site on virtual machines.
We run the VM's using Windows Server 2003 making full use of any hypervisor functionality. As far as we can tell this is near as damn it to having a physical box and it's a damn site easier to manage.

The web team maintain that it's not the same as having a physical box and have claimed mistakes have been made in the past due to visual differences on the VM's to actual people's machines but have not been able to provide proof of even one. When pushed they can barely tell us what version of IE it was let alone if it was XP or Vista.

What I'm looking for is something to go to them with and say "Look, Company X with a million employees and a billion hits a day tests their stuff using VM's and it's good enough for them."

Ideally we'd have a bank of machines with every possible configuration on but we just don't physically have the space. This doesn't stop them complaining.

Is anyone out there able to say with a degree of certainty AND PROOF that a VM running with all the hypervisor gubbins is a near perfect rendition of a physical box?

[TdC]

I work for such a company, but I must say I don't understand the problem: running in a VM or base hardware shouldn't make a difference to the user's perceived functionality, content or delivery. Ofc, you should test and develop against an OS / webserver that will reflect the production environment exactly, but hardware or not should make no difference. besides that the discussion is ghei imo.

basically I can't help because I can't tell you what you want to hear.

[WPKenny]

Originally Posted by TdC

I work for such a company, but I must say I don't understand the problem: running in a VM or base hardware shouldn't make a difference to the user's perceived functionality, content or delivery. Ofc, you should test and develop against an OS / webserver that will reflect the production environment exactly, but hardware or not should make no difference. besides that the discussion is ghei imo.

basically I can't help because I can't tell you what you want to hear.

pooh. Are you able to tell me how your company tests their website? i.e. physical machines of varying configs or loads of VM's on RDC?

[TdC]

well, we're huge, so what basically happens is a project gets set up. We have a bunch of architects mumble about sizing, hardware choice and flows of communication. We buy a shitload of machines and set up a DTAP environment. While all this is happening, I just heard from one of our WebSphere devs, the devs work on their workstations, presumably against VM's or against their private testing servers.

Once DTAP is in place and installed, the devs move their stuff from local testing to D, and perform what they call an integration test which means faffing about making things work. Once they have that they use the D->T->A flow to get stuff production ready for the the move to the holy grail that is P.

If it is my department that does any of that, and it probably is if we're having the interweb touch our machines (through a shitload of firewalls, load balancers, reverse proxies and whathaveyou), it will probably be on Sun stuff. There will be full compartmentalization of systems, eg webservers will have no content and be in an isolated network area, content servers will be elsewhere again isolated, database servers will be isolated, there will probably be WebSphere somewhere, LDAP (maybe), and possibly connections to things like HP Non-Stop or mainframes. But hey, security through complexity and all that

edit: oh, um, while our machines tend to be the same for a given task, we don't particularly test websites against varying hardware configs, because the hardware shouldn't influence the site if the underlying configurations are the same. As we roll out 10's or 100's of machines for a given project, we use a software distribution mechanism which always delivers the same, or a limited difference, outcome. Eg, all our webservers will always be the same, qua OS and services.

[Gahn]

Originally Posted by WPKenny

I was wondering if anyone out there worked for a firm reasonably large firm that has a significant web presence.

I work at a place that's got 350+ employees and one of the largest web presence in it's industry sector (i.e. top 3 in google natural rankings + paid for rankings).

It's all Apple Mac based. As in everyone has a mac desktop or laptop machine. It's a historical thing and something that's not likely to change in the next 5 - 10 years at least.

There's an heated discussion going on between the IT department and the Web team about testing our web site on virtual machines.
We run the VM's using Windows Server 2003 making full use of any hypervisor functionality. As far as we can tell this is near as damn it to having a physical box and it's a damn site easier to manage.

The web team maintain that it's not the same as having a physical box and have claimed mistakes have been made in the past due to visual differences on the VM's to actual people's machines but have not been able to provide proof of even one. When pushed they can barely tell us what version of IE it was let alone if it was XP or Vista.

What I'm looking for is something to go to them with and say "Look, Company X with a million employees and a billion hits a day tests their stuff using VM's and it's good enough for them."

Ideally we'd have a bank of machines with every possible configuration on but we just don't physically have the space. This doesn't stop them complaining.

Is anyone out there able to say with a degree of certainty AND PROOF that a VM running with all the hypervisor gubbins is a near perfect rendition of a physical box?

Bollox: i run Microsoft Crm 4 on Boxes and VMs without any difference at all.
You can only have problems if the Web App have to access data and the DB Server is virtualized as well.

[WPKenny]

Cheers guys.

It seems that as far as most people are concerned there is no difference.

I'm just left with trying to prove it to them.

Apparently they spent 3 days trying to fix an issue with "lines" in a text box when scrolling before they realised it was to do with refresh rate on the RDC connection. So trying to convince them that anything with the word "Virtual" in is anything less than "pretend" is proving quite a task and if they get their way it means a shite load more hassle for us.

[TdC]

ah, so they are looking at a browser on another box via RDC that is looking at the website? but then you're introducing a whole extra set of variables!

[WPKenny]

Originally Posted by TdC

ah, so they are looking at a browser on another box via RDC that is looking at the website? but then you're introducing a whole extra set of variables!

Yup. Right now they're using RDC (could just as easily be VNC) to connect to a VM running all the hypervisor doo-dahs. So in theory that VM is "as good as" a physical box. They dispute this and maintain there's a difference although haven't been able to come up with any proof, even so they're demanding physical machines. Not only is this a pain in the bum to organise there's a physical limitation on space for all these machines since you're looking at about 5 machines with monitors, mice and keyboards as a minimum. Space is already stretched, physical networking availability is also stretched. If we can convince them that VM are the way forward it does away with all this fecking about.

We're also working towards PCI compliance so managing X amount of physical windows machine on a network geared towards macs could prove to be a whole other problem.

[Vae]

If they only want them for testing websites can't they just have a few laptops on a wireless network conenction from their own desks?

[TdC]

while I agree with your VM statements I feel that the website must be tested directly. any extra communication layers may mess up your findings.

also, I must add that humans test only for usability and perceived performance over here. the rest is done by loading the site with stuff like loadrunner or whatever it's called again (I always forget)

[Dark Orb Choir]

isnt this what we have test labs for, we used to call them ring fenced environments, a duplicate of everythnig in a production environment just outside of the normal lan/wan........


imho virtualization isnt like the real thing, its like sex with a condom on

[Kryten]

I've heard some bollocks in my time but that's a funny one. Virtualisation is good enough that it's used more commonly than powerful single servers. As long as the VM's are based in the same place as the physical machines they would otherwise be replacing for these testing purposes (a side of it I have no experience in unfortunately) there would be *no conceivable difference* in performance, reliability or validity of the data gained or recorded.
Meaning, it sounds to me as if they're "afraid", so to speak, of the VM's. Would be interesting to get the input of some other companies on that though.

[WPKenny]

A heavy side to their testing is the appearance of the website. Buttons positions, does this text line up with that? etc. They maintain that the VM's skew pages to make them look "not as intended" even if functionality remains. Again, they cant provide proof of this.

[TdC]

in teh light of what Kryt said, I can find myself somewhat: I knew that visualization would make my job more of a hassle, and denied myself a lot of the good things that come along with it, preferring to focus more on -in my case- how many people I had to call to arrange a server with 20 VM's to be patched :/

[TdC]

Originally Posted by WPKenny

A heavy side to their testing is the appearance of the website. Buttons positions, does this text line up with that? etc. They maintain that the VM's skew pages to make them look "not as intended" even if functionality remains. Again, they cant provide proof of this.

they're full of shit. ask them when they last had a good poo would you?