cockups

[k9awya]

i was doing charlton athletic footy clubs dinner invites and they were meant to be in silver, i did them in gold.

leet.

 

[nath]

£20k for a new motor
£7k for a new brake
£85k to get the twisted chassis fixed
£50 for a night on the piss for the 'witnesses' to help them forget what they'd seen (we blamed it on the yanks).

That's gunna take some beating, tbh.

 

[Munkey]

I told the CIA there were Weapons of Mass Destruction in Iraq

 

[00dave]

My former bosses made a colossal cockup once. Working at a leisure centre one of the attractions was one of those laser shooting games. (pain in the arse to fix those things) the bosses who inherrited the business from their father decided in all their wisdom to send 1000 (yes 1 thousand) free game tickets to the local schools to coincide with half term. They then wondered why for the next month they recieved about £10 a day from the site that could pull in £500 a day over half term, but were still having to pay 12 hours worth of staff wages, wear and tear, and electricity costs. What's worse was that the tickets were printed on white paper about the size of a credit card with simple black font like this and not lamenated so as you can imagine more than 1000 were taken in.

Me myself I never make cockups at work, at least none I'm going to tell you lot about. After wednesday I'll be working for the queen/government/MOD so let's hope I don't make too many more.

 

[Cyfr]

Oh.. and on work experience I made coffee with 2 sugers instead of 0!

 

[kanonfodda]

SoWat wins

When I worked for a software supplier (we supplied software to dentists), I was tasked with sending out pricing updates for NHS patients (cost of filling etc). I had to update all the prices manually, only I got a few wrong. Twice.

We had to send out 300 disks, 3 times, all labeled and sealed in jiffy bags manually

I was a temp at the time, after that I got a permanent post :s

 

[Whipped]

Sent a print command I couldn;t revoke and accidentally printed an entire data base. A bad day for trees.

Students here keep doing that. If you try to print a single page from our online timetabling system, it has a nasty habit of churning out all 900 pages at a cost of 5p a page. Or 50p if the person is unlucky enough to have selected the colour printer

Oh, and best one i've come across in my short life - cleaner unplugged our main router from the mains so she could use the vacum cleaner.... and didn't plug it back in. Think the boss had a word, since she enver did it again.

We get a smiilar thing all the time in our student union. The router for the building is in a small cabinet in one of the offices. It's plugged in to a wall socket that also has a PC plugged into it. A couple of months ago someone brought a kettle in (I think you can see where this is going). At least once a month we are flooded with phone calls from the union telling us the network has died on them. I then have to wonder over and unplug the kettle to plug the router back in

 

[granny]

Whats the biggest cockup you ever made

Got married.

Whats the biggest cockup you ever made at work

Oh, OK, hmm let me think... so many to choose from Oo. I think the most annoying was when we were doing an angiogenesis (new blood vessel growth) assay which involved a very expensive kit which basically took 10 days to get some human cells to grow into tube-like structures. We wanted to treat one set of them with something called VEGF (a growth factor) and the VEGF needed to be made into solution in something else called PBS (phosphate buffered saline). Rather than making up the PBS fresh (a 10 minute job) I found an old (5 years old) bottle in the fridge and used that. Of course it was infected by this time and the cells all got nasty bacteria all over them and it ruined the entire experiment. Sigh.

 

[rynnor]

The worst I ever did was to cause a very important program to crashout for about 12 hours one night - I personally blame the pillock who was covering the live system that night though (not me btw).

It pales in comparison to my mates, one of whom lost the Bank £90,000 one night and another who introduced a bug at Friday Lunchtime before a bank holiday that captured all the cards of a particular Bank when they were put into one of our ATM's - we realised pretty quick and took it out within an hour but by then nearly 1500 cards had been captured - our bank had to apologise to the other bank and a whole special incident team had to be set up to handle it plus the huge number of extremely angry customer calls...

Nice one

 

[yaruar]

Though I've made my fair share I can't quite work out how or which ones I can safely tell out of my own long list. However I do remember a classic one back at uni, I will call the culprit 'Student A'

• Picture one experimental research lab.
  
• Watch 3 doctorates working for 3 months building a new ultra sensitive measuring device.
  
• Watch 'Student A' carry said device across lab to install into experiment.
  
• Be amazed as £100,000 worth of kit smashes onto the floor.
  
• See 'Student A' assisting in the 1500 man hours to build new unit.

At my uni one of the phd research students blew our mass spectrometer through the lab wall by accidently creating nitroglyricin in it one evening by forgetting to clean it from previous experiments..

As for my own moments of wanting to rewind time.

Whilst working in support on my first job I was supporting a 60 million pound SAP development project being done jointly with AC..

Whilst upgrading all the office pc's, a series of unfortunate things happened and I managed to lose all the Lead consultants work from the previous year.. With no backups....
Then, in my last job before this, I flagged a space issue on our exchange server to my boss. There were 100gb+ of log transaction files. So my boss deletes the files. But didn't check and also deleted the mail databases..

Then he paniced and asked me for the backups and I pointed out there never had been a backup....

5 years of email down the pan. After 4 months of different data recovery techniques we had nothing.

That was the most horrible day of my life!

 

[Clown]

I called my teacher 'mum'

 

[Whipped]

I called our lolipop lady mum. Oh wait! She was my mum

 

[SoWat]

5 years of email down the pan. After 4 months of different data recovery techniques we had nothing.

That was the most horrible day of my life!

I got a panic attack just reading that!

 

[mank!]

Some years ago, our American developer team refused to put a 'full speed' test mode onto one of our million-pound industrial mailers.

As I was on the developer list for the source code I decided to write it myself (I'd always fancied having a crack at controlling hardware via 8051 and Parallel Interface boards).

Only thing was, I misjudged the brake 'tap time' (which is what I wanted to test), and ended up fully applying the brake... at the same time as the motor was running at full speed.

£20k for a new motor
£7k for a new brake
£85k to get the twisted chassis fixed
£50 for a night on the piss for the 'witnesses' to help them forget what they'd seen (we blamed it on the yanks).

I presume I'm the only one who didn't understand this?

 

[Tom]

I reckon so mank!

Sounds like hes talking about a big conveyor belt type mail machine.

 

[RandomBastard]

I imagine it to be a large automated mail out and frank and sort machine. So people dont have to put mass mail outs (like telephone bills) in envelopes just prints straight off and gets packected up.

 

[Gumbo]

On the subject of printing things wrongly, our slightly strange network covers a load of branches, and if you want to print something from a certain area of the system you have to change the printer to the one in your branch before hitting print.

Well of course people often forget this step, and just hit print causing a page to appear in the computer room at head office.

This was until the default printer got changed to the printer in the Financial Directors office, and suffice to say that she has a rather fearsome reputation, and your username appears at the top of all prints.

Apparently these accidental printing mishaps have declined markedly since the change was introduced.

As far as mistakes I have made go, well in my line of work a 25% error rate is considered normal, and I'm currently running at about 17%, so I'm happy.

 

[SoWat]

Allow me to expand your minds

The machine was not a conveyor type thing, it was a chain driven monster (about 70feet long). It had lots of hoppers for inserts and stuff.

The system was controlled via a PC, with commands given to output devices (solenoids, motors, etc) via a 32 port serial card. The signals come out of the com ports, through an 805x board, and then to a Parallel interface board which actually sends the signals to the components. (it works the other way too, with inputs such as photocells sending information on a reverse route).

I wanted to test what would happen if the machine 'jerked' at high speed. I intended to do this by sending a 'tap' signal to the brake (i.e. apply the brake for a fraction of a second. Unfortunately I failed to take heed of the fact that anything over 30msecs was a complete revolution of the encoder... and I had set the 'tap' to 50msecs. This meant, in effect, that the brake stayed on. Under normal circumstances the clutch is disengaged prior to any braking, unfortunately I had the thing going at full pelt (13,000 cycles per hour)... cue severely wounded machine.

 

[Doh_boy]

I bet it sounded as if the world was going to end, when it broke!

This best one I can think of is this:

I worked a company who made testing software for WAP (internet over the phones) and one such tool was a load tester. It was pretty much a DOS program which can be used to emulate users on a company's server (wap gateway). One day as I was testing it at 'unusually high loads' I managed to send the 10,000 user requests (at the same time) to teletext's wap site! Usually we sent it internally to another programe called the 'responder'. I had it going for 15 mins until the sys admin came up and asked me what the fuck did I think I was doing! :]

Everyone wondered why the interent went off for half an hour!

 

[Zephirus]

the best thing i can think of is having to explain to a rather panicky it guy that the flashing yellow led's on his server farm's 20 odd servers weren't actually a) flashing or b) led's. They were actually the yellow coloured reset button's.

having to clear up the results of someone killing syncd on a server that subsequently crashed slightly later, resulting in metadata loss for all the servers filesystems. (all the files had no time/date stamps).. oracle had a bit of a job dealing with that

As for mistakes by me.. I once took a whole county out for over 48 hours, because a colleague got confused with areas other than that, just generally minimal stuff, like yanking disks out of live servers and the like.

 

[EvilMonkeh]

Not at work, but at school. I discovered our network allowed winpopup. I loged on to two pc's using my login, (ofc you can't send messages to yourself, so I sent it to the workgroup). I then showed a friend who proceeded to do lots of naughty stuff which got sent to all the pc's in the school. I remember my head saying to me 'No Andrew, I don't need a shit''

hehehe
ive accidently done that before at school and the whole school (200 pcs) got the msg "arse" but luckily i was on someone elses computer

 

[Delboy]

Got up late for a 5am shift, was paged by a desperate user on the way to work. And to top it off, later that weewk I missed a couple of servers that failed their backups.

 

[Xavier]

I've only just spotted this thread so I'll contribute my mishaps albeit a little later than the majority.

Prior to my journalistic shenanigans, I worked as an ASP/MSSQL developer and server administrator for a major development house in the mobile industry. Most of my time was spent writing content management code, or front end UI's to a handful of our bespoke apps.

One of those applications was an OTA configurator - a program which sent out SMS messages to configure your mobile phone for wap access, email, photo messaging etc. and the hooks into the app were made via a SQL database. After capturing the users mobile number and type of OTA configuration they wanted (determined by device and network) a row was written to a table on the SQL server and a stored procedure executed to send the queued texts.

All was fine and dandy until one day we were approached to produce a batch processing application, which was to be fed a set of numbers and phone types and configure them in a single hit. I don't know if it was down to low caffeine levels or what, but I got a loop somewhere in my stored procedure very wrong and send a sustained burst of texts to my personal mobile. Because we'd just shifted from using a cellular engine to send our own SMS to using a third party service, the SQL server had a direct IP connection to the remote box and queued up a good 15k messages before I was able to intervene. Thankfully we got a very good price on bulk messaging but the goof still ended up quite pricey.

Orange weren't chuffed either, they had to get a pair of engineers to work over the weekend to flush the DoS style attack that I'd dispatched against my own number... heh.




Second noteworthy goof, whiile working at the same place... Working in the server room for once my project manager trundled in and asked if I'd look at one of the webservers which had slowed down to a crawl. I logged in over terminal services and in the end opted to reboot it so that I could have a proper look at event manager, however W2K server went squiffy, so I asked my PM to cycle the power on the machine as he was closer. Having just moved the servers around in the rack he got a bit confused and rather than ask me which machine it was (we'd named them all after Pokemon and I don't think he'd quite caught up...) he hit the power button closest to him, which turned out to be that of our core UPS, taking the entire production environment and some 40 high-traffic websites totally offline.

Xav

 

[TdC]

um, iirc I've made exactly one mistake at workies:

a certain project had the insane and stupid wish to have their three servers configured to a certain non-standard patch level. they managed to get 'da man' to agree to forbid me and mine to upgrade/patch the machines beyond the specifications set by whoever made their app, thus effectively forbidding us to admin properly.
normally we'd fight this tooth and nail but as these machines are 'financially delicate' we had to give in to the wishes of the unwashed ("give in" as in da man calling me personally to tell me to do the install or I would be amazed to learn how much pain a human can endure before I'd be fired). in the book 'hunt for the red october' by clancy, a certain scene has an engineer and a captain being told by their political officer that it would be 'politically unsound' to stop the submarine to perform vital checks on the running reactor. I felt the way I imagine the engineer and captain felt.

anyway, to create the non-standardly patched machines we had to do everything by hand which is shit. my personal machine was nearly done, with 3 20hr days under my belt at 3am on a sunday morning I decided that I only had one patch more to install. I did it the "quick" way, the dangerous way, the way that I'd never do, the way that would earn a minion a severe talking to and coffee-monkey rights for a year, the way that can't be backed out in case something went wrong. well...I wasn't pleased to see that I'd selected the incorrect patch. wasn't pleased to find that it really is impossible to back out a kernel patch from a running kernel that has been installed the quick and evil way, and I was most most unhappy at having to call everyone at 4am sunday morning to tell them I'd fubar'd the machine. d'OH!!

 

[RandomBastard]

TdC : doh eh? I don't remember that scene in hfro, are you sure you dont mean some other book? as the political officer on red octorber (krazny oktabre or sumit in russkie) was murdered in the story very early on i believe, and the reactor checks were fake to get the crew off no?

 

[Tom]

I think hes referring to Titanic, where the representative of White Star urges the Captain to proceed with all boilers running full steam.

 

[TdC]

hehe no, I mean the scene in the book aboard the ES Petrosomething just prior to it's cold-water incident. doesn't occur in the film iirc.

 

[DaGaffer]

Not one of my own (thank God), but the most expensive mistake I've ever seen was years ago when I worked at British Aerospace; the very first prototype airframe for the Sea Harrier FRS2 (the one the Navy upgraded to after the Falklands) was on the back of a low loader being brought from Kingston to Brough (near Hull, where I worked). I happened to be walking past as the lorry driver delivering the plane was reversing into the 'shed' where the plane's to be delivered. Unfortunately, he managed to miss the entrance as he's reversing, and stuffed over a million quid's worth of airframe into a wall, resulting in a Harrier two feet shorter than it was supposed to be...

 

[ECA]

a sovereign is a gold coin worth 45-65 quid on average depending on type/date/condition.

 

[old.Osy]

Happened just two days ago

The pc holding the cash-in software went slow and crashed a lot, so I had to go check it up. Decided it was clogged beyond repair, so I used the backup image to create a fresh environment. Was tired, and I forgot about the database

Had to endure good 15 minutes of boss yelling at me. A good 2 years of data gone. Well I had a back-up, but it was dated 23-10-2003. Which doesn't make up for it.