Jan 4th Mass Link Deads

[bob007]

A word from our sponsors.

Is this a goa problem.

I run a tracert after each ld an its clear, 20 - 40 ms, 15 - 17 hops

Is there a problem at Wanndo we should know about ?

 

[cHodAX]

Last 4 days have been unplayable, are GOA going to refund those days? At £150,000 in subs every month they need to get thier act together and fix this damn thing once and for all!!

 

[hellraisermk2]

Didn't you know? It's this patch!!! That's right, in America they had the same lag and it mysteriously dissapeared next patch!!!111

Talk about utter bullshit. If it was the patch causing the lag, then why the feck didn't someone find out why and correct it, or at least go straight to 1.54 (God forbid that we should actually see it).

It's absolute rubbish. Goa/ Wanadoo/ Whoever, get your act together. A few thousand other people and myself don't want to pay for a pathetic service where even small scale RvR causes two thirds of the group to completely LD, and the other members to lag out in different directions.

 

[bob007]

Last few days, theres been some router problems, not down to GoA. This is just about tonight. There no router problem yet fgs all over excal are link deading for nothing.

 

[SilverHood]

goa should spend their money on getting us top routers rather than the crap 3rd world junk we're currently using. Apaprently.

 

[old.Glendower]

ha!

It is two fukking soup cans with string running between them

 

[Whoodoo_RD]

Re: ha!

Originally posted by old.Glendower
It is two fukking soup cans with string running between them

Dont forget the 48K Sinclair Spectrum router machines they use.

 

[bob007]

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Ian James>tracert 193.252.123.33

Tracing route to 193.252.123.33 over a maximum of 30 hops

1 11 ms 10 ms 10 ms 10.46.0.1
2 12 ms 17 ms 20 ms bagu-t2cam1-a-ge-wan34-106.inet.ntl.com [80.5.16
2.25]
3 17 ms 22 ms 26 ms bagu-t2cam1-b-v99.inet.ntl.com [80.5.163.10]
4 21 ms 20 ms 22 ms man-t2core-b-ge-wan63.inet.ntl.com [213.104.242.
173]
5 12 ms 25 ms 11 ms man-bb-b-so-210-0.inet.ntl.com [62.253.184.61]
6 21 ms 19 ms 47 ms 213.206.159.245
7 23 ms 25 ms 27 ms sl-bb20-lon-11-0.sprintlink.net [213.206.128.56]

8 27 ms 34 ms 25 ms sl-bb21-par-14-0.sprintlink.net [213.206.129.70]

9 27 ms 37 ms 33 ms sl-gw10-par-15-0.sprintlink.net [217.118.224.42]

10 30 ms 28 ms 30 ms sle-franc1-1-0.sprintlink.net [213.206.131.42]
11 30 ms 27 ms 29 ms P8-0.PASBB1.Pastourelle.opentransit.net [193.251
.240.101]
12 27 ms 48 ms 36 ms P4-0.PASCR3.Pastourelle.opentransit.net [193.251
.241.162]
13 45 ms 41 ms 51 ms P12-0.BAGCR1.Bagnolet.opentransit.net [193.251.2
41.130]
14 130 ms 111 ms 68 ms P3-0.BAGCR3.Bagnolet.opentransit.net [193.251.24
1.110]
15 84 ms 123 ms 85 ms P1-0.BAGBB1.Bagnolet.opentransit.net [193.251.24
1.145]
16 131 ms 125 ms 58 ms WanadooPortails1.GW.opentransit.net [193.251.251
.58]
17 37 ms 84 ms 33 ms 193.252.122.10
18 * * * Request timed out.
19 * 30 ms 28 ms 193.252.123.33

Trace complete.



Tracing route to 193.252.123.33 over a maximum of 30 hops

1 10 ms 12 ms 14 ms 10.46.0.1
2 20 ms 19 ms 12 ms bagu-t2cam1-a-ge-wan34-106.inet.ntl.com [80.5.16
2.25]
3 12 ms 12 ms 13 ms bagu-t2cam1-b-v99.inet.ntl.com [80.5.163.10]
4 18 ms 14 ms 17 ms man-t2core-b-ge-wan63.inet.ntl.com [213.104.242.
173]
5 12 ms 13 ms 12 ms man-bb-b-so-210-0.inet.ntl.com [62.253.184.61]
6 19 ms 20 ms 21 ms 213.206.159.245
7 41 ms 21 ms 19 ms sl-bb20-lon-11-0.sprintlink.net [213.206.128.56]

8 * 26 ms 30 ms sl-bb21-par-14-0.sprintlink.net [213.206.129.70]

9 32 ms 27 ms 31 ms sl-gw10-par-15-0.sprintlink.net [217.118.224.42]

10 42 ms 26 ms 42 ms sle-franc1-1-0.sprintlink.net [213.206.131.42]
11 30 ms 32 ms 28 ms P13-0.PASCR2.Pastourelle.opentransit.net [193.25
1.241.169]
12 29 ms 28 ms 28 ms P11-0.PASCR1.Pastourelle.opentransit.net [193.25
1.241.97]
13 72 ms 39 ms 37 ms P4-0.BAGCR3.Bagnolet.opentransit.net [193.251.24
1.118]
14 26 ms 29 ms 26 ms P1-0.BAGBB1.Bagnolet.opentransit.net [193.251.24
1.145]
15 35 ms 32 ms 26 ms WanadooPortails1.GW.opentransit.net [193.251.251
.58]
16 29 ms 34 ms 29 ms 193.252.122.10
17 30 ms 29 ms 28 ms 193.252.123.33

_______________________________________
above are 2 tracert's

First is directly after a ld on sunday the 5th.

2nd is 1 min after


Am sure this shows a server problem rather then a routing problem, Please anything from GoA.

 

[old.Glendower]

good show

Nice job on providing some proof!

Thanks for posting.

 

[old.windforce]

arghhhh nerf routing


[212.142.28.122]
4 42 ms 32 ms 20 ms srp8-0.ah00rt02.brain.upc.nl [212.142.32.50]
5 37 ms 36 ms 30 ms srp0-0.am00rt01.brain.upc.nl [212.142.32.1]
6 37 ms 43 ms 36 ms srp0-0.am00rt06.brain.upc.nl [212.142.32.44]
7 45 ms 38 ms 33 ms nl-ams01a-rd1-pos-3-0.aorta.net [213.46.161.53]

8 111 ms 111 ms 107 ms us-nyc01a-rd1-pos-0-0.aorta.net [213.46.160.194]

9 530 ms 543 ms 559 ms gige3-1.ipcolo2.NewYork1.Level3.net [64.158.176.
37]
10 542 ms 548 ms 539 ms gigabitethernet2-1.core2.NewYork1.Level3.net [64
159.17.70]
11 551 ms 598 ms 547 ms NYKBB4.New-york.opentransit.net [209.244.160.146

12 540 ms 535 ms 532 ms P2-0.NYKCR2.New-york.opentransit.net [193.251.24
.233]
13 643 ms 625 ms 657 ms P4-0.PASCR1.Pastourelle.opentransit.net [193.251
241.133]
14 615 ms 657 ms 617 ms P4-0.BAGCR3.Bagnolet.opentransit.net [193.251.24
.118]
15 618 ms 624 ms 622 ms P1-0.BAGBB1.Bagnolet.opentransit.net [193.251.24
.145]
16 619 ms 630 ms * WanadooPortails1.GW.opentransit.net [193.251.251
58]
17 605 ms 619 ms 612 ms 193.252.122.10
18 620 ms 610 ms 613 ms 193.252.123.13


being routed to america and back sucks

 

[-Nxs-]

Originally posted by SilverHood
goa should spend their money on getting us top routers rather than the crap 3rd world junk we're currently using. Apaprently.

Maybe you should moan at your ISP and not GOA, I've been playing all week with little if any lag, i have not gone LD once this week.

Its NOT a problem with GOA's routers, ISP

 

[cHodAX]

Originally posted by -Nxs-


Maybe you should moan at your ISP and not GOA, I've been playing all week with little if any lag, i have not gone LD once this week.

Its NOT a problem with GOA's routers, ISP

Then WTF are people in the UK, Germany, Sweden, Finland and The Netherlands all reporting the EXACT same problem??

 

[ghoul]

Originally posted by -Nxs-


Maybe you should moan at your ISP and not GOA, I've been playing all week with little if any lag, i have not gone LD once this week.

Its NOT a problem with GOA's routers, ISP

There is to many people on different ISP's going LD.
Blaming it on ISP's is wrong.

 

[Silent Shadow]

It's GOA's servers for sure...

 

[-Nxs-]

Lol just get yaself on a propper ISP

Im STILL playing happily all weekend and NO linkdeads not even one... and very little if any lag - performance counter for ping and packet loss is most of the time green.

and no im not on a special line, its just a standard ADSL home package with nildram, when i was on BT i was going LD at least twice a week, now ive moved to a half decent ISP there are no problems at all, maybe my ISP has more tolerant equipment ? who knows, just stating a fact that my connection is fine

 

[Whoodoo_RD]

Originally posted by -Nxs-
Lol just get yaself on a propper ISP

Im STILL playing happily all weekend and NO linkdeads not even one... and very little if any lag - performance counter for ping and packet loss is most of the time green.

and no im not on a special line, its just a standard ADSL home package with nildram, when i was on BT i was going LD at least twice a week, now ive moved to a half decent ISP there are no problems at all, maybe my ISP has more tolerant equipment ? who knows, just stating a fact that my connection is fine

Are you illiterate???? Evidence from the traces proves the problem is NOT with ISPs, ppl from all over europe on all kinds of connections are complaining, be it 56k to 1meg pipes, are all suffering. GOA / Wannadoo are the source of the problems, try reading the whole thread.

As for your conn, maybe your just lucky, or maybe your not on at the times when its been at its worst.

 

[-Nxs-]

Originally posted by Whoodoo_RD


Are you illiterate???? Evidence from the traces proves the problem is NOT with ISPs, ppl from all over europe on all kinds of connections are complaining, be it 56k to 1meg pipes, are all suffering. GOA / Wannadoo are the source of the problems, try reading the whole thread.

As for your conn, maybe your just lucky, or maybe your not on at the times when its been at its worst.

Ive been online most of the weekend esp daytime and evening, awwwwww is poor ickle Whoodoo_RD going LD... Im normaly one for slagging off GOA at any opertunity, but I feel like a change here, my conn has been fine for the last 2 weeks since i changed iSP, i think the worst was a 10sec red packet loss but no LD.

 

[Ragnarok1978]

Originally posted by -Nxs-
Lol just get yaself on a propper ISP

Im STILL playing happily all weekend and NO linkdeads not even one... and very little if any lag - performance counter for ping and packet loss is most of the time green.

and no im not on a special line, its just a standard ADSL home package with nildram, when i was on BT i was going LD at least twice a week, now ive moved to a half decent ISP there are no problems at all, maybe my ISP has more tolerant equipment ? who knows, just stating a fact that my connection is fine

It ain't that easy for everyone mate.

For example, pretty much one and all danish ISP route thru either sprintlink (teh horror) or gblx.net (also bad, getting better tho) when going to france. Just because a handfull gamers complain, they aren't gonan change their routings, profit plays a big part here.

And even tho my own traffic goes thru gblx, I hardly ever get problems till I hit opentransit which are the biggest sinners for creating lag on the DAoC servers.

 

[Roo Stercogburn]

On Saturday I had something happen which pretty much confirmed for me it's not just a general internet conditions problem.

Bear in mind that routers and the general internetworking software and hardware make no distiction between different data content, they will behave the same way regardless of what the data packets contain.

So...

During one of the double lag spikes, my char was on a horse. The horse and every piece of scenery pretty much froze in place. However, I was in the middle of typing something and this appeared on channel straight away. The freeze continued for about another minute, but during that time I was able to type on all channels and the response was displayed on screen almost immediately. Since as I said, the internet networking makes no distinction between data content, this points to a server based problem, where the server was only able to process some of the data coming through but no others. It seems as if the chat processing was working fine, but all other game updates were failing.

This is not an internet condition and cannot be explained away as an internet condition related problem. It might have *possibly* been a side effect from some networking related issues where job queing was failing on the server because info coming from the internet was lagging badly then a load hitting the server at the same time, but I'm not convinced. If there was prioritisation taking over, there should have been *some* gameworld updates taking place during the time everything was frozen rather than just chat info sailing through every time. I am very familiar with the effects of massive lag: your keyboard response slows down and though slow and jerky, gameworld updates continue to happen either completely or not at all.

I logged out and immediately posted this to Right Now in addition to some of the other things I had to say on the subject.

 

[Slipurson]

HMMM

Are u blameing GOA for response times over 600 even 5-6 hops before their server ?

But at the same time it is not our ISPs since i had 2 DAoC running at the same time, (mine and my Girlfriends) and she had no lag whatsoever the whole weekend and i have had 2-3 Lds, the only thing different for me and her was that we were in diff zones, most of the time she was in JH crafting and i was all over the place , and the LDs i had was when much ppl were at the same place... so it seams like the servers can not handle to many connections going to the same "zone"...

and for the "are GOA going to refund..." plz think before typing... refunds for 4-5 Lds ? (if u had more then just don't keep logging in since GOA "suck" so much)....

 

[Whoodoo_RD]

Originally posted by -Nxs-
awwwwww is poor ickle Whoodoo_RD going LD...

Twice LD all weekend, but witness to many other ppl dissapearing in mass lumps.

 

[Fafnir]

for once i've been spared of the worst lag this weekend, just getting a few lag spikes, and 2 l'd while i've seen people ld dussin of times.

 

[Jupitus]

Originally posted by Roo Stercogburn
On Saturday I had something happen which pretty much confirmed for me it's not just a general internet conditions problem.

Bear in mind that routers and the general internetworking software and hardware make no distiction between different data content, they will behave the same way regardless of what the data packets contain.

So...

During one of the double lag spikes, my char was on a horse. The horse and every piece of scenery pretty much froze in place. However, I was in the middle of typing something and this appeared on channel straight away. The freeze continued for about another minute, but during that time I was able to type on all channels and the response was displayed on screen almost immediately. Since as I said, the internet networking makes no distinction between data content, this points to a server based problem, where the server was only able to process some of the data coming through but no others. It seems as if the chat processing was working fine, but all other game updates were failing.


I logged out and immediately posted this to Right Now in addition to some of the other things I had to say on the subject.

Roo... this is a side effect of a lag spike, and it is weird, I agree, but basically you stay on the horse through the lag spike and then appear stationary on it afterwards, even though you have in fact lagged off it. I have had this a few times now, and you will be able to talk or anything else after the lag spike, and eventually the horse will vanish and you will be standing there. I even recognise the signs and just jump off and run these days instead of waiting for the horse to vanish.

The symptoms being displayed appear in opentransit's network in my experience. Sudden spike of packet loss at their nodes causing large numbers of users to go LD. Not all users will be affected by this... many ISPs in Europe may route to GOA through backbone providers other than Opentransit, or even still use Opentransit but different nodes within their network. To say that it has to be GOA for a subset of users to be affected and others not is not valid.

 

[Envenom2]

hm dunno about any lag i havent played Daoc in over a month and its the best thing ibe done ^^ daoc sucks no depth no nothing and they cant even run a server properly ^^ rolllllll on swg

 

[censi]

Erm chaps its the open transit routers...

Since about new year there have been intermittent problems with them....

About once every 1-3 hours they jump up to about 40% PL...

I have never seen this many mass LD's than I have done this week...

Again not a word on the GOA web site...

All you can do is keep spamming them emails with the traceroute info in it....

I empasize the word SPAM...

A good tip I have learnt, never send GOA 1 mail about a problem, send them 10. Dirty i know but it works.

Also point out in the mail that its not just you but 95% of the server being effected, and if they dont post information on the website within 24 hours you will be cancelling your subsciption....

Comms problems happen as we all know... but they should be investigating them in a timely manner and keeping us informed of their progress and what the actual details of the problem are....

They get away with it because you let them... Now stop your moaning here, get a traceroute the next time it happens and complain and keep complaining....

GOA are lazy....

 

[Roo Stercogburn]

I'm not saying there hasn't been problems with internet conditions.

However...

...if it was entirely down to an internet condition, it would block all traffic, not selectively choose which type of data gets to and from the client to the server. Chat channels updated with no visible delay, even coming from other players while everything else in game is completely frozen. Some of these were entries I typed AFTER the lag spike hit and I deliberately kept typing in on different channels throughout the lag spike. All updated perfectly.

I've seen the frozen horse routine before, if not for this anomoly wouldn't really think much of it. I'm also familiar with keyboard effects when lag hits... its normally impossible to type.

-

(Relevant info all done with, rest of this post is musing, you may want to skip to next post / find a new thread to read / take the dog for a walk... erm, even more than usual with my posts )

-

I'm intrigued about the server processes in place for handling delayed data (eg because of lag). I know SWG is going to use a queing system for each game upate but I wondered how DAoC does it. If DAoC uses a queing system then concevably 'chat' queue items could go into a different queue than other gameworld updates and appear to update normally while the rest of the game appears frozen since there is a backlog of queued updates waiting to go through.
Does the server then have to make decisions about which updates to ditch and which to keep or does it put the client updates on hold while it resolves the queue issues? This might cause the some clients to appear to freeze while it gets on with it then once the issue is sorted, the game jumps to the point where the server updated to, with default actions for certain types of activity (eg getting thrown off horses, even though you don't actually LD).

Even though a lag spike may have only taken moments I bet the backlog generated by thousands of clients updating even for a second or two could take the server a while to clean up - remember while its cleaning up, new updates would be coming in all the time, so the easiest (perhaps not best) way to resolve the issue may be to prevent any updates until the queue is down to an acceptable size again.

Is there code that will allow certain data to update on the server while putting other game updates on hold until the situation is sorted? Does it even measure the quality and rate of throughput per client or does it just count the time since the last client-to-server data packet was received to decide if it has timed out? I'm talking here about the game server engine itself, not the operating system or hardware.

However, if there is a queing system and chat items have their own queue, and the types of freezes are a result of the server struggling to clean out queued data when lag hits, then perhaps higher spec server hardware is the solution? Or is the hardware platform at max current available spec and its being stressed because the game engine simply can't handle it?

What kind of load balancing and spare capacity is in place for these occasional blips, ie what % of 'normal' full server and network capacity is there above that for these blips.

Erm, there's more than this but I'll stop now because you're probably gnawing your leg off with boredom

 

[old.Glendower]

Actually, I thought it rather interesting.

Nobody at GOA will understand it though.

 

[Thorarin]

Originally posted by Roo Stercogburn
...if it was entirely down to an internet condition, it would block all traffic, not selectively choose which type of data gets to and from the client to the server. Chat channels updated with no visible delay, even coming from other players while everything else in game is completely frozen.

Bad example, I'd say.
DAoC uses two different types of connections to communicate with the server, TCP and UDP. The difference between them is that TCP guarantees that all data arrives, and arrives in the right order (at least from the client point of view). This is assuming the network connection is not completely broken of course (TCP is a connection-oriented protocol). The logical usage for this type of connection is things such as text from chat channels.

UDP is a connectionless protocol and does not guarantee that data arrives in order, or that data arrives at all for that matter. World data is sent like this, because the server can't afford to buffer all this data in case of a slow connection. On a 200 people relic raid a modem user would fall more and more behind until finally he sees what happened about 15 minutes ago, but without any apparent lag-effects such as characters walking further than they actually do. With UDP, the excess data is easily dropped, creating such effects, but staying current.

Now, introduce some packet loss in the connection between the server and you. All the TCP data (chat text amongst other things most likely) still arrives fine, although probably a bit later, because certain packets need to be resent. UDP doesn't resend any packets, they are lost forever, so if the packet telling you that Invader X stopped running at location Y never arrives, you see him continuing to run at you. If you then receive a packet that he's casting a spell, you see him running at you while casting a spell (CHEATS!), etc


I hope this also answers your question about queueing. As far as I can see, there is none for world updates, while queueing for chat channels is inherent to the protocol.

Edit:

I've seen the frozen horse routine before, if not for this anomoly wouldn't really think much of it. I'm also familiar with keyboard effects when lag hits... its normally impossible to type.

With regard to the keyboard effects, I think you are thinking of telnet connections and the like (hell, I even get it while running SETI@Home and typing in a command prompt at the same time). This does not apply to DAoC though, as you type your entire sentence on the client, which then sends it to the server in one go.

Telnet clients go like

"I'd like a capital A please"
...
"Ok, I got your request, hold on. Here, a capital A!"
...
"Gee, thanks. Now I can put a capital A on my screen"

 

[Roo Stercogburn]

Interesting Thorarin, but I think I've probably not explained it very well. This lag spike was different from any other lag spike I've ever had: normally everything freezes , including chat. This was the first time updates to one part of the game continued normally while the rest was locked, which marked it out as different. I've never seen the chat windows continue while the game was frozen before. Don't know if this happens to other peeps a lot, but it was my first time.

And regardless of the protocol or port that the data uses, it still has to get to the server in the first place to update onscreen... which it was doing. All I was pointing out was that the data was reaching the server in a timely fashion, ie, the network was working. And btw, regardless of TCP resending packets... if the connection is down, it doesn't matter how often it tries to resend them, they won't get there and TCP will eventually time out.

For the protocol to make a difference to the connection to the server for the type of data (if they do use different protocols for chat and gameworld updates), TCP and UDP would have to be routed differently. (Side issue: if they were routed differently this would invalidate any evidence from TRACERT and pings since these utilities will only check one route at a time).

So - we have one route to the server, and some of the data gets through in a timely fashion. Ie, network is fine. At least in this particular instance. Chances are internet conditions may have set it all off, but the server was then getting upset and unable to cope afterwards.

Part of the reason I'm interested in the queueing is because since the amount of data the server receives over time might not be even close to constant, but rather dipping and increasing wildly when the internet lags, any queue handling may struggle when the amount of data received peaks. Because we're only talking about data that has actually reached the server, this is where server hardware may struggle a bit, and contribute to make the delay to game updates slower. So you get an initial lag, then lots of data hitting the server at once, then the server has to handle it, and grinds a bit while it catches up.

Though its not likely to happen, I'd love to have a Mythic or GOA techie actually tell us how the system handles these things

 

[Thorarin]

Originally posted by Roo Stercogburn
So - we have one route to the server, and some of the data gets through in a timely fashion. Ie, network is fine. At least in this particular instance. Chances are internet conditions may have set it all off, but the server was then getting upset and unable to cope afterwards.

Not necessarily. If half the packets are dropped, TCP traffic will still have enough troughput for some chat text, but world updates would be very bad. They wouldn't freeze completely though, which is what you're saying I think..

And yeah, the server is likely to have a higher load just after a lagspike

Also, I wouldn't bet on any GOA person explaining this, they just translate some texts and run a couple of executables on a couple of computers