iPlayer shaped?

[Sean]

I'm on Karoo Max Option Two.  I sync, stable, at 4.5mbps.  I have no speed problems with the 100MB karoo test downloads, no packet loss either.

13:00 Saturday:


18:30 Today:


The stream 1, 2 and 3 come from the CDNs that host the streams (Akamai, Level 3, Limelight).  The first is their normal BBC site.  The streams are half the connection, and the results today are actually better than normal.

18:30 Today, Youtube, Wireshark packet loss with excessive buffering on a none HD stream:

[dylan]

Could this indicate (at least in the case of iPlayer + Akamai) that the route to the Akamai edge server has less bandwidth available to it than the route to the BBC server?

Perhaps Karoo needs to pay for more transit from Fused Networks (Telecomplete) http://www.fused-networks.com/information/100104/100185/100191/transit/ or even better, would it be possible for Karoo to host some local caching servers themselves to keep traffic on the KC network?

Dyls

Karoo to BBC

  3    19 ms    19 ms    20 ms  10.102.240.129
  4    35 ms   204 ms   202 ms  10.102.240.221
  5    25 ms    29 ms    25 ms  10.102.59.7
  6    34 ms    26 ms    27 ms  bbc-gw1-linx.prt0.thdoe.bbc.co.uk [195.66.226.103]
  7    27 ms    31 ms    31 ms  212.58.238.133
  8    29 ms    28 ms    28 ms  te12-1.hsw0.cwwtf.bbc.co.uk [212.58.239.222]
  9    31 ms    28 ms    28 ms  212.58.255.12
 10    28 ms    28 ms    36 ms  www.bbc.co.uk [212.58.246.161]

Karoo to Akamai

  3    23 ms    25 ms    22 ms  10.102.241.117
  4    21 ms    20 ms    20 ms  10.102.240.221
  5    25 ms    25 ms    28 ms  10.102.58.7
  6    25 ms    26 ms    25 ms  linx2.telecomplete.net [195.66.226.139]
  7    25 ms    26 ms    26 ms  193.0.255.25
  8    27 ms    26 ms    25 ms  v101-r4.thn.telecomplete.net [213.160.96.121]
  9    68 ms    65 ms    69 ms  v100-r5.thn.telecomplete.net [213.160.96.49]
 10    25 ms    26 ms    30 ms  akamai-thn.telecomplete.net [213.160.97.30]
 11    26 ms    26 ms    29 ms  cp41752.edgefcs.net [88.221.84.110]

[Adrian]

Just looking at some BGP details, once I've made head and tails of it I'll post more

[Adrian]

BGP Looking glass details from LonAP:

BGP routing table entry for 87.102.0.0/17, version 26743
Paths: (4 available, best #3, table Default-IP-Routing-Table)
  Advertised to update-groups:
     1         
  6461 12390 12390 12390 12390 12390
    193.203.5.16 from 193.203.5.16 (64.125.0.165)
      Origin IGP, metric 0, localpref 990, valid, external
      Community: 6461:1001 6461:2004 6461:2107 6461:2215 6461:2434 8330:4
  6461 12390 12390 12390 12390 12390, (received-only)
    193.203.5.16 from 193.203.5.16 (64.125.0.165)
      Origin IGP, metric 0, localpref 100, valid, external
      Community: 6461:1001 6461:2004 6461:2107 6461:2215 6461:2434
  12390 12390 12390 12390
    193.203.5.18 from 193.203.5.18 (212.50.161.251)
      Origin IGP, metric 0, localpref 990, valid, external, best
      Community: 8330:4
  12390 12390 12390 12390, (received-only)
    193.203.5.18 from 193.203.5.18 (212.50.161.251)
      Origin IGP, metric 0, localpref 100, valid, external
collector

BGP from Linx:

BGP routing table entry for 87.102.0.0/17, version 23975377
Paths: (7 available, best #3, table Default-IP-Routing-Table)
  Advertised to update-groups:
     1         
  6461 12390 12390 12390 12390 12390
    195.66.224.76 from 195.66.224.76 (64.125.0.251)
      Origin IGP, metric 28, localpref 100, valid, external
      Community: 5459:1 5459:60
  6461 12390 12390 12390 12390 12390, (received-only)
    195.66.224.76 from 195.66.224.76 (64.125.0.251)
      Origin IGP, metric 28, localpref 100, valid, external
  3356 12390
    195.66.224.77 from 195.66.224.77 (4.68.2.7)
      Origin IGP, metric 0, localpref 100, valid, external, best
      Community: 5459:1 5459:60
  3356 12390, (received-only)
    195.66.224.77 from 195.66.224.77 (4.68.2.7)
      Origin IGP, metric 0, localpref 100, valid, external
  12390 12390 12390 12390 12390
    195.66.224.119 from 195.66.224.119 (212.50.161.251)
      Origin IGP, metric 0, localpref 100, valid, external
      Community: 5459:1 5459:60
  12390 12390 12390 12390 12390, (received-only)
    195.66.224.119 from 195.66.224.119 (212.50.161.251)
      Origin IGP, metric 0, localpref 100, valid, external
  3356 12390
    195.66.226.77 (metric 2) from 195.66.232.239 (195.66.232.239)
      Origin IGP, metric 0, localpref 100, valid, internal
      Community: 5459:3 5459:60

Why is the routing prepended to use 3rd party transit (AboveNet/AS6461) effectively encouraging traffic to come back using a transit provider, rather than your own network (Kingston/AS12390) ?

[~C~]

Because you may want to balance your ingress traffic across multiple transit providers and IX peers, not pull all your traffic in via IX peering where you will most likely have less bandwidth than that at your transit points. Maybe more bandwidth at Linx, LoNAP etc is needed??

~C~

[Adrian]

I was under the impression it would be cheaper and likely to have more bandwidth on your own network and at the peering points than you would from transit providers which may cost more?

I am also under the impression more bandwidth is needed at LINX, etc...

Someone has mentioned that the reason for this arrangement could be down to capacity issues at the peering points.

[Adrian]

Different for the earlier netblocks though:

212.50.160.0/19 and 213.249.128.0/18

BGP routing table entry for 212.50.160.0/19, version 24180368
Paths: (7 available, best #3, table Default-IP-Routing-Table)
  Advertised to update-groups:
     1         
  6461 12390
    195.66.224.76 from 195.66.224.76 (64.125.0.251)
      Origin IGP, metric 28, localpref 100, valid, external
      Community: 5459:1 5459:60
  6461 12390, (received-only)
    195.66.224.76 from 195.66.224.76 (64.125.0.251)
      Origin IGP, metric 28, localpref 100, valid, external
  3356 12390
    195.66.224.77 from 195.66.224.77 (4.68.2.7)
      Origin IGP, metric 0, localpref 100, valid, external, best
      Community: 5459:1 5459:60
  3356 12390, (received-only)
    195.66.224.77 from 195.66.224.77 (4.68.2.7)
      Origin IGP, metric 0, localpref 100, valid, external
  12390 12390 12390 12390 12390
    195.66.224.119 from 195.66.224.119 (212.50.161.251)
      Origin IGP, metric 0, localpref 100, valid, external
      Community: 5459:1 5459:60
  12390 12390 12390 12390 12390, (received-only)
    195.66.224.119 from 195.66.224.119 (212.50.161.251)
      Origin IGP, metric 0, localpref 100, valid, external
  3356 12390
    195.66.226.77 (metric 2) from 195.66.232.239 (195.66.232.239)
      Origin IGP, metric 0, localpref 100, valid, internal
      Community: 5459:3 5459:60

BGP routing table entry for 213.249.128.0/18, version 24180408
Paths: (7 available, best #3, table Default-IP-Routing-Table)
  Advertised to update-groups:
     1         
  6461 12390
    195.66.224.76 from 195.66.224.76 (64.125.0.251)
      Origin IGP, metric 28, localpref 100, valid, external
      Community: 5459:1 5459:60
  6461 12390, (received-only)
    195.66.224.76 from 195.66.224.76 (64.125.0.251)
      Origin IGP, metric 28, localpref 100, valid, external
  3356 12390
    195.66.224.77 from 195.66.224.77 (4.68.2.7)
      Origin IGP, metric 0, localpref 100, valid, external, best
      Community: 5459:1 5459:60
  3356 12390, (received-only)
    195.66.224.77 from 195.66.224.77 (4.68.2.7)
      Origin IGP, metric 0, localpref 100, valid, external
  12390 12390 12390 12390 12390
    195.66.224.119 from 195.66.224.119 (212.50.161.251)
      Origin IGP, metric 0, localpref 100, valid, external
      Community: 5459:1 5459:60
  12390 12390 12390 12390 12390, (received-only)
    195.66.224.119 from 195.66.224.119 (212.50.161.251)
      Origin IGP, metric 0, localpref 100, valid, external
  3356 12390
    195.66.226.77 (metric 2) from 195.66.232.239 (195.66.232.239)
      Origin IGP, metric 0, localpref 100, valid, internal
      Community: 5459:3 5459:60

[~C~]

It is certainly cheaper to pass traffic over your IX peering and in an ideal world, LINX, LoNAP and MaNAP would all be on 10GigE. *However*, as they are only on GigE at the moment then there is only a finite amount of traffic that can be pulled over those interfaces. Rather than maxing out IX peering links at 1Gig it's better to run them at a lower capacity to allow for bursts of traffic and then put the rest of your traffic onto your transit links. As most of the providers at the IX points are also connected to the same T1 transit providers (in this case Level3 and Abovenet). Like I said, it a balancing act getting the right amount of ingress/egress over all your links and also making sure that the traffic comes into the right POP site (ie. London or Manchester). There's not as much going on at MaNAP as there is at LINX so that tends to force a majority of the traffic through London. If capacity on the Hull-London links is running high but Hull-Manchester isn't then it makes sense to move some traffic over to Manchester to even things up. How do you do that? You prepend out of your London links (LINX, Abovenet) but don't on your Manchester links (MaNAP, Level3).

Some ISP's will ignore prepends and set a higher local-pref on their LINX routes (for example) so getting the right balance of traffic into the network requires careful monitoring and slight adjustments, ie. prepend one prefix but not the other then monitor traffic levels across all links, until you get the right balance and no one link is going to run at 99% while another sits at 30% (for example).

~C~

[Adrian]

Thank you for the insight, C

[dylan]

Thanks C, this is all very interesting, the good majority of people don't know much about ISP level networking.

One question though, is it not possible to operate a local Akamai cache (or other CDN server) to avoid sending traffic down highly contended links?

Dyls

[~C~]

Dylan,

It probably is possible. Will it happen? Dunno to be honest!!

 

~C~

[friendlykcengineer]

Hi All,

I have passed on your observations re the peering / transit to the relevant team for comment.
~C~ , your analysis shows awesome in-depth knowledge!
FYI we did have a short-lived but technically successful trial of local iPlayer caching, but a general roll-out is on hold pending commercial decisions beyond the control of Karoo. I'm not able to comment further on this, so please don't ask!
Leaving aside the transit / caching debate, I'd still like to get some hard data on who is having issues and when, so far not a lot of feedback.

Cheers

FKCE

[friendlykcengineer]

Just seen the other thread re different speeds across the KC network.
Will try and get my head round that info and factor it in.

[friendlykcengineer]

Hi All,

After discussion with our peering guru's it appears that there could be differences in speeds experienced by our customers during peak demand periods depending on the route taken from the national network into the Hull network. They are aware of this and have been using pre-pending to try and balance traffic to avoid pinch-points in recent weeks / months. They have also moved some traffic away from peering to transit (which is obviously more expensive) as part of these short-term measures. A long term solution has been in the planning stages for some time, and will be implemented in the next few weeks.

This could explain some of the different speeds you've been reporting, and the effects could appear pretty random across the network, but no particular area should be any worse affected than others.

Will post again when I get more info.


Cheers

FKCE

[icoot]

HU17

Karoo letter tells me i can get 15MG on ADSL+ but ..

I WAS happy with my max 2 package on 8MB which router says im connecting at alas from about 1pm till 1am I get the below....