Why non-ISP's should peer

or,

how non-ISP's can usefully gain value from a connection to one of the major peering points in New Zealand

Preamble

Peering (the exchange of zero-cost traffic between consenting parties), as opposed to transit (payment by a customer to a network provider, to carry traffic over the providers infrastructure) is principally a social, political and economic problem. The technical procedures needed to implement peering are well defined and relatively straightforward, when compared with the machinations required to get agreement to exchange routes with a prospective peer.

Policies towards peering in New Zealand vary widely amongst ISP's. The smaller ISP's (generally those that don't have direct Southern Cross capacity, and which are locally owned) will usually cheerfully peer with all and sundry. Larger organisations with asperations toward telco status can be more difficult to convince to peer, but still not impossible - you just have to put in more effort. In general, peering in New Zealand is no harder, and arguably easier to organise than elsewhere in the world. The high cost differential between international and national bandwidth provides a strong incentive for network providers to keep New Zealand traffic within New Zealand, and that encourages those providers to look upon peering favourably.

About the exchanges

In basic operation, the exchanges are pretty similar. eBGP requires that all the peers be on the same IP subnet, on the same broadcast media - you can't easily do multipartite peering if there are routers in the way of some of the peers. Thus, large metro ethernets are good candidates for exchange points. On each exchange there are two route servers - Linux PC's that run the Quagga BGP listener - these machines have identical configs located in different buildings.

There are small differences in the exchanges - WIX is distributed throughout Wellington on the CityLink ethernet fabric, you have to be a CityLink customer to use it. While APE was originally more localised in the Sky Tower in Auckland with connections to it provided by a variety of operators, connectivity to the APE switches is now also available to any CityLink customer in the Auckland CBD. Pretty much anybody on CityLink in Wellington can join WIX, the relatively low cost of entry means there are many peers of all sorts of sizes, as the cost of participating in WIX is a negligible increase over the basic costs of getting a CityLink connection. (There are also smaller regional exchanges in development for Palmerston North, Christchurch and Invercargill.)

The higher cost involved in getting a circuit to APE has meant that there is a smaller number of more committed peerers - per peer, more traffic is moved over APE than over WIX. While the cost of getting a connection to the APE switches is coming down it still is higher than in Wellington. As such, this document will tend towards discussing how and why an entity should connect to APE - for most organisations with the wherewithal to do so, peering on WIX needs little justification. (These arguments also apply to the regional exchanges.)

The exchanges support a couple of modes of operation:

1. bilateral peering between consenting parties.
   
2. multilateral peering via the route servers at each exchange.

Bilateral peering tends to suit organisations with legal departments who have time and energy to spend on organising and managing their peering agreements.

Multilateral peering tends to appeal to organisations who don't have a dedicated peering department, especially as the number of peers increase (the n-squared nature of bilateral arrangements bites pretty quickly as the number of peers increases).

Different organisations therefore have different peering policies. Small ISP's and non ISP organisations tend to favour relying on the route servers (especially in Wellington, where there are now over 80 non-ISP peers) - you expend 5% of the effort, for 90% of the value.

Medium sized ISP's use the route servers, but also use bilateral sessions for peers of extra importance. Small and medium ISP's tend to be fairly promiscuous - they'll peer with anybody who wants to.

Large ISP's prefer to only enter into bilateral sessions with other large ISP's, but with a bit of effort it is possible to get them to bend their definition of large so that they'll peer more widely.

Basic Peering

We'll define basic peering as "announcing some routes to the route servers, and listening to everything the route servers send" - it's lazy man peering. It doesn't get you every possible peer on APE or WIX - there are some ISP's that only advertise to the route servers, and don't listen to announcements from the route servers, and others that don't peer at all with the route servers. However, it does get you well underway, for little effort.

Setting up basic peering

There are a few things that you need to assemble, before you can start peering on APE or WIX.

1. a connection of reasonable bandwidth (at least 10Mb ethernet, more usually 100Mb/Gig ethernet) to APE or WIX
   
2. a router than can handle BGP-4, and with enough throughput for the traffic loads you expect to put on it. Most peers use Cisco routers, or Zebra based PC's (Linux/FreeBSD), with a smattering of Juniper and Allied Telesyn gear dotted around.
   
3. an AS number. These can be obtained from APNIC, either directly or through an APNIC member (CityLink can obtain one for you, as, usually, can your ISP). On WIX, we have historically used private ASN to enable peering for non ISP's, but are moving towards using public ASN for all peering in both Auckland and Wellington. If you plan to do any kind of multihoming between multiple ISP's, then you definitely want the public ASN.
   
4. some IP numbers. The more the better - /29 and below are accepted by the route servers, but some ISP's only accept /24 and shorter. This may seem like a self evident requirement, but a suprising number of CityLink users in Wellington try and peer with only a /32, when it doesn't really make sense to do so.

Assuming that you've got points 1-4 above sorted, then you're ready for basic peering on APE or WIX. You fill out the forms, use the recipes on the website and in the confirmation email to configure BGP-4 on your router, and be happy. The route servers employ strict filtering on inbound announcements from peers, so you will have to let CityLink know what IP networks you plan to advertise to the route servers. The route servers are configured via RPSL, which allows for automated generation of the BGP configs from a database driven policy specification.

See the Getting Started document for more information on this.

Filling in the gaps

Once you've got basic peering with the route servers going, you'll notice fairly quickly that you get some oddities with traffic - depending on who's listening and who's advertising, traffic can take assymetric routes, and you'll also notice some holes in your route table where you might think certain ISP's should be. The trick now, is to encourage those ISP's who aren't listening to your announcements into the route server to do so.

Often this is relatively straightforward - an email to NZNOG from CityLink staff asking everybody to listen to the new prefix announcement, coupled with a few phone calls can be all it takes to get your routes out to many of the ISP's who don't have a default "listen-all" stance. However there are going to be some ISP's who just won't listen to your announcements - and unfortunately, these tend to be the biggest ISP's.

So, we need a way to make the big boys listen to your announcements. Fortunately, this isn't that hard. All you need is:

• a goodly pile of interesting content that users out on the Internet in New Zealand want to get at.

Non ISP's that are net sinks of traffic requesting peering are not looked upon with favour by big ISP's - they have little bargaining power, since they basically want something for free, and give little value to the ISP in return for the peering. The lack of favour increases as the disparity between source and sink increases - the more of a leech you are, the less likely you are to get peering.

However, if you are a net source of traffic, then the boot moves rapidly to the other foot. If you're a provider of services that NZ Internet users want to see, then that gives you significant bargaining power over the ISP's. If that traffic is primarily of interest to local users (ie has little international component to cloud the issue), and expecially if that traffic is timing sensitive (ie, if it's intolerant of packet reordering or latency variation), then so much the better. The ideal situation is to offer a high bandwidth service that is performance sensitive to a bunch of users who'll whine at their ISP if it doesn't work well. Something like video conferencing or media streaming (eg lecture delivery for distance learning) is ideal, but sizeable http or ftp traffic will also do the trick. The content doesn't need to be available 24x7, but needs to be in demand for enough time for the ISP's to notice that they're hurting (ie, peak loads of a few seconds aren't sufficient, you need sustained demand over many minutes/hours).

So, you get basic peering going, and then you get your pile of content organised. You announce to the world (via NZNOG and possibly via direct contact with ISP's) that you expect significant local demand for your content, and that they should listen to the announcements for your prefix on APE. On the webpage that gives access to that content, you put a big note saying "if you're having trouble getting to the content, please call your ISP and grizzle about why they don't peer properly at APE - there's nothing we can do". After a few rounds of this, most ISP's will get the peering going - they strongly dislike calls that go like this: "Hey, this works fine for my friend who's on ISP X, but I can't reach it through you". If you have some mechanism to inform your users, then publically encouraging them to change ISP to the ISP's that do peer can be an effective tool to "correct" the ISP's that still aren't peering.

This may seem slightly facile, but it has been shown time and again to be an effective technique for getting peering sessions established in NZ - we got more peering sessions setup in the afternoon of the LoTR launch in 2003, than in the previous six months. Generally, you can get pretty much any ISP to setup any peering session if they perceive an emergency (because then the tech guys don't have to get permission), and then once that session is up, then it's in place infrastructure, and they don't bother getting permission to remove it. At a technical NOC level, most ISP's are in favour of peering, it's the policy makers further up the chain that can be problematic.

Redundancy

Once on the exchange, you're in a good position to organise peering sessions with ISP's where they deliver you a full route table, or at least a default route. So peering on the exchanges is a great foundation for setting up redundant Internet access - you can use your exchange connection to add significant reliability to your Internet connection.