On-link vs Off-Link Packet Delivery: Unicast, Multicast, Anycast

By David Beck, a Men&Mice trainer and course developer

Delivering IP packets within a subnet (on-link) is different than delivering between subnets (off-link). This article examines those differences for unicast, multicast, and anycast destination addresses. The goal is to shine a light on the differences for anycasts and to explain on-link anycasting. Everything is applicable to both IPv4 and IPv6, with one exception that is clearly noted. However, that difference is why this is fundamentally an IPv6 article. The IPv6 terminology "link" will be used in lieu of "subnet." "Node" will be used in lieu of "host."

UNICASTS

A unicast address uniquely identifies one interface on one node. It is used to deliver a packet to only one interface. It is the type of address that initially comes to mind, when one hears "IP address." A sender with a unicast destined packet looks up the packet's destination address in its routing table, and selects the longest-match.

The longest-match entry may indicate that the destination is on-link (OL). For an OL destination the sender uses the data link layer (DLL) protocol, e.g. Ethernet, to deliver the packet to the final destination. Alternatively, the longest-match may indicate that the destination is off-link (XL, external-link). The routing table entry for an XL destination includes an OL router that is closer to the destination. The sender uses the DLL protocol to deliver the packet to the router. The router repeats the same process sending the packet toward the final destination.

An internet is two or more links (networks) joined by a router. It is the need to deliver unicast packets to XL nodes, that spurred the creation of internet protocols such as IP. If all destinations were OL, packet delivery could be done with only the DLL protocol and DLL addresses. (Note that the Internet is a huge internet with links throughout the world joined by routers.)

MULTICASTS

A multicast address identifies multiple interfaces on multiple nodes. A packet is delivered to all identified interfaces. In both IPv4 and IPv6, multicast addresses are distinct from unicasts and easily identifiable. IPv4 multicasts are 224.0.0.0/4 (addresses starting with 224-239). IPv6 multicasts are ff00::/8. Multicasting was not always a part of IP. It was added at the end of the 1980s.

Delivery of OL IP multicasts relies on the capability of the underlying DLL protocol to support multicasting, or to at least support broadcasting. Happily, Ethernet and 802.11 wireless have multicasting functionality and there are multicast MAC addresses. This makes OL multicasting delivery simple. IP multicast addresses are mapped to MAC multicast addresses through a formula, for IPv4 that is defined in RFC "Host Extensions for IP Multicasting" (http://www.rfc-editor.org/rfc/rfc1112.txt), and for IPv6 in the RFC “Transmission of IPv6 Packets over Ethernet Networks” (http://www.rfc-editor.org/rfc/rfc2464.txt). A node assigned a multicast IP address configures its network interface to listen for the corresponding MAC multicast address. When a packet is sent to an OL IP multicast address, it is packed in a frame destined for the corresponding MAC multicast. All nodes listening for the MAC address, receive and process the frame and the packet. OL multicasting is easily implemented, and widely used.

Both IPv4 and IPv6 routing protocols rely on OL multicasting. A router sends one packet to notify all others of routing protocol information, such as a network topology change. (OL delivery of IP multicasts, with underlying DLL protocols that don't support multicasting or broadcasting, can be very challenging. That little nightmare is happily well outside the scope of this article.)

XL multicasting, with a multicast address assigned to nodes on different links, is far more challenging. A special multicasting routing protocol must be implemented. The sender generates one packet; multicasting routers must duplicate the packet to deliver it to nodes on different links. XL multicasting is not widely used. The mass majority of organizations doesn't implement multicasting routing protocols. Multicasting routing protocols are not used in the open Internet.

ANYCASTS

Now we're reaching the heart of this article.

Like a multicast, an anycast is an address assigned multiple interfaces on multiple nodes. Unlike a multicast, an anycast packet is delivered to only one node. The sender doesn't care which node receives the packet, all the destinations are equivalent.

Like multicasting, there is a wide chasm between OL and XL anycasting. For multicasting, OL is easy and widely used. XL multicasting is challenging and rare. It is the opposite for anycasting. XL anycasting is easy.

XL anycasts are used for root and TLD (Top Level Domain) DNS servers. For example, the F root server (f.root-servers.net) has the IPv4 address 192.5.5.241. The address is an anycast. It is currently assigned to fifty-five DNS servers throughout the world (http://www.isc.org/f-root/). For DNS, all these servers are identical. When a packet is sent to 192.5.5.241, the natural process of unicast routing delivers it to only one node (one DNS server). Since all the servers have the same information, it is unimportant which is reached. In order for administrators to manage individual servers, each also has a unicast address, but that address is not intended for answering DNS queries.

XL anycast addresses are indistinguishable from unicast addresses. When a unicast address is assigned to a second node, the address becomes an anycast. The various nodes assigned the address don't know that it is an anycast and not a unicast. No special handling is required by nodes assigned an anycast address. No special handling is required by a sender generating a packet to an anycast destination. There is no protocol supporting anycasts. None is needed. Normal unicast routing handles delivery of XL anycasts. The only requirements are that the nodes sharing an address are on different links, and routing is properly configured. There isn't even an RFC defining the technical specifics for XL anycasting. Note that several RFCs discuss anycasting. For example the RFC "Operation of Anycast Services" is a Best Current Practices (BCP) document (http://www.rfc-editor.org/rfc/rfc4786.txt).

While XL anycasting is found in IPv4 and IPv6, OL anycasting is only implemented in IPv6. Nodes on the same link share an address. A packet addressed to an OL anycast is still delivered to only one interface. However, because the nodes are on the same link, routing can't handle delivery. Unlike XL anycasting, OL anycasting requires a technical specification. RFC "IP Version 6 Addressing Architecture" (http://www.rfc-editor.org/rfc/rfc4291.txt) specifies OL anycasting.

A node assigned an OL anycast sends Neighbor Advertisements (NAs). The NAs associate the anycast address with the node's own DLL unicast address. The NAs are fundamentally the same as the node would send for an assigned unicast address. Mapping an OL anycast, to the DLL unicast of a node, was a fundamental implementation decision taken by the IPv6 designers. Without prejudice as to whether it was a good or bad decision, it is noteworthy that there were other options. For example, First Hop Redundancy Protocols (FHRPs), essentially implement an OL anycast address, but with a completely different approach. The IETF standardized FHRP is RFC specified: "Virtual Router Redundancy Protocol (VRRP) Version 3 for IPv4 and IPv6" (http://www.rfc-editor.org/rfc/rfc5798.txt).

By definition, an anycast address is assigned to multiple nodes. Each node assigned an OL anycast sends NAs associating the address with its own DLL unicast. These NAs are basically competing with each other. Other nodes on the link associate the OL anycast with one DLL unicast address, with one of the nodes assigned the OL anycast. If, three nodes on an Ethernet share an anycast address, another OL node will use the unicast MAC address of one of the three. When a packet is sent to the anycast address, the DLL deliveries it to one node. In this way, the anycast requirement of delivery to a single node, is met.

It is worth considering why DLL unicasting is used for OL anycast delivery. While it isn't ideal, it is necessary because DLL protocols don't have anycast capabilities. If, for example, Ethernet had anycasting addresses and anycast functionality, an OL anycast could be mapped to a MAC anycast. OL anycasting would then be as trivial as OL unicasting, and OL multicasting. Since DLL anycasting doesn't exist, OL anycasts must be mapped to what DLLs provide, unicasts, broadcasts, or multicasts. DLLs broadcasting and multicasting functionality cannot be used, because they violate the anycast requirement of deliver to only one node. So delivery of an OL anycast is necessarily implemented by a DLL unicast.

XL anycast addressing require no special handling. OL anycast addressing does. A node assigned an OL anycast must be explicitly informed that the address is an anycast, requiring IPv6 systems to implement a technique to indicate anycasts. The OL anycast indication suppresses Duplicate Address Detection (DAD). When an IPv6 unicast is assigned, DAD tests if the address is assigned to another node on the link. The address isn't used if DAD reports duplication. Anycasts are purposely assigned to multiple nodes, so DAD is disabled for OL anycasts. Additionally, OL anycast NAs are sent with a slight delay, and, more importantly, with the Override flag cleared in the NA message.

A set Override flag tells the receiver to replace a previously cached entry for the advertised IPv6 address. For a unicast NA, the Override flag is set, so that a receiver uses the new advertisement. This makes sense since the unicast NA is coming from the same node that sent the older cached information. Newer unicast information is better. For an OL anycast, each node assigned the address sends an NA with its own DLL address. The cleared Override flag in these NAs means a receiver will use the DLL address from the first NA that arrives. Older anycast information is better.

The cleared Override flag prevents oscillating between different DLL addresses for the anycast destination. It additionally means that, at any given time, senders on a link are reaching different nodes that share the anycast address. This can be viewed positively, as it provides load balancing. However, it can't be controlled and all senders could also be sending to one DLL address (to one node). Unfortunately, if a cached OL anycast destination becomes unreachable, it can't be replaced until the cached entry times out. So for the unlucky nodes on the link with the unreachable address cached, the OL anycast is unreachable, but for all other nodes on the link the anycast is reachable.

So unlike an XL anycast, an OL anycast requires both special configuration and special handling on the nodes assigned the address. Neither OL or XL anycasts require special handling by senders.

RFCs define IPv6 OL anycast addresses. Being defined distinguishes them from other anycasts, which are purposely indistinguishable from unicasts. The most common OL anycast is the Subnet-Router anycast (SRA). Every link has an SRA address. All the interface identifier bits set to 0, identifies the SRA. For the link 2001:db8:cafe:fee::/64, the SRA is 2001:db8:cafe:fee::/128. The SRA is defined in RFC 4291. RFC "Reserved IPv6 Subnet Anycast Addresses" (http://www.rfc-editor.org/rfc/rfc2526.txt) reserves the highest 128 addresses on each link for OL anycasts.

Initially it may seem that the SRA would be an ideal way to implement a FHRP. However, the widely used FHRPs, HSRP, VRRP, GLBP and CARP all support IPv6 without using the SRA. Where is the SRA used? Saying it has not been widely implemented, is overstating its current uses. Depending on feedback, perhaps another article will delve deeper into the SRA.

So it ends on a whimper. IPv6 has added OL anycasting, but it is unused by any significant application, and its usefulness is questionable.

SUMMARY

IP unicasting, both OL (on-link) and XL (external-link, off-link), is easy and widely implemented.
OL IP multicasting is easy and common. XL multicasting is more involved and uncommon.
OL IP anycasting is extremely rarely used and limited to IPv6. XL anycasting is easy, and although not common, widely used for DNS.