Building IPv6 and Dual-Stack Backbones on AWS

Last Updated: 22/06/2024

This blog will focus on building IPv6-only and dual-stack backbone networks, including inspection patterns for East-West and North-South IPv6 traffic. There will be heavy focus on brownfield environments and how to introduce IPv6 and dual-stack into an existing IPv4-only network. I will not be covering application specific IPv6 and will focus on the core networking side of things.

This blog is intended for Network/Security/Infrastructure/Cloud Architects and Engineers, and I will assume that you have a solid understanding of AWS Networking Services (such as VPCs, Direct Connects, Transit Gateway etc.) and are familiar with IPv6 concepts. If you’re not familiar with general IPv6 concepts, please check out IPv6: Starter to Master.

IPv6 Allocation

AWS allows you to bring-your-own IPv6 Addresses (BYOIPv6) to AWS or you can request an IPv6 block. AWS does not support the use of Unique Local Addresses (ULAs) and only support Global Unicast Addresses (GUAs).

VPCs in AWS support IPv4-only or dual-stack networking, they currently do not support IPv6-only CIDRs at the VPC level. You can allocate an IPv6 Global Unicast Address (GUA) block during the creation of the VPC, and you can edit an existing IPv4-only VPC and allocate an IPv6 Block.

There are two methods of IPv6 Allocation to VPCs in AWS:

1. Direct Allocation. This will allocate a random /56 IPv6 block to the VPC during creation and allows you to have fixed /64 for subnets by default, but configuration in /4 blocks for larger subnets. Note, if you operate multiple VPCs, each /56 IPv6 CIDR allocated to the VPCs will be non-contiguous, meaning appropriate summary routes cannot be formed. I recommend this approach to be avoided.
2. From AWS IP Address Manager (IPAM).
   • BYOIPv6. If you have your own IPv6 block, you can import this into a Top-Level IPAM Pool, and structure your pools according (such as Regional-Level Pools). The smallest IPv6 address range that you can bring is /48 for CIDRs that are publicly advertised by AWS, and /56 for CIDRs that are not publicly advertised by AWS. You will need to provide a x.509 certificate to prove you own the IPv6 block.
   • AWS Allocated Contiguous IPv6 block. AWS allocate a default size of a /52 IPv6 block to a Regional-Level Pool. Top-Level Pools are not supported for AWS Allocated IPv6 Blocks. You can request up to a /40 per Region, subject to AWS approval. AWS Allocated IPv6 blocks are automatically published to the Internet, and this cannot be disabled.

For IPAM Allocated VPC CIDRs (both BYOIPv6 and AWS Allocated), you are able to create VPCs of sizes between /44 and /60, and subnets of sizes between /44 and /64, in increments of /4 versus the fixed size of /56 for VPC through direct allocation. Though direct allocation does also permit varying subnet sizes. Both options have a minimum size of /64 subnets to ensure there a 64-bit Interface ID can be formed.

While VPCs can be IPv4-only or dual-stack, subnets support IPv4-only, dual-stack and IPv6-only. You can add an IPv6 CIDR to an existing IPv4-only subnet to make it dual-stack, but you cannot remove the IPv4 CIDR block of an existing subnet, even if there are no IPv4 resources inside them. To create IPv6-only subnets, you must create new subnets. You can have a mix of all three network structures, within the same VPC.

There are currently no charges for IPv6 Addresses in AWS.

DHCPv6 for IPv6 Host Address Allocation

AWS does support the use of DHCPv6 but there are some important considerations, especially in a dual-stack subnet:

• An EC2 instance in a dual-stack subnet can only retrieve its IPv6 address from the IPv6 DHCP server. It cannot retrieve any additional network configurations from the IPv6 DHCP server, such as DNS server names or domain names.
• An EC2 instance in a IPv6-only subnet can retrieve its IPv6 address from the IPv6 DHCP server and can retrieve additional networking configuration information, such as DNS server names and domain names.
• For an EC2 instance in an IPv6-only subnet, the IPv4 DHCP Server will return 169.254.169.253 as the name server if “AmazonProvidedDNS” is explicitly mentioned in the DHCP option set. If “AmazonProvidedDNS” is missing from the option set, the IPv4 DHCP Server won’t return an address whether other IPv4 name servers are mentioned in the option set or not.

The use of DHCPv6 is optional, and you may statically configure the host OS with an IP address. However, the VPC anti-spoofing mechanism enforces the IP address configured on the network interface controller (NIC) to match the one configured on the underlying elastic network interface. AWS recommends enabling DHCP, even for resources that require static IP addresses, and managing static IP assignment at the elastic network interface level.

In IPv4, when a public IP address is allocated to the EC2 via its elastic network interface, this is lost upon restart/termination. To have a static IPv4 public address, you need to use an Elastic IP that is assigned to the elastic network interface. In IPv6, the Global-Unicast Address (GUA) assigned to the EC2 is retained throughout its lifetime, therefore the concept of IPv6 Elastic IPs do not exist.

Exposure of AWS Services

In IPv4, AWS provide numerous services exposed to users through the use the 169.254.x Link-Local Addresses. There are equivalent IPv6 Addresses for each of these services but only for Nitro Systems. These services are exposed using IPv6 Unique-Local Addresses (ULAs) and not through the use of IPv6 Link-Local Addresses (LLAs) which you may have expected.

Below are the equivalent IPv6 Address for the AWS provided services.

DNS queries for both IP address-based naming (IPBN) and resource-based naming (RBN) DNS hostnames coexist to ensure backward compatibility and to allow you to migrate from IPBN to RBN. For private DNS hostnames based on IPBN, you cannot configure whether a DNS A record query for the instance is responded to or not. DNS A record queries are always responded to. In contrast, for private DNS hostnames based on RBN, you can configure whether DNS A and/or DNS AAAA queries for the instance are responded to or not.

AWS VPC Peering for IPv6

VPC Peering’s are IP protocol agnostic, therefore they support IPv6. Both IPv4 and IPv6 traffic can traverse the same peering connection. Similar to IPv4, you should ensure there is static IPv6 route within the VPC route tables. VPC Peering’s are not a common pattern since the introduction of AWS Transit Gateways, but there are still some uses cases. See Are VPC Peerings still a thing? for example use cases.

AWS Transit Gateway for IPv6

Transit Gateway attachments are IP protocol agnostic themselves and you can route IPv4 and IPv6 on the same attachment, however IPv6 addresses need to be assigned to the Elastic Network Interfaces (ENI) of the attachment, thus you need to explicitly configure assignment for the elastic network interfaces by modifying the attachment and enabling IPv6 support.

You cannot create a transit gateway attachment using IPv6-only subnets. The subnets must be dual-stack to support the use of IPv6. This is due to IPv4 being enabled by default on Transit Gateway attachments, and this cannot be modified.

Connect Type Attachments (used for native SD-WAN connectivity using GRE) support IPv6.  To enable this, the Transit Gateway Peers support IPv6 using Multi-Protocol BGP (MP-BGP) and a /125 CIDR block from the well-known fd00::/8 unique local address range is allocated.

Below is an example of Connect Type Attachment between a Transit Gateway and SD-WAN Appliance:

Source: https://docs.aws.amazon.com/whitepapers/latest/ipv6-on-aws/amazon-vpc-connectivity-options-for-ipv6.html

AWS Virtual Private Gateways do not support IPv6 Site-to-Site (S2S) VPN, thus AWS Transit Gateway should be leveraged which supports IPv6 S2S VPN. A single VPN connection can carry IPv4 or IPv6 traffic, but not both at the same time. When configuring a connection, you specify either IPv4 or IPv6 for use inside the tunnel. The underlying connectivity (outside tunnel IPs) occurs via IPv4 only. This means that both the AWS and customer VPN terminating devices need to be addressable via public IPv4 addresses. 

Below is an example of S2S VPN for Transit Gateways:

Source: https://docs.aws.amazon.com/whitepapers/latest/ipv6-on-aws/hybrid-connectivity-design.html

Dual-Stack Decentralised Ingress/Egress Patterns

Public Facing Resources / Public Subnets

Resources with IPv6 Global-Unicast Addresses (GUAs) function identical to resources with public IPv4, thus their networking structures are also fairly similar. Resources with IPv6 GUA Addresses (similar to IPv4 Public Addresses) are publicly routable addresses, thus can route directly to the Internet. Internet Ingress/Egress traverses an Internet Gateway (IGW) and a single IGW supports both IPv4 and IPv6 traffic, there is no additional configuration on the IGW that is required to support this. Similar to IPv4 where you need to add a default route (/0) destined for the Internet Gateway, you should add a default (::/0) route for IPv6-destined traffic with the next hop as the (same) Internet Gateway.

Internal-only and Outbound-only Resources / Private Subnets

In this section I will cover two types of Private resources, ones that are isolated with no inbound and outbound Internet access, and ones that are private with outbound only internet access.

With IPv4 resources, allocating an RFC1918 IPv4 Address makes them private given these are not publicly routable addresses and makes them isolated resources (assume you have no inbound reverse proxy etc.). To give egress-only access to an IPv4 private resource, you route the traffic through a NAT Gateway which provides one-way outbound access to the Internet.

In IPv6, the concept of Public/Private is slightly different given all resources within a VPC are given a Global-Unicast Address (GUA) which is publicly routable assuming these are advertised (or at least publicly routable capable).

To make an IPv6 resources isolated, you need to ensure there are no routes to an Internet Gateway given the address will likely be advertised. To implement Egress-Only Internet access for IPv6 resources, you can place them in a ‘private’ subnet, but route Internet-destined traffic through an Egress-Only Internet Gateway (EO-IGW). There is no use of NAT within an IPv6 environment. If you have dual-stack private resources, you can still route IPv4 traffic to a NAT Gateway, then onto a standard Internet Gateway.

Here is an example of a Private resource with dual-stack networking for egress-only Internet access:

Source: https://docs.aws.amazon.com/whitepapers/latest/ipv6-on-aws/amazon-vpc-internet-access.html

1. Access to IPv4 Public Addresses will use the EC2s IPv4 Address and route to the NAT Gateway via the default route (/0). From the NAT Gateway, traffic will be routed to the Internet Gateway.
2. Access to IPv6 Global Unicast Addresses will use the EC2s IPv6 Address and route directly to the Egress-only Internet Gateway. There is no need for NAT, and outside-initiated connections (inbound) are not permitted.

IPv6-only Decentralised Ingress/Egress Patterns and IPv4 Interoperability

The section above assumes that a resource is running dual-stack, thus has IPv4 and IPv6 addresses, but what if we only had an IPv6 address? How would we communicate with IPv4 only addresses (RFC1918 and Internet IPv4 Addresses)? In this section we will focus on AWS’s implementation of interoperability between IPv6 and IPv4.

While there are many techniques for interoperability, AWS uses the NAT64 + DNS64 approach. If you’re unfamiliar with this concept, please check out IPv6: Starter to Master for a detailed walkthrough.

AWS NAT Gateways support NAT64 by default, there is no option to configure this. All NAT Gateways, including existing ones support this even if you deployed it back in 2015 when NAT Gateways were first introduced. Using the well-known 64:ff9b::/96 prefix for IPv4-embedding, you can route this prefix to a NAT Gateway which performs NAT64 and the forward to either;

1. An Internet Gateway to access an IPv4-only resources on the Internet
2. The implicit router within the VPC for intra-VPC connectivity. This is automatically provided via the local route on the VPC route table.
3. Another resource (such as a Transit Gateway) for IPv4-only addresses outside the VPC.

Traffic to IPv6 endpoints should still be routed directly to the Internet Gateway via the default route (::/0) for native IPv6 connectivity. If your IPv6 resource is private, then the default route (::/0) should be destined to an Egress-Only Internet Gateway (EO-IGW). You must ensure that DNS64 is enabled on the subnet, which is disabled by default.

Below is an example of an IPv6-only resource accessing IPv4-only resources, both internal and external:

Source: https://docs.aws.amazon.com/whitepapers/latest/ipv6-on-aws/designing-dns-for-ipv6.html

In this example, you will see the 64:ff9b::/96 prefix being routed to NAT Gateway for NAT64 translation to access IPv4 addresses. This well-known prefix is returned with an embedded IPv4 Address by a DNS64 resolver if an AAAA record doesn’t exist, but an A record does.

This assumes that a DNS Record exists, for IPv4 literal addresses, you will need to consider other approaches such as 464XLAT, which is an extension of NAT64 including a client-side translation. More information can be found on IPv6: Starter to Master

Dual-Stack Centralised Ingress/Egress Patterns

IPv6 Centralised Ingress Patterns

This type of architecture is a little more complicated compared to IPv4 Centralised Ingress Architectures. The main reason being, you cannot route traffic from an Application Load Balancer to resources outside of its VPC (even to a static IPv6 address). While you can use a third-party reverse proxy to route to static IPv6 addresses outside the VPC, this is no use if you are using elastic resources such as Auto-Scaling groups that do not maintain fixed IPs, thus there is no native mechanism of referencing the backend targets.

A workaround to this is to use AWS Private Link which creates a VPC Endpoint local to the Centralised Ingress VPC which connects to a Network Load Balancer that fronts the application and the communication occurs via the AWS Backbone.

Source: IPv6 deployment models for AWS Network Firewall | Networking & Content Delivery (amazon.com)

In this example, AWS Network Firewall is used as a perimeter Firewall, but the same architecture is used without the Firewall. You simply do not associate an Edge route table to the Internet Gateway and allow direct communication between the Internet Gateway and Application Load Balancer.

Personally, I am not a great fan of IPv6 Centralised Ingress patterns as I think they’re a little complicated. I would recommend a decentralised ingress, and if you want to have a perimeter firewall, you can use AWS Network Firewall per VPC, or better use a WAF on an Application Load Balancer. AWS do not support creating Firewall endpoints outside of the VPC in which the Firewall is deployed in, thus you need a distributed Firewall set-up. Third Party Firewalls do support distributed set-ups using Private Link and Gateway Load Balancer Endpoints.

IPv6 East-West Inspection Patterns

Hybrid Connectivity – Direct Connect & Virtual Interfaces

The IP Protocol is a Layer 3 construct, thus many concepts related to the Direct Connects such as physical connections, link aggregation groups (LAGs), VLANs, and jumbo frame are no different from IPv4 use cases.

Where IPv6 does differ is when it comes to addressing and configuration of BGP peering’s on top of a virtual network interface (VIF). There are three types of VIFs:

• Private
• Transit
• Public

You can provision either 0 or 1 peering per address family on to a given VIF, so it is possible to retrofit IPv6 onto an existing VIF without the need to re-provision or deploy a new one.

Transit and Private VIF IPv6 peering’s — In IPv4 you are free to choose your own addressing for the logical point-to-point. In IPv6, AWS automatically allocates a /125 CIDR for each VIF, and it’s not possible to specify custom IPv6 addresses.

When connecting a Virtual Private Gateway and a Direct Connect Gateway, you use the concept of ‘allowed prefixes’ to publish a summary route from the Direct Connect Gateway to on-premise routers. There is a slight change in behaviour with IPv4 and IPv6 allowed prefix lists. In IPv4, not specifying a prefix means no filter is applied, thus 0.0.0.0/0 is published. In IPv6, it is the opposite, not specifying a prefix means a full filter is applied thus its action is an implicit deny.

With Transit Gateways, the behaviour of allowed prefixes of the Direct Connect Gateways remain the same, acting as a static origination of routes instead of a filter.

An “Allowed Prefix” IPv6 CIDR must be of length /64 or less specific, but you cannot specify “::/0”.

In terms of advertising prefixes from your on-premises facility to AWS, there are no CIDR restrictions, though standard Direct Connect quotas apply. Any prefix advertised is local to your AWS network and not propagated beyond its boundary.

If you’re using AWS-assigned CIDR blocks for the VPCs, or BYOIPv6 with CIDRs that are marked as advertisable by AWS, you will receive over the public VIF peering the summary prefixes that contain the CIDRs of your VPCs. When using public VIFs in conjunction with private/transit VIFs, take into account that your device will receive the same prefixes (the ones for your VPCs) over both types of VIFs. At this point, route filtering on your customer device needs to be taken into consideration, to ensure symmetric flow of traffic over the different virtual interfaces.