08 Mar Azure ExpressRoute – Part 2
This is the second part of the multi-part series of articles where we’re going to review some of the most important aspects of Microsoft Azure ExpressRoute service.
In part 1 of this article series, we looked at the ‘what’ aspect of the ExpressRoute service. This article will continue to do that and will look at expanding on what we learned earlier.
ExpressRoute Location
Each ExpressRoute circuit has location (sometimes called peering location / meet-me-location), which is the co-location facilities where Microsoft Enterprise Edge (MSEE) devices are located. ExpressRoute locations are the entry point to Microsoft’s network and are globally distributed, enabling customers to connect to Microsoft’s network around the world. These locations are where ExpressRoute partners and ExpressRoute Direct customers issue cross connections to Microsoft’s network.
In general, the ExpressRoute location does not need to match the Azure region – Azure regions and ExpressRoute locations are two distinct and different concepts. The ExpressRoute circuits peering / meet-me-location is chosen based on the physical location of customer’s on-prem data centers.
Here’s the Azure region to ExpressRoute peering locations mapping within Azure geopolitical region.
Customers can access Azure services across all regions within a geopolitical region if they connect to at least one ExpressRoute location within the geopolitical region.
When ordering multiple ExpressRoute circuits (parallel paths), customers have following choices, each has its own advantage and downside:
ExpressRoute Location
Same Metro
If a metro has multiple ExpressRoute peering locations (e.g., Sydney and Sydney2 or Amsterdam and Amsterdam2) and the circuits are created at different peering locations (preferred), they can be linked to the same VNet in Azure. The advantage of this design is when application failover happens, end-to-end latency between your on-premises applications and Microsoft stays approximately the same. However, if there is a natural disaster such as an earthquake, connectivity for both paths may no longer be available.
Different Metros (Same Geo-political region)
This design sits in the middle of previous and the next option, offering best of the two worlds. Of course, it boils down to the customer requirements and what best addresses those.
Different Geopolitical Region
To choose an ER location outside of the geo-political region requires ER Premium SKU for both circuits in the parallel paths. The advantage of this configuration is the chances of a natural disaster causing an outage to both links are much lower but at the cost of increased latency end-to-end.
Note: The redundant ER circuits can be ordered via different (or same) service providers. Ordering via different provider, theoretically makes it more resilient to provider-level issues.
Note: The “Peering / Meet-me Location” indicates the physical location where customer / connectivity provider peer with Microsoft. This is not linked to “Location” property of the ExpressRoute object created in Azure portal, which refers to the geography where the Azure Network Resource Provider is located. While they are not related, it is a good practice to choose a Network Resource Provider geographically close to the Peering Location of the circuit.
ExpressRoute Connectivity Types
Microsoft rely on partnering with Service (connectivity) Providers to create BGP peering (Routing Domains) to enable customer connectivity to Azure. Azure ER Connectivity can be accomplished in any of the four ways:
- Cloud Exchange Colocation (Virtual Cross-connect): Customer with data centers co-located in a facility with a cloud exchange, can order virtual cross-connections to the Microsoft cloud through the co-location provider’s Ethernet exchange. Colocation Provider can offer either Layer 2 cross-connections, or managed Layer 3 cross-connections between customer’s data center in the co-location facility and the Microsoft cloud.
- Point-to-point Ethernet Network: A direct Ethernet connection by the Exchange Provider connects the customer data center to Microsoft Azure. Point-to-point Ethernet providers (Exchange Providers) can offer Layer 2 connections, or managed Layer 3 connections between customer’s data center and the Microsoft cloud.
- Any-to-any (MPLS / IP VPN) Network: Leverages Multiprotocol Label Switching (MPLS) technology to provide WAN connectivity between the customer network and Microsoft Azure. Azure is interconnected to customer’s WAN just like any other branch office. WAN providers (Network Service Providers) typically offer managed Layer 3 connectivity.
- ExpressRoute Direct: Customer’s connect directly into the Microsoft’s global network at a peering location distributed across the world.
Note: Microsoft do not support layer 2 (VLANs) connectivity extensions into Azure using ExpressRoute.
ExpressRoute Connectivity Elements
ExpressRoute connectivity involves number of parties and quite a number of connectivity elements, each under the control of different parties. Understanding these elements is critical to not just deployment, but the operational management of the ER circuit(s).
ExpressRoute connectivity traditionally involves three distinct network zones, as follows:
- Customer Network
- Provider Network
- Microsoft Data center
The following diagram shows the logical connectivity of a customer network to Microsoft network using ExpressRoute:
In the preceding diagram, the numbers indicate key network points. Depending on the ExpressRoute connectivity model (Cloud Exchange Co-location, Point-to-Point Ethernet Connection, or Any-to-any (IPVPN)), the network points 3 and 4 may be switches (Layer 2 devices) or routers (Layer 3 devices). In the direct connectivity model, there are no network points 3 and 4; instead CEs (2) are directly connected to MSEEs via dark fiber. The key network points illustrated are as follows:
- Customer compute device (for example, a server or PC)
- CEs: Customer edge routers
- PEs (CE facing): Provider edge routers/switches that are facing customer edge routers. Referred to as PE-CEs in this document.
- PEs (MSEE facing): Provider edge routers/switches that are facing MSEEs. Referred to as PE-MSEEs in this document.
- MSEEs: Microsoft Enterprise Edge (MSEE) ExpressRoute routers
- Virtual Network (VNet) Gateway
- Compute device on the Azure VNet
If the Cloud Exchange Co-location, Point-to-Point Ethernet, or direct connectivity models are used, CEs (2) establish BGP peering with MSEEs (5).
If the Any-to-any (IPVPN) connectivity model is used, PE-MSEEs (4) establish BGP peering with MSEEs (5). PE-MSEEs propagate the routes received from Microsoft back to the customer network via the IPVPN service provider network.
Note: For high availability, Microsoft establishes a fully redundant parallel connectivity between MSEEs (5) and PE-MSEEs (4) pairs. A fully redundant parallel network path is also encouraged between customer network and PE-CEs pair.
ExpressRoute Peering (Routing Domains)
Azure ExpressRoute supports two types of peering connections:
- Private Peering: Private peering is used to provide connectivity between customers’ on-premise networks and Azure VNets (IaaS services and PaaS services that have Private Endpoint enabled).
- Microsoft Peering: Microsoft peering is used to provide connectivity between on premise networks and Azure SaaS / PaaS services (public end-points), such as Office 365, Azure Active Directory. General recommendation from Microsoft is to access Office 365 services over the Internet – more here and here.
Note: Microsoft accepts any prefix size (up to /32) advertised over BGP sessions on both the Microsoft and the private peering.