01 Mar Azure ExpressRoute – All you Need to Know
As all roads lead to the cloud, the cloud needs clear expressway for your data and apps to move – both in transit (during big ticket migrations) as well as steady state (when you have a Hybrid estate for a foreseeable future and you want seamless connectivity between the cloud and on-prem).
Quoting directly from Microsoft documentation – “ExpressRoute lets you extend your on-premises networks into the Microsoft cloud over a private connection with the help of a connectivity provider. With ExpressRoute, you can establish connections to Microsoft cloud services, such as Microsoft Azure and Microsoft 365.
Connectivity can be from an any-to-any (IP VPN) network, a point-to-point Ethernet network, or a virtual cross-connection through a connectivity provider at a colocation facility. ExpressRoute connections don’t go over the public Internet. This allows ExpressRoute connections to offer more reliability, faster speeds, consistent latencies, and higher security than typical connections over the Internet.”
The reason this topic made it to our list of blogs we’d like to share is because:
- Getting ExpressRoute (ER) deployed, requires the organisation to make some critical decisions beforehand.
- Involves multiple moving parts and working with various stakeholders to get it going.
- While I worked on my first engagement (back couple of years), it took me a while to get comfortable with the terminology used in official documentation. For example, difference between an ER circuit and ER connection. Through this series or articles, I’ll try to clarify that including a list all the components involved in an ExpressRoute (ER) path that need to be looked at when designing highly available and redundant pair of ER circuits.
- Will be honest here! Initially, I struggled with it a lot – there’s simply too much on the plate to eat. However, I believe I’ve got enough now to be able to help others – pay it forward!
Confession: there’s some really great documentation already in existence on this topic, including the official Microsoft Azure document library. However, we wanted to create a targeted, well-structured, and simplified version for a ready reckoner! This article will reference (and credit) back to appropriate Microsoft ExpressRoute documentation.
Introduction to Azure ExpressRoute
When an organisation hosts services and application in Azure, Azure is considered as a logical extension of their on-premises network. Multiple options exist to establish network connectivity between Azure and the on-premises environments. The topology that results from establishing this connectivity is commonly known as a hybrid cloud (combination of private and public cloud environments). Network connectivity is required between on-premises data center(s) and Azure to enable seamless access to applications / resources across the hybrid cloud.
Microsoft Azure offers the following three connectivity options between on-prem environments and Azure:
1. ExpressRoute (ER): A dedicated, private, high-speed connection facilitated by a connectivity provider. Traffic does not traverse over the Internet. Connection speed ranges from 50 Mbps to 10 Gbps (100 Gbps with ExpressRoute Direct).
Important: An ExpressRoute circuit represents a logical (not physical) connection between your on-premises infrastructure and Microsoft cloud services through a connectivity provider (ExpressRoute circuits do not map to a physical entities – a single ER circuit has built-in availability, which means dual connections).
2. Site-to-Site (S2S) VPN: An IPsec (IKE v1 and IKE v2) encrypted connection over Internet to enable connectivity between on-premises network to Azure VNet. This type of connectivity is limited by number of connections and throughput (ranging between 100 Mbps to 1 Gbps) depending on the Azure VPN Gateway SKU is used. This type of connectivity requires a VPN gateway or Routing and Remote Access Services (RRAS) on the on-premises side.
3. Point-to-Site (P2S) VPN: A VPN connection over SSTP (Secure Socket Tunnelling Protocol) or IKE v2 that facilitates connectivity from an individual client computer (e.g., working from hone scenarios) to Azure VNet.
The focus of this article (as you would have probably guessed already) is ExpressRoute connectivity type, and that too very specific to Private peering – more on peering types, later in this article. Microsoft peering deserves a dedicated article, and I don’t dare take that away from it.
First thing first, when starting to plan the ExpressRoute connectivity, the articles here and here quickly get you started by helping you visualize the ER provisioning workflow:
There’s still some planning required outside of what’s discussed above – hold on to you seats…
Planning ExpressRoute Connectivity
Before ordering ExpressRoute connectivity, an organization needs to plan the following areas:
- ExpressRoute circuit peering / meet-me location for primary as well as disaster recovery sites
- ER Provider of choice in the chosen peering region(s)
- Type of ER connectivity required – Regular Vs ER Direct
- ER SKU – Local Vs Standard Vs Premium
- ER Data Plan / Billing Model – Metered Vs Unlimited
- ER circuit bandwidth
- Additional features like – ER GlobalReach Add-on
- ER Gateway SKU and performance to match the ER circuit bandwidth
- Gateway subnet sizing
- Public IP address resource for the Gateway VMs
- Naming convention for the Azure components needed to enable ER connectivity
- Investigate the on-prem router / firewall for appropriate number and type of ports available to terminate ER connectivity into
- IP Schema required to enable ER connectivity – Subnets and VLAN ID to be used while creating peering
- BGP route advertisements to be done over each ER circuit and failover config (AS Path prepend / More specific route / Connection weight)
- Lead time to lay out the physical connectivity – this is quite important, especially, if the organization is running against time
This series of articles will only cover the most important aspects from then list above.
ExpressRoute SKU and Pricing
When it comes to ER circuit SKUs, there are myriad of options, which is good, but it takes some working-out to choosing the right for yourself.
Table below describes the available ExpressRoute SKUs:
ER Circuit SKU
Description
Limitations (if any)
Standard
A Standard ER circuit gives customers access to all Azure regions in a geopolitical area (e.g., Europe, USA)
Max. number of VNets allowed to connect: 10
Max. number of routes advertised to Azure private peering: 4000
Max. number of routes advertised to Microsoft peering: 200
Premium
A Premium ER circuit allows access to all Azure regions (across a geopolitical area) globally.
ExpressRoute premium charges apply on top of ExpressRoute standard circuit charges (it’s an add-on)
Max. number of VNets allowed to connect: Ranging between 20 to 100 depending on circuit speed.
Max. number of routes advertised to Azure private peering: 10000
Max. number of routes advertised to Microsoft peering: 200
Local
A Local ER circuit at an ER peering location gives customers access only to one or two Azure regions in or near the same metro. The main advantage of Local SKU is that customers don’t pay egress charges as long as they use ER circuit with speeds between 50 Mbps to 500 Mbps.
It is an economical solution if customers have massive amount of data to transfer that they can bring over a private connection to an ExpressRoute peering location near their desired Azure regions.
Number of VNets allowed to connect: 10
Customers can only advertise routes (over Microsoft and private peering) from the corresponding local region of the ExpressRoute circuit. They won’t be able to receive routes for other regions different than the defined Local region.
Obviously, ER Global Reach features is not available on Local SKU.
It is not available at a peering location where there is no Azure region in that state or province or country/region.
Note: When BGP sessions exceed the BGP limits, the excess sessions will be dropped. They will be reset once the prefix count goes below the limit.
Remember, the ExpressRoute cost has following two components:
1. ER provider fee: Check with you provider about this component.
2. Microsoft ER fee:
- A fixed monthly port fee (dual ports for HA / circuit)
- Data transfer fee
Table below is a quick summary of options available to Azure customers who are looking to deploy Azure ExpressRoute service:
Microsoft allows increasing the bandwidth of an existing ExpressRoute circuit (on a best effort basis) via the Azure portal, or by using PowerShell. If there is capacity available on the physical port on which customers’ circuit was created, the change succeeds. However, if change fails, it means either there isn’t enough capacity left on the current port and the customer needs to create a new ExpressRoute circuit with the higher bandwidth, or that there is no additional capacity at that location, in which case customers can’t increase the bandwidth.
If the bandwidth change succeeds, customers will also need to follow up with the connectivity provider to ensure that they update the throttles within their networks to support the bandwidth increase. Please note that bandwidth of the ExpressRoute circuit cannot be reduced – if that’s required, customers can create a new ExpressRoute circuit with lower bandwidth and delete the old circuit.
Note: ExpressRoute circuits are configured to allow a burst up to two times the bandwidth limit procured for no additional cost (by using the bandwidth available on the secondary connection of your ExpressRoute circuit). Customers need to check with their service provider to see if they support this capability.
Important: Certain properties like bandwidth, SKU, and billing model of an ExpressRoute circuit can be modified without impacting connectivity. However, the following points need to be kept in mind:
- Bandwidth of an ExpressRoute circuit can only be increased if there is capacity available on the port. Downgrading the bandwidth of a circuit is not supported.
- ExpressRoute data plan can be changed from Metered Data to Unlimited Data. However, changing from Unlimited Data to Metered Data is not supported – the existing circuit needs to be deleted and a new one created with metered pricing plan.