AVS Routing Design Concern

Question

AVS Routing Design Concern

Ali El Husseiny 60

Hey AVS People,

I’d like to briefly explain my current setup and highlight a specific routing concern I have.

We have an AVS environment connected to an ExpressRoute gateway in a transit VNet, which also hosts a Route Server and a BGP NVA—both of which are peered with each other using BGP.

The transit VNet is peered with the hub VNet, with gateway transit disabled. In the hub, we have:

A Route Server (with an eBGP session established with the BGP NVA in the transit VNet)

A Perimeter Firewall

An ExpressRoute Gateway

A Core Firewall

Our on-premises connectivity is established via IPsec over ExpressRoute, terminating at the perimeter firewall in the hub.

My question: Traffic from AVS traverses through the transit VNet to the hub. In the hub, the Route Server and ExpressRoute Gateway establish an iBGP session by default (since they share the same ASN in Azure). However, since our on-prem traffic is actually reachable through the perimeter firewall via IPsec, how can I make sure that traffic from AVS destined for on-prem is routed to the perimeter firewall first or maybe core firewall first then perimeter not sure, rather than directly to the ER gateway?

Appreciate your help in clarifying this path.

Accepted answer

0 additional answers

Your answer

Answer 1

Thanmayi Godithi 230 Microsoft External Staff Moderator

Hi @Ali El Husseiny,

Thanks for reaching out on Microsoft Q&A forum.

I understand your AVS (Azure VMware Solution) environment is connected to your on-premises network via a transit virtual network (VNet), which peers with a hub VNet hosting critical network components such as a Route Server, perimeter firewalls, and an ExpressRoute Gateway. Your primary requirement is to ensure that traffic originating from AVS to on-premises flows through the designated perimeter firewall rather than taking the default path directly through the ExpressRoute Gateway, which may bypass your network inspection controls.

Based on your topology, there are several supported approaches to help enforce this traffic flow behaviour:

Option 1: User-Defined Routes (UDRs) on GatewaySubnet

One straightforward approach is to create UDRs on the GatewaySubnet in the hub VNet. Although GatewaySubnet does not support a 0.0.0.0/0 UDR for ExpressRoute, you can define specific UDRs for your on-premises address ranges. For instance, if your on-premises range is 192.168.0.0/16, you can create a UDR with that prefix and set the next hop type to "Virtual Appliance" with the next hop IP as the firewall's private IP address. Ensure that route propagation is enabled on the subnet to allow coexistence of both system and custom routes.

This will ensure that traffic from AVS is routed through the transit VNet and hub VNet to the firewall before reaching the on-premises network.

For more information, refer to the documentation on User-defined routes in Azure.

Option 2: BGP Route Advertisement with Custom Next Hop

Since you are using BGP-capable network virtual appliances (NVAs) in the transit VNet, you can leverage Azure Route Server’s ability to honour custom next-hop IPs. This approach allows the NVA to advertise on-premises prefixes to the Route Server, while explicitly setting the firewall IP as the next hop. The Route Server will then propagate these routes to other peers, and Azure will honour the custom next hop during route selection.

This dynamic routing approach ensures AVS traffic is routed via the firewall path instead of directly via ExpressRoute, while still maintaining flexibility and scalability.

You can find further details in Injecting default routes via Azure Route Server.

Option 3: Route Priority Manipulation

Understanding Azure’s route selection behaviour is essential. The platform always prefers the longest prefix match (LPM). When multiple routes have the same prefix length, Azure prioritizes them as follows: User-defined routes take precedence over BGP routes, which in turn take precedence over system routes.

You can use this to your advantage by either using more specific prefixes in UDRs or BGP advertisements, or by overriding broader ExpressRoute-learned routes through user-defined ones. This method is particularly effective when you need deterministic routing control.

Refer to Azure routing overview for more guidance.

Option 4: Configuring Route Server to Influence Routing Preference

Given that you are using Azure Route Servers in both the hub and transit VNets, it is recommended to configure the route server to prefer VPN/NVA paths over ExpressRoute by adjusting the routing preference setting. Additionally, applying AS path prepending to the ExpressRoute Gateway route advertisements will make those paths less preferred compared to those learned from the firewall via the NVA.

Ensure that the Route Server peers with distinct ASNs for the NVA and the ExpressRoute Gateway, and that the BGP sessions are configured correctly to support this preference behaviour.

Recommended Architecture Flow

For your specific setup, we recommend a combination of the above methods. In the transit VNet, configure your BGP NVA to advertise on-premises prefixes to the Route Server with the firewall’s IP as the next hop. In the hub VNet, allow the Route Server to propagate these routes to connected networks. As a fallback mechanism, apply UDRs on the GatewaySubnet for critical on-premises prefixes to statically enforce the firewall route in case dynamic advertisements are not available.

This results in the following desired traffic flow: AVS → Transit VNet → Hub VNet → Route Server → Perimeter Firewall → (Optional Core Firewall) → On-premises.

Additional Considerations

The iBGP session between the ExpressRoute Gateway and Route Server will still exist, but your custom routes will be preferred due to specificity or route type precedence.
Ensure that the "Use remote virtual network’s gateway or Route Server" setting is enabled on the VNet peering's between spoke, transit, and hub networks.
Use the “Effective Routes” view on test VMs to verify that routes are being resolved as expected.
ExpressRoute can still function as a backup route path if the primary firewall-inspected path is unavailable.

Kindly let us know if the above helps or you need further assistance on this issue.

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Ali El Husseiny 60 Reputation points

2025-07-30T13:13:21.93+00:00
Dear Thanmayi Godithi,

Thank you for your reply !

So just to confirm what I have understood :

Create a UDR (User-Defined Route) table with route propagation enabled, and assign it to the ExpressRoute Gateway subnet in the hub.

From the NVA in the Transit VNet, we advertise on-premises prefixes via BGP, setting the firewall’s private IP in the hub as the next hop. ( question here: cant we instead just create a UDR on the NVA outside interface saying that on prem traffic to go to the firewall ip in the hub as next hop and in this outside NIC do we need load balancer ?)

Since there's an eBGP session between the Transit NVA and the Route Server in the hub, the advertised routes propagate into the hub with the firewall IP as the next hop.

AVS → Transit VNet ( here we either advertise on prem or create udr to onprem if possible)→ Hub VNet → Route Server → Perimeter Firewall → (Optional Core Firewall) → On-premises.
Thanmayi Godithi 230 Reputation points Microsoft External Staff Moderator

2025-08-01T06:49:03.5+00:00
Hi @Ali El Husseiny,

Your grasp of the overall architecture flow is accurate:

UDR on ExpressRoute Gateway subnet with route propagation enabled

BGP advertisement from Transit NVA to Route Server with firewall IP as next hop

Route propagation from Transit to Hub Route Server via eBGP

Traffic flow: AVS → Transit VNet → Hub VNet → Route Server → Perimeter Firewall → On-premises

Addressing Your Question: UDR vs BGP Advertisement

Regarding your question about creating a UDR on the NVA's outside interface instead of BGP advertisement:

Short Answer: While technically possible, BGP advertisement is the preferred and more scalable approach.

Here's why:

UDR Approach Limitations:

Static Nature: UDRs are static and require manual updates when on-premises prefixes change

Scalability: Managing multiple on-premises prefixes becomes complex with individual UDRs

No Automatic Failover: If your NVA fails, UDRs don't provide automatic path selection

Route Propagation Issues: UDRs on the NVA don't automatically propagate to other VNets or Route Servers

BGP Advertisement Advantages:

Dynamic Updates: Routes are automatically updated when topology changes

Route Attributes: You can influence path selection using BGP attributes (AS-path prepending, communities, etc.)

Automatic Failover: BGP provides convergence and failover capabilities

Centralized Management: Route Server propagates routes to all connected VNets automatically

Load Balancer Question:

For the outside NIC of your NVA, you typically don't need a load balancer unless:

You're running multiple NVA instances for high availability

You need to distribute traffic across multiple firewall instances

Your design requires active-active NVA deployment

Best Practice Implementation:

Primary Method: Use BGP advertisement from your Transit NVA to Route Server

Backup Method: Create specific UDRs on the ExpressRoute Gateway subnet for critical on-premises prefixes

Route Priority: Remember that UDRs take precedence over BGP routes when prefixes match exactly

Hybrid Approach:

Option 1 (Preferred): BGP Advertisement Transit NVA → BGP to Route Server → Advertise on-prem prefixes with firewall next-hop

Option 2 (Backup/Specific Routes): UDR + BGP - Use BGP for dynamic route advertisement - Use UDRs for specific static routes that need special handling - Apply UDRs on critical subnets for guaranteed path enforcement.

Kindly let us know if the above helps or you need further assistance on this issue.

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.
Ali El Husseiny 60 Reputation points

2025-08-01T10:32:40.84+00:00

Thanmayi Godithi I am so thankful your prompt replies and follow up on my questions .

Lastly I will go with BGP advertisement on the transit NVA but here just to let you know that we are talking about two instances of NVA in the transit having ebgp with route server in the tranist and also in the hub so which mode is recommended to be chosen active/active or active/passive? and from both modes we are obliged to use a load balancer in the outside nic and why ?
Ali El Husseiny 60 Reputation points

2025-08-04T06:48:16.8666667+00:00

Thanmayi Godithi appreciated if I can have a response for my last comment so we can close this thread please

Thanmayi Godithi 230 Microsoft External Staff Moderator

Hi @Ali El Husseiny,

Thank you for confirming the direction. Since you're deploying two NVA instances in the Transit VNet, and each of them is establishing eBGP sessions with both the Route Server in the Transit VNet and the Route Server in the Hub VNet, your decision between active/active and active/passive mode is important for routing behaviour and availability.

In an active/active configuration, both NVAs are active simultaneously and participate in BGP route advertisement and traffic forwarding. This setup offers higher availability and improved throughput. However, a Standard Load Balancer is required on the external (outside) interface to distribute traffic evenly across both NVAs and to monitor their health. This ensures consistent traffic flow and failover in case one NVA becomes unavailable.

In an active/passive configuration, only one NVA handles traffic at any given time while the second instance remains on standby. In this setup, a load balancer is optional. You can route traffic directly to the active instance using UDRs or BGP policies, but you must implement a failover mechanism to switch to the standby NVA if needed—either via BGP preference changes or route updates.

Summary:

	Deployment Mode	NVA Usage	Load Balancer on Outside NIC	BGP Sessions	Failover Mechanism
	Active/Active	Both NVAs active	Required	Both NVAs to both RS	LB probes & BGP convergence
	Active/Passive	One NVA active, one standby	Optional	One NVA to both RS	BGP route update/manual

Recommendation:

Use active/active with a load balancer if your goal is high availability, better performance, and automatic failover.
Use active/passive without a load balancer if you prefer a simpler setup and can manage failover through routing adjustments or monitoring tools.

Kindly let us know if the above helps or you need further assistance on this issue.

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.

Ali El Husseiny 60 Reputation points

2025-08-04T18:04:04.8666667+00:00

Thanmayi Godithi thank you clear for the outside interface , the same applies for the inside of the transit bgp nva whether running active/active (i need also load balancer here) or active/passive (load balancer is optional)?
Thanmayi Godithi 230 Reputation points Microsoft External Staff Moderator

2025-08-05T05:38:15.5633333+00:00
Hi @Ali El Husseiny,

Yes, the same principle applies to the inside (internal) interfaces of your Transit NVAs.

If you're running active/active mode, you should deploy an internal Standard Load Balancer (ILB) in front of the NVAs' inside NICs. This ensures return traffic is consistently routed through the same NVA that handled the incoming traffic, helping maintain symmetry, enabling load distribution, and supporting automatic failover through health probes. This is especially important in scenarios where traffic returns from on-premises or other VNets to AVS or workloads inside Azure.

In active/passive mode, the internal load balancer is optional. You can use a UDR or BGP route preference to send traffic directly to the active NVA. However, you must implement a failover mechanism (such as BGP route preference change) to switch to the passive NVA during failure.

To summarize:

Active/Active → Internal Load Balancer is required for proper load distribution and session consistency.

Active/Passive → Internal Load Balancer is optional, but failover handling must be configured manually or via BGP.

Supporting References

Azure Route Server documentation outlines how to deploy NVAs behind an internal load balancer with Next Hop IP support for both active/active and active/passive topologies Microsoft Learn.

The Load Balancer HA ports guidance explains how internal Standard Load Balancer with HA ports can ensure flow symmetry and high-availability of NVAs Microsoft Learn.

Kindly let us know if the above helps or you need further assistance on this issue.

Please "Accept the answer" if the information helped you. This will help us and others in the community as well.
Ali El Husseiny 60 Reputation points

2025-08-05T09:07:03.89+00:00

Hi Thanmayi Godithi ,

the bgp nva can handle assymetric routing because we dont care here for session tracking ,etc because there is no inspection happening there. so does it mean we can ignore the load balancers in both sides?
Thanmayi Godithi 230 Reputation points Microsoft External Staff Moderator

2025-08-05T14:53:26.39+00:00
Hi @Ali El Husseiny,

Yes, if your BGP NVAs are fully stateless (no session or connection inspection/tracking) and do not require source-IP stickiness, you can typically avoid deploying load balancers on both sides, assuming all routing decisions are handled by BGP and not by UDRs (User Defined Routes).

Key Pointers:

If any subnet in your topology needs UDRs for routing to the NVAs, you’ll still need a load balancer as UDR can only point to a single IP (the LB frontend).

If you introduce stateful services or firewalls in the future, or require session tracking, you will need to reintroduce load balancers and symmetric routing.

Without a load balancer, failover is handled by withdrawal of BGP routes (when an NVA is down), so route-convergence/failover is dictated by BGP timers.

Summary: If your network only relies on BGP routing, is tolerant of asymmetric flows, and all NVAs are stateless, you can safely remove the load balancers and let BGP handle high availability and path selection. Review your environment to make sure there are no leftover UDRs or stateful dependencies before removing the LBs.

Kindly let us know if the above helps or you need further assistance on this issue.

Please "Accept the above answer" if the information helped you. This will help us and others in the community as well.

Share via

AVS Routing Design Concern

0 additional answers

Your answer