MPLS Network Design with Four Labels Stacked

The other day I was troubleshooting a particular MPLS network and I was wondering if it was supposed to have either three labels or four labels stacked in that design. As it turns out, three labels is the normal situation, i.e. when there are no network failure events, and four labels when RSVP link-protection (facility backup) kicks in. If you don’t need a detailed explanation about this design and configuration, just jump straight to Table 1, you’ll probably find what you want, enjoy!

This MPLS network design has both RSVP and LDP enabled as illustrated in Figure 1. RSVP is running in the core (vMX1 through vMX6), for LSP traffic engineering, and all PEs (vMX7 and vMX8) run LDP as the MPLS infrastructure protocol and they provide L3VPN as a service. To make these LDP LSPs end-to-end, vMX1 and vMX6 are tunneling LDP over RSVP. The third label in the MPLS stack appers as a result of this LDP tunneling. As you’ll see shortly, the fourth label will appear in the label stack when the link being protected by facility backup fails, which is the link ge-0/0/0.45 in orange between vMX4 and vMX5 - Figure 1. Table 1 summarizes the purpose of each label involved in this MPLS network:

MPLS Label Stack Reason why the label is pushed
1st innermost label (L3VPN) L3VPN vpnv4 label between vMX7 and vMX8
2nd label (LDP) LDP LSP between PE routers (vMX7 and vMX8) and P routers, directly connected to PEs, for example, vMX1 and vMX6
3rd label (RSVP) RSVP LSP with LDP tunneling. It tunnels the 2nd label by swaping the LDP label and pushing the RSVP label on top of that, e.g., RSVP LSP VMX1_TO_VMX6 and RSVP LSP VMX6_TO_VMX1 (Figure 1)
4th outermost label (RSVP Facility Backup) Bypass RSVP LSP, facility backup. Another RSVP label is pushed on top of the 3rd label, when the downstream protected link fails
Table 1 - MPLS label stack in this network topology.

Info
If fast-reroute were set instead of facility backup (i.e. bypass tunnels/LSPs), the fourth label wouldn’t show up because detour tunnels/LSPs don’t push another label in the stack. More details on 1.

Topology

Figure 1 shows the topology, basically, it is composed of LDP domains tunneled over an RSVP MPLS Core with facility backup (orange link down bellow) between vMX4 and vMX5.

Figure 1

Configuration

The following sections show the configuration of vMX1 (P1), vMX2 (P2), vMX7 (PE1), vMX8 (PE2) and specific configuration of vMX4 and vMX5.

Facility Backup Junos on vMX4 and vMX5

Facility backup link-protection is enabled on ge-0/0/0.45 between vXM4 and vMX5:

1
2
3
4
root@vMX5# show protocols rsvp | display set
set protocols rsvp interface ge-0/0/0.45 link-protection
set protocols rsvp interface ge-0/0/0.25
set protocols rsvp interface ge-0/0/0.56
1
2
3
4
root@vMX4# show protocols rsvp | display set
set protocols rsvp interface ge-0/0/0.14
set protocols rsvp interface ge-0/0/0.45 link-protection
set protocols rsvp interface ge-0/0/0.24

Junos vMX1 (P1)

LDP tunneling is enabled by setting ldp-tunneling under the VMX1_TO_VMX6 LSP. Notice that LDP are only enabled on PE facing interfaces and loopbacks. Conversely, since this core is meant to run RSVP, P facing interfaces have RSVP enabled. vMX1 requests link-protection downstream by setting this attribute under [protocols mpls label-switched-patch VMX1_TO_VMX6].

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
set system host-name vMX1
set interfaces ge-0/0/0 vlan-tagging
set interfaces ge-0/0/0 unit 12 vlan-id 12
set interfaces ge-0/0/0 unit 12 family inet address 10.1.2.1/24
set interfaces ge-0/0/0 unit 12 family mpls
set interfaces ge-0/0/0 unit 14 vlan-id 14
set interfaces ge-0/0/0 unit 14 family inet address 10.1.4.1/24
set interfaces ge-0/0/0 unit 14 family mpls
set interfaces ge-0/0/0 unit 17 vlan-id 17
set interfaces ge-0/0/0 unit 17 family inet address 10.1.7.1/24
set interfaces ge-0/0/0 unit 17 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 77.77.77.1/24
set interfaces lo0 unit 0 family inet address 1.1.1.1/32
set routing-options autonomous-system 100
set protocols rsvp interface ge-0/0/0.14
set protocols rsvp interface ge-0/0/0.12
set protocols mpls label-switched-path VMX1_TO_VMX6 to 6.6.6.6
set protocols mpls label-switched-path VMX1_TO_VMX6 ldp-tunneling
set protocols mpls label-switched-path VMX1_TO_VMX6 link-protection
set protocols mpls label-switched-path VMX1_TO_VMX6 primary PATH_TO_VMX6
set protocols mpls path PATH_TO_VMX6 4.4.4.4 strict
set protocols mpls path PATH_TO_VMX6 5.5.5.5 strict
set protocols mpls interface ge-0/0/0.14
set protocols mpls interface ge-0/0/0.12
set protocols mpls interface ge-0/0/0.17
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface ge-0/0/0.14 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/0.12 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/0.17 interface-type p2p
set protocols ospf area 0.0.0.0 interface lo0.0 passive
set protocols ldp interface ge-0/0/0.17
set protocols ldp interface lo0.0

Junos vMX6 (P6)

vMX6 (P6) has a similar configuration to vMX1 (P1) (more details explained in the previous subsection), except the RSVP LSP VMX6_TO_VMX1 goes in the opposite direction of VMX1_TO_VMX6. Check Figure 1 out for a visual representation of this difference.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
set system host-name vMX6
set interfaces ge-0/0/0 vlan-tagging
set interfaces ge-0/0/0 unit 36 vlan-id 36
set interfaces ge-0/0/0 unit 36 family inet address 10.3.6.6/24
set interfaces ge-0/0/0 unit 36 family mpls
set interfaces ge-0/0/0 unit 56 vlan-id 56
set interfaces ge-0/0/0 unit 56 family inet address 10.5.6.6/24
set interfaces ge-0/0/0 unit 56 family mpls
set interfaces ge-0/0/0 unit 68 vlan-id 68
set interfaces ge-0/0/0 unit 68 family inet address 10.6.8.6/24
set interfaces ge-0/0/0 unit 68 family mpls
set interfaces lo0 unit 0 family inet address 6.6.6.6/32
set protocols rsvp interface ge-0/0/0.56
set protocols rsvp interface ge-0/0/0.36
set protocols mpls label-switched-path VMX6_TO_VMX1 to 1.1.1.1
set protocols mpls label-switched-path VMX6_TO_VMX1 ldp-tunneling
set protocols mpls label-switched-path VMX6_TO_VMX1 link-protection
set protocols mpls label-switched-path VMX6_TO_VMX1 primary PATH_TO_VMX1
set protocols mpls path PATH_TO_VMX1 5.5.5.5 strict
set protocols mpls path PATH_TO_VMX1 4.4.4.4 strict
set protocols mpls interface ge-0/0/0.56
set protocols mpls interface ge-0/0/0.36
set protocols mpls interface ge-0/0/0.68
set protocols ospf traffic-engineering
set protocols ospf area 0.0.0.0 interface ge-0/0/0.56 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/0.36 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/0.68 interface-type p2p
set protocols ospf area 0.0.0.0 interface lo0.0 passive
set protocols ldp interface ge-0/0/0.68
set protocols ldp interface lo0.0

Junos vMX7 (PE1)

vMX7 (PE1) is running LDP exclusively and L3VPN as a service, peering vpnv4 with vMX8 (PE2). Plus, a connected route in the Customer1 VRF will be annouced for checking connectivity.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
set system host-name vMX7
set interfaces ge-0/0/0 vlan-tagging
set interfaces ge-0/0/0 unit 17 vlan-id 17
set interfaces ge-0/0/0 unit 17 family inet address 10.1.7.7/24
set interfaces ge-0/0/0 unit 17 family mpls
set interfaces ge-0/0/0 unit 47 vlan-id 47
set interfaces ge-0/0/0 unit 47 family inet address 10.4.7.7/24
set interfaces ge-0/0/0 unit 47 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 77.77.77.1/24
set interfaces lo0 unit 0 family inet address 7.7.7.7/32
set routing-options autonomous-system 100
set protocols mpls interface ge-0/0/0.17
set protocols mpls interface ge-0/0/0.47
set protocols bgp group internal_vpnv4 type internal
set protocols bgp group internal_vpnv4 local-address 7.7.7.7
set protocols bgp group internal_vpnv4 family inet-vpn unicast
set protocols bgp group internal_vpnv4 neighbor 8.8.8.8
set protocols ospf area 0.0.0.0 interface ge-0/0/0.17 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/0.47 interface-type p2p
set protocols ospf area 0.0.0.0 interface lo0.0 passive
set protocols ldp interface ge-0/0/0.17
set protocols ldp interface ge-0/0/0.47
set protocols ldp interface lo0.0
set routing-instances Customer1 instance-type vrf
set routing-instances Customer1 interface ge-0/0/1.0
set routing-instances Customer1 route-distinguisher 7.7.7.7:100
set routing-instances Customer1 vrf-target target:65000:1
set routing-instances Customer1 vrf-table-label

Junos vMX8 (PE2)

vMX8 (PE2) pretty much mirrors vMX1 (PE1)'s configuration, more details on the previous subsection.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
set system host-name vMX8
set interfaces ge-0/0/0 vlan-tagging
set interfaces ge-0/0/0 unit 38 vlan-id 38
set interfaces ge-0/0/0 unit 38 family inet address 10.3.8.8/24
set interfaces ge-0/0/0 unit 38 family mpls
set interfaces ge-0/0/0 unit 68 vlan-id 68
set interfaces ge-0/0/0 unit 68 family inet address 10.6.8.8/24
set interfaces ge-0/0/0 unit 68 family mpls
set interfaces ge-0/0/1 unit 0 family inet address 88.88.88.1/24
set interfaces lo0 unit 0 family inet address 8.8.8.8/32
set routing-options autonomous-system 100
set protocols mpls interface ge-0/0/0.38
set protocols mpls interface ge-0/0/0.68
set protocols bgp group internal_vpnv4 type internal
set protocols bgp group internal_vpnv4 local-address 8.8.8.8
set protocols bgp group internal_vpnv4 family inet-vpn unicast
set protocols bgp group internal_vpnv4 neighbor 7.7.7.7
set protocols ospf area 0.0.0.0 interface ge-0/0/0.38 interface-type p2p
set protocols ospf area 0.0.0.0 interface ge-0/0/0.68 interface-type p2p
set protocols ospf area 0.0.0.0 interface lo0.0 passive
set protocols ldp interface ge-0/0/0.38
set protocols ldp interface ge-0/0/0.68
set protocols ldp interface lo0.0
set routing-instances Customer1 instance-type vrf
set routing-instances Customer1 interface ge-0/0/1.0
set routing-instances Customer1 route-distinguisher 8.8.8.8:100
set routing-instances Customer1 vrf-target target:65000:1
set routing-instances Customer1 vrf-table-label

Packet Capture

The following sections will show the MPLS label stack of this network with packet captures.

Initial state no network event failures

Initially, assuming we haven’t had any network failure events, everything is UP, we’d normally see three labels in the MPLS stack (RSVP/LDP/L3VPN), as you can see in Figure 2, which shows a particular ICMP capture from a host in the VRF CustomerA of the MPLS L3VPN:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
root@vMX7> show route table bgp.l3vpn.0
bgp.l3vpn.0: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
8.8.8.8:100:88.88.88.0/24
*[BGP/170] 00:06:09, localpref 100, from 8.8.8.8
AS path: I, validation-state: unverified
> to 10.1.7.1 via ge-0/0/0.17, Push 16, Push 299808(top)
root@vMX7> show route table Customer1.inet.0
Customer1.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
77.77.77.0/24 *[Direct/0] 00:07:11
> via ge-0/0/1.0
77.77.77.1/32 *[Local/0] 00:07:12
Local via ge-0/0/1.0
88.88.88.0/24 *[BGP/170] 00:06:16, localpref 100, from 8.8.8.8
AS path: I, validation-state: unverified
> to 10.1.7.1 via ge-0/0/0.17, Push 16, Push 299808(top)
1
2
3
4
5
6
root@vMX7> ping routing-instance Customer1 88.88.88.1 rapid
PING 88.88.88.1 (88.88.88.1): 56 data bytes
!!!!!
--- 88.88.88.1 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 49.178/56.957/71.660/9.214 ms

vMX7, in the VRF, pushes two labels 299808/16. When this packet comes to vMX1,
as you can see bellow, label 299808 is swapped with 299776 and 299792 is pushed on top, Figure 2 shows this packet after vMX1 performs this push operation. So the final result, with tree labels is stacked like this 299792/299776/16:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
root@vMX1> show route forwarding-table label 299808
Routing table: default.mpls
MPLS:
Destination Type RtRef Next hop Type Index NhRef Netif
299808 user 0 10.1.4.4 Swap 299776, Push 299792(top) 578 2 ge-0/0/0.14
Routing table: __mpls-oam__.mpls
MPLS:
Destination Type RtRef Next hop Type Index NhRef Netif
default perm 0 dscd 529 1
root@vMX1> show route table inet.3 6.6.6.6 detail
inet.3: 5 destinations, 8 routes (3 active, 0 holddown, 4 hidden)
6.6.6.6/32 (3 entries, 2 announced)
State: <FlashAll>
*RSVP Preference: 7/1
Next hop type: Router
Address: 0x9740d20
Next-hop reference count: 7
Next hop: 10.1.4.4 via ge-0/0/0.14, selected
Label-switched-path VMX1_TO_VMX6
Label operation: Push 299792
Label TTL action: prop-ttl
Load balance label: Label 299792: None;
Session Id: 0x0
State: <Active Int>
Local AS: 100
Age: 25:50 Metric: 3
Validation State: unverified
Task: RSVP
Announcement bits (2): 0-OSPF 2-Resolve tree 1
AS path: I
LDP Preference: 9
Next hop type: Router
root@vMX1>

Figure 2

Failure State - When the RSVP protected interface is disabled

First, let’s disable ge-0/0/0.45 on vMX5 to simulate this network failure to see the fourth label in the stack, before I do that, let’s double check that the bypass tunnel have been precomputed successfully by the CSPF on both vMX4 and vMX5:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
root@vMX4> show mpls lsp
Ingress LSP: 0 sessions
Total 0 displayed, Up 0, Down 0
Egress LSP: 1 sessions
To From State Rt Style Labelin Labelout LSPname
4.4.4.4 5.5.5.5 Up 0 1 SE 3 - Bypass->10.4.5.4
Total 1 displayed, Up 1, Down 0
Transit LSP: 2 sessions
To From State Rt Style Labelin Labelout LSPname
1.1.1.1 6.6.6.6 Up 0 1 SE 299776 3 VMX6_TO_VMX1
6.6.6.6 1.1.1.1 Up 0 1 SE 299792 299792 VMX1_TO_VMX6
Total 2 displayed, Up 2, Down 0
root@vMX4>
root@vMX5> show mpls lsp
Ingress LSP: 0 sessions
Total 0 displayed, Up 0, Down 0
Egress LSP: 1 sessions
To From State Rt Style Labelin Labelout LSPname
5.5.5.5 4.4.4.4 Up 0 1 SE 3 - Bypass->10.4.5.5
Total 1 displayed, Up 1, Down 0
Transit LSP: 2 sessions
To From State Rt Style Labelin Labelout LSPname
1.1.1.1 6.6.6.6 Up 0 1 SE 299776 299776 VMX6_TO_VMX1
6.6.6.6 1.1.1.1 Up 0 1 SE 299792 3 VMX1_TO_VMX6
Total 2 displayed, Up 2, Down 0
root@vMX5>

We’re all set. As you can see bellow, since the link between vMX4 and vMX5 is gone, vMX4 will sent traffic through vMX2 using the bypass LSP Bypass->10.4.5.5, which pushes another RSVP label 299792 on top of that. The final MPLS stack, in this particular link, will be (RSVP_Bypass/RSVP/LDP/L3VPN) 299792/299792/299796/16 as shown in Figure 3

1
2
3
4
5
6
7
8
root@vMX5> configure
Entering configuration mode
[edit]
root@vMX5# set interfaces ge-0/0/0.45 disable
[edit]
root@vMX5# commit
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
root@vMX4> show route table mpls.0 label 299792
mpls.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both
299792 *[RSVP/7/1] 00:52:44, metric 1
> to 10.4.5.5 via ge-0/0/0.45, label-switched-path VMX1_TO_VMX6
to 10.2.4.2 via ge-0/0/0.24, label-switched-path Bypass->10.4.5.5
root@vMX4> show route table mpls.0 label 299792 detail
mpls.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
299792 (1 entry, 1 announced)
*RSVP Preference: 7/1
Next hop type: Router
Address: 0x96a8154
Next-hop reference count: 1
Next hop: 10.4.5.5 via ge-0/0/0.45 weight 0x1, selected
Label-switched-path VMX1_TO_VMX6
Label operation: Swap 299792
Load balance label: Label 299792: None;
Session Id: 0x0
Next hop: 10.2.4.2 via ge-0/0/0.24 weight 0x8001
Label-switched-path Bypass->10.4.5.5
Label operation: Swap 299792, Push 299792(top)
Label TTL action: prop-ttl, prop-ttl(top)
Load balance label: Label 299792: None; Label 299792: None;
Session Id: 0x0
State: <Active Int Accounting>
Age: 52:48 Metric: 1
Validation State: unverified
Task: RSVP
Announcement bits (1): 0-KRT
AS path: I
root@vMX4>

Figure 3

References