Thursday, September 26, 2013

Adventures with DMVPN

As part of a new deployment at work, I am about to roll out a DMVPN network.  I am not a security guy, although I've dabbled a bit with IPSec VPNs, so I decided I needed to do some labbing on my GNS3 setup.  If anyone has any comments or suggestions, I'd really appreciate them.

So here is the situation.  I have some remote sites that need to connect to VDI infrastructure at the Head Office.  The sites will have a thin client, and only need access to two subnets.  The Head Office is behind a managed firewall (a 1921 router) that provides NAT.  This router is not accessible - it is managed by our provider.

Finally, for HA I've decided to use two DMVPN clouds (i.e. each spoke connects one tunnel to each hub, and failover is done using routing - in my case EIGRP - and HSRP).

Below is the diagram of the topology I built.


Note that the switches are there so I can visualize the connections; in reality, VLANs on L3 switches would be used.

On the left, is my "Head Office" with the subnets 10.0.8.0/24 and 10.0.108.0/24 that my sites need to connect to.  These subnets have a default route out to the "Internet" in 10.0.1.0/24.  Routers R1 and R2 are my DMVPN hubs with and "Outside" interface on 10.0.2.0/24 and and "Inside" on 10.0.1.0/24.

R4 is the provider's router, connecting the Head Office to the Internet.  R5 represents the "Internet" and my spoke sites are on the right.  The spokes never need to talk to each other.

Here we go.

First, we set up the basics in the head office.

R3:

!
! Interfaces for DC subnets (10.0.8.x and 10.0.108.x) and the outbound facing interface (10.0.1.x)
!
interface FastEthernet0/0
 ip address 10.0.1.2 255.255.255.0
 duplex auto
 speed auto
!
interface FastEthernet0/1
 ip address 10.0.108.1 255.255.255.0
 duplex auto
 speed auto
!
interface FastEthernet1/0
 ip address 10.0.8.1 255.255.255.0
 duplex auto
 speed auto
!
!
! Default router goes to the ISP perimeter router
!
ip route 0.0.0.0 0.0.0.0 10.0.1.1
!
! Routes for the tunnelled sites point to the HSRP address on the DMVPN Hub routers
!
ip route 10.43.0.0 255.255.0.0 10.0.1.250
ip route 10.99.0.0 255.255.0.0 10.0.1.250
!
!

Next the basics for our "Internet" router.

R5:

!
! As all connected devices are using NAT, we don't need any routes, just the connected interfaces.
!
!
interface FastEthernet0/0
 ip address 172.16.6.1 255.255.255.0
 duplex auto
 speed auto
!
interface FastEthernet0/1
 ip address 192.168.2.249 255.255.255.0
 duplex auto
 speed auto
!
!

Now our "ISP" perimeter router needs to be configured.

R4:

!
! External "Internet" interface (192.168.2.254), and the internal DC interfaces; direct outbound (10.0.1.x) and our "DMZ" (10.0.2.x).  Set up NAT inside and outside.
!
interface FastEthernet0/0
 ip address 10.0.2.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly
 duplex auto
 speed auto
!
interface FastEthernet0/1
 ip address 192.168.2.254 255.255.255.0
 ip nat outside
 ip virtual-reassembly
 duplex auto
 speed auto
!
interface FastEthernet1/0
 ip address 10.0.1.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly
 duplex auto
 speed auto
!
ip nat inside source list NAT interface FastEthernet0/1 overload
!
ip access-list extended NAT
 permit ip 10.0.0.0 0.0.255.255 any
!
!
! Default route to our "Internet" router, and routes to our DC subnets.
!
ip route 0.0.0.0 0.0.0.0 192.168.2.249
ip route 10.0.8.0 255.255.255.0 10.0.1.2
ip route 10.0.108.0 255.255.255.0 10.0.1.2
!
!
! Finally, the NAT configuration for our IPSec Tunnels to the DMVPN Hubs.  Note that we only need the UDP ports, as ESP is encapsulated for NAT-T.
!
ip nat inside source static udp 10.0.2.251 500 192.168.2.250 500 extendable
ip nat inside source static udp 10.0.2.251 4500 192.168.2.250 4500 extendable
ip nat inside source static udp 10.0.2.252 500 192.168.2.252 500 extendable
ip nat inside source static udp 10.0.2.252 4500 192.168.2.252 4500 extendable
!


Now, we begin the basics of the remote site router.  Site ID is 99, so we are using 10.99.x.x as the internal address space.

R7:

!
! Internal and External Interfaces and NAT, and a default route to the "Internet"
!
!
interface FastEthernet0/0
 ip address 172.16.6.99 255.255.255.0
 ip nat outside
 ip virtual-reassembly
 duplex auto
 speed auto
!
interface FastEthernet0/1
 ip address 10.99.25.1 255.255.255.0
 ip nat inside
 ip virtual-reassembly
 duplex auto
 speed auto
!
!
ip route 0.0.0.0 0.0.0.0 172.16.6.1
!
!
ip nat inside source list NAT interface FastEthernet0/0 overload
!
! Deny NAT from internal subnets to our DC private space, permit everything else.
!
ip access-list extended NAT
 deny   ip 10.99.25.0 0.0.0.255 10.0.0.0 0.0.255.255
 permit ip 10.99.0.0 0.0.255.255 any
!

Now to the heart of the matter.  We are going to now set up our DMVPN Hubs.

First, we set up the internal (10.0.1.x) and external (10.0.2.x) interfaces, employing HSRP on the internals side.  Adding in a default route and internal routes to the DC subnets.

R1:

interface FastEthernet0/0
 ip address 10.0.1.251 255.255.255.0
 duplex auto
 speed auto
 standby 1 ip 10.0.1.250
 standby 1 priority 110
 standby 1 preempt
 standby 1 track FastEthernet0/1 25
!
interface FastEthernet0/1
 ip address 10.0.2.251 255.255.255.0
 duplex auto
 speed auto
!
!
ip route 0.0.0.0 0.0.0.0 10.0.2.1
ip route 10.0.8.0 255.255.255.0 10.0.1.2
ip route 10.0.108.0 255.255.255.0 10.0.1.2
!
!

R2:

interface FastEthernet0/0
 ip address 10.0.1.252 255.255.255.0
 duplex auto
 speed auto
 standby 1 ip 10.0.1.250
 standby 1 priority 90
 standby 1 preempt
 standby 1 track FastEthernet0/1 25
!
interface FastEthernet0/1
 ip address 10.0.2.252 255.255.255.0
 duplex auto
 speed auto
!
ip route 0.0.0.0 0.0.0.0 10.0.2.1
ip route 10.0.8.0 255.255.255.0 10.0.1.2
ip route 10.0.108.0 255.255.255.0 10.0.1.2
!

Now, setting up the crypto on the Hubs.  Note transport mode is required for NAT-T.


R1:

!
crypto isakmp policy 10
 encr 3des
 authentication pre-share
crypto isakmp key tunnelkey1 address 0.0.0.0 0.0.0.0
crypto isakmp keepalive 10
!
!
crypto ipsec transform-set DMVPNESP esp-3des esp-sha-hmac
 mode transport
!
crypto ipsec profile MFS-DMVPN
 set transform-set DMVPNESP
!
!

R2:

!
crypto isakmp policy 20
 encr 3des
 authentication pre-share
crypto isakmp key tunnelkey2 address 0.0.0.0 0.0.0.0
crypto isakmp keepalive 10
!
!
crypto ipsec transform-set DMVPNESP2 esp-3des esp-sha-hmac
 mode transport
!
crypto ipsec profile MFS-DMVPN2
 set transform-set DMVPNESP2
!
!
Next, setting up the Hub Tunnels.  One tunnel will use 10.88.0.0/24, the other 10.89.0.0/24:

R1:

interface Tunnel0
 ip address 10.88.0.1 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp map multicast dynamic
 ip nhrp network-id 12345
 ip nhrp holdtime 600
 tunnel source FastEthernet0/1
 tunnel mode gre multipoint
 tunnel key 100000
 tunnel path-mtu-discovery
 tunnel protection ipsec profile MFS-DMVPN shared
!

R2:

interface Tunnel1
 ip address 10.89.0.1 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp map multicast dynamic
 ip nhrp network-id 12346
 ip nhrp holdtime 600
 tunnel source FastEthernet0/1
 tunnel mode gre multipoint
 tunnel key 100001
 tunnel path-mtu-discovery
 tunnel protection ipsec profile MFS-DMVPN2 shared
!

Finally the routing protocol.  For simplicity, I am using EIGRP.  I redistribute my statics for the DC subnets using a route-map.  In addition, I added a ditstribute-list route-map to stop re-advertising inbound routes (I am not disabling split-horizon as spoke-to spoke is not allowed or required, but if tunnels are asymmetric - some on Hub 1, others on Hub 2, the routes will be readvertised, and I don't want it to do that).

R1:

!
router eigrp 8888
 redistribute static metric 1500 1 255 1 1500 route-map DC-STATICS
 passive-interface default
 no passive-interface FastEthernet0/0
 no passive-interface Tunnel0
 network 10.0.1.0 0.0.0.255
 network 10.88.0.0 0.0.0.255
 distribute-list route-map DC-ROUTES out Tunnel0
 no auto-summary
!
!
ip prefix-list DC-ROUTES seq 5 permit 10.0.1.0/24
ip prefix-list DC-ROUTES seq 10 permit 10.0.8.0/24
ip prefix-list DC-ROUTES seq 15 permit 10.0.108.0/24
ip prefix-list DC-ROUTES seq 20 deny 0.0.0.0/0 le 32
!
ip prefix-list DC-STATICS seq 5 permit 10.0.8.0/24
ip prefix-list DC-STATICS seq 10 permit 10.0.108.0/24
ip prefix-list DC-STATICS seq 15 deny 0.0.0.0/0 le 32
!
!
!
!
route-map DC-ROUTES permit 5
 match ip address prefix-list DC-ROUTES
!
route-map DC-ROUTES deny 10
!
route-map DC-STATICS permit 5
 match ip address prefix-list DC-STATICS
!
route-map DC-STATICS deny 10
!


R2:

!
router eigrp 8888
 redistribute static metric 1000 1 255 1 1500 route-map DC-STATICS
 passive-interface default
 no passive-interface FastEthernet0/0
 no passive-interface Tunnel1
 network 10.0.1.0 0.0.0.255
 network 10.89.0.0 0.0.0.255
 distribute-list route-map DC-ROUTES out Tunnel1
 no auto-summary
!
!
!
ip prefix-list DC-ROUTES seq 5 permit 10.0.1.0/24
ip prefix-list DC-ROUTES seq 10 permit 10.0.8.0/24
ip prefix-list DC-ROUTES seq 15 permit 10.0.108.0/24
ip prefix-list DC-ROUTES seq 20 deny 0.0.0.0/0 le 32
!
ip prefix-list DC-STATICS seq 5 permit 10.0.8.0/24
ip prefix-list DC-STATICS seq 10 permit 10.0.108.0/24
ip prefix-list DC-STATICS seq 15 deny 0.0.0.0/0 le 32
!
!
!
!
route-map DC-ROUTES permit 5
 match ip address prefix-list DC-ROUTES
!
route-map DC-ROUTES deny 10
!
route-map DC-STATICS permit 5
 match ip address prefix-list DC-STATICS
!
route-map DC-STATICS deny 10
!

Finally, the Spoke Router.

R7:

!
! Crypto.  As I am not really familiar with IPSec, I created two profiles, matching the ones on the Hubs.  I don't know if any of it is superfluous.
!
!
crypto isakmp policy 10
 encr 3des
 authentication pre-share
!
crypto isakmp policy 20
 encr 3des
 authentication pre-share
crypto isakmp key tunnelkey2 address 192.168.2.252
crypto isakmp key tunnelkey1 address 192.168.2.250
crypto isakmp keepalive 10
!
!
crypto ipsec transform-set DMVPNESP2 esp-3des esp-sha-hmac
 mode transport
crypto ipsec transform-set DMVPNESP esp-3des esp-sha-hmac
 mode transport
!
crypto ipsec profile MFS-DMVPN
 set transform-set DMVPNESP
!
crypto ipsec profile MFS-DMVPN2
 set transform-set DMVPNESP2
!

!
! Now, the first tunnel.  Note the different delays for EIGRP so that one is preferred over the other.
!
!
!
interface Tunnel0
 ip address 10.88.0.99 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp map 10.88.0.1 192.168.2.250
 ip nhrp map multicast 192.168.2.250
 ip nhrp network-id 12345
 ip nhrp holdtime 600
 ip nhrp nhs 10.88.0.1
 delay 1000
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 100000
 tunnel path-mtu-discovery
 tunnel protection ipsec profile MFS-DMVPN shared
!
!
!And the second.
!
!
interface Tunnel1
 ip address 10.89.0.99 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp map 10.89.0.1 192.168.2.252
 ip nhrp map multicast 192.168.2.252
 ip nhrp network-id 12346
 ip nhrp holdtime 600
 ip nhrp nhs 10.89.0.1
 delay 2000
 tunnel source FastEthernet0/0
 tunnel mode gre multipoint
 tunnel key 100001
 tunnel path-mtu-discovery
 tunnel protection ipsec profile MFS-DMVPN2 shared
!
!
!Next the EIGRP configuration.  Just running as a stub.
!
!
router eigrp 8888
 passive-interface default
 no passive-interface Tunnel0
 no passive-interface Tunnel1
 network 10.88.0.0 0.0.0.255
 network 10.89.0.0 0.0.0.255
 network 10.99.0.0 0.0.255.255
 no auto-summary
 eigrp stub connected summary
!
And we are done.  I repeated the spoke config for a second site, as per the diagram.

So, starting up the routers, we find the tunnels come up.  On R1 and R2, I see two peers each; one for each spoke.

R1#sho dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding
        UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface: Tunnel0, IPv4 NHRP Details
Type:Hub, NHRP Peers:2,

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1      172.16.6.2      10.88.0.43    UP 00:02:37     D
     1     172.16.6.99      10.88.0.99    UP 00:02:34     D



R2#show dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding
        UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface: Tunnel1, IPv4 NHRP Details
Type:Hub, NHRP Peers:2,

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1      172.16.6.2      10.89.0.43    UP 00:03:37     D
     1     172.16.6.99      10.89.0.99    UP 00:03:32     D
And in the routing tables (only R1 shown), I see my remote site networks.

R1#show ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 10.0.2.1 to network 0.0.0.0

     10.0.0.0/24 is subnetted, 8 subnets
S       10.0.8.0 [1/0] via 10.0.1.2
C       10.0.2.0 is directly connected, FastEthernet0/1
C       10.0.1.0 is directly connected, FastEthernet0/0
D       10.43.25.0 [90/26882560] via 10.88.0.43, 00:04:01, Tunnel0
C       10.88.0.0 is directly connected, Tunnel0
D       10.89.0.0 [90/26882560] via 10.0.1.252, 00:03:58, FastEthernet0/0
S       10.0.108.0 [1/0] via 10.0.1.2
D       10.99.25.0 [90/26882560] via 10.88.0.99, 00:03:57, Tunnel0
S*   0.0.0.0/0 [1/0] via 10.0.2.1

Out at the spoke, I can see my two tunnels up:

R7#sho dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
        N - NATed, L - Local, X - No Socket
        # Ent --> Number of NHRP entries with same NBMA peer
        NHS Status: E --> Expecting Replies, R --> Responding
        UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface: Tunnel0, IPv4 NHRP Details
Type:Spoke, NHRP Peers:1,

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1   192.168.2.250       10.88.0.1    UP 00:06:16     S

Interface: Tunnel1, IPv4 NHRP Details
Type:Spoke, NHRP Peers:1,

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb
 ----- --------------- --------------- ----- -------- -----
     1   192.168.2.252       10.89.0.1    UP 00:06:14     S

And I see the redistributed statics for my DC subnets.

R7#sho ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 172.16.6.1 to network 0.0.0.0

     172.16.0.0/24 is subnetted, 1 subnets
C       172.16.6.0 is directly connected, FastEthernet0/0
     10.0.0.0/24 is subnetted, 6 subnets
D EX    10.0.8.0 [170/25856256] via 10.88.0.1, 00:01:24, Tunnel0
D       10.0.1.0 [90/25858560] via 10.88.0.1, 00:06:33, Tunnel0
C       10.88.0.0 is directly connected, Tunnel0
C       10.89.0.0 is directly connected, Tunnel1
D EX    10.0.108.0 [170/25856256] via 10.88.0.1, 00:06:33, Tunnel0
C       10.99.25.0 is directly connected, FastEthernet0/1
S*   0.0.0.0/0 [1/0] via 172.16.6.1
R7#
And to confirm I am receiving routes from both hubs, I can check the EIGRP topology:

R7#sho ip eigrp 8888 topology
IP-EIGRP Topology Table for AS(8888)/ID(172.16.6.99)

Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply,
       r - reply Status, s - sia Status

P 10.0.8.0/24, 1 successors, FD is 25856256
        via 10.88.0.1 (25856256/1706752), Tunnel0
        via 10.89.0.1 (26112256/2560256), Tunnel1

P 10.0.1.0/24, 1 successors, FD is 25858560
        via 10.88.0.1 (25858560/28160), Tunnel0
        via 10.89.0.1 (26114560/28160), Tunnel1
P 10.88.0.0/24, 1 successors, FD is 25856000
        via Connected, Tunnel0
P 10.89.0.0/24, 1 successors, FD is 26112000
        via Connected, Tunnel1
P 10.0.108.0/24, 1 successors, FD is 25856256
        via 10.88.0.1 (25856256/1706752), Tunnel0
        via 10.89.0.1 (26112256/2560256), Tunnel1

P 10.99.25.0/24, 1 successors, FD is 28160
        via Connected, FastEthernet0/1
R7#

So, now for a quick test using the attached VPCs.  A traceroute from the spoke to a subnet at the hub:

VPCS[3]> show
...
VPCS3  10.99.25.5/24        10.99.25.1        00:50:79:66:68:02  20002  30002
       fe80::2050:79ff:fe66:6802/64
VPCS4  10.0.8.5/24          10.0.8.1          00:50:79:66:68:03  20003  30003
       fe80::2050:79ff:fe66:6803/64

VPCS[3]> tracert 10.0.8.5
traceroute to 10.0.8.5, 64 hops max, press Ctrl+C to stop
 1   10.99.25.1   1.664 ms  2.975 ms  17.017 ms
 2   10.88.0.1   25.941 ms  31.516 ms  23.962 ms
 3   10.0.1.2   43.469 ms  26.979 ms  25.521 ms
 4   10.0.8.5   41.547 ms  36.017 ms  27.056 ms

So far so good.  Now I will shut down R1, and the failover should occur and the traceroute should use Tunnel1 instead of Tunnel0 (10.89.0.1 instead of 10.88.0.1).  First, the routing table will show the routes coming in from R2:

R7#sho ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route

Gateway of last resort is 172.16.6.1 to network 0.0.0.0

     172.16.0.0/24 is subnetted, 1 subnets
C       172.16.6.0 is directly connected, FastEthernet0/0
     10.0.0.0/24 is subnetted, 6 subnets
D EX    10.0.8.0 [170/26112256] via 10.89.0.1, 00:00:24, Tunnel1
D       10.0.1.0 [90/26114560] via 10.89.0.1, 00:00:24, Tunnel1
C       10.88.0.0 is directly connected, Tunnel0
C       10.89.0.0 is directly connected, Tunnel1
D EX    10.0.108.0 [170/26112256] via 10.89.0.1, 00:00:24, Tunnel1
C       10.99.25.0 is directly connected, FastEthernet0/1
S*   0.0.0.0/0 [1/0] via 172.16.6.1
And the traceroute:

VPCS[3]> tracert 10.0.8.5
traceroute to 10.0.8.5, 64 hops max, press Ctrl+C to stop
 1   10.99.25.1   6.257 ms  2.014 ms  4.906 ms
 2   10.89.0.1   52.001 ms  25.993 ms  22.953 ms
 3   10.0.1.2   33.960 ms  27.989 ms  26.974 ms
 4   10.0.8.5   36.913 ms  29.943 ms  30.820 ms

 So that was a quick trip through labbing up my DMVPN design.  I've used a lot of defaults, and some of the NHRP things are not quite clear.  I fear I don't know enough about IPSec for in-depth troubleshooting, but hopefully this will transition smoothly into production now that I've proved the concepts.  Once again, any feedback is welcome, either in the comments or via twitter (@mengelm).










No comments:

Post a Comment