README.kernel
1KERNEL REQUIREMENTS
2===================
3
4The linux kernel has had various major regressions, performance
5issues and subtle bugs (especially in pmtu). Here is a short list
6of some -stable kernels and the first point release that is supposedly
7working well with opennhrp/dmvpn:
8 3.12.8 or later
9 3.14.54 or later
10 3.18.22 or later[1]
11
12[1] But you need to apply the following two backported commits:
13 3cdaa5be9e ipv4: Don't increase PMTU with Datagram Too Big message
14 cb6ccf09d6 route: Use ipv4_mtu instead of raw rt_pmtu
15
16See below for list of known issues in various kernel versions.
17
18Kernels earlier than 3.12 need CONFIG_ARPD enabled in the configuration.
19Many distributions do not enable it by default, and you may need to
20compile your own kernel.
21
22KERNEL BUGS
23===========
24
25DMVPN and mGRE support in the kernel has been brittle. There are various
26regressions in multiple kernel versions.
27
28This list tries to collect them to one source of information:
29
30- forward pmtu is disabled intentionally (but tunnel devices rely on it)
31 Broken since 3.14-rc1:
32 commit "ipv4: introduce ip_dst_mtu_maybe_forward and protect forwarding path against pmtu spoofing"
33 Workaround:
34 Set sysctl net.ipv4.ip_forward_use_pmtu=1
35 (Should fix kernel to have this by default on for tunnel devices)
36
37- subtle path mtu mishandling issues
38 Broken since (uncertain)
39 Fixed in 4.1-rc2:
40 commit "ipv4: Don't increase PMTU with Datagram Too Big message."
41 commit "route: Use ipv4_mtu instead of raw rt_pmtu"
42
43- fragmentation of large packets inside tunnel not working
44 Broken since 3.11-rc1
45 commit "ip_tunnels: Use skb-len to PMTU check."
46 Fixed in 3.14.54, 3.18.22, 4.1.9, 4.2-rc3
47 commit "ip_tunnel: fix ipv4 pmtu check to honor inner ip header df"
48
49- ipsec will crash during xfrm gc
50 Broke since 3.15-rc1
51 commit "flowcache: Make flow cache name space aware"
52 Fixed in 3.18.10, 4.0
53 commit "flowcache: Fix kernel panic in flow_cache_flush_task"
54
55- TSO on GRE tunnels failed, and resulted in very slow performance
56 Broke since 3.14.24, 3.18-rc3
57 commit "gre: Use inner mac length when computing tunnel length"
58 Fixed in 3.14.30, 3.18.4
59 commit "gre: fix the inner mac header in nbma tunnel xmit path"
60 commit "gre: Set inner mac header in gro complete"
61
62- NAPI GRO handling was broken; causing immediate crash (32-bit only?)
63 Broken since 3.13-rc1
64 commit "net: gro: allow to build full sized skb"
65 Fixed 3.14.5, 3.15-rc7
66 commit "net: gro: make sure skb->cb[] initial content has not to be zero"
67
68- ip_gre dst caching broke NBMA GRE tunnels
69 Broken since 3.14-rc1
70 Fixed in 3.14.5, 3.15-rc6
71 commit "ipv4: ip_tunnels: disable cache for nbma gre tunnels"
72
73- Few packets can be lost when neighbor entry is in NUD_PROBE state,
74 and there is continuous traffic to it.
75 Broken since dawn of time
76 Fixed in 3.15-rc1
77 commit "neigh: probe application via netlink in NUD_PROBE"
78
79- GRO was implemented for GRE, but the hw capabilities were not updated
80 correctly. In practice forwarding from non-GRE (physical) interface
81 to GRE interface with gro/gso/tx offloads enabled (also on the target
82 interface) does not work properly.
83 Broken around 3.9 to 3.11, need to check details.
84
85- recvfrom() returned incorrect NBMA address, breaking NAT detection
86 Broken since 3.10-rc1
87 commit "GRE: Refactor GRE tunneling code."
88 Fixed in 3.10.27, 3.12.8, 3.13-rc7
89 commit "ip_gre: fix msg_name parsing for recvfrom/recvmsg"
90
91- sendto() was broken causing opennhrp not work at all
92 Broken since 3.10-rc1
93 commit "GRE: Refactor GRE tunneling code."
94 Fixed in 3.10.12, 3.11-rc6
95 commit "ip_gre: fix ipgre_header to return correct offset"
96
97- PMTU was broken due to GRE driver rewrite
98 Broken since 3.10-rc1
99 commit "GRE: Refactor GRE tunneling code."
100 Fixed in 3.11-rc1
101 commit "ip_tunnels: Use skb-len to PMTU check."
102
103- PMTU was broken due to routing cache removal
104 Broken since 3.6-rc1
105 commit "ipv4: Cache input routes in fib_info nexthops"
106 Fixed in 3.11-rc1
107 commit "ipv4: use next hop exceptions also for input routes"
108 + 3 other commits
109 Patches exist for 3.10, but they were not approved to 3.10-stable.
110
111- Race condition during bootup: changing ARP flag did not flush
112 existing neighbor entries, causing problems if traffic was routed
113 to gre interface before opennhrp was running.
114 Broken since dawn of time
115 Fixed in 3.11-rc1
116 commit "arp: flush arp cache on IFF_NOARP change"
117
118- Crash in IPsec
119 Broken since 3.9-rc1
120 commit "xfrm: removes a superfluous check and add a statistic"
121 Fixed in 3.10-rc3
122 commit "xfrm: properly handle invalid states as an error"
123
124- An incorrect ip_gre change broke NHRP traffic over GRE
125 Broken since 3.8-rc2
126 commit "ip_gre: make ipgre_tunnel_xmit() not parse network header as IP unconditionally"
127 Fixed in 3.8.5, 3.9-rc4
128 commit "Revert "ip_gre: make ipgre_tunnel_xmit() not parse network header as IP unconditionally""
129
130- Multicast traffic over mGRE was broken.
131 Broken since 2.6.34-rc2
132 commit "gre: fix hard header destination address checking"
133 Fixed in 2.6.39-rc2
134 commit "net: gre: provide multicast mappings for ipv4 and ipv6"
135
136- Serious performance issues causing small throughput on medium to large DMVPN networks
137 Broken since dawn of time
138 Fixed in 2.6.35
139 multiple commits rewriting ipsec caching
140
141- Even though around 2.6.24 is the first version where opennhrp started
142 to work, there has been various PMTU, performance, and functionality
143 bugs before 2.6.34. That's one of the first version I consider stable
144 wrt. to opennhrp functionality.
145
146
README.nhrpd
1Quagga / NHRP Design and Configuration Notes
2============================================
3
4Quagga/NHRP is an NHRP (RFC2332) implementation for Linux. The primary
5use case is to implement DMVPN. The aim is thus to be compatible with
6Cisco DMVPN (and potentially with FlexVPN in the future).
7
8
9Current Status
10--------------
11
12- IPsec integration with strongSwan (requires patched strongSwan)
13- IPv4 over IPv4 NBMA GRE
14- IPv6 over IPv4 NBMA GRE -- majority of code exist; but is not tested
15- Spoke (NHC) functionality complete
16- Hub (NHS) functionality complete
17- Multicast support is not done yet
18 (so OSPF will not work, use BGP for now)
19
20The code is not (yet) compatible with Cisco FlexVPN style DMVPN. It
21would require relaying IKEv2 routing messages from strongSwan to nhrpd
22and parsing that. It is doable, but not implemented for the time being.
23
24
25Routing Design
26--------------
27
28In contrast to opennhrp routing design, Quagga/NHRP routes each NHRP
29domain address individually (similar to Cisco FlexVPN).
30
31To create NBMA GRE tunnel you might use following:
32 ip tunnel add gre1 mode gre key 42 ttl 64 dev eth0
33 ip addr add 10.255.255.2/32 dev gre1
34 ip link set gre1 up
35
36This has two important differences compared to opennhrp setup:
37 1. The 'tunnel add' now specifies physical device binding. Quagga/NHRP
38 wants to know stable protocol address to NBMA address mapping. Thus,
39 add 'dev <physdev>' binding, or specify 'local <nbma-address>'. If
40 neither of this is specified, NHRP will not be enabled on the interface.
41 Alternatively you can skip 'dev' binding on tunnel if you allow
42 nhrpd to manage it using 'tunnel source' command (see below).
43
44 2. The 'addr add' now has host prefix. In opennhrp you would have used
45 the GRE subnet prefix length here instead, e.g. /24.
46
47Quagga/NHRP will automatically create additional host routes pointing to
48gre1 when a connection with these hosts is established. The gre1 subnet
49should be announced by routing protocol. This allows routing protocol
50to decide which is the closest hub and get the gre addresses' traffic.
51
52The second benefit is that hubs can then easily exchange host prefixes
53of directly connected gre addresses. And thus routing of gre addresses
54inside hubs is based on routing protocol's shortest path choice -- not
55on random choice from next hop server list.
56
57
58Configuring nhrpd
59-----------------
60
61The configuration is done using vtysh, and most commands do what they
62do in Cisco. As minimal configuration example one can do:
63 configure terminal
64 interface gre1
65 tunnel protection vici profile dmvpn
66 tunnel source eth0
67 ip nhrp network-id 1
68 ip nhrp shortcut
69 ip nhrp registration no-unique
70 ip nhrp nhs dynamic nbma hubs.example.com
71
72There's important notes about the "ip nhrp nhs" command:
73
74 1. The 'dynamic' works only against Cisco (or nhrpd), but is not
75 compatible with opennhrp. To use dynamic detection of opennhrp hub's
76 protocol address use the GRE broadcast address there. For the above
77 example of 10.255.255.0/24 the configuration should read instead:
78 ip nhrp nhs 10.255.255.255 nbma hubs.example.com
79
80 2. nbma <FQDN> works like opennhrp dynamic-map. That is, all of the
81 A-records are configured as NBMA addresses of different hubs, and
82 each hub protocol address will be dynamically detected.
83
84
85Hub functionality
86-----------------
87
88Sending Traffic Indication (redirect) notifications is now accomplished
89using NFLOG.
90
91Use:
92iptables -A FORWARD -i gre1 -o gre1 \
93 -m hashlimit --hashlimit-upto 4/minute --hashlimit-burst 1 \
94 --hashlimit-mode srcip,dstip --hashlimit-srcmask 16 --hashlimit-dstmask 16 \
95 --hashlimit-name loglimit-0 -j NFLOG --nflog-group 1 --nflog-range 128
96
97or similar to get rate-limited samples of the packets that match traffic
98flow needing redirection. This kernel NFLOG target's nflog-group is configured
99in global nhrp config with:
100 nhrp nflog-group 1
101
102To start sending these traffic notices out from hubs, use the nhrp per-interface
103directive:
104 ip nhrp redirect
105
106opennhrp used PF_PACKET and tried to create packet filter to get only
107the packets of interest. Though, this was bad if shortcut fails to
108establish (remote policy, or both are behind NAT or restrictive
109firewalls), all of the relayaed traffic would match always.
110
111
112Getting information via vtysh
113-----------------------------
114
115Some commands of interest:
116 - show dmvpn
117 - show ip nhrp cache
118 - show ip nhrp shortcut
119 - show ip route nhrp
120 - clear ip nhrp cache
121 - clear ip nhrp shortcut
122
123
124Integration with strongSwan
125---------------------------
126
127Contrary to opennhrp, Quagga/NHRP has tight integration with IKE daemon.
128Currently strongSwan is supported using the VICI protocol. strongSwan
129is connected using UNIX socket (hardcoded now as /var/run/charon.vici).
130Thus nhrpd needs to be run as user that can open that file.
131
132Currently, you will need patched strongSwan. The working tree is at:
133 http://git.alpinelinux.org/cgit/user/tteras/strongswan/log/?h=tteras
134
135And the branch with patches against latest release are:
136 http://git.alpinelinux.org/cgit/user/tteras/strongswan/log/?h=tteras-release
137
138