現象

在主機內能 ping 通互聯網的地址，但是從互聯網拉取文件時一直超時。

排查過程

網絡拓撲為： 互聯網 <-> 邏輯網關 <-> 主機 ，即主機訪問互聯網時需要經過邏輯網關，邏輯網關里配置了用于訪問互聯網的NAT相關規則。

在主機內使用 curl 或 wget 時，增加日志級別，可以觀察到連接已經建立了，但是后續卻沒有數據，直到超時退出。

由于出網經過邏輯網關，因此在邏輯網關內抓包觀察，以下是分別在邏輯網關兩側的抓包結果，先看互聯網側的抓包結果，如下所示：

15:18:12.793219 IP 192.168.10.39.36302 > 116.253.29.205.443: Flags [S], seq 2488013264, win 65535, options [mss 1460,sackOK,TS val 4141372887 ecr 0,nop,wscale 8], length 0
15:18:12.821648 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [S.], seq 3457463034, ack 2488013265, win 42340, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0
15:18:12.822230 IP 192.168.10.39.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, length 0
15:18:12.910971 IP 192.168.10.39.36302 > 116.253.29.205.443: Flags [P.], seq 1:196, ack 1, win 256, length 195
15:18:12.939368 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], ack 196, win 83, length 0

15:18:12.946179 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:18:12.946198 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1461:2921, ack 196, win 83, length 1460
15:18:12.949090 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [P.], seq 2921:4374, ack 196, win 83, length 1453
15:18:13.009258 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [P.], seq 2921:4374, ack 196, win 83, length 1453
15:18:13.240253 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:18:13.704255 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:18:14.672258 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:18:16.528279 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:18:20.240243 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:18:27.856268 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:18:42.704333 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460

15:18:42.850948 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [F.], seq 4374, ack 196, win 83, length 0
15:18:42.852265 IP 192.168.10.39.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0
15:18:42.880688 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:19:43.032513 IP 192.168.10.39.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0
15:19:43.060908 IP 116.253.29.205.443 > 192.168.10.39.36302: Flags [R], seq 3457463035, win 0, length 0

其中，192.168.10.39 是 NAT 轉換后的地址，從上面可以看到 1-5 行是三次握手，建立連接的日志；第 7-17 行是服務器發送數據的日志，其中可以看到 1:1461 的包一直在重發（因為沒有收到響應）；19-23 行是服務器通知邏輯網關 關閉連接的日志。

再看內網側的抓包結果，如下所示：

15:18:12.793177 IP 10.0.0.6.36302 > 116.253.29.205.443: Flags [S], seq 2488013264, win 65535, options [mss 1460,sackOK,TS val 4141372887 ecr 0,nop,wscale 8], length 0
15:18:12.821677 IP 116.253.29.205.443 > 10.0.0.6.36302: Flags [S.], seq 3457463034, ack 2488013265, win 42340, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0
15:18:12.822207 IP 10.0.0.6.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, length 0
15:18:12.910939 IP 10.0.0.6.36302 > 116.253.29.205.443: Flags [P.], seq 1:196, ack 1, win 256, length 195
15:18:12.939388 IP 116.253.29.205.443 > 10.0.0.6.36302: Flags [.], ack 196, win 83, length 0

15:18:42.850974 IP 116.253.29.205.443 > 10.0.0.6.36302: Flags [F.], seq 4374, ack 196, win 83, length 0
15:18:42.852238 IP 10.0.0.6.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0
15:19:43.032468 IP 10.0.0.6.36302 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0
15:19:43.060929 IP 116.253.29.205.443 > 10.0.0.6.36302: Flags [R], seq 3457463035, win 0, length 0

其中，10.0.0.6 是 NAT 轉換前的地址，1-5 行也是建立連接的日志，然后 7-10 行就已經是關閉連接的日志了。和前面的抓包結果對比可以很直觀的發現，服務器發送過來的數據包，內部網卡都沒有收到，導致最后服務器關閉了連接。

那么現在的問題就是為什么內部網卡會收不到這些數據包？

重新看日志的時候發現，建立連接的時候，雙方約定了mss（最大報文段長度）為1460 （對應日志 [mss 1460,sackOK,TS val 4141372887 ecr 0,nop,wscale 8]），說明主機內的網卡MTU（最大傳輸單元）應該是 1500 字節，但是，邏輯網關的網卡 MTU 是 1400 ，這就導致了如果收到的報文長度是大于 1400 字節的就會被丟棄，這和前面觀察到的現象基本一致，并且服務器發來的報文長度就是1460（seq 1:1461）。

由于前面抓包的時候限制了協議，這里限制 host 后再抓包看下結果：

tcpdump -nn -i net1 host 116.253.29.205

15:38:11.303605 IP 192.168.10.39.36306 > 116.253.29.205.443: Flags [S], seq 2941142219, win 65535, options [mss 1460,sackOK,TS val 4142571398 ecr 0,nop,wscale 8], length 0
15:38:11.320427 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [S.], seq 2764254742, ack 2941142220, win 42340, options [mss 1460,nop,nop,sackOK,nop,wscale 9], length 0
15:38:11.320981 IP 192.168.10.39.36306 > 116.253.29.205.443: Flags [.], ack 1, win 256, length 0
15:38:11.411741 IP 192.168.10.39.36306 > 116.253.29.205.443: Flags [P.], seq 1:196, ack 1, win 256, length 195
15:38:11.428650 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], ack 196, win 83, length 0
15:38:11.431748 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:11.431785 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:11.431764 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1461:2921, ack 196, win 83, length 1460
15:38:11.431812 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:11.433477 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [P.], seq 2921:4374, ack 196, win 83, length 1453
15:38:11.433488 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:11.470625 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [P.], seq 2921:4374, ack 196, win 83, length 1453
15:38:11.470662 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:11.694652 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:11.694692 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:12.134651 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:12.134690 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:13.014626 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:13.014662 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:14.806622 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:14.806661 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:18.326657 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:18.326696 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:25.366616 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:25.366658 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:39.702639 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:39.702686 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:38:41.337719 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [F.], seq 4374, ack 196, win 83, length 0
15:38:41.338581 IP 192.168.10.39.36306 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0
15:38:41.355434 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [.], seq 1:1461, ack 196, win 83, length 1460
15:38:41.355472 IP 192.168.10.39 > 116.253.29.205: ICMP 192.168.10.39 unreachable - need to frag (mtu 1400), length 556
15:39:41.624210 IP 192.168.10.39.36306 > 116.253.29.205.443: Flags [.], ack 1, win 256, options [nop,nop,sack 1 {4374:4375}], length 0
15:39:41.641021 IP 116.253.29.205.443 > 192.168.10.39.36306: Flags [R], seq 2764254743, win 0, length 0

這次的抓包結果就能說明確實是因為邏輯網關網卡的 MTU 為 1400 導致的 。

解決方法

知道原因后，解決方法基本就清晰了，在報文經過邏輯網關時修改 MSS 的值即可，使其根據路徑MTU中的最小值進行計算，這里有個專業的術語叫 ?**MSS鉗制**?，具體的操作指令如下所示。

iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu