tl;dr: When using some Mifi routers on Ubuntu, they have a firmware issue where new WebRTC connections (Hangouts, Zoom, Freedom Teleop, etc) can trigger a router IP table reset in the router or cell network infrastructure and public IP change immediately after connection, making it fail most of the time.

diagnosis: On the system resources page, in the network tab, if you see "IP" events where the address changed right after connecting to WebRTC, then this issue exists.

solution: Enable IP table “Masquerading” on the robot by running iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE each time on boot so that the internally seen network routing is not changed directly after the WebRTC connection is made, removing the failure.

Problem background

When using a Verizon MiFi, such as the 7700 or 8800, initiating a WebRTC connection triggers the MiFi to renew its public IP address, causing a repeated disconnect as the WebRTC routing then fails. It is reported by many people when connecting Google Hangouts and other standard communications systems. It does not currently have a solution from Verizon.

The end result is that it is not possible to stably connect in Google Hangouts, Freedom Teleoperation or many other WebRTC-based systems. If a connection does hold, then it stays connected, but this can be 1 in 5 or 1 in 20 times.

Below are public requests over the last 2 years:

• Github - webrtc / samples
• Stackoverflow - ip masquerading fix problematic WebRTC
• Reddit - MiFi refuses to connect to WebRTC
• Verizon Support - WebRTC Connection disables Verizon MIFI

This failure is not seen on:

• Other carrier’s MiFis (Not guaranteed, but seems to work on most hardware and sims)
• Peplink and other cellular routers using Verizon SIM cards (Could mean the mifi firmware is the root cause rather than Verizon network)
• Windows and MacOS. (Again not guaranteed, but seems to work)
• WIFI networks connected to LAN

Root cause

The root cause of this issue is that the Mifi’s NAT code, routing table implementation, cell-towers or another element of the system do not correctly set up and stably keep the routing of data the same through a session.

Specifically, when a new WebRTC connection is made, the MiFi re-requests, resets, or is given a new IP address which makes the peer-to-peer WebRTC data routing fail because an intermediate IP address changes.

It seems that the root cause is that the routing tables defined by the MiFi router change or fail for an unspecified reason directly after a WebRTC connection request is received, causing WebRTC direct connections to fail.

All other connections such as SSH and HTTP also fail for a short period of time, showing that it is not just a single port, but the actual core connection going offline.

Best Known Method 2020

The current solution which has been tested on Verizon 8800 and 7700 Jetpacks is to enable IP table masquerading for the connection so that the routing between machines still works, even if the cellular network changes its underlying public connection IPs.

📘

The world is complex

There may be other root causes or solutions for different hardware setups, so if you have see other behavior, or this does not fix performance issues, please reach out.

Also, as 5G and new infrastructure is rolled out, this should ideally go away as a necessary implementation due to moving to IPv6 and new router + network infrastructure.

What is IP table masquerading?

Masquerading allows a computer to use a private (reserved) IP network address on your LAN and have your Linux-based router perform some clever, real-time translation of IP addresses and ports.

More detailed descriptions:

• Oreilly Linux Network Administration Guide
• Oreilly Configuring IP Masquerade

Why Masquerading works

It works by bypassing the problematic NAT which lives in the router (Or Verizon infrastructure) so that when a new connection is made, the routing tables still function correctly and it is not disconnected.

Analysis and testing

Below is a test where a robot running on a Verizon MiFi was used to run 4 groups of 5 connection attempts over WebRTC. The 1st and 3rd groups were done with standard IP tables which used the MiFi for routing and the 2nd and 4th groups were done with IP Masquerading turned on. The upper half of the table shows a flag circled in red each time the IP address changed. It can be clearly seen that the IP address never changed on a connection when Masquerading was turned on, however it changed on 8 of 10 attempts when it was turned off.

The unit for testing was a Verizon Jetpack 7730L attached to an Ubuntu 16.04-based ARM architecture mobile robot running ROS.

How to enable masquerading

You can enable Masquerading by typing in the below command or adding it to your docker file's entry point:

iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE

NOTES:

• If you connect to the Verizon MiFi on a different adapter than wlan0, please change the name to that one, such as eth0.
• The above command is ephemeral and won’t last between reboots so needs to be triggered each time the computer or container is booted.

Disable masquerading

Change the -A (add) to a -D (delete) and rerun the command.

iptables -t nat -D POSTROUTING -o wlan0 -j MASQUERADE

Check your routing rules

You can run the below command to list your post routing rules before and after enabling masquerading.

iptables -t nat -v -L POSTROUTING -n --line-number

You should see a rule such as the below one added at the end.

4        0     0 MASQUERADE  all  --  *      wlan0   0.0.0.0/0            0.0.0.0/0

How do I permanently enable masquerading on reboot?

It is not possible to have iptables rules permanent across reboots, but you can add a loading script which reloads the previous command each time.

echo "iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE" >> /etc/rc.local
sudo reboot

Reboot to validate that it applies it correctly on startup. Then check the routing table after reboot to make sure there is a MASQUERADE rule as seen above using iptables -t nat -v -L POSTROUTING -n --line-number.

Enabling it for docker containers

If you are using docker containers in you setup, just make sure to include the command that enables masquerading it in the entry-point script.