UNetLab: Fixing external connectivity (pnet) issues when running VMware Workstation on Linux host

In this quick post, I’ll address an external connectivity issue on UNetLab pnet networks that I’ve stumbled upon recently and how you can fix it, in case you’re running VMware Workstation on Linux. Specifically, the problem has to do with the fact that you can’t ping from UNetLab nodes to your Linux host, or vice versa, even though all nodes are connected to pnet1 (in my case), which in turn, is connected to vmnet1 on VMware Workstation.

Long story short, the problem has to do with Linux access permissions on vmnet* network adapters. In my case, I setup Workstation as a systemd service. So, whenever my Linux host boots up all vmware daemons are started accordingly. As a result, all vmware network adapters vmnet* are owned by the root user, which turns out to be a problem because this adapter needs to be set in promiscuous mode in execution time, and since I’m running workstation with my standard user I’m not allowed to so.

As you’ll see shortly, I decided to work around this by changing the ownership of /dev/vmnet1 to my standard user, since this is my personal laboratory this solution is totally acceptable. Before I show you the final solution, let me walk you through the problem. Initially, /dev/vmnet1 is owned by the root user:

1
2
3
[19:52:14] at volos ➜ ~ ll /dev/vmnet1
crw------- 1 root root 119, 1 Oct 12 19:51 /dev/vmnet1
[19:52:18] at volos ➜ ~

If you were to verify L3 reachability, you’d notice that you couldn’t ping between your host machine and UNL nodes even though ARP resolution was successful. For example, my Linux host running VMware Workstation, has the IP 192.168.99.1 set on vmnet1 and one of the UNL nodes, a Juniper vMX, has the IP set as 192.168.99.101.

1
2
3
4
5
[19:58:26] at volos ➜ ~ ping 192.168.99.101
PING 192.168.99.101 (192.168.99.101) 56(84) bytes of data.
^C
--- 192.168.99.101 ping statistics ---
28 packets transmitted, 0 received, 100% packet loss, time 27011ms
1
2
3
4
5
6
7
8
9
10
11
[19:58:52] at volos ➜  ~ sudo tcpdump -i vmnet1 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vmnet1, link-type EN10MB (Ethernet), capture size 262144 bytes
19:59:08.641326 ARP, Request who-has 192.168.99.101 tell volos, length 28
19:59:09.644668 ARP, Request who-has 192.168.99.101 tell volos, length 28
19:59:10.647999 ARP, Request who-has 192.168.99.101 tell volos, length 28
19:59:12.578038 ARP, Request who-has 192.168.99.101 tell volos, length 28
19:59:12.597030 ARP, Reply 192.168.99.101 is-at 00:05:86:71:86:07 (oui Unknown), length 46
[20:02:07] at volos ➜  ~ ip neigh
192.168.99.101 dev vmnet1 lladdr 00:05:86:71:86:07 REACHABLE
192.168.99.2 dev vmnet1 lladdr 00:0c:29:7f:97:19 REACHABLE

On, vMX1, it’s possible to see the ping fails, but ARP resolution works as expected:

1
2
3
4
5
6
7
root@vMX1> ping 192.168.99.1
PING 192.168.99.1 (192.168.99.1): 56 data bytes
^C
--- 192.168.99.1 ping statistics ---
10 packets transmitted, 0 packets received, 100% packet loss
root@vMX1>

If you were to trace where the ICMP packets were being dropped, you’d see that it would be on vmnet1. On UNL, pnet1 bridge was bridging ICMP packets as expected:

1
2
3
4
5
6
7
8
9
root@unl01:~# sudo tcpdump -i pnet1 icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pnet1, link-type EN10MB (Ethernet), capture size 65535 bytes
23:01:21.505168 IP 192.168.99.101 > 192.168.99.1: ICMP echo request, id 24843, seq 0, length 64
23:01:22.512921 IP 192.168.99.101 > 192.168.99.1: ICMP echo request, id 24843, seq 1, length 64
23:01:23.523552 IP 192.168.99.101 > 192.168.99.1: ICMP echo request, id 24843, seq 2, length 64
23:01:24.533095 IP 192.168.99.101 > 192.168.99.1: ICMP echo request, id 24843, seq 3, length 64
23:01:25.543998 IP 192.168.99.101 > 192.168.99.1: ICMP echo request, id 24843, seq 4, length 64
23:01:26.553985 IP 192.168.99.101 > 192.168.99.1: ICMP echo request, id 24843, seq 5, length 64

Fixing Permissions

In order to fix the permissions on /dev/vmnet1, since I run vmware on systemd, I decided to chown /dev/vmnet1 to my standard user in the end of the start function in the /etc/init.d/vmware script. Consequently, whenever the service is started my standard user will get ownership of /dev/vmnet1 and I’ll be able to run VMware Workstation without being root and I’ll be able to use this adapter in promiscuous mode:

1
2
3
4
5
6
7
8
9
10
vmwareService() {
case "$1" in
start)
....
....
....
if [ -e /dev/vmnet1 ]; then
chown arcanjo:arcanjo /dev/vmnet1
fi

Now, you simply have to restart vmware service, and /dev/vmnet1 will be owned by your standard user, arcanjo in my particular example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
[20:04:35] sudo systemctl restart vmware
[20:04:44] sudo systemctl status vmware
● vmware.service - VMware daemon
Loaded: loaded (/usr/lib/systemd/system/vmware.service; enabled; vendor preset: disabled)
Active: active (exited) since Wed 2016-10-12 20:04:44 BRT; 6s ago
Process: 2760 ExecStop=/etc/init.d/vmware stop (code=exited, status=0/SUCCESS)
Process: 2884 ExecStart=/etc/init.d/vmware start (code=exited, status=0/SUCCESS)
Main PID: 2884 (code=exited, status=0/SUCCESS)
Tasks: 14 (limit: 4915)
CGroup: /system.slice/vmware.service
├─2983 /usr/lib/vmware/bin/vmware-vmblock-fuse -o subtype=vmware-vmblock,default_permissions,allow_other /var/run/vmblock-fuse
├─3018 /usr/bin/vmnet-bridge -s 6 -d /var/run/vmnet-bridge-0.pid -n 0
├─3032 /usr/bin/vmnet-netifup -s 6 -d /var/run/vmnet-netifup-vmnet1.pid /dev/vmnet1 vmnet1
├─3040 /usr/bin/vmnet-dhcpd -s 6 -cf /etc/vmware/vmnet1/dhcpd/dhcpd.conf -lf /etc/vmware/vmnet1/dhcpd/dhcpd.leases -pf /var/run/vmnet-dhcpd-vmnet1.pid vmnet1
├─3042 /usr/bin/vmnet-netifup -s 6 -d /var/run/vmnet-netifup-vmnet2.pid /dev/vmnet2 vmnet2
├─3050 /usr/bin/vmnet-dhcpd -s 6 -cf /etc/vmware/vmnet2/dhcpd/dhcpd.conf -lf /etc/vmware/vmnet2/dhcpd/dhcpd.leases -pf /var/run/vmnet-dhcpd-vmnet2.pid vmnet2
├─3052 /usr/bin/vmnet-netifup -s 6 -d /var/run/vmnet-netifup-vmnet3.pid /dev/vmnet3 vmnet3
├─3060 /usr/bin/vmnet-dhcpd -s 6 -cf /etc/vmware/vmnet3/dhcpd/dhcpd.conf -lf /etc/vmware/vmnet3/dhcpd/dhcpd.leases -pf /var/run/vmnet-dhcpd-vmnet3.pid vmnet3
├─3063 /usr/bin/vmnet-natd -s 6 -m /etc/vmware/vmnet8/nat.mac -c /etc/vmware/vmnet8/nat/nat.conf
├─3065 /usr/bin/vmnet-netifup -s 6 -d /var/run/vmnet-netifup-vmnet8.pid /dev/vmnet8 vmnet8
├─3073 /usr/bin/vmnet-dhcpd -s 6 -cf /etc/vmware/vmnet8/dhcpd/dhcpd.conf -lf /etc/vmware/vmnet8/dhcpd/dhcpd.leases -pf /var/run/vmnet-dhcpd-vmnet8.pid vmnet8
└─3091 /usr/sbin/vmware-authdlauncher
Oct 12 20:04:45 volos vmnet-dhcpd[3068]: Internet Software Consortium DHCP Server 2.0
Oct 12 20:04:45 volos vmnet-dhcpd[3068]: Copyright 1995, 1996, 1997, 1998, 1999 The Internet Software Consortium.
Oct 12 20:04:45 volos vmnet-dhcpd[3068]: All rights reserved.
Oct 12 20:04:45 volos vmnet-dhcpd[3068]:
Oct 12 20:04:45 volos vmnet-dhcpd[3068]: Please contribute if you find this software useful.
Oct 12 20:04:45 volos vmnet-dhcpd[3068]: For info, please visit http://www.isc.org/dhcp-contrib.html
Oct 12 20:04:45 volos vmnet-dhcpd[3068]:
Oct 12 20:04:45 volos vmware[2884]: [29B blob data]
Oct 12 20:04:45 volos vmware[2884]: [41B blob data]
Oct 12 20:04:45 volos vmware[2884]: [36B blob data]
[20:04:51] at volos ➜ ~
[20:04:52] at volos ➜ ~ ll /dev/vmnet1
crw------- 1 arcanjo arcanjo 119, 1 Oct 12 19:51 /dev/vmnet1
[20:04:57] at volos ➜ ~

If I were to test L3 reachability again, it works perfectly:

1
2
3
4
5
6
7
8
9
10
11
12
13
[22:33:58] at volos ➜ ~ ping 192.168.99.101
PING 192.168.99.101 (192.168.99.101) 56(84) bytes of data.
64 bytes from 192.168.99.101: icmp_seq=1 ttl=64 time=1.59 ms
64 bytes from 192.168.99.101: icmp_seq=2 ttl=64 time=1.45 ms
^C
--- 192.168.99.101 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 1.454/1.524/1.595/0.080 ms
[22:34:06] at volos ➜ ~ ip neigh
192.168.99.101 dev vmnet1 lladdr 00:05:86:71:ce:07 REACHABLE
192.168.99.2 dev vmnet1 lladdr 00:0c:29:7f:97:19 REACHABLE
192.168.25.1 dev wlp4s0 lladdr 2c:e4:12:7f:59:ad REACHABLE
[22:34:07] at volos ➜ ~

From vMX1’s (UNL node) perspective:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
root@vMX1> ping 192.168.99.1
PING 192.168.99.1 (192.168.99.1): 56 data bytes
64 bytes from 192.168.99.1: icmp_seq=0 ttl=64 time=1.965 ms
64 bytes from 192.168.99.1: icmp_seq=1 ttl=64 time=2.231 ms
64 bytes from 192.168.99.1: icmp_seq=2 ttl=64 time=2.707 ms
^C
--- 192.168.99.1 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.965/2.301/2.707/0.307 ms
root@vMX1> show arp
MAC Address Address Name Interface Flags
00:50:56:c0:00:01 192.168.99.1 192.168.99.1 ge-0/0/7.0 none
root@vMX1>

Final Thoughts

In conclusion, the external connectivity issue had nothing to do with UNL itself and was just a specific detail with Linux permissions and the way VMware Workstation is setup by default on Linux. So, I decided to write this post mentioning UNL just to facilitate for Linux users who might face this problem of pnet networks and bridging vmnet on VMware Workstation. I hope you guys enjoyed this post.