Adam Ayala

Dev/Ops Engineer

Linux SysAdmin

Sotware Developer

Infrastructure Analyst

Adam Ayala

Dev/Ops Engineer

Linux SysAdmin

Sotware Developer

Infrastructure Analyst

Blog Post

Split Tunneling a VPN on an Ubuntu Server

August 6, 2019 How-To, Linux, Ubuntu, Uncategorized

I decided I needed a single bare metal setup rather than using LXC/LXD containers to separate certain apps on my server. This required a split tunnel VPN. This enables the server to route certain traffic through a VPN connection, while everything else will have direct access, bypassing the VPN. Network traffic will be securely separated.

Install OpenVPN from the OpenVPN repository. I’ll use the actual OpenVPN repository just to install the latest version.

wget https://swupdate.openvpn.net/repos/repo-public.gpg -O - | sudo apt-key add -
echo "deb http://build.openvpn.net/debian/openvpn/stable xenial main" | sudo tee -a /etc/apt/sources.list.d/openvpn.list
sudo apt-get update && sudo apt-get install openvpn -

I create the systemd service that will start the required OpenVPN configuration on system start. This script will also restart the OpenVPN service if the service was terminated for some reason. Logs will be located in syslog (/var/log/syslog). For debugging purposes I’ll also add them to the OpenVPN config file later.

[Unit]
Description=OpenVPN connection to %i
Documentation=man:openvpn(8)
Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO
After=network.target

[Service]
RuntimeDirectory=openvpn
PrivateTmp=true
KillMode=mixed
Type=forking
ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn --script-security 2 --config /etc/openvpn/%i.conf --writepid /run/openvpn/%i.pid
PIDFile=/run/openvpn/%i.pid
ExecReload=/bin/kill -HUP $MAINPID
WorkingDirectory=/etc/openvpn
Restart=on-failure
RestartSec=3
ProtectSystem=yes
LimitNPROC=10
DeviceAllow=/dev/null rw
DeviceAllow=/dev/net/tun rw

[Install]
WantedBy=multi-user.target

Enable the openvpn@openvpn.service to start on boot.

sudo systemctl enable openvpn@openvpn.service

Create PIA (Private Internet Access)Configuration File for Split Tunneling

I use Private Internet Access as my VPN, but this should work with any VPN. The next step is to modify the configuration file provided by PIA to adjust it for the Split Tunneling. In this guide we will use the Sweden VPN server, but you can use any of the available servers. For best VPN performance (especially for torrents) I strongly recommend using UDP protocol, and not TCP. I’m going to also copy over the necessary certificates for PIA’s servers.

Get the Required Certificates for PIA

cd /tmp sudo wget https://www.privateinternetaccess.com/openvpn/openvpn.zipsudo unzip openvpn.zip
sudo cp crl.rsa.2048.pem ca.rsa.2048.crt /etc/openvpn/

Create Modified PIA Configuration File for Split Tunneling

We make the following changes to the default PIA configuration file. Add route-noexec to prevent the server from push “redirect-gateway” and make the client send all traffic over VPN by default.  Add auth-nocache to prevent caching passwords in memory. Add the call for login.txt file with the username and password to make automatic login possible. Add script-security 2 to allow client to call externals scripts (up and down).
Call the first script, iptables.sh, to mark packets for the VPN user (OpenVPN allows only one up call). Add down script, update-resolv-conf, to restore DNS servers when disconnecting from VPN. You can change the hostname highlighted in red to the server of your choice. For the list of available locations and hostnames look at PIA’s site and select the hostname of your choice.

We are editing the openvpn.conf file which is launched by the systemd service we created earlier (openvpn@openvpn.service). This way we ensure the up and down scripts we made are also executed with the OpenVPN systemd service is restarted.

Create the OpenVPN configuration file

sudo nano /etc/openvpn/openvpn.conf

Copy and paste the following, adjust the country you want to use by replacing sweden.privateinternetaccess.com

client
dev tun
proto udp
remote sweden.privateinternetaccess.com 1198
resolv-retry infinite
nobind
persist-key
persist-tun
cipher aes-128-cbc
auth sha1
tls-client
remote-cert-tls server
auth-user-pass /etc/openvpn/login.txt
auth-nocache
comp-lzo
verb 1
reneg-sec 0
crl-verify /etc/openvpn/crl.rsa.2048.pem
ca /etc/openvpn/ca.rsa.2048.crt
disable-occ
script-security 2
route-noexec

#up and down scripts to be executed when VPN starts or stops
up /etc/openvpn/iptables.sh
down /etc/openvpn/update-resolv-conf

#debugging
log /var/log/openvpn.log
pull-filter ignore "auth-token"

Make OpenVPN Auto Login on Service Start
The username and password for PIA will be stored in a login.txt file, this way OpenVPN can auto connect on service start. Create the txt file

sudo nano /etc/openvpn/login.txt

Enter your PIA username and password

USERNAME
PASSWORD

Configure VPN DNS Servers to Stop DNS Leaks

DNS Leaks are often the main reason your real identity gets exposed even if using VPN. The update-resolv-conf script that comes with OpenVPN will automatically apply the preferred DNS servers when OpenVPN connects.

This script will make sure that when using OpenVPN you are not subject to DNS leaks. We will use PIA’s DNS Servers (209.222.18.222 and 209.222.18.218) and Google’s (8.8.8.8) as a third option. You are free to use the DNS servers you trust and prefer. It is advised to change the local DNS to a public even if you are not using VPN. If you are behind a router (and you probably are), it is also a good practice to configure public DNS address on the router too.Note: make sure you are using a static IP on your machine or reserved DHCP also known as static DHCP. Do not configure the static IP on your server, as resolvconf will not work then. You should set the static IP from your router.

Open the update-resolv-conf file

sudo nano /etc/openvpn/update-resolv-conf

Add the following under the example DNS settings:

foreign_option_1='dhcp-option DNS 209.222.18.222'<br>foreign_option_2='dhcp-option DNS 209.222.18.218'<br>foreign_option_3='dhcp-option DNS 8.8.8.8'

Split Tunneling with IPtables and Routing Tables

We will use iptables to mark packets from a user (in our case a “vpn” user), and then use routing tables to route these marked packets through the OpenVPN interface, while allowing unmarked packets direct access to the Internet. Create the user vpn. All of the applications you want tunneled over VPN will run as this user. Create vpn user with no login option for security reasons as there is no need to log in to the system as the vpn user.

sudo adduser --disabled-login vpn

We are going to use the vpn user to run certain services, it is recommended to add your regular user to the vpn group and vpn user to your regular user’s group to avoid any permission issues.

Replace username with the user you would like to add to the vpn group

sudo usermod -aG vpn username

Replace group with the group name of your regular user that you would like to add the vpn user to

sudo usermod -aG group vpn

Get Routing Information for the iptables Script

We need the local IP and the name of the network interface. Again, make sure you are using a static IP on your machine or reserved DHCP also known as static DHCP, but configured on your router.

ip route list

The output will be similar to this

default via 192.168.1.1 dev enp0s25<br>192.168.1.0/24 dev enp0s25 proto
kernel scope link src 192.168.1.150

enp0s25 is the network interface (NETIF), and 192.168.1.150 is the local IP address (LOCALIP). You will need to enter these two into the following script, which we are going to create now.

Flush current iptables rules

sudo iptables -F

Add the following rule, which will block vpn user’s access to Internet (except the loopback device).

sudo iptables -A OUTPUT ! -o lo -m owner --uid-owner vpn -j DROP

Now install iptables-persistent to save this single rule that will be always applied on each system start.

sudo apt-get install iptables-persistent -y

During the install, iptables-persistent will ask you to save current iptables rules to /etc/iptables/rules.v4 if you choose YES at the prompt.

Now when system starts, vpn user is not able to access Internet. If the OpenVPN service is started successfully, then this rule is flushed (only until the next system restart), and the Split Tunnel rules are applied.

Scripts for managing iptables and routing.

The first script will mark the packets for vpn user, the second script will take care of proper routing. Create the iptables script

sudo nano /etc/openvpn/iptables.sh

Copy the following to the iptables.sh script, and make sure you enter the network interface and the local IP we identified earlier. You can see the comments for each section about the function of the given part of the script. If you are interested in more details about iptables, a good starting point is the Official Ubuntu Documentation. Remember, this script will flush your existing iptables rules (UFW included), therefore you need to append your own rules into this script if you need any additional firewall rules.

#! /bin/bash

export INTERFACE="tun0"
export VPNUSER="vpn"
export LOCALIP="192.168.1.0/24"
export NETIF="enp0s25"

# flushes all the iptables rules, if you have other rules to use then add them into the script
iptables -F -t nat
iptables -F -t mangle
iptables -F -t filter

# mark packets from $VPNUSER
iptables -t mangle -A OUTPUT -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT ! --dest $LOCALIP -m owner --uid-owner $VPNUSER -j MARK --set-mark 0x1
iptables -t mangle -A OUTPUT --dest $LOCALIP -p udp --dport 53 -m owner --uid-owner $VPNUSER -j MARK --set-mark 0x1
iptables -t mangle -A OUTPUT --dest $LOCALIP -p tcp --dport 53 -m owner --uid-owner $VPNUSER -j MARK --set-mark 0x1
iptables -t mangle -A OUTPUT ! --src $LOCALIP -j MARK --set-mark 0x1
iptables -t mangle -A OUTPUT -j CONNMARK --save-mark

# allow responses
iptables -A INPUT -i $INTERFACE -m conntrack --ctstate ESTABLISHED -j ACCEPT

# block everything incoming on $INTERFACE to prevent accidental exposing of ports
iptables -A INPUT -i $INTERFACE -j REJECT

# let $VPNUSER access lo and $INTERFACE
iptables -A OUTPUT -o lo -m owner --uid-owner $VPNUSER -j ACCEPT
iptables -A OUTPUT -o $INTERFACE -m owner --uid-owner $VPNUSER -j ACCEPT

# all packets on $INTERFACE needs to be masqueraded
iptables -t nat -A POSTROUTING -o $INTERFACE -j MASQUERADE

# reject connections from predator IP going over $NETIF
iptables -A OUTPUT ! --src $LOCALIP -o $NETIF -j REJECT

# Start routing script
/etc/openvpn/routing.sh

exit 0

Make the iptables script executable

sudo chmod +x /etc/openvpn/iptables.sh

With the routing rules we configure the route for the packets we just marked with the first script.

sudo nano /etc/openvpn/routing.sh

Paste the following script which makes the default route after the VPN the loopback interface, effectively nullifying the traffic if the VPN connection goes down.

#! /bin/bash
VPNIF="tun0"
VPNUSER="vpn"
GATEWAYIP=$(ifconfig $VPNIF | egrep -o '([0-9]{1,3}\.){3}[0-9]{1,3}' | egrep -v '255|(127\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})' | tail -n1)
if [[ `ip rule list | grep -c 0x1` == 0 ]]; then
ip rule add from all fwmark 0x1 lookup $VPNUSER
fi
ip route replace default via $GATEWAYIP table $VPNUSER
ip route append default via 127.0.0.1 dev lo table $VPNUSER
ip route flush cache

# run update-resolv-conf script to set VPN DNS
/etc/openvpn/update-resolv-conf

exit 0

Finally, make the script executable

sudo chmod +x /etc/openvpn/routing.sh


We will need a routing table by adding the table name to the rt_tables file (in our case it is vpn). Open rt_tables

sudo nano /etc/iproute2/rt_tables

Add the vpn user table at the bottom of the file

200 vpn

It should look similar to this:

#
# reserved values
#
255 local
254 main
253 default
0 unspec
#
# local
#
#
1 inr.ruhep
200 vpn

Finally, we need to change the default level of reverse path filtering to ensure the kernel routes the traffic correctly. By default it is set to value of 1 that is “strict mode”. It is not necessary to disable reverse path filtering completely (setting to “0”), but we need to set it to level 2, “loose mode”.

Create a reverse path filter file for the vpn user

sudo nano /etc/sysctl.d/9999-vpn.conf

Copy the following, make sure you use the correct network interface name in the third line (remember the ip route list command from before and the output, in my case it was enp0s25)

net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.enp0s25.rp_filter = 2

To apply new sysctl rules run:

sudo sysctl --system

Testing the VPN Split Tunnel
I recommend a system restart, and if everything was configured properly, you should have a running OpenVPN service enabled for the vpn user and all the other users on your server should have direct access to Internet. Now lets check if everything is correct. Login as your regular user over SSH, and check OpenVPN service status

sudo systemctl status openvpn@openvpn.service

This should return something like this:

openvpn@openvpn.service - OpenVPN connection to client
Loaded: loaded (/etc/systemd/system/openvpn@openvpn.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2016-09-05 11:25:18 CEST; 1 day 4h ago
Docs: man:openvpn(8)
https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
https://community.openvpn.net/openvpn/wiki/HOWTO
Process: 3223 ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn
Main PID: 3266 (openvpn)
CGroup: /system.slice/system-openvpn.slice/openvpn@openvpn.service
└─3266 /usr/sbin/openvpn --daemon ovpn-client --status /run/openvpn/client.status 10 --cd /etc/openvpn

Sep 05 11:25:21 server ovpn-client[3266]: OPTIONS IMPORT: route options modified
Sep 05 11:25:21 server ovpn-client[3266]: OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified

If the service is not running you can check if there is a log of the error in /var/log/syslog . For troubleshooting you can set output verbosity in the openvpn.conf file to a higher level. Set it to 3 and check the syslog again. Remember to set verbosity level back to 1 if you don’t need more detailed logs anymore.

Check IP address

Using the SSH session for the regular user, check the IP address

curl ipinfo.io

It will return your IP and depending on how much information is provided, the country should be listed in each case. Obviously, it should be your ISP now and your location.

Now check the IP address of the vpn user with

sudo -u vpn -i -- curl ipinfo.io

If everything went fine, it should return the IP address and the country of the VPN server you selected. If you used Sweden server, then the country should be “SE”. It is very important that the IP address for user vpn should be different then your regular user’s IP.

In my case for user vpn and using Sweden PIA server I have the following output

{
"ip": "X.XXX.XXX.XX",
"hostname": "No Hostname",
"city": "",
"region": "",
"country": "SE",
"loc": "59.3294,18.0686",
"org": "AS57858 Inter Connects Inc"

Obviously, the “x.xxx.xxx.xx” part is my assigned VPN IP address which is different then my public IP, and you can see the country as SE which is Sweden.

Check DNS Server
Finally, check if the DNS for VPN is properly configured, type

sudo -u vpn -i -- cat /etc/resolv.conf

The output should be

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 209.222.18.222
nameserver 209.222.18.218
nameserver 8.8.8.8

If you see the above DNS servers then your DNS for VPN is configured correctly.

Taggs:
Write a comment