Linux Policy Based Routing
How do you make each IP addresses function independently of the other IP addresses on the machine?
Linux offer a wide range of utilities and tools to accomplish almost any task, including this one. The tool we need to look at for this task is called ip. The man page for ip says "ip - show / manipulate routing, devices, policy routing and tunnels". We will use the policy routing aspect of this tool to accomplish our task. By default linux has one routing table, and 99% of the time that is all you need. But what happens in the above situation where you need to have multiple independently functioning IP addresses on the same machine? Traffic coming to the machine on a non-default IP address may leave the machine on the interface with the default route possibly using the IP address of that interface. This can cause problems with stateful firewall inspections upstream, or in the case of some UDP traffic the response may have the wrong source IP address. There can be any number of reasons why this will not work, and many reasons why you want it to work.
To setup policy based routing (PBR) there are some housekeeping tasks required. The systems I have setup PBR on are generally some variant of RedHat/CentOS, or Ubuntu/Mint. The concepts are the same for all modern linux distros, YMMV. Under the directory /etc/iproute2 there is one file you will need to edit: rt_tables. rt_tables simply enumerates your routing tables, by matching a routing table name to a routing table number. When using the ip utility it will be referring to the route table(s) by name, but you can also use numbers. I prefer to use names since they are descriptive, and help you, or someone else, understand what you are trying to accomplish at some point in the future.
I have used this technique for 2 or more IP addresses on the same server, and in theory you can set up more than 240 separate routing policies.
For this example we will call the routing policies: main, TABLE2, and TABLE3. Linux comes with three routing tables built in: local, main, and default. Default is not the default, main is, go figure. The "default" routing table, main, comes ready to use on the system. We will not alter the main(default) policy here, we assume that is setup and already in use via the standard system setup routines. This example will be using the following networks:
The first task is to tell the system about the two new tables we wish to use. The /etc/iproute2/rt_tables file is where this is done. The ip command will refer to this file to convert the table name to a table number. The rt_tables file needs to have the following lines added:
You have to assign a unique table ID to each table and the number has to be between 1 and 253. For the purposes of this example I have picked 12 and 13. The default routing table is 253, and you should find that listed in the rt_tables file already. After the file has been updated it will look like (this is taken from a mint 17.1 distro):
Each routing table needs to have a default route setup, and you would accomplish this with the following commands:
Notice how we did not include an entry for the system default routing table. The default table is already populated via the standard system configuration setup. You can add any route statement here for a any ip ranges, but for this example we are only concerned with network traffic coming and going to a particular device/IP (devX) and how it will get back to its own gateway.
The next step is to add rules that will tell the system when to use these alternate routing tables as opposed to the default routing table:
These rules simply state that if the source address for a packet leaving the machine matches 10.0.2.2 or 10.0.3.2 use the maching PBR table. The "prio" is the priority set within the Route Policy DataBase (RPDB). The RPDB is scanned for matching rules in the order of increasing priority. Picking a value greater than 0 and less than 32766 should work, I chose 1000 for this example. You can use the same value for both rule lines. For more information on RPDB refer to the ip man page.
Simply adding these four lines listed above to your local /etc/rc.local file is enough to set the machine up at boot to work with policy based routing.
One thing to note about this setup is that if another device on 10.0.1.x/24 attempts to connect to 10.0.2.2 the response will originate from 10.0.2.2, and travel back through 10.0.2.1 gateway and via whatever path exists to reach 10.0.1.x. The response will not come from the network device associated with 10.0.1.2. This is by design. If the response were to be emitted from 10.0.1.2 there are iptable implications, along with upstream firewall issues that could cause problems.
I have used this technique on linux systems that connected to multiple independent service providers with physical devices, served multiple IP tunnels over OpenVPN, as well as multiple IP addresses on several VLANs using the same physical device.