The author of the original article - https://t.me/Pryslonskyi
All rights belongs to him. Here is only .md adaptation and translation.
- Introduction
- Starting point
- Preliminary Setup
- Multi-WAN configuration
- Configuring ISP connections and implementing routing based on marks
This article is intended for novice administrators of Mikrotik RouterOS (hereafter ROS). It covers routing with failover for several unrelated multihomed AS internet channels operating on the IPv4 protocol. It also includes settings for basic security. The article does not describe "automatic" traffic balancing across channels due to the fact that implementing it with such initial data involves a series of compromises and limitations that require separate consideration. The article does provide tools for "manual" balancing, and the settings presented serve as a solid foundation for adding "automatic" balancing by at least half a dozen methods.
For our example, we've chosen a five-port Mikrotik router with ROS v6.45+ or v7+. It will route traffic between two local networks (LAN1 and LAN2) and three providers (ISP1, ISP2, ISP3). The connection to ISP1 has a static private address, ISP2 is a public address obtained via DHCP, and ISP3 is a public address with PPPoE authorization. The connection diagram is illustrated in the figure:
+------------------------------------------------------------------+
| Internet |
+-----------+---------------------+-------------------+------------+
/ | \
+----------------+ +----------------+ +----------------+
| ISP1 | | ISP2 | | ISP3 |
| 100.66.66.1/30 | | Public IPv4 | | Public IPv4 |
+-------+--------+ +-------+--------+ +-------+--------+
| | |
| | |
| | |
| | |
.....................................................................
. +--------+----------+ +---------+---------+ +---------+---------+ .
. | Ether1 | | Ether2 | | Ether3 | .
. | 100.66.66.2/30 | | DHCP | | PPPoE | .
. +-------------------+ +-------------------+ +-------------------+ .
. +-------------------+ +-------------------+ .
. | Ether4 | | Ether5 | .
. | 192.168.88.254/24 | MikroTik | 172.16.1.0/23 | .
. +---------+---------+ +---------+---------+ .
.....................................................................
| |
| |
| |
| |
+--------+--------+ +--------+--------+
| LAN1 | | LAN2 |
| 192.168.88.0/24 | | 172.16.0.0/23 |
+-----------------+ +-----------------+
The tasks are:
- Enable automatic failover to a backup provider. The primary provider is ISP2, with ISP1 as the first backup and ISP3 as the second backup.
- Set up LAN1 network access to the internet exclusively through ISP1.
- Allow for priority routing of traffic from local networks to the internet through a selected provider based on an address-list.
- Include the ability to publish services from the local network to the internet (DSTNAT).
- Configure the firewall filter to ensure a minimum level of security from the Internet.
- Enable the router to send its own traffic through any of the three providers, depending on the selected source address.
- Ensure that response packets are routed back through the channel from which they originated, including LAN.
Note: We will be configuring the router from scratch to ensure there are no surprises due to changes in the default "out-of-the-box" configurations that may vary from version to version. The settings will be applied using commands in the Winbox terminal. The physical connection for configuration will be made through a direct connection to the ether5 interface.
Reflecting on what multi-WAN is, whether it's a problem or just a web of conspiracies woven by clever folks around
A curious and attentive admin, setting up such a scheme on their own, might suddenly realize that things are working fine as is. Yes, without those custom routing tables and route rules that most articles on this topic are full of. Shall we verify?
Can we set up addressing on the interfaces and default gateways? Yes:
For ISP1, we set an address and a gateway with distance=2 and check-gateway=ping.
For ISP2, the default DHCP client setup means the distance will be 1.
For ISP3, in the PPPoE client settings, with add-default-route=yes, we set default-route-distance=3.
Let's not forget to set up NAT for outgoing traffic:
/ip firewall nat add action=masquerade chain=srcnat out-interface-list=WAN
As a result, LAN users happily browse their content via the primary provider ISP2, with channel reservation through the check gateway mechanism.
Task 1 accomplished. Where's the multi-WAN with its labels? Nowhere to be seen...
Next, we need to route specific LAN clients through ISP1:
/ip firewall mangle add action=route chain=prerouting dst-address-list=!BOGONS \
passthrough=yes route-dst=100.66.66.1 src-address-list=Via_ISP1
/ip firewall mangle add action=route chain=prerouting dst-address-list=!BOGONS \
passthrough=no route-dst=100.66.66.1 src-address=192.168.88.0/24
Tasks 2 and 3 accomplished. Labels, marks, route rules, where are you?!
Need to allow access to a favorite OpenVPN server at address 172.17.17.17 for internet clients? Easy:
/ip cloud set ddns-enabled=yes
Clients use the output of ":put [ip cloud get dns-name]" as their peer.
We set up port forwarding from the internet:
/ip firewall nat add action=dst-nat chain=dstnat dst-port=1194 \
in-interface-list=WAN protocol=udp to-addresses=172.17.17.17
Task 4 done.
We configure the firewall and other security measures for task 5, while happily noting that everything's already working for the users, and we reach for our favorite drink...
Oh, we forgot about the tunnels.
Did the L2TP client, set up according to some Googled article, connect to our favorite Dutch VDS? Yes.
Did the L2TP server with IPsec come up, and are clients connecting via the DNS name from IP Cloud (see above)? Yes.
Leaning back in our chair, sipping our drink, we lazily consider tasks 6 and 7. Do we really need to bother? Everything's working fine as is... So, if it's not needed, that's all there is to it. Multi-WAN implemented.
What is multi-WAN? It's connecting multiple internet channels to a single router.
You might as well stop reading the article here, as what else could there be beyond some flashy but dubious tricks?
For those who are still here, interested in tasks 6 and 7 and feeling the itch of perfectionism, let's dive deeper.
The primary task in implementing multi-WAN is to ensure correct traffic routing. Specifically, regardless of which provider channel(s) Note 3 the default route on our router points to, it must return the response through the same channel from which the packet was received. The task is clear. So, where's the problem? In a simple local network, the task is the same, but no one bothers with additional settings, and there are no issues. The difference is that any routable node on the Internet is accessible through each of our channels, not just one specific channel as in a simple local network. The "trouble" arises when a request comes in on the IP address for ISP3, but in our case, the response would go out through the ISP2 channel, as that's where the default gateway is pointed. It would be sent out and then dropped by the provider as invalid.
Having identified the problem, how do we solve it? The solution involves ensuring that outgoing traffic is routed back through the same ISP that the incoming request used. This can be achieved through various methods, such as policy-based routing, where rules are set up to match incoming traffic and ensure responses use the same route, or by using connection marking in conjunction with routing marks to keep track of the initial path of a connection and ensure the response follows the same path back.
Let's break down the solution into three stages:
-
Preliminary Setup: In this stage, we'll define the router's basic configurations, including the local network, firewall, address lists, hairpin NAT, and more.
-
Multi-WAN configuration: During this stage, we will mark and sort the required connections into routing tables. This involves identifying and classifying traffic to ensure that it is routed through the correct ISP, based on criteria such as source or destination addresses.
-
ISP Connection: In this final stage, we'll configure the interfaces that provide Internet connectivity, implement routing, and activate the Internet channel backup mechanism.
Note: The choice of three different types of ISP connections is deliberate, to illustrate that setting up multi-WAN configurations with dynamic addresses is not insurmountable and to demonstrate one of the solution approaches.
Important! The channel switching mechanism, determined by the route cost distance, utilizes the check gateway mechanism. The scripts provided in the article are not related to channel reservation but are intended to automate the resolution of dynamic IP issues on the ISP2 and ISP3 channels.
/system reset-configuration skip-backup=yes no-defaults=yes
Agree with "Dangerous! Reset anyway? [y/N]:" and, after rebooting, connect via Winbox using MAC. At this point, the configuration and user base are cleared.
1.2. In ROS 6.49+, decline the password change for the default user (select "Cancel" in Winbox and press "Ctrl+C" in the terminal). Create a new user:
/user add group=full name=knight password=ultrasecret comment="Not horse"
Log in as this user and remove the default one:
/user remove admin
Note: The author considers deleting the default user more secure than disabling it and recommends this practice.
1.3. Create basic interface lists for easier management in the firewall, discovery settings, and other MAC servers:
/interface list add name=WAN comment="For Internet"
/interface list add name=LAN comment="For Local Area"
Label interfaces with comments:
/interface ethernet set ether1 comment="to ISP1"
/interface ethernet set ether2 comment="to ISP2"
/interface ethernet set ether3 comment="to ISP3"
/interface ethernet set ether4 comment="to LAN1"
/interface ethernet set ether5 comment="to LAN2"
And populate the interface lists:
/interface list member add interface=ether1 list=WAN comment="to ISP1"
/interface list member add interface=ether2 list=WAN comment="to ISP2"
/interface list member add interface=ether3 list=WAN comment="to ISP3"
/interface list member add interface=ether4 list=LAN comment="to LAN1"
/interface list member add interface=ether5 list=LAN comment="to LAN2"
Note: Writing clear comments is worth the time and greatly aids in understanding the configuration. For security reasons, it's recommended to add the ether3 interface to the "WAN" interface list, even though it won't carry IP protocol traffic. Remember to add the PPP interface to the "WAN" list once it's set up.
/ip neighbor discovery-settings set discover-interface-list=!WAN
/tool mac-server set allowed-interface-list=LAN
/tool mac-server mac-winbox set allowed-interface-list=LAN
/ip firewall filter add action=accept chain=input \
comment="Related Established Untracked Allow" \
connection-state=established,related,untracked
(Allows established, related, and untracked connections, initiated from connected networks or by the router itself.)
/ip firewall filter add action=accept chain=input \
comment="ICMP from ALL" protocol=icmp
(Allows all ICMP traffic, useful for troubleshooting MTU issues.)
/ip firewall filter add action=drop chain=input comment="All other WAN Drop" \
in-interface-list=WAN
(Drops all other incoming traffic from the Internet.)
/ip firewall filter add action=accept chain=forward \
comment="Established, Related, Untracked allow" \
connection-state=established,related,untracked
(Allows established and related connections passing through the router.)
/ip firewall filter add action=drop chain=forward comment="Invalid drop" \
connection-state=invalid
(Drops invalid connections passing through the router. Recommended by Mikrotik but may block legitimate traffic in some routing scenarios.)
/ip firewall filter add action=drop chain=forward \
comment="Drop all from WAN not DSTNATed" connection-nat-state=!dstnat \
connection-state=new in-interface-list=WAN
(Prevents packets from the Internet that haven't gone through DSTNAT from passing through the router, protecting local networks from external threats.)
Note: Assume LAN1 and LAN2 are trusted networks, and traffic between them and from them is not filtered.
/ip firewall address-list
add address=0.0.0.0/8 comment="\"This\" Network" list=BOGONS
add address=10.0.0.0/8 comment="Private-Use Networks" list=BOGONS
add address=100.64.0.0/10 comment="Shared Address Space. RFC 6598" list=BOGONS
add address=127.0.0.0/8 comment=Loopback list=BOGONS
add address=169.254.0.0/16 comment="Link Local" list=BOGONS
add address=172.16.0.0/12 comment="Private-Use Networks" list=BOGONS
add address=192.0.0.0/24 comment="IETF Protocol Assignments" list=BOGONS
add address=192.0.2.0/24 comment=TEST-NET-1 list=BOGONS
add address=192.168.0.0/16 comment="Private-Use Networks" list=BOGONS
add address=198.18.0.0/15 comment="Network Interconnect Device Benchmark Testing"\
list=BOGONS
add address=198.51.100.0/24 comment=TEST-NET-2 list=BOGONS
add address=203.0.113.0/24 comment=TEST-NET-3 list=BOGONS
add address=224.0.0.0/4 comment=Multicast list=BOGONS
add address=192.88.99.0/24 comment="6to4 Relay Anycast" list=BOGONS
add address=240.0.0.0/4 comment="Reserved for Future Use" list=BOGONS
add address=255.255.255.255 comment="Limited Broadcast" list=BOGONS
(This list includes addresses and networks not routable on the Internet, and we will adhere to this convention.)
Note: This list may change, so it's advisable to check for updates periodically.
/ip dns set servers=1.1.1.1,8.8.8.8
Note: In the current version of ROS, statically assigned DNS servers take precedence over those provided dynamically. The system sends the name resolution request to the first server in the list, moving to the next server if the current one is unavailable. The timeout is significant—over 5 seconds—and the system does not automatically revert to a previously unavailable server once it comes back online. Given this behavior and the presence of multi-WAN, the author recommends not using DNS servers provided by ISPs.
/ip address add interface=ether4 address=192.168.88.254/24 comment="LAN1 IP"
/ip address add interface=ether5 address=172.16.1.0/23 comment="LAN2 IP"
For ROS6:
/ip route rule add dst-address=192.168.88.0/24 table=main comment="to LAN1"
/ip route rule add dst-address=172.16.0.0/23 table=main comment="to LAN2"
For ROS7:
/routing/rule/add action=lookup dst-address=192.168.88.0/24 table=main comment="to LAN1"
/routing/rule/add action=lookup dst-address=172.16.0.0/23 table=main comment="to LAN2"
Note: This is a simple and quick way to access local network addresses using the external IP addresses of the router's interfaces, regardless of which channel the default route leads to.
/ip firewall nat add action=src-nat chain=srcnat comment="Hairpin to LAN1" out-interface=ether4 src-address=192.168.88.0/24 to-addresses=192.168.88.254
/ip firewall nat add action=src-nat chain=srcnat comment="Hairpin to LAN2" out-interface=ether5 src-address=172.16.0.0/23 to-addresses=172.16.1.0
Note: This allows users from LAN1 and LAN2 to access servers within the same network segment via the external IP (dstnat).
/routing/table/add disabled=no fib name=to_isp1
/routing/table/add disabled=no fib name=to_isp2
/routing/table/add disabled=no fib name=to_isp3
Note: In ROS7, the approach to marks and routing tables has been reimagined. For instance, in ROS6, the new-routing-mark option in the mangle rule's mark-routing action directly created a new mark and an associated routing table in the system. In ROS7, akin to iproute2 in Linux, custom tables must be created beforehand. Only then do operations with them become available in the corresponding rules.
RouterOS packet flow theory.
To implement the correct multi-WAN setup, ensuring responses are sent back through the same channel from which requests were received, we'll use two RouterOS tools: connection mark and routing mark. Connection mark allows us to tag a specific connection, which can then be used as a condition for applying a routing mark. With a routing mark, we can then direct traffic within ip route and route rules. Having understood the tools, the next steps involve deciding which connections to mark and precisely where to apply these marks.
Firstly, we need to tag all connections that come into the router from the Internet via the respective ISP channel. In our scenario, this means creating three marks corresponding to the number of channels: "conn_isp1", "conn_isp2", and "conn_isp3".
The nuance with the second part lies in recognizing that incoming connections will be of two types: transit (passing through the router to the local network) and those intended for the router itself (like management or service traffic). The connection mark mechanism operates within the mangle table of RouterOS.
Lets look at the flow of routed forward packet:
-
The packet enters prerouting processing:
a. check if there is a hotspot and modify the packet for hotspot use;
b. process packet through RAW prerouting chain;
c. send the packet through connection tracking;
d. process packet through Mangle prerouting chain;
e. process packet through NATs dst-nat chain; -
Run packet through routing table to make routing decision;
-
The packet enters the forward process:
a. check TTL value;
b. process packet through Mangle forward chain;
c. process packet through the Filter forward chain;
d. send the packet to accounting processes; -
A packet enters postrouting process:
a. process packet through Mangle postrouting chain;
b. process packet through NATs src-nat chain;
c. if there is a hotspot undo any modifications made in hotspot-in;
d. process packet through queue tree (HTB Global);
e. process packet through simple queues; -
Check if there is IPsec and then process through IPsec policies.
And input:
When a packet's destination is a router (routing input):
-
Packet enters prerouting processing:
a. check if there is a hotspot and modify the packet for hotspot use;
b. process packet through RAW prerouting chain;
c. send a packet through connection tracking;
d. process packet through Mangle prerouting chain;
e. process packet through NATs dst-nat chain; -
Run packet through routing table to make routing decision;
-
A Packet enters the input process;
a. process packet through Mangle input chain;
b. process packet through Filter input chain;
c. process packet through queue tree (HTB Global);
d. process packet through simple queues; -
Check if there is IPsec and then process through IPsec policies.
Also output:
When a packet is originated from the router (routing output):
-
The packet is originated from the router itself:
a. the packet goes through the routing table to make a routing decision -
A packet enters the output process: a. process packet through the Bridge decision;
b. send the packet through connection tracking;
c. process packet through the Mangle output chain;
d. process packet through the Filter output chain;
e. send the packet to routing adjustment ( policy routing); -
The packet enters postrouting process:
a. process packet through Mangle postrouting chain;
b. process packet through NATs src-nat chain;
c. if there is a hotspot undo any modifications made in hotspot-in;
d. process packet through queue tree (HTB Global);
e. process packet through simple queues; -
Check if there is IPsec and then process through IPsec policies;
Following the packet flow, we observe that a packet arriving at the "input interface" passes through the "Prerouting" chain before being divided into transit and local traffic at the "Routing Decision" point. Therefore, to address both scenarios efficiently, we'll utilize the Connection Mark in the Prerouting chain of the Mangle table.
Note: In RouterOS, "Routing Marks" are referred to as "Table" in the ip/routes/rules (routing/rules in ROS v7) section, while in other sections, they are called "Routing Mark". This discrepancy might cause some confusion, but essentially, they refer to the same concept and serve as the equivalent of rt_tables in iproute2 on Linux.
For ISP1:
/ip firewall mangle add action=mark-connection chain=prerouting \
comment="Connmark in from ISP1" connection-mark=no-mark in-interface=ether1 \
new-connection-mark=conn_isp1 passthrough=no
For ISP2:
/ip firewall mangle add action=mark-connection chain=prerouting \
comment="Connmark in from ISP2" connection-mark=no-mark in-interface=ether2 \
new-connection-mark=conn_isp2 passthrough=no
For ISP3:
/ip firewall mangle add action=mark-connection chain=prerouting \
comment="Connmark in from ISP3" connection-mark=no-mark in-interface=pppoe-isp3 \
new-connection-mark=conn_isp3 passthrough=no
Note: If you're following these settings step by step, you might encounter an error when entering the third command due to the absence of the "pppoe-isp3" interface, which will be configured later in section 3.3.2. For now, you can replace "pppoe-isp3" with "ether3". Remember to update this to the correct interface name after completing section 3.3.2.
Using connection-mark=no-mark instead of connection-state=new is a more universal approach to avoid marking already marked connections.
passthrough=no is used because, in this implementation method, re-marking is excluded, and processing can be stopped after the first rule match to speed things up.
Keep in mind that at this stage, we are not yet intervening in the routing itself. These steps are preparatory. The next phase will involve handling transit traffic returning through established connections to recipients in the local network, i.e., packets that have followed the path: “Input Interface” => “Prerouting” => “Routing Decision” => “Forward” => “Post Routing” => “Output Interface” and have reached their destination in the local network.
Important! In RouterOS, there is no logical distinction between external and internal interfaces. If we trace the path of a response packet, it follows the same logical route as the request: “Input Interface” => “Prerouting” => “Routing Decision” => “Forward” => “Post Routing” => “Output Interface”, with the only difference being that the “Input Interface” for the request is an ISP interface, and for the response, it's a LAN interface.
To direct the return transit traffic through the corresponding routing tables, the following commands are used in the RouterOS firewall mangle section:
For traffic returning via ISP1:
/ip firewall mangle add action=mark-routing chain=prerouting \
comment="Routemark transit out via ISP1" connection-mark=conn_isp1 \
dst-address-type=!local in-interface-list=!WAN new-routing-mark=to_isp1 passthrough=no
For traffic returning via ISP2:
/ip firewall mangle add action=mark-routing chain=prerouting \
comment="Routemark transit out via ISP2" connection-mark=conn_isp2 \
dst-address-type=!local in-interface-list=!WAN new-routing-mark=to_isp2 passthrough=no
For traffic returning via ISP3:
/ip firewall mangle add action=mark-routing chain=prerouting \
comment="Routemark transit out via ISP3" connection-mark=conn_isp3 \
dst-address-type=!local in-interface-list=!WAN new-routing-mark=to_isp3 passthrough=no
Note: in-interface-list=!WAN indicates we're only dealing with traffic from the local network, and dst-address-type=!local ensures we're not targeting traffic destined for the router's own interfaces.
The same principle applies to local packets that have reached the router through the path:
“Input Interface” => “Prerouting” => “Routing Decision” => “Input” => “Local Process”
Important!: The response will follow the path:
“Local Process” => “Routing Decision” => “Output” => “Post Routing” => “Output Interface”
To ensure the return local traffic is directed through the correct routing tables, the following commands are set up in the RouterOS firewall mangle section:
For local traffic returning via ISP1:
/ip firewall mangle add action=mark-routing chain=output \
comment="Routemark local out via ISP1" connection-mark=conn_isp1 \
dst-address-type=!local new-routing-mark=to_isp1 passthrough=no
For local traffic returning via ISP2:
/ip firewall mangle add action=mark-routing chain=output \
comment="Routemark local out via ISP2" connection-mark=conn_isp2 \
dst-address-type=!local new-routing-mark=to_isp2 passthrough=no
For local traffic returning via ISP3:
/ip firewall mangle add action=mark-routing chain=output \
comment="Routemark local out via ISP3" connection-mark=conn_isp3 \
dst-address-type=!local new-routing-mark=to_isp3 passthrough=no
At this point, the preparation for sending responses back through the internet channel from which the request originated can be considered complete. Everything is marked, tagged, and ready for routing.
An excellent "side" effect of such configuration is the ability for DstNAT port forwarding to work simultaneously with both ISPs (ISP2, ISP3), except ISP1, where we have a non-routable address. This feature is crucial for scenarios such as a mail server with two MX records pointing to different Internet channels or a service accessible via a DNS name that resolves to the IP addresses of both providers.
To address nuances in the interaction of local networks with the router's external IPs, solutions from sections 1.8.2 and 3.1.2.6 can be utilized.
Furthermore, this marking tool can also be employed to address task 3. This can be implemented as follows:
To route traffic from local clients specified in address lists to the corresponding routing tables, use the following commands in the RouterOS firewall mangle section:
For traffic designated to go through ISP2:
/ip firewall mangle add action=mark-routing chain=prerouting \
comment="Address List via ISP1" dst-address-list=!BOGONS new-routing-mark=to_isp1 \
passthrough=no src-address-list=Via_ISP1
For traffic designated to go through ISP3:
/ip firewall mangle add action=mark-routing chain=prerouting \
comment="Address List via ISP3" dst-address-list=!BOGONS new-routing-mark=to_isp3 \
passthrough=no src-address-list=Via_ISP3
Note: These rules are provided as examples. It's important to consider the following:
- With only two internet channels, it's sensible to set just one rule for the backup channel.
- With multiple channels, it doesn't make sense to create a rule for the primary provider.
- The current failover settings allow marked traffic to exit through the backup channel(s) if the "target" internet channel fails.
- Remember that rules are processed in sequence, in the order they are written. This is crucial if different address lists contain overlapping addresses/networks. Without passthrough, the first matching rule will take effect; with passthrough, the last one will. Keep this in mind when specifying overlapping network ranges in different "address-list" to avoid unintended routing behaviors.
/ip address add interface=ether1 address=100.66.66.2/30 comment="ISP1 IP"
/interface bridge add name=br-lo comment="Loopback interface"
/ip route add distance=254 gateway=br-lo comment="Emergency route"
Note: This route allows traffic from local processes to pass the Route Decision stage regardless of the status of any provider's channels. The nuance with outgoing local traffic is that for a packet to go anywhere, there must be an active route to the default gateway in the main routing table. If it's absent, the packet will simply be discarded.
To enhance the check gateway tool for deeper channel status analysis, consider using recursive routes. The essence of this method is specifying a route through an intermediate gateway. Consequently, the check gateway will verify the availability of this intermediate gateway, not the ISP's gateway. For ISP1, ISP2, and ISP3, the "test" gateways will be 4.2.2.1, 4.2.2.2, and 4.2.2.3, respectively (these are public addresses of Level3DNS). You can choose any other accessible addresses based on your preferences.
/ip route add check-gateway=ping comment="For recursion via ISP1" \
distance=1 dst-address=4.2.2.1 gateway=100.66.66.1 scope=11
Note: In ROS7, the conditions for constructing recursive routes have changed slightly from ROS6. A universal solution suitable for both versions involves ensuring the 'scope' value of the "base route" is higher than the 'target-scope' and equal to the 'target-scope' of the recursive route.
/ip route add check-gateway=ping comment="Unmarked via ISP1" \
distance=2 gateway=4.2.2.1 target-scope=11
Note: The use of distance=2 in routing configurations is deliberate, as ISP1 is designated as the first backup provider according to the task requirements.
For ROS6:
/ip route add comment="Marked via ISP1 Main" distance=1 gateway=4.2.2.1 \
routing-mark=to_isp1 target-scope=11
For ROS7 (replace 'routing-mark' with 'routing-table'):
/ip route add comment="Marked via ISP1 Main" distance=1 gateway=4.2.2.1 \
routing-table=to_isp1 target-scope=11
Note: This is where we start utilizing the preparatory work done in section 2. Through this route, all traffic marked with "to_isp1" will be directed to the gateway of the first provider, regardless of the current active default gateway in the main table.
For ROS6:
/ip route add comment="Marked via ISP2 Backup1" distance=2 gateway=4.2.2.1 \
routing-mark=to_isp2 target-scope=11
/ip route add comment="Marked via ISP3 Backup1" distance=2 gateway=4.2.2.1 \
routing-mark=to_isp3 target-scope=11
For ROS7 (replace 'routing-mark' with 'routing-table'):
/ip route add comment="Marked via ISP2 Backup1" distance=2 gateway=4.2.2.1 \
routing-table=to_isp2 target-scope=11
/ip route add comment="Marked via ISP3 Backup1" distance=2 gateway=4.2.2.1 \
routing-table=to_isp3 target-scope=11
Note: These routes are also necessary for backing up traffic from local networks that are members of the address list "to_isp".*
For ROS6:
/ip route rule add comment="From ISP1 IP to Inet" src-address=100.66.66.2 table=to_isp1
For ROS7:
/routing/rule/add action=lookup comment="From ISP1 IP to Inet" \
src-address=100.66.66.2 table=to_isp1
Note: Combined with rules from section 1.8.2, this ensures the traffic exits through the intended channel with the specified source. This is critical for establishing tunnels (EoIP, IP-IP, GRE, L2TP, etc.) where the local side's IP address is specified. Since ip route rules are executed from top to bottom until the first match, this rule should be placed after the rules from section 1.8.2.
/ip firewall nat add action=src-nat chain=srcnat comment="NAT via ISP1" \
ipsec-policy=out,none out-interface=ether1 to-addresses=100.66.66.2
Note: NAT all outgoing traffic except for what matches IPsec policies. Prefer using 'src-nat' over 'masquerade' when possible, as 'masquerade' is slower and more resource-intensive because it recalculates the NAT address for each new connection. However, remember that 'masquerade' has features like clearing connections if a channel drops and nuances regarding address selection if there are multiple addresses from different networks on the interface.
3.1.4. Direct clients from the list, prohibited from exiting through other providers, straight to the gateway of ISP1
/ip firewall mangle add action=route chain=prerouting \
comment="Address List via ISP1 only" dst-address-list=!BOGONS passthrough=no \
route-dst=100.66.66.1 src-address-list=Via_only_ISP1 place-before=0
Note: The action=route has a higher priority and is applied before other routing rules. The place-before=0 places this rule at the top of the list, ensuring it's processed first.
Since ISP2 provides settings via DHCP, it's practical to handle the configuration and any potential changes through a script triggered by the DHCP client's events. This approach ensures that any updates provided by ISP2, such as changes in the IP address, subnet mask, or gateway, are automatically applied without the need for manual intervention.
ROS6:
/ip dhcp-client
add add-default-route=no disabled=no interface=ether2 script=":if (\$bound=1) do={\r\
\n /ip route remove [ find gateway=\"4.2.2.2\" ]; /ip route remove [ find where \
dst-address ~\"4.2.2.2\" ]\r\
\n /ip route add check-gateway=ping comment=\"For recursion via ISP2\" distance=1 \
dst-address=4.2.2.2/32 gateway=\$\"gateway-address\" scope=11\r\
\n /ip route add check-gateway=ping comment=\"Unmarked via ISP2\" \
distance=1 gateway=4.2.2.2 target-scope=11\r\
\n /ip route add comment=\"Marked via ISP2 Main\" distance=1 gateway=4.2.2.2 \
routing-mark=to_isp2 target-scope=11\r\
\n /ip route add comment=\"Marked via ISP1 Backup1\" distance=2 gateway=4.2.2.2 \
routing-mark=to_isp1 target-scope=11\r\
\n /ip route add comment=\"Marked via ISP3 Backup2\" distance=3 gateway=4.2.2.2 \
routing-mark=to_isp3 target-scope=11\r\
\n :if [:tobool ([/ip firewall nat find comment=\"NAT via ISP2\"])] do={\r\
\n /ip firewall nat set [find comment=\"NAT via ISP2\"] action=src-nat \
chain=srcnat ipsec-policy=out,none out-interface=\$\"interface\" \
to-addresses=\$\"lease-address\" \r\
\n } else={/ip firewall nat add action=src-nat chain=srcnat ipsec-policy=out,none \
out-interface=\$\"interface\" to-addresses=\$\"lease-address\" \
comment=\"NAT via ISP2\"}\r\
\n :if [:tobool ([/ip route rule find comment=\"From ISP2 IP to Inet\"])] do={\r\
\n /ip route rule set [find comment=\"From ISP2 IP to Inet\"] \
src-address=\$\"lease-address\" table=to_isp2\r\
\n } else={/ip route rule add comment=\"From ISP2 IP to Inet\" \
src-address=\$\"lease-address\" table=to_isp2}\r\
\n} else={\r\
\n /ip route remove [find gateway=\"4.2.2.2\"]; /ip route remove [find where \
dst-address ~\"4.2.2.2\"]\r\
\n /ip firewall nat remove [find comment=\"NAT via ISP2\"]\r\
\n /ip route rule remove [find comment=\"From ISP2 IP to Inet\"]\r\
\n}\r\
\n" use-peer-dns=no use-peer-ntp=no
ROS7:
/ip dhcp-client
add add-default-route=no dhcp-options=clientid,clientid interface=ether2 \
script=":if (\$bound=1) do={\r\
\n /ip route remove [ find gateway=\"4.2.2.2\" ]; /ip route remove \
[ find where dst-address ~\"4.2.2.2\" ]\r\
\n /ip route add check-gateway=ping comment=\"For recursion via ISP2\" distance=1 \
dst-address=4.2.2.2/32 gateway=\$\"gateway-address\" scope=11\r\
\n /ip route add check-gateway=ping comment=\"Unmarked via ISP2\" \
distance=1 gateway=4.2.2.2 target-scope=11\r\
\n /ip route add comment=\"Marked via ISP2 Main\" distance=1 gateway=4.2.2.2 \
routing-table=to_isp2 target-scope=11\r\
\n /ip route add comment=\"Marked via ISP1 Backup1\" distance=2 gateway=4.2.2.2 \
routing-table=to_isp1 target-scope=11\r\
\n /ip route add comment=\"Marked via ISP3 Backup2\" distance=3 gateway=4.2.2.2 \
routing-table=to_isp3 target-scope=11\r\
\n :if [:tobool ([/ip firewall/nat/ find comment=\"NAT via ISP2\"])] do={\r\
\n /ip firewall nat set [find comment=\"NAT via ISP2\"] action=src-nat \
chain=srcnat ipsec-policy=out,none out-interface=\$\"interface\" \
to-addresses=\$\"lease-address\" \r\
\n } else={/ip firewall nat add action=src-nat chain=srcnat ipsec-policy=out,none \
out-interface=\$\"interface\" to-addresses=\$\"lease-address\" \
comment=\"NAT via ISP2\"}\r\
\n :if [:tobool ([/routing/rule find comment=\"From ISP2 IP to Inet\"])] do={\r\
\n /routing/rule/set [find comment=\"From ISP2 IP to Inet\"] \
action=lookup src-address=\$\"lease-address\" table=to_isp2\r\
\n } else={/routing/rule/add action=lookup comment=\"From ISP2 IP to Inet\" \
src-address=\$\"lease-address\" table=to_isp2 }\r\
\n} else={\r\
\n /ip route remove [find gateway=\"4.2.2.2\"]; /ip route remove [find where \
dst-address ~\"4.2.2.2\"]\r\
\n /ip firewall nat remove [find comment=\"NAT via ISP2\"]\r\
\n /routing/rule/remove [find comment=\"From ISP2 IP to Inet\"]\r\
\n}\r\
\n" use-peer-dns=no use-peer-ntp=no
Here is the content of the DHCP client script for ROS6 and ROS7, focusing on the body of the script:
ROS6:
:if ($bound=1) do={
/ip route remove [ find gateway="4.2.2.2" ]; /ip route remove [ find where \
dst-address ~"4.2.2.2" ]
/ip route add check-gateway=ping comment="For recursion via ISP2" distance=1 \
dst-address=4.2.2.2/32 gateway=$"gateway-address" scope=11
/ip route add check-gateway=ping comment="Unmarked via ISP2" distance=1 \
gateway=4.2.2.2 target-scope=11
/ip route add comment="Marked via ISP2 Main" distance=1 gateway=4.2.2.2 \
routing-mark=to_isp2 target-scope=11
/ip route add comment="Marked via ISP1 Backup1" distance=2 gateway=4.2.2.2 \
routing-mark=to_isp1 target-scope=11
/ip route add comment="Marked via ISP3 Backup2" distance=3 gateway=4.2.2.2 \
routing-mark=to_isp3 target-scope=11
:if [:tobool ([/ip firewall nat find comment="NAT via ISP2"])] do={
/ip firewall nat set [find comment="NAT via ISP2"] action=src-nat chain=srcnat \
ipsec-policy=out,none out-interface=$"interface" to-addresses=$"lease-address"
} else={ /ip firewall nat add action=src-nat chain=srcnat ipsec-policy=out,none \
out-interface=$"interface" to-addresses=$"lease-address" comment="NAT via ISP2"}
:if [:tobool ([/ip route rule find comment="From ISP2 IP to Inet"])] do={
/ip route rule set [find comment="From ISP2 IP to Inet"] \
src-address=$"lease-address" table=to_isp2
} else={ /ip route rule add comment="From ISP2 IP to Inet" \
src-address=$"lease-address" table=to_isp2}
} else={
/ip route remove [find gateway="4.2.2.2"]; /ip route remove [find where \
dst-address ~"4.2.2.2"]
/ip firewall nat remove [find comment="NAT via ISP2"]
/ip route rule remove [find comment="From ISP2 IP to Inet"]
}
ROS7:
:if ($bound=1) do={
/ip route remove [ find gateway="4.2.2.2" ]; /ip route remove \
[ find where dst-address ~"4.2.2.2" ]
/ip route add check-gateway=ping comment="For recursion via ISP2" distance=1 \
dst-address=4.2.2.2/32 gateway=$"gateway-address" scope=11
/ip route add check-gateway=ping comment="Unmarked via ISP2" distance=1 \
gateway=4.2.2.2 target-scope=11
/ip route add comment="Marked via ISP2 Main" distance=1 gateway=4.2.2.2 \
routing-table=to_isp2 target-scope=11
/ip route add comment="Marked via ISP1 Backup1" distance=2 gateway=4.2.2.2 \
routing-table=to_isp1 target-scope=11
/ip route add comment="Marked via ISP3 Backup2" distance=3 gateway=4.2.2.2 \
routing-table=to_isp3 target-scope=11
:if [:tobool ([/ip firewall/nat/ find comment="NAT via ISP2"])] do={
/ip firewall nat set [find comment="NAT via ISP2"] action=src-nat chain=srcnat \
ipsec-policy=out,none out-interface=$"interface" to-addresses=$"lease-address"
} else={ /ip firewall nat add action=src-nat chain=srcnat ipsec-policy=out,none \
out-interface=$"interface" to-addresses=$"lease-address" comment="NAT via ISP2" }
:if [:tobool ([/routing/rule find comment="From ISP2 IP to Inet"])] do={
/routing/rule/set [find comment="From ISP2 IP to Inet"] action=lookup \
src-address=$"lease-address" table=to_isp2
} else={ /routing/rule/add action=lookup comment="From ISP2 IP to Inet" \
src-address=$"lease-address" table=to_isp2 }
} else={
/ip route remove [ find gateway="4.2.2.2" ]; /ip route remove [ find where \
dst-address ~"4.2.2.2" ]
/ip firewall nat remove [find comment="NAT via ISP2"]
/routing/rule/remove [find comment="From ISP2 IP to Inet"]
}
Note: The first part of the script is executed upon successful DHCP lease acquisition, and the second part upon lease release.note 2
Important! In some cases, the gateway might not respond to ICMP requests, which can occur with LTE connections in passthrough mode or if the provider filters ICMP at the gateway. In such scenarios, for the route "For recursion via ISP(x)", instead of check-gateway=ping, you could use check-gateway=arp.
Given that ISP3 provides dynamic settings, it makes sense to implement necessary changes through scripts that are triggered after the PPP interface goes up and down.
Note: The PPP interface can be specified as a gateway instead of an IP address. However, in such a case, using a recursive route becomes unfeasible. Therefore, we capture the provider's side IP address in a variable and use it in the same manner as we do for other providers.
For ROS7:
/ppp profile
add comment="for PPPoE to ISP3" interface-list=WAN name=isp3_client \
on-down="/ip route remove [find gateway=\"4.2.2.3\"]\r\
\n/ip route remove [find where dst-address ~\"4.2.2.3\"]\r\
\n/ip firewall nat remove [find comment=\"NAT via ISP3\"]\r\
\n/ip route rule remove [find comment=\"From ISP3 IP to Inet\"]" \
on-up="/ip route remove [find gateway=\"4.2.2.3\"]; \r\
\n/ip route remove [find where dst-address ~\"4.2.2.3\"]\r\
\n/ip route add check-gateway=ping comment=\"For recursion via ISP3\" distance=1 \
dst-address=4.2.2.3/32 gateway=\$\"remote-address\" scope=11\r\
\n/ip route add check-gateway=ping comment=\"Unmarked via ISP3\" \
distance=3 gateway=4.2.2.3 target-scope=11\r\
\n/ip route add comment=\"Marked via ISP3 Main\" distance=1 gateway=4.2.2.3 \
routing-mark=to_isp3 target-scope=11\r\
\n/ip route add comment=\"Marked via ISP1 Backup2\" distance=3 gateway=4.2.2.3 \
routing-mark=to_isp1 target-scope=11\r\
\n/ip route add comment=\"Marked via ISP2 Backup2\" distance=3 gateway=4.2.2.3 \
routing-mark=to_isp2 target-scope=11\r\
\n/ip firewall mangle set [find comment=\"Connmark in from ISP3\"] \
in-interface=\$\"interface\"\r\
\n:if [:tobool ([/ip firewall nat find comment=\"NAT via ISP3\"])] do={\r\
\n /ip firewall nat set [find comment=\"NAT via ISP3\"] action=src-nat chain=srcnat \
ipsec-policy=out,none out-interface=\$\"interface\" to-addresses=\$\"local-address\"} \
else={\r\
\n /ip firewall nat add action=src-nat chain=srcnat ipsec-policy=out,none \
out-interface=\$\"interface\" to-addresses=\$\"local-address\" \
comment=\"NAT via ISP3\"}\r\
\n:if [:tobool ([/ip route rule find comment=\"From ISP3 IP to Inet\"])] \
do={\r\
\n /ip route rule set [find comment=\"From ISP3 IP to Inet\"] \
src-address=\$\"local-address\" table=to_isp3} else={\r\
\n /ip route rule add comment=\"From ISP3 IP to Inet\" \
src-address=\$\"local-address\" table=to_isp3 }"
For ROS7:
/ppp profile
add interface-list=WAN name=isp3_client on-down=\
"/ip route remove [find gateway=\"4.2.2.3\"]\r\
\n/ip route remove [find where dst-address ~\"4.2.2.3\"]\r\
\n/ip firewall nat remove [find comment=\"NAT via ISP3\"]\r\
\n/routing/rule/ remove [find comment=\"From ISP3 IP to Inet\"]" \
on-up="/ip route remove [find gateway=\"4.2.2.3\"]; \r\
\n/ip route remove [find where dst-address ~\"4.2.2.3\"]\r\
\n/ip route add check-gateway=ping comment=\"For recursion via ISP3\" distance=1 \
dst-address=4.2.2.3/32 gateway=\$\"remote-address\" scope=11\r\
\n/ip route add check-gateway=ping comment=\"Unmarked via ISP3\" \
distance=3 gateway=4.2.2.3 target-scope=11\r\
\n/ip route add comment=\"Marked via ISP3 Main\" distance=1 gateway=4.2.2.3 \
routing-table=to_isp3 target-scope=11\r\
\nip route add comment=\"Marked via ISP1 Backup2\" distance=3 gateway=4.2.2.3 \
routing-table=to_isp1 target-scope=11\r\
\nip route add comment=\"Marked via ISP2 Backup2\" distance=3 gateway=4.2.2.3 \
routing-table=to_isp2 target-scope=11\r\
\nip firewall mangle set [find comment=\"Connmark in from ISP3\"] \
in-interface=\$\"interface\"\r\
\n:if [:tobool ([/ip firewall/nat/ find comment=\"NAT via ISP3\"])] do={\r\
\n /ip firewall nat set [find comment=\"NAT via ISP3\"] action=src-nat chain=srcnat \
ipsec-policy=out,none out-interface=\$\"interface\" to-addresses=\$\"local-address\"} \
else={\r\
\n /ip firewall nat add action=src-nat chain=srcnat ipsec-policy=out,none \
out-interface=\$\"interface\" to-addresses=\$\"local-address\" \
comment=\"NAT via ISP3\"}\r\
\n:if [:tobool ([/routing/rule/ find comment=\"From ISP3 IP to Inet\"])] do={\r\
\n /routing/rule/ set [find comment=\"From ISP3 IP to Inet\"] action=lookup \
src-address=\$\"local-address\" table=to_isp3} else={\r\
\n /routing/rule/ add action=lookup comment=\"From ISP3 IP to Inet\" \
src-address=\$\"local-address\" table=to_isp3 }"
Here's the content for the PPP profile script body for both ROS6 and ROS7, detailing actions for when the PPP interface comes up and goes down.
ROS6(On Up):
/ip route remove [find gateway="4.2.2.3"];
/ip route remove [find where dst-address ~"4.2.2.3"]
/ip route add check-gateway=ping comment="For recursion via ISP3" distance=1 \
dst-address=4.2.2.3/32 gateway=$"remote-address" scope=11
/ip route add check-gateway=ping comment="Unmarked via ISP3" distance=3 \
gateway=4.2.2.3 target-scope=11
/ip route add comment="Marked via ISP3 Main" distance=1 gateway=4.2.2.3 \
routing-mark=to_isp3 target-scope=11
/ip route add comment="Marked via ISP1 Backup2" distance=3 gateway=4.2.2.3 \
routing-mark=to_isp1 target-scope=11
/ip route add comment="Marked via ISP2 Backup2" distance=3 gateway=4.2.2.3 \
routing-mark=to_isp2 target-scope=11
/ip firewall mangle set [find comment="Connmark in from ISP3"] in-interface=$"interface"
:if [:tobool ([/ip firewall nat find comment="NAT via ISP3"])] do={
/ip firewall nat set [find comment="NAT via ISP3"] action=src-nat chain=srcnat \
ipsec-policy=out,none out-interface=$"interface" to-addresses=$"local-address"} else={
/ip firewall nat add action=src-nat chain=srcnat ipsec-policy=out,none \
out-interface=$"interface" to-addresses=$"local-address" comment="NAT via ISP3"}
:if [:tobool ([/ip route rule find comment="From ISP3 IP to Inet"])] do={
/ip route rule set [find comment="From ISP3 IP to Inet"] \
src-address=$"local-address" table=to_isp3} else={
/ip route rule add comment="From ISP3 IP to Inet" src-address=$"local-address" \
table=to_isp3 }
ROS6(On Down):
/ip route remove [find gateway="4.2.2.3"]
/ip route remove [find where dst-address ~"4.2.2.3"]
/ip firewall nat remove [find comment="NAT via ISP3"]
/ip route rule remove [find comment="From ISP3 IP to Inet"]
ROS7(On Up):
/ip route remove [find gateway="4.2.2.3"];
/ip route remove [find where dst-address ~"4.2.2.3"]
/ip route add check-gateway=ping comment="For recursion via ISP3" distance=1 \
dst-address=4.2.2.3/32 gateway=$"remote-address" scope=11
/ip route add check-gateway=ping comment="Unmarked via ISP3" distance=3 \
gateway=4.2.2.3 target-scope=11
/ip route add comment="Marked via ISP3 Main" distance=1 gateway=4.2.2.3 \
routing-table=to_isp3 target-scope=11
ip route add comment="Marked via ISP1 Backup2" distance=3 gateway=4.2.2.3 \
routing-table=to_isp1 target-scope=11
ip route add comment="Marked via ISP2 Backup2" distance=3 gateway=4.2.2.3 \
routing-table=to_isp2 target-scope=11
ip firewall mangle set [find comment="Connmark in from ISP3"] in-interface=$"interface"
:if [:tobool ([/ip firewall/nat/ find comment="NAT via ISP3"])] do={
/ip firewall nat set [find comment="NAT via ISP3"] action=src-nat chain=srcnat \
ipsec-policy=out,none out-interface=$"interface" to-addresses=$"local-address"} else={
/ip firewall nat add action=src-nat chain=srcnat ipsec-policy=out,none \
out-interface=$"interface" to-addresses=$"local-address" comment="NAT via ISP3"}
:if [:tobool ([/routing/rule/ find comment="From ISP3 IP to Inet"])] do={
/routing/rule/ set [find comment="From ISP3 IP to Inet"] action=lookup \
src-address=$"local-address" table=to_isp3} else={
/routing/rule/ add action=lookup comment="From ISP3 IP to Inet" \
src-address=$"local-address" table=to_isp3 }
ROS7(On Down):
/ip route remove [find gateway="4.2.2.3"]
/ip route remove [find where dst-address ~"4.2.2.3"]
/ip firewall nat remove [find comment="NAT via ISP3"]
/routing/rule/ remove [find comment="From ISP3 IP to Inet"]
Note: The line
/ip firewall mangle set [find comment=«Connmark in from ISP3»] in-interface=$«interface»;ensures proper handling of interface renaming by working with its ID rather than the display name. This approach maintains the correct association of connection marks with the dynamic PPP interface, essential for consistent policy application across interface reconnections.
/interface pppoe-client add allow=mschap2 comment="to ISP3" disabled=no \
interface=ether3 name=pppoe-isp3 password=isp3_pass profile=isp3_client \
add-default-route=no user=isp3_client
Note: Some providers might "forget" to send the "remote-address" parameter. In such cases, the configuration script may not work correctly, and you might see an error like this in the logs:
pppoe,ppp,info pppoe-isp3: could not determine remote address, using xxx.xxx.xxx.xxx
To resolve this issue, disable check-gateway for the route with the comment "For recursion via ISP3" and set any fictitious IP address in the PPP profile:
/ip route unset [find comment="For recursion via ISP3"] check-gateway
/ppp profile set isp3_client remote-address=169.254.69.96
The line
/ip firewall mangle set [find comment=«Connmark in from ISP3»] in-interface=$«interface»;
ensures correct processing of interface renaming by working with its system identifier rather than its display name.
Final Touch - Setting Up the Clock:
ROS6:
/system ntp client set enabled=yes \
server-dns-names=0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org
ROS7:
/system ntp client set enabled=yes \
servers=0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org
For Those Who Read to the End:
The method of multi-WAN implementation presented here is the author's personal preference and is not the only possible way. RouterOS's toolkit is extensive and flexible, which, while potentially daunting for beginners, also accounts for its popularity. Continue to explore, experiment, and discover new tools and solutions. For instance, in this multi-WAN implementation, you could replace the Check-gateway with recursive routes using Netwatch. Additionally, to enhance reliability, you might employ a cascade of recursive routes, reducing the number of false switches during temporary unavailability of the host selected as the "base". Even with ultra-reliable anycast addresses, such incidents can occasionally occur.
Notes
Check-gateway is a mechanism that allows for the deactivation of a route after two consecutive failed checks of gateway availability. The check is performed every 10 seconds, plus the response timeout, resulting in an actual switch-over timing ranging from 20 to 30 seconds. If this switch-over timing is insufficient, one option is to use the Netwatch tool, which allows for manual setting of the check timer and the use of custom scripts for a more detailed analysis of the state.
The check-gateway mechanism does not trigger with periodic packet losses in the channel.
Important! The activation/deactivation of a route by the check-gateway mechanism leads to the activation/deactivation of all other routes going through the same gateway. Therefore, it is not necessary to enable check-gateway for all related routes.
There can be failures in the DHCP mechanism, which appear as a client stuck in the renew state. In such cases, the second part of the script may not execute, but this will not hinder traffic flow, as the corresponding recursive route monitors the state.
(Equal Cost Multi-Path) — ROS allows for setting a route with multiple gateways and the same distance. In this case, connections will be distributed across the channels using a round-robin algorithm, proportional to the number of specified gateways.