fortigate - webfilter - url filter cheat sheet

fortigate - webfilter - url filter cheat sheet

#=====================================================================
# Log&Report - Security Events - Web Filter, filter a specific URL. v7.2.3

#launch CLI from fortigate GUI
config webfilter profile #Configure Web filter profiles
get #list all profiles
edit profile-name
show # see current setting
set log-all-url enable
set extended-log enable
end
#=====================================================================
#Troubleshoot  view the log of a blocked website in the CLI

# execute log filter category utm-webfilter
# execute log display

1: date=2019-04-22 time=13:46:25 logid="0316013056" type="utm" subtype="webfilter" eventtype="ftgd_blk" level="warning" vd="vdom1" eventtime=1555965984972459609 policyid=1 sessionid=659263 srcip=10.1.200.15 srcport=49234 srcintf="wan2" srcintfrole="wan" dstip=54.183.57.55 dstport=80 dstintf="wan1" dstintfrole="wan" proto=6 service="HTTP" hostname="www.fortinet.com" profile="webfilter" action="blocked" reqtype="direct" url="/" sentbyte=386 rcvdbyte=0 direction="outgoing" msg="URL belongs to a denied category in policy" method="domain" cat=52 catdesc="Information Technology"

#=====================================================================
#Troubleshoot webfilter CLI FortiOS 7.2x

Test #1:  Is the service enabled? Make sure that at least one firewall policy has a Web Filter and SSL/SSH Inspection profile enabled
If the output shows that the service is not enabled, create a firewall policy and enable Web Filtering inspection there. 

# diagnose debug rating 

Test #2: Can the FortiGate get to the Internet DNS by IP?
Some ISPs and networks block ICMP (ping) traffic. This should be taken into account before considering the test to have failed.

# execute ping 8.8.8.8

Test #3: Can the FortiGate resolve FQDNs?
Some ISPs and networks block ICMP (ping) traffic. 
This should be taken into account before considering the test to have failed. 
The important part of this test is that the unit successfully resolves an FQDN to an IP, not that the ping suceeds.

# exec ping google.com

Test #4: Can the FortiGate resolve a specific host name?
Above mentioned FQDNs might not be pingable, it is an expected behavior.Key point here is to see, if these FQDNs are resolved

# exec ping service.fortiguard.net
# exec ping update.fortiguard.net
# exec ping guard.fortinet.net

change the Fortiguard Web Filtering Port in CLI the following way:

change the Fortiguard Web Filtering Port in CLI the following way

# config system fortiguard
    (fortiguard) set port 53
    (fortiguard) end

In case changing the Web Filtering port cannot solve the problem with Web Filtering, try to change the source port range for self-originated traffic

# config system global
(global) set ip-src-port-range 1031-4999
(global) end
# diagnose test application urlfilter 99

# diagnose webfilter stats list root
# diagnose webfilter stats list #view web filter statistics

System > Replacement Messages > HTTP section > URL Block Page #customize the URL web page blocked message
==========================================================================================================
diagnose test application urlfilter -1
diagnose test application urlfilter 0 #stop
diagnose test application urlfilter 3 #only works if WEB filtering cache is enabled
==========================================================================================================
#diagnose test application urlfilter <Test Level>

1.   This Menu
2.   Clear WF cache
3.   Display WF cache contents
4.   Display WF cache TTL list
5.   Display WF cache LRU list
6.   Display WF cache in tree format6.   Display WF cache in tree format6.   Display WF cache in tree format
7.   Toggle switch for dumping unrated packet
8.   Increase timeout for polling
9.   Decrease timeout for polling
10.  Print debug values
11.  Clear Spam Filter cache
12.  Clear AV Query cache
13. Toggle switch for dumping expired license packets
14.  Show running timers (except request timers)
144. Show running timers (including request timers)
15.  Send INIT requests.
16.  Display WF cache contents of prefix type
99.  Restart the urlfilter daemon.
==========================================================================================================
#Troubleshooting rating issues can be done using the following diagnose command 
diagnose debug application urlfilter -1
diagnose debug en

#Be sure to disable debugging once done:
diag deb dis
diag deb app urlfilter 0
==========================================================================================================
#Troubleshooting rating issues can be done using the following diagnose command 
diagnose debug application urlfilter -1
diagnose debug en

#Be sure to disable debugging once done:
diag deb dis
diag deb app urlfilter 0
#=====================================================================
#display all utm-webfilter logs with the destination ip address 40.85.78.63
# execute log filter category 3       
# execute log filter field dstip 40.85.78.63
# execute log display

#display all utm-webfilter logs with destination ip address 40.85.78.63 that are not from September 13, 2019
# execute log filter free-style "(date 2019-09-13 not) and (dstip 40.85.78.63)" 

# execute log filter field url http://example.com/phpmyadmin/
#=====================================================================
#By design, the FortiGate will use secure HTTPS connection for FortiGuard web filter and anti-spam services and the option to change the protocol has been removed from GUI.

To change the fortiguard port, you have to disable "fortiguard-anycast" option under fortiguard settings then only; the FortiOS will allow to change the protocol to UDP.
# config system fortiguard
    set fortiguard-anycast disable                
    set port 8888
    set protocol udp
end
#=====================================================================

Carrier Grade Network Address Translation (CGNAT) Explained:

Carrier Grade Network Address Translation (CGNAT) is a type of Network Address Translation (NAT) used by service providers to manage the limited availability of IPv4 addresses. It is especially relevant in situations where the demand for IP addresses exceeds the available supply, which is a common issue due to the depletion of IPv4 addresses.

Key Features of CGNAT:

Address Translation:

CGNAT allows multiple private IP addresses within a local network to be translated into a smaller pool of public IP addresses when accessing resources on the internet.
Port Block Allocation:

CGNAT often involves the allocation of port ranges to individual users or sessions. This allows for a more granular and efficient use of public IP addresses.
Scalability:

CGNAT is highly scalable and enables service providers to support a large number of users within their network using a limited set of public IP addresses.
Conservation of IPv4 Addresses:

By dynamically allocating a port block to a user or session, CGNAT conserves public IP addresses. This is crucial in a scenario where IPv4 addresses are scarce.
Subscriber Identification:

CGNAT solutions often involve subscriber-based policies. Each subscriber (user or device) is assigned a unique set of port blocks, allowing for effective identification and tracking of traffic.
Use Cases for CGNAT:

Internet Service Providers (ISPs):

ISPs implement CGNAT to efficiently manage the IP address space, especially in regions where IPv4 addresses are exhausted. It enables them to provide internet connectivity to a large number of subscribers.
Mobile Networks:

In mobile networks, where a massive number of devices connect to the internet, CGNAT helps mobile operators extend the usability of their IPv4 address space.
Enterprise Networks:

Large enterprises may deploy CGNAT to handle address shortages and facilitate internet access for numerous internal users.
Temporary Deployments:

CGNAT can be used in temporary deployments or events where a sudden influx of devices requires internet connectivity without the need for a large pool of public IP addresses.
Scenario Explanation:

In the provided FortiGate scenario:

The firewall policy "LAN-to-Internet" specifies that traffic from the internal network (10.0.1.0/24) should be allowed to access any destination on the internet (0.0.0.0/0).
The CGNAT configuration involves the dynamic allocation of port blocks (port-block-allocation) from the specified IP pool (cgnat-pba-pool).
This setup allows FortiGate to effectively use a limited range of public IP addresses, ensuring that multiple users can access the internet simultaneously. Port blocks are dynamically assigned to users, providing a scalable and efficient solution for managing internet-bound traffic in a scenario where public IP addresses are a limited resource.
#=====================================================================
Here's an explanation of Carrier Grade Network Address Translation (CGNAT) and its use cases, along with details of how FortiGate handles traffic in the given configuration:

Carrier Grade Network Address Translation (CGNAT):

A large-scale NAT technique employed by ISPs and other network providers to conserve public IPv4 addresses.
It allows multiple subscribers to share a smaller pool of public IP addresses, extending the use of the limited IPv4 address space.
Subscriber:

A user or device connected to a network that utilizes CGNAT.
Subscribers typically have private IP addresses within their local network.
CGNAT translates these private addresses to shared public IP addresses when communicating with the internet.
Use Cases for CGNAT:

ISPs: Conserve IPv4 addresses for residential and business customers.
Mobile Network Operators (MNOs): Handle the large number of mobile devices on their networks efficiently.
Enterprise Networks: Securely connect a large number of devices to the internet without requiring individual public IPs.
IoT (Internet of Things) Networks: Manage a vast number of connected devices with limited IPv4 addresses.
FortiGate Actions in the Given Configuration:

Firewall Policy Check:
Applies the "LAN-to-Internet" policy to allow traffic from the subscriber's subnet to the internet.
NAT and IP Pool Allocation:
Performs CGNAT, translating the subscriber's private IP to a shared public IP from the "cgnat-pba-pool".
Allocates a block of 128 ports from a public IP to the subscriber (up to 8 blocks per subscriber).
Traffic Forwarding:
Replaces the source IP and port in the subscriber's traffic with the allocated public IP and port.
Forwards the modified traffic to the internet.
Return Traffic Handling:
Reverses the NAT process for responses, mapping the public IP and port back to the subscriber's private IP and port.
Forwards the responses to the subscriber's device.
#=====================================================================
Here's the specific order of HTTP inspection when multiple features are enabled in a web filter profile on FortiGate:

Static URL Filter: This is the first layer of filtering. It checks if a website's URL matches any explicitly defined allow or block rules in the static URL filter list.

FortiGuard Category Filter: If the URL doesn't match a static rule, FortiGate consults FortiGuard's categorization database. It categorizes websites into various categories (e.g., social media, news, adult content) and applies filtering based on the profile's settings for each category.

Advanced Filters: If the URL passes the first two filters, FortiGate applies any enabled advanced filters, such as:

Safe Search: Restricts search results to filter out potentially offensive or inappropriate content.
Anti-Virus: Scans web content for malware.
Data Leak Prevention (DLP): Prevents sensitive data from being transmitted.
Application Control: Controls access to specific web applications.
Why the other options are incorrect:
#=====================================================================