In modern network security, URL filtering is essential for controlling and monitoring web access. When an explicit proxy is paired with a Palo Alto Networks firewall and Deep Packet Inspection (DPI)—typically involving SSL/TLS decryption—is enabled, the firewall can perform highly granular URL filtering. This post dives into how URL filtering works in this setup, the mechanics behind it, and how it differs from non-decrypted URL filtering. We’ll also provide a comparison table to highlight key differences, tailored for networking professionals looking to optimize their firewall policies.
What is Deep Packet Inspection (DPI) in This Context?
For this discussion, we assume Deep Packet Inspection (DPI) refers to the firewall’s ability to decrypt and inspect SSL/TLS-encrypted traffic, including the full HTTP request and response payloads. With DPI enabled, a Palo Alto firewall can inspect the HTTP Host header, URL path, and even content, offering more precise control than non-decrypted URL filtering, which relies solely on metadata like the Server Name Indication (SNI).
When an explicit proxy is involved, the firewall’s DPI capabilities interact with the proxy’s role as an intermediary, enabling detailed inspection of the traffic. Let’s explore how this works and the specific mechanisms involved.
How URL Filtering Works with Explicit Proxy and DPI (Decryption)
In an explicit proxy setup, client devices are configured to send all web requests to the proxy server, which forwards them to the destination server. When DPI with SSL/TLS decryption is enabled on the Palo Alto firewall, the firewall can inspect the full HTTP request, including the URL path, even for HTTPS traffic. Here’s the process:1. Client Sends HTTP CONNECT Request
- When a user accesses an HTTPS website (e.g., https://www.example.com/path/to/page), their browser sends an HTTP CONNECT request to the explicit proxy. For example:
CONNECT www.example.com:443 HTTP/1.1 Host: www.example.com - This request is sent in plaintext and specifies the destination hostname and port. The firewall, if positioned between the client and the proxy, can see this request and extract the hostname (www.example.com) for preliminary URL filtering.
2. Proxy Establishes HTTPS Connection
- The proxy responds with an HTTP/1.1 200 Connection Established message, establishing a tunnel for the HTTPS traffic.
- The client sends a Client Hello packet as part of the SSL/TLS handshake, containing the Server Name Indication (SNI) (e.g., www.example.com).
- The proxy forwards this Client Hello to the destination server (or initiates its own Client Hello on behalf of the client).
3. Firewall Performs SSL/TLS Decryption
- The Palo Alto firewall, positioned either between the proxy and the internet or between the client and the proxy, intercepts the HTTPS traffic and performs SSL Forward Proxy decryption (if configured).
- Decryption Process:
- The firewall acts as a man-in-the-middle (MITM), terminating the SSL/TLS session from the client (or proxy) and establishing a new session with the destination server.
- The firewall uses its SSL Forward Proxy configuration, which includes a trusted CA certificate installed on client devices or the proxy. This allows the firewall to generate a dynamic certificate for the destination server (e.g., www.example.com) and present it to the client or proxy, enabling decryption without triggering certificate warnings.
- Once decrypted, the firewall can inspect the full HTTP request, including:
- HTTP Host Header: Specifies the hostname (e.g., www.example.com).
- URL Path: Specifies the specific resource (e.g., /path/to/page).
- HTTP Method and Headers: Additional context about the request.
- Content: The payload of the request or response, if needed for advanced inspection.
4. URL Filtering with Decrypted Traffic
- With the decrypted HTTP request, the firewall extracts the full URL (e.g., https://www.example.com/path/to/page) and queries the PAN-DB cloud-based URL filtering database (or a local cache) to categorize the URL.
- The PAN-DB database maps URLs to categories (e.g., “social-media,” “news,” “malware”), and the firewall applies the configured URL filtering policy (e.g., allow, block, alert, or override).
- Because the firewall can see the full URL, including the path, it can enforce policies at a more granular level than non-decrypted filtering, which is limited to the hostname.
5. Policy Enforcement and Logging
- The firewall applies the URL filtering policy based on the category and logs the details, including the full URL, category, and action taken, in the URL Filtering logs.
- If the policy blocks the request, the firewall can send a custom block page to the client via the proxy.
- The decrypted traffic is re-encrypted by the firewall before being sent to the destination server (or back to the proxy), ensuring end-to-end security.
6. Proxy’s Role in the Process
- The explicit proxy acts as an intermediary, forwarding the client’s HTTPS traffic to the destination server. If the firewall is between the client and the proxy, it can decrypt the traffic before it reaches the proxy. If the firewall is between the proxy and the internet, it decrypts the proxy’s outbound traffic.
- The proxy may also perform its own URL filtering or caching, but the firewall’s DPI capabilities provide deeper inspection and override any proxy-based filtering if configured to do so.
Key Packet Header InformationWhen DPI with decryption is enabled, the firewall inspects the following:
- HTTP CONNECT Request (before the TLS handshake):
- Sent in plaintext by the client to the proxy.
- Contains the Host header (e.g., www.example.com).
Example:
CONNECT www.example.com:443 HTTP/1.1
Host: www.example.com- Client Hello Packet (if decryption is not yet applied):
- Contains the Server Name Indication (SNI) (e.g., www.example.com) in plaintext.
- Used as a fallback if decryption is not enabled or fails.
- Decrypted HTTP Request:
- After decryption, the firewall inspects the full HTTP request, including:
- Host Header: Host: www.example.com
- URL Path: /path/to/page
- Full URL: https://www.example.com/path/to/page
- After decryption, the firewall inspects the full HTTP request, including:
Example:
GET /path/to/page HTTP/1.1
Host: www.example.com
User-Agent: Mozilla/5.0 ...URL Filtering with Decryption vs. Without Decryption
The following table compares URL filtering with an explicit proxy when DPI (decryption) is enabled versus when it is not enabled:
| Aspect | Decryption | No Decryption |
|---|---|---|
| Inspection Depth | Full HTTP request (Host header, URL path, method, content) | Limited to SNI (hostname) or CN in the SSL certificate |
| Granularity | Can filter based on full URL (e.g., https://www.example.com/path/to/page) | Limited to hostname (e.g., www.example.com) |
| HTTP CONNECT Visibility | Can inspect CONNECT request and decrypted HTTP request | Can inspect CONNECT request and SNI in Client Hello |
| SNI Dependency | Not dependent on SNI; uses decrypted HTTP Host header and URL path | Heavily dependent on SNI; falls back to CN if SNI is absent |
| Encrypted SNI (ESNI/ECH) Impact | Not affected, as decryption reveals the full URL | Affected; ESNI/ECH encrypts SNI, potentially preventing hostname identification |
| Performance Impact | Higher CPU/memory usage due to decryption and re-encryption | Lower resource usage, as only metadata is inspected |
| Privacy Considerations | Requires trusted CA certificate on clients; may raise privacy concerns | No decryption, so fewer privacy concerns |
| Configuration Complexity | Requires SSL Forward Proxy setup, CA certificate deployment, and client trust | Simpler; only requires URL filtering profile and PAN-DB license |
| Use Case | Ideal for environments needing granular path-based filtering or content inspection | Suitable for basic hostname-based filtering with minimal overhead |
| Log Detail | Logs full URL, category, and action; can include content-based details | Logs hostname, category, and action; no path or content details |
| Bypass Risk | Lower risk, as full URL is inspected | Higher risk if SNI is absent or ESNI/ECH is used |
Best Practices:
- Configure SSL Decryption Properly:
- Deploy a trusted CA certificate on all client devices or the proxy to avoid certificate warnings.
- Use Selective Decryption to decrypt only specific URL categories (e.g., “social-media,” “high-risk”) to balance performance and security.
- Verify Proxy Integration:
- Ensure the firewall is positioned to inspect traffic either between the client and proxy or between the proxy and the internet.
- If the proxy performs its own decryption, coordinate with the firewall to avoid duplicate decryption, which can impact performance.
- Optimize PAN-DB Lookups:
- Enable caching in the URL filtering profile to reduce latency for frequently accessed URLs.
- Use the Test A Site tool (urlfiltering.paloaltonetworks.com) to verify URL categorizations.
- Monitor and Log:
- Enable detailed logging for decrypted traffic to gain visibility into full URLs and user activity.
- Use the Log container page only option to reduce log volume for non-critical categories.
- Prepare for ESNI/ECH:
- As Encrypted Client Hello (ECH) adoption grows, non-decrypted URL filtering may become less effective. Plan to enable decryption for critical use cases to maintain visibility.
- Test and Validate:
- Use packet captures (e.g., Wireshark) to verify that the firewall is correctly inspecting HTTP CONNECT requests and decrypted HTTP headers.
- Test URL filtering policies with sample URLs to ensure expected behavior.
Practical Example
Suppose a user behind an explicit proxy tries to access https://www.facebook.com/messages:
- The client sends an HTTP CONNECT request to the proxy: CONNECT www.facebook.com:443 HTTP/1.1.
- The proxy establishes an HTTPS tunnel, and the client sends a Client Hello with server_name: www.facebook.com.
- The firewall, configured for SSL Forward Proxy, decrypts the traffic, revealing the HTTP request:
GET /messages HTTP/1.1 Host: www.facebook.com - The firewall extracts the full URL (https://www.facebook.com/messages) and queries PAN-DB, which categorizes it as “social-media.”
- If the policy blocks “social-media” URLs with /messages, the firewall blocks the request and logs the full URL.
Conclusion
When an explicit proxy is used with a Palo Alto firewall and DPI (SSL/TLS decryption) is enabled, URL filtering becomes highly granular, allowing inspection of the full URL, including the path, via the decrypted HTTP request. This contrasts with non-decrypted URL filtering, which relies on the SNI or CN and is limited to hostname-based filtering. By leveraging DPI, networking professionals can enforce precise policies, enhance visibility, and mitigate risks from encrypted traffic. However, decryption introduces performance and privacy considerations that must be carefully managed.