In a previous analysis, detailed in The Curious Case of Deny Logs in Palo Alto SSL Decryption: A Troubleshooting Journey (published March 12, 2025), we explored an unusual issue with a Palo Alto Networks (PAN) firewall: traffic logs consistently reported “deny” entries for AWS Route 53 health checks despite explicit “allow” policies and uninterrupted service operation. As of March 19, 2025, Palo Alto Networks support has identified PAN-273949 as a potential root cause, suggesting a firmware update to PAN-OS 11.1.6-h3 may address this SSL decryption logging discrepancy. This section provides an interim update on our planned approach, the proposed resolution, and anticipated outcomes.
Background: SSL Decryption Configuration and Observed Behavior
Our environment leverages a PAN firewall for SSL decryption, utilizing a wildcard certificate (*.mydomain.com) to secure traffic routed through a reverse proxy. AWS Route 53 health checks, configured to poll web services every 30 seconds and validate specific response strings, ensure operational integrity. While these health checks executed successfully—confirmed by uninterrupted production services—the firewall’s traffic logs unexpectedly logged these transactions as “deny” actions. Security policies were explicitly set to “allow,” and no operational impact was observed, highlighting a disconnect between logged events and actual traffic handling.
Initial Findings and Support Insight
Initial troubleshooting efforts, as documented previously, ruled out policy misconfiguration. Security policies permitted traffic from Route 53 IP ranges to the reverse proxy, and temporarily disabling all security profiles did not eliminate the “deny” logs. Further investigation into system logs revealed intermittent SNMP daemon (snmpd) errors, prompting a case with Palo Alto Networks support. Their analysis pointed to PAN-273949, a known issue affecting SNMP logging processes, which they linked to SSL decryption anomalies. Specifically, support indicated that this bug could cause the firewall to misreport allowed, decrypted traffic as “denied” in logs, despite successful processing—a scenario consistent with our observations.
Proposed Resolution: Firmware Update to PAN-OS 11.1.6-h3
Palo Alto Networks has recommended upgrading to PAN-OS 11.1.6-h3, a hotfix release that includes the resolution for PAN-273949. This update addresses a defect in the SNMP daemon’s interaction with SSL decryption, ensuring that log entries accurately reflect policy actions. The proposed fix aims to correct the logging misalignment, where decrypted health check traffic—successfully allowed through the reverse proxy—was erroneously tagged as “deny” due to an internal processing error. We have scheduled this firmware update for a future maintenance window to validate its effectiveness, balancing the need for resolution against operational stability.
Planned Validation and Expected Outcomes
The firmware upgrade to PAN-OS 11.1.6-h3 is slated for later this month, pending a suitable maintenance period to minimize disruption to production services. Given the current lack of operational impact—the “deny” logs are a visibility issue rather than a functional failure—the update is not yet implemented, and its success remains unconfirmed. Post-upgrade, we anticipate the following potential outcomes:
- Resolution Confirmed: Traffic logs align with “allow” policies, eliminating “deny” entries for health checks, and snmpd errors cease, validating PAN-273949 as the root cause.
- Partial Resolution: The update mitigates some logging discrepancies, but residual issues (e.g., certificate handling quirks) require further investigation.
- No Change: The “deny” logs persist, indicating a misdiagnosis or an unrelated SSL decryption configuration issue, necessitating additional diagnostics.
Post-update validation will include monitoring traffic logs, reviewing snmpd logs for residual errors, and confirming health check functionality.
Implications for IT Operations
This issue underscores the importance of accurate logging in network security appliances. Discrepancies between logged events and actual traffic behavior, as observed with SSL decryption in this case, can erode trust in monitoring tools and complicate troubleshooting efforts. The PAN-273949 fix highlights Palo Alto Networks’ commitment to refining SSL decryption functionality—a critical component for organizations leveraging wildcard certificates and health check monitoring. For IT professionals managing similar configurations, this serves as a reminder to correlate log data with operational outcomes and to prioritize timely firmware updates when addressing known defects.
We will provide a follow-up analysis once the PAN-OS 11.1.6-h3 upgrade is complete, detailing the results and any additional insights gained. For those encountering analogous SSL decryption logging challenges, we invite you to share your experiences or recommendations in the comments section below.