F5 iRule HSL and Syslog Log Forwarding Documentation¶
1. Overview¶
This documentation explains how to capture HTTP/HTTPS traffic using iRule on F5 Load Balancer and forward it to a log server using HSL (High Speed Logging) and Syslog protocol to send Apifort.
1.1 Purpose¶
- Capture HTTP request and response data
- Collect header information
- Capture request/response bodies (JSON, XML, form-data)
- Safely transmit data by chunking
2. iRule Configuration¶
2.1 Complete Irule¶
Tcl
proc isCapturableContentType {contentType} {
set contentType [string tolower $contentType]
foreach type $::static::capturable_content_types {
if {[string match *$type* $contentType]} {
return 1
}
}
return 0
}
when RULE_INIT {
set static::max_collect_len_req 1024000
set static::max_collect_len_res 1024000
set static::server_ip "10.1.130.92"
set static::server_port "10514"
set static::capturable_content_types {json xml x-www-form-urlencoded}
set static::catalog_id "83bd0a2e-f7eb-47fd-8169-791390be7597"
set static::delimiter "__"
set static::hsl_start "${static::delimiter}HSL_START${static::delimiter}"
set static::request_header_start "${static::delimiter}RQH${static::delimiter}"
set static::request_header_end "${static::delimiter}RQHE${static::delimiter}"
set static::response_header_start "${static::delimiter}RSH${static::delimiter}"
set static::response_header_end "${static::delimiter}RSHE${static::delimiter}"
set static::header_name "${static::delimiter}HN${static::delimiter}"
set static::header_value "${static::delimiter}HV${static::delimiter}"
set static::request_payload "${static::delimiter}RQP${static::delimiter}"
set static::response_payload "${static::delimiter}RSP${static::delimiter}"
set static::hsl_end "${static::delimiter}HSL_END${static::delimiter}"
}
when CLIENT_ACCEPTED {
}
when HTTP_REQUEST {
set sessionId "[IP::client_addr][TCP::client_port][IP::local_addr][TCP::local_port][expr { int(1000000 * rand()) }]"
binary scan [md5 $sessionId] H* correlationId junk
set request_time [clock clicks -milliseconds]
set reqHeaderString "${static::request_header_start}"
set contentTypeHeaderValue ""
if {[HTTP::has_responded]} {
event disable all
return
}
foreach aHeader [HTTP::header names] {
if { [string tolower $aHeader] == "content-type" } {
set contentTypeHeaderValue [HTTP::header value $aHeader]
}
set lwcasekey [string map -nocase {"\"" "\\\""}[string tolower $aHeader]]
set value [string map -nocase {"\"" "\\\""} [HTTP::header value $aHeader]]
set headers "${static::header_name}${lwcasekey}${static::header_value}${value}"
append reqHeaderString $headers
}
append reqHeaderString ${static::request_header_end}
set uri [HTTP::uri]
set method [HTTP::method]
set client_addr [IP::client_addr]
set local_port [TCP::local_port]
set method [HTTP::method]
set host [HTTP::host]
set request_time [clock clicks -milliseconds]
set request_payload ""
set request_truncated 0
set log_message ""
set captureBody [call isCapturableContentType $contentTypeHeaderValue]
if {$captureBody} {
HTTP::collect $static::max_collect_len_req
}
}
when HTTP_REQUEST_DATA {
if {$captureBody && [HTTP::payload length] > 0 } {
set capture_length [HTTP::payload length]
if { $capture_length > $static::max_collect_len_req } {
set request_truncated 1
# Set request_payload to empty if body exceeds max length
set request_payload ""
} else {
set request_payload [b64encode [string range "[HTTP::payload]" 0 $capture_length]]
}
}
}
when HTTP_RESPONSE {
set response_time [clock clicks -milliseconds]
set resHeaderString "${static::response_header_start}"
set contentTypeHeaderValue ""
foreach aHeader [HTTP::header names] {
if { [string tolower $aHeader] == "content-type" } {
set contentTypeHeaderValue [HTTP::header value $aHeader]
}
set lwcasekey [string map -nocase {"\"" "\\\""}[string tolower $aHeader]]
set value [string map -nocase {"\"" "\\\""} [HTTP::header value $aHeader]]
set headers "${static::header_name}${lwcasekey}${static::header_value}${value}"
append resHeaderString $headers
}
append resHeaderString "${static::response_header_end}"
set captureResponseBody [call isCapturableContentType $contentTypeHeaderValue]
if {$captureResponseBody} {
HTTP::collect $static::max_collect_len_res
}
set data_sent 0
set response_truncated 0
if { [HTTP::header exists "Content-Length"] && [HTTP::header value "Content-Length"] == 0 } {
set response_payload ""
set log_message "${static::hsl_start} $static::catalog_id $method $uri $client_addr [HTTP::status] $request_time $response_time $request_truncated $response_truncated $reqHeaderString $resHeaderString ${static::request_payload}$request_payload ${static::response_payload}$response_payload ${static::hsl_end}"
# Inline chunking logic
set chunk_size 800
set total_length [string length $log_message]
set total_pages [expr {int(($total_length + $chunk_size - 1) / $chunk_size)}]
for {set i 0} {$i < $total_pages} {incr i} {
set start [expr {$i * $chunk_size}]
set end [expr {$start + $chunk_size - 1}]
set chunk [string range $log_message $start $end]
set page_num [expr {$i + 1}]
#log local0. "CORRELATIONID__${correlationId}__PAGE_${page_num}__TOTAL_${total_pages}__${chunk}\n"
log $static::server_ip:$static::server_port local0. "CORRELATIONID__${correlationId}__PAGE_${page_num}__TOTAL_${total_pages}__${chunk}\n"
after 1
}
set data_sent 1
}
}
when HTTP_RESPONSE_DATA {
set response_payload ""
if { $captureResponseBody && [HTTP::payload length] > 0 } {
set capture_length [HTTP::payload length]
if { $capture_length > $static::max_collect_len_res } {
set response_truncated 1
set response_payload ""
} else {
set response_payload [b64encode [string range "[HTTP::payload]" 0 $capture_length]]
}
}
if { $data_sent != 1 } {
#log local0. "$response_payload"
set log_message "${static::hsl_start} $static::catalog_id $method $uri $client_addr [HTTP::status] $request_time $response_time $request_truncated $response_truncated $reqHeaderString $resHeaderString ${static::request_payload}$request_payload ${static::response_payload}$response_payload ${static::hsl_end}"
set chunk_size 800
set total_length [string length $log_message]
set total_pages [expr {int(($total_length + $chunk_size - 1) / $chunk_size)}]
for {set i 0} {$i < $total_pages} {incr i} {
set start [expr {$i * $chunk_size}]
set end [expr {$start + $chunk_size - 1}]
set chunk [string range $log_message $start $end]
set page_num [expr {$i + 1}]
#log local0. "CORRELATIONID__${correlationId}__PAGE_${page_num}__TOTAL_${total_pages}__${chunk}\n"
log $static::server_ip:$static::server_port local0. "CORRELATIONID__${correlationId}__PAGE_${page_num}__TOTAL_${total_pages}__${chunk}\n"
after 1
}
}
}
2.2 Basic Parameters¶
Tcl
# Log server connection information
set static::server_ip "10.1.37.55" # Apifort Application server host address
set static::server_port "10514" # Apifort Application server port
# Catalog identifier
set static::catalog_id "d6c89239-95be-48b7-bb7b-8656d881f787" # Catalog where traffic will be sent
# Performance parameters
set static::max_collect_len_req 102400 # Maximum request body size (100KB)
set static::max_collect_len_res 102400 # Maximum response body size (100KB)
set chunk_size 800 # Log message chunk size
2.3 Performance Optimization¶
Chunk Size Optimization¶
Tcl
# Current value
set chunk_size 800
# Recommended value (based on F5 MTU)
set chunk_size 1860
Note: F5's MTU (Maximum Transmission Unit) is typically 1500 bytes. After accounting for Ethernet and IP headers, it can safely be increased to 1860 bytes.
Body Collection Limits¶
The following values can be increased based on system load:
- max_collect_len_req: Request body collection limit
- max_collect_len_res: Response body collection limit
Recommendations:
- Low load: 102400 (100KB)
- Medium load: 512000 (500KB)
- High load: 1024000 (1MB)
3. iRule Components¶
3.1 Content Type Check¶
Tcl
proc isCapturableContentType {contentType} {
set contentType [string tolower $contentType]
foreach type $::static::capturable_content_types {
if {[string match *$type* $contentType]} {
return 1
}
}
return 0
}
Capturable content types:
- application/json
- application/xml or text/xml
- application/x-www-form-urlencoded
3.2 Session and Correlation ID Generation¶
Tcl
set sessionId "[IP::client_addr][TCP::client_port][IP::local_addr][TCP::local_port][expr { int(1000000 * rand()) }]"
binary scan [md5 $sessionId] H* correlationId junk
A unique correlation ID is generated for each HTTP transaction. This ID is used to reassemble chunked log messages.
3.3 Log Message Format¶
Log messages are structured using special delimiters:
Text Only
__HSL_START__ catalog_id method uri client_addr status request_time response_time
request_truncated response_truncated __RQH__headers__RQHE__ __RSH__headers__RSHE__
__RQP__request_payload __RSP__response_payload __HSL_END__
4. Log Transmission Mechanism¶
4.1 Chunking Logic¶
Large log messages are split according to the specified chunk_size value:
Tcl
set chunk_size 800
set total_length [string length $log_message]
set total_pages [expr {int(($total_length + $chunk_size - 1) / $chunk_size)}]
for {set i 0} {$i < $total_pages} {incr i} {
set start [expr {$i * $chunk_size}]
set end [expr {$start + $chunk_size - 1}]
set chunk [string range $log_message $start $end]
set page_num [expr {$i + 1}]
log $static::server_ip:$static::server_port local0. "CORRELATIONID__${correlationId}__PAGE_${page_num}__TOTAL_${total_pages}__${chunk}\n"
after 1
}
4.2 Log Format¶
Each chunk is sent in this format:
Text Only
CORRELATIONID__<correlation_id>__PAGE_<current_page>__TOTAL_<total_pages>__<chunk_data>
5. Installation and Configuration¶
5.1 Creating the iRule¶
- Log in to F5 management interface
- Navigate to Local Traffic > iRules
- Click Create button
- Paste the iRule code and save
5.2 Assigning iRule to Virtual Server¶
- Navigate to Local Traffic > Virtual Servers
- Select the relevant Virtual Server
- Go to Resources tab
- Select your iRule in the iRules section
- Click Update button
5.3 Configure Apifort¶
- Check the SWITCH_SYSLOG and make sure 10514 port is exposed on docker-compose.yml.
YAML
environments:
SWITCH_SYSLOG: ${SWITCH_SYSLOG:-true}
ports:
- "10514:10514/tcp"
- "10514:10514/udp"
networks:
- apifort-network
6. Monitoring and Troubleshooting¶
6.1 Log Verification¶
Bash
# Check logs on F5
tail -f /var/log/ltm
# View sent logs
tcpdump -i any -nn -s0 -w /var/tmp/hsl_capture.pcap host 10.1.37.55 and port 10514
6.2 Performance Monitoring¶
Bash
# Check CPU usage
tmsh show sys cpu
# Memory usage
tmsh show sys memory
# iRule statistics
tmsh show ltm rule <rule_name> stats
7. Security Recommendations¶
- Sensitive Data Masking: Sensitive data like credit cards and passwords should not be logged
- SSL/TLS Usage: Encrypted communication between F5 and log server is preferred
- Rate Limiting: Apply rate limiting to prevent excessive log generation
8. Performance Optimization Recommendations¶
8.1 Chunk Size Optimization¶
- Set chunk_size appropriate to MTU value (recommended: 1860)
- Test according to network infrastructure
8.2 Body Collection Limits¶
- Adjust according to system resources
- Filter unnecessarily large payloads
8.3 Content Type Filtering¶
- Capture only necessary content types
- Exclude binary content (images, videos)
9. Maintenance and Updates¶
9.1 Regular Checks¶
- Log server accessibility
- F5 system resources
- iRule performance metrics
9.2 Version Management¶
- Version control iRule changes
- Test in staging environment
- Prepare rollback plan
10. Sample Log Output¶
Text Only
CORRELATIONID__a3b5c7d9e1f3__PAGE_1__TOTAL_3__HSL_START__ d6c89239-95be-48b7-bb7b-8656d881f787 POST /api/v1/users 192.168.1.100 200 1625234567890 1625234567950 0 0 __RQH____HN__content-type__HV__application/json__HN__authorization__HV__Bearer token123__RQHE__ __RSH____HN__content-type__HV__application/json__RSHE__ __RQP__eyJ1c2VybmFtZSI6InRlc3QifQ== __RSP__eyJzdGF0dXMiOiJzdWNjZXNzIn0= __HSL_END__
11. Best Practices¶
11.1 Resource Management¶
- Monitor F5 CPU and memory usage regularly
- Adjust collection limits based on actual load
- Implement proper error handling
11.2 Log Retention¶
- Define log retention policies
- Implement log rotation on the receiving server
- Archive old logs appropriately
11.3 Testing Strategy¶
- Test iRule in non-production environment first
- Perform load testing with expected traffic patterns
- Validate log parsing on the receiving end