Skip to content

Citrix ADC

Overview

This guide details the implementation of a high-performance, security-hardened transaction correlation framework for Citrix ADC (NetScaler). It is designed specifically for enterprise environments where data privacy, auditability, and absolute correlation at high TPS are critical.

Principles of Design

  • Transaction-Scoped Variable: Ensures that the correlation ID stays "locked" to a specific request-response pair, even if the response is delayed by up to 60 seconds.
  • Privacy First: All traffic logs are redirected to a remote syslog server. Local binary logging is disabled to prevent privileged users from accessing sensitive traffic data on the appliance.
  • Precision Tracing: Uses the system-provided HTTP.REQ.TXID for globally unique, UUID-style identifiers.

Before You Begin

Prerequisites & Resource Requirements

  • Citrix ADC Version: 13.0 build 64.x or later is required for full HTTP.REQ.TXID support.
  • Memory Allocation: Each transaction scoped variable consumes approximately 1KB of memory per concurrent transaction. For 10,000 concurrent transactions, ensure at least 10MB of free memory in the NSPE (NetScaler Packet Engine).
  • Features: Requires Rewrite and Audit Logging features to be enabled (enable ns feature REWRITE, AUDIT).
  • Network: The ADC must have low-latency UDP/TCP access to the Syslog Collector. For high-volume environments, a dedicated 1Gbps/10Gbps management interface is recommended.

Performance Considerations

Deploying detailed logging and rewrite policies has a measurable impact on the appliance performance:

Component Performance Impact Recommendation
Transaction Variables Low CPU, Moderate Memory Use transaction scope (not global) to avoid lock contention.
Rewrite Policies Moderate CPU Evaluate TRUE rules quickly. Avoid complex regex in log strings.
Audit Logging Moderate CPU/Network Capturing bodies increases packet processing time. Default is 512 bytes for production safety.
Syslog Overload Network I/O Use UDP for maximum speed. At 1,000+ TPS, consider log sampling.

Load Consideration

In high-load scenarios (1,000+ TPS), it is recommended to monitor CPU usage on the Packet Engines (stat ns) after enabling the policies. If CPU spikes above 80%, reduce the BODY() capture size or use sampled logging.

Deployment

Step 1: Remote Syslog Channel & Isolation

To ensure a 100% clean traffic-only stream, we use "Level-Gating". We set our custom logs to the ERROR level and restrict the syslog action to ignore the standard INFORMATIONAL system noise.

Bash
# 1a. Add the remote log collector (With ERROR-level Gate)
add audit syslogAction APIFort_Syslog_Server <REMOTE_COLLECTOR_IP> -serverPort 10514 -logLevel ERROR -logFacility LOCAL1 -managementlog NONE -mgmtlogLevel NONE -acl ENABLED -userDefinedAuditlog YES

# 1b. Create and bind the syslog policy
add audit syslogPolicy APIFort_Syslog_Pol true APIFort_Syslog_Server
bind audit syslogGlobal -policyName APIFort_Syslog_Pol -priority 100

Filtering Noise

This configuration works as a high-pass filter. Since Citrix management logs (AAA, CLI) are typically INFORMATIONAL (6), they are blocked by our ERROR (3) gate, ensuring only your correlated traffic data reaches the collector.

Step 1.1: Verify Connectivity to Collector

Before applying the full configuration, verify that the ADC can reach the collector on port 10514. Since standard utilities like nc (netcat) are often missing from the Citrix shell, use the "Service Workaround" to check reachability:

Bash
# Create a temporary TCP service to test reachability
add service test_syslog_reachability <REMOTE_COLLECTOR_IP> TCP 10514

# Check the state (Wait ~5 seconds for the probe)
show service test_syslog_reachability
  • Expected Result: State: UP. This confirms the ADC can establish a TCP handshake with the collector.
  • Failed Result: State: DOWN (Timeout). Check Azure NSG Outbound rules or the collector's firewall.

Cleanup:

Bash
rm service test_syslog_reachability

Step 2: Correlation Variable & ID Generation

Define the transaction-scoped variable and the assignment policy that populates it with the unique TXID.

Bash
# Define the variable with 'transaction' scope
add ns variable apifort_corrid_var -type "text(256)" -scope transaction

# Create the assignment (extracting TXID)
add ns assignment apifort_corrid_assign -variable "$apifort_corrid_var" -set "HTTP.REQ.TXID"

Step 3: Audit Message Actions (Log Formats)

Create the message actions that format the log output sent to syslog.

Bash
# Request logging (Client_IP + Headers + 512 bytes of Body)
add audit messageaction apifort_log_req ERROR "\"REQ | CorrID=\" + $apifort_corrid_var + \" | Client_IP=\" + CLIENT.IP.SRC + \" | Headers=\" + HTTP.REQ.FULL_HEADER.BEFORE_STR(\"\r\n\r\n\") + \" | Body=\" + HTTP.REQ.BODY(512)" -logtoNewnslog NO

# Response logging (Headers + 512 bytes of Body)
add audit messageaction apifort_log_res ERROR "\"RES | CorrID=\" + $apifort_corrid_var + \" | Headers=\" + HTTP.RES.FULL_HEADER.BEFORE_STR(\"\r\n\r\n\") + \" | Body=\" + HTTP.RES.BODY(512)" -logtoNewnslog NO

Step 4: Rewrite Policies & Bindings

These policies trigger the ID assignment and the logging message actions.

Bash
# Policy for Correlation ID Assignment
add rewrite policy apifort_pol_assign_corrid TRUE apifort_corrid_assign

# Policy for Request Logging (Triggers apifort_log_req)
add rewrite policy apifort_pol_log_req TRUE NOREWRITE -logAction apifort_log_req

# Policy for Response Logging (Triggers apifort_log_res)
add rewrite policy apifort_pol_log_res TRUE NOREWRITE -logAction apifort_log_res

# Injection into backend headers (Optional but recommended for end-to-end tracing)
add rewrite action apifort_act_inject_corrid insert_http_header X-Correlation-ID "$apifort_corrid_var"
add rewrite policy apifort_pol_inject_corrid TRUE apifort_act_inject_corrid

Step 5: Binding to the Virtual Server

Bind the policies to your target Vserver (e.g., an HTTP/SSL load balancer).

Bash
# Bind Request Phase (in correct execution order)
bind lb vserver <YOUR_VSERVER_NAME> -policyName apifort_pol_assign_corrid -priority 10 -gotoPriorityExpression NEXT -type REQUEST
bind lb vserver <YOUR_VSERVER_NAME> -policyName apifort_pol_inject_corrid -priority 20 -gotoPriorityExpression NEXT -type REQUEST
bind lb vserver <YOUR_VSERVER_NAME> -policyName apifort_pol_log_req -priority 100 -gotoPriorityExpression NEXT -type REQUEST

# Bind Response Phase
bind lb vserver <YOUR_VSERVER_NAME> -policyName apifort_pol_log_res -priority 101 -gotoPriorityExpression NEXT -type RESPONSE

Security Hardening

Security Requirements for Enterprise

For Finance/Bank/Gov deployments, ensure the following settings are strictly applied:

  1. logtoNewnslog NO: Prevents sensitive traffic data from being saved in binary format on the ADC disk.
  2. LOCAL1 Facility: Ensures that traffic audit logs do not mix with system administrative logs in /var/log/ns.log.
  3. ACL ENABLED: Only allows logging traffic from the secure management subnet to the collector.

High-Throughput Optimization (1,000+ TPS)

For extremely high-volume environments, apply these additional optimizations:

A. Body Sampling

Instead of logging every request, log only on errors (4xx/5xx) or use a sampling rate:

Bash
# Log ONLY on HTTP errors
set rewrite policy apifort_pol_log_req -rule "HTTP.RES.STATUS.GE(400)"

B. Payload Minimization

If CPU on the Packet Engine spikes, disable body capture and log only headers:

Bash
set audit messageaction apifort_log_req -stringBuilderExpr "\"REQ | CorrID=\" + $apifort_corrid_var + \" | \" + HTTP.REQ.FULL_HEADER"

C. Resource Monitoring

Monitor these counters to ensure the logging pipeline is healthy:

  • stat ns: Check PE CPU usage.
  • stat audit: Check for Log messages dropped or Transmit errors.
  • stat rewrite: Check hits/hits rate.

Verification

Run the following command on the ADC CLI to verify hits:

Bash
show rewrite policy | grep apifort

Verification Check

Check your remote syslog collector for log pairs sharing the identical CorrID at level <139>.

Performance Benchmarks & Estimations

The following benchmarks were conducted on a Citrix ADC VPX instance to measure the "Logging Tax" (CPU overhead) of this framework.

Benchmark Results (Measured at ~100 TPS)

Scenario Packet CPU (Avg) CPU Overhead (Tax) Description
Baseline 0.20% - No policies bound (Raw traffic).
Header-Only 0.25% +0.05% Logging headers only (No body).
Standard (512B) 0.30% +0.10% Headers + 512 bytes of body (Production Default).
Heavy (2048B) 0.35% +0.15% Headers + 2048 bytes of body.

Scalability Estimations (Linear Projection)

Based on the measured overhead, the following table estimates the impact on the Packet Engine (PE) CPU at higher transaction volumes:

Throughput (TPS) Estimated CPU Overhead (Standard 512B)
100 TPS ~0.1%
1,000 TPS ~1.0%
5,000 TPS ~5.0%
10,000 TPS ~10.0%

Note on Estimations

These estimations assume a standard HTTP request size. Real-world impact may vary based on SSL/TLS handshake frequency and average packet size. The use of transaction scope variables and TXID ensures that performance stays linear even at very high concurrency.