Overview¶

HTTPCap is a high-performance network traffic capture application that collects HTTP/HTTPS traffic from network interfaces and streams it to Apache Kafka for further processing. The application is distributed as a Docker container with a Docker Compose configuration for easy deployment and management.

Features¶

Real-time network traffic capture from network interfaces
HTTP/HTTPS traffic parsing with decompression support
IPIP tunnel decapsulation support
High-performance packet processing with configurable worker pools
Kafka integration for reliable data streaming
Extensive filtering capabilities to exclude unwanted traffic
Resource management with configurable memory and buffer limits
Metrics and monitoring support
Real IP detection from HTTP headers

Installation¶

Prerequisites¶

Docker Engine (version 27.0 or higher)
Docker Compose (version 2.31 or higher)
Root or sudo privileges (required for network packet capture)

Installation Steps¶

Extract the distribution package

Bash

tar -xzf httpcap-distribution.tar.gz
cd httpcap

Load the Docker image
Bash
```
docker load -i httpcap2206.tar.gz
```

Configuration¶

HTTPCap is configured through environment variables in the Docker Compose file. All configuration parameters have sensible defaults but can be customized based on your requirements.

Network Configuration¶

Parameter	Default	Description
`INPUT_RAW`	`eth0`	Network interface to capture traffic from "Format: "eth0:80,eth0:8080,eth1:443" or just "eth0,eth1""
`INPUT_RAW_ENGINE`	`af_packet`	Packet capture engine (Options: pcap,af_packet)
`IPIP`	`false`	Enable IPIP tunnel decapsulation
`REAL_IP_HEADER`	`x-real-ip`	HTTP header to extract real client IP
`INPUT_RAW_BPF_FILTER`	(see below)	BPF filter to exclude specific hosts

Default BPF Filter: The application excludes traffic from internal monitoring and infrastructure hosts by default:

Text Only

not (host 10.95.91.34 or host 10.131.28.43 or host 10.131.30.40 or host 10.131.1.186 or host 10.6.240.99 or host 10.93.221.189 or host 10.94.42.62 or host 10.98.134.210 or host 10.6.88.150 or host 10.112.1.22 or host 10.6.88.151 or host 10.6.88.152)

Performance Tuning¶

Parameter	Default	Description
`PROCESSING_WORKERS`	`3`	Number of packet processing workers
`ASSEMBLER_WORKERS`	`5`	Number of TCP stream assembly workers
`ASSEMBLER_WORKER_PKT_BUF`	`30000`	Packet buffer size per worker
`ASSEMBLER_OUT_BUF`	`30000`	Output buffer size
`ASSEMBLER_FLUSH_INTERVAL`	`500ms`	Stream flush interval
`ASSEMBLER_FLUSH_AGE`	`5s`	Maximum age before stream flush
`ASSEMBLER_MAX_STREAM_BUFFER`	`5mb`	Maximum buffer size per TCP stream (fixed)
`BUFFER_CAP`	`128KiB`	General buffer capacity
`HEAP_LIMIT`	`16GiB`	Maximum heap memory limit

HTTP Processing¶

Parameter	Default	Description
`MAX_BODY_BYTES`	`3MiB`	Maximum HTTP body size to capture
`MAX_HEADER_BYTES`	`64KiB`	Maximum HTTP header size

Kafka Output Configuration¶

Parameter	Default	Description
`OUTPUT_KAFKA_HOST`	`192.155.23.11:9094`	Kafka broker address
`OUTPUT_KAFKA_TOPIC`	`apifort.httpcap_logs`	Kafka topic for output
`OUTPUT_KAFKA_USERNAME`	`app`	Username of Kafka
`OUTPUT_KAFKA_PASSWORD`	``	Password of Kafka
`OUTPUT_KAFKA_USE_SASL`	`true`	Kafka authentication protocol
`OUTPUT_KAFKA_SASL_MECH`	`SASL/PLAIN`	Kafka authentication mechanism
`OUTPUT_KAFKA_COLLECTION_ID`	`d9e796d6-a6c9-4af7-84f1-e5814339941b`	Api Catalog identifier
`OUTPUT_KAFKA_MAX_MESSAGE_BYTES`	`5mb`	Maximum Kafka message size (fixed)
`OUTPUT_KAFKA_FLUSH_FREQUENCY`	`50ms`	Kafka producer flush frequency
`OUTPUT_KAFKA_FLUSH_BYTES`	`2mb`	Flush when this many bytes are pending
`OUTPUT_KAFKA_FLUSH_MESSAGES`	`500`	Flush when this many messages are pending
`OUTPUT_KAFKA_CHANNEL_BUF`	`30000`	Kafka channel buffer size
`OUTPUT_KAFKA_PRODUCERS`	`3`	Number of Kafka producers
`OUTPUT_KAFKA_COMPRESSION`	`snappy`	Kafka compression algorithm

Monitoring and Logging¶

Parameter	Default	Description
`METRICS_OUTPUT`	`console`	Metrics output destination
`VERBOSE`	`1`	Logging verbosity level (0-5)

Content Filtering¶

Parameter	Default	Description
`FILTER_BLOCK_EXTS`	(see below)	File extensions to filter out
`FILTER_BLOCK_CTYPES`	(see below)	Content-Types to filter out
`FILTER_BLOCK_ACCEPT`	(see below)	Accept headers to filter out
`FILTER_BLOCK_MULTIPART`	(see below)	Multipart content patterns to filter

Deployment¶

Basic Deployment¶

Verify the Apifort Incoming Traffic settings:

Check the SWITCH_KAFKA and TRAFFIC_SOURCE parameters from docker-compose.yml in Apifort Directory

YAML

SWITCH_KAFKA: ${SWITCH_KAFKA:-true}
TRAFFIC_SOURCE: ...,httpcap|kafka|kafka:9092|apifort_consumers|apifort.api.httpcap_logs,...

Update docker-compose.yml (optional) to override default values:

YAML

environment:
  # Network configuration
  INPUT_RAW: eth0
  IPIP: "false"

  # Kafka configuration
  OUTPUT_KAFKA_HOST: your-kafka-broker:9092
  OUTPUT_KAFKA_TOPIC: your-topic

  # Performance tuning
  PROCESSING_WORKERS: 5
  HEAP_LIMIT: 16GiB

Start HTTPCap:
Bash
```
docker-compose up -d
```

Verify HTTPCap is running:

Bash

docker-compose ps
docker-compose logs -f httpcap

Advanced Configuration¶

IPIP Tunnel Support¶

For environments using IPIP encapsulation:

YAML

environment:
  IPIP: "true"

Custom BPF Filters¶

To exclude additional hosts or networks:

YAML

environment:
  INPUT_RAW_BPF_FILTER: "not (host 10.0.0.1 or net 192.168.0.0/24)"

Real IP Header Configuration¶

If your infrastructure uses a different header for real client IPs:

YAML

environment:
  REAL_IP_HEADER: "x-forwarded-for"

High-Traffic Environments¶

For high-traffic environments, increase worker counts and buffer sizes:

YAML

environment:
  PROCESSING_WORKERS: 20
  ASSEMBLER_WORKERS: 15
  ASSEMBLER_WORKER_PKT_BUF: 50000
  HEAP_LIMIT: 32GiB

Traffic Filtering¶

HTTPCap includes extensive filtering capabilities to reduce noise and focus on relevant traffic.

File Extension Filtering¶

The application filters out common static assets by file extension:

Web assets: html, htm, css, js, map
Images: jpg, jpeg, png, gif, svg, webp, ico, bmp, tiff
Fonts: ttf, otf, woff, woff2, eot
Audio: mp3, wav, ogg, flac, aac
Video: mp4, m4v, avi, mov, mkv, webm, ogv, flv, ts
Documents: pdf, doc, docx, xls, xlsx, ppt, pptx
Archives: zip, rar, 7z, tar, gz, bz2
Data: json, xml, rss, atom, manifest, txt, rtf, csv, yml, yaml, toml, ini, cfg, config
Other: php, asp, aspx, jsp, cfm, m3u8, exe, msi, dmg, pkg, deb, rpm, swf, log, bak, sql

Content-Type Filtering¶

Responses with the following content-types are filtered:

Text types: text/html, text/css, text/javascript, text/calendar, text/vcard
Application types: application/javascript, application/x-javascript, application/pdf, application/zip, etc.
Image types: image/jpeg, image/png, image/gif, image/svg+xml, etc.
Video types: video/mp4, video/mpeg, video/ogg, video/webm, etc.
Audio types: audio/mpeg, audio/ogg, audio/wav
Font types: font/ttf, font/otf, font/woff, font/woff2
Streaming: text/event-stream, application/x-mpegURL, video/mp2t
Other: application/wasm, application/ld+json, multipart types

Custom Filtering¶

To customize filtering, override the default filter parameters:

YAML

environment:
  FILTER_BLOCK_EXTS: "jpg,png,css,js"
  FILTER_BLOCK_CTYPES: "image/jpeg,image/png"

Output Format¶

HTTPCap outputs captured HTTP traffic in JSON format to Kafka. Each message includes:

Request and response headers
Request and response bodies (up to configured limits)
Timing information
Connection metadata
Collection ID for data organization
Real client IP (when available)

Troubleshooting¶

Viewing Startup Command¶

HTTPCap prints the full command with all parameters at startup. Check the logs to see the exact configuration:

Bash

docker-compose logs httpcap | head -50

Common Issues¶

1. Permission Denied Errors¶

Problem: Cannot capture packets from network interface

Solution: Ensure the container runs with proper privileges:

YAML

privileged: true
cap_add:
  - NET_ADMIN
  - NET_RAW

2. High Memory Usage¶

Problem: Container using too much memory

Solution: Adjust heap limit and buffer sizes:

YAML

environment:
  HEAP_LIMIT: 8GiB
  ASSEMBLER_WORKER_PKT_BUF: 10000
  ASSEMBLER_OUT_BUF: 10000

3. Kafka Connection Issues¶

Problem: Cannot connect to Kafka broker

Solution: Verify Kafka connectivity:

Bash

# Test from within container
docker-compose exec httpcap nc -zv your-kafka-host 9092

4. Missing Traffic¶

Problem: Expected traffic not being captured

Solution: Check:

Correct interface name (INPUT_RAW)
BPF filter isn't excluding your traffic
The traffic is actually HTTP/HTTPS
IPIP setting matches your network configuration

Debugging¶

Enable verbose logging for troubleshooting:

YAML

environment:
  VERBOSE: 5
  METRICS_OUTPUT: console

Check application logs:

Bash

docker-compose logs -f httpcap

Performance Optimization¶

Network Interface Selection¶

Use the correct interface where your traffic flows
Consider network topology when using IPIP decapsulation

Worker Tuning¶

Processing Workers: Set to number of CPU cores
Assembler Workers: Set to 50-75% of processing workers
Buffer Sizes: Increase for high-traffic environments

Memory Management¶

Monitor heap usage and adjust HEAP_LIMIT accordingly
Balance buffer sizes with available memory
Use metrics to identify bottlenecks

Kafka Optimization¶

Increase producers for higher throughput
Adjust flush settings based on latency requirements
Use compression to reduce network bandwidth

Maintenance¶

Updating the Application¶

Stop the current instance:
Bash
```
docker-compose down
```

Load the new image:

Bash

docker load -i httpcap-new-version.tar.gz

Update docker-compose.yml with new image name
Start the updated application:
Bash
```
docker-compose up -d
```

Monitoring¶

Monitor application health through:

Docker logs: docker-compose logs -f
Container stats: docker stats httpcap-instance
Kafka lag monitoring on the configured topic
System metrics (CPU, memory, network)

Backup and Recovery¶

Since HTTPCap is stateless, no backup is required. To recover:

Ensure Kafka cluster is healthy
Restart the HTTPCap container
Verify traffic capture resumes

Security Considerations¶

Network Access: HTTPCap requires privileged access to capture network packets
Data Privacy: Captured traffic may contain sensitive information
Host Isolation: Consider network segmentation for the capture host
BPF Filters: Use filters to exclude sensitive internal traffic

Support¶

For issues or questions:

Check the troubleshooting section
Review application logs with verbose mode enabled
Verify all prerequisites are met
Ensure network connectivity to Kafka cluster
Check the startup command in logs for configuration verification

Appendix¶

Built-in Features¶

The following features are always enabled in the current version:

Promiscuous mode: Interface captures all traffic
Response tracking: Matches requests with responses
JSON output format: Kafka messages in JSON
Decompression: Automatic decompression of compressed responses
AF_PACKET engine: High-performance packet capture on Linux

System Requirements¶

Minimum Requirements(Up to 2000 TPS):

CPU: 2 cores
RAM: 4GB

Recommended for Production(+5000 TPS):

CPU: 8+ cores
RAM: 16GB+