datagen

Purpose

datagen populates ClickHouse with realistic synthetic data — hosts, clients, certificates, TLS handshakes, and TLS scans — so a development or demo environment has something to look at without needing live network traffic. Generated handshakes are modeled on real browser/tool JA3 fingerprints (Chrome, Firefox, Safari, Edge, curl, Python Requests, Go’s HTTP client) plus a small percentage of known-malicious fingerprints (TrickBot, Emotet, Dridex, Cobalt Strike, Metasploit) so malware-detection and compliance views have something to trigger on.

Usage

datagen [options]

Flag	Default	Description
`-h`	`localhost`	ClickHouse hostname
`-p`	`9000`	ClickHouse port
`-U`	`default`	ClickHouse user
`-W`	—	ClickHouse password (prompted interactively if omitted)
`-db`	`insights`	ClickHouse database name
`-q`	`10000`	Number of TLS handshakes to generate
`-hosts`	`50`	Number of unique hosts to generate
`-clients`	`100`	Number of unique clients to generate
`-d`	`7`	Number of days to spread random timestamps over
`-debug`	`false`	Enable debug logging
`-realistic`	`false`	Generate hostnames under security-profile domains (`.recommended.test`, `.secure.test`, `.weak.test`, `.insecure.test`) for use against the `tlslab` Docker test network
`-docker-hosts`	—	Output path for a dnsmasq hosts file (only used with `-realistic`)

Realistic mode

With -realistic, generated hostnames are distributed across four security profiles — 30% recommended, 35% secure, 25% weak, 10% insecure — plus a fixed set of certificate edge-case hosts (expired.certs.test, selfsigned.certs.test, wronghost.certs.test, shortkey.certs.test, sha1.certs.test, expiring-soon.certs.test). Combined with -docker-hosts, it writes a dnsmasq hosts file mapping each profile to a fixed IP on the 172.30.0.0/24 test network (.10–.14), matching the addresses tlslab’s Docker test endpoints listen on.

Examples

# Quick local dataset: 90 days, 300 hosts, 5000 handshakes
go run cmd/datagen/datagen.go -d 90 -q 5000 -hosts 300 -db insights -U default

# Larger dataset with explicit ClickHouse connection details
go run cmd/datagen/datagen.go -h localhost -p 19000 -U insights -db insights \
  -d 30 -q 10000 -hosts 50 -clients 100

# Realistic mode for testing against the tlslab Docker network
go run cmd/datagen/datagen.go -realistic -docker-hosts /tmp/dnsmasq-hosts.conf

Before regenerating a large dataset, clear the existing tables:

TRUNCATE TABLE tls_hellos;
TRUNCATE TABLE ja3_fingerprints;
TRUNCATE TABLE ja3_malicious;

tlslab ja3import