insights

Purpose

insights is the main REST API server. It serves the endpoints the web dashboard uses to query captured traffic, hosts, certificates, compliance status, and SCA results, and it handles authentication (JWT sessions backed by FusionAuth, with optional two-factor authentication). It connects to ClickHouse for all data and to Redis for background task enqueueing (TLS/nmap scans triggered from the dashboard) and the WebSocket status feed.

Usage

insights -c <config.yaml> [-debug]

Flag	Default	Description
`-c`	`/etc/insights.yaml`	Path to the configuration file
`-debug`	`false`	Enable debug-level logging

The config file also carries a debug: true setting; either the flag or the config value turns on debug logging.

Configuration

configs/insights.yaml is the template. Key sections:

name: insights
debug: true
listen: "localhost:3000"
uploadpath: ~/tmp
backup_path: ~/tmp/backups

db:                # ClickHouse (note: uses the generic `db` key, not `clickhouse`)
  host: localhost
  port: 19000
  database: insights
  username: insights
  password: secret

fusionauth:
  baseurl: http://localhost:9011
  apikey: <fusionauth-api-key>
  applicationid: <application-uuid>
  clientsecret: <client-secret>
  tenantid:

redis:
  host: localhost
  port: 6379
  channel: insights

sca:
  webhook_base_url: ""
  oauth:
    encryption_key: "${SCA_ENCRYPTION_KEY}"   # 32-byte hex string
    github:
      client_id: "${GITHUB_CLIENT_ID}"
      client_secret: "${GITHUB_CLIENT_SECRET}"
      redirect_url: "https://your-domain/oauth/github/callback"
    gitlab:
      client_id: "${GITLAB_CLIENT_ID}"
      client_secret: "${GITLAB_CLIENT_SECRET}"
      redirect_url: "https://your-domain/oauth/gitlab/callback"

On startup, insights connects to ClickHouse, loads kickstart defaults into the ClickHouse config table (see internal/config), and then starts serving HTTP on listen.

API surface

All routes are mounted under /api and require a JWT (Authorization: Bearer <token>) except /auth/*, the OAuth callback endpoints, and the webhook endpoints. Route groups include:

/api/clients, /api/hosts, /api/handshakes, /api/certificates — query captured traffic and inventory
/api/hosts/{uuid}/tlsscan, /api/hosts/{uuid}/nmapscan — queue a scan for a specific host
/api/traffic/{hourly,sixhour,daily,weekly,monthly} — time-bucketed traffic volume for dashboard charts
/api/compliance/* — compliance stats, findings, framework/rule management (see Compliance)
/api/cbom/* — Cryptographic Bill of Materials generation/download
/api/sca/* — repository connections, OAuth, scan triggers and results
/api/chat/* — an AI assistant that can execute read-only SQL against the schema for ad-hoc questions
/api/upload/pcap, /api/pcap/* — PCAP file upload and management (see pcapworker)
/api/status, /api/status/queues, /api/status/workers — service and asynq queue health
/auth/login, /auth/refresh, /auth/two-factor — authentication
/oauth/github, /oauth/gitlab (+ /callback) — SCA repository OAuth
/webhooks/github, /webhooks/gitlab — repository push webhooks that trigger SCA scans

A global rate limit of 100 requests/minute per IP applies to /api/*; a stricter 10 requests/minute applies to /auth/* to slow brute-force login attempts.

Example

go run cmd/insights/insights.go -c configs/insights.yaml -debug

capture