Ingesting into a SIEM

First connect your Kafka consumer. For IBM QRadar, follow Connecting IBM QRadar to prepare the keystore and truststore and configure its Kafka collector. Use the topics and consumer group issued in connection.yml.

Select a parser for each feed

Use the topic-to-feed mapping in connection.yml to select the appropriate parser. Configure field mappings from that feed's reference: record shape, field names, types, optional values, and event semantics depend on the feed. Separate parsers may be needed for feeds from the same service.

Retain the original payload alongside normalized SIEM fields. Preserve documented types and treat absent optional fields according to the schema. Accept additional fields without failing ingestion. Use documented structured fields for rules and correlation, with human-readable text for analyst display where the feed provides it.

Preserve source time

Map the feed's documented event-time field to your SIEM's event time. Preserve the original value and precision in the raw record if the SIEM's indexed time has lower precision. Keep ingestion time separately so delivery delay remains visible.

Check what the time represents: a source event, an observation, or an aggregation window can require different dashboard and correlation rules. Use the timing assurance applicable to the feed. See Time provenance for CNX's timing reference and the currently documented service-specific assurance.

Apply the feed's delivery rules

Configure replay and deduplication using the identifiers and guarantees in the feed reference. Some feeds provide a stable event identifier; others describe observations or aggregates. Apply sampling and aggregation rules when calculating totals or interpreting missing records.

Use documented correlation identifiers to establish relationships between records. Kafka arrival order does not establish a sequence across hosts or topics. Consult Delivery and compatibility for the shared retention window and consumer behavior.

Validate and monitor ingestion

Validate representative records from each subscribed feed against its schema. Check parsing, event time, optional fields, and the rules used by dashboards. Monitor parse failures alongside consumer lag, authentication errors, and storage failures. Commit offsets after successful processing according to your collector's durability model, and retain imported data under your own retention policy.

For DNS mappings and examples, see Ingesting DNS telemetry and the DNS feed reference. Use the feed catalog to find each feed family's schema.