Connecting a Kafka consumer

Your telemetry config and credentials repositories are hosted on CNX's Git server. Sign in through your organisation's SSO integration. Coordinate with the internal team responsible for SSO and CNX repository access to obtain write access to config and read access to credentials for the people managing the integration.

CNX enables the telemetry subscription and provides your repository links during onboarding. Choose the feeds from Available feeds; availability depends on the feeds enabled for your account.

SSO provides access to the repositories used to manage the subscription. Your Kafka consumer authenticates separately with the client certificate and private key configured below.

1. Generate a key and certificate request

Generate a private key and a PEM certificate signing request (CSR) locally using your organization's approved tools. Keep the private key in your consumer's secret storage. Commit only the CSR to the config repository. CNX uses its public key to issue your client certificate and assigns the subscriber identity and permissions.

2. Configure the subscription

Add the CSR under csr/ and create subscriptions.yml:

version: 1
subscriptions:
  - id: siem-primary
    name: "Primary SIEM"
    csr: csr/siem-primary.pem
    feeds:
      - dns.logs.ops
      - dns.logs.platform
      - dns.queries.detail
      - dns.queries.summary

Push the files directly to the config repository's main branch. The subscriber configuration does not require a merge request. See Subscription configuration for fields and the credentials repository layout.

3. Retrieve the issued credentials

Read subscribers/siem-primary/ in the credentials repository after CNX publishes the accepted configuration. Use connection.yml for the exact bootstrap endpoint, physical topic names, and consumer group. Check the certificate validity and the accepted feeds in manifest.yml.

Retrieve cnx_ca.crt and cnx_ica_service.crt from the repository root for broker trust. The root also contains cnx_ica_external_client.crt, the issuer of your client certificate. client-fullchain.crt already bundles your leaf with that client-issuing intermediate.

The credentials repository README links to this guide. CNX delivers PEM certificates and connection.yml supplies your connection values. If your collector requires a Java keystore, follow Connecting IBM QRadar to convert the certificates locally with your private key.

4. Configure the consumer

Set these values in your Kafka client or SIEM collector:

Enable broker hostname verification. Allow outbound connectivity to the bootstrap endpoint and every broker endpoint advertised in Kafka metadata. Bootstrap connectivity alone does not confirm that the consumer can fetch records from all topic partitions.

5. Validate ingestion

Confirm that each selected topic can be described and consumed. Check that records match the schema for the selected feed and your subscription's scope. Use the feed's documented validation procedure or representative events to check parsing and ingestion. An idle topic alone does not establish a connection failure.

Map event fields using Ingesting into a SIEM. Decide whether a new consumer starts at the earliest retained record or from new records, using your client's offset-reset configuration.

6. Operate continuously

Monitor consumer health, partition lag, authentication errors, and certificate expiry. Kafka retains two hours of data; keep lag comfortably within that window and store consumed records under your own retention policy.

CNX publishes renewed certificates to the credentials repository when fewer than 30 days remain. Retrieve the replacement, update your collector's certificate or keystore, and verify consumption before the old certificate expires. Keep the corresponding private key available. For a key rotation, generate a new key and CSR and update the subscription's CSR file.

Use a separate subscription for another independent SIEM or analytics destination. Consumers sharing one group divide partitions between them.