Kafka

Anomali Security Analytics can ingest event log data from Apache Kafka. Kafka is a log source type that supports the following features:

  • Drilldown: Security Analytics supports drilldown to raw logs if you enable the option to store raw events.

  • Backfill: The backfill feature allows you to ingest event logs from a time period before the configuration was activated.

Network Topology and Requirements

Security Analytics listens to Universal Link for updates from log sources. Universal Link sends queries to Kafka to collect events.

  • Make sure your firewall allows connectivity between the Kafka broker and both Security Analytics and Universal Link. Direct communication with Security Analytics occurs during the initial log source configuration when Security Analytics sends queries to Kafka to get Kafka topics. The periodic communication to ingest logs does not involve direct communication between Security Analytics and Kafka.

  • Universal Link sends periodic queries to Kafka port 9092/tcp to collect events.

  • Universal Link must be able to send requests and data outbound to api.threatstream.com and AWS S3.
  • Security Analytics listens to Universal Link for updates from log sources on port 443/https. Typically, your network and security team creates rules to allow this traffic when Universal Link is first installed.

  • Refer to Anomali Security Analytics Cloud IP Addresses, Ports, and URLs for Security Analytics network details that your network and security needs to know to allow communication from your enterprise resources to your Security Analytics deployment.

Before You Begin

  • Have connection details at hand. During the configuration, you specify the IP address and port for the Kafka Broker service and select a Kafka topic.

Step 1: Create the Security Analytics Log Source Configuration

To add the log source:

  1. Click the Settings icon in the left navigation panel and select Security Analytics.

  2. Click Links.

  3. Locate the Universal Link configuration for which you want to add a log source and click the hyperlink name in the Name column. The Link must be online.

  4. Click the Available tab to display the log source types available. The set of available log source types are different for Cloud Link and Universal Link deployed on premises.

  1. Click the Kafka icon.

  2. Complete the configuration as described in the following table.

    Setting Description
    Data Source Name Type a name that facilitates Link management. Only ASCII characters are allowed.
    Data Source Description

    (Optional) A short description for the log source.

    Default Time Zone

    Select the desired timezone for the log source from the drop-down list. Type the first few characters of the timezone for a quick search.

    The default is Universal Link System Time Zone, which is the timezone where Universal Link is installed.

    Exclude Internal Network Events

    Select this option if you do not want to include traffic within your internal network in IOC matches.
    Store raw events

    Select this option to collect and store the entire event log that is available from the log source. Requires Universal Link 5.2 or later.

    When this option is enabled, and you click Get Raw Logs to drill down from Security Analytics results, Security Analytics displays the log as seen on the original device.

    When this option is not enabled, Security Analytics filters the fields collected and stored to include only fields that can be mapped to the Anomali Security Analytics eventlog schema. When this option is not enabled, and you click Get Raw Logs to drill down from Security Analytics results, Security Analytics can display only the log fields collected and used in Security Analytics.

    Note: Collection and storage of raw event logs utilizes greater bandwidth and storage resources than collection of logs that have been filtered.

    Kafka broker server

    Specify the IP address or fully-qualified domain name for the Kafka broker server in one of the following formats: 

    <IP_address>:<port>
<localhost>:<port>
    Default: localhost:9092
    Kafka Topic

    Select the Kafka topic from the list to subscribe to. Click the refresh icon to fetch the topics from the server. The default is logstash.

    Important: Do not create multiple Kafka log sources using the same topic. Universal Link only supports one unique topic per Kafka log source. This applies whether you are using a Kafka cluster or a standalone Kafka deployment.
    Backfill Earliest Events

    Click this option if you want to fetch the earliest events from the Kafka topic when this log source is first activated. This will constitute the first batch of events from the Kafka log source. Thereafter, only the most recent events are fetched.

    If not selected, Universal Link will fetch the most recent events when this log source is first activated.

    Use Custom Mapping Select this option if you want to specify custom mapping.

  3. Click Next if you are using custom mapping; otherwise, click Save.

  4. To use custom mapping:

    If there are live events in the Kafka server, or if you enabled backfill, Security Analytics fetches live or backfill events from the Kafka server. These will be used as sample data. If the Kafka server is unavailable, either paste the sample content directly into the Sample Content field, or browse and select a file on your local system that has a sample row of data.

  1. Click Next to display the parser setup page.

    Select the parser and set the field delimiter. Click Run to test that your settings produce the desired output.

    Parser Type Delimiter
    CEF Uses default delimiters.
    CSV

    The delimiter characters include Pipe,Comma, Semi-colon, Space, or a Custom delimiter of your choice. Select one.
    JSON Uses default delimiters.
    Key/Value

    This type has two sets of delimiters:

    • Field Delimiter: Comma, Semi-colon, Space, Pipe, or a Custom delimiter of your choice.

    • Value Delimiter: Colon, Equal, Double Equals, or Custom.

    Select one from each set.
    LEEF Field Delimiter: Pipe, Comma, Semi-colon, Space, Tab, or a Custom delimiter of your choice. Select one.
    Dissect Specify a Dissect expression in the Input box.
    Regex/Grok Specify a regex/grok expression in the Input box.
    Ruby Specify a Ruby expression in the Input box.

  2. Click Next to display the mapping page.

    Map the data as required. The mapping information shows the default field mapping values. The top section shows the raw data from your log source. The left side of the screen lists the raw data (Source) and a likely match with one of the fields (Destination) in Security Analytics. The section on the right side shows a system-generated mapping. If needed, use the Source and Destination drop down menus to manually remap the data.

  1. Click Save.

Step 2: Activate the Security Analytics Log Source Configuration

By default, a newly-added data source is not active.

To activate a log source:

  1. Click the Settings icon in the left navigation panel and select Security Analytics.

  2. Click Links.

  3. Locate the Link configuration you want to review and click the hyperlink name in the Name column.

  4. Locate the log source you want to activate and toggle the activation switch shown in the following figure.

    When you activate the log source, the associated Link configuration must be regenerated.

  5. Click Regenerate Now in the Link tools panel.

Step 3: Verify Logs Are Ingested into Security Analytics

You can expect logs to be transmitted to Security Analytics shortly after the log source has been activated and the Universal Link regenerated.

To verify logs are ingested:

  1. Click the Settings icon in the left navigation panel and select Security Analytics.

  2. Click Links.

  3. Click the hyperlink in the Name column for the Universal Link associated with the log source.

  4. Review the timestamp for the last data received and the number of events in the last batch of data to confirm event logs are ingested as expected.

Troubleshooting Tips

  • During Kafka logsource processing (for example, while updating the list of Kafka topics), make sure the Kafka server is available and the required port is open. Otherwise, you will get an error message similar to the following:
    Failed to get Kafka topics. Error: NoBrokersAvailable