Full-Text Log Indexing

Anomali Search offers full-text indexing on eventlog data, a type of search capability that allows you to directly perform a keyword search on the raw message and ensures faster search results, while reducing the AVC units you consume.

Full-text indexing in Anomali is near real-time, with most events indexed within 10 minutes after ingestion.

Key Technical Guidelines

When Schema RBAC is active for your organization, the schema filters in your assigned roles determine which data this feature returns. An eventlog filter limits your results to only the eventlog data your role permits, and an OCSF filter limits your results to only the OCSF data your role permits. If a query runs against a schema for which your role has no filter assigned, the query returns a forbidden error. For details, see Role-Based Access Control for Schemas.

Full-text indexing entails the following technical highlights:

• Full-text indexing is applicable to both the eventlog and ocsf schemas. See OCSF Schema Overview for more information on OCSF Version 1.2 parsing and naming conventions.

• You must place the desired keyword at the beginning of the query to leverage full-text indexing. To use the OCSF schema, add the case-insensitive schema name "ocsf" as a prefix before the desired keyword.

• You must enclose the desired keyword in double quotes ("") to leverage full-text indexing.

• Full-text indexing does not support wildcards, partial matches, or substrings. Only exact matches are supported. This applies to sourcetype naming as well.

• All full-text searches ignore character case. For example, "Error" and "error" are treated identically.

• For OCSF schema fields, array indexes start at array[1] and not array[0].

• The full-text indexer performs tokenization using the following definitions:

  • Valid characters: letters, digits, dot (.), dashes (-), underscore (_).

  • Delimiters: spaces, punctuations, symbols, special characters.

• Full-text indexing is supported by AQL search queries across the Anomali platform.

How to Use

Example 1: Enter a desired keyword enclosed in double quotes (""), just like you would in a Google-like search.

"john.doe@acmecorp.com"

Example 2: As an example, combine AQL operators on the raw message, such as the where operator, with the desired keyword at the start of the query.

"Alice.Bob" | where sourcetype="windows_logs" 

"Alice.Bob" | aggr count as total_count by sourcetype | sort cnt desc | limit 10

Example 3: The desired keyword must be at the start of a query. Here, the email address is after the where operator, so full-text indexing is not applied.

| where message contains "John.Doe@acmecorp.com"

Example 4: Full-text indexing applies to keywords at the start. Here, "abcd" will utilize full-text indexing, but not "efgh".

"abcd" | where message contains "efgh" | limit 10

Example 5: Full-text indexing applies to the ocsf schema. Here, the indexer looks for a UUID in an ingested log.

ocsf "d3590ed6-52b3-4102-aeff-aad2292ab01c" | aggr count as cnt by category_name

Example 6: Wildcard notation or substring searches are not allowed:

• Search keyword "acme" will not return results when your log data contains "john.doe@acmecorp.com".