Feedback:

dedup

Do not apply the dedup operator to the message field. Doing so impacts search performance because each raw event is held in memory for this operation. Anomali recommends following the AQL Best Practices to reduce the dataset size by limiting or filtering it before using dedup.

Use the dedup operator to remove duplicate records or entries from a dataset and leave only unique entries based on specified criteria.

Syntax

|fields <field> [,<field>,...] |dedup <field> [,<field>, ...]

Usage

You must place the fields operator prior to the dedup operator in your query to ensure accurate functionality.

Example

Example 1: This shows one event log for each sourcetype.

Copy 
|fields sourcetype  |dedup sourcetype

Example 2: This shows one event log for each action.

Copy 
|where action is not null |fields action |dedup action

Example 3: Filter privileged user accounts and replace specific username values.

Note: For OCSF schema fields, array indexes start at array[1] and not array[0].

Copy 
ocsf | where actor.user.name IN ('root', 'Administrator', 'system') | dedup actor.user.name | replace 'system' with 'test_of_system_replace' in actor.user.name