Feedback:

Send email

fit DensityFunction

Use the DensityFunction algorithm to model normal historical behavior using Kernel Density Estimation (KDE) and generate anomaly scores for new data.

Syntax

Model fitting phase

| fit densityfunction [bandwidth] [threshold] [lower_threshold] [upper_threshold] <numeric_field> [by <group_field> [, <group_field>...]] [into '<model_name>']

bandwidth

The smoothing factor for the Kernel Density Estimation, controlling the smoothness or extent of details of the density curve.

Lower bandwidth: sharper, more sensitive density curve (detects small variations).

Higher bandwidth: smoother, more generalized curve (ignores small fluctuations).
threshold

A general anomaly detection cutoff. Any density score below 0.01 is usually considered anomalous.
lower_threshold A lower bound for extremely rare events. This helps separate critical anomalies from normal low-density variations.
upper_threshold A high-density threshold. Useful for validation, debugging, or categorizing events as strongly normal.
numeric_field Field whose distribution you want to model.
by <group_field> Build separate density models per group. For example, one model per sourcetype.
into '<model_name>' Saves the trained model under this name so apply can use it later. Must be a string without spaces, wrapped with single or double quotes.

Model apply phase

| apply <model_name> as <output_field> <numeric_field> <group_field>[, <group_field>...]

<model_name> Name of a previously saved DensityFunction model.
as <output_field> Name of the field that will hold the score.
<numeric_field> The same field that was used during the fit phase.
<group_field> The same grouping fields used during the fit phase.

How DensityFunction Works

Training Phase (Model Fitting)

During the fitting phase, DensityFunction analyzes historical data and learns a probability density function that represents “normal” behavior. The function:

  • Accepts one or more numeric fields as training input.

  • (Optional) accepts grouping fields to build separate baseline models for each user, host, device, or other entity.

  • Uses KDE with a Gaussian kernel to generate a smooth density curve from raw observations.

  • Allows configuration of key parameters such as:

    • Bandwidth: controls how smooth or detailed the density curve is.

    • Kernel type: by default, Gaussian is the kernel type.

    • (Optional) Sampling options: synthetic sampling from the density curve.

    • Random seed: ensures reproducibility.

Once training is complete, you can optionally save the model and reused it to score new data.

Apply Phase (Scoring)

In the apply phase, the trained DensityFunction model:

  • Evaluates each new incoming event.

  • Determines whether the event fits within the learned normal distribution.

  • Generates a density score indicating how typical or unusual the new event is:

    • Higher density indicates a highly expected or normal outcome.

    • Lower density indicates a rare or anomalous outcome.

  • This score can be used to:

    • Trigger alerts

    • Feed dashboards

    • Power downstream detection logic

    • Enhance contextual analysis

When grouping fields is used during training, new data is automatically routed to the appropriate group-specific model.

Usage

DensityFunction is designed for scenarios where you need to:

  • Understand normal behavior in your environment over time.

  • Detect anomalies such as spikes, dips, or unusual distributions.

  • Capture cyclical patterns, including:

    • Hourly authentication activity

    • Daily or weekly access patterns

    • Seasonally recurring event frequencies

  • Identify outliers that may indicate security issues, misconfigurations, or system failures.

DensityFunction is particularly effective when analyzing cyclical or periodic security activity, such as login behavior by hour of day, network access patterns by weekday, or repeated operational workflows that follow predictable rhythms.

Usage Recommendation

  • Ensure consistent lengths of input-output fields, valid JSON, and avoid empty datasets.

  • KDE requires enough data per group. Very small groups lead to poor density estimation.

  • Avoid grouping fields with extremely high cardinality unless necessary.

  • Ensure numeric training fields are properly normalized or aggregated.

Performance considerations

  • KDE for 1 model is fast, but KDE for thousands of groups becomes heavy.

  • 100k groups is feasible possible but performance might degrade.

  • Performance might degrade exponentially with:

    • More grouping fields

    • High cardinality values (for example, user IDs)

    • Large datasets

Example

Example 1: In the following example, the model learns normal event-count patterns per sourcetype over 10-minute intervals and then scores new intervals to identify whether each count is typical or anomalous.

Model fitting phase

Copy 
| bin event_time span=10m | aggr count as cnt by event_time, sourcetype
| fit densityfunction cnt by sourcetype into 'den_func_model'

Model apply phase

Copy 
| bin event_time span=10m | aggr count as cnt by event_time, sourcetype
| apply den_func_model as final_densi