Sample Patterns for Dissect, Grok, Regex, and Ruby Parsers

Custom log sources can implement user-defined parsers. Anomali provides sample patterns for programmable sub parsers.

Dissect

Dissect is a Logstash filter used to extract fields from log data. The following table provides examples of patterns commonly used to parse event logs.

Dissect Pattern Example
%{clientip}

1.2.3.4
%{status} 200
%{request} /english/venues/cities/images/montpellier/18.gif
%{size} 3171
%{httpversion} 1.1
%{method} GET
%{url} http://www.google.com/bot.html
%{@timestamp} 30/Apr/1998:22:00:52 +0000

Grok

Grok is a Logstash filter used to extract fields from log data. The following table provides examples of patterns commonly used to parse event logs.

Grok Pattern Example
%{URIPATH:uri_path} /v2/blacklist/
%{NUMBER:response_status} 15824, 0.043
%{WORD:method} GET,production
%{GREEDYDATA:body} any sentence
%{INT:port} 1234
%{IPORHOST:ip} 55.3.244.1
%{URI:url} http://sylvaclouds.eu/new1/IMG-0001-documents.exe
%{DATESTAMP:datetime} 2017-09-27 0:21:12
%{SYSLOGTIMESTAMP:date} Jun 2 14:55:46

Regex

Regex is short for regular expression. A regular expression specifies a string search pattern. The following table provides examples of regex patterns commonly used to extract fields from unstructured data.

Regex Pattern Example

(?<ip>([0-9]{1,3}.){3}[0-9]{1,3})

 38.99.50.98
(?<port>\d+) 8554
(?<method>\w+) GET
(?<action>\w+\s\w+). session miss.^M
(?<url>(?:https?:)(?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_-]*)+) http://sylvaclouds.eu/new1/IMG-0001-documents.exe
(?<hostname>[\w-]+) elb-shop-com
(?<status>\d{1,3}) 200
(?<customer_uri>[^/]+$) customerURI=/All Customers/ANSI/ANSI/740663
(?<uri_path>(?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_-]*)+) /Users/jdoe/Downloads/110
(?<hash>(^[a-fA-F0-9]{32}$)|(^[a-fA-F0-9]{40}$)|(^[a-fA-F0-9]{64}$)) Matches md5 or sha1 or sha256
(?<event_time>([\d{2,4}-]+){2}\d{2}\s([\d:]+){2}\d{2}) 2017-09-25 23:27:36
(?<timestamp>[\d-]+\w[\d:.-]+) 2019-10-20T23:00:26.059475-07:00
\<\d+\>(?<datetime>\d{2,4}[-/\s][A-Za-z]{3,9}[-/\s]\d+\s[\d:]+) <14>2019 Jun 22 16:24:05
(?<datetime>[A-Za-z]{3,9}\s[A-Za-z]{3,9}\s[0-9]{1,2}\s[\d:]+) Fri Aug 5 22:11:53 2011

Ruby

Ruby is a programming language that can be used to extract fields from unstructured data. The following table provides examples of Ruby logic commonly used to parse unstructured data.

Ruby Pattern Example
def filter(event)
date = event.get("date")
time = event.get("time")
if (not date.nil?) and (not time.nil?)
date_time = "#{date}T#{time}"
event.set("date_time", date_time)
end
return [event]
end

Merges two fields:

date=2018-07-10 time=22:31:14
def filter(event)
if event.get('srcip') == 'localhost'
return []
end
return [event]
end		 Drops events that matches srcip=localhost.

def filter(event)
date = event.get("<189>date")
date = date.sub!("2018", "2021")
event.set("date", date)
return [event]
end

 Replaces date from 2018 to 2021.

def filter(event)
service = event.get("service")
_1, _2 = service.split(":")
event.set("service_port", _2)
return [event]
end

 Splits line using delimiter (eg - service=udp/port:7001).

def filter(event)
url = event.get("cs_Referer")
if (not url.nil?) and url.start_with?("https://urlscan.io")
tags = Array(event.get("tags")) # note that Array(nil) => []
tags << "drop_this_event"
event.set("tags", tags)
end
return [event]
end

 Drops events that match cs_referer=https://urlscan.io using tags.

def filter(event)
events = []
["1st event", "2nd event", "3rd event"].each do |name|
e = event.clone
e.set('event_name', name)
events << e
end
return events
end

Splits a single event into three events.