Snapshot

To retrieve a snapshot of the threat intelligence from the ThreatStream cloud.

Snapshot is a point-in-time copy of the threat intelligence on ThreatStream.

ThreatStream provides multiple methods for retrieving threat intelligence. Use the following criteria to determine which method is appropriate for your needs:

Task Appropriate Method
Downloading small intelligence sets on an ad-hoc basis. Use the Intelligence API. See Intelligence for more information.
Downloading large intelligence sets–those spanning thousands of pages–on an ad-hoc basis. Use the Snapshot API.
Downloading large intelligence sets–those spanning thousands of pages–on a regular basis. Use ThreatStream Integrator. Contact your Anomali Support representative for more information.

You can set filters to specify criteria by which intelligence should be retrieved, as described in the table below.

Note: Your organization can have no more than three snapshot processes running at once.

Request

/api/v1/snapshot/

HTTP Methods: GET, POST

Attributes: See the table below

Attribute Element Description
custom_config

A map containing the following elements: filter, max_results, whitelist/blacklist, field_whitelist/field_blacklist, chunk_size, format and use_org_whitelist.

For example,

"custom_config": {"filter": "state=\"active\"", "whitelist": {"itype": ["mal_ip", "mal_domain"]}, "field_blacklist": ["asn", "country"], "chunk_size": "1MB", "format": "json", "use_org_whitelist": true}

filter

Filter string to use as the search criteria for the intelligence.

Example:... {"filter": "state=\"active\" AND confidence >=70 AND date_last >=2015-05-01T00:00:00", ...}

See Intelligence Fields in ThreatStream for a complete list of fields you can specify.
max_results

Specify a maximum number of results to return.

Example:

max_results=50000
whitelist

Only include indicators that match the specified fields and values.

Syntax:"whitelist": {"<field_name_1>": ["value_1", "value_2"...], "<field_name_2>": ["value_1", "value_2"...]}

If multiple fields are specified in a list, then the “AND” boolean operation is assumed among the fields.

Examples:

  1. "whitelist": {"itype": ["mal_ip", "mal_domain"]}

    Only indicators of type mal_ip and mal_domain will be included in the snapshot. All others indicator types (itypes) except these two will not be included.

  2. "whitelist": {"country": ["CN"], "itype": ["mal_ip", "mal_domain"]}

    Only indicators for which country = CN AND indicator type is either mal_ip or mal_domain will be included in the snapshot.

Notes:

  • You can only use either the whitelist or the blacklist attribute in a snapshot call, but not both.
  • See Intelligence Fields in ThreatStream  for a complete list of fields you can specify.
blacklist

Exclude indicators that match the specified fields and values.

Syntax:"blacklist": {"<field_name_1>": ["value_1", "value_2"...], "<field_name_2>": ["value_1", "value_2"...]}

If multiple fields are specified in a list, then the “AND” boolean operation is assumed among the fields.

Examples:

  1. "blacklist": {"itype": ["mal_ip", "mal_domain"]}

    Indicators of type mal_ip and mal_domain will be excluded from the snapshot. All others indicator types (itypes) will be included in the snapshot.

  2. "blacklist": {"country": ["CN"], "itype": ["mal_ip", "mal_domain"]}

    Indicators for which country = CN AND indicator type is either mal_ip or mal_domain will be excluded from the snapshot. All other indicators will be included in the snapshot.

Notes:

  • You can only use either the whitelist or the blacklist attribute in a snapshot call, but not both.
  • See Intelligence Fields in ThreatStream for a complete list of fields you can specify.
field_whitelist

Include only these fields in the results. Remove all other fields.

For example, if "field_whitelist": ["asn", "severity", "domain"], only these fields will be included for each indicator in the snapshot.

Notes:

  • A field_whitelist can be used with a whitelist or a blacklist.
  • When specifying a field_whitelist, make sure that you include at least one of the intelligence fields. For csv and json formats, these fields include domain, email, srcip, md5, srcip, url. If you do so, no intelligence will be returned. For csv_v2 and json_v2, the intelligence field for all observables is value.
  • Ensure that the combination of (whitelist + field_whitelist) does not result in a field set that will always have empty results. For example, "whitelist": {"itype": ["mal_ip", "mal_domain"]} + "field_whitelist": ["asn", "severity", "url"], will always result in no matches. Whitelist will only show mal IP and mal domain information and the field_whitelist will only show asn, severity, and url information, resulting in empty results.
  • See Intelligence Fields in ThreatStream for a complete list of fields you can specify.
field_blacklist

Exclude these fields from the records that are returned.

For example, if "field_blacklist": ["confidence", "severity"], these fields will be excluded for each indicator in the snapshot.

Notes:

  • A field_blacklist can be used with a whitelist or a blacklist.
  • When specifying a field_blacklist, make sure that you do not blacklist all of the intelligence fields. For csv and json formats, these fields include domain, email, srcip, md5, srcip, url. For csv_v2 and json_v2, the intelligence field for all observables is value.
  • Ensure that the combination of (whitelist + field_blacklist) does not result in a field set that will always have empty results. For example, "whitelist": {"itype": ["apt_domain", "mal_domain"]} + "field_blacklist": ["asn", "severity", "domain"], will always result in no matches. Whitelist will only show apt and mal domain information and the field_blacklist will hide the domain information, resulting in empty results.
  • See Intelligence Fields in ThreatStream for a complete list of fields you can specify.
chunk_size

Specify the size of the snapshot file, in KB or MB. The maximum supported value is 200 MB.

To retrieve a successful snapshot, you must specify either chunk_size or chunk_ioc_count. When a value is not specified for this attribute, the API call returns an error. Do not specify values for both attributes.

Note: If the chunk_size is small and cannot fit in an entire snapshot, you will need to retrieve multiple chunks until all data has been retrieved. If more data is available, the value of the Status field is "inprogress". Continue to retrieve the data until Status field value is "completed".
chunk_ioc_count

Specify the number of Indicators of Compromise (IOCs) the snapshot can contain. The maximum supported value is 250,000.

To retrieve a successful snapshot, you must specify either chunk_size or chunk_ioc_count. Requests fail without values specified for one of these attributes. Do not specify values for both attributes.

format

Format of the snapshot.

Available formats: csv, csv_v2, json, json_v2, stix

Note: If you specify csv or json, only domain, email, MD5, IP, and URL observables are included in snapshots. Therefore to ensure complete data is returned Anomali recommends using csv_v2 or json_v2.

  use_org_whitelist

Specify whether your organization Import Whitelist should be applied to the snapshot.

Possible values: true, false

When set to true, observables configured on your Import Whitelist are filtered out of the snapshot.

If no value is specified, your Import Whitelist is not applied to the snapshot.

intel_version integer A field to specify the intelligence version—v1 or v2. Currently, only v1 is supported.

Response

Format: CSV, JSON, or STIX

Attributes: See the table below

Filtering of attributes: See the table below

Attribute Type Filter Operators Description
id integer "lt", "lte", "gt", "gte" A unique identifier for the snapshot.
location string Not available for filtering Location header that specifies the URL to use for checking the status of snapshot creation and downloading the snapshot files.
resource_uri string Not available for filtering URI for the snapshot on ThreatStream.
sha256sum string "exact"

The sha256sum (to verify the integrity) of the snapshot file that is available for download.

Note: This attribute is only available in the response.

snapshot_ts datetime "lt", "lte", "gt", "gte" Timestamp when the snapshot was started on ThreatStream.
status string Not available for filtering

Status of snapshot creation. Status is "inprogress" when snapshot files are being created. Once all files have been created, the Status is "completed". Status may be "errors" if the snapshot creation failed for any reason.

Possible values: inprogress, completed, errors

Example

To obtain a full snapshot, in JSON format, in 1 MB chunks, that meets the specified criteria:

Copy 
curl -XPOST 'https://api.threatstream.com/api/v1/snapshot/' -H 'Authorization: apikey <username>:<api_key>' -H 'Content-Type: application/json' -d '{ 
   "intel_version":1,
   "custom_config":{ 
      "filter":"state=\"active\"",
      "whitelist":{ 
         "itype":[ 
            "mal_ip",
            "mal_domain"
         ]
      },
      "field_blacklist":[ 
         "asn",
         "country"
      ],
      "format":"json_v2",
      "use_org_whitelist":true,
      "chunk_size":"1MB"
   }
}'

A successful response contains a Location header, such as https://api.threatstream.com/api/v1/snapshot/2/.

The Location header contains the ID of the snapshot. In the above example, https://api.threatstream.com/api/v1/snapshot/2/, 2 is the snapshot ID. Use this ID when making HTTP GET requests to retrieve the snapshot. If the snapshot contains multiple files, the HTTP GET response will contain URLs of all the files along with the Status. If all files could not be retrieved with one HTTP GET request, Status will be "inprogress". Continue to make requests until all files have been retrieved. Once all files have been retrieved, the Status will change to "completed".

Copy 
curl 'https://api.threatstream.com/api/v1/snapshot/2/' -H 'Authorization: apikey <username>:<api_key>'

The above example assumes that the Location header is https://api.threatstream.com/api/v1/snapshot/2/.

Output Sample

The calls to the status URL will return JSON and STIX output similar to the following:

Copy 
{ 
   "status":"inprogress",
   "files":[ 
      { 
         "download_url":"https://<download_URL_1>",
         "sha256":"<SHA-256 sum of the file to be downloaded>",
         "total_count":<count>
      },
      { 
         "download_url":"https://<download_URL_2>",
         "sha256":"<SHA-256 sum of the file to be downloaded>",
         "total_count":<count>
      }
   ]
}

 

Copy 
{ 
   "status":"inprogress",
   "files":[ 
      { 
         "download_url":"https://<download_URL_1>",
         "sha256":"<SHA-256 sum of the file to be downloaded>",
         "total_count":<count>
      },
      { 
         "download_url":"https://<download_URL_2>",
         "sha256":"<SHA-256 sum of the file to be downloaded>",
         "total_count":<count>
      }
   ]
}

Error Codes

If the snapshot creation process fails, HTTP status codes reflect the reason for failure.