Import (with approval)

To import threat data (observables) into ThreatStream and require approval of the imported data through the ThreatStream UI. Unstructured data can be specified in any format, such as JSON, CSV, PDF, or plain text, and can be contained in a file or directly specified in the API call. You can also import data from plain text intelligence streams by specifying a URL. Once data has been imported, you must login to the ThreatStream UI and approve the import.

This method can be used to import unstructured and structured data. See Supported Data Formats for more information.

When importing unstructured data, all intelligence is associated with the same attributes—threat type, classification, confidence. For example, if a file has two IP addresses, each IP will be assigned the same indicator type based on the threat type you specify. See Request Attributes for a list of global settings that can be specified for imported observables.

However, when importing structured data, you can specify a different itype and tags for each observable in the structured file, which is different from the global settings. When a local setting is specified, it overrides the global setting.

Request

/api/v2/intelligence/import/

HTTP Method: POST

Attributes: See the table below

Request Attributes

These attributes specify global settings for imported observables. If an imported observable does not have an explicitly specified value for these settings, the global setting is applied. However, if an imported observable does have an explicitly specified value, it overrides the global value.

Supported Data Formats

This API call can be used to import unstructured and structured data.

Unstructured Data

There are no specific requirements for unstructured data.

Structured Data

Follow these guidelines when importing structured data:

• The structured data must be contained in a valid CSV file with the following header line: value, itype, tags. Optional columns include private_tags, is_anonymous, and tlp.

  value	itype	tags	private_tags	is_anonymous	tlp
  1.2.3.4	brute_ip	exploitation, phishing		true	green
  https://example.com	apt_url	phishing	sensitive		amber
  4d68462ca6071f9b6ad0fabea8bd1c9f	mal_md5			false	red
  %.skydome.com	mal_wildcard_domain	malware_domains
  https://bluebird.com/%	suspicious_wildcard_url	sus_urls

• The value header label is mandatory. This column must contain data.

• For domain, email, IP, MD5, and URL observables, itype, tags, and private_tags header labels can be omitted if no data will be specified for these columns. For all other observables, you must specify a value for itype. See Indicator Types in ThreatStream.
• tags is a comma-separated list of tags. Tags included in the tags column are assigned the Anomali Community visibility setting and visible to all users with access to the observable.
• private_tags is a comma-separated list of tags. Tags included in the private_tags column are assigned the My Organization visibility setting and visible only to users in your organization.
• is_anonymous sets whether the organization and user information for the observable is anonymized. Possible values include true and false. If no value is specified, observables will not be anonymized.
• tlp sets a TLP color for the observable. Possible values include clear, green, amber, amber+strict, and red.
• The data specified in itype and tags columns overrides the global settings specified in the Request Attributes settings. For example, if the structured CSV contains the following: value=1.2.3.4, itype=phish_ip, and Request Attribute ip_mapping=mal_ip, then itype for the observable 1.2.3.4 will be set to phish_ip because the setting in the file (local value) will override the Request Attribute setting (global value).
• If the itype and tags columns are empty, the data for these columns is derived from the Request Attributes settings. For example, if value=1.2.3.4, itype is not specified in the file, tags is not specified in the file, and the Request Attribute ip_mapping=mal_ip, then itype for the observable will be set to mal_ip.

Examples

1. To import observables from a file. The file can contain unstructured or structured data.

   curl example:

   Copy

   curl -X POST 'https://api.threatstream.com/api/v2/intelligence/import/'  -F "confidence=30" -F "classification=private"  -F "threat_type=malware" -F "file=@<filename>" -H 'Authorization: apikey <username>:<api_key>'

   Python example:

   def import_with_approval_using_file(username, api_key):
       auth = 'apikey {}:{}'.format(username, api_key)
       headers = {'Authorization': auth}
       api_url = "https://api.threatstream.com/api/v2/intelligence/import/"
       tags = [{'name': 'private_tag', 'tlp': 'red'}]
       payload={
           'classification': 'private',
           'confidence': 70,
           'threat_type': 'malware',
           'tags': json.dumps(tags)
       }
       filename = '<name_of_your_file>' #this can be a .txt, .csv or .pdf file
       files = {'file': open(filename, 'rb')}
   response = requests.post(api_url, headers=headers, data=payload, files=files)
   print(response.text)

2. To import observables directly, by specifying them in the API call:

   curl example:

   Copy

   curl -X POST 'https://api.threatstream.com/api/v2/intelligence/import/'  -F "confidence=70" -F "classification=private" -F "threat_type=apt" -F "datatext=<text_to_parse_and_import>" -H 'Authorization: apikey <username>:<api_key>'

   Python example:

   def import_with_approval_using_datatext(username, api_key):
       auth = 'apikey {}:{}'.format(username, api_key)
       headers = {'Authorization': auth}
       url = "https://api.threatstream.com/api/v2/intelligence/import/"
       tags = [{'name': 'public_tag'}, {'name': 'private_tag', 'tlp': 'red'}]
       payload={
           'classification': 'private',
           'confidence': 65,
           'threat_type': 'malware',
           'tags': json.dumps(tags),
           'datatext': <text_to_parse_and_import>'
       }
   response = requests.post(url, headers=headers, data=payload)
   print(response.text)
   

3. To import observables from a plain text intelligence stream by specifying a URL:

   curl example:

   Copy

   curl -X POST 'https://api.threatstream.com/api/v2/intelligence/import/'  -F "confidence=60" -F "classification=private"  -F "threat_type=c2" -F "url=<url_for_plain_text_stream>" -H 'Authorization: apikey <username>:<api_key>'

   Python example:

   def import_with_approval_using_url(username, api_key):
       auth = 'apikey {}:{}'.format(username, api_key)
       headers = {'Authorization': auth}
       api_url = "https://api.threatstream.com/api/v2/intelligence/import/"
       tags = [{'name': 'public_tag'}, {'name': 'private_tag', 'tlp': 'red'}]
       url = "<your_url>"
       payload={
           'classification': 'private',
           'confidence': 70,
           'threat_type': 'malware',
           'tags': json.dumps(tags),
           'url': url
        }
   response = requests.post(api_url, headers=headers, data=payload)
   print(response.text)

4. To import observables associated with tags that are private to your organization:

   Copy

   curl -X POST 'https://api.threatstream.com/api/v2/intelligence/import/' -F "threat_type=malware" -F "datatext=<text_to_parse_and_import>" -F "classification=private" -F "confidence=40" -F 'tags=[{"name":"my_tag", "tlp":"red"},{"name":"another_private_tag", "tlp":"red"}]' -H 'Authorization: apikey <username>:<api_key>'

5. To import observables and share them with trusted circles whose ids are 1 and 2:

   Copy

   curl -X POST 'https://api.threatstream.com/api/v2/intelligence/import/' -F "confidence=80" -F "classification=private" -F "trustedcircles=1,2" -F "threat_type=phish" -F "datatext=<text_to_parse_and_import>" -H 'Authorization: apikey <username>:<api_key>'

6. To automatically approve imported observables. An import job is created and observables are set to active:

   Copy

   curl -X POST 'https://api.threatstream.com/api/v2/intelligence/import/' -F "classification=private" -F "threat_type=malware" -F 'tags=[{"name":"tag1","tlp":"red"}]' -F "datatext=<text_to_parse_and_import>" -F "default_state=active" -H 'Authorization: apikey <username>:<api_key>'

   Copy

   curl -X POST 'https://api.threatstream.com/api/v2/intelligence/import/' -F "file=<filename>" -F "confidence=30" -F "classification=private" -F "threat_type=malware" -F "data_type=unstructured" -H "Authorization: apikey <username>:<api_key>"

7. To import observables and create an association between the import job and an Actor whose ID is 77:

   Copy

   curl -X POST 'https://api.threatstream.com/api/v2/intelligence/import/' -F "actors=77" -F "threat_type=apt" -F "datatext=<text_to_parse_and_import>" -H 'Authorization: apikey <username>:<api_key>'

8. To import wildcard observables in a CSV file. The file includes structured data.

   Copy

   curl -X POST 'https://api.threatstream.com/api/v2/intelligence/import/' -F "confidence=30" -F "classification=private" -F "threat_type=malware" -F "file=@<filename>" -H 'Authorization: apikey <username>:<api_key>'

    

Response

HTTP 202 Accepted {"import_session_id": IMPORT_SESSION_ID, "success": STATUS, "job_id": BACKGROUND_JOB_ID}

where

import_session_id is the ID for the import job. You will need this ID to check the status of the import job.

job_id is an identifier for the job on ThreatStream.

Example Response:

{"job_id": "6194c7c0-53ae-41fc-a6f0-dc3d174f02e2", "success": true, "import_session_id": "482"}

Checking status of an Import job

Use this API call to periodically poll the status of an import job:

curl -X GET 'https://api.threatstream.com/api/v1/importsession/import_session_id/' -H 'Authorization: apikey <username>:<api_key>'

where import_session_id is the identifier that was returned in the response of the Import call.

This call returns one of the following statuses in the "status" attribute:

• processing—the import process is in progress. Keep polling periodically.
• errors—the import process failed.

• done—the import process has completed.

  At this point, the call also returns numIndicators and numRejected attributes that denote the number of observables that were accepted for importing and the observables that were rejected, respectively. This information is the same as the one displayed on the Import Review page of the ThreatStream UI.

Getting Observable ID in an Import Job

Use this API to get IDs of the observables in an import job.

curl -X GET 'https://api.threatstream.com/api/v1/importsession/<import_session_id>/intelligence' -H 'Authorization: apikey <username>:<api_key>'

where <import_session_id> is the identifier of the import job.

This call returns the information about the import job. You can view IDs of the observables in the "id" attribute.

Approving All Observables in an Import Job

Use this API call to approve all observables in an import job:

curl -X PATCH 'https://api.threatstream.com/api/v1/importsession/<import_session_id>/approve_all' -H 'Authorization: apikey <username>:<api_key>'

where <import_session_id> is the identifier of the import job.

Approving Selected Observables in an Import Job

Use this API call to approve selected observables in an import job:

curl -X PATCH 'https://api.threatstream.com/api/v1/importsession/<import_session_id>/approve_selected/?ids=<id1>,<id2>' -H 'Authorization: apikey <username>:<api_key>'

where

<import_session_id> is the identifier of the import job.

ids=<id1>,<id2> is a comma-separated list of IDs of the observables to be imported.