API Integrations

Sift Security functionality can be extended by adding ‘integrations’ to other third-party applications. This document describes how a developer can leverage this feature to add an integration into Sift Security.

There are a few broad considerations when designing an integration with Sift:

  • These integrations run in a docker container, such that these docker containers run with the required scripts/executables as ENTRYPOINT’s. We will cover this and other Docker specifications in sections below.
  • Sift’s Integration engine calls/runs these integrations and assigns ‘work’ to them, which means that these processes should be short lived and should exit after finishing the process. The interactions between Sift’s Integration Engine and these integrations happen over STDIN and STDOUT, specific details of which would be covered in following sections. These STDIN/STDOUT interactions happen using a specific language defined by our JSON schema, which we discuss in the following sections with a case study.

With these broader considerations in mind, let us dive into the process of creating these integrations (or what we like to call them – agents).

Add Integration

Sift Security walks you through adding your integration through a guided step-by-step process which can be accessed under the Configuration tab, and Integrations sub-tab in the Sift Security user interface.

To begin, simply click on the ADD INTEGRATION button.

_images/figure-1.png

Click on ADD INTEGRATION to enter the wizard

1. Integration Name

Next, provide a unique name that your integration would be known as.

_images/figure-2.png

Note that this name is what your integration’s action will be displayed as. For example, if we choose Slack Integration as our Integration Name, following would be the displayed name of our integration.

_images/figure-3.png

2. User Interface

Next, you see the User Interface configuration box:

_images/figure-4.png
  • Here, you can select the Take User Input? checkbox if you require some sort of user input in a form for this integration.

  • We identify the tab/section which this integration is going to work on by configuring the Section option. You can only choose one of the Sections on which this integrations will function.

  • Next, you would need to configure the Section Element. You will notice that the options in the Section Element column change based on the chosen value in Section. This is because Section Element represents the UI entities within that Section.

    For example, the Canvas section has many types of Section Elements, namely nodes (which denote graph vertices in our graph canvas), edges (edges in the graph) and canvas (which represents a graph canvas as a whole), whereas the Collaborate section only has report as an element. This represents the setup where you can trigger your integration by clicking on this Section Element, and by selecting your integration from the Actions list. An example of such a list under Actions is:

    _images/figure-5.png
  • In addition to that, one can provide additional filters wherever they apply. For example, if you wanted your integration to show up only on IP address type of nodes, you could fill in the Section Element Type field with the value ip (as depicted in the following image). These values essentially serve as filters for the types of entities.

    _images/figure-6.png

3. Form Setup

In case your integration wants certain fields to show up on the UI or certain values to be prefilled, this is the place to set it up.

Let’s look at this picture and dissect it into its individual components, while using this as an example of what this form will end up looking like:

_images/figure-7.png
  • Field Name: This is the field name with which corresponding values will be associated. For example, if you chose the field name to be message, the JSON that will be assigned to you would look something like this:

    {"type":"work","agentID":"b556135e-06f3-4507-8363-5e04c8648970","payload":[{"name":"message","value":"Hello World! - from vivek@siftsec.com"}],"job_id":1507824886086}
    

    Note that the value in above JSON is associated with the name “message”. (Note: We will keep revisiting this “work” message throughout this document). By clicking on the Add New Field button, you can request multiple such fields.

  • Field Label: This is the display name of this field. In this example, since we chose the Field Label to be “Message”, you can observe “Message” being the displayed as the Field Name in the UI.

    _images/figure-8.png
  • UI Element: It’s here that you would specify the UI input types. These values directly correspond to the HTML input types such as text, textarea, select, and date. Note that you can configure the ‘select’ input type to configure a drop-down list type of input. You could then specify the items on your list by providing “Option Labels” and “Option Values” for the elements of your drop-down list.

  • Value(Optional): This is the prefill value of this field. This is where you can specify certain special SIFT-* keywords such that Sift Security will pre-fill your field with these special values. Here is the full list of these special keywords.

    Special SIFT-* Keywords
    Keyword Meaning
    SIFT-REPORT-URL URL of the report on which this integration is operating
    SIFT-REPORT-NAME Name of the report on which this integration is operating
    SIFT-USER-NAME Username of the user who is logged in and is triggering this action
    SIFT-TIMELINE-SELECTION-START Start time of the timeline selection on the canvas tab
    SIFT-TIMELINE-SELECTION-END End time of the timeline selection on the canvas tab
    SIFT-SELECTED-NODE The entire JSON blob that represents the node in the graph database on which the integration is operating
    SIFT-SELECTED-NODE-ID The ID in the graph database for the node on which the integration is operating
    SIFT-SELECTED-NODE-DISPLAY-NAME The displayed value of the node on which the integration is operating. For example, for an IP address node, this field would give us the actual IP address that the node represents
    SIFT-SELECTED-NODE-TYPE The data type of the node. This could be things like ip, hostname, email_address etc. Look at our data model for more details and complete list of these types
    SIFT-CANVAS-NAME The name of the canvas

    Another use case of these special keywords is that you can construct arbitrary strings from them. For example, your “Value” could be :

    Hello world ! from SIFT-USER-NAME . Click here for navigating to the report on Sift Security: SIFT-REPORT-URL
    

    This would manifest as the following string after Sift has replaced these keywords with corresponding strings.

    Hello world ! from vivek@siftsec.com . Click here for navigating to the report on Sift Security: https://myinstance.siftsec.com/index.html#/reports/434a80c0-997c-11e7-a364-47559a1cdfe1
    
  • Regex(Optional): You can specify the regex pattern that the user-filled values must adhere to.

  • Validation Message(Optional): In case the user-filled values fail to adhere to your regex pattern, you can display this validation message.

  • Tooltip(Optional):The tooltip that acts as help text on your field. In the example that we’ve been following, we’ve specified the Tooltip value to be: “The message you want to send to slack”. Notice how it manifests in the UI in the following image:

    _images/figure-9.png

4. Environment Variables

As mentioned before, we run these integrations in docker containers. We provide this section such that users can specify key-value pairs that are likely to change multiple times. This allows our users to add/edit these configurations without having to hardcode them in their scripts. We simply pass these values as Environment Variables to the docker containers such that the scripts can make use of them.

Let’s take the following example into consideration.

_images/figure-10.png

We will pass the key-value pair of url and its value as environment variable in the docker container. In our example Python script, we simply access this value by using the os module, using os.environ['url']

5. Upload

_images/figure-11.png

Upload requirements:

  • This must be a .zip file (not more than 100 mb in size) which, when unzipped, should have the following contents in the top level directory:

    • A Dockerfile

    • All the supporting files and scripts needed to build the image

      Note that we build the docker image on this top level directory, which means it is imperative that there is a Dockerfile in it. We extract this .zip file, change directory into the top level directory, and simply run: docker build . You can use the same method to test whether your zip file conforms to these criteria.

Dockerfile requirements:

  • We run the Docker images as executables, which means that these Dockerfiles should have the ENTRYPOINT instruction that allows the given behavior
    • This ENTRYPOINT should be an executable which should expect an input through STDIN and should return an output on STDOUT.

STDIN/STDOUT requirements:

  • Your script/executable will always receive one of two types of messages, which are of the following structure:

    • Describe messages: Sift’s Integration Engine sends these messages to all the integration executables to check whether they are functional.

      • This type of input to the integrations will always be the following JSON:

        {"type":"describe"}
        
      • Note that this message should always be responded with the following JSON:

        {"type":"describe"}
        
    • Work messages: These are the messages where all the payload is provided to the integrations to act upon.

      • Note the “type” field of this JSON, which is “work”

        {"type":"work","agentID":"b556135e-06f3-4507-8363-5e04c8648970","payload":[{"name":"message","value":"Hello World! - from vivek@siftsec.com"}],"job_id":1507824886086}
        
        • type: will always be work
        • agentID: unique ID that Sift’s Integration Engine assigns to the integrations
        • payload: the key-value pairs as requested by the integration during setup
        • job_id: unique ID that Sift’s Integration Engine assigns to each work-job
      • The key-value pairs that the integration needs are contained in the “payload” section of this JSON message.

      • The response to these work messages should be structured as follows:

        {"type":"work","agentID":"b556135e-06f3-4507-8363-5e04c8648970","payload":[{"name":"message","value":"Hello World! - from vivek@siftsec.com"}],"job_id":1507824886086}
        
        • job_id: should be the same as the one provided in the corresponding work message
        • type: should always be info
        • payload: This contains the message you want to display at the completion of the task. The fields that are allowed here are :
          • message: Display string
          • url(optional): You can use this field if there is a URL to expose at the end of the job
          • url_label(optional): You can use field to assign a label to the aforementioned URL
        • action: This is the display method of the message in payload. The only supported action currently is “popup-message”
        • timestamp(optional): current time in epoch-milliseconds
        • return_code(optional): 0 if successful, any other code represents failure

Example

Let’s walk through a case study of adding an integration that allows us to share reports created on Sift Security via Slack.

1. Write your script

You can choose any programming language you like, but let’s take Python (2.7) for this example.

Requirements:

  • Read from STDIN (Get one of ‘describe’ or ‘work’ type messages)
  • Write to STDOUT (Respond with one of ‘heartbeat’ or ‘info’)
  • Create a webhook on slack that allows us to post messages via REST API

Additional notes:

  • Notice that instead of hardcoding the Slack webhook URL, we read it from as an OS Environment Variable in this script
import os
import json
import time
import datetime
import requests


def post_to_slack(message):
  req = requests.post(os.environ['url'], data=('{"text": "' + message + '"}"'))
  return req.status_code

if __name__ == "__main__":

  input_json = raw_input()
  parsed_input = json.loads(input_json)

  if parsed_input["type"] == "describe":
      registration = dict()
      registration["type"] = "heartbeat"
      print json.dumps(registration)

  else:
      job_id = parsed_input["job_id"]
      msg = parsed_input["payload"][0]["value"]
      returned_code = post_to_slack(msg)
      info = dict()
      info["type"] = "info"
      info["job_id"] = job_id
      info["agent"] = "slack-agent"
      info["timestamp"] = long(time.time() * 1000)
      info["action"] = "popup-message"
      info["payload"] = {}
      if returned_code == 200:
          info["payload"]["message"] = "Message posted to slack successfully"
          info["return_code"] = 0
      else:
          info["payload"]["message"] = "Could not post to slack; Error code " + str(returned_code) + ". Please check your hook again"
          info["returned_code"] = 1
      print json.dumps(info)

2. Package as Docker and Test

In order to package this script into a set of instructions that can build a Docker image, we will take the following steps:

  1. Create a directory
mkdir slackimage
  1. This directory contains these files for the sake of this example
$ ls
Dockerfile slack.py  steps.sh
  1. Our Dockerfile is a simple instruction set that is:
FROM centos
ADD steps.sh /
ADD slack.py /
RUN /bin/bash steps.sh
RUN yum install -y python-requests
ENTRYPOINT ["python", "-u", "/slack.py"]
  1. Our steps.sh file is a simple set of instructions that get our container ready with all the dependencies:
yum install -y epel-release
yum install -y python-pip
  1. Now, we zip this directory using:
zip -r slackintegration.zip slackimage

3. Enter the wizard

Navigate to the Configuration tab on the Sift Security portal, followed by the Integrations sub-tab. Click on the ADD INTEGRATION button to enter the guided setup.

Step 1:

  • Provide the integration name: Let’s call it “Slack Integration”

Step 2:

  • Check the “Take User Input?” button
  • Choose “collaborate” as the “Section” and “report” as the “Section Element”

Step 3:

Fill in the following values:

  • Field Name : message
  • Field Label : Message
  • UI Element : text
  • Value: Hello world ! - from SIFT-USER-NAME

Step 4:

Fill in the following values:

  • Name: url
  • Value: <your slack webhook URL>

Step 5: Upload your zip file

Navigate to the Collaborate tab and click on any report, and you will notice “Slack Integration” under “Action”.

_images/figure-12.png

Click on “Slack Integration” and you see the following message :

_images/figure-8.png

Hit Submit and Voila! Our agent is successfully added to Sift Security.