Skip to content

Latest commit

 

History

History
206 lines (152 loc) · 6.64 KB

configure-ontology.md

File metadata and controls

206 lines (152 loc) · 6.64 KB
Raw

Configure Custom Graphistry Ontology

The Graphistry ontology is a set of mappings that guide automatic visualization and simplify data worklows. This document overviews the ontology and how to extend. For the formal format, see Graphistry's convict-format specification.

Out-of-the-box ontologies

Graphistry supports out-of-the-box ontologies of common systems:

  • SIEM Models: Much of Splunk CIM, ArcSight CEF, and ElasticSearch Common Schema
  • Vendor-specific models, such as FireEye HX/NX/iSIGHT, AWS CloudWatch, and WinLog
  • Classifying IPs as internal vs. external by RFC 1918

See below for the list of built-in types they map to.

Define custom ontologies

  1. Edit data/investigations/config/config.json as per below
  2. Restart docker service pivot: docker-compose restart pivot

Generally, you can limit the amount of work by mapping custom column names to built-in types, and thereby reuse their preconfigured settings.

Ontology types

Primary

Key ontology defines:

  • For each type, such as user:
    • Default icon: string name supported by Font Awesome 4, such as user-o
    • Default color: string hex value, such as #F00 for red
    • Default size: number, typically between 10 and 200
    • Displayed title: prioritized cascade based on entity type and available column names
  • New types: For the automatic table -> graph transform (aka hypergraph transform), the mapping from table column names to node entity types.

Secondary

Additional settings exist such specific to individual layouts and connectors

How to extend the ontology

Easiest: Ask Graphistry to do it for you!

Ideally, you can provide representative sample data that has the columns and values of interest, and if a data schema is available, that too.

Ex: For Splunk users wanting support for a new product, provide the output of

  • search index=some_product | fields * | dedup 20 event_type | head 1000
  • select all columns in the Field Selector
  • download the CSV

Add new types

For example, to create a new node type ip,

  1. Extend data/investigations/config/config.json:
{
  ...
  "ontology": {
      "icons": {
         "ip": "device",
         ...,
      },
      "colors": {
          "ip": "#F00",
          ...
      },
      "sizes": {
          "ip": 100
      }
   ...
}
  1. Restart the pivot service:

user@server.com:/var/graphistry $ docker-compose stop pivot nginx && docker-compose up -d

Override default node/edge titles

Graphistry picks the displayed title for each node and edge through the first match on the following cascade:

  • By type match: Does the element's type value have a corresponding byType binding?
  • By field match: Does the element contain a column name in byField?
  • By pointTitle , edgeTitle, if available
  • Use an element ID provided with the graph
  • Use an element ID generated by the system

Ex:

{
  ...
  "ontology": {
      "titles": {
          "byType": {
              "geo": "address",
              "user": "name"
          },
          "byField": ["src_ip", "dest_ip"]
      }
  }
}

Configure new columns / new hypergraph transforms

The existing ontology may already have all the types you want, but a new data source may have columns that need to be mapped into it.

For example, to recognize src_ip and dest_ip columns as both generating ip-type nodes:

  1. Extend data/investigations/config/config.json:
{
   ...
   "ontology": {
       "products": [
           {
               "name": "my_extension_1",
               "colTypes": {
                   "src_ip": "ip",
                    "dest_ip": "ip"
               }
           }
       ]
   }
}
  1. Restart the pivot service:
user@server.com:/var/graphistry $ docker-compose stop pivot nginx && docker-compose up -d

Built-in types

The current set of built-in types is below. Upon system start, Graphistry emits the list of Ontology types for your installed version. You can also add your own (see above).

We recommend using built-in types when possible. Each type comes with a built-in color, icon, size, and mappings from common data sources to it. This saves you work now, and as more connectors become supported and new features are added, you will automatically benefit from them in the future as well.

[
    "actor", "agent", "alert", "amazon", "amex", "arn", "asn", "availabilityzone", 
    "baidu", "bucket", 
    "cidr", "city", "cloud", "cny", "code", "container", "continent", "cookie", "count", "country", 
    "direction", "discover", "domain", "domainReputation", 
    "email", "error", "eur", "event", "extension", 
    "facebook", "file", "filePath", "filepath", "flag", "flickr", 
    "gateway", "gbp", "geo", "github", "google", "googleplus", "group", 
    "hash", "hashReputation", "host", "httpMethod", "httpmethod", 
    "id", "ils", "image", "inr", "instagram", "instance", "ip", "ipReputation", 
    "jcb", "jpy", 
    "key", "krw", 
    "language", "linkedin", "log", 
    "mac", "machine_type", "machinetype", "mastercard", "medium", "message", "money", 
    "name", "netbios", "networkinterface", "number", 
    "organization", "os", 
    "packer", "path", "payload", "paypal", "phone", "pinterest", "pod", "port", "process", "program", "protocol", 
    "qq", "quora", 
    "reddit", "role", "rub", 
    "score", "size", "skype", "slack", "snapchat", "state", "stripe", "subnet", 
    "tag", "telegram", "time", "timezone", "toolkit", "try", "tumblr", "twitch", "twitter", 
    "uri", "urifragment", "uripath", "uriquery", "url", "urlReputation", "usd", "user", "useragent", 
    "vendor", "version", "vine", "visa", "volume", "vpc", 
    "wechat", "weibo", "whatsapp", 
    "xbt", 
    "yahoo", "youtube"
]

Layouts and IPs

Layouts have additional options. The most common to modify is to flag values for being "inside" in the network map layout.

You can put any regular expression here:


  "layouts": {
    "network": {
        "ipInternalAcceptList": ['/10\.*/', '/127.0.0.1/']
    }
  },

Testing your ontology

  1. Syntax errors:

Graphistry tries to detect syntax error, and upon one, logs the error and stops. To see what is going on:

docker ps <- see if pivot is unhealthy or in a restart loop docker-compose logs pivot <- see the precise error message

  1. Satisfactory configuration

We recommend creating a Manual Data pivot. For example, to test various ip columns, use the following:

  • Query: [ {"src_ip": "10.10.0.0", "dest_ip": "10.10.0.1", "ip": "10.10.0.2"} ]
  • JQ: .
  • Nodes: src_ip, dest_ip, ip