The Graphistry ontology is a set of mappings that guide automatic visualization and simplify data worklows. This document overviews the ontology and how to extend. For the formal format, see Graphistry's convict-format specification.
Graphistry supports out-of-the-box ontologies of common systems:
- SIEM Models: Much of Splunk CIM, ArcSight CEF, and ElasticSearch Common Schema
- Vendor-specific models, such as FireEye HX/NX/iSIGHT, AWS CloudWatch, and WinLog
- Classifying IPs as internal vs. external by RFC 1918
See below for the list of built-in types they map to.
- Edit
data/investigations/config/config.json
as per below - Restart docker service
pivot
:docker-compose restart pivot
Generally, you can limit the amount of work by mapping custom column names to built-in types, and thereby reuse their preconfigured settings.
Key ontology
defines:
- For each type, such as
user
:- Default icon: string name supported by Font Awesome 4, such as user-o
- Default color: string hex value, such as
#F00
for red - Default size: number, typically between 10 and 200
- Displayed title: prioritized cascade based on entity type and available column names
- New types: For the automatic table -> graph transform (aka hypergraph transform), the mapping from table column names to node entity types.
Additional settings exist such specific to individual layouts and connectors
Ideally, you can provide representative sample data that has the columns and values of interest, and if a data schema is available, that too.
Ex: For Splunk users wanting support for a new product, provide the output of
search index=some_product | fields * | dedup 20 event_type | head 1000
- select all columns in the Field Selector
- download the CSV
For example, to create a new node type ip
,
- Extend
data/investigations/config/config.json
:
{
...
"ontology": {
"icons": {
"ip": "device",
...,
},
"colors": {
"ip": "#F00",
...
},
"sizes": {
"ip": 100
}
...
}
- Restart the pivot service:
user@server.com:/var/graphistry $ docker-compose stop pivot nginx && docker-compose up -d
Graphistry picks the displayed title for each node and edge through the first match on the following cascade:
- By type match: Does the element's
type
value have a correspondingbyType
binding? - By field match: Does the element contain a column name in
byField
? - By
pointTitle
,edgeTitle
, if available - Use an element ID provided with the graph
- Use an element ID generated by the system
Ex:
{
...
"ontology": {
"titles": {
"byType": {
"geo": "address",
"user": "name"
},
"byField": ["src_ip", "dest_ip"]
}
}
}
The existing ontology may already have all the types you want, but a new data source may have columns that need to be mapped into it.
For example, to recognize src_ip
and dest_ip
columns as both generating ip
-type nodes:
- Extend
data/investigations/config/config.json
:
{
...
"ontology": {
"products": [
{
"name": "my_extension_1",
"colTypes": {
"src_ip": "ip",
"dest_ip": "ip"
}
}
]
}
}
- Restart the pivot service:
user@server.com:/var/graphistry $ docker-compose stop pivot nginx && docker-compose up -d
The current set of built-in types is below. Upon system start, Graphistry emits the list of Ontology types
for your installed version. You can also add your own (see above).
We recommend using built-in types when possible. Each type comes with a built-in color, icon, size, and mappings from common data sources to it. This saves you work now, and as more connectors become supported and new features are added, you will automatically benefit from them in the future as well.
[
"actor", "agent", "alert", "amazon", "amex", "arn", "asn", "availabilityzone",
"baidu", "bucket",
"cidr", "city", "cloud", "cny", "code", "container", "continent", "cookie", "count", "country",
"direction", "discover", "domain", "domainReputation",
"email", "error", "eur", "event", "extension",
"facebook", "file", "filePath", "filepath", "flag", "flickr",
"gateway", "gbp", "geo", "github", "google", "googleplus", "group",
"hash", "hashReputation", "host", "httpMethod", "httpmethod",
"id", "ils", "image", "inr", "instagram", "instance", "ip", "ipReputation",
"jcb", "jpy",
"key", "krw",
"language", "linkedin", "log",
"mac", "machine_type", "machinetype", "mastercard", "medium", "message", "money",
"name", "netbios", "networkinterface", "number",
"organization", "os",
"packer", "path", "payload", "paypal", "phone", "pinterest", "pod", "port", "process", "program", "protocol",
"qq", "quora",
"reddit", "role", "rub",
"score", "size", "skype", "slack", "snapchat", "state", "stripe", "subnet",
"tag", "telegram", "time", "timezone", "toolkit", "try", "tumblr", "twitch", "twitter",
"uri", "urifragment", "uripath", "uriquery", "url", "urlReputation", "usd", "user", "useragent",
"vendor", "version", "vine", "visa", "volume", "vpc",
"wechat", "weibo", "whatsapp",
"xbt",
"yahoo", "youtube"
]
Layouts have additional options. The most common to modify is to flag values for being "inside" in the network map layout.
You can put any regular expression here:
"layouts": {
"network": {
"ipInternalAcceptList": ['/10\.*/', '/127.0.0.1/']
}
},
- Syntax errors:
Graphistry tries to detect syntax error, and upon one, logs the error and stops. To see what is going on:
docker ps
<- see if pivot
is unhealthy or in a restart loop
docker-compose logs pivot
<- see the precise error message
- Satisfactory configuration
We recommend creating a Manual Data pivot. For example, to test various ip
columns, use the following:
- Query:
[ {"src_ip": "10.10.0.0", "dest_ip": "10.10.0.1", "ip": "10.10.0.2"} ]
- JQ:
.
- Nodes:
src_ip
,dest_ip
,ip