0.0.33 release

Synapse v0.0.33 Release Notes

New Features

• #502 - Added the dir:/// handler for opening a Cortex (currently SQLite backed) by file path.
• #502 - Added a Telepath reminder API to facillitate server side statefullness on Proxy reconnect.
• #502 - Added a Cortex metadata directory configable option and helpers for CoreModules to access that directory.
• #507 - Added inet:dns:rev6 form for recording IPV56 PTR lookups.

Enhancements

• #502 - Added a reqPerm() API helper to require a user have a given permission.
• #502 - Removed old/broken session management code.

Bugs

• #170, #501 - Replaced the Python re module with regex. This addresses a unicode parsing error in the re module which prevented the correct identification of some punycode encoded FQDN values.
• #508 - Add a signal handler for SIGTERM to the Eventbus.main() function. This allows for gracefully shutting down a dmon which was started in a Docker contain. Previously, SIGTERM was not caught and caused the Python process to close ungracefully.
• #509 - Removed Python 3.7 from test matrix until synapse.async library is removed.
• #509 - Fixed a bug in formTufoByProp which allowed the formation of nodes which were valid props, but not actually forms.
• #509 - Fixed a bug in storm that prevented setting read-only properties on nodes which may not have had the read-only property present.

Documentation

• #503 - Added docstrings to inet.py, dns.py and files.py models.
• #505 - Added link to docker for the vertexproject/synapse images.
• #510 - Added docs for running PostgreSQL cortex tests manually with Docker.
• #511 - Added code style guidelines to indicate the preference of the regex module over the use of re.

0.0.32 release

Synapse v0.0.32 Release Notes

New Features

• #480 - Added a fully asynchronous push FIFO structure in order to support future Synapse built services.
• #490 - Added a make:json typecast which can be used to cast an object into a JSON string.
• #492 - Added a JSONL to messagepack tool. synapse.tools.json2mpk can be used to convert a JSONL file to a stream of messagepack's blobs.
• #496 - Added a ingest helper setGestFunc to the IngestApi mixin. This allows a function to be registered which performs data ingest without relying on a full ingest definition being created.
• #480 - Configable objects have a new method, reqConfOpts(). This method checks all configabl options; if an option has the property 'req' which evaluates to True and the value is not set on the object, a ReqConfOpt exception is thrown. This can be used to enforce an object to have specific configuration options set.

Enhancements

• #490 - Ensured Synapse was generating pretty JSON strings in places where a human may end up reading the JSON directly.
• #497 - Made axon exception logging more verbose.
• #489 - Docker images are now built in DockerCloud using a Dockerfile contained in the Synapse repository. The vertexproject/synapse image will use the python:3.6.3-slim base image moving forward, as to keep container size smaller.
• #480 - During a graceful shutdown, an atexit handler will now attempt to fini() all EventBus objects which have not been fini()'d and have had the self._fini_atexit flag set to True on them.
• #480 - AtomFile objects may now be truncated to reduce their size.

Bugs

• #487 - Removed Python 2.7 from the list of suppported Python versions in setup.py trove classifiers.
• #491 - Fixed a race condition in splicepump tests for node:created values.
• #494 - Added a minimum and maximum value for the IntType integer value to ensure it is bound within a signed 64big value. This is reflective of storage limitations of the SQLITE and PSQL storage backings. This ensures that we cannot make a node in one storage backing that cannot be moved to another storage backing because of storage-specific issues.
• #499 - The storm pivot operator was not runt-node aware; so it was unable to pivot to runt nodes. This has been fixed.
• #498 - Telepath's Proxy object was unable to successfully reconnect to a shared object if the Proxy object had event handlers registered to it. The order of operations for handling a reconnection has been changed to allow this to function properly.

Documentation

• #488 - Updated scheduler persec/loop function docstrings to clarify the return values and ability to cancel future tasks.

0.0.31 release

Synapse v0.0.31 Release Notes

New Features

• #477 - Added node:created universal TUFO property. This is set when a node is created via formNodeByProp, and enables lifting/sorting nodes by the time they are created. Existing nodes will have node:created props set on them based on the Cortex timestamp value of their node:form property. Since this requires lifting every tufo:form row in a Cortex, it is reccomended that this is first tested in a copy of any production cortexes before doing a production deployment; so any neccesary outage windows can be planned.
• #484 - The Cortex axon:url configable option now accepts a URL to a service bus. It will create an AxonCluster object if that is the case.

Enhancements

• #478 - Properties which are read-only will be able to be set on a node if that property does not exist on the node.
• #485 - Test context managers in synapse.lib.iq now properly clean up after themselves in the event of a test failure/error.
• #485 - Added setTstEnvars context manager to the SynTest class to enable running tests with specific environmental variables set.

Bugs

• #459, #478 - Refactor how nodes are created using formTufoByProp. This has the impact that nodes automatically created via the autoadds mechanism will now have any secondary properties available added to them from the process of doing data normalization.

Documentation

• #483 - Remove outdated readme examples
• #486 - Add docstrings to inet:iface properties.

0.0.30 release

Synapse v0.0.30 Release Notes

New Features

• #473 - Added it:prod:soft, it:prod:softver, it:hostsoft types and associated forms, to allow tracking software, versions of software and software installed on a given host.
• #473 - Added it:semver data type for doing type normalization of Semantic Version numbers. Added helper functions for both Semantic version parsing and generic version parsing to synapse.lib.version
• #473 - Added it:version:brute typecast to attempt parsing a version string into a normalized system value that can be used to do ordered comparison of version strings.
• #476 - Added inet:iface type and form for modeling a network interface on a particular device being bound to a particular IP, host, phone or wifi SSID. Added inet:wifi:ssid type.
• #476 - Added ps:contact type and form to act as a conglomerate of contact information for a individual.
• #476 - Added tel:mob:tac, tel:mod:imei and tel:mob:imsi types and forms for modeling cellphone related information. This includes parsing and validation of pre-2004 IMEI/IMSI numbers.
• #482 - Moved test helper functions from synapse.tests.common to synapse.lib.iq so other users of Synapse can reuse our pre-existing test helpers (SynTest, TestEnv and TstOutput).

Enhancements

• #465 - Added axon:listener, axon:tags and axon:syncopts to the AxonHost class, so these default values can be passed to Axons made by an AxonHost.
• #479 - Add test for calling the storm task() operator on a remote cortex with a local calback handler.

Bugs

• #475 - Changed PropValuType to use reqPropNorm instead of getPropNorm to enforce that the property referred to BY the type must be a modeled property.

0.0.29 release

Synapse v0.0.29 Release Notes

New Features

• #471 - The dmon tool, synapse.tools.dmon can now accept the log level via a environmental variable, SYN_DMON_LOG_LEVEL. This can be added as an environmental variable in a docker compose file using the Vertex Project Synapse Docker images to configure the logging level.

Enhancements

• #467 - Added it:exec:proc:path, it:exec:proc:src:proc and it:exec:proc:src:exe properties to the it:exec:proc form. Removed the it:exec:subproc form since it is not needed with the :src: properties on it:exec:proc.
• #467 - Removed the computer science model (compsci.py) since it was superseded by the host execution model.
• #467 - Added inet:flow:src:exe and inet:flow:dst:exe properties to inet:flow to allow modeling data between file:bytes nodes.
• #468, #469, #472 - Added pytest-xdist to the testrunner.sh script to speed up local (dev) test runs of synapse.
• #470 - Remove unnecessary docker related functionality.

Bugs

• #466, #474 - Fixed bug in non-blocking SSL link which would sometimes prevent data from being transmitted

Documentation

• #462 - Added documentation for the host execution model in infotech.py.

0.0.28 release

Synapse v0.0.28 Release Notes

New Features

• #456 - A global thread pool has been added to Synapse and a Task object convention added for executing tasks in the pool. This is in preparation of future feature support.

Enhancements

• #461 - The storm query operator refs() now also lifts nodes by prop-valu combination in order to get nodes which may refer to the source nodes. This allows identifying XREF nodes which point TO the inbound node.
• #463 - The TimeType now norms the string "now" as the current system time.
• #464 - Added a "guid" helper for Ingest to assist in making GuidType nodes without having to form strings out of variables.

0.0.27 release

Synapse v0.0.27 Release Notes

New Features

• #446, #450 - Adds the ability for GuidType nodes to normalize a list of items, in order to generate stable guids for potentially re-encounterable data. This only works when generating a property norm value (getPropNorm) and does not work for purely type normalization (getTypeNorm). Storm keyword list argument parsing can be used to generate stable GUID using the CLI, Ingest or Storm mechanisms.
• #452 - Synapse now stores the current version of the Synapse library in the Cortex blob store at the end of Cortex initialization. This was done in order to prepare for eventually enforcing required upgrade paths for data migrations or other features.
• #447 - Added the new form inet:web:postref XREF to track an inet:web:post which refers to another node.
• #447 - Added the new form inet:web:action GUID to to track an arbitrary action by an ``inet:web:acct`. The actions tracked by this are by defined by [Synapse] user convention.
• #447 - Added the new form inet:web:actref XREF to track how an inet:web:action may have interacted with another node.
• #454 - Added the inet:web:chprofile GUID to track previous values of a inet:web:acct node, representing changes to user accounts or profiles.
• #454 - Added the inet:web:post:repost property to track the concept of a inet:web:post being a copy of another post.
• #455 - Added a pair of Storm (and Cortex) Configable options to enable and set logging levels for Storm queries. These are storm:query:log:en and storm:query:log:level. This logs what the query is and what the user execution context was.
• #426 - Axon and AxonHost objects are now Configable objects with configuration definitions that are used to define their behavior.

Enhancements

• #442 - Python 2.7 support dropped from Synapse.
• #447 - Migrated all inet:net* forms to the inet:web:* space. The following is a map of the migrated forms and their corresponding new forms:

('inet:netuser', 'inet:web:acct')
('inet:netgroup', 'inet:web:group')
('inet:netmemb', 'inet:web:memb')
('inet:follows', 'inet:web:follows')
('inet:netpost', 'inet:web:post')
('inet:netfile', 'inet:web:file')
('ps:hasnetuser', 'ps:haswebacct')
('ou:hasnetuser', 'ou:haswebacct')

These forms will automatically be migrated in existing Cortexes. If XREF types were used to point to any of these forms and the cortex was not first migrated to v0.0.26, the XREF type migration will fail. It is recommended that users first upgrade to v0.0.26 prior to upgrading to v0.0.27.

• #447 - Added Storage.updateProperty() and Storage.updatePropertyValu() APIs to the Cortex storage layer for doing bulk property and property-by-value updates. These are explicitly NOT exposed in the Cortex class.
• #449 - Thinned out some components of the EventBus class for performance reasons. This did result in the removal of the synapse.eventbus.on() decorator for decorating functions to be used as event callbacks.
• #456 - Removed unused Synapse modules: synapse.hivemind, synapse.mindmeld, synapse.lib.moddef.
• #426 - Logging in tests is now controlled by the environmental variable SYN_TEST_LOG_LEVEL which, as an integer, will set the logging level used by the root logger.
• #426 - The environmental variable SYN_TEST_SKIP_LONG can be set to a non-zero integer to skip potentially long running tests. This can shave up to a minute of test execution time.
• #426 - Axons now have the in-memory cache enabled on their Cortex by default.
• #458 - The inet:web:acct:occupation property has been changed from a str:txt type to str:lwr to allow for better foldability between user-declared occupations.

Bugs

• #443 - Make the daemon return more useful error messages when an exception has occurred during execution of a remote request.
• #444, #445 - Allow an inet:srv4 type to accept an integer string as input. Also adds additional boundary checking when norming an ip:port string to ensure that irreversible inputs are not accepted.
• #453 - Cleaned up skifIfNoInternet() test helpers. They will now be allowed to fail unless the SYN_TEST_SKIP_INTERNET environmental variable is set to a non-zero integer.
• #426 - The synapse.lib.heap.Heap class was not properly responding to heap:resize events. This was remedied.
• #426 - Wrapped a .items() iterator in synapse.daemon.OnHelp with a list to prevent a RuntimeError.
• #426 - Fix the synapse.lib.service.SvcProxy.getSynSvcs() method to return a Telepath safe list instead of a dict.values() view object.
• #426 - Fix the synapse.lib.service.SvcProxy.init to strap in event handlers AFTER initializing instance variables to avoid a race condition on startup.
• #426 - AxonHost now waits before advertising itself on the bus, and properly calculates the number of axons it needs to make. This addressed an issue where the AxonHost was generating a non-deterministic number of Axons.
• #426 - The Axon’s thread to make clones for itself on a ServiceBus now waits until a remote clone is made. This addressed an issue where the Axon would make extra clones for itself.

Documentation

• #448 - Added in-model documentation for the file: model defined in files.py.
• #451 - Added user guide information for Storm lift operations, lift(), guid() and alltags().
• #426 - Docstrings in the synapse.axon module were rewritten or added when needed.

0.0.26 release

Synapse v0.0.26 Release Notes

New Features

• #438 - Added PropValu datatype to synapse. This allows a secondary (or primary) property to be modeled as a string in the form property=<repr valu>. This type also yields sub of "prop" representing the property, and "strval" or "intval" being the system normalized value of the property. This allow for node creation where a reference to another node property is needed but cannot be defined up front in the model. The additional subs allow for filtering/pivoting operations on nodes which use the PropValu type. A simple example of the string form for a PropValue is inet:ipv4=1.2.3.4. A more complex example of the string form for a PropValu is inet:passwd=oh=my=graph!
• #438 - XREF types have been updated to use the PropValu datatype instead of storing data in the property columns. This removes any extra-model data from the property columns in Cortex rows, and allows the implementation of pivotable XREF nodes. The string syntax for XREF creation was changed to be in line with the Comp datatype syntax, which looks now looks like ``. This will affect any ingests or programmatic creation of xref nodes done by users. An example of the string form of a Xref now looks like the following: (98db59098e385f0bfdec8a6a0a6118b3,"inet:passwd=oh=my=graph!"). Note that the PropValue portion of the value is quote delimited.

Enhancements

• #438 - The refs() storm operator is updated to be aware of secondary properties which are PropValu types and will pivot off of them, even if the props themselves are not forms.
• #438 - The file:imgof and file:txtref nodes no longer have glob property xref:*, instead :xref is now a PropValu and has the additional :xref:prop, :xref:intval and :xref:strval secondary props. This does require a data migration, so deployment of v0.0.26 should be tested on Cortexes which use those forms prior to production use.
• #438 - Migrated unittests to using getRamCore() helper when possible.
• #438 - Added unittest self.len() helper. Started some migration there.
• #439 - Cleaned up Socket() class implementation to be better, stronger and faster.
• #440 - Moved the syn:type, syn:form and syn:prop forms (and other items declared in DataModel) to be part of the "syn" model. These core elements are now themselves introspect-able.
• #440 - Added syn:prop:base and syn:prop:relname to syn:prop nodes, so those nodes can now be lifted by a basename or a relative property name.
• #441 - Added Python 3.7 RC1 to the test matrix. Pinned postgres test image to postgres:9.6.
• The Storm runtime now respects the storm:limit:lift configable value as the default limit when lifting nodes.

Bugs

• #440 - Changed how runtime nodes (runts) used to represent the data model in a cortex are created. These are now made from the type and property definitions based on the loaded data model; having been guided by the data model being processed. This restores things like "syn:prop:form" which was accidentally dropped from nodes.

0.0.25 release

Synapse v0.0.25 Release Notes

New Features

• #404 - Added model for inet:web:netlogon to track netuser's logging into services.
• #433 - Added inet:dns:mx, inet:dns:cname, inet:dns:soa, inet:dns:txt node types for recording different DNS responses.
• #436 - Added the ability to define triggers. Triggers are predefined actions which can react to events and trigger storm queries, in order to automate activities and actions.
• #436 - Rewrote the user authentication system to be more integrated with Cortex (and used within the trigger subsystem). This allows adding user roles and permissions to add, delete, or update nodes.

Enhancements

• #404 - Added inet:whois:nsrec comp nodes to track nameserver's associated with whois record. This removed the inet:whois:rec:ns* properties and migrates existing props into the new nodes.

Bugs

• #437 - Fix the tag interval filtering in the Storm query system.

Documentation

• #435 - Rewrote docstrings for synapse.lib.queue subsystem.

0.0.24 release

Synapse v0.0.24 Release Notes

Enhancements

• #430 - CortexTest class was split into CortexBaseTest and CortexTest. CortexBaseTest is used for running the
  basic_core_expectations tests with the different storage backing; and CortextTest is used for more generic API tests.
  This allows running the basic test suite against the different storage types without running the entire test suite, in
  order to isolate possible storage related issues.
• #434 - Migrated from using nosetestes to using py.test as the Synapse testrunner for CI.

Bugs

• #422 - Fixed the order of autoadds being added to the DB during the addTufoEvents / addTufoEvent Cortex API. This
  could manifest itself as tufos made by these events to not contain their properties.
• #428 - Fix a issue with the LMDB cortex where parameters passed to pylmdb were not bool types (as that library
  expected), causing the parameters to be ignored in effect.

Documentation

• #422 - The inet:dns model docstrings were updated to be more comprehensive.