0.0.53 release

Synapse 0.0.53 Release Notes

Enhancements

• #734 - Add a smaller default LMDB map size for lmdb databases which do not need to be large.

Bugs

• #733 - Fix cryotank indexing test races.
• #735 - Fix a race which can occur during synapse.lib.net.Plex teardown.
• #736 - Fix neuron test race conditions.

0.0.52 release

Synapse 0.0.52 Release Notes

Summary

OSX Support is preliminary restored in this release. Windows is still not supported, however we are aware of users successfully using Synapse under the Windows Subsystem for Linux (WSL) available in Windows 10.

New Features

• #697, #722 - The CryoTank now supports selective field indexing using Synapse type normalization. This allows for records to be retrieved based on indexed values.
  A remote API has been added for this and is available via the CryoClient for managing indexing on tanks managed by a CryoCell.
• #723 - Add a get:opers() command to Storm which allows the introspection of which registered storm operators are available to a Cortex.
• #725 - Added aCortex RBAC system for controlling node creation, deletion, property setting and tagging. Currently, this is applicable to users using SSL to share out Cortexes, since user authentication is done via SSL certificate common name. This does require explicit configuration to use. See synapse.lib.iq.SynTest.getSslCore() for a configuration example. The RBAC interface can also be configured via cmdr.
• #727 - Move from using select.Epoll() to selectors.DefaultSelector() for the global plex defined in synapse.lib.net. This allows restoring OS X use of Synapse.

Enhancements

• #724 - Add cell Ctor type when doing cell registration. This allows a Neuron and CellPool to have awareness of what types of objects have registered with it.
• #726 - Refactored CryoCell and CryoClient.

Bugs

• #728 - Fix a race condition in cryotank index testing.
• #535, #730 - Fix Cmdr CLI loop for OSX. Thanks @MichaelSquires for the bug report.
• #731, #732 - Fix unittests so that they run on OSX.

0.0.51 release

Synapse v0.0.51 Release Notes

Enhancements

• #719 - When doing a type norm for time, include the valu being normed in the BadTypeValu.
• #720 - When a node is deleted using Cortex.delTufo(), the node form, primary property, and current user scope are logged at the info level. This also applies to nodes deleted via Storm and splices.
• #721 - Add test showing that a splice contains the current user.

0.0.50 release

Synapse v0.0.50 Release Notes

New Features

• #714 - inet:addr type was modified to represent a URL like format to represent tcp/udp/icmp clients and servers. The inet:client and inet:server types were added, which are subs of the inet:addr type.
• #714 - Added inet:servfile comp form. This is the intersection of an inet:server and a file:bytes node. It a file available from a given server over an arbitrary protocol.
• #714 - Added inet:download guid form to represent an instance of a inet:client downloading an arbitrary file:bytes from a inet:server at some point in time.
• #717 - Storm operators may now send status messages along with their results. These messages are displayed after any nodes are printed. This is a backwards-compatible change. There are no built-in Storm operators which use this currently.

Enhancements

• #709, #711, #712 - Move to CircleCi as a CI testrunner. Remove all Drone / Travis CI code.
• #715 - Remove the automatic EventBus event propagation from a Cortex to a loaded CoreModule. This has an unnecessary performance impact and is not needed for a CoreModule to subscribe to events from the Cortex. A CoreModule implementation has a reference to the Cortex directly and can listen for the events as needed.
• #717 - The SynTest.getDmonCore() API now sets the local scope variable syn:core to refer to the actual Cortex object which backs the Proxy object yielded by the API. This allows test writers using that API to access the underlying Cortex to perform function calls which may not be possible to execute over the Proxy.
• #714 - Update the it:av:sig form to be the intersection of a it:prod:soft and a str:lwr. Previously, this was a sepr type containing an ou:alias but that does not account for an organization having multiple products. The it:av:filehit also was changed from a sepr to a comp type since signature names are not safe to use in seprs as sepr character may appear in the names. There are no migrations in place for these modeling changes. Please reach out to Vertex Project on Slack if you have data using these forms that requires migration.
• #714 - Update the synapse.models.inet.ipv6norm() (used for normalizing IPv6 types) now also returns a ipv4 sub if the normalized address is in the IPv4 address space.
• #718 - Remove the Storm vartree implementation. It was incomplete and will be replaced in the future with an implicit pivot syntax which will achieve the same effect in a cleaner fashion.

Bugs

• #710 - Fix a test which was leaking a CellPool object which was attempting a reconnect loop which would never reconnect.
• #716 - Normalize tags when calling Cortex.getTufosByTag or doing tag filtering in Storm.

0.0.49 release

Synapse v0.0.49 Release Notes

V0.0.49 is primarily a bugfix release to address regressions and issues from v0.0.47.

Enhancements

• #706 - The Cortex.formNodeByProp() API will now call setTufoProps() on a node if it already exists and properties are passed into formNodeByProp. This allows properties to be added in a “last caller wins” scenario.
• #706 - The node:add splice now accepts a tags value which contains a list of tags to add to the node created by the splice.

Bugs

• #705 - Fix an issue in Configable\’s @confdef decorator helper for identifying config options in classes. This method was previously skipping Telepath proxy objects during introspection; and a regression was introduced which didn’t properly skip non-callable object methods. This caused an issue when doing introspection on objects which had Telepath Proxy objects as the value of class attributes. This regression has been fixed by properly skipping non-callable class members.
• #706 - Splice changes introduced in v0.0.47 were improperly wrapping messages. This meant that the Cortex.splice() and Cortex.splices() APIs required splice messages as though they came off of the EventBus. This required sending duplicate, redundant data over the network during remote calls. This splice reactors now expect unpacked splices. This does not affect splices written to splice files or splices converted to the new format, since those splices were already unwrapped.
• #707 - The Telepath socket multiplexor (synapse.lib.socket.Plex) had an issue added where xlist sockets may not have been properly culled from the list of Sockets maintained by the plex. The xlist sockets were the same as the rlist sockets, so this has been simplified. In addition, an identity check for fileno was changed to a equality check.

Documentation

• #706 - Examples of all differentsplice message forms have been added in a Cortex test test_cortex_splices. This can be used as an immediate reference to people doing manual splice creation.
• #708 - Added synapse.cryotank.CryoCell to the automatically generated Configable docs.

0.0.48 release

Synapse v0.0.48 Release Notes

New Features

• #698 - Add source, seen, record and recref forms to allow for tracking data sources and references in a Cortex hypergraph.
• #698 - Add ou:meet, ou:meet:attendee, ou:conference, and ou:conference:attendee forms to allow for tracking meetings and conferences, as well as their attendees.

Enhancements

• #702 - synapse.lib.datapath now has a DataPath() class which is used to represent the steps in a given path. This allows pre-computation of a given datapath to reduce overhead when the same path may need to be parsed multiple times.

Bugs

• #703 - Ensure that the AxonCell was sending the correct marshalled exceptions.
• #704 - Change the auth value of the Cortex configuration for a cell:pool to be a Base64 encoded version of the msgpack encoded auth dictionary. This allows for configuring the cell:pool via a JSON Daemon configuration file.

0.0.47 release

Synapse v0.0.47 Release Notes

Summary

There are a sizeable amount of changes included in v0.0.47. A few items are highlighted here; please see the sections after the summary for a full list of new features and enhancements.

Notable New features

• A new data storage object, the CryoTank, has been introduced. This utilizes LMDB to perform fast storage and retrieval of structured records. This can be used to quickly store and retrieve large feeds of data.
• A new service architecture has been implemented in Synapse. This uses an asynchronous, generator based approach to networking and remote calling. This has been utilized to reimplement the Axon storage (now AxonCell), as well as implementing the networking protocol for the CryoTank, the CryoCell. Detailed documentation for this service architecture will be included in future Synapse release.
• A new LMDB key-value store helper has been added, the KvStor. This is used to implement persist dictionary and set objects, which are used as helpers to the Cell. Previously, a service could use their own Cortex as a simple KV store, but that was potentially heavy thing to do. This alternative is considerably lighter for these simple use cases.
• Storm now supports the use of subquery filters. An example query is the following: inet:dns:a -{ :ipv4 -> inet:ipv4 +cc=us } which will lift all inet:dns:a nodes, then filter nodes whose IPv4 address has a us country code. Detailed documentation for this will be added in a future Synapse release.

Backwards Incompatible Changes

• A global socket multiplexor, implemented using EPOLL, has been added to Synapse. This change means that Synapse will only work on Linux at the moment. The Vertex Project does plan on restoring Windows and OSX support in a future release.
• The Cortex Splice format has changed, in order to support better filtration. This requires that splice producers and splice consumers update simultaneously to v0.0.47. If users have existing splice logs that need to be updated, a helper function has been written to to convert an existing splice log file to the new format. See the notes below for additional information.
• The following Synapse components have been removed or altered significantly:
  • synapse.axon has been rewritten substantially. In addition, it no longer imports synapse.lib.hashset using a star import.
  • synapse.lib.persist has been removed. It is no longer used in core Synapse code.
  • synapse.lib.heap has been removed. It contained an implementation error and is no longer used in core Synapse code.
  • synapse.lib.atomfile has been changed to currently only support Linux.

Known Issues

• #700 - The axon:upload handler on the AxonCell can allow bytes to be stored twice in the same BlobStor. This may result in a BlobStor having bytes which the AxonCell does not have direct knowledge of. No data is lost here, but extra disk space may be consumed.

New Features

• #637, #650, #695 - Added synapse.lib.cell. This contains the Cell class and related helper classes. The Cell is the base class for a microservices architecture. Similar to a CoreModule, the Cell is designed to be subclassed and have some functions overridden. See Cell docstrings for additional notes.
• #637, #695 - Added synapse.neuron module. This contains the Neuron() class which is used as a service directory for Synapse Cell based services. The Neuron is responsible for doing service provisioning and service name resolution. Additional documentation related to Neuron, Cell architectures will be included in a future Synapse release.
• #650, #679 - The synapse.axon.Axon implementation was removed, and replaced by a new AxonCell and BlobCell implementations. This decouples the Axon idea from the Cortex implementation, and makes the AxonCell solely responsible for storing files and doing retrieval by SHA256 hash. The AxonCell acts as a indexing master, while multiple BlobCells need to be used in order to store files. The AxonClient helper is available to interact with a AxonCell, while the BlobClient is available to interact with the BlocCell. The use of this new storage implementation does require the deployment of a Neuron, since these are all Cell based services. Additional documentation related to Axon architecture will be included in a future Synapse release.
• #637, #654, #655, #674 - Added a new module, synapse.cryotank. It contains a CryoTank() class, which is used for storing structured data (anything that can be msgpacked) into a LMDB database. It also contains a CryoCell() class for creation, management, deletion of CryoTank objects, as well as for putting data into remote CryoTank’s. It has a CryoClient() helper object which can be used to interact with the remote CryoCell.
• #648, #649, #672, #674, #675 - Added synapse.tools.cryo.cat to look at data from remote CryoTanks, dump that data to disk in different formats, or add data to a remote CryoTank when reading data from stdin.
• #681 - Add cellpool:conf config option to the Cortex. This is a dictionary which takes three value, a auth blob for talking to a Neuron, as well as the host and port used to connect to the Neuron. The configuration of a cellpool:conf, and subsequentially axon:name, is required in order to allow a Cortex to save bytes to a AxonCell using the formNodeByBytes() and formNodeByFd() APIs.
• #637 - Added RetnWait() to the synapse.lib.thread. This can (and should) be used as a context manager, which can emulate a synchronous callback occuring, while waiting for a local thread.event to be set.
• #650 - Added synapse.lib.lmdb. This contains several LMDB related classes for future use with new Cortex storage layers. The APIs for these objects should not be considered stable for third party use at this time.
• #637, #642, #658 - Added a new module, synapse.lib.net, to handle asynchronous networking code. This includes a new EPoll based multiplexer for Socket IO. An instance of this plex is used as part of a global plex (located in synapse.glob). This change currently makes Synapse only compatible with Linux operating systems. Eventually, we’ll re-add Windows and OSX support in a future release.
• #687 - Added ps:persona and ps:persona:has forms to represent the idea of a persona.
• #637 - Added a new way to marshall exceptions, synapse.common.getexcfo() which will be the preferred way to marshall exceptions moving forward for Synapse code.
• #636, #638, #676, #678, #689 - Added the synapse.lib.crypto module. This contains several new modules. synapse.lib.crypto.tinfoil contains a simple AESGCM encryption/decryption class called TinFoilHat. synapse.lib.crypto.ecc contains helpers for doing ECC key pair generation and doing ECDHE key exchange. synapse.lib.crypto.vault contains a Vault() class. The Vault() can be used to store key pairs and do key signing locally.
• #634 - The Storm CLI command jointags() was split into jointags() and pivottags() which behave similarly to the pivot() and refs() commands, respectively. The old default behavior for jointags() was moved to pivottags(), and the current jointags() behavior now reflects the old jointags(keep_nodes=1) behavior.
• #628, #659 - Added synapse.lib.kv. This module contains the KvStor() class; which implements a LMDB backed kv-store. An instance of the KvStor() object can be used to get KvLook(), KvDict() and KvSet() objects. The KvLook can be used to do key=valu type storage, which is directly backed by the LMDB database. The KvDict can be used for key=valu storage, but it also keeps a in-memory dictionary of the contents to avoid database lookups. The KvSet can be treated similarly to a set() object, but addition and removal is backed by the LMDB database.
• #691 - Added a initial HTTP model for tracking to the inet model. Added inet:http:request, inet:http:response, inet:http:header, inet:http:param, inet:http:reqhead, and inet:http:reqparam forms.
• #606 - Added synapse.lib.membrane and Membrane() class. The Membrane object can be used as a filter on the events fired by a EventBus to determine whether or not they are sent to a function provided to the Membrane() object.
• #606 - Added a membranes config option to the Cortex which applies rules to splice events and puts splices into a named Fifo for later consumption.

Enhancements

• #667 - Added subquery based filtration for Storm. Subqueries take all of the query nodes, apply either a positive or negative a filter to those nodes, and then add matching nodes back to the query results.
• #685 - Migrated ou:has* forms to a generic ou:org:has Xref form. Add seen:min and seen:max secondary properties.
• #650 - Added genraises() API to synapse.lib.iq.SynTest. This mimics the raises() API, but expects the function to be a generator. This consumes the generator in a bound local and expects that to raise.
• #650 - Added getAxonCore() API to synapse.lib.iq.SynTest to get a TstEnv instance which has a Axon, Cortex, Blob, Neuron, Daemon and clients configured already.
• #650 - Added synapse.common.ehex()/synapse.common.uhex() wraps around binascii.hexlify() and binascii.unhexlify() APIs.
• #650 - Added synapse.common.buid() to generate a 32 byte hash. Unlike the guid() API, this returns bytes. It can take a msgpackable object to generate a stable buid.
• #650 - Added synapse.common.spin() to c...

0.0.46 release

v0.0.46 Release Notes

Bugs

• Pull in version bounds for third-party packages from the master branch. This addresses an immediate issue where Synapse fails to work with a pypi installation and Tornado 5.0.0 is installed.

0.0.45 release

Synapse v0.0.45 Release Notes

New Features

• #661 - Added the inet:group form and type. This is analogous to the inet:user form for tracking a given group name.

Enhancements

• #666 - Add ps:person:name:en and various name:en subprops props to distinguish between localized and English versions of names.
• #661 - The inet:web:group form was made more robust. The second field of the sepr type was renamed to :id to represent a site specific identifier, freeing the :name property up to represent a descriptive name of the group. :name:en was added to provide for a English version of the name. :signup, :signup:ipv4 and signup:ipv6 props were added to track when groups were created. :loc and :latlong props were added to allow for some location based tracking. :seen:min and :seen:max secondary properties were added to track when a group was active.
• #662 - The python msgpack library was version locked to 0.5.1.
• #663 - Added :name:en and :realname:en secondary props to inet:web:acct for tracking English specific versions of a name.

Bugs

• #624 - The ps:person was missing the :name:middle field. This has been added.
• #660 - The ou:member form was broken. This has been changed to be a Comp type and had the secondary property :end added to it.

0.0.44 release

Synapse v0.0.44 Release Notes

Notices related to v0.0.44

The minimum version of msgpack used by Synapse has been upgraded to be at least 0.5.0. This is complicated by the fact that the msgpack-python package was renamed to msgpack by its maintainers. Installation of Synapse from PyPi using the sdist and wheel packages should upgrade msgpack in a working state, and docker image users should not be affected by this change. Users which directly use a checkout of the Synapse Github repository should uninstall msgpack-python and then install msgpack. This can be done with the following commands (you may need to adjust them for your installation; accounting for aliases or the use of python3, etc):

• python -m pip uninstall msgpack-python
• python -m pip install msgpack

The synapse.lib.heap.Heapfile and synapse.axon.Axon structures are planned to undergo a significant rewrite, possibly landing as early as v0.0.45. These rewrites will not be backwards compatible, in order to account for design issues in these structures. If you are using these classes for doing data storage, please reach out to us via Slack as soon as possible (see https://github.com/vertexproject/synapse/blob/master/README.rst for a link to join our Synapse Slack chatroom).

New Features

• #592 - Added GUID type it:auth:passwdhash to store password hashes without requiring the plaintext password.
• #604 - Added synapse.lib.atomic.Counter() class as a thread safe counter object.
• #595 - Added the rcode secondary property to the inet:dns:look form to allow encoding the DNS response code for a given lookup.
• #595 - The it:dev:regkey type is now a lowercased string. Existing it:dev:regkey and it:dev:regval:key values will be automatically lowercased as a model migration.
• #607 - Added Axon.getAxonStatus() and AxonHost.getAxonHostStatus() APIs to get runtime information about Axons.

Enhancements

• #591 - Removed the Cortex CLI command nextseq in favor of using the Storm macro command nexttag().
• #591 - Add Storm test coverage.
• #593 - Added puts() API to the synapse.lib.fifo.Fifo object to the Fifo structure to allow for bulk object adding.
• #596 - Added Axon tests to show a clone sync operation restarting after a persist offset file is removed.
• #601 - Ensure the synapse.lib.heap.Heap file storage only grows upward and cannot be resized down by replaying a resize event.
• #602 - Set backoff / backoff-retry values for the Drone ci configuration for the git plugin.
• #603 - The synapse.lib.tags.tufoHasTag() and synapse.lib.tufo.tagged() APIs were equivalent functions. tufoHasTag has been removed.
• #598, #609 - Updated msgpack-python>=0.4.8 to msgpack=>0.5.1 as a dependency.
• #598 - Updated synaspe.lib.msgpack.Unpk msgpack unpacker helper to use the new tell() API introduced in msgpack 0.5.0.
• #598 - Added a future-proofing msgpack test to ensure we break when the msgpack python API eventually changes to assuming a strict utf8 compliance and can then versionlock our use of msgpack.
• #605 - Added simple integrity checking to the synapse.lib.heap.HeapFile structure.

Bugs

• #591 - synapse.lib.cache.KeyCache behavior updated to no longer store cache valus if they are None. If None was cached, that value was unable to be updated later.
• #594, #597 - Fix Ingest user docs which were using outdated forms. Thanks @tn3rt for the bug report!
• #599 - Fix AxonMixin.eatfd() and AxonMixin.eatbytes() to allow file descriptors and bytes which have zero bytes to be consumed.

Documentation

• #605 - Rewrote API docstrings for synapse.lib.heap.HeapFile.