0.0.43 release

Synapse v0.0.43 Release Notes

New Features

• #590 - Added synapse.lib.remcycle.fetch() API which allows retrieving a URL using Tornado and executing a callback to process the response.

Enhancements

• #589 - Added the ability to set a tags directive in a Ingest forms definition; which allows setting per-form tags without adding the tags to the current or child scopes.

0.0.42 release

Synapse v0.0.42 Release Notes

New Features

• #588 - Added a RFC2822 address parser and type/form, inet:rfc2822:addr. This normalizes and parses string identifiers and attempts to extract email addresses as a secondary property.

Bugs

• #587 - Make synapse.lib.msgpack helper functions resilient to unmatched unicode surrogate characters. This also affects the synapse.lib.socket.Socket and synapse.lib.persist.Dir msgpack unpackers. This is done by passing unicode_errors='surrogatepass' to the msgpack pack/unpack functions. This makes synapse more resilient to malformed string data which may be encountered in the real world.

Documentation

• #587 - Removed synapse.statemach. It was previously used for providing object persistence at an API level but has been unused within Synapse and was generally not a safe tool to use for third party use.

0.0.41 release

Synapse v0.0.41 Release Notes

New Features

• #574 - Added EventBus.onWith() API. This is a context manager which acts like .on(), but the callback is removed when the context manager is exited.
• #575 - Added synapse.lib.iq.CmdGenerator() class to allow testing CLI command loops using unittest mock.
• #577 - Added synapse.lib.certdir.genClientCert() API to allow creation of a PKCS12 certificate bundle for a user certificate, private key and CA cert.
• #577 - Added a --p12 option to the easycert tool. This allows a user to bundle their certificate, private key and CA cert into a PKC12 formatted file.
• #578 - Added the Storm macro syntax <- to represent a join() operation.

Enhancements

• #576 - The Daemon now fini's EventBus() objects in LIFO order when it is fini'd. In other words, objects created in a dmon configuration are torn down in reverse order that they are created.
• #576, #581 - Increased test coverage for CLI related tests.
• #577 - Updated tests for synapse.lib.certdir to ensure that the certificates made by certdir were correct.
• #577 - Updated tests for synapse.tools.easycert to ensure that the certificates made by easycert were correct.
• #578 - Updated the storm join() operator syntax to behave exactly like the pivot() operator with respect to source and destination properties. join() still is an additive operator which does not consume the source nodes.
• #583 - synapse.lib.remcycle.Hypnos now registers and persists ingest definitions as syn:ingest nodes in its Cortex.

Bugs

• #579 - Fix a reference to onCtx (the original name for the EventBus.onWith() function).
• #584 - Cortex Storage backings had different behaviors when limit=0 was passed to functions which joined rows together to make tufos. This has been fixed, so that a limit=0 API parameter will return 0 rows.

Documentation

• #573 - Added telepath docstrings for evalurl().
• #577 - Rewrote API documentation for synapse.lib.certdir.
• #586 - Updated the Storm join() documentation to reflect changes to its arguments.

0.0.40 release

Synapse v0.0.40 Release Notes

New Features

• #564 - Added a int:2:str10 typecast to convert a integer value to the base10 string value.
• #569 - The Telepath Proxy object now fires the local event tele:sock:runsockfini when the Proxy is attempting to reconnect to its remote Daemon. This can be hooked by implementers to detect the loss of an existing connection to a remote Daemon.
• #572 - The synapse.axon.AxonCluster helper object is now a subclass of EventBus. This will now automatically close its corresponding SvcProx instance on its own .fini().
• #572 - Added the synapse.lib.iq.objhierarchy() function to extract a type hierarchy from nested tufo-style objects. This can be a useful for debugging.

Enhancements

• #565 - Increase Tornado related test timeouts from 5 to 30 seconds.
• #570 - Additional debug logging added to synapse.axon.Axon actions.
• #572 - Additional debug logging added to synapse.lib.persist.Dir actions.

Bugs

• #561 - When a telepath Proxy object is made, by default it will create its own socket Plex object. This Plex is now fini’d when the proxy is fini’d.
• #568 - Add synapse.glob to synapse.lib.socket imports to fix a missing reference.
• #567- The synapse.lib.persist.Dir.items() method yielded offsets which could switch from being relative offsets to being absolute offsets. This has been changed to always yield absolute offsets.
• #570 - The synapse.axon.Axon._fireAxonClone thread now uses the tele:sock:runsockfini event to detect that a disconnect has occurred for its remote Proxy, and uses that to break out of its innermost synapse.lib.persist.Dir.items() loop. The previous behavior would fail to reconnect if part of an Axon cluster went down and then came back up, as the Proxy contained potentially invalid link information instead of getting an updated link tufo from the Axon’s ServiceBus.
• #571 - Fini Axon threads after calling self.syncdir.fini() to ensure that any threads which are in the real time event pump from the syncdir can safely close their resources before attempting to stop any threads.

Documentation

• #572 - Miscellaneous API docstring improvements.

0.0.39 release

Synapse v0.0.39 Release Notes

New Features

• #553 - Synapse Docker images will now have -debug images available as well for all master and tagged builds. This image is built off of the vertexproject/pydebug:3.6-slim image. This image has full GDB support for Python enabled. See https://github.com/vertexproject/pydebug for more information.

Enhancements

• #551 - Make 4096 bit RSA keys by default when using the synaspse.tools.easycert tool. Thanks @galyac!
• #550 - Add additional debug logging to Synapse when starting up a Cortex, executing remote methods via Telepath and around SSL errors.
• #554 - Validate the Storm tree() recurlim argument is greater than or equal to zero.
• #557 - inet:dns:look:tcp4:ipv4 and inet:dns:look:tcp4:udp4 sub properties of port and ipv4 have been added, so a DNS response can be recorded from a server even if the server source port is unknown.
• #559 - Add a limit parameter to the Storm totags() operator.
• #560 - Add unit test for synapse.lib.dark.

Bugs

• #555 - Set the x509 extension number of certificates made by synaspse.tools.easycert to version 3 (0x02). This allows Chrome to validate the certificates starting in Chrome 63.
• #558 - When an exception is raised during the synapse.lib.socket.Plex._plexMainLoop call to select.select(), perform a check to see if any sockets are fini'd or have fileno() == -1 and perform a cleanup operation on that socket. This can reduce a race condition where a bad socket can get wedged and prevent the multiplexor from working correctly.
• #558 - Change the synapse.link.ssl.Socket.send() implementation to call self.sock.send() itself instead of calling through the default Socket.send() method. This allows us to properly catch SSLWantReadError and SSLWantWriteError exceptions; instead of allowing teh default method to catch them as OSError exceptions and fini the socket. The previous behavior lead to SSL Sockets being unstable.
• #556 - Fix the Storm filter function for +range() so that filter operator works properly.
• #562 - Fix the Storm filter function for +in() so that filter operator works properly.

Documentation

• #552 - Add documentation for the Storm operators to the user guide: pivot(), join(), refs(), fromtags(), totags(), jointags(), and tree().

0.0.38 release

Synapse v0.0.38 Release Notes

New Features

• #545 - Added storm macro function get:tasks and an API for introspecting tasks which have been registered on a Cortex.

Enhancements

• #544 - Added new fields (url and whois:fqdn) to whois:contact.
• #547 - Enabled pyup monitoring for Synapse release notes and added config file to disable pyup update checks.
• #549 - Removed cryptography from setup.py.

Documentation

• #548 - Added CHANGELOG.md to maintain release notes within the repository.

0.0.37 release

Synapse v0.0.37 Release Notes

New Features

• #542 - The Daemon now automatically calls item.fini() for items which are made from a Daemon configuration which are EventBus objects when the Daemon is fini()'d. This allows the Daemon to tear down all instances of eventbus objects which it makes without having to share the object and set the onfini option when configuring the share.

Enhancements

• #541 - Added exe, proc and host secondary properties to the inet:dns:look format to allow capturing a DNS lookup which may have originated from a file, process or a host.
• #540 - When the socket multiplexer does an accept() call, the remote ip and port are logged at the debug (logging.DEBUG) log level.

Bugs

• #58, 537 - Fixed IPv6 type norm() operations for OSX by using the ipaddress library instead of the s_socket.inet_ntop() function. Thanks @MichaelSquires !
• #543 - Recent changes to pytest (included in the base image used for doing CI testing) changed how logging is performed. This disables those pytest changes.

0.0.36 release

Synapse v0.0.36 Release Notes

New Features

• #529 - Synapse Docker container vertexproject/synapse is now built off of a base container, vertexproject/synaspe-base-image:py36. This container is hosted from https://github.com/vertexproject/synapse-base-image and is also used for CI testing. The synapse-base-image contains all of the dependencies required for Synapse, as well as having software updated via apt-get. The base container also has :py35 and :py34 tags available as well.
• #523 - Added Cortex.extCoreFifo() to put of a list of items in a named Cortex FIFO.
• #523 - Added DataModel.addPropTypeHook() to allow a DataModel user to fire a function whenever a type is used to define a property. This can be used to define callbacks by a CoreModule.
• #523, #538 - Added synapse.lib.db to handle pooled connections to databases. Added synapse.lib.sqlite to handle SQLite specific DB optimizations.
• #523 - Added synapse.lib.gis to handle geospatial computations.
• #523 - Added synaspe.lib.iq.TestSteps helper. This allows for for interlocking events for multithreaded tests.
• #523 - Multiple improvements to CoreModule’s. They may now get a unique _mod_iden property by an implementor. This value can be retrieved with the CoreModule.getModIden() API. Added getModProp() and setModProp() APIs so that the CoreModule can store data in the attached Cortex object. Added a finiCoreModule() API which is automatically registered as a fini function for the CoreModule. CoreModule implementors can override this API in order to have resources torn down automatically.
• #523 - Added synapse.lib.revision module to provide helpers for doing revision path enforcement.
• #523 - Added syn:alias node types to allow for a global GUID alias for a given iden.
• #523 - Added synapse.models.geospace.LatLongType and synapse.models.geospace.DistType for normalizing Latitude/longitude data and distances.
• #523 - Added geo:nloc form to allow tracking the physical location of a given node over time.
• #539 - Added inet:wifi:ap node type to allow the intersection of a SSID and a BSSID value.
• #539 - Added tel:mob:imid form to represent the knowledge of an IMEI and IMSI together.
• #539 - Added tel:mob:imsiphone node type to represent the knowledge of an IMEI and a telephone together.

Enhancements

• #528 - When a property value is included in the Storm stats() operator, that value is now normed using getPropNorm().
• #529, #532 - The SvcProxy now refires syn:svc:init and syn:svc:fini events, so users of the SvcProxy may now react to those events to know that a service has been added or removed from the SvcBus.
• #534 - Log messages for exceptions which occur on the EventBus which cause an exception in dist() function now include the repr of the EventBus object so it is clear what type of object had the error, and the mesg itself.
• #534 - traceback.print_exc() calls have been replaced by `logger.exception()`` calls so traceback information is directed through logging mechanisms, instead of being printed to stdout.
• #534 - The Axon.has() api now validates the hvalu parameter is not None before querying the Axon db via getTufosByProp, to ensure that we are not returning an arbitrary axon:blob tufo.
• #523 - Cortex.formTufoByProp() will now fire node:set:prop events for each property in a newly created node. This does not affect splice generation.
• #523 - Cortex.delTufo() will now fire node:set:prop events for each secondary property in the deleted node to indicate the new-valu is None.
• #523 - Added SynTest.getDirCore() and SynTest.getTestSteps() helpers for getting directory backed Cortexes and TestStep objects, respectively.
• #523 - CoreModule.getModPath() Now returns None if the Cortex the module is loaded in is not a directory backed Cortex.
• #523 - synapse.lib.queue.Queue now has a size() API and a __len__ implementation which allows inspection of how many items are in the internal collections.deque object.
• #523 - Added synapse.lib.scope.pop() and synapse.lib.scope.Scope.pop() methods, which allow either a thread or object local scope to have a named object pop’ed out of it, similar to dict.pop().
• #523 - Storm syntax integer parser now supports parsing negative values (starting with a - sign), parsing values which start with 0x as hex values, parsing values which start with 0b as binary strings, and parsing floats properly.
• #539 - CompType forms now accept dictionary of values as input. They keys which map to fields and optfields are used to form the node.
• #539 - Added ipv4, tcp4, and udp4 secondary properties to inet:dns:look. These represent the IP address which requested the look, and the servers which may have responded to the look.

Bugs

• #529 - The SvcProxy object was incorrectly registering services by name, as well as tags, in its' ByTag helper. This was causing the SvcProxy to think additional services were still available after they were no longer available to the SvcBus. This has been corrected, and the SvcProxy no longer misuses the ByTag helper.
• #531 - The atexit handler for the EventBus had a bad reference which could have triggered a NameError on shutdown. This has been fixed.
• #533 - Change the Axon test test_axon_host_spinbackup to use waiters on syn:svc:init events to address a race condition.
• #534 - Additional proxy objects are fini'd during Axon tests.
• #523 - Socket.send() now catches OSError and ConnectionError exceptions and fini’s the socket if they occur.

Documentation

• #527 - Added Storm documentation for the stats() operator.
• #534 - Docstrings for synapse.lib.persist.Dir.items(), synapse.lib.service.SvcProxy.callByTag, synapse.lib.service.runSynSvc have been rewritten.

0.0.35 release

Synapse v0.0.35 Release Notes

New Features

• #524 - The Cortex class has a new API getCoreMods, which returns a list of the currently loaded CoreModules in the Cortex.

Enhancements

• #522 - Exceptions raised during the thread Pool _run_work function are logged with additional information about what failed to run.

Bugs

• #522 - The synapse.lib.msgpack.en() function's use of the global msgpack.Packer object was wrapped in a try/except block; so that in the event of an exception during packing, we call the reset method to clear internal buffers of the object. It was possible that a serialization failure leaves data in the object, which would then be passed along to a subsequent caller. See msgpack/msgpack-python#258 for example code showing this issue.
• #522 - Ensure that the axonbus Proxy objects made by Axon and Axonhost objects are fini'd.
• #522 - Fini more objects during Axon and Telepath tests which were not properly fini'd.
• #525 - The Axon _fireAxonClones function did not wait for its existing clones to come online (since they are handled by threads) befor entering the _findAxonClones routine. This could have caused the Axon to attempt to make additional clones for itself until the number of clones the Axon had loaded met the axon:clones option. The _fireAxonClones clones routine now waits 60 seconds for each previously known clone to come online before attempting to bring new clones online for itself.
• #526 - Pypi package had included a scripts package. This included development related scripts and was not intended for redistribution; and it collides with an existing scripts package on Pypi.

Documentation

• #522 - Update docstrings for telepath.openurl and telepath.openlink APIs.

0.0.34 release

Synapse v0.0.34 Release Notes

New Features

• #504 - Universal node properties, tufo:form and node:created, are now model properties. Those properties do not have a form associated with them. In addition, the universal node property node:ndef was added. This is the guid derived from the primary property and primary property together, giving a way to universally represent a node value in a anonymous form. Universal properties are now added to the the datamodel documentation generated by autodoc. The associated migration for adding node:ndef values to nodes migrates all forms loaded into the Cortex datamodel at the time of startup. Depending on the size of a Cortex, this migration may take a long time to complete and it is encouraged that large (10 million+ node) Cortexes have a test migration done on a backup of the Cortex.
• #515 - Add a inet:addr type, which normalizes both IPV4 and IPV6 values to a single IPV6 value, which will produce a IPV4 sub if the address is part of the v6 -> v4 mapped space.
• #515 - Add a inet:dns:req form to record a DNS request which was made by an IP at a given time.
• #515 - Add a inet:dns:type type to enumerate different types of DNS requests.

Enhancements

• #516 - The task:<taskname> events fired by the Storm task() operator includes all the nodes in the current query set under the nodes value, instead of firing a single event per node under the node key.
• #504 - The msgpack helpers, synapse.common.msgenpack, synapse.common.msgunpack and synapse.common.msgpackfd were removed. They are duplicates of functionality present in synapse.lib.msgpack content. They are replaced by synpase.lib.msgpack.en, synapse.lib.msgpack.un and synapse.lib.fd respectively.

Bugs

• #517 - The Cortex.delTufoTag API did not return the tufo to the caller. It now returns the modified tufo to the caller.
• #518 - Ensure Axon resources are fini'd during Axon related tests.
• #519 - The tests for normalizing the string 'now' as a time type are more forgiving of system load.

Documentation

• #512 - Added style guide notes to prefer returning None over raising exceptions.
• #513 - Added filter documentation for storm
• #520 - Added a link to the Synapse slack chat to the readme.rst file. Invite your friends, they're welcome here!