Merging main to demo (#856)

* Explicitly keep track of all Processes in the Executable to make sure all Processes are assigned the Runtime Signed-off-by: Risbud, Sumedh <sumedh.risbud@intel.com> * A utility to measure the size of a python object recursively. Useful for example, to measure the size of NxBoard object after compilation Signed-off-by: Risbud, Sumedh <sumedh.risbud@intel.com> * Removed the utility to estimate Python object size Signed-off-by: Risbud, Sumedh <sumedh.risbud@intel.com> * Fixed Runtime unittest to accommodate passing an explicit ProcessList to the Executable Signed-off-by: Risbud, Sumedh <sumedh.risbud@intel.com> * Removed unused import Signed-off-by: Risbud, Sumedh <sumedh.risbud@intel.com> * LearningDense bit-accurate (#812) * minor change in dependency computation * updating stochastic round type hint * small fix in clip_weights * progress in making tests pass * fixing Sparse init * trying tests * adapting init method of LearningDense Process --------- Co-authored-by: PhilippPlank <32519998+PhilippPlank@users.noreply.github.com> * Bump fonttools from 4.41.1 to 4.43.0 (#824) Bumps [fonttools](https://github.com/fonttools/fonttools) from 4.41.1 to 4.43.0. - [Release notes](https://github.com/fonttools/fonttools/releases) - [Changelog](https://github.com/fonttools/fonttools/blob/main/NEWS.rst) - [Commits](fonttools/fonttools@4.41.1...4.43.0) --- updated-dependencies: - dependency-name: fonttools dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump gitpython from 3.1.37 to 3.1.41 (#825) Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.37 to 3.1.41. - [Release notes](https://github.com/gitpython-developers/GitPython/releases) - [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES) - [Commits](gitpython-developers/GitPython@3.1.37...3.1.41) --- updated-dependencies: - dependency-name: gitpython dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * SigmaS4Delta Neuronmodel and Layer with Unittests (#830) * first wokring version * S4D model cleaned * update license * fix imports * linting * incorporate reviews * update docstring * Bump pillow from 10.0.1 to 10.2.0 (#832) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.0.1 to 10.2.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.0.1...10.2.0) --- updated-dependencies: - dependency-name: pillow dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: PhilippPlank <32519998+PhilippPlank@users.noreply.github.com> * [QUBO] Solution readout via spikeIO for multi-chip support (#820) * 32bit receiver * linting --------- Co-authored-by: Risbud, Sumedh <sumedh.risbud@intel.com> * Bump cryptography from 41.0.6 to 42.0.0 (#834) Bumps [cryptography](https://github.com/pyca/cryptography) from 41.0.6 to 42.0.0. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@41.0.6...42.0.0) --- updated-dependencies: - dependency-name: cryptography dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cryptography from 42.0.0 to 42.0.2 (#836) Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.0 to 42.0.2. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@42.0.0...42.0.2) --- updated-dependencies: - dependency-name: cryptography dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cryptography from 42.0.2 to 42.0.4 (#837) Bumps [cryptography](https://github.com/pyca/cryptography) from 42.0.2 to 42.0.4. - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@42.0.2...42.0.4) --- updated-dependencies: - dependency-name: cryptography dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Spiker with 32bit * Alternative to Injector/Extractor Processes (#835) * prototype implementing injector/extractor function, not wrapped in a Process * modified injector and extractor classes * fixed linting --------- Co-authored-by: PhilippPlank <32519998+PhilippPlank@users.noreply.github.com> Co-authored-by: Philipp Plank <philipp.plank@intel.com> * fix security issues * Fix workflows (#844) * prototype implementing injector/extractor function, not wrapped in a Process * modified injector and extractor classes * fixed linting * fix workflows --------- Co-authored-by: gkarray <ghassen.karray@intel.com> * Bump pillow from 10.2.0 to 10.3.0 (#847) Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.2.0 to 10.3.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@10.2.0...10.3.0) --- updated-dependencies: - dependency-name: pillow dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump idna from 3.6 to 3.7 (#848) Bumps [idna](https://github.com/kjd/idna) from 3.6 to 3.7. - [Release notes](https://github.com/kjd/idna/releases) - [Changelog](https://github.com/kjd/idna/blob/master/HISTORY.rst) - [Commits](kjd/idna@v3.6...v3.7) --- updated-dependencies: - dependency-name: idna dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update ci.yml (#853) * ATRLIF neuron model (#846) * added process and CPU process models of ATRLIF neuron; added a Jupyter notebook to demonstrate the properties of the ATRLIF neuron * testing of the tutorial * tests for process and model added; copyright notes added; cleanup * codacy-related fixed * comment formatting and copyright notices adjusted * Add the models and process of conv_in_time in src/lava/proc/conv_in_time (#833) * add the models and process of conv_in_time in src/lava/proc/conv_in_time * remove unused library * remove Trailing whitespace * add unittest for conv in time and related pytorch ground truth * add fixed_pt version of conv in time * change input to spike_input * add from lava.proc.conv import utils * remove unwanted comments * fixed some linting errors * Start all comments with upper case character & change the year for all copyright headers to 2024 for new files * remove whitespace * continuation line under-indented * Trailing whitespace * shorten variables names --------- Co-authored-by: bamsumit <bam_sumit@hotmail.com> Co-authored-by: PhilippPlank <32519998+PhilippPlank@users.noreply.github.com> * Fixed/updated poetry.lock to match the TOML file Signed-off-by: Risbud, Sumedh <sumedh.risbud@intel.com> * Delinting Signed-off-by: Risbud, Sumedh <sumedh.risbud@intel.com> * Fixed a unittest by adding .name attribute to a Mock object Signed-off-by: Risbud, Sumedh <sumedh.risbud@intel.com> --------- Signed-off-by: Risbud, Sumedh <sumedh.risbud@intel.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Ghassen Karray <36999864+gkarray@users.noreply.github.com> Co-authored-by: PhilippPlank <32519998+PhilippPlank@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: smm-ncl <145007783+smm-ncl@users.noreply.github.com> Co-authored-by: Philipp Stratmann <86950058+phstratmann@users.noreply.github.com> Co-authored-by: Philipp Plank <philipp.plank@intel.com> Co-authored-by: gkarray <ghassen.karray@intel.com> Co-authored-by: Jannik Luboeinski <33398515+jlubo@users.noreply.github.com> Co-authored-by: Zeyu Liu <liuzeyu@usc.edu> Co-authored-by: bamsumit <bam_sumit@hotmail.com>
lava-nc · May 16, 2024 · a91206d · a91206d
1 parent 974b487
commit a91206d
Show file tree

Hide file tree

Showing 46 changed files with 3,405 additions and 503 deletions.
diff --git a/.github/workflows/cd.yml b/.github/workflows/cd.yml
@@ -1,4 +1,5 @@
 name: Run CD
+permissions: read-all
 on:
   workflow_dispatch:
 

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -1,4 +1,5 @@
 name: Run CI
+permissions: read-all
 on:
   push:
     branches:
@@ -54,7 +55,7 @@ jobs:
     runs-on: ${{ matrix.operating-system }}
     strategy:
       matrix:
-        operating-system: [ubuntu-latest, windows-latest, macos-latest]
+        operating-system: [ubuntu-latest, windows-latest, macos-13]
     steps:
     - uses: actions/checkout@v3
       with:

diff --git a/.github/workflows/codacy_coverage_reporter.yml b/.github/workflows/codacy_coverage_reporter.yml
@@ -1,5 +1,5 @@
 name: Codacy Coverage Reporter
-
+permissions: read-all
 on:
   workflow_run:
      workflows: ["Run CI"]

diff --git a/.github/workflows/issues.yml b/.github/workflows/issues.yml
@@ -1,5 +1,5 @@
 name: Add new issues to the NCL planning project and label them
-
+permissions: {}
 on:
   issues:
     types:
@@ -10,6 +10,7 @@ on:
 jobs:
   add-to-project:
     name: Add issue to project
+    permissions: {}
     runs-on: ubuntu-latest
     steps:
       - uses: actions/add-to-project@v0.4.0

diff --git a/.gitignore b/.gitignore
@@ -145,3 +145,5 @@ dmypy.json
 .idea/
 .vscode/
 .history/
+.flakeheaven_cache/
+tutorials/in_depth/results/
diff --git a/poetry.lock b/poetry.lock
diff --git a/src/lava/magma/compiler/channel_map.py b/src/lava/magma/compiler/channel_map.py
@@ -4,7 +4,7 @@
 
 import itertools
 import os
-import pickle
+import pickle  # noqa # nosec
 import typing as ty
 from collections import defaultdict
 from dataclasses import dataclass
@@ -33,6 +33,7 @@ class Payload:
 def lmt_init_id():
     return -1
 
+
 class ChannelMap(dict):
     """The ChannelMap is used by the SubCompilers during compilation to
     communicate how they are planning to partition Processes onto their
@@ -193,15 +194,15 @@ def read_from_cache(self,
             src_port_process: AbstractProcess = procname_to_proc_map[
                 src_port_info[0]]
             src: AbstractPort = getattr(src_port_process,
-                                             src_port_info[1])
+                                        src_port_info[1])
             dst_port_process: AbstractProcess = procname_to_proc_map[
                 dst_port_info[0]]
             dst: AbstractPort = getattr(dst_port_process,
-                                             dst_port_info[1])
+                                        dst_port_info[1])
             for port_pair, pld in self.items():
                 s, d = port_pair.src, port_pair.dst
                 if s.name == src.name and d.name == dst.name and \
                     s.process.name == src_port_process.name and \
-                    d.process.name == dst_port_process.name:
+                        d.process.name == dst_port_process.name:
                     pld.src_port_initializer = payload.src_port_initializer
-                    pld.dst_port_initializer = payload.dst_port_initializer
+                    pld.dst_port_initializer = payload.dst_port_initializer
diff --git a/src/lava/magma/compiler/compiler.py b/src/lava/magma/compiler/compiler.py
@@ -5,7 +5,7 @@
 import itertools
 import logging
 import os
-import pickle
+import pickle  # noqa # nosec
 import typing as ty
 from collections import OrderedDict, defaultdict
 
@@ -231,9 +231,8 @@ def _compile_proc_groups(
         if self._compile_config.get("cache", False):
             cache_dir = self._compile_config["cache_dir"]
             if os.path.exists(os.path.join(cache_dir, "cache")):
-                with open(os.path.join(cache_dir, "cache"), "rb") \
-                  as cache_file:
-                    cache_object = pickle.load(cache_file)
+                with open(os.path.join(cache_dir, "cache"), "rb") as cache_file:
+                    cache_object = pickle.load(cache_file)  # noqa # nosec
 
                 proc_builders_values = cache_object["procname_to_proc_builder"]
                 proc_builders = {}
@@ -283,8 +282,7 @@ def _compile_proc_groups(
             # Validate All Processes are Named
             procname_to_proc_builder = {}
             for p, pb in proc_builders.items():
-                if p.name in procname_to_proc_builder or \
-                  "Process_" in p.name:
+                if p.name in procname_to_proc_builder or "Process_" in p.name:
                     msg = f"Unable to Cache. " \
                           f"Please give unique names to every process. " \
                           f"Violation Name: {p.name=}"

diff --git a/src/lava/magma/core/learning/utils.py b/src/lava/magma/core/learning/utils.py
@@ -29,23 +29,24 @@ def stochastic_round(values: np.ndarray,
     return (values + (random_numbers < probabilities).astype(int)).astype(int)
 
 
-def apply_mask(int_number: int, nb_bits: int) -> int:
+def apply_mask(item: ty.Union[np.ndarray, int], nb_bits: int) \
+        -> ty.Union[np.ndarray, int]:
     """Get nb_bits least-significant bits.
 
     Parameters
     ----------
-    int_number : int
-        Integer number.
+    item : np.ndarray or int
+        Item to apply mask to.
     nb_bits : int
         Number of LSBs to keep.
 
     Returns
     ----------
-    result : int
+    result : np.ndarray or int
         Least-significant bits.
     """
     mask = ~(~0 << nb_bits)
-    return int_number & mask
+    return item & mask
 
 
 def float_to_literal(learning_parameter: float) -> str:

diff --git a/src/lava/magma/core/model/py/connection.py b/src/lava/magma/core/model/py/connection.py
@@ -441,8 +441,8 @@ def _update_synaptic_variable_random(self) -> None:
         pass
 
     def _update_dependencies(self) -> None:
-        self.x0[self.tx > 0] = True
-        self.y0[self.ty > 0] = True
+        self.x0 = self.tx > 0
+        self.y0 = self.ty > 0
 
     @abstractmethod
     def _compute_trace_histories(self) -> typing.Tuple[np.ndarray, np.ndarray]:

diff --git a/src/lava/magma/core/process/ports/ports.py b/src/lava/magma/core/process/ports/ports.py
@@ -432,6 +432,17 @@ class OutPort(AbstractIOPort, AbstractSrcPort):
     sub processes.
     """
 
+    def __init__(self, shape: ty.Tuple[int, ...]):
+        super().__init__(shape)
+        self.external_pipe_flag = False
+        self.external_pipe_buffer_size = 64
+
+    def flag_external_pipe(self, buffer_size=None):
+        self.external_pipe_flag = True
+
+        if buffer_size is not None:
+            self.external_pipe_buffer_size = buffer_size
+
     def connect(
             self,
             ports: ty.Union["AbstractIOPort", ty.List["AbstractIOPort"]],
@@ -493,6 +504,15 @@ def __init__(
         super().__init__(shape)
         self._reduce_op = reduce_op
 
+        self.external_pipe_flag = False
+        self.external_pipe_buffer_size = 64
+
+    def flag_external_pipe(self, buffer_size=None):
+        self.external_pipe_flag = True
+
+        if buffer_size is not None:
+            self.external_pipe_buffer_size = buffer_size
+
     def connect(self,
                 ports: ty.Union["InPort", ty.List["InPort"]],
                 connection_configs: ty.Optional[ConnectionConfigs] = None):

diff --git a/src/lava/magma/runtime/runtime.py b/src/lava/magma/runtime/runtime.py
@@ -26,7 +26,7 @@
 if ty.TYPE_CHECKING:
     from lava.magma.core.process.process import AbstractProcess
 from lava.magma.compiler.channels.pypychannel import CspRecvPort, CspSendPort, \
-    CspSelector
+    CspSelector, PyPyChannel
 from lava.magma.compiler.builders.channel_builder import (
     ChannelBuilderMp, RuntimeChannelBuilderMp, ServiceChannelBuilderMp,
     ChannelBuilderPyNc)
@@ -38,7 +38,7 @@
     ChannelType
 from lava.magma.compiler.executable import Executable
 from lava.magma.compiler.node import NodeConfig
-from lava.magma.core.process.ports.ports import create_port_id
+from lava.magma.core.process.ports.ports import create_port_id, InPort, OutPort
 from lava.magma.core.run_conditions import (AbstractRunCondition,
                                             RunContinuous, RunSteps)
 from lava.magma.compiler.channels.watchdog import WatchdogManagerInterface
@@ -308,6 +308,10 @@ def _build_processes(self):
                     proc._runtime = self
                     exception_q = Queue()
                     self.exception_q.append(exception_q)
+
+                    # Create any external pypychannels
+                    self._create_external_channels(proc, proc_builder)
+
                     self._messaging_infrastructure.build_actor(target_fn,
                                                                proc_builder,
                                                                exception_q)
@@ -323,6 +327,44 @@ def _build_runtime_services(self):
                                 rs_builder,
                                 self.exception_q[-1])
 
+    def _create_external_channels(self,
+                                  proc: AbstractProcess,
+                                  proc_builder: AbstractProcessBuilder):
+        """Creates a csp channel which can be connected to/from a
+        non-procss/Lava python environment. This enables I/O to Lava from
+        external sources."""
+        for name, py_port in proc_builder.py_ports.items():
+            port = getattr(proc, name)
+
+            if port.external_pipe_flag:
+                if isinstance(port, InPort):
+                    pypychannel = PyPyChannel(
+                        message_infrastructure=self._messaging_infrastructure,
+                        src_name="src",
+                        dst_name=name,
+                        shape=py_port.shape,
+                        dtype=py_port.d_type,
+                        size=port.external_pipe_buffer_size)
+
+                    proc_builder.set_csp_ports([pypychannel.dst_port])
+
+                    port.external_pipe_csp_send_port = pypychannel.src_port
+                    port.external_pipe_csp_send_port.start()
+
+                if isinstance(port, OutPort):
+                    pypychannel = PyPyChannel(
+                        message_infrastructure=self._messaging_infrastructure,
+                        src_name=name,
+                        dst_name="dst",
+                        shape=py_port.shape,
+                        dtype=py_port.d_type,
+                        size=port.external_pipe_buffer_size)
+
+                    proc_builder.set_csp_ports([pypychannel.src_port])
+
+                    port.external_pipe_csp_recv_port = pypychannel.dst_port
+                    port.external_pipe_csp_recv_port.start()
+
     def _get_resp_for_run(self):
         """
         Gets response from RuntimeServices