-
Notifications
You must be signed in to change notification settings - Fork 5
/
README
205 lines (152 loc) · 7.93 KB
/
README
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
.. figure:: https://cdn.rawgit.com/IRC-SPHERE/HyperStream/dfbac332/hyperstream.svg
:alt: HyperStream logo
HyperStream logo
HyperStream
===========
|DOI| |Join the chat at https://gitter.im/IRC-SPHERE-HyperStream/Lobby|
|Build Status| |Dependency Status| |Test Coverage| |Issue Count|
|Documentation Status|
Hyperstream is a large-scale, flexible and robust software package for
processing streaming data.
- HyperStream `homepage <https://irc-sphere.github.io/HyperStream/>`__
- Tutorial
`notebooks <http://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/>`__
- Gitter `chat room <https://gitter.im/IRC-SPHERE-HyperStream/Lobby>`__
- Developer
`documentation <http://hyperstream.readthedocs.io/en/latest/>`__
Hyperstream overcomes the limitations of other computational engines and
provides high-level interfaces to execute complex nesting, fusion, and
prediction both in online and offline forms in streaming environments.
Although developed specifically for SPHERE, Hyperstream is a general
purpose tool that is well-suited for the design, development, and
deployment of algorithms and predictive models in a wide space of
sequential predictive problems.
This software has been designed from the outset to be
domain-independent, in order to provide maximum value to the wider
community. Key aspects of the software include the capability to create
complex interlinked workflows, and a computational engine that is
designed to be "compute-on-request", meaning that no unnecessary
resources are used.
Installation
============
Docker images
-------------
If you do not want to install all the packages separately you can use
our Docker bundle available
`here <https://github.com/IRC-SPHERE/Hyperstream-Dockerfiles>`__.
Local machine
-------------
Install via pip
::
pip install hyperstream
python -c 'from hyperstream import HyperStream'
To get the latest version
::
pip install -U git+git://github.com/IRC-SPHERE/HyperStream.git#egg=hyperstream
pip install -r requirements.txt
Or clone the repository
.. code:: bash
git clone git@github.com:IRC-SPHERE/HyperStream.git
cd HyperStream
virtualenv venv
. venv/bin/activate
pip install -r requirements.txt
python -c 'from hyperstream import HyperStream'
Additionally, one of the requirements to run Hyperstream is a MongoDB
server. By default, Hyperstream tries to connect to the port 27017 on
the localhost.
To access via a cloud provider, you might try mLab, MongoDB Atlas, or
search for a different one. If you are using mLab, here are a few notes:
- Set up an account via the mLab web site instructions. When asked to
pick a server type (e.g. Amazon, Google, etc) you can just choose free
option of 500MB. This is more than enough to get started.
- mLab will ask you to create a database; any name is fine, but make
sure you write down what it is.
- After creating a database, note that you’ll need to create at least
one database user in order to access the database.
- You can test your database connection using MongoDB’s built-in command
line tools
To install MongoDB locally go to the `official
documentation <https://docs.mongodb.com/manual/installation/>`__. E.g.
in a Debian OS it is possible to install with the following command
.. code:: bash
sudo apt-get install mongodb
Once the MongoDB server is installed, it can be started with the
following command
.. code:: bash
service mongod start
Running tests
=============
Run the following command
::
nosetests
Note that for the MQTT logging test to succeed, you will need to have an
MQTT broker running (e.g. Mosquitto). For example:
::
docker run -ti -p 1883:1883 -p 9001:9001 toke/mosquitto
or on OSX you will need pidof and mosquitto:
::
brew install pidof
brew install mosquitto
brew services start mosquitto
Tutorials
=========
The following tutorials show how to use HyperStream in a step-by-step
guide.
- `Tutorial 1:
Introduction <https://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/tutorial_01.ipynb>`__
- `Tutorial 2: Creating
tools <https://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/tutorial_02.ipynb>`__
- `Tutorial 3: Stream
composition <https://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/tutorial_03.ipynb>`__
- `Tutorial 4: Real-time
streams <https://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/tutorial_04.ipynb>`__
- `Tutorial 5:
Workflows <https://nbviewer.jupyter.org/github/IRC-SPHERE/HyperStream/blob/tutorials/examples/tutorial_05.ipynb>`__
.. code:: python
from hyperstream import HyperStream, StreamId, TimeInterval
from hyperstream.utils import utcnow, UTC
from datetime import timedelta
hs = HyperStream(loglevel=20)
M = hs.channel_manager.memory
T = hs.channel_manager.tools
clock = StreamId(name="clock")
clock_tool = T[clock].window().last().value()
ticker = M.get_or_create_stream(stream_id=StreamId(name="ticker"))
now = utcnow()
before = (now - timedelta(seconds=30)).replace(tzinfo=UTC)
ti = TimeInterval(before, now)
clock_tool.execute(sources=[], sink=ticker, interval=ti, alignment_stream=None)
print(list(ticker.window().tail(5)))
[StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 39, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 39, tzinfo=<UTC>)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 40, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 40, tzinfo=<UTC>)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 41, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 41, tzinfo=<UTC>)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 42, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 42, tzinfo=<UTC>)), StreamInstance(timestamp=datetime.datetime(2017, 6, 30, 16, 23, 43, tzinfo=<UTC>), value=datetime.datetime(2017, 6, 30, 16, 23, 43, tzinfo=<UTC>))]
HyperStream Viewer
==================
The `HyperStream
Viewer <https://github.com/IRC-SPHERE/HyperStreamViewer>`__ is a
python/Flask web-app for interacting with HyperStream. In order to keep
HyperStream to a minimum, this web-app is released as a separate
repository that takes the core as a dependency.
License
=======
This code is released under the `MIT
license <https://github.com/IRC-SPHERE/HyperStream/blob/master/LICENSE>`__.
Acknowledgements
================
This work has been funded by the UK Engineering and Physical Sciences
Research Council (EPSRC) under Grant
`EP/K031910/1 <http://gow.epsrc.ac.uk/NGBOViewGrant.aspx?GrantRef=EP/K031910/1>`__
- "SPHERE Interdisciplinary Research Collaboration".
.. |DOI| image:: https://zenodo.org/badge/DOI/10.5281/zenodo.242227.svg
:target: https://doi.org/10.5281/zenodo.242227
.. |Join the chat at https://gitter.im/IRC-SPHERE-HyperStream/Lobby| image:: https://badges.gitter.im/IRC-SPHERE-HyperStream/Lobby.svg
:target: https://gitter.im/IRC-SPHERE-HyperStream/Lobby?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge
.. |Build Status| image:: https://travis-ci.org/IRC-SPHERE/HyperStream.svg?branch=master
:target: https://travis-ci.org/IRC-SPHERE/HyperStream
.. |Dependency Status| image:: https://www.versioneye.com/user/projects/58e423cb26a5bb005220301e/badge.svg?style=flat-square
:target: https://www.versioneye.com/user/projects/58e423cb26a5bb005220301e
.. |Test Coverage| image:: https://codeclimate.com/github/IRC-SPHERE/HyperStream/badges/coverage.svg
:target: https://codeclimate.com/github/IRC-SPHERE/HyperStream/coverage
.. |Issue Count| image:: https://codeclimate.com/github/IRC-SPHERE/HyperStream/badges/issue_count.svg
:target: https://codeclimate.com/github/IRC-SPHERE/HyperStream
.. |Documentation Status| image:: https://readthedocs.org/projects/hyperstream/badge/?version=latest
:target: http://hyperstream.readthedocs.io/en/latest/?badge=latest