Skip to content

Commit

Permalink
[nop] Update README template
Browse files Browse the repository at this point in the history
  • Loading branch information
@ptaoussanis
ptaoussanis committed Aug 1, 2023
1 parent ad8f637 commit 513d217
Show file tree
Hide file tree
Showing 3 changed files with 52 additions and 201 deletions.
0 LICENSE → LICENSE.txt
View file
File renamed without changes.
253 changes: 52 additions & 201 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,230 +1,81 @@
<a href="https://www.taoensso.com" title="More stuff by @ptaoussanis at www.taoensso.com">
<img src="https://www.taoensso.com/taoensso-open-source.png" alt="Taoensso open-source" width="400"/></a>
<a href="https://www.taoensso.com/clojure" title="More stuff by @ptaoussanis at www.taoensso.com"><img src="https://www.taoensso.com/open-source.png" alt="Taoensso open source" width="340"/></a>
[**Documentation**](#documentation) | [Latest releases](#latest-releases) | [Get support][GitHub issues]

**[CHANGELOG]** | [API] | current [Break Version]:
# Nippy

```clojure
[com.taoensso/nippy "3.2.0"]
```
#### The fastest serialization library for Clojure

<!-- ![build status](https://github.com/ptaoussanis/nippy/workflows/build/badge.svg?branch=master) -->
Clojure's rich data types are awesome. And its [reader](https://clojure.org/reference/reader) allows you to take your data just about anywhere. But the reader can be painfully slow when you've got a lot of data to crunch (like when you're serializing to a database).

> See [here](https://taoensso.com/clojure/backers) if you're interested in helping support my open-source work, thanks! - Peter Taoussanis
Nippy is an attempt to provide a reliable, high-performance **drop-in alternative to the reader**.

## _SECURITY ADVISORY_
Used by [Carmine](https://github.com/taoensso/carmine), [Faraday](https://github.com/ptaoussanis/faraday), [PigPen](https://github.com/Netflix/PigPen), [Onyx](https://github.com/onyx-platform/onyx), [XTDB](https://github.com/xtdb/xtdb), and others.

Users of Nippy <= `v2.15.0-RC1` should **please upgrade ASAP** due to a **Remote Code Execution (RCE) vulnerability** when deserializing data from an **untrusted source**.
## Latest release/s

See [here](https://github.com/ptaoussanis/nippy/issues/130) for details, including upgrade instructions.
- `2022-07-18` `3.2.0`: [changes](../../releases/tag/3.2.0)

# Nippy: the fastest serialization library for Clojure
[![Main tests][Main tests SVG]][Main tests URL]
[![Graal tests][Graal tests SVG]][Graal tests URL]

Clojure's [rich data types] are *awesome*. And its [reader] allows you to take your data just about anywhere. But the reader can be painfully slow when you've got a lot of data to crunch (like when you're serializing to a database).
See [here][GitHub releases] for earlier releases.

Nippy is an attempt to provide a reliable, high-performance **drop-in alternative to the reader**. Used by the [Carmine Redis client], the [Faraday DynamoDB client], [PigPen], [Onyx] and others.
## Why Nippy?

## Features
* Small, uncomplicated **all-Clojure** library
* **Terrific performance** (the fastest for Clojure that I'm aware of)
* Comprehesive **support for all standard data types**
* **Easily extendable to custom data types** (v2.1+)
* Java's **Serializable** fallback when available (v2.5+)
* **Reader-fallback** for all other types (including Clojure 1.4+ tagged literals)
* **Full test coverage** for every supported type
* Fully pluggable **compression**, including built-in high-performance [LZ4] compressor
* Fully pluggable **encryption**, including built-in high-strength AES128 enabled with a single `:password [:salted "my-password"]` option (v2+)
* Utils for **easy integration into 3rd-party tools/libraries** (v2+)
- Small, simple **all-Clojure** library
- **Terrific performance**: the [best](#performance) for Clojure that I'm aware of
- Comprehensive support for **all standard data types**
- Easily extendable to **custom data types**
- **Robust test suite**, incl. full coverage for every supported type
- Auto fallback to Java Serializable when available
- Auto fallback to Clojure Reader for all other types (including tagged literals
- Pluggable **compression** with built-in [LZ4](https://code.google.com/p/lz4/)
- Pluggable **encryption** with built-in AES128
- Tools for easy + robust **integration into 3rd-party libraries**, etc.

## Getting started

Add the necessary dependency to your project:

```clojure
Leiningen: [com.taoensso/nippy "3.2.0"] ; or
deps.edn: com.taoensso/nippy {:mvn/version "3.2.0"}
```

And setup your namespace imports:

```clojure
(ns my-ns (:require [taoensso.nippy :as nippy]))
```

### De/serializing

As an example of what it can do, let's take a look at Nippy's own reference stress data:

```clojure
nippy/stress-data
=>
{:nil nil
:true true
:false false
:boxed-false (Boolean. false)

:char \ಬ
:str-short "ಬಾ ಇಲ್ಲಿ ಸಂಭವಿಸ"
:str-long (apply str (range 1000))
:kw :keyword
:kw-ns ::keyword
:kw-long (keyword
(apply str "kw" (range 1000))
(apply str "kw" (range 1000)))

:sym 'foo
:sym-ns 'foo/bar
:sym-long (symbol
(apply str "sym" (range 1000))
(apply str "sym" (range 1000)))

:regex #"^(https?:)?//(www\?|\?)?"

:many-small-numbers (vec (range 200))
:many-small-keywords (->> (java.util.Locale/getISOLanguages)
(mapv keyword))
:many-small-strings (->> (java.util.Locale/getISOCountries)
(mapv #(.getDisplayCountry (java.util.Locale. "en" %))))

:queue (enc/queue [:a :b :c :d :e :f :g])
:queue-empty (enc/queue)
:sorted-set (sorted-set 1 2 3 4 5)
:sorted-map (sorted-map :b 2 :a 1 :d 4 :c 3)

:list (list 1 2 3 4 5 (list 6 7 8 (list 9 10 '(()))))
:vector [1 2 3 4 5 [6 7 8 [9 10 [[]]]]]
:map {:a 1 :b 2 :c 3 :d {:e 4 :f {:g 5 :h 6 :i 7 :j {{} {}}}}}
:set #{1 2 3 4 5 #{6 7 8 #{9 10 #{#{}}}}}
:meta (with-meta {:a :A} {:metakey :metaval})
:nested [#{{1 [:a :b] 2 [:c :d] 3 [:e :f]} [#{{}}] #{:a :b}}
#{{1 [:a :b] 2 [:c :d] 3 [:e :f]} [#{{}}] #{:a :b}}
[1 [1 2 [1 2 3 [1 2 3 4 [1 2 3 4 5]]]]]]

:lazy-seq (repeatedly 1000 rand)
:lazy-seq-empty (map identity '())

:byte (byte 16)
:short (short 42)
:integer (int 3)
:long (long 3)
:bigint (bigint 31415926535897932384626433832795)

:float (float 3.14)
:double (double 3.14)
:bigdec (bigdec 3.1415926535897932384626433832795)

:ratio 22/7
:uri (URI. "https://clojure.org/reference/data_structures")
:uuid (java.util.UUID/randomUUID)
:date (java.util.Date.)

;;; JVM 8+
:time-instant (java.time.Instant/now)
:time-duration (java.time.Duration/ofSeconds 100 100)
:time-period (java.time.Period/of 1 1 1)

:bytes (byte-array [(byte 1) (byte 2) (byte 3)])
:objects (object-array [1 "two" {:data "data"}])

:stress-record (StressRecord. "data")
:stress-type (StressType. "data")

;; Serializable
:throwable (Throwable. "Yolo")
:exception (try (/ 1 0) (catch Exception e e))
:ex-info (ex-info "ExInfo" {:data "data"})}
```

Serialize it:

```clojure
(def frozen-stress-data (nippy/freeze nippy/stress-data))
=> #<byte[] [B@3253bcf3>
```

Deserialize it:

```clojure
(nippy/thaw frozen-stress-data)
=> {:bytes (byte-array [(byte 1) (byte 2) (byte 3)])
:nil nil
:boolean true
<...> }
```

Couldn't be simpler!

See also the lower-level `freeze-to-out!` and `thaw-from-in!` fns for operating on `DataOutput` and `DataInput` types directly.

### Encryption (v2+)

Nippy also gives you **dead simple data encryption**. Add a single option to your usual freeze/thaw calls like so:

```clojure
(nippy/freeze nippy/stress-data {:password [:salted "my-password"]}) ; Encrypt
(nippy/thaw <encrypted-data> {:password [:salted "my-password"]}) ; Decrypt
```
## Performance

There's two default forms of encryption on offer: `:salted` and `:cached`. Each of these makes carefully-chosen trade-offs and is suited to one of two common use cases. See the `aes128-encryptor` [API] docs for a detailed explanation of why/when you'd want one or the other.
Since its earliest versions, Nippy has consistently been the **fastest serialization library for Clojure** that I'm aware of. It offers:

### Custom types (v2.1+)
- Roundtrip times **>12x faster** than `tools.reader` with **60% smaller** data size.
- Roundtrip times **>2x faster** than `data.fressian` with **30% smaller** data size.

```clojure
(defrecord MyType [data])
![benchmarks-png](benchmarks.png)

(nippy/extend-freeze MyType :my-type/foo ; A unique (namespaced) type identifier
[x data-output]
(.writeUTF data-output (:data x)))
The [benchmark code](https://github.com/taoensso/nippy/blob/master/src/taoensso/nippy/benchmarks.clj) can be easily run in your own environment.

(nippy/extend-thaw :my-type/foo ; Same type id
[data-input]
(MyType. (.readUTF data-input)))
## Documentation

(nippy/thaw (nippy/freeze (MyType. "Joe"))) => #taoensso.nippy.MyType{:data "Joe"}
```
- [Full documentation][GitHub wiki] (**getting started** and more)
- Auto-generated API reference: [Codox][Codox docs], [clj-doc][clj-doc docs]

## Performance
## Funding

Nippy is currently the **fastest serialization library for Clojure** that I'm aware of, and offers roundtrip times between **~10x and ~15x** faster than Clojure's `tools.reader.edn`, with a **~40% smaller output size**.
You can [help support][sponsor] on this project, thank you!! 🙏

![benchmarks-png]
## License

[Detailed benchmark info] is available on Google Docs.
Copyright &copy; 2012-2023 [Peter Taoussanis][].
Licensed under [EPL 1.0](LICENSE.txt) (same as Clojure).

## Contacting me / contributions
<!-- Common -->

Please use the project's [GitHub issues page] for all questions, ideas, etc. **Pull requests welcome**. See the project's [GitHub contributors page] for a list of contributors.
[GitHub releases]: ../../releases
[GitHub issues]: ../../issues
[GitHub wiki]: ../../wiki

Otherwise, you can reach me at [Taoensso.com]. Happy hacking!
[Peter Taoussanis]: https://www.taoensso.com
[sponsor]: https://www.taoensso.com/sponsor

\- [Peter Taoussanis]
<!-- Project -->

## License
[Codox docs]: https://taoensso.github.io/nippy/
[clj-doc docs]: https://cljdoc.org/d/com.taoensso/nippy/

Distributed under the [EPL v1.0] \(same as Clojure).
Copyright &copy; 2012-2022 [Peter Taoussanis].
[Clojars SVG]: https://img.shields.io/clojars/v/com.taoensso/nippy.svg
[Clojars URL]: https://clojars.org/com.taoensso/nippy

<!--- Standard links -->
[Taoensso.com]: https://www.taoensso.com
[Peter Taoussanis]: https://www.taoensso.com
[@ptaoussanis]: https://www.taoensso.com
[More by @ptaoussanis]: https://www.taoensso.com
[Break Version]: https://github.com/ptaoussanis/encore/blob/master/BREAK-VERSIONING.md

<!--- Standard links (repo specific) -->
[CHANGELOG]: https://github.com/ptaoussanis/nippy/releases
[API]: http://ptaoussanis.github.io/nippy/
[GitHub issues page]: https://github.com/ptaoussanis/nippy/issues
[GitHub contributors page]: https://github.com/ptaoussanis/nippy/graphs/contributors
[EPL v1.0]: https://raw.githubusercontent.com/ptaoussanis/nippy/master/LICENSE
[Hero]: https://raw.githubusercontent.com/ptaoussanis/nippy/master/hero.png "Title"

<!--- Unique links -->
[rich data types]: http://clojure.org/reference/datatypes
[reader]: http://clojure.org/reference/reader
[Carmine Redis client]: https://github.com/ptaoussanis/carmine
[Faraday DynamoDB client]: https://github.com/ptaoussanis/faraday
[PigPen]: https://github.com/Netflix/PigPen
[Onyx]: https://github.com/onyx-platform/onyx
[LZ4]: https://code.google.com/p/lz4/
[benchmarks-png]: https://github.com/ptaoussanis/nippy/raw/master/benchmarks.png
[Detailed benchmark info]: https://docs.google.com/spreadsheet/ccc?key=0AuSXb68FH4uhdE5kTTlocGZKSXppWG9sRzA5Y2pMVkE
[Main tests SVG]: https://github.com/taoensso/nippy/actions/workflows/main-tests.yml/badge.svg
[Main tests URL]: https://github.com/taoensso/nippy/actions/workflows/main-tests.yml
[Graal tests SVG]: https://github.com/taoensso/nippy/actions/workflows/graal-tests.yml/badge.svg
[Graal tests URL]: https://github.com/taoensso/nippy/actions/workflows/graal-tests.yml
Binary file modified benchmarks.png
View file
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 513d217

Please sign in to comment.