-
Notifications
You must be signed in to change notification settings - Fork 162
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
10 changed files
with
408 additions
and
433 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,32 @@ | ||
--- | ||
id: codecs | ||
title: "Codecs" | ||
--- | ||
|
||
Once we have our schema, we can combine it with a codec. A codec is a combination of a schema and a serializer. Unlike codecs in other libraries, a codec in ZIO Schema has no type parameter: | ||
|
||
```scala | ||
trait Codec { | ||
def encoder[A](schema: Schema[A]): ZTransducer[Any, Nothing, A, Byte] | ||
def decoder[A](schema: Schema[A]): ZTransducer[Any, String, Byte, A] | ||
|
||
def encode[A](schema: Schema[A]): A => Chunk[Byte] | ||
def decode[A](schema: Schema[A]): Chunk[Byte] => Either[String, A] | ||
} | ||
``` | ||
|
||
It basically says: | ||
- `encoder[A]`: Given a `Schema[A]` it is capable of generating an `Encoder[A]` ( `A => Chunk[Byte]`) for any Schema. | ||
- `decoder[A]`: Given a `Schema[A]` it is capable of generating a `Decoder[A]` ( `Chunk[Byte] => Either[String, A]`) for any Schema. | ||
|
||
Example of possible codecs are: | ||
|
||
- CSV Codec | ||
- JSON Codec (already available) | ||
- Apache Avro Codec (in progress) | ||
- Apache Thrift Codec (in progress) | ||
- XML Codec | ||
- YAML Codec | ||
- Protobuf Codec (already available) | ||
- QueryString Codec | ||
- etc. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
--- | ||
id: combining-different-encoders | ||
title: "Combining Different Encoders" | ||
--- | ||
|
||
Let's take a look at a round-trip converting an object to JSON and back, then converting it to a protobuf and back. This is a simple example, but it shows how to combine different encoders to achieve a round-trip. | ||
|
||
```scala | ||
object CombiningExample extends zio.App { | ||
import zio.schema.codec.JsonCodec | ||
import zio.schema.codec.ProtobufCodec | ||
import ManualConstruction._ | ||
import zio.stream.ZStream | ||
|
||
override def run(args: List[String]): UIO[ExitCode] = for { | ||
_ <- ZIO.unit | ||
_ <- ZIO.debug("combining roundtrip") | ||
person = Person("Michelle", 32) | ||
|
||
personToJson = JsonCodec.encoder[Person](schemaPerson) | ||
jsonToPerson = JsonCodec.decoder[Person](schemaPerson) | ||
|
||
personToProto = ProtobufCodec.encoder[Person](schemaPerson) | ||
protoToPerson = ProtobufCodec.decoder[Person](schemaPerson) | ||
|
||
newPerson <- ZStream(person) | ||
.tap(v => ZIO.debug("input object is: " + v)) | ||
.transduce(personToJson) | ||
.transduce(jsonToPerson) | ||
.tap(v => ZIO.debug("object after json roundtrip: " + v)) | ||
.transduce(personToProto) | ||
.transduce(protoToPerson) | ||
.tap(v => ZIO.debug("person after protobuf roundtrip: " + v)) | ||
.runHead | ||
.some | ||
.catchAll(error => ZIO.debug(error)) | ||
_ <- ZIO.debug("is old person the new person? " + (person == newPerson).toString) | ||
_ <- ZIO.debug("old person: " + person) | ||
_ <- ZIO.debug("new person: " + newPerson) | ||
} yield ExitCode.success | ||
} | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,146 @@ | ||
--- | ||
id: getting-started | ||
title: "Getting Started" | ||
--- | ||
|
||
To get started, first we need to understand that a ZIO Schema is basically built-up from these three | ||
sealed traits: `Record[R]`, `Enum[A]` and `Sequence[Col, Elem]`, along with the case class `Primitive[A]`. Every other type is just a specialisation of one of these (or not relevant to get you started). | ||
|
||
We will take a look at them now. | ||
|
||
## Basic Building Blocks | ||
|
||
### Schema | ||
|
||
The core data type of ZIO Schema is a `Schema[A]` which is **invariant in `A`** by necessity, because a Schema allows us to derive operations that produce an `A` but also operations that consume an `A` and that imposes limitations on the types of **transformation operators** and **composition operators** that we can provide based on a `Schema`. | ||
|
||
It looks kind of like this (simplified): | ||
|
||
```scala | ||
sealed trait Schema[A] { self => | ||
def zip[B](that: Schema[B]): Schema[(A, B)] | ||
|
||
def transform[B](f: A => B, g: B => A): Schema[B] | ||
} | ||
``` | ||
|
||
### Records | ||
|
||
Our data structures usually are composed of a lot of types. For example, we might have a `User` | ||
type that has a `name` field, an `age` field, an `address` field, and a `friends` field. | ||
|
||
```scala | ||
case class User(name: String, age: Int, address: Address, friends: List[User]) | ||
``` | ||
|
||
This is called a **product type** in functional programming. The equivalent of a product type in ZIO Schema is called a record. | ||
|
||
In ZIO Schema such a record would be represented using the `Record[R]` typeclass: | ||
|
||
```scala | ||
object Schema { | ||
sealed trait Record[R] extends Schema[R] { | ||
def fields: Chunk[Field[_]] | ||
def construct(value: R): Chunk[Any] | ||
def defaultValue: Either[String, R] | ||
} | ||
} | ||
|
||
``` | ||
|
||
### Enumerations | ||
|
||
Other times, you might have a type that represents a list of different types. For example, we might have a type, like this: | ||
|
||
```scala | ||
sealed trait PaymentMethod | ||
|
||
object PaymentMethod { | ||
final case class CreditCard(number: String, expirationMonth: Int, expirationYear: Int) extends PaymentMethod | ||
final case class WireTransfer(accountNumber: String, bankCode: String) extends PaymentMethod | ||
} | ||
``` | ||
|
||
In functional programming, this kind of type is called a **sum type**: | ||
- In Scala 2, this is called a **sealed trait**. | ||
- In Scala3, this is called an **enum**. | ||
|
||
In ZIO Schema we call these types `enumeration` types, and they are represented using the `Enum[A]` type class. | ||
|
||
```scala | ||
object Schema extends SchemaEquality { | ||
sealed trait Enum[A] extends Schema[A] { | ||
def annotations: Chunk[Any] | ||
def structure: ListMap[String, Schema[_]] | ||
} | ||
} | ||
``` | ||
|
||
### Sequence | ||
|
||
Often you have a type that is a collection of elements. For example, you might have a `List[User]`. This is called a `Sequence` and is represented using the `Sequence[Col, Elem]` type class: | ||
|
||
```scala | ||
object Schema extends SchemaEquality { | ||
|
||
final case class Sequence[Col, Elem]( | ||
elementSchema: Schema[Elem], | ||
fromChunk: Chunk[Elem] => Col, | ||
toChunk: Col => Chunk[Elem] | ||
) extends Schema[Col] { | ||
self => | ||
override type Accessors[Lens[_, _], Prism[_, _], Traversal[_, _]] = Traversal[Col, Elem] | ||
override def makeAccessors(b: AccessorBuilder): b.Traversal[Col, Elem] = b.makeTraversal(self, schemaA) | ||
override def toString: String = s"Sequence($elementSchema)" | ||
} | ||
} | ||
``` | ||
|
||
### Optionals | ||
|
||
A special variant of a collection type is the `Optional[A]` type: | ||
|
||
```scala | ||
object Schema extends SchemaEquality { | ||
|
||
final case class Optional[A](codec: Schema[A]) extends Schema[Option[A]] { | ||
self => | ||
|
||
private[schema] val someCodec: Schema[Some[A]] = codec.transform(a => Some(a), _.get) | ||
|
||
override type Accessors[Lens[_, _], Prism[_, _], Traversal[_, _]] = | ||
(Prism[Option[A], Some[A]], Prism[Option[A], None.type]) | ||
|
||
val toEnum: Enum2[Some[A], None.type, Option[A]] = Enum2( | ||
Case[Some[A], Option[A]]("Some", someCodec, _.asInstanceOf[Some[A]], Chunk.empty), | ||
Case[None.type, Option[A]]("None", singleton(None), _.asInstanceOf[None.type], Chunk.empty), | ||
Chunk.empty | ||
) | ||
|
||
override def makeAccessors(b: AccessorBuilder): (b.Prism[Option[A], Some[A]], b.Prism[Option[A], None.type]) = | ||
b.makePrism(toEnum, toEnum.case1) -> b.makePrism(toEnum, toEnum.case2) | ||
} | ||
|
||
} | ||
``` | ||
|
||
### Primitives | ||
|
||
Last but not least, we have primitive values. | ||
|
||
```scala | ||
object Schema extends SchemaEquality { | ||
final case class Primitive[A](standardType: StandardType[A]) extends Schema[A] { | ||
type Accessors[Lens[_, _], Prism[_, _], Traversal[_, _]] = Unit | ||
|
||
override def makeAccessors(b: AccessorBuilder): Unit = () | ||
} | ||
} | ||
``` | ||
|
||
Primitive values are represented using the `Primitive[A]` type class and represent the elements, | ||
that we cannot further define through other means. If we visualize our data structure as a tree, primitives are the leaves. | ||
|
||
ZIO Schema provides a number of built-in primitive types, that we can use to represent our data. | ||
These can be found in the `StandardType` companion-object. | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,97 @@ | ||
--- | ||
id: motivation | ||
title: "Understanding The Motivation Behind ZIO Schema" | ||
--- | ||
|
||
ZIO Schema is a library used in many ZIO projects such as _ZIO Flow_, _ZIO Redis_, _ZIO Web_, _ZIO SQL_ and _ZIO DynamoDB_. It is all about reification of our types. Reification means transforming something abstract (e.g. side effects, accessing fields, structure) into something "real" (values). | ||
|
||
## Reification: Functional Effects | ||
|
||
In functional effects, we reify by turning side-effects into values. For example, we might have a simple statement like; | ||
|
||
```scala | ||
println("Hello") | ||
println("World") | ||
``` | ||
|
||
In ZIO we reify this statement to a value like | ||
|
||
```scala | ||
val effect1 = Task(println("Hello")) | ||
val effect2 = Task(println("World")) | ||
``` | ||
|
||
And then we are able to do awesome things like: | ||
|
||
```scala | ||
(Task(println("Hello")) zipPar Task(println("World"))).retryN(100) | ||
``` | ||
|
||
## Reification: Optics | ||
|
||
In Scala, we have product types like this case class of a `Person`: | ||
|
||
```scala | ||
final case class Person(name: String, age: Int) | ||
``` | ||
|
||
This case class has two fields: | ||
|
||
- A field `name` of type `String` | ||
- A field `age` of type `Int` | ||
|
||
The Scala language provides special support to access the fields inside case classes using the dot syntax: | ||
|
||
```scala | ||
val person = Person("Michelle", 32) | ||
val name = person.name | ||
val age = person.age | ||
``` | ||
|
||
However, this is a "special language feature", it's not "real" like the side effects we've seen in the previous example (`println(..) vs. Task(println(...))`). | ||
|
||
Because these basic operations are not "real," we are unable to create an operator that we can use, for example, we cannot combine two fields that are inside a nested structure. | ||
|
||
The solution to this kind of problem is called an "Optic". Optics provide a way to access the fields of a case class and nested structures. There are three main types of optics: | ||
- `Lens`: A lens is a way to access a field of a case class. | ||
- `Prism`: A prism is a way to access a field of a nested structure or a collection. | ||
- `Traversal`: A traversal is a way to access all fields of a case class, nested structures or collections. | ||
|
||
Optics allow us to take things which are not a first-class **concept**, and turn that into a first-class **value**, namely the concept of | ||
- drilling down into a field inside a case class or | ||
- drilling down into a nested structure. | ||
|
||
Once we have a value, we can compose these things together to solve hard problems in functional programming, e.g. | ||
- handling nested case class copies, | ||
- iterations down deep inside on elements of a nested structure or collections | ||
|
||
For more information on optics, refer to the [ZIO Optics](https://zio.dev/zio-optics/) documentation. | ||
|
||
|
||
## Reification: Schema | ||
|
||
So far we've looked at how to | ||
- reify side-effects into values (ZIO) | ||
- how to reify accessing and modifying fields inside case classes or arbitrary structures by turning these operations into values as well (Optics) | ||
|
||
**ZIO Schema** is now about how to **describe entire data structures using values**. | ||
|
||
The "built-in" way in scala on how to describe data structures are `case classes` and `classes`. | ||
|
||
For example, assume we have the `Person` data type, like this: | ||
|
||
```scala | ||
final case class Person(name: String, age: Int) | ||
``` | ||
|
||
It has the following information: | ||
|
||
- Name of the structure: `Person` | ||
- Fields: `name` and `age` | ||
- Type of the fields: `String` and `Int` | ||
- Type of the structure: `Person` | ||
|
||
ZIO Schema tries to reify the concept of structure for datatypes by turning the above information into values. | ||
|
||
Not only for case classes, but also for other types like collections, tuples, enumerations etc. | ||
|
Oops, something went wrong.