Skip to content

Commit

Permalink
Support for nested and inlined maps (inside other maps) (#252)
Browse files Browse the repository at this point in the history
### What's done:
- Added a support for nested anonymous Maps
  • Loading branch information
orchestr7 authored Jan 7, 2024
1 parent 3d079cd commit 1edd2a6
Show file tree
Hide file tree
Showing 6 changed files with 227 additions and 49 deletions.
52 changes: 42 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,13 +42,12 @@ However, to reduce the scope, ktoml now supports only the following platforms:

Other platforms could be added later on the demand (just create a corresponding issue) or easily built by users on their machines.

:globe_with_meridians: ktoml supports Kotlin 1.8.21
:globe_with_meridians: ktoml supports Kotlin 1.9.22

## Current limitations
:heavy_exclamation_mark: Please note, that TOML standard does not define Java-like types: `Char`, `Short`, etc.
You can check types that are supported in TOML [here](https://toml.io/en/v1.0.0#string).
We will support all Kotlin primitive types in the future with the non-strict configuration of ktoml, but now
only String, Long, Double and Boolean are supported from the list of Kotlin primitives.
:heavy_exclamation_mark: Please note, that TOML standard does not define Java-like types: `Char`, `Short`, etc.
You can check types that are supported in TOML standard [here](https://toml.io/en/v1.0.0#string).
However, in Ktoml, our goal is to comprehensively support all primitive types offered by Kotlin.

**General** \
We are still developing and testing this library, so it has several limitations: \
Expand All @@ -73,6 +72,7 @@ We are still developing and testing this library, so it has several limitations:
:white_check_mark: Local Time (to `LocalTime` of [kotlinx-datetime](https://github.com/Kotlin/kotlinx-datetime)) \
:white_check_mark: Multiline Strings \
:white_check_mark: Arrays (including multiline arrays) \
:white_check_mark: Maps (for anonymous key-value pairs) \
:x: Arrays: nested; of Different Types \
:x: Nested Inline Tables \
:x: Array of Tables \
Expand Down Expand Up @@ -264,11 +264,6 @@ Toml(
)
```

## Exceptions
Ktoml will produce different exceptions in case of the invalid input. Please note, that some of strict checks can be enabled or disabled (please see
`Configuration` section of this readme). We intentionally made only two parental sealed exceptions public:
`TomlDecodingException` and `TomlEncodingException` - you can catch them in your code. All other exceptions inherit one of these two and will not be public.

## How ktoml works: examples
:heavy_exclamation_mark: You can check how below examples work in [decoding ReadMeExampleTest](https://github.com/akuleshov7/ktoml/blob/main/ktoml-core/src/commonTest/kotlin/com/akuleshov7/ktoml/decoders/ReadMeExampleTest.kt) and [encoding ReadMeExampleTest](https://github.com/akuleshov7/ktoml/blob/main/ktoml-core/src/commonTest/kotlin/com/akuleshov7/ktoml/encoders/ReadMeExampleTest.kt).

Expand Down Expand Up @@ -499,3 +494,40 @@ with the following code:
Toml.encodeToString<MyClass>(/* your encoded object */)
```
</details>

## Q&A

<details>
<summary>I want to catch ktoml-specific exceptions in my code, how can I do it?</summary>

Ktoml may generate various exceptions when encountering invalid input. It's important to note that certain strict checks can be enabled or disabled (refer to the `Configuration` section in this readme). We have intentionally exposed only two top-level exceptions, namely `TomlDecodingException` and `TomlEncodingException`, for public use. You can catch these exceptions in your code, as all other exceptions inherit from one of these two and will not be publicly accessible.
</details>

<details>
<summary>What if I do not know the names for keys and tables in my TOML, and therefore cannot specify a strict schema for decoding? Can I still decode it somehow?</summary>

Certainly. In such cases, you can decode all your key-values into a `Map`. However, it's important to be aware that both ktoml and kotlinx will be unable to enforce type control in this scenario. Therefore, you should not expect any "type safety." For instance, even when dealing with a mixture of types like Int, Map, String, etc., such as:

```toml
[a]
b = 42
c = "String"
[a.innerTable]
d = 5
[a.otherInnerTable]
d = "String"
```

You can still decode it using `Toml.decodeFromString<MyClass>(data)` where:

```kotlin
// MyClass(a={b=42, c=String, innerTable={d=5}, otherInnerTable={d=String}})
@Serializable
data class MyClass(
val a: Map<String, Map<String, String>>
)
```


However, be aware that this may lead to unintended side effects. Our recommendation is to decode only key-values of the **same** type for a more predictable outcome.
</details>
Original file line number Diff line number Diff line change
Expand Up @@ -209,8 +209,8 @@ public class TomlMainDecoder(
/**
* A hack that comes from a compiler plugin to process Inline (value) classes
*/
override fun decodeInline(inlineDescriptor: SerialDescriptor): Decoder =
iterateOverTomlStructure(inlineDescriptor, true)
override fun decodeInline(descriptor: SerialDescriptor): Decoder =
iterateOverTomlStructure(descriptor, true)

/**
* this method does all the iteration logic for processing code structures and collections
Expand Down Expand Up @@ -247,7 +247,7 @@ public class TomlMainDecoder(
// This logic is a special case when user would like to parse key-values from a table to a map.
// It can be useful, when the user does not know key names of TOML key-value pairs, for example:
// if parsing
StructureKind.MAP -> TomlMapDecoder(nextProcessingNode)
StructureKind.MAP -> TomlMapDecoder(nextProcessingNode, config)

else -> {
val firstTableChild = nextProcessingNode.getFirstChild() ?: throw InternalDecodingException(
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
package com.akuleshov7.ktoml.decoders

import com.akuleshov7.ktoml.exceptions.UnsupportedDecoderException
import com.akuleshov7.ktoml.TomlInputConfig
import com.akuleshov7.ktoml.exceptions.IllegalTypeException
import com.akuleshov7.ktoml.exceptions.InternalDecodingException
import com.akuleshov7.ktoml.tree.nodes.TomlKeyValue
import com.akuleshov7.ktoml.tree.nodes.TomlStubEmptyNode
import com.akuleshov7.ktoml.tree.nodes.TomlTable
import kotlinx.serialization.DeserializationStrategy
import kotlinx.serialization.ExperimentalSerializationApi
Expand All @@ -14,26 +17,35 @@ import kotlinx.serialization.modules.SerializersModule
* Sometimes, when you do not know the names of the TOML keys and cannot create a proper class with field names for parsing,
* it can be useful to read and parse TOML tables to a map. This is exactly what this TomlMapDecoder is used for.
*
* PLEASE NOTE, THAT IT IS EXTREMELY UNSAFE TO USE THIS DECODER, AS IT WILL DECODE EVERYTHING, WITHOUT ANY TYPE-CHECKING
*
* @param rootNode toml table that we are trying to decode
* @param decodingElementIndex for iterating over the TOML table we are currently reading
* @param kotlinxIndex for iteration inside the kotlinX loop: [decodeElementIndex -> decodeSerializableElement]
* @param config TomlInput config
*/
@ExperimentalSerializationApi
public class TomlMapDecoder(
private val rootNode: TomlTable,
private val config: TomlInputConfig,
private var decodingElementIndex: Int = 0,
private var kotlinxIndex: Int = 0,
) : TomlAbstractDecoder() {
override val serializersModule: SerializersModule = EmptySerializersModule()

override fun decodeElementIndex(descriptor: SerialDescriptor): Int {
// we will iterate in the following way:
// for [map]
// a = 1
// b = 2
// kotlinxIndex will be 0, 1, 2 ,3
// and decodingElementIndex will be 0, 1 (as there are only two elements in the table: 'a' and 'b')
decodingElementIndex = kotlinxIndex / 2
// stubs are internal technical nodes that are not needed in this scenario
if (rootNode.children[decodingElementIndex] is TomlStubEmptyNode) {
skipStubs()
} else {
// we will iterate in the following way:
// for [map]
// a = 1
// b = 2
// kotlinxIndex will be (0, 1), (2 ,3)
// and decodingElementIndex will be 0, 1 (as there are only two elements in the table: 'a' and 'b')
decodingElementIndex = kotlinxIndex / 2
}

if (decodingElementIndex == rootNode.children.size) {
return CompositeDecoder.DECODE_DONE
Expand All @@ -48,24 +60,43 @@ public class TomlMapDecoder(
deserializer: DeserializationStrategy<T>,
previousValue: T?
): T {
val returnValue = when (val processedNode = rootNode.children[decodingElementIndex]) {
// stubs are internal technical nodes that are not needed in this scenario
skipStubs()
return when (val processedNode = rootNode.children[decodingElementIndex]) {
// simple decoding for key-value type
is TomlKeyValue -> processedNode
else -> throw UnsupportedDecoderException(
""" Attempting to decode <$rootNode>; however, custom Map decoders do not currently support nested structures.
Decoding is limited to plain structures only:
[map]
a = 1
b = 2
c = "3"
"""
)
is TomlKeyValue -> ((if (index % 2 == 0) processedNode.key.toString() else processedNode.value.content)) as T
is TomlTable -> if (index % 2 == 0) {
processedNode.name as T
} else {
TomlMapDecoder(processedNode, config).decodeSerializableValue(deserializer)
}
else -> throw InternalDecodingException("Trying to decode ${processedNode.prettyStr()} with TomlMapDecoder, " +
"but faced an unknown type of Node")
}

return ((if (index % 2 == 0) returnValue.key.toString() else returnValue.value.content)) as T
}

override fun decodeKeyValue(): TomlKeyValue {
TODO("No need to implement decodeKeyValue for TomlMapDecoder as it is not needed for such primitive decoders")
throw IllegalTypeException("""
You are trying to decode a nested Table ${rootNode.fullTableKey} with a <Map> type to some primitive type.
For example:
[a]
[a.b]
a = 2
should be decoded to Map<String, Map<String, Long>>, but not to Map<String, Long>
""", rootNode.lineNo)
}

/**
* TomlStubs are internal technical nodes that should be skipped during the decoding process.
* And so we need to skip them with our iteration indices:
* decodingElementIndex had step equals to 1
* kotlinxIndex has step equals to 2 (because in kotlinx.serialization Maps have x2 index: one for key and one for value)
*/
private fun skipStubs() {
if (rootNode.children[decodingElementIndex] is TomlStubEmptyNode) {
++decodingElementIndex
kotlinxIndex += 2
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
package com.akuleshov7.ktoml.decoders

import com.akuleshov7.ktoml.Toml
import com.akuleshov7.ktoml.exceptions.IllegalTypeException
import kotlinx.serialization.ExperimentalSerializationApi
import kotlinx.serialization.Serializable
import kotlinx.serialization.decodeFromString
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFailsWith

class NestedMapTest {
@Serializable
data class NestedTable(
val outer: Map<String, Map<String, Long>>
)

@Serializable
data class SimpleNestedTable(
val map: Map<String, Long>,
)

@ExperimentalSerializationApi
@Test
fun nestedInvalidMapping() {
val data = """
[map]
[map.a]
b = 1
""".trimIndent()

assertFailsWith<IllegalTypeException> {
Toml.decodeFromString<SimpleNestedTable>(data)
}
}

@ExperimentalSerializationApi
@Test
fun testSimpleNestedMaps() {
val data = """
[outer]
[outer.inner1]
a = 5
b = 5
""".trimIndent()

val result = Toml.decodeFromString<NestedTable>(data)
assertEquals(NestedTable(outer = mapOf("inner1" to mapOf("a" to 5, "b" to 5))), result)
}

@ExperimentalSerializationApi
@Test
fun testNestedMaps() {
val data = """
[outer]
[outer.inner1]
a = 5
b = 5
[outer.inner2]
c = 7
d = 12
""".trimIndent()

val result = Toml.decodeFromString<NestedTable>(data)
assertEquals(NestedTable(outer = mapOf("inner1" to mapOf("a" to 5, "b" to 5), "inner2" to mapOf("c" to 7, "d" to 12))), result)
}

@ExperimentalSerializationApi
@Test
fun nestedMapFromReadme() {
@Serializable
data class MyClass(
val a: Map<String, Map<String, String>>
)

val data = """
[a]
b = 42
c = "String"
[a.innerTable]
d = 5
[a.otherInnerTable]
d = "String"
""".trimIndent()

println(Toml.decodeFromString<MyClass>(data))
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
package com.akuleshov7.ktoml.decoders

import com.akuleshov7.ktoml.Toml
import kotlinx.serialization.ExperimentalSerializationApi
import kotlinx.serialization.Serializable
import kotlinx.serialization.decodeFromString
import kotlin.test.Test
import kotlin.test.assertEquals

class NestedTableTest {
@Serializable
data class NestedTable(
val outer: Outer
)

@Serializable
data class Outer(
val inner: Inner
)

@Serializable
data class Inner(
val a: Int
)

@ExperimentalSerializationApi
@Test
fun testDottedKeys() {
val data = """
[outer]
[outer.inner]
a = 5
""".trimIndent()

val result = Toml.decodeFromString<NestedTable>(data)
assertEquals(NestedTable(Outer(Inner(a = 5))), result)
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -112,20 +112,6 @@ class PlainMapDecoderTest {
Toml.decodeFromString<TestDataMap>(data)
}

data = """
[map]
[map.a]
b = 1
[map.b]
c = 1
text = "Test"
number = 15
""".trimIndent()

assertFailsWith<UnsupportedDecoderException> {
Toml.decodeFromString<TestDataMap>(data)
}

data = """
text = "Test"
number = 15
Expand Down

0 comments on commit 1edd2a6

Please sign in to comment.