From 0050143b222bc5be15bc3f27e832071eb281d770 Mon Sep 17 00:00:00 2001 From: paomian Date: Fri, 6 Sep 2024 16:52:14 +0800 Subject: [PATCH 1/7] chore: add doc for dryrun pipeline api --- docs/user-guide/logs/manage-pipelines.md | 127 +++++++++++++++++- .../user-guide/logs/manage-pipelines.md | 127 +++++++++++++++++- 2 files changed, 252 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/logs/manage-pipelines.md b/docs/user-guide/logs/manage-pipelines.md index 5c36c3cc5..8276c748c 100644 --- a/docs/user-guide/logs/manage-pipelines.md +++ b/docs/user-guide/logs/manage-pipelines.md @@ -87,4 +87,129 @@ Output: Readable timestamp (UTC): 2024-06-27 12:02:34.257312110Z ``` -The output `Readable timestamp (UTC)` represents the creation time of the pipeline and also serves as the version number. \ No newline at end of file +The output `Readable timestamp (UTC)` represents the creation time of the pipeline and also serves as the version number. + +## Test a Pipeline + +First, create a pipeline using the following command: + +```shell +curl -X "POST" "http://localhost:4000/v1/events/pipelines/test?db=public" \ + -H 'Content-Type: application/x-yaml' \ + -d $'processors: + - date: + field: time + formats: + - "%Y-%m-%d %H:%M:%S%.3f" + ignore_missing: true + +transform: + - fields: + - id1 + - id2 + type: int32 + - fields: + - type + - log + - logger + type: string + - field: time + type: time + index: timestamp' +{"pipelines":[{"name":"test","version":"2024-09-03 08:47:38.686403312"}],"execution_time_ms":2} +``` + +Then, test the pipeline using the following command: + +```shell +curl -X "POST" "http://localhost:4000/v1/events/pipelines/dryrun?pipeline_name=test" \ + -H 'Content-Type: application/json' \ + -d $'{"time":"2024-05-25 20:16:37.217","id1":"2436","id2":"2528","type":"I","logger":"INTERACT.MANAGER","log":"ClusterAdapter:enter sendTextDataToCluster\\n"}' +``` + +The output is as follows: + +```json +{ + "rows": [ + [ + { + "data_type": "INT32", + "key": "id1", + "semantic_type": "FIELD", + "value": 2436 + }, + { + "data_type": "INT32", + "key": "id2", + "semantic_type": "FIELD", + "value": 2528 + }, + { + "data_type": "STRING", + "key": "type", + "semantic_type": "FIELD", + "value": "I" + }, + { + "data_type": "STRING", + "key": "log", + "semantic_type": "FIELD", + "value": "ClusterAdapter:enter sendTextDataToCluster\n" + }, + { + "data_type": "STRING", + "key": "logger", + "semantic_type": "FIELD", + "value": "INTERACT.MANAGER" + }, + { + "data_type": "TIMESTAMP_NANOSECOND", + "key": "time", + "semantic_type": "TIMESTAMP", + "value": "2024-05-25 20:16:37.217+0000" + } + ] + ], + "schema": [ + { + "colume_type": "FIELD", + "data_type": "INT32", + "fulltext": false, + "name": "id1" + }, + { + "colume_type": "FIELD", + "data_type": "INT32", + "fulltext": false, + "name": "id2" + }, + { + "colume_type": "FIELD", + "data_type": "STRING", + "fulltext": false, + "name": "type" + }, + { + "colume_type": "FIELD", + "data_type": "STRING", + "fulltext": false, + "name": "log" + }, + { + "colume_type": "FIELD", + "data_type": "STRING", + "fulltext": false, + "name": "logger" + }, + { + "colume_type": "TIMESTAMP", + "data_type": "TIMESTAMP_NANOSECOND", + "fulltext": false, + "name": "time" + } + ] +} +``` + +The output shows that the pipeline successfully processed the log data. The `rows` field contains the processed data, and the `schema` field contains the schema information of the processed data. You can use this information to verify the correctness of the pipeline configuration. \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md index f0175c261..add6917bd 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md @@ -87,4 +87,129 @@ timestamp_ns="1719489754257312110"; readable_timestamp=$(TZ=UTC date -d @$((${ti Readable timestamp (UTC): 2024-06-27 12:02:34.257312110Z ``` -输出的 `Readable timestamp (UTC)` 即为 Pipeline 的创建时间同时也是版本号。 \ No newline at end of file +输出的 `Readable timestamp (UTC)` 即为 Pipeline 的创建时间同时也是版本号。 + +## 测试 Pipeline + +首先，使用以下命令创建一个 Pipeline： + +```shell +curl -X "POST" "http://localhost:4000/v1/events/pipelines/test?db=public" \ + -H 'Content-Type: application/x-yaml' \ + -d $'processors: + - date: + field: time + formats: + - "%Y-%m-%d %H:%M:%S%.3f" + ignore_missing: true + +transform: + - fields: + - id1 + - id2 + type: int32 + - fields: + - type + - log + - logger + type: string + - field: time + type: time + index: timestamp' +{"pipelines":[{"name":"test","version":"2024-09-03 08:47:38.686403312"}],"execution_time_ms":2} +``` + +然后，使用以下命令测试该 Pipeline： + +```shell +curl -X "POST" "http://localhost:4000/v1/events/pipelines/dryrun?pipeline_name=test" \ + -H 'Content-Type: application/json' \ + -d $'{"time":"2024-05-25 20:16:37.217","id1":"2436","id2":"2528","type":"I","logger":"INTERACT.MANAGER","log":"ClusterAdapter:enter sendTextDataToCluster\\n"}' +``` + +输出如下： + +```json +{ + "rows": [ + [ + { + "data_type": "INT32", + "key": "id1", + "semantic_type": "FIELD", + "value": 2436 + }, + { + "data_type": "INT32", + "key": "id2", + "semantic_type": "FIELD", + "value": 2528 + }, + { + "data_type": "STRING", + "key": "type", + "semantic_type": "FIELD", + "value": "I" + }, + { + "data_type": "STRING", + "key": "log", + "semantic_type": "FIELD", + "value": "ClusterAdapter:enter sendTextDataToCluster\n" + }, + { + "data_type": "STRING", + "key": "logger", + "semantic_type": "FIELD", + "value": "INTERACT.MANAGER" + }, + { + "data_type": "TIMESTAMP_NANOSECOND", + "key": "time", + "semantic_type": "TIMESTAMP", + "value": "2024-05-25 20:16:37.217+0000" + } + ] + ], + "schema": [ + { + "colume_type": "FIELD", + "data_type": "INT32", + "fulltext": false, + "name": "id1" + }, + { + "colume_type": "FIELD", + "data_type": "INT32", + "fulltext": false, + "name": "id2" + }, + { + "colume_type": "FIELD", + "data_type": "STRING", + "fulltext": false, + "name": "type" + }, + { + "colume_type": "FIELD", + "data_type": "STRING", + "fulltext": false, + "name": "log" + }, + { + "colume_type": "FIELD", + "data_type": "STRING", + "fulltext": false, + "name": "logger" + }, + { + "colume_type": "TIMESTAMP", + "data_type": "TIMESTAMP_NANOSECOND", + "fulltext": false, + "name": "time" + } + ] +} +``` + +输出显示该 Pipeline 成功处理了日志数据。`rows` 字段包含已处理数据，`schema` 字段包含已处理数据的模式信息。您可以使用这些信息来验证 Pipeline 配置的正确性。 \ No newline at end of file From 38d9dc8c6930b602b62f61c86a3ff099fcedfe53 Mon Sep 17 00:00:00 2001 From: paomian Date: Fri, 6 Sep 2024 19:03:20 +0800 Subject: [PATCH 2/7] chore: example of adding error fixing to the pipeline dryrun http api --- docs/user-guide/logs/manage-pipelines.md | 126 +++++++++++++++++- .../user-guide/logs/manage-pipelines.md | 124 ++++++++++++++++- 2 files changed, 248 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/logs/manage-pipelines.md b/docs/user-guide/logs/manage-pipelines.md index 8276c748c..5899c63d2 100644 --- a/docs/user-guide/logs/manage-pipelines.md +++ b/docs/user-guide/logs/manage-pipelines.md @@ -212,4 +212,128 @@ The output is as follows: } ``` -The output shows that the pipeline successfully processed the log data. The `rows` field contains the processed data, and the `schema` field contains the schema information of the processed data. You can use this information to verify the correctness of the pipeline configuration. \ No newline at end of file +The output shows that the pipeline successfully processed the log data. The `rows` field contains the processed data, and the `schema` field contains the schema information of the processed data. You can use this information to verify the correctness of the pipeline configuration. + +### Test a Failed Pipeline + +Assume that the pipeline configuration is as follows: + + +```bash +curl -X "POST" "http://localhost:4000/v1/events/pipelines/test" \ + -H 'Content-Type: application/x-yaml' \ + -d $'processors: + - date: + field: time + formats: + - "%Y-%m-%d %H:%M:%S%.3f" + ignore_missing: true + - gsub: + fields: + - message + pattern: "\\\." + replacement: + - "-" + ignore_missing: true + +transform: + - fields: + - message + type: string + - field: time + type: time + index: timestamp' +``` + +The pipeline configuration contains an error. The `gsub` Processor expects the `replacement` field to be a string, but the current configuration provides an array. As a result, the pipeline creation fails with the following error message: + + +```json +{"error":"Failed to parse pipeline: 'replacement' must be a string"} +``` + +Therefore, We need to modify the configuration of the `gsub` Processor and change the value of the `replacement` field to a string type. + +```bash +curl -X "POST" "http://localhost:4000/v1/events/pipelines/test" \ + -H 'Content-Type: application/x-yaml' \ + -d $'processors: + - date: + field: time + formats: + - "%Y-%m-%d %H:%M:%S%.3f" + ignore_missing: true + - gsub: + fields: + - message + pattern: "\\\." + replacement: "-" + ignore_missing: true + +transform: + - fields: + - message + type: string + - field: time + type: time + index: timestamp' +``` + +The Pipeline has been successfully created at this point, and We can test the Pipeline using the `dryrun` interface. We will test it with erroneous log data where the value of the message field is in numeric format, causing the pipeline to fail during processing. + + +```bash +curl -X "POST" "http://localhost:4000/v1/events/pipelines/dryrun?pipeline_name=test" \ + -H 'Content-Type: application/json' \ + -d $'{"message": 1998.08,"time":"2024-05-25 20:16:37.217"}' + +{"error":"Failed to execute pipeline, reason: gsub processor: expect string or array string, but got Float64(1998.08)"} +``` + +The output indicates that the pipeline processing failed because the `gsub` Processor expects a string type rather than a floating-point number type. We need to adjust the format of the log data to ensure the pipeline can process it correctly. +Let's change the value of the message field to a string type and test the pipeline again. + +```bash +curl -X "POST" "http://localhost:4000/v1/events/pipelines/dryrun?pipeline_name=test" \ + -H 'Content-Type: application/json' \ + -d $'{"message": "1998.08","time":"2024-05-25 20:16:37.217"}' +``` + +At this point, the Pipeline processing is successful, and the output is as follows: + +```json +{ + "rows": [ + [ + { + "data_type": "STRING", + "key": "message", + "semantic_type": "FIELD", + "value": "1998-08" + }, + { + "data_type": "TIMESTAMP_NANOSECOND", + "key": "time", + "semantic_type": "TIMESTAMP", + "value": "2024-05-25 20:16:37.217+0000" + } + ] + ], + "schema": [ + { + "colume_type": "FIELD", + "data_type": "STRING", + "fulltext": false, + "name": "message" + }, + { + "colume_type": "TIMESTAMP", + "data_type": "TIMESTAMP_NANOSECOND", + "fulltext": false, + "name": "time" + } + ] +} +``` + +It can be seen that the `.` in the string `1998.08` has been replaced with `-`, indicating a successful processing of the Pipeline. \ No newline at end of file diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md index add6917bd..da5a68870 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md @@ -212,4 +212,126 @@ curl -X "POST" "http://localhost:4000/v1/events/pipelines/dryrun?pipeline_name=t } ``` -输出显示该 Pipeline 成功处理了日志数据。`rows` 字段包含已处理数据，`schema` 字段包含已处理数据的模式信息。您可以使用这些信息来验证 Pipeline 配置的正确性。 \ No newline at end of file +输出显示该 Pipeline 成功处理了日志数据。`rows` 字段包含已处理数据，`schema` 字段包含已处理数据的模式信息。您可以使用这些信息来验证 Pipeline 配置的正确性。 + +### 测试 Pipeline 失败 + +接下来，我们测试一个失败的 Pipeline。假设我们的 Pipeline 配置文件如下： + +```bash +curl -X "POST" "http://localhost:4000/v1/events/pipelines/test" \ + -H 'Content-Type: application/x-yaml' \ + -d $'processors: + - date: + field: time + formats: + - "%Y-%m-%d %H:%M:%S%.3f" + ignore_missing: true + - gsub: + fields: + - message + pattern: "\\\." + replacement: + - "-" + ignore_missing: true + +transform: + - fields: + - message + type: string + - field: time + type: time + index: timestamp' +``` + +Pipeline 配置存在错误。`gsub` processor 期望 `replacement` 字段为字符串，但当前配置提供了一个数组。因此，管道创建失败，并显示以下错误消息： + + +```json +{"error":"Failed to parse pipeline: 'replacement' must be a string"} +``` + +因此，您需要修改 `gsub` processor 的配置，将 `replacement` 字段的值更改为字符串类型。 + +```bash +curl -X "POST" "http://localhost:4000/v1/events/pipelines/test" \ + -H 'Content-Type: application/x-yaml' \ + -d $'processors: + - date: + field: time + formats: + - "%Y-%m-%d %H:%M:%S%.3f" + ignore_missing: true + - gsub: + fields: + - message + pattern: "\\\." + replacement: "-" + ignore_missing: true + +transform: + - fields: + - message + type: string + - field: time + type: time + index: timestamp' +``` + +此时 Pipeline 创建成功，可以使用 `dryrun` 接口测试该 Pipeline。我们使用一个错误的日志数据来测试，message 字段的值为数字类型，这会导致 Pipeline 处理失败。 + +```bash +curl -X "POST" "http://localhost:4000/v1/events/pipelines/dryrun?pipeline_name=test" \ + -H 'Content-Type: application/json' \ + -d $'{"message": 1998.08,"time":"2024-05-25 20:16:37.217"}' + +{"error":"Failed to execute pipeline, reason: gsub processor: expect string or array string, but got Float64(1998.08)"} +``` + +输出显示 Pipeline 处理失败，因为 `gsub` Processor 期望的是字符串类型，而不是浮点数类型。我们需要修改日志数据的格式，确保 Pipeline 能够正确处理。 +我们再将 message 字段的值修改为字符串类型，然后再次测试该 Pipeline。 + +```bash +curl -X "POST" "http://localhost:4000/v1/events/pipelines/dryrun?pipeline_name=test" \ + -H 'Content-Type: application/json' \ + -d $'{"message": "1998.08","time":"2024-05-25 20:16:37.217"}' +``` + +此时 Pipeline 处理成功，输出如下： + +```json +{ + "rows": [ + [ + { + "data_type": "STRING", + "key": "message", + "semantic_type": "FIELD", + "value": "1998-08" + }, + { + "data_type": "TIMESTAMP_NANOSECOND", + "key": "time", + "semantic_type": "TIMESTAMP", + "value": "2024-05-25 20:16:37.217+0000" + } + ] + ], + "schema": [ + { + "colume_type": "FIELD", + "data_type": "STRING", + "fulltext": false, + "name": "message" + }, + { + "colume_type": "TIMESTAMP", + "data_type": "TIMESTAMP_NANOSECOND", + "fulltext": false, + "name": "time" + } + ] +} +``` + +可以看到，`1998.08` 字符串中的 `.` 已经被替换为 `-`，Pipeline 处理成功。 \ No newline at end of file From 5089993678c6a39002a284e918d299429c577df3 Mon Sep 17 00:00:00 2001 From: paomian Date: Tue, 10 Sep 2024 11:45:34 +0800 Subject: [PATCH 3/7] chore: changes to pipeline-related text. increase readability of documentation --- docs/user-guide/logs/manage-pipelines.md | 133 +---------------- .../user-guide/logs/manage-pipelines.md | 137 ++---------------- 2 files changed, 18 insertions(+), 252 deletions(-) diff --git a/docs/user-guide/logs/manage-pipelines.md b/docs/user-guide/logs/manage-pipelines.md index 5899c63d2..170cdc316 100644 --- a/docs/user-guide/logs/manage-pipelines.md +++ b/docs/user-guide/logs/manage-pipelines.md @@ -89,132 +89,11 @@ Readable timestamp (UTC): 2024-06-27 12:02:34.257312110Z The output `Readable timestamp (UTC)` represents the creation time of the pipeline and also serves as the version number. -## Test a Pipeline +## Debug -First, create a pipeline using the following command: +First, please refer to the [Quick Start example](/user-guide/logs/quick-start.md#write-logs-by-pipeline) to see the correct execution of the Pipeline. -```shell -curl -X "POST" "http://localhost:4000/v1/events/pipelines/test?db=public" \ - -H 'Content-Type: application/x-yaml' \ - -d $'processors: - - date: - field: time - formats: - - "%Y-%m-%d %H:%M:%S%.3f" - ignore_missing: true - -transform: - - fields: - - id1 - - id2 - type: int32 - - fields: - - type - - log - - logger - type: string - - field: time - type: time - index: timestamp' -{"pipelines":[{"name":"test","version":"2024-09-03 08:47:38.686403312"}],"execution_time_ms":2} -``` - -Then, test the pipeline using the following command: - -```shell -curl -X "POST" "http://localhost:4000/v1/events/pipelines/dryrun?pipeline_name=test" \ - -H 'Content-Type: application/json' \ - -d $'{"time":"2024-05-25 20:16:37.217","id1":"2436","id2":"2528","type":"I","logger":"INTERACT.MANAGER","log":"ClusterAdapter:enter sendTextDataToCluster\\n"}' -``` - -The output is as follows: - -```json -{ - "rows": [ - [ - { - "data_type": "INT32", - "key": "id1", - "semantic_type": "FIELD", - "value": 2436 - }, - { - "data_type": "INT32", - "key": "id2", - "semantic_type": "FIELD", - "value": 2528 - }, - { - "data_type": "STRING", - "key": "type", - "semantic_type": "FIELD", - "value": "I" - }, - { - "data_type": "STRING", - "key": "log", - "semantic_type": "FIELD", - "value": "ClusterAdapter:enter sendTextDataToCluster\n" - }, - { - "data_type": "STRING", - "key": "logger", - "semantic_type": "FIELD", - "value": "INTERACT.MANAGER" - }, - { - "data_type": "TIMESTAMP_NANOSECOND", - "key": "time", - "semantic_type": "TIMESTAMP", - "value": "2024-05-25 20:16:37.217+0000" - } - ] - ], - "schema": [ - { - "colume_type": "FIELD", - "data_type": "INT32", - "fulltext": false, - "name": "id1" - }, - { - "colume_type": "FIELD", - "data_type": "INT32", - "fulltext": false, - "name": "id2" - }, - { - "colume_type": "FIELD", - "data_type": "STRING", - "fulltext": false, - "name": "type" - }, - { - "colume_type": "FIELD", - "data_type": "STRING", - "fulltext": false, - "name": "log" - }, - { - "colume_type": "FIELD", - "data_type": "STRING", - "fulltext": false, - "name": "logger" - }, - { - "colume_type": "TIMESTAMP", - "data_type": "TIMESTAMP_NANOSECOND", - "fulltext": false, - "name": "time" - } - ] -} -``` - -The output shows that the pipeline successfully processed the log data. The `rows` field contains the processed data, and the `schema` field contains the schema information of the processed data. You can use this information to verify the correctness of the pipeline configuration. - -### Test a Failed Pipeline +### Debug create a Pipeline Assume that the pipeline configuration is as follows: @@ -279,7 +158,11 @@ transform: index: timestamp' ``` -The Pipeline has been successfully created at this point, and We can test the Pipeline using the `dryrun` interface. We will test it with erroneous log data where the value of the message field is in numeric format, causing the pipeline to fail during processing. +The Pipeline has been successfully created at this point. + +#### Debug writing logs + +We can test the Pipeline using the `dryrun` interface. We will test it with erroneous log data where the value of the message field is in numeric format, causing the pipeline to fail during processing. ```bash diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md index da5a68870..6e18e6851 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md @@ -89,134 +89,13 @@ Readable timestamp (UTC): 2024-06-27 12:02:34.257312110Z 输出的 `Readable timestamp (UTC)` 即为 Pipeline 的创建时间同时也是版本号。 -## 测试 Pipeline +## 问题调试 -首先，使用以下命令创建一个 Pipeline： +首先，请参考 [快速入门示例](/user-guide/logs/quick-start.md#使用-pipeline-写入日志)来查看 Pipeline 正确的执行情况。 -```shell -curl -X "POST" "http://localhost:4000/v1/events/pipelines/test?db=public" \ - -H 'Content-Type: application/x-yaml' \ - -d $'processors: - - date: - field: time - formats: - - "%Y-%m-%d %H:%M:%S%.3f" - ignore_missing: true - -transform: - - fields: - - id1 - - id2 - type: int32 - - fields: - - type - - log - - logger - type: string - - field: time - type: time - index: timestamp' -{"pipelines":[{"name":"test","version":"2024-09-03 08:47:38.686403312"}],"execution_time_ms":2} -``` - -然后，使用以下命令测试该 Pipeline： - -```shell -curl -X "POST" "http://localhost:4000/v1/events/pipelines/dryrun?pipeline_name=test" \ - -H 'Content-Type: application/json' \ - -d $'{"time":"2024-05-25 20:16:37.217","id1":"2436","id2":"2528","type":"I","logger":"INTERACT.MANAGER","log":"ClusterAdapter:enter sendTextDataToCluster\\n"}' -``` - -输出如下： - -```json -{ - "rows": [ - [ - { - "data_type": "INT32", - "key": "id1", - "semantic_type": "FIELD", - "value": 2436 - }, - { - "data_type": "INT32", - "key": "id2", - "semantic_type": "FIELD", - "value": 2528 - }, - { - "data_type": "STRING", - "key": "type", - "semantic_type": "FIELD", - "value": "I" - }, - { - "data_type": "STRING", - "key": "log", - "semantic_type": "FIELD", - "value": "ClusterAdapter:enter sendTextDataToCluster\n" - }, - { - "data_type": "STRING", - "key": "logger", - "semantic_type": "FIELD", - "value": "INTERACT.MANAGER" - }, - { - "data_type": "TIMESTAMP_NANOSECOND", - "key": "time", - "semantic_type": "TIMESTAMP", - "value": "2024-05-25 20:16:37.217+0000" - } - ] - ], - "schema": [ - { - "colume_type": "FIELD", - "data_type": "INT32", - "fulltext": false, - "name": "id1" - }, - { - "colume_type": "FIELD", - "data_type": "INT32", - "fulltext": false, - "name": "id2" - }, - { - "colume_type": "FIELD", - "data_type": "STRING", - "fulltext": false, - "name": "type" - }, - { - "colume_type": "FIELD", - "data_type": "STRING", - "fulltext": false, - "name": "log" - }, - { - "colume_type": "FIELD", - "data_type": "STRING", - "fulltext": false, - "name": "logger" - }, - { - "colume_type": "TIMESTAMP", - "data_type": "TIMESTAMP_NANOSECOND", - "fulltext": false, - "name": "time" - } - ] -} -``` - -输出显示该 Pipeline 成功处理了日志数据。`rows` 字段包含已处理数据，`schema` 字段包含已处理数据的模式信息。您可以使用这些信息来验证 Pipeline 配置的正确性。 - -### 测试 Pipeline 失败 +### 调试创建 Pipeline -接下来，我们测试一个失败的 Pipeline。假设我们的 Pipeline 配置文件如下： +在创建 Pipeline 的时候你可能会遇到错误，例如使用如下配置创建 Pipeline： ```bash curl -X "POST" "http://localhost:4000/v1/events/pipelines/test" \ @@ -251,7 +130,7 @@ Pipeline 配置存在错误。`gsub` processor 期望 `replacement` 字段为字 {"error":"Failed to parse pipeline: 'replacement' must be a string"} ``` -因此，您需要修改 `gsub` processor 的配置，将 `replacement` 字段的值更改为字符串类型。 +因此，你需要修改 `gsub` processor 的配置，将 `replacement` 字段的值更改为字符串类型。 ```bash curl -X "POST" "http://localhost:4000/v1/events/pipelines/test" \ @@ -278,7 +157,11 @@ transform: index: timestamp' ``` -此时 Pipeline 创建成功，可以使用 `dryrun` 接口测试该 Pipeline。我们使用一个错误的日志数据来测试，message 字段的值为数字类型，这会导致 Pipeline 处理失败。 +此时 Pipeline 创建成功，可以使用 `dryrun` 接口测试该 Pipeline。 + +#### 调试日志写入 + +接下来使用一个错误的日志数据来测试，该日志中 message 字段的值为数字类型，这会导致 Pipeline 处理失败。 ```bash curl -X "POST" "http://localhost:4000/v1/events/pipelines/dryrun?pipeline_name=test" \ From d6ee5cda1061532cfad3c421d518d7d3a4efe6ea Mon Sep 17 00:00:00 2001 From: paomian Date: Tue, 10 Sep 2024 11:50:02 +0800 Subject: [PATCH 4/7] chore: fix by pr commit --- .../current/user-guide/logs/manage-pipelines.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md index 6e18e6851..1cc6d7778 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md @@ -123,7 +123,7 @@ transform: index: timestamp' ``` -Pipeline 配置存在错误。`gsub` processor 期望 `replacement` 字段为字符串，但当前配置提供了一个数组。因此，管道创建失败，并显示以下错误消息： +Pipeline 配置存在错误。`gsub` processor 期望 `replacement` 字段为字符串，但当前配置提供了一个数组。因此，该 Pipeline 创建失败，并显示以下错误消息： ```json From 8f4065394ed2b0c2f31bbc28e144fbcd1114c4e4 Mon Sep 17 00:00:00 2001 From: paomian Date: Tue, 10 Sep 2024 18:58:04 +0800 Subject: [PATCH 5/7] chore: fix by pr commit --- docs/user-guide/logs/manage-pipelines.md | 6 +++--- .../current/user-guide/logs/manage-pipelines.md | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/user-guide/logs/manage-pipelines.md b/docs/user-guide/logs/manage-pipelines.md index 170cdc316..f524cc23d 100644 --- a/docs/user-guide/logs/manage-pipelines.md +++ b/docs/user-guide/logs/manage-pipelines.md @@ -93,9 +93,9 @@ The output `Readable timestamp (UTC)` represents the creation time of the pipeli First, please refer to the [Quick Start example](/user-guide/logs/quick-start.md#write-logs-by-pipeline) to see the correct execution of the Pipeline. -### Debug create a Pipeline +### Debug creating a Pipeline -Assume that the pipeline configuration is as follows: +You may encounter errors when creating a Pipeline. For example, when creating a Pipeline using the following configuration: ```bash @@ -160,7 +160,7 @@ transform: The Pipeline has been successfully created at this point. -#### Debug writing logs +### Debug writing logs We can test the Pipeline using the `dryrun` interface. We will test it with erroneous log data where the value of the message field is in numeric format, causing the pipeline to fail during processing. diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md index 1cc6d7778..d3dbbd438 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md @@ -159,7 +159,7 @@ transform: 此时 Pipeline 创建成功，可以使用 `dryrun` 接口测试该 Pipeline。 -#### 调试日志写入 +### 调试日志写入 接下来使用一个错误的日志数据来测试，该日志中 message 字段的值为数字类型，这会导致 Pipeline 处理失败。 From e1ac887841aab57b99fcfa9425edad37e12565af Mon Sep 17 00:00:00 2001 From: paomian Date: Tue, 10 Sep 2024 19:04:18 +0800 Subject: [PATCH 6/7] chore: add note for pipeline dryrun api --- docs/user-guide/logs/manage-pipelines.md | 4 +++- .../current/user-guide/logs/manage-pipelines.md | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/logs/manage-pipelines.md b/docs/user-guide/logs/manage-pipelines.md index f524cc23d..840894c2f 100644 --- a/docs/user-guide/logs/manage-pipelines.md +++ b/docs/user-guide/logs/manage-pipelines.md @@ -158,12 +158,14 @@ transform: index: timestamp' ``` -The Pipeline has been successfully created at this point. +Now that the Pipeline has been created successfully, you can test the Pipeline using the `dryrun` interface. ### Debug writing logs We can test the Pipeline using the `dryrun` interface. We will test it with erroneous log data where the value of the message field is in numeric format, causing the pipeline to fail during processing. +**This API is only used to test the results of the Pipeline and does not write data to GreptimeDB.** + ```bash curl -X "POST" "http://localhost:4000/v1/events/pipelines/dryrun?pipeline_name=test" \ diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md index d3dbbd438..676834bef 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md @@ -161,7 +161,9 @@ transform: ### 调试日志写入 -接下来使用一个错误的日志数据来测试，该日志中 message 字段的值为数字类型，这会导致 Pipeline 处理失败。 +我们可以使用 `dryrun` 接口测试 Pipeline。我们将使用错误的日志数据对其进行测试，其中消息字段的值为数字格式，会导致 Pipeline 在处理过程中失败。 + +**此接口仅仅用于测试 Pipeline 的处理结果，不会将数据写入到 GreptimeDB 中。** ```bash curl -X "POST" "http://localhost:4000/v1/events/pipelines/dryrun?pipeline_name=test" \ From 1cc00005f834bb4991f1da8330c8b5d46437b10e Mon Sep 17 00:00:00 2001 From: Yiran Date: Wed, 11 Sep 2024 10:49:03 +0800 Subject: [PATCH 7/7] Apply suggestions from code review --- docs/user-guide/logs/manage-pipelines.md | 2 +- .../current/user-guide/logs/manage-pipelines.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/user-guide/logs/manage-pipelines.md b/docs/user-guide/logs/manage-pipelines.md index 840894c2f..67209a053 100644 --- a/docs/user-guide/logs/manage-pipelines.md +++ b/docs/user-guide/logs/manage-pipelines.md @@ -164,7 +164,7 @@ Now that the Pipeline has been created successfully, you can test the Pipeline u We can test the Pipeline using the `dryrun` interface. We will test it with erroneous log data where the value of the message field is in numeric format, causing the pipeline to fail during processing. -**This API is only used to test the results of the Pipeline and does not write data to GreptimeDB.** +**This API is only used to test the results of the Pipeline and does not write logs to GreptimeDB.** ```bash diff --git a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md index 676834bef..5d6514072 100644 --- a/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md +++ b/i18n/zh/docusaurus-plugin-content-docs/current/user-guide/logs/manage-pipelines.md @@ -163,7 +163,7 @@ transform: 我们可以使用 `dryrun` 接口测试 Pipeline。我们将使用错误的日志数据对其进行测试，其中消息字段的值为数字格式，会导致 Pipeline 在处理过程中失败。 -**此接口仅仅用于测试 Pipeline 的处理结果，不会将数据写入到 GreptimeDB 中。** +**此接口仅仅用于测试 Pipeline 的处理结果，不会将日志写入到 GreptimeDB 中。** ```bash curl -X "POST" "http://localhost:4000/v1/events/pipelines/dryrun?pipeline_name=test" \