refactor: move remote WAL under the disaster recovery #1186

nicecui · 2024-09-19T08:11:57Z

What's Changed in this PR

as title

Checklist

Please confirm that all corresponding versions of the documents have been revised.
Please ensure that the content in sidebars.ts matches the current document structure when you changed the document structure.
This change requires follow-up update in localized docs.

coderabbitai · 2024-09-19T08:12:05Z

Important

Review skipped

Auto reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share

Tips

Chat

There are 3 ways to chat with CodeRabbit:

Review comments: Directly reply to a review comment made by CodeRabbit. Example:
-- I pushed a fix in commit <commit_id>, please review it.
-- Generate unit testing code for this file.
- Open a follow-up GitHub issue for this discussion.
Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
-- @coderabbitai generate unit testing code for this file.
-- @coderabbitai modularize this function.
PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
-- @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
-- @coderabbitai read src/utils.ts and generate unit testing code.
-- @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
-- @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

@coderabbitai pause to pause the reviews on a PR.
@coderabbitai resume to resume the paused reviews.
@coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
@coderabbitai full review to do a full review from scratch and review all the files again.
@coderabbitai summary to regenerate the summary of the PR.
@coderabbitai resolve resolve all the CodeRabbit review comments.
@coderabbitai configuration to show the current CodeRabbit configuration for the repository.
@coderabbitai help to get help.

Other keywords and placeholders

Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (`.coderabbit.yaml`)

You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
Please see the configuration documentation for more information.
If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

Visit our Documentation for detailed information on how to use CodeRabbit.
Join our Discord Community to get help, request features, and share feedback.
Follow us on X/Twitter for updates and announcements.

cloudflare-workers-and-pages · 2024-09-19T08:19:05Z

Deploying greptime-docs with Cloudflare Pages

Latest commit:	`d77ab7a`
Status:	✅ Deploy successful!
Preview URL:	https://8ba49fb3.greptime-docs.pages.dev
Branch Preview URL:	https://oper-dr.greptime-docs.pages.dev

View logs

sunng87 · 2024-09-19T17:17:54Z

remote wal is not for DR only, we'd better keep it standalone.

fengjiachun · 2024-09-20T04:11:50Z

i18n/zh/docusaurus-plugin-content-docs/current.json

@@ -64,7 +64,7 @@
    "description": "The label for category Disaster Recovery in sidebar docs"
  },
  "sidebar.docs.category.Remote WAL": {
-    "message": "Remote WAL",
+    "message": "远程 WAL",


Remote is fine; translation isn't needed.

fengjiachun · 2024-09-20T04:12:05Z

i18n/zh/docusaurus-plugin-content-docs/current/user-guide/operations/admin.md

@@ -36,7 +36,7 @@ ORDER BY datanode_id ASC
 ## 数据管理

 * [存储位置说明](/user-guide/concepts/storage-location.md)。
-* 通过 [设置Remote WAL](./remote-wal/quick-start.md) 实现 GreptimeDB 的集群容灾。
+* 通过 [设置Remote WAL](/user-guide/operations/disaster-recovery/remote-wal/quick-start.md) 实现 GreptimeDB 的集群容灾。


Suggested change

* 通过 [设置Remote WAL](/user-guide/operations/disaster-recovery/remote-wal/quick-start.md) 实现 GreptimeDB 的集群容灾。

* 通过 [设置 Remote WAL](/user-guide/operations/disaster-recovery/remote-wal/quick-start.md) 实现 GreptimeDB 的集群容灾。

fengjiachun · 2024-09-20T04:12:46Z

...h/docusaurus-plugin-content-docs/current/user-guide/operations/disaster-recovery/overview.md

@@ -42,7 +42,7 @@ GreptimeDB 基于存储计算分离的云原生架构设计：

 * **Frontend**：数据插入和查询的服务层，将请求转发到 Datanode 并处理和合并 Datanode 的响应。
 * **Datanode**：GreptimeDB 的存储层，是一个 LSM 存储引擎。Region 是在 Datanode 中存储和调度数据的基本单元。Region 是一个表分区，是一组数据行的集合。Region 中的数据保存在对象存储中（例如 AWS S3）。未刷新的 Memtable 数据被写入 WAL，并可以在灾难发生时恢复。
-* **WAL**：持久化内存中未刷新的 Memtable 数据。当 Memtable 被刷新到 SSTable 文件时，WAL 将被截断。它可以是基于本地磁盘的（本地 WAL）或基于 Kafka 集群的（远程 WAL）。
+* **WAL**：持久化内存中未刷新的 Memtable 数据。当 Memtable 被刷新到 SSTable 文件时，WAL 将被截断。它可以是基于本地磁盘的（本地 WAL）或[基于 Kafka 集群的（远程 WAL）](./remote-wal/quick-start.md)。


Suggested change

* **WAL**：持久化内存中未刷新的 Memtable 数据。当 Memtable 被刷新到 SSTable 文件时，WAL 将被截断。它可以是基于本地磁盘的（本地 WAL）或[基于 Kafka 集群的（远程 WAL）](./remote-wal/quick-start.md)。

* **WAL**：持久化内存中未刷新的 Memtable 数据。当 Memtable 被刷新到 SSTable 文件时，WAL 将被截断。它可以是基于本地磁盘的（Local WAL）或[基于 Kafka 集群的（Remote WAL）](./remote-wal/quick-start.md)。

fengjiachun · 2024-09-20T04:13:12Z

...lugin-content-docs/current/user-guide/operations/disaster-recovery/remote-wal/quick-start.md

@@ -1,13 +1,13 @@
 # 快速开始

-## 什么是 Remote WAL
+## 什么是远程 WAL（Remote WAL）


Suggested change

## 什么是远程 WAL（Remote WAL）

## 什么是 Remote WAL

fengjiachun · 2024-09-20T04:13:28Z

...lugin-content-docs/current/user-guide/operations/disaster-recovery/remote-wal/quick-start.md


 [WAL](/contributor-guide/datanode/wal.md#introduction)(Write-Ahead Logging) 是 GreptimeDB 中的一个关键组件，它持久记录每一次数据修改，以确保不会丢失缓存在内存中的数据。我们在  [Datanode](/user-guide/concepts/why-greptimedb.md)  服务中用持久的嵌入式存储引擎 [raft-engine](https://github.com/tikv/raft-engine) 将 WAL 实现为一个模块。在公共云中部署 GreptimeDB 时，我们可以在云存储（AWS EBS、GCP 持久盘等）中持久存储 WAL 数据，以实现 0 RPO。然而，由于 WAL 与 Datanode 紧密耦合，导致部署过程中的 RTO（Recovery Time Objective）较长。此外，由于 raft-engine 无法支持多日志订阅，这使得实现 region 热备份和 region 迁移变得困难。

 为了解决上述问题，我们决定设计并实现一个远程 WAL。远程 WAL 将 WAL 从 Datanode 分离到远程服务，我们选择了 Apache Kafka 作为远程服务。Apache Kafka 在流处理中被广泛采用，展现出卓越的分布式容错能力和基于主题的订阅机制。在发布 v0.5.0 版本时，我们引入了 Apache Kafka 作为 WAL 的可选存储引擎。


-## 运行带有 Remote WAL 的 Standalone GreptimeDB
+## 运行带有远程 WAL 的 Standalone GreptimeDB


Suggested change

## 运行带有远程 WAL 的 Standalone GreptimeDB

## 运行带有 Remote WAL 的 Standalone GreptimeDB

nicecui · 2024-09-23T07:37:19Z

remote wal is not for DR only, we'd better keep it standalone.

nicecui added 8 commits September 19, 2024 11:13

the deployments navigation

ae4e62c

the zh docs

195be02

polish the content

16e9e5c

fix dead links

e408538

en overview for the deployments section

a650a95

zh overview

ec849c0

typo

91c67b4

move remote WAL under the disaster recovery

d77ab7a

nicecui requested review from killme2008 and sunng87 September 19, 2024 08:11

nicecui requested a review from fengjiachun as a code owner September 19, 2024 08:11

fengjiachun reviewed Sep 20, 2024

View reviewed changes

Base automatically changed from deployments to main September 20, 2024 06:37

nicecui closed this Sep 23, 2024

nicecui deleted the oper/dr branch October 18, 2024 06:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: move remote WAL under the disaster recovery #1186

refactor: move remote WAL under the disaster recovery #1186

nicecui commented Sep 19, 2024

coderabbitai bot commented Sep 19, 2024

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (`.coderabbit.yaml`)

Documentation and Community

cloudflare-workers-and-pages bot commented Sep 19, 2024

sunng87 commented Sep 19, 2024

fengjiachun Sep 20, 2024

fengjiachun Sep 20, 2024

fengjiachun Sep 20, 2024

fengjiachun Sep 20, 2024

fengjiachun Sep 20, 2024

nicecui commented Sep 23, 2024

	* 通过 [设置Remote WAL](/user-guide/operations/disaster-recovery/remote-wal/quick-start.md) 实现 GreptimeDB 的集群容灾。
	* 通过 [设置 Remote WAL](/user-guide/operations/disaster-recovery/remote-wal/quick-start.md) 实现 GreptimeDB 的集群容灾。

	* WAL：持久化内存中未刷新的 Memtable 数据。当 Memtable 被刷新到 SSTable 文件时，WAL 将被截断。它可以是基于本地磁盘的（本地 WAL）或[基于 Kafka 集群的（远程 WAL）](./remote-wal/quick-start.md)。
	* WAL：持久化内存中未刷新的 Memtable 数据。当 Memtable 被刷新到 SSTable 文件时，WAL 将被截断。它可以是基于本地磁盘的（Local WAL）或[基于 Kafka 集群的（Remote WAL）](./remote-wal/quick-start.md)。

	## 什么是远程 WAL（Remote WAL）
	## 什么是 Remote WAL

	## 运行带有远程 WAL 的 Standalone GreptimeDB
	## 运行带有 Remote WAL 的 Standalone GreptimeDB

refactor: move remote WAL under the disaster recovery #1186

refactor: move remote WAL under the disaster recovery #1186

Conversation

nicecui commented Sep 19, 2024

What's Changed in this PR

Checklist

coderabbitai bot commented Sep 19, 2024

Review skipped

Chat

CodeRabbit Commands (Invoked using PR comments)

Other keywords and placeholders

CodeRabbit Configuration File (.coderabbit.yaml)

Documentation and Community

cloudflare-workers-and-pages bot commented Sep 19, 2024

Deploying greptime-docs with Cloudflare Pages

sunng87 commented Sep 19, 2024

fengjiachun Sep 20, 2024

Choose a reason for hiding this comment

fengjiachun Sep 20, 2024

Choose a reason for hiding this comment

fengjiachun Sep 20, 2024

Choose a reason for hiding this comment

fengjiachun Sep 20, 2024

Choose a reason for hiding this comment

fengjiachun Sep 20, 2024

Choose a reason for hiding this comment

nicecui commented Sep 23, 2024

CodeRabbit Configuration File (`.coderabbit.yaml`)