Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

更新的原子性 #3

Open
htfy96 opened this issue Aug 29, 2016 · 9 comments
Open

更新的原子性 #3

htfy96 opened this issue Aug 29, 2016 · 9 comments

Comments

@htfy96
Copy link
Contributor

htfy96 commented Aug 29, 2016

symbolic link替换

Symbolic link覆盖在主流Linux平台上为原子操作:cp -fT

实在要严格还可以renameat2:

RENAME_EXCHANGE
Atomically exchange oldpath and newpath. Both pathnames must exist but may be of different types (e.g., one could be a non-empty directory and the other a symbolic link).

所以主要问题就是如何CoW

CoW

CoW有几种实现的方式:

  • Tunasync之前的版本是基于Btrfs文件系统自带的特性。好处是简单、直接,缺陷是需要依赖特定的文件系统
  • 除了Btrfs之外,还可以有AUFS、Devicemapper、vfs、Zfs、overlay等方式搞。。(没错这个列表就是从Docker那边抄的)
  • 所以懒得打字了……抄了张图……
    2016-08-29 20-18-06

剩下具体怎么搞需要更多的调研……

原子更新有没有必要

+ 解决掉很多难搞的二段式更新问题,全部一段更新
- 额外的空间,稳定性……现在的lug不需要任何 privilege ,之所以用rlimit而没上cgroup就是因为这样
@shanzi
Copy link
Member

shanzi commented Aug 29, 2016

感觉 rsync 的痕迹已经不是很大了。感觉初期可以先不考虑原子性的问题。。。

@htfy96
Copy link
Contributor Author

htfy96 commented Aug 29, 2016

@shanzi 那其实现在这样就可以尝试上线跑一跑了(雾

@shanzi
Copy link
Member

shanzi commented Aug 29, 2016

@htfy96 来来来~不过你可以再在群里跟 B 哥讨论下,我听说它们现在也没有做原子性的维护,好像会引入很多复杂性。

@htfy96
Copy link
Contributor Author

htfy96 commented Aug 30, 2016

Overlayfs调研

https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt
https://wiki.archlinux.org/index.php/Overlay_filesystem

使用很简单。挂lowerdir=原镜像目录(只读),upperdir=一个空dir,workdir=一个临时空目录,都到/merge就行了。然后rsync 到/merge,差分的东西会被写到upperdir中。

比较麻烦的是怎么把更新后的目录/merge放回来……毕竟不能直接交换连接,否则会出现overlay叠overlay的情况。

Docker的做法是把源镜像目录的普通文件都hard link到/temp,然后再把upperdir的差分写回到/temp上去,最后用/temp替换到指向原镜像目录的link。

这样的结果可能就是会2x diffsize,而且大量hard link的代价是不是很捉急也是未知数。

Overlayfs CoW的基本单位是文件,而不是block,所以对大个文件的写入比较捉急,但是好处是实现简单。

对当前系统的影响:由于User namespace没进内核,要想允许mount要么授权,要么--privilege ,要么在外面搞一个server RPC把东西挂进去

@riaqn
Copy link

riaqn commented Aug 30, 2016

弱弱地问一下是什么东西的原子性? 是说一个repo的原子性? 上游有原子性吗?

2016-08-29 6:42 GMT-06:00 Zheng Luo notifications@github.com:

@shanzi https://github.com/shanzi 那其实现在这样就可以尝试上线跑一跑了(雾


You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#3 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACWSVgzigLvFjOe1KTSPRzCfM2-xWXvvks5qktPMgaJpZM4Jva8Z
.

@htfy96
Copy link
Contributor Author

htfy96 commented Aug 30, 2016

@riaqn repo的一致性由写入的原子性可以保证……当然也可以通过二段式更新糊过去……

下面是Debian的指南,注意到要么二段式更新糊,要么主动接收push:

If you do use your own scripts, the minimal functionality that must be supported is as follows:

MUST perform a 2-stage sync

The archive mirroring must be done in 2 stages. The first rsync run must ignore the index files. The correct exclude options for the first rsync run are:

   --exclude Packages* --exclude Sources* --exclude Release* --exclude InRelease --exclude i18n/* --exclude ls-lR*

The first stage must not delete any files.

The second stage should then transfer the above excluded files and delete files that no longer belong on the mirror.

Rationale: if archive mirroring is done in a single stage, there will be periods of time during which the index files will reference files not yet mirrored.

MUST not ignore pushes while running (for push mirroring).

@riaqn
Copy link

riaqn commented Aug 30, 2016

所以我们在讨论的原子性是?

2016-08-29 20:56 GMT-06:00 Zheng Luo notifications@github.com:

@riaqn https://github.com/riaqn repo的一致性由写入的原子性可以保证……当然也可以通过二段式更新糊过去……

下面是Debian的指南 https://www.debian.org/mirror/ftpmirror,注意到要么二段式更新糊,
要么主动接收push:

If you do use your own scripts, the minimal functionality that must be
supported is as follows:

MUST perform a 2-stage sync

The archive mirroring must be done in 2 stages. The first rsync run must
ignore the index files. The correct exclude options for the first rsync run
are:

--exclude Packages* --exclude Sources* --exclude Release* --exclude InRelease --exclude i18n/* --exclude ls-lR*

The first stage must not delete any files.

The second stage should then transfer the above excluded files and delete
files that no longer belong on the mirror.

Rationale: if archive mirroring is done in a single stage, there will be
periods of time during which the index files will reference files not yet
mirrored.

MUST not ignore pushes while running (for push mirroring).


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#3 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACWSVpiT_-dHrGluDJfaOKCmWntMM8S6ks5qk5v2gaJpZM4Jva8Z
.

@htfy96
Copy link
Contributor Author

htfy96 commented Aug 30, 2016

@riaqn
因为Rsync不能保证更新的顺序,所以可能出现先sync了Index,再sync了真正文件的情况,从而导致客户端先下载到了有8.0版本软件foo Index,但foo8.0还没有同步完成,造成文件下载失败。而

二段式更新先更新文件(Debian是保留旧版本包的),再更新Index。尽管这样还是有可能在更新文件和更新Index之间,Index发生了变化,但概率小了很多。

我们这个原子性是 希望使Index和真正文件在一个原子操作内更新,这样效果上和二段式更新等价,因为我们很难搞定上游的一致性……

@shanzi
Copy link
Member

shanzi commented Feb 6, 2018

考虑了一下,即使 Mirror 是原子更新的,还是有一定概率用户遇到一致性问题。

假定用户的一次install是一个session,这个session可能有多个请求,最初的请求可能是update local index,之后去访问package,如果在这中间出现原子的更新,还是有很微小的概率出现问题。

感觉一致性的问题,如果真的想开坑解决(just for fun)的话,可以考虑在 web server 的层面做 session sticky。保证一个session的所有请求访问的是同一个mirror snapshot。不过这个坑就比较大了,还涉及到文件系统的组织问题。

初步的话,原子更新用2 stage 就可以了。不过我最近在构思一个mirrors的大大大大坑,虽然不一定会真的下手做,然而感觉可以讨论下。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants