Skip to content

QuickStart

Jump to bottom
agapple edited this page Mar 4, 2016 · 2 revisions

几点说明

a. 数据迁移的方案可参见设计文档，oracle全量基于JDBC拉取数据，增量基于物化视图来实现，所以这里需要给oracle数据库账号开启特殊权限

GRANT SELECT,INSERT,UPDATE,DELETE ON XXX TO XXX; #常见CRUD权限
GRANT CREATE ANY MATERIALIZED VIEW TO XXX;
GRANT DROP ANY MATERIALIZED VIEW TO XXX;

启动步骤

1. 下载yugong

直接下载，可访问：https://github.com/alibaba/yugong/releases，会列出所有历史的发布版本包的下载，比如以1.0.0版本为例子：

wget https://github.com/alibaba/yugong/releases/download/yugong-1.0.0/yugong-1.0.0.tar.gz

 or

自己编译

git clone git@github.com:alibaba/yugong.git
cd yugong; 
mvn clean install -Dmaven.test.skip -Denv=release

 编译完成后，会在根目录下产生target/yugong-$version.tar.gz

2. 解压缩

mkdir /tmp/yugong
tar zxvf yugong-$version.tar.gz  -C /tmp/yugong

3. 配置修改

vi conf/yugong.properties
# 源数据库,需要修改成自己的数据库信息
yugong.database.source.username=test
yugong.database.source.password=test
yugong.database.source.type=ORACLE
yugong.database.source.url=jdbc:oracle:thin:@127.0.0.1:1521:test
yugong.database.source.encode=UTF-8
yugong.database.source.poolSize=30

目标数据库,需要修改成自己的数据库信息


yugong.database.target.url=jdbc:mysql://127.0.0.1:3306/test
yugong.database.target.username=test
yugong.database.target.password=test
yugong.database.target.type=DRDS
yugong.database.target.encode=UTF-8
yugong.database.target.poolSize=30


yugong.table.batchApply=true
yugong.table.onceCrawNum=1000
yugong.table.tpsLimit=0


use connection default schema


yugong.table.ignoreSchema=false


skip Applier Load Db failed data


yugong.table.skipApplierException=false


需要迁移的数据库表


yugong.table.white=yugong_example_oracle
yugong.table.black=


tables use multi-thread enable or disable


yugong.table.concurrent.enable=true


tables use multi-thread size


yugong.table.concurrent.size=5


retry times


yugong.table.retry.times = 3


retry interval or sleep time (ms)


yugong.table.retry.interval = 1000


MARK/FULL/INC/ALL(REC+FULL+INC)/CHECK/CLEAR


yugong.table.mode=ALL


yugong extractor


yugong.extractor.dump=false
yugong.extractor.concurrent.enable=true
yugong.extractor.concurrent.size=20
yugong.extractor.noupdate.sleep=1000
yugong.extractor.noupdate.thresold=0
yugong.extractor.once=false


{0} is all columns , {1}.{2} is schemaName.tableName , {3} is primaryKey


#yugong.extractor.sql=select /+parallel(t)/ {0} from {1}.{2} t
#yugong.extractor.sql=select * from (select {0} from {1}.{2} t where {3} > ? order by {3} asc) where rownum <= ?


yugong applier


yugong.applier.concurrent.enable=true
yugong.applier.concurrent.size=20
yugong.applier.dump=false


stats


yugong.stat.print.interval=5
yugong.progress.print.interval=1


alarm email


yugong.alarm.email.host = smtp.163.com
yugong.alarm.email.username = test@163.com
yugong.alarm.email.password =
yugong.alarm.email.stmp.port = 465

 说明：

  1. 需要修改源和目标数据库的账号信息
  2. 需要修改yugong.table.white信息，登记需要同步的测试表

4. 准备测试表

a.  在源库oracle上创建一张待同步表

create table yugong_example_oracle
 (   
     id NUMBER(11)  ,
     name varchar2(32) ,
     alias_name  char(32) default ' ' not null,
     amount number(11,2),
     score  number(20), 
     text_b blob,
     text_c clob,
     gmt_create date not null,
     gmt_modified date not null,
     CONSTRAINT yugong_example_oracle_pk_id  PRIMARY   KEY (id) 
 );

 b.  在目标库MySQL上创建一张目标表

create table yugong_example_mysql
 (   
     id bigint(20) unsigned auto_increment,
     display_name varchar(128) ,
     amount varchar(32),
     score bigint(20) unsigned , 
     text_b blob,
     text_c text,
     gmt_create timestamp not null,
     gmt_modified timestamp not null,
     gmt_move timestamp not null,
     CONSTRAINT yugong_example_mysql_pk_id  PRIMARY KEY (id) 
 );

 c.  在源库构造几条测试数据

insert into yugong_example_oracle values(1,'ljh','agapple',10.2,100, NULL , NULL ,sysdate,sysdate)
insert into yugong_example_oracle values(2,'yugong','yugong',16.88,2088, NULL , NULL ,sysdate,sysdate)

5. 准备启动 

sh bin/startup.sh

6. 查看日志

查看总日志

vi logs/yugong/table.log
2016-03-05 01:33:02.714 [main] INFO  com.taobao.yugong.YuGongLauncher - ## start the YuGong.
2016-03-05 01:33:02.835 [main] INFO  com.taobao.yugong.controller.YuGongController - check source database connection ... 
2016-03-05 01:33:02.859 [main] INFO  com.taobao.yugong.controller.YuGongController - check source database is ok
2016-03-05 01:33:02.859 [main] INFO  com.taobao.yugong.controller.YuGongController - check target database connection ... 
2016-03-05 01:33:02.880 [main] INFO  com.taobao.yugong.controller.YuGongController - check target database is ok
2016-03-05 01:33:02.882 [main] INFO  com.taobao.yugong.controller.YuGongController - check source tables read privileges ... 
2016-03-05 01:33:03.027 [main] INFO  com.alibaba.druid.pool.DruidDataSource - {dataSource-1} inited
2016-03-05 01:33:03.550 [main] INFO  com.taobao.yugong.controller.YuGongController - check source tables is ok. 
2016-03-05 01:33:04.213 [main] INFO  com.taobao.yugong.controller.YuGongController - ## prepare start tables[1] with concurrent[5]
2016-03-05 01:33:04.685 [YuGongInstance-ALIBABA.YUGONG_EXAMPLE_ORACLE] INFO  com.taobao.yugong.controller.YuGongInstance - table[ALIBABA.YUGONG_EXAMPLE_ORACLE] is start2016-03-05 01:33:04.685 [main] INFO  com.taobao.yugong.YuGongLauncher - ## the YuGong is running now ......
2016-03-05 01:33:04.690 [main] INFO  com.taobao.yugong.YuGongLauncher - 
[YuGong Version Info]
[version ]
[hexVeision] 
[date    ]2016-03-05 01:09:25
[branch  ]master
[url     ]git@github.com:alibaba/yugong.git

2016-03-05 01:34:04.219 [pool-2-thread-2] INFO  com.taobao.yugong.common.stats.ProgressTracer - {未启动:0,全量中:0,增量中:0,已追上:1,异常数:0}
2016-03-05 01:34:04.227 [pool-2-thread-2] INFO  com.taobao.yugong.common.stats.ProgressTracer - 已完成:[ALIBABA.YUGONG_EXAMPLE_ORACLE]

查看表同步日志

2016-03-05 01:33:04.359 [main] INFO  c.t.y.e.o.OracleFullRecordExtractor$ContinueExtractor - ALIBABA.YUGONG_EXAMPLE_ORACLE start postion:0
2016-03-05 01:33:04.365 [main] INFO  com.alibaba.druid.pool.DruidDataSource - {dataSource-2} inited
2016-03-05 01:33:04.685 [main] INFO  com.taobao.yugong.controller.YuGongInstance - table[ALIBABA.YUGONG_EXAMPLE_ORACLE] start successful. extractor:com.taobao.yugong.extractor.oracle.OracleAllRecordExtractor , applier:com.taobao.yugong.applier.AllRecordApplier
2016-03-05 01:33:04.802 [YuGongInstance-ALIBABA.YUGONG_EXAMPLE_ORACLE] INFO  c.t.yugong.extractor.oracle.OracleAllRecordExtractor - table [ALIBABA.YUGONG_EXAMPLE_ORACLE] full extractor is end , next auto start inc extractor
2016-03-05 01:33:05.010 [YuGongInstance-ALIBABA.YUGONG_EXAMPLE_ORACLE] INFO  c.t.y.e.oracle.OracleMaterializedIncRecordExtractor - table[ALIBABA.YUGONG_EXAMPLE_ORACLE] now is NO_UPDATE ...

 出现了:

  1. full extractor is end , next auto start inc extractor #代表全量迁移已完成,自动进入增量模式
  2. now is NO_UPDATE #代表增量表暂时无日志

7.  oracle上执行增量变更

在源库oracle上对源表进行增量变更

insert into yugong_example_oracle values(3,'test','test',88,188, NULL , NULL ,sysdate,sysdate) 
update yugong_example_oracle set alias_name = 'superman' where id = 1

 查看表同步日志

2016-03-05 01:34:21.496 [YuGongInstance-ALIBABA.YUGONG_EXAMPLE_ORACLE] INFO  c.t.y.e.oracle.OracleMaterializedIncRecordExtractor - table[ALIBABA.YUGONG_EXAMPLE_ORACLE] now is CATCH_UP ...

 会瞬间出现now is CATCH_UP，代表刚完成处理了增量数据，并且当前没有新的增量. 

8.  查看mysq目标库数据

9. 关闭

sh bin/stop.sh

10. 整理同步逻辑

整个测试例子包含特性：

  •  schema/table名不同. oracle中为yugong_example_oracle，mysql中为test.yugong_example_mysql
  •  字段名字不同.  oracle中的name字段，映射到mysql的display_name
  •  字段逻辑处理.  mysql的display_name字段数据来源为oracle库的:name+'('alias_name+')'
  •  字段类型不同.  oracle中的amount为number类型，映射到mysql的amount为varchar文本型
  •  源库多一个字段. oracle中多了一个alias_name字段
  •  目标库多了一个字段. mysql中多了一个gmt_move字段，(简单的用迁移时的当前时间进行填充)

 具体的转换逻辑定义，可查看： 

vi conf/translator/YugongExampleOracleDataTranslator.java


 

Clone this wiki locally