使用DATAX进行数据同步
1.安装数据库相关软件
本次同步任务是mysql–> mongo;所以需要安装mysql与mongo
/1/software/安装Mysql脚本.md
/1/software/安装Mongo脚本.md
docker-compose
2.安装数据同步组件
本次同步使用的工具是datax,并且选择带界面的UI进行同步（datax-web）
2.1 配置文件编写
cd ~ && mkdir dataweb && cd $_
cat > ./bootstrap.properties << EOF
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=datax_web_db
DB_USERNAME=datax_web
DB_PASSWORD=123456
EOF
此处得多等待一些时间，等mysql启动完全
docker exec -it mysql mysql -uroot -p123456
2.2 装载datax-web数据库
CREATE DATABASE datax_web_db DEFAULT CHARACTER SET utf8mb4;
CREATE USER 'datax_web'@'%' IDENTIFIED WITH mysql_native_password BY '123456';
GRANT ALL PRIVILEGES ON datax_web_db.* TO 'datax_web'@'%';
FLUSH PRIVILEGES;
exit
2.2 datax-web的docker-compose启动脚本
services:
datax_web:
ulimits:
nofile:
soft: 1024
hard: 1024
container_name: datax-web
network_mode: host
volumes:
- ./bootstrap.properties:/home/datax/datax-web-2.1.2/modules/datax-admin/conf/bootstrap.properties
image: linshellfeng/datax_web:3.0.1
2.3 启动datax-web,接下来要提取数据库到mysql
docker-compose up -d
2.3.1 利用docker文件传递，复原datax-web数据库
#从datax-web容器中将sql文件拷贝到mysql容器
docker cp datax-web:/home/datax/datax-web-2.1.2/bin/db/datax_web.sql ~/mysql/data/
docker exec -it mysql mysql -uroot -p123456
use datax_web_db;
source /var/lib/mysql/datax_web.sql
exit
2.4 更新mongo驱动
docker exec -it datax-web bash
find / -name mongo-java-driver-\*.jar
cd /home/datax/datax/lib/
curl -O https://repo1.maven.org/maven2/org/mongodb/mongo-java-driver/3.12.14/mongo-java-driver-3.12.14.jar
mv mongo-java-driver-3.2.2.jar mongo-java-driver-3.2.2.jar.bk
cd /home/datax/datax-web-2.1.2/modules/datax-admin/lib/
curl -O https://repo1.maven.org/maven2/org/mongodb/mongo-java-driver/3.12.14/mongo-java-driver-3.12.14.jar
cd /home/datax/datax/lib/
cp /home/datax/datax-web-2.1.2/modules/datax-admin/lib/mongo-java-driver-3.12.14.jar ./
mv ./mongo-java-driver-3.4.2.jar ./mongo-java-driver-3.4.2.jar.bk
exit
2.4.1 更新完了，勿忘重启
docker restart datax-web
#进入容器查看日志
docker exec -it datax-web bash
find / -name console.out
tail -f /home/datax/datax-web-2.1.2/modules/datax-admin/bin/console.out
#待出现访问地址web-URL: http://127.0.0.1:9527/index.html 即可证明完全启动，点击下方CTRL+C，结束日志观看，继续点击退出，退出容器
2.5 准备同步数据及账号
CREATE DATABASE test;
use test;
CREATE TABLE comment (
id int NOT NULL,
name varchar(255) NOT NULL,
remarks varchar(255) DEFAULT NULL,
type varchar(255) DEFAULT NULL,
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci COMMENT='注释解释';
INSERT INTO comment (id, name, remarks, type) VALUES (1, 'DE', '字段加默认值，强制传值', 'condition');
INSERT INTO comment (id, name, remarks, type) VALUES (2, 'E', '等于', 'condition');
INSERT INTO comment (id, name, remarks, type) VALUES (3, 'NP', '不分页', 'table');
INSERT INTO comment (id, name, remarks, type) VALUES (4, 'E', '导出', 'table');
INSERT INTO comment (id, name, remarks, type) VALUES (5, 'L', '带分页', 'table');
INSERT INTO comment (id, name, remarks, type) VALUES (6, 'B', '带基础模型', 'table');
INSERT INTO comment (id, name, remarks, type) VALUES (7, 'R', '关系模型', 'field');
INSERT INTO comment (id, name, remarks, type) VALUES (8, 'T', '不存储', 'field');
INSERT INTO comment (id, name, remarks, type) VALUES (9, 'I', '不接受前端传值前端也看不到', 'field');
INSERT INTO comment (id, name, remarks, type) VALUES (10, 'BE', '范围过滤', 'condition');
INSERT INTO comment (id, name, remarks, type) VALUES (11, 'NJA', '强制不接受前端传值但回显给前端 [可新增、不修改]', 'field');
INSERT INTO comment (id, name, remarks, type) VALUES (12, 'L', '模糊匹配', 'condition');
INSERT INTO comment (id, name, remarks, type) VALUES (13, 'M', '移动', 'table');
INSERT INTO comment (id, name, remarks, type) VALUES (14, '字典', '关联字典', 'field');
INSERT INTO comment (id, name, remarks, type) VALUES (15, '关联', '关联表', 'field');
INSERT INTO comment (id, name, remarks, type) VALUES (16, 'IN', 'IN查询', 'condition');
INSERT INTO comment (id, name, remarks, type) VALUES (1013, 'S', '只展示', 'field');
INSERT INTO comment (id, name, remarks, type) VALUES (2013, 'S', '有父表', 'table');
INSERT INTO comment (id, name, remarks, type) VALUES (2014, 'H', '抑制输出', 'field');
INSERT INTO comment (id, name, remarks, type) VALUES (2015, 'EI', '忽略导出字段', 'field');
INSERT INTO comment (id, name, remarks, type) VALUES (2016, 'IR', '忽略关系输出', 'field');
INSERT INTO comment (id, name, remarks, type) VALUES (2017, 'EO', '导出字段', 'field');
INSERT INTO comment (id, name, remarks, type) VALUES (2018, 'P', '只分页', 'table');
2.5.1 装载测试数据
docker exec -it mysql mysql -uroot -p123456
source /var/lib/mysql/data.sql
exit
2.5.2 准备Mongo账号
** 直接用root会报如下错误：**
{ "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18, "codeName" : "AuthenticationFailed" }}}]
2.5.2.1 给mongo新建数据库及用户
docker exec -it mongo bash
mongo
use admin
db.auth("root","123456")
db.getUsers()
2.5.2.2 执行测试库添加用户
use test
db.createUser({
user: "syncdata",
pwd: "123456",
roles: [{
role: "dbOwner",
db: "test"
}]
})
db.getUsers()
2.5.3 datax端口开放在9527,访问datax-web
9527/index.html
2.5.4 打开之后填写账号密码: admin/123456
2.5.4.1 新建项目
2.5.4.2 新建任务
2.5.4.3 底部写任务脚本文件
{
"job": {
"content": [
{
"reader": {
"name": "mysqlreader",
"parameter": {
"username": "root",
"password": "123456",
"column": ["id","name","remarks","type"],
"connection": [
{
"table": [
"comment"
],
"jdbcUrl": [
"jdbc:mysql://127.0.0.1:3306/test?useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=UTC"
]
}
],
"splitPk": "id"
}
},
"writer": {
"name": "mongodbwriter",
"parameter": {
"address": [
"127.0.0.1:27017"
],
"userName": "syncdata",
"userPassword": "123456",
"dbName": "test",
"collectionName": "comment",
"column": [
{
"name": "_id",
"type": "int"
},
{
"name": "name",
"type": "string"
},
{
"name": "remarks",
"type": "string"
},
{
"name": "type",
"type": "string"
}
],
"writeMode": {
"isReplace": "true",
"replaceKey": "_id"
}
}
}
}
],
"setting": {
"speed": {
"channel": 1
}
}
}
}
2.5.5 配置完成点击“执行一次”
2.5.6 查看同步日志
2.5.7 任务成功执行完后查询，查看Mongo里是否有数
docker exec -it mongo bash
mongo
use test
db.auth("syncdata","123456")
show dbs
show tables
db.comment.find()
exit
2.6 问题排查脚本
docker exec -it datax-web bash
ps axu
find / -name *entrypoint.sh
find / -name console.out
find / -name *.sql
文章版权声明：除非注明，否则均为边学边练网络文章，版权归原作者所有
