datax安装及批量生成json任务文件，以sqlservrreader和mysqlwriter为例_datax的json文件自动生成

; section名称
[sqlserver]
url = jdbc:sqlserver://hadoop103:1433;DatabaseName=test
username = sa
password = yH5nm57BSkFjmL8rBpRH
table = t_info
column = id,order_date,purchaser,quantity,product_id
where =

;key 只能是小写
[mysql]
url = jdbc:mysql://hadoop102:3306/test?useUnicode=true&characterEncoding=utf-8
username = root
password = 123321
table = t_info
column = id,order_date,purchaser,quantity,product_id
;preSql = delete from t_info WHERE order_date = CURDATE();
preSql =
writeMode = insert

2. 使用python脚本，读取配置文件自动生成datax的json文件config.json

这个json文件即datax的job任务，可以直接运行同步数据

vim generate_json.py

import json
datax_json = {
“job”: {
“content”: [
{
“reader”: {
“name”: “sqlserverreader”,
“parameter”: {
“connection”: [
{
“jdbcUrl”: [],
“table”: []
}
],
“password”: “”,
“username”: “”,
“column”: []
}
},
“writer”: {
“name”: “mysqlwriter”,
“parameter”: {
“column”: [],
“connection”: [
{
“jdbcUrl”: “”,
“table”: []
}
],
“password”: “”,
“preSql”: [],
“session”: [],
“username”: “”,
“writeMode”: “”
}
}
}
],
“setting”: {
“speed”: {
“channel”: “5”
}
}
}
}

def get_config():
import configparser
file = ‘prod\datax.ini’
con = configparser.ConfigParser()
con.read(file, encoding=‘utf-8-sig’)
sections = con.sections()

获取特定section

items_sqlserver = con.items(‘sqlserver’) # 返回结果为元组
config_sqlserver = dict(items_sqlserver)

items_mysql = con.items(‘mysql’) # 返回结果为元组
config_mysql = dict(items_mysql)

key 只能是小写

return {‘config_sqlserver’:config_sqlserver,‘config_mysql’:config_mysql}

if name == ‘main’:
config = get_config()

sqlserver reader

datax_json[‘job’][‘content’][0][‘reader’][‘parameter’][‘connection’][0][‘jdbcUrl’].append(config[‘config_sqlserver’][‘url’])
datax_json[‘job’][‘content’][0][‘reader’][‘parameter’][‘connection’][0][‘table’].append(config[‘config_sqlserver’][‘table’])
datax_json[‘job’][‘content’][0][‘reader’][‘parameter’][‘password’] = config[‘config_sqlserver’][‘password’]
datax_json[‘job’][‘content’][0][‘reader’][‘parameter’][‘username’] = config[‘config_sqlserver’][‘username’]
datax_json[‘job’][‘content’][0][‘reader’][‘parameter’][‘where’] = config[‘config_sqlserver’][‘where’]
datax_json[‘job’][‘content’][0][‘reader’][‘parameter’][‘column’] = list(config[‘config_sqlserver’][‘column’].split(‘,’))

mysql writer

datax_json[‘job’][‘content’][0][‘writer’][‘parameter’][‘connection’][0][‘jdbcUrl’] = config[‘config_mysql’][‘url’]
datax_json[‘job’][‘content’][0][‘writer’][‘parameter’][‘connection’][0][‘table’].append(config[‘config_mysql’][‘table’])
datax_json[‘job’][‘content’][0][‘writer’][‘parameter’][‘password’] = config[‘config_mysql’][‘password’]
datax_json[‘job’][‘content’][0][‘writer’][‘parameter’][‘username’] = config[‘config_mysql’][‘username’]
datax_json[‘job’][‘content’][0][‘writer’][‘parameter’][‘preSql’].append(config[‘config_mysql’][‘presql’]) # 配置里面key只能识别为小写
datax_json[‘job’][‘content’][0][‘writer’][‘parameter’][‘writeMode’] = config[‘config_mysql’][‘writemode’]# 配置里面key只能识别为小写
datax_json[‘job’][‘content’][0][‘writer’][‘parameter’][‘column’] = list(config[‘config_mysql’][‘column’].split(‘,’))

with open (‘prod\config.json’, ‘w’) as f:
f.write(json.dumps(datax_json))

print(datax_json)

3. 按照步骤2生成的json文件如下所示：

{
   “job”: {
       “content”: [{
           “reader”: {
               “name”: “sqlserverreader”,
               “parameter”: {
                   “connection”: [{
                       “jdbcUrl”: [“jdbc:sqlserver://hadoop103:1433;DatabaseName=test”],
                       “table”: [“t_info”]
                   }],
                   “password”: “yH5nm57BSkFjmL8rBpRH”,
                   “username”: “sa”,
                   “column”: [“id”, “order_date”, “purchaser”, “quantity”, “product_id”],
                   “where”: “”
               }
           },
           “writer”: {
               “name”: “mysqlwriter”,
               “parameter”: {
                   “column”: [“id”, “order_date”, “purchaser”, “quantity”, “product_id”],
                   “connection”: [{
                       “jdbcUrl”: “jdbc:mysql://hadoop102:3306/test?useUnicode=true&characterEncoding=utf-8”,
                       “table”: [“t_info”]
                   }],
                   “password”: “123321”,
                   “preSql”: [“”],
                   “session”: [],
                   “username”: “root”,
                   “writeMode”: “insert”
               }
           }
       }],
       “setting”: {
           “speed”: {
               “channel”: “5”
           }
       }
   }
}