Question

问题在于：

从多个表AA，BB，CC，DD中选择几个结果集... MM在一系列相同的mysql数据库db1
从数据字典中为这些表选择一些派生值例如。大小，用户更新权限等
请注意用于每个连接的变量，例如db版本，套接字，端口号，表，架构等
注意一些系统env变量。例如df - 数据挂载点
请注意用户执行的时间等
将每个数据库的每个表的组合结果集插入到一行中另一个db。

许多示例都是通过选择和插入数据库而不是组合结果（在Python中）。

部分工作的示例代码，可以轻松选择并将单独的数据集插入到数据库中。

使用PyMySQL库和Python 3.4

#!/usr/bin/python3

import glob
import pymysql.cursors

#add functions here
#should really add one per sql statement but compressed some examples into     this function

def getdata (connection):
     #comment
     #sql1 = "select count(a) as \"rowcount\", a, b, min(date_entered) as \"earliest\", max(date_entered) as \"latest\" from db1.AA group by a,b"
     sql1 = "select count(a) as \"rowcount\", a, min(date_entered) as \"earliest\", max(date_entered) as \"latest\" from db1.BB group by a"
     #sql1 = "select count(a) as \"rowcount\", a, min(date_entered) as \"earliest\", max(date_entered)  as \"latest\" from db1.CC group by a"
     #sql1 = "SELECT table_name  as \"table\",  round(((data_length+index_length)/1024/1024),2) 'size' FROM information_schema.TABLES WHERE table_name = 'AA'"
    #sql1 = "SELECT table_name  as \"table\",  round(((data_length+index_length)/1024/1024),2) 'size' FROM information_schema.TABLES WHERE table_name = 'CC'"
    #sql1 = "SELECT table_name  as \"table\",  round(((data_length+index_length)/1024/1024),2) 'size' FROM information_schema.TABLES WHERE table_name = 'BB'"    
    cursor1=connection.cursor()
    cursor1.execute(sql1)
    result = cursor1.fetchall()
    return result

def getdata2 (connection):
    #will be the gathering of variables and environment
    sql2 = "show variables like 'socket'"
    cursor1=connection.cursor()
    cursor1.execute(sql2)
    result = cursor1.fetchall()
    return result

#keep the writes to the end
def putdata (connection,row):
    #the insert statements of the separate selects are
#    sql="INSERT INTO ZZ (rowcount,a,b,earliest,latest) VALUES (%(rowcount)s,%(a)s,%(b)s,%(earliest)s,%(latest)s)"
    sql="INSERT INTO ZZ (rowcount,a,earliest,latest) VALUES (%(rowcount)s,%(a)s,%(earliest)s,%(latest)s)"
#    sql="INSERT INTO ZZ (table,size) VALUES (%(table)s,%(size)s)"    
#    sql="INSERT INTO ZZ (socket) VALUES (%(filename)s)"
    #ideally they should include all the parameters
    cursor2=connection.cursor()
    cursor2.execute(sql,row)
    connection.commit()
#should really add an exception here for the write db connection not being there
connection2 = pymysql.connect(unix_socket = '/var/run/mysqld/mysqld.sock',
                          host='localhost',
                          db='db2',
                          charset='utf8mb4',
                          use_unicode= 'true',
                          user='user',
                          password='password',
                          cursorclass=pymysql.cursors.DictCursor)

#There are about 50 instances
pattern = '/var/run/mysqld/*.sock'

x = 0 # just for debug
for filename in glob.glob(pattern):
    x = x + 1 # just for debug
try:
    connection1 = pymysql.connect(
                             unix_socket = filename,
                             host='localhost',
                             db='db1',
                             charset='utf8mb4',
                             use_unicode= 'true',
                             user='user',
                             password='password',
                             cursorclass=pymysql.cursors.DictCursor)
    print(filename)  # just for debug so can see where get to
    print ('\t', x) # just for debug so can see where get to
    data=getdata(connection1)
    for row in data:
        putdata(connection2,row)
        print(row)


except BaseException as e:
    print("Error connecting to database",filename)
    pass

print("success")

结果示例如下：

/var/run/mysqld/mysql_socketinfo.sock
 58
{'a': '123abc', 'earliest': datetime.datetime(2017, 10, 3, 8, 45, 5), 'latest': datetime.datetime(2017, 11, 27, 12, 55, 31), 'rowcount': 13}
{'a': '456def', 'earliest': datetime.datetime(2016, 10, 20, 14, 21, 1), 'latest': datetime.datetime(2016, 11, 23, 14, 49, 21), 'rowcount': 468}
{'a': '789ghi', 'earliest': datetime.datetime(2016, 12, 5, 21, 56, 20), 'latest': datetime.datetime(2017, 6, 5, 16, 42, 58), 'rowcount': 30}

正在插入的表格：

CREATE TABLE `ZZ` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `a` varchar(60) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
  `b` varchar(60) CHARACTER SET utf8 COLLATE utf8_unicode_ci NOT NULL DEFAULT '',
  `schema` varchar(45) DEFAULT NULL,
  `table` varchar(45) DEFAULT NULL,
  `rowcount` bigint(255) DEFAULT NULL,
  `size` bigint(255) DEFAULT NULL,
  `earliest` datetime DEFAULT '1970-01-01 00:00:00',
  `latest` datetime DEFAULT '1970-01-01 00:00:00',
  `hostname` varchar(45) NOT NULL,
  `socket` varchar(45) NOT NULL,
  `c` varchar(45) DEFAULT NULL,
  `when` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1

使用python将几个mysql结果集与一些变量连接并插入另一个db

0 个答案: