通过Python子进程的mysqldump失败,并出现long where子句

时间:2018-11-26 21:25:02

标签: python mysql subprocess

我对Python还是有些陌生,并且正在编写Flask API与MySQL进行通信并执行每天通过票务系统获取的常规命令。前提是:

  • 请求连同查询一起发送到API
  • 代码正则表达式是WHERE子句
  • 通过子进程启动mysqldump以备份受影响的表的那些行
  • 然后执行查询本身

我遇到的问题是当提供了long where子句时(大约2,850个字符 2,865个字符或更多,所以不是很长),mysqldump本身会执行不行。查询本身仍然执行并返回结果,尽管转储未完成。

我已经尝试在subprocess库中使用callPopencommunicatewait的几乎所有组合,但似乎没有处理这些较大的查询。

在测试时,我正在执行的查询非常快(SELECTmysql客户端中的执行时间为0.00秒),因此它们甚至都不是长时间运行/超时的。现在,我正在用一个简单的SELECT语句进行测试,该语句在WHERE子句中传递了一系列ID值。如果要添加单个ID,转储将停止工作,但API仍会返回查询结果。我还可以将mysqldump命令传递给子进程,并在目标服务器上的Shell中执行,并在.015s中完成。因此,似乎代码编写正确,可以按需来回传递,但是当WHERE子句开始变长时,会发生某些事情。

(在较小的查询中)当前有效的代码如下:

from flask import request, jsonify, abort
from sshtunnel import SSHTunnelForwarder
import configparser, re, subprocess, paramiko

...

server = SSHTunnelForwarder(
    (proxy, 22),
    ssh_username=user,
    ssh_pkey=key,
    remote_bind_address=(db_host, 3306)
)

server.start()

conn = db.MySQL(
    '127.0.0.1',
    db_user,
    db_pass,
    database,
    server.local_bind_port
).client

cursor = conn.cursor()

regexp = re.compile("where", re.IGNORECASE)
query_split = regexp.split(query)
where_clause = query_split[-1].strip()

bak_file = open('/mnt/backup/' + ticket_no + '.bak.sql', 'w+')

mysqldump = '/usr/bin/mysqldump %s %s -u%s -p%s -h%s -P%d \
    --single-transaction --master-data=2 --replace -t -w"' % (
    database, table, db_user, db_pass, '127.0.0.1',
    server.local_bind_port) + where_clause + '"'

proc = subprocess.Popen(mysqldump, stdout=bak_file, stderr=subprocess.STDOUT, shell=True)
proc.wait()

bak_file.close()

try:
    cursor.execute(query)
    ret = cursor.fetchone()[0]

#                conn.commit()
    conn.rollback()
except (db.MySQLdb.Error, db.MySQLdb.Warning) as e:
    conn.rollback()
    conn.close()
    ret = e[1]

server.stop()

return jsonify({'result': ret})

数据以JSON形式发送。如前所述,将单个ID添加到以下列表会导致其失败。即使将2个字符添加到ID(2900-> 290011)也会失败(添加1个字符仍然可以使用)。

{
"ticket": "0000000000",
"host": "servername",
"db": "dbname",
"query": "select count(*) from table WHERE Id in ('2025','2026','2027','2028','2029','2032','2033','2034','2036','2037','2038','2041','2042','2043','2044','2045','2046','2047','2048','2049','2050','2051','2057','2059','2060','2061','2062','2063','2064','2065','2067','2069','2084','2086','2087','2089','2091','2092','2093','2095','2096','2097','2098','2100','2101','2103','2104','2110','2111','2112','2113','2114','2115','2116','2117','2118','2120','2121','2122','2123','2124','2125','2126','2127','2128','2129','2130','2131','2144','2146','2160','2161','2163','2164','2165','2166','2167','2168','2169','2170','2172','2173','2174','2175','2176','2177','2178','2179','2180','2181','2182','2183','2184','2191','2193','2200','2201','2202','2204','2206','2208','2211','2212','2214','2215','2217','2226','2227','2228','2229','2230','2233','2234','2235','2236','2237','2241','2242','2244','2245','2246','2248','2250','2251','2252','2256','2257','2258','2260','2262','2264','2266','2267','2268','2269','2271','2272','2273','2274','2275','2276','2278','2280','2281','2282','2283','2284','2286','2287','2288','2289','2291','2292','2293','2294','2295','2296','2297','2298','2299','2300','2301','2302','2303','2304','2308','2309','2310','2312','2314','2322','2323','2326','2327','2328','2329','2330','2331','2332','2333','2334','2335','2336','2337','2338','2339','2341','2342','2343','2344','2345','2346','2347','2348','2349','2350','2351','2352','2353','2354','2355','2356','2358','2360','2363','2364','2365','2367','2368','2369','2372','2373','2375','2376','2378','2380','2381','2383','2384','2385','2386','2387','2388','2390','2391','2392','2393','2398','2399','2400','2401','2402','2405','2406','2407','2408','2409','2410','2411','2413','2414','2416','2417','2418','2421','2424','2427','2428','2429','2430','2432','2433','2434','2435','2436','2437','2438','2439','2440','2441','2447','2448','2450','2456','2457','2458','2459','2460','2461','2469','2470','2472','2473','2475','2477','2478','2479','2480','2481','2482','2487','2497','2498','2499','2500','2502','2505','2506','2507','2508','2509','2511','2512','2515','2518','2519','2520','2521','2523','2526','2527','2528','2529','2531','2532','2533','2534','2536','2539','2541','2542','2544','2549','2550','2551','2554','2555','2556','2559','2566','2594','2596','2598','2599','2600','2602','2603','2605','2607','2608','2609','2610','2611','2619','2620','2622','2623','2624','2631','2632','2633','2634','2635','2636','2638','2668','2670','2671','2672','2673','2675','2676','2677','2680','2685','2686','2688','2690','2709','2711','2737','2740','2744','2748','2764','2765','2768','2769','2770','2772','2773','2774','2775','2777','2778','2779','2787','2794','2795','2796','2807','2815','2839','2840','2841','2853','2854','2858','2859','2860','2861','2862','2865','2867','2871','2872','2874','2875','2880','2882','2885','2890','2891','2892','2895','2905','2906','2907')"
}

注意::由于防火墙的限制,查询代码通过SSHTunnel连接,我认为这可能是问题的一部分,也可能是问题的一部分。但是,正如我提到的,使用较小的查询,一切运行正常。

更新1:我做了进一步的测试,发现我可以通过的最大长度WHERE子句是 2,864个字符/字节。当我达到2865时,它就会中断。但是,当我在命令行上运行WHERE时,较长的mysqldump子句可以很好地运行。

更新2:我在-vvv命令中添加了mysqldump标志。在备份失败的情况下,我的转储文件中包含以下内容:

-- Connecting to 127.0.0.1...
-- Starting transaction...
-- Setting savepoint...
-- Retrieving table structure for table tablename...
-- SELECT query...

成功转储之后的下一行将显示为:

-- Retrieving rows...

0 个答案:

没有答案