改进Python + Influxdb导入性能

时间:2017-04-02 14:38:59

标签: python performance weblogic jython influxdb

以下脚本是

的摘录

https://github.com/RittmanMead/obi-metrics-agent/blob/master/obi-metrics-agent.py

 List <WebElement> allElement = driver.findElements(By.xpath("//a/span[contains(text(),',')]/parent::*")); // will get all anchor tag which having , and store in a list

 System.out.println(allElement.size());
 for(WebElement element:allElement)
 {
    System.out.println(element.getAttribute("href"));
    driver = new ChromeDriver();
    driver.get(element.getAttribute("href"));
 }

Influxdb目前正以大约5点/秒(总共1:45分钟)加载数据。 有什么方法可以提高性能? 我还附上了数据的样本摘录

Data Set

编辑1:

我被建议使用“请求”模块来保持连接活动 在我创建的帖子上

https://groups.google.com/forum/#!topic/comp.lang.python/D0XJPLOLXgo

但是,weblogic使用Jython 2.2.1作为标准(即使在最新版本中)。我读了Maarten Smeets的一篇文章

https://technology.amis.nl/2015/10/04/how-to-use-wlst-as-a-jython-2-7-module/

允许将weblogic部署为Jython 2.7模块。 所以我下载了&amp;安装了Jython 2.7并将weblogic导入为模块。

我能够下载,安装和导入请求作为模块,但是,我遇到了标准weblogic功能的问题

import calendar, time
import sys
import getopt
import httplib

print '---------------------------------------'

try:
    wls_user = "user"
    wls_pw = "pw"
    url  = "url"
    outputFormat="Influxdb"
    targetHost="hostname"
    targetPort="port"
    targetDB="dbname"
except:
    print ''

print wls_user, wls_pw,url, outputFormat,targetHost,targetPort,targetDB

now_epoch = calendar.timegm(time.gmtime())*1000

influx_msgs=''
connect(wls_user,wls_pw,url)
results = displayMetricTables('Oracle_BI*','dms_cProcessInfo')
for table in results:
    tableName = table.get('Table')
    rows = table.get('Rows')
    rowCollection = rows.values()
    iter = rowCollection.iterator()
    while iter.hasNext():
        row = iter.next()
        rowType = row.getCompositeType()
        keys = rowType.keySet()
        keyIter = keys.iterator()
        inst_name= row.get('Name').replace(' ','-')
        try:
            server= row.get('Servername').replace(' ','-').replace('/','_')
        except:
            try:
                server= row.get('ServerName').replace(' ','-').replace('/','_')
            except:
                server='unknown'
        try:
            host= row.get('Host').replace(' ','-')
        except:
            host=''
        while keyIter.hasNext():
            columnName = keyIter.next()
            value = row.get(columnName )
            if columnName.find('.value')>0:
                metric_name=columnName.replace('.value','')
                if value is not None:
                    if value != 0:
                        if outputFormat=='InfluxDB':
                            influx_msg= ('%s,server=%s,host=%s,metric_group=%s,metric_instance=%s value=%s %s') % (metric_name,server,host,tableName,inst_name,  value,now_epoch*1000000)
                            influx_msgs+='\n%s' % influx_msg
                            conn = httplib.HTTPConnection('%s:%s' % (targetHost,targetPort))
                            ## TODO pretty sure should be urlencoding this ...
                            a=conn.request("POST", ("/write?db=%s" % targetDB), influx_msg)
                            r=conn.getresponse()

之前没有任何问题的工作

根据Maarten的帖子,我确实尝试使用前缀** wl。** displayMetricTables, 但这不起作用

任何和所有帮助将不胜感激

由于 P

0 个答案:

没有答案