仅在需要/特定时间间隔时将收集的RAW数据插入Influxdb

时间:2019-02-13 04:47:50

标签: grafana influxdb collectd

在尝试向influxdb中插入数据以从grafana提取数据时,我们面临着问题。截至目前,正在进行一项测试,我们将从主机收集的数据直接插入到influxdb db中,事实是,这变得非常巨大,并且这仅是来自单个主机的信息,我们的PROD中有数百个主机环境是我们的主要目标。

我们试图做的是仅在需要调查中断时才插入数据,为此,我们计划从当前正在创建的RAW文件中进行插入。我们遇到了一个解决方案,但NMON可以满足我们的需要,但NMON文件却可以满足我们的需要,但老实说,我对Golang并不了解,这是他们用于该语言的语言,或者至少我是这么认为的。

我在这个社区中发现有一个PARSER插件是telegraf的一部分,但我找不到如何使用它来从collects提供的RAW文件中获取信息并将其插入到influxdb中

如果有人能就此事给我提建议,我将不胜感激。

谢谢。

输出示例:

    ################################################################################
    # Collectl:   V3.6.5-2  HiRes: 1  Options: -D
    # Host:       localhost  DaemonOpts: -f /var/log/collectl -r00:00,7,60 -m -F60 -s+YZ -i60
    # Booted:     1548722801.34 [20190128-18:46:41]
    # Distro:     Red Hat Enterprise Linux Server release 6.9 (Santiago)  Platform: VMware Virtual Platform
    # Date:       20190130-230000  Secs: 1548910800 TZ: -0600
    # SubSys:     bcdfijmnstYZ Options:  Interval: 60:60 NumCPUs: 10  NumBud: 3 Flags: ix
    # Filters:    NfsFilt:  EnvFilt:  TcpFilt: ituc
    # HZ:         100  Arch: x86_64-linux-thread-multi PageSize: 4096
    # Cpu:        GenuineIntel Speed(MHz): 2992.968 Cores: 1  Siblings: 1 Nodes: 1
    # Kernel:     2.6.32-696.20.1.el6.x86_64  Memory: 198339428 kB  Swap: 17825788 kB
    # NumDisks:   22 DiskNames: sda sdb sdc sde sdd dm-0 dm-1 dm-2 dm-3 dm-4 dm-5 dm-6 dm-7 dm-8 dm-9 dm-10 dm-11 dm-12 dm-13 dm-14 dm-15 dm-16
    # NumNets:    3 NetNames: lo:?? eth0:10000 eth1:10000
    # NumSlabs:   201 Version: 2.1
    # SCSI:       CD:1:00:00:00 DA:2:00:00:00 DA:2:00:01:00 DA:2:00:02:00 DA:2:00:03:00 DA:2:00:04:00
    ################################################################################
    >>> 1548910800.001 <<<
    buddy Node 0, zone      DMA      1      1      1      3      3      2      1      1      0      0      2
    buddy Node 0, zone    DMA32     14     18     15     17     14      7      7     11      8      4    106
    buddy Node 0, zone   Normal    403    386   5583   3572  13604  10598   7080   3627   2055      2     27
    cpu  22087348 256 7084173 153333789 15626 4857 2989099 0 0
    cpu0 2343703 23 756440 15044112 1503 394 389489 0 0
    cpu1 1424427 40 403584 16804532 1978 2 13682 0 0
    cpu2 1400317 30 399191 16835804 1824 1 12608 0 0
    cpu3 2373492 20 777245 15024216 1405 441 357515 0 0
    cpu4 2305879 21 754504 15129573 1544 395 334811 0 0
    cpu5 2406073 20 826459 14953667 1377 1854 337925 0 0
    cpu6 2524873 19 813584 14794741 1330 456 388679 0 0
    cpu7 2377199 26 774277 14997185 1586 440 369269 0 0
    cpu8 2431979 19 818481 14856251 1454 501 415840 0 0
    cpu9 2499402 33 760403 14893703 1620 370 369277 0 0
    intr 5904658682
    ctxt 4292174746
    processes 1824586
    procs_running 3
    procs_blocked 0
    int    0:        844          0          0          0          0          0          0          0          0          0   IO-APIC-edge      timer
    int    1:         10       1146          0          0          0          0          0          0          0          0   IO-APIC-edge      i8042
    int    8:          1          0          0          0          0          0          0          0          0          0   IO-APIC-edge      rtc0
    int    9:          0          0          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   acpi
    int   12:        110          0          0       1080          0          0          0          0          0          0   IO-APIC-edge      i8042
    int   14:          0          0          0          0          0          0          0          0          0          0   IO-APIC-edge      ata_piix
    int   15:         95          0          0          0          0     639862          0          0          0          0   IO-APIC-edge      ata_piix
    int   16:          4          0          0          0          0          0          0          0          0          2   IO-APIC-fasteoi   vmwgfx
    int   24:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
    int   25:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
    int   26:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
    int   27:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
    int   28:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
    int   29:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
    int   30:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
    int   31:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
    int   32:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
    int   33:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
    int   34:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
    int   35:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
    int   36:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
    int   37:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
    int   38:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
    int   39:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
    int   40:          0          0          0          0          0          0          0          0          0          0   PCI-MSI-edge      pciehp
disk  253       3 dm-3 817 0 6530 1019 1 0 8 6 0 187 1025
disk  253       4 dm-4 261 0 3298 154 78648 0 629184 94613 0 12780 94788
disk  253       5 dm-5 122 0 970 30 1 0 8 0 0 30 30
disk  253       6 dm-6 128 0 1018 49 1 0 8 6 0 55 55
disk  253       7 dm-7 122 0 970 43 1 0 8 0 0 43 43
disk  253       8 dm-8 14705 0 173314 18512 30827989 0 246623912 309099632 0 580567 309838167
disk  253       9 dm-9 257 0 2050 167 1 0 8 0 0 80 167
disk  253      10 dm-10 142 0 1130 58 112 0 896 81 0 107 139
disk  253      11 dm-11 26758 0 1183658 26638 9156 0 73248 27014 0 15012 53652
disk  253      12 dm-12 133 0 1058 59 6 0 48 2 0 49 61
disk  253      13 dm-13 133 0 1058 56 6 0 48 2 0 46 58
disk  253      14 dm-14 5088 0 615178 3932 107319 0 858552 1285013 0 3111 1288946
disk  253      15 dm-15 15240 0 410298 10749 1538387 0 12307096 7551698 0 121712 7567479
disk  253      16 dm-16 544 0 20778 753 989864 0 7918912 2153755 0 479168 2155269
load 0.21 0.16 0.10 1/2757 24089
tcp-Ip: Forwarding DefaultTTL InReceives InHdrErrors InAddrErrors ForwDatagrams InUnknownProtos InDiscards InDelivers OutRequests OutDiscards OutNoRoutes ReasmTimeout ReasmReqds ReasmOKs ReasmFails FragOKs FragFails FragCreates
tcp-Ip: 2 64 2345808100 0 65987 0 0 0 2345023989 2542097859 0 88 0 1374787 657103 0 658849 0 1378045
tcp-Icmp: InMsgs InErrors InDestUnreachs InTimeExcds InParmProbs InSrcQuenchs InRedirects InEchos InEchoReps InTimestamps InTimestampReps InAddrMasks InAddrMaskReps OutMsgs OutErrors OutDestUnreachs OutTimeExcds OutParmProbs OutSrcQuenchs OutRedirects OutEchos OutEchoReps OutTimestamps OutTimestampReps OutAddrMasks OutAddrMaskReps
tcp-Icmp: 9644 24 1453 0 0 0 0 4223 3966 2 0 0 0 10125 0 1934 0 0 0 0 3966 4223 0 2 0 0
tcp-Tcp: RtoAlgorithm RtoMin RtoMax MaxConn ActiveOpens PassiveOpens AttemptFails EstabResets CurrEstab InSegs OutSegs RetransSegs InErrs OutRsts
tcp-Tcp: 1 200 120000 -1 189518 203747 116 1705 1538 2337716464 2532808789 6750 1 1108
tcp-Udp: InDatagrams NoPorts InErrors OutDatagrams RcvbufErrors SndbufErrors
tcp-Udp: 7155937 2287 12361 9272237 1 0
fs-ds 279615    266755  45      0       0       0
fs-is 121177    118
fs-fnr 9728     0       6815744
nfsc-net 0 0 0 0
nfsc-rpc 499842 1085 499886
nfsc-proc2 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
nfsc-proc3 22 0 105381 0 2333 383140 6 1704 0 0 0 0 0 0 0 0 0 0 44 7201 20 10 0

1 个答案:

答案 0 :(得分:0)

Telegraf's docs 显示如何读取收集的文件的示例, 您使用文件输入,并使用data_format = "collectd"来使用收集的解析器。

旁注,直接InfluxDB can accept CollectD。 (这可能需要此file,但似乎未包含在我的安装中)

要进行速率限制,请查看收集的interval,或者如果您更喜欢使用telegraf,它也有such an option
我不确定influxdb的选项,但是您可能会对它们的downsampling and retention功能感兴趣。