Kubernetes,Fluentd:匹配`stdout`日志

时间:2020-10-19 19:31:33

标签: kubernetes microservices fluentd

我在Kubernetes集群中有一个以DaemonSet运行的Fluentd。多个应用程序以微服务的形式部署在群集上,该群集跨名称空间,名称与微服务名称相同。

微服务(名称为payments)将日志输出到stdout。我想解析该微服务日志并将其放置在Fluentd中的某个目的地。问题是我只需要特定的微服务payments日志。如果我使用:

<source>
  @type tail
  @id in_tail_container_logs
  path /var/log/containers/*.log
  pos_file /var/log/fluentd-containers.log.pos
  tag kubernetes.*
  read_from_head true
  ....
</source>

它将扫描将日志输出到stdout我想过滤的特定微服务日志的所有Pod。我们有办法在Fluentd中做到这一点吗?谢谢

2 个答案:

答案 0 :(得分:1)

每个Pod在Option Explicit 'This code was originally written by Terry Kreft. 'It is not to be altered or distributed, 'except as part of an application. 'You are free to use it in any application, 'provided the copyright notice is left unchanged. ' 'Code Courtesy of 'Terry Kreft Private Const STARTF_USESHOWWINDOW& = &H1 Private Const NORMAL_PRIORITY_CLASS = &H20& Private Const INFINITE = -1& Private Type STARTUPINFO cb As Long lpReserved As String lpDesktop As String lpTitle As String dwX As Long dwY As Long dwXSize As Long dwYSize As Long dwXCountChars As Long dwYCountChars As Long dwFillAttribute As Long dwFlags As Long wShowWindow As Integer cbReserved2 As Integer lpReserved2 As Long hStdInput As LongPtr hStdOutput As LongPtr hStdError As LongPtr End Type Private Type PROCESS_INFORMATION hProcess As LongPtr hThread As LongPtr dwProcessID As Long dwThreadID As Long End Type 'Added Private Type SECURITY_ATTRIBUTES nLength As Long lpSecurityDescriptor As LongPtr bInheritHandle As Long End Type Private Declare PtrSafe Function WaitForSingleObject Lib "kernel32" (ByVal _ hHandle As LongPtr, ByVal dwMilliseconds As Long) As Long 'Type not defined Declare PtrSafe Function CreateProcessA Lib "kernel32" _ (ByVal lpApplicationName As String, ByVal lpCommandLine As String, _ lpProcessAttributes As SECURITY_ATTRIBUTES, lpThreadAttributes As SECURITY_ATTRIBUTES, _ ByVal bInheritHandles As Long, ByVal dwCreationFlags As Long, lpEnvironment As Any, _ ByVal lpCurrentDriectory As String, lpStartupInfo As STARTUPINFO, _ lpProcessInformation As PROCESS_INFORMATION) As LongPtr ' Original 'Private Declare Function CreateProcessA Lib "kernel32" (ByVal _ lpApplicationName As Long, ByVal lpCommandLine As String, ByVal _ lpProcessAttributes As Long, ByVal lpThreadAttributes As Long, _ ByVal bInheritHandles As Long, ByVal dwCreationFlags As Long, _ ByVal lpEnvironment As Long, ByVal lpCurrentDirectory As Long, _ lpStartupInfo As STARTUPINFO, lpProcessInformation As _ PROCESS_INFORMATION) As Long Private Declare PtrSafe Function CloseHandle Lib "kernel32" (ByVal _ hObject As LongPtr) As Long Public Sub ShellWait(Pathname As String, Optional WindowStyle As Long) Dim proc As PROCESS_INFORMATION Dim start As STARTUPINFO Dim ret As LongPtr 'Not used, but needed Dim si1 As SECURITY_ATTRIBUTES Dim si2 As SECURITY_ATTRIBUTES ' Initialize the STARTUPINFO structure: With start .cb = Len(start) If Not IsMissing(WindowStyle) Then .dwFlags = STARTF_USESHOWWINDOW .wShowWindow = WindowStyle End If End With 'Set the structure size si1.nLength = Len(si1) si2.nLength = Len(si2) ' Start the shelled application: ret = CreateProcessA(vbNullString, Pathname, si1, si2, False, _ NORMAL_PRIORITY_CLASS, 0&, vbNullString, start, proc) 'TEST SECURITY_ATTRIBUTES Data Types ' Wait for the shelled application to finish: ret = WaitForSingleObject(proc.hProcess, INFINITE) ' TEST proc.hProcess is LongPtr ret = CloseHandle(proc.hProcess) ' TEST proc.hProcess is LongPtr End Sub 的容器目录中创建一个日志文件

它应该在container文件夹中具有匹配的日志文件,例如,如果您的部署以名称/var/lib/docker/containers开头,我们可以通过提供该服务捕获该特定服务的日志

payments

path /var/log/containers/payments*.log

答案 1 :(得分:0)

不要在文件级别实现分隔。相反,请使用kubernetes_metadata流利过滤器,该过滤器会使用与Pod相关的元数据来丰富每个日志记录。

示例配置:https://github.com/fluent/fluentd-kubernetes-daemonset/blob/e8cb994ae938be784a55dfada297a7f4192a411c/docker-image/v1.10/debian-papertrail/conf/kubernetes.conf#L191-L201

示例输入/输出:https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter/tree/84f66a8f9e06ab5b5211053fcce4cd8ab4bd74ba#example-inputoutput

有了这个,就有几种方法可以过滤掉特定的服务:

  • 将所有日志发送到日志记录后端,例如elasticsearch,然后按元数据进行过滤。
  • 使用过滤器@type grep通过流利的元数据进行grep操作,并仅发布您的服务。
  • 使用过滤器@type rewrite_tag_filter将元数据转换为标签,然后使用流利的匹配规则来分隔日志。
相关问题