使用filebeat进行多行匹配案例

使用filebeat进行多行匹配

一 、 测试tomcat 的错误日志

(1)对于tomcat错误日志生成

vim /baimei/softwares/tomcat/conf/server.xml

 

(2)启动tomcat

 

/baimei/softwares/tomcat/bin/startup.sh

(3) 查看错误日志：

cat /baimei/softwares/tomcat/logs/catalina.out 

 

 

31-May-2023 14:35:22.192 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
31-May-2023 14:35:22.247 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 1329 ms
01-Jun-2023 20:53:15.926 SEVERE [main] org.apache.tomcat.util.digester.Digester.fatalError Parse Fatal Error at line 161 column 8: The content of elements must consist of well-formed character data or markup.
	org.xml.sax.SAXParseException; systemId: file:/baimei/softwares/apache-tomcat-8.5.89/conf/server.xml; lineNumber: 161; columnNumber: 8; The content of elements must consist of well-formed character data or markup.
		at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:204)
		at com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.fatalError(ErrorHandlerWrapper.java:178)
		at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:400)
		at com.sun.org.apache.xerces.internal.impl.XMLErrorReporter.reportError(XMLErrorReporter.java:327)
		at com.sun.org.apache.xerces.internal.impl.XMLScanner.reportFatalError(XMLScanner.java:1472)
		at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.startOfMarkup(XMLDocumentFragmentScannerImpl.java:2637)
		at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2734)
		at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:605)
		at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:507)
		at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:867)
		at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:796)
		at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:142)
		at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1216)
		at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:644)
		at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1496)
		at org.apache.catalina.startup.Catalina.load(Catalina.java:618)
		at org.apache.catalina.startup.Catalina.load(Catalina.java:669)
		at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:307)
		at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:477)
01-Jun-2023 20:53:15.935 WARNING [main] org.apache.catalina.startup.Catalina.load Catalina.start using conf/server.xml: The content of elements must consist of well-formed character data or markup.

 

 

（4）filebeat采集tomcat错误日志

 

 

filebeat.inputs:
- type: log
  paths:
  - "/baimei/softwares/tomcat/logs/catalina.out*"
  # 表示指定多行匹配的类型
  multiline.type: pattern
  # 指定多行匹配的模式
  multiline.pattern: '^[0-9]{2}'
  # 详见图解: https://www.elastic.co/guide/en/beats/filebeat/7.17/multiline-examples.html
  multiline.negate: true
  multiline.match: after

output.console:
  pretty: true

 

启动：

filebeat -e -c  /root/config/multi.yaml | grep message

删除日志：

rm -rf /var/lib/filebeat/*

 

 

我们可以看不到， 错误的日志， 以 空格  at  org 开头的 非 数字开头的都合并到一起了。

那么匹配模式可以参考官网：

https://www.elastic.co/guide/en/beats/filebeat/7.17/multiline-examples.html

其他案例：

将下面的json数据匹配为3行：

cat /tmp/studnet.info

{
  "name": "孙建超",
  "hobby": ["抽烟","喝酒","烫头"]
}
{
  "name": "王宗玉",
  "hobby": ["点烟","倒酒","染发"]
}
{
  "name": "杨雄",
  "hobby": ["买烟","买酒","买染发膏"]
}

 

filebeat 配置：

[root@baimeidashu-elk111 ~/config/testmult]#cat 1.yaml 
filebeat.inputs:
- type: log
  paths:
  - "/tmp/studnet.info"
  multiline.type: pattern
  multiline.pattern: '^{'
  multiline.negate: true
  multiline.match: after

output.console:
  pretty: tru

 

 

删除：

rm -rf /var/lib/filebeat/*

启动：

filebeat -e -c  /root/config/testmult/1.yaml | grep message

 

方法2：

[root@baimeidashu-elk111 ~/config/testmult]#cat 2.yaml 
filebeat.inputs:
- type: log
  paths:
  - "/tmp/studnet.info"
  multiline.type: pattern
  multiline.pattern: '}$'
  multiline.negate: true
  multiline.match: before

output.console:
  pretty: true

 

方法3：

[root@baimeidashu-elk111 ~/config/testmult]#cat 3.yaml 
filebeat.inputs:
- type: log
  paths:
  - "/tmp/studnet.info"
  multiline.type: pattern
  multiline.pattern: '^}'
  multiline.negate: true
  multiline.match: before

output.console:
  pretty: true

 

方法4：

[root@baimeidashu-elk111 ~/config/testmult]#cat 4.yaml 
filebeat.inputs:
- type: log
  paths:
  - /tmp/studnet.info
  multiline.type: pattern
  multiline.pattern: '{'
  multiline.negate: true
  multiline.match: after
  multiline.flush_pattern: '}'

output.console:
  pretty: true

 

 

方法5： （最不推荐这种）

filebeat.inputs:
- type: log
  paths:
  - /tmp/studnet.info
  multiline:
    type: count
    count_lines: 4
output.console:
  pretty: true

控制行数，这种非常的不确定，除非规律明确。

 

欢迎来撩 ： 汇总all