Created
February 12, 2015 10:56
-
-
Save anonymous/9afe80ad604f9a3d3c00 to your computer and use it in GitHub Desktop.
Logstash Multiline parsing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| input { | |
| stdin {codec => multiline{ | |
| pattern => "^ -%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601}" | |
| negate => true | |
| what => "previous" | |
| } }} | |
| filter { | |
| grok { | |
| match => [ "message", "^ -%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601:time} \[%{WORD:main}\] %{LOGLEVEL:loglevel}%{SPACE}%{SPACE}\(%{JAVACLASS:class}\) %{DATA:mydata}((?m)\n\t%{GREEDYDATA:stack}) " ] | |
| } | |
| date { | |
| match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"] | |
| } | |
| } | |
| output { | |
| elasticsearch { | |
| host => "localhost" | |
| } | |
| stdout { codec => rubydebug} | |
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "@timestamp" => "2015-02-12T09:55:16.056Z", | |
| "message" => " - 2014-04-29 13:04:23,700 [main] INFO (api.batch.ThreadPoolWorker) Loading properties from classpath resource file:/env3/logiciels/splapp/ | |
| config/threadpoolworker.properties\n", | |
| "@version" => "1", | |
| "host" => "myHost", | |
| "time" => "2014-01-14 11:09:38,623", | |
| "main" => "main", | |
| "loglevel" => "ERROR", | |
| "class" => "support.context.ContextFactory", | |
| "mydata" => "Error getting connection to database jdbc:oracle:thin, with user cisuser and driver oracle" | |
| "stack" => "" | |
| } | |
| { | |
| "@timestamp" => "2015-02-12T09:55:16.056Z", | |
| "message" => " - 2014-01-14 11:09:38,623 [main] ERROR (support.context.Contextto database jdbc:oracle:thin, with user cisuser and driver oracle\n\tat oracle.jdbc.dron(SQLStateMapping.java:70)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:131)\n\tat oracle.jdbc.driver.DaseError.java:141)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:151)\n\tat oracle.jdbc.driver.DatabaseErrova:161)\n", | |
| "@version" => "1", | |
| "host" => "myHost", | |
| "time" => "2014-01-14 11:09:38,623", | |
| "main" => "main", | |
| "loglevel" => "ERROR", | |
| "class" => "support.context.ContextFactory", | |
| "mydata" => "Error getting connection to database jdbc:oracle:thin, with user cisuser and driver oracle" | |
| "stack" => "at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)\n\tat oracle.jdbc.driver.DatabaseE.java:131)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:141)\n\tat oracle.jdbc.driver.DatabaseError.newSQ\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:161)\n" | |
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| - 2014-04-29 13:04:23,700 [main] INFO (api.batch.ThreadPoolWorker) Loading properties from classpath resource file:/env3/logiciel/splapp/standalone/config/threadpoolworker.properties | |
| - 2014-01-14 11:09:38,623 [main] ERROR (support.context.ContextFactory) Error getting connection to database jdbc:oracle, with user cisuser and driver oracle.jdbc.driver.OracleDriver | |
| at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70) | |
| at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:131) | |
| at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:141) | |
| at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:151) | |
| at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:161) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "@timestamp" => "2015-02-12T09:55:16.056Z", | |
| "message" => " - 2014-04-29 13:04:23,700 [main] INFO (api.batch.ThreadPoolWorker) Loading properties from classpath resource file:/env3/logiciels/splapp/config/threadpoolworker.properties\n - 2014-01-14 11:09:38,623 [main] ERROR (support.context.Contextto database jdbc:oracle:thin, with user cisuser and driver oracle\n\tat oracle.jdbc.dron(SQLStateMapping.java:70)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:131)\n\tat oracle.jdbc.driver.DaseError.java:141)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:151)\n\tat oracle.jdbc.driver.DatabaseErrova:161)\n", | |
| "@version" => "1", | |
| "host" => "myHost", | |
| "time" => "2014-01-14 11:09:38,623", | |
| "main" => "main", | |
| "loglevel" => "ERROR", | |
| "class" => "support.context.ContextFactory", | |
| "mydata" => "Error getting connection to database jdbc:oracle:thin, with user cisuser and driver oracle" | |
| "stack" => "at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)\n\tat oracle.jdbc.driver.DatabaseE.java:131)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:141)\n\tat oracle.jdbc.driver.DatabaseError.newSQ\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:161)\n" | |
| } |
When using the multiline codec, you cannot currently define an array or use a wildcard in the path => configuration due to logstash-plugins/logstash-input-file#10
The workaround is to use the multiline filter
Okay I tested the filter multiline with a wildcard it actually works. I hope there are no bugs related to it because when I'll start parsing millions of log lines I won't be able to tell whether lines were merged from different files.
Thank you wiibaa!
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi wiibaa, Thank you so much this helped me get the results i want. I added the "break_on_match => false" parameter so that grok won't break on the first match. and I used the file input so i won't have to copy-paste one line at a time.
Can you tell me what are the bugs I may encounter while using the input multiline codec? I believe you mentioned something about it before. Thank you again.