Created
February 12, 2015 10:56
-
-
Save anonymous/9afe80ad604f9a3d3c00 to your computer and use it in GitHub Desktop.
Logstash Multiline parsing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| input { | |
| stdin {codec => multiline{ | |
| pattern => "^ -%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601}" | |
| negate => true | |
| what => "previous" | |
| } }} | |
| filter { | |
| grok { | |
| match => [ "message", "^ -%{SPACE}%{SPACE}%{TIMESTAMP_ISO8601:time} \[%{WORD:main}\] %{LOGLEVEL:loglevel}%{SPACE}%{SPACE}\(%{JAVACLASS:class}\) %{DATA:mydata}((?m)\n\t%{GREEDYDATA:stack}) " ] | |
| } | |
| date { | |
| match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"] | |
| } | |
| } | |
| output { | |
| elasticsearch { | |
| host => "localhost" | |
| } | |
| stdout { codec => rubydebug} | |
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "@timestamp" => "2015-02-12T09:55:16.056Z", | |
| "message" => " - 2014-04-29 13:04:23,700 [main] INFO (api.batch.ThreadPoolWorker) Loading properties from classpath resource file:/env3/logiciels/splapp/ | |
| config/threadpoolworker.properties\n", | |
| "@version" => "1", | |
| "host" => "myHost", | |
| "time" => "2014-01-14 11:09:38,623", | |
| "main" => "main", | |
| "loglevel" => "ERROR", | |
| "class" => "support.context.ContextFactory", | |
| "mydata" => "Error getting connection to database jdbc:oracle:thin, with user cisuser and driver oracle" | |
| "stack" => "" | |
| } | |
| { | |
| "@timestamp" => "2015-02-12T09:55:16.056Z", | |
| "message" => " - 2014-01-14 11:09:38,623 [main] ERROR (support.context.Contextto database jdbc:oracle:thin, with user cisuser and driver oracle\n\tat oracle.jdbc.dron(SQLStateMapping.java:70)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:131)\n\tat oracle.jdbc.driver.DaseError.java:141)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:151)\n\tat oracle.jdbc.driver.DatabaseErrova:161)\n", | |
| "@version" => "1", | |
| "host" => "myHost", | |
| "time" => "2014-01-14 11:09:38,623", | |
| "main" => "main", | |
| "loglevel" => "ERROR", | |
| "class" => "support.context.ContextFactory", | |
| "mydata" => "Error getting connection to database jdbc:oracle:thin, with user cisuser and driver oracle" | |
| "stack" => "at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)\n\tat oracle.jdbc.driver.DatabaseE.java:131)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:141)\n\tat oracle.jdbc.driver.DatabaseError.newSQ\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:161)\n" | |
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| - 2014-04-29 13:04:23,700 [main] INFO (api.batch.ThreadPoolWorker) Loading properties from classpath resource file:/env3/logiciel/splapp/standalone/config/threadpoolworker.properties | |
| - 2014-01-14 11:09:38,623 [main] ERROR (support.context.ContextFactory) Error getting connection to database jdbc:oracle, with user cisuser and driver oracle.jdbc.driver.OracleDriver | |
| at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70) | |
| at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:131) | |
| at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:141) | |
| at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:151) | |
| at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:161) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| { | |
| "@timestamp" => "2015-02-12T09:55:16.056Z", | |
| "message" => " - 2014-04-29 13:04:23,700 [main] INFO (api.batch.ThreadPoolWorker) Loading properties from classpath resource file:/env3/logiciels/splapp/config/threadpoolworker.properties\n - 2014-01-14 11:09:38,623 [main] ERROR (support.context.Contextto database jdbc:oracle:thin, with user cisuser and driver oracle\n\tat oracle.jdbc.dron(SQLStateMapping.java:70)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:131)\n\tat oracle.jdbc.driver.DaseError.java:141)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:151)\n\tat oracle.jdbc.driver.DatabaseErrova:161)\n", | |
| "@version" => "1", | |
| "host" => "myHost", | |
| "time" => "2014-01-14 11:09:38,623", | |
| "main" => "main", | |
| "loglevel" => "ERROR", | |
| "class" => "support.context.ContextFactory", | |
| "mydata" => "Error getting connection to database jdbc:oracle:thin, with user cisuser and driver oracle" | |
| "stack" => "at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)\n\tat oracle.jdbc.driver.DatabaseE.java:131)\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:141)\n\tat oracle.jdbc.driver.DatabaseError.newSQ\n\tat oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:161)\n" | |
| } |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Okay I tested the filter multiline with a wildcard it actually works. I hope there are no bugs related to it because when I'll start parsing millions of log lines I won't be able to tell whether lines were merged from different files.
Thank you wiibaa!