Options of syslog-parser parsers
parser: Parse and segment structured messages > Parsing syslog messages > Options of syslog-parser parsers
The syslog-parser has the following options.
default-facility()
Description: This parameter assigns a facility value to the messages received from the file source if the message does not specify one.
default-priority()
Description: This parameter assigns an emergency level to the messages received from the file source if the message does not specify one. For example, default-priority(warning).
flags()
Description: Specifies the log parsing options of the source.
- assume-utf8: The assume-utf8 flag assumes that the incoming messages are UTF-8 encoded, but does not verify the encoding. If you explicitly want to validate the UTF-8 encoding of the incoming message, use the validate-utf8 flag.
- empty-lines: Use the empty-lines flag to keep the empty lines of the messages. By default, syslog-ng OSE removes empty lines automatically.
- expect-hostname: If the expect-hostname flag is enabled, syslog-ng OSE will assume that the log message contains a hostname and parse the message accordingly. This is the default behavior for TCP sources. Note that pipe sources use the no-hostname flag by default.
- kernel: The kernel flag makes the source default to the LOG_KERN | LOG_NOTICE priority if not specified otherwise.
- no-hostname: Enable the no-hostname flag if the log message does not include the hostname of the sender host. That way, syslog-ng OSE assumes that the first part of the message header is ${PROGRAM} instead of ${HOST}. For example:
source s_dell { network(port(2000) flags(no-hostname)); }; - no-multi-line: The no-multi-line flag disables line-breaking in the messages: the entire message is converted to a single line. Note that this happens only if the underlying transport method actually supports multi-line messages. Currently the file() and pipe() drivers support multi-line messages.
- no-parse: By default, syslog-ng OSE parses incoming messages as syslog messages. The no-parse flag completely disables syslog message parsing and processes the complete line as the message part of a syslog message. The syslog-ng OSE application will generate a new syslog header (timestamp, host, and so on) automatically and put the entire incoming message into the MESSAGE part of the syslog message (available using the ${MESSAGE} macro). This flag is useful for parsing messages not complying to the syslog format.If you are using the flags(no-parse) option, then syslog message parsing is completely disabled, and the entire incoming message is treated as the ${MESSAGE} part of a syslog message. In this case, syslog-ng OSE generates a new syslog header (timestamp, host, and so on) automatically. Note that since flags(no-parse) disables message parsing, it interferes with other flags, for example, disables flags(no-multi-line).
- dont-store-legacy-msghdr: By default, syslog-ng stores the original incoming header of the log message. This is useful if the original format of a non-syslog-compliant message must be retained (syslog-ng automatically corrects minor header errors, for example, adds a whitespace before msg in the following message: Jan 22 10:06:11 host program:msg). If you do not want to store the original header of the message, enable the dont-store-legacy-msghdr flag.
- sanitize-utf8: When using the sanitize-utf8 flag, syslog-ng OSE converts non-UTF-8 input to an escaped form, which is valid UTF-8.
- store-raw-message: Save the original message as received from the client in the ${RAWMSG} macro. You can forward this raw message in its original form to another syslog-ng node using the syslog-ng() destination, or to a SIEM system, ensuring that the SIEM can process it. Available only in 3.16 and later.
- syslog-protocol: The syslog-protocol flag specifies that incoming messages are expected to be formatted according to the new IETF syslog protocol standard (RFC5424), but without the frame header. Note that this flag is not needed for the syslog driver, which handles only messages that have a frame header.
- validate-utf8: The validate-utf8 flag enables encoding-verification for messages formatted according to the new IETF syslog standard (for details, see IETF-syslog messages). If theBOM1character is missing, but the message is otherwise UTF-8 compliant, syslog-ng automatically adds the BOM character to the message.
template()
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
Parsing messages with comma-separated and similar values
parser: Parse and segment structured messages > Parsing messages with comma-separated and similar values
The syslog-ng OSE application can separate parts of log messages (that is, the contents of the ${MESSAGE} macro) at delimiter characters or strings to named fields (columns). One way to achieve this is to use a csv (comma-separated-values) parser (for other methods and possibilities, see the other sections of parser: Parse and segment structured messages. The parsed fields act as user-defined macros that can be referenced in message templates, file- and tablenames, and so on.
Parsers are similar to filters: they must be defined in the syslog-ng OSE configuration file and used in the log statement. You can also define the parser inline in the log path.
To create a csv-parser(), you have to define the columns of the message, the separator characters or strings (also called delimiters, for example, semicolon or tabulator), and optionally the characters that are used to escape the delimiter characters (quote-pairs()).
Declaration:
parser{ csv-parser( columns(column1, column2, ...) delimiters(chars(" "), strings(" ")) ); };
Column names work like macros.
Names starting with a dot (for example, .example) are reserved for use by syslog-ng OSE. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard vs. soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
Example: Segmenting hostnames separated with a dash
The following example separates hostnames like example-1 and example-2 into two parts.
parser p_hostname_segmentation {
csv-parser(columns("HOSTNAME.NAME", "HOSTNAME.ID")
delimiters("-")
flags(escape-none)
template("${HOST}"));
};
destination d_file {
file("/var/log/messages-${HOSTNAME.NAME:-examplehost}");
};
log {
source(s_local);
parser(p_hostname_segmentation);
destination(d_file);
};
Example: Parsing Apache log files
The following parser processes the log of Apache web servers and separates them into different fields. Apache log messages can be formatted like:
"%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %T %v"
Here is a sample message:
192.168.1.1 - - [31/Dec/2007:00:17:10 +0100] "GET /cgi-bin/example.cgi HTTP/1.1" 200 2708 "-" "curl/7.15.5 (i4 86-pc-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8c zlib/1.2.3 libidn/0.6.5" 2 example.mycompany
To parse such logs, the delimiter character is set to a single whitespace (delimiters(" ")). Whitespaces between quotes and brackets are ignored (quote-pairs('""[]')).
parser p_apache {
csv-parser(
columns("APACHE.CLIENT_IP", "APACHE.IDENT_NAME", "APACHE.USER_NAME",
"APACHE.TIMESTAMP", "APACHE.REQUEST_URL", "APACHE.REQUEST_STATUS",
"APACHE.CONTENT_LENGTH", "APACHE.REFERER", "APACHE.USER_AGENT",
"APACHE.PROCESS_TIME", "APACHE.SERVER_NAME")
flags(escape-double-char,strip-whitespace)
delimiters(" ")
quote-pairs('""[]')
);
};
The results can be used for example to separate log messages into different files based on the APACHE.USER_NAME field. If the field is empty, the nouser name is assigned.
log {
source(s_local);
parser(p_apache);
destination(d_file);
};
destination d_file {
file("/var/log/messages-${APACHE.USER_NAME:-nouser}");
};
Example: Segmenting a part of a message
Multiple parsers can be used to split a part of an already parsed message into further segments. The following example splits the timestamp of a parsed Apache log message into separate fields.
parser p_apache_timestamp {
csv-parser(
columns("APACHE.TIMESTAMP.DAY", "APACHE.TIMESTAMP.MONTH", "APACHE.TIMESTAMP.YEAR", "APACHE.TIMESTAMP.HOUR", "APACHE.TIMESTAMP.MIN", "APACHE.TIMESTAMP.MIN", "APACHE.TIMESTAMP.ZONE")
delimiters("/: ")
flags(escape-none)
template("${APACHE.TIMESTAMP}")
);
};
log {
source(s_local);
parser(p_apache);
parser(p_apache_timestamp);
destination(d_file);
};
Further examples:
- For an example on using the greedy option, see Example: Adding the end of the message to the last column.
Options of CSV parsers
parser: Parse and segment structured messages > Parsing messages with comma-separated and similar values > Options of CSV parsers
The syslog-ng OSE application can separate parts of log messages (that is, the contents of the ${MESSAGE} macro) at delimiter characters or strings to named fields (columns). One way to achieve this is to use a csv (comma-separated-values) parser (for other methods and possibilities, see the other sections of parser: Parse and segment structured messages. The parsed fields act as user-defined macros that can be referenced in message templates, file- and tablenames, and so on.
Parsers are similar to filters: they must be defined in the syslog-ng OSE configuration file and used in the log statement. You can also define the parser inline in the log path.
To create a csv-parser(), you have to define the columns of the message, the separator characters or strings (also called delimiters, for example, semicolon or tabulator), and optionally the characters that are used to escape the delimiter characters (quote-pairs()).
Declaration:
parser{ csv-parser( columns(column1, column2, ...) delimiters(chars(" "), strings(" ")) ); };
Column names work like macros.
Names starting with a dot (for example, .example) are reserved for use by syslog-ng OSE. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard vs. soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
columns()
Description: Specifies the name of the columns to separate messages to. These names will be automatically available as macros. The values of these macros do not include the delimiters.
delimiters()
Description: The delimiter is the character or string that separates the columns in the message. If you specify multiple characters using the delimiters(chars("")) option, every character will be treated as a delimiter. To separate the columns at the tabulator (tab character), specify \t. For example, to separate the text at every hyphen (-) and colon (:) character, use delimiters(chars("-:")), Note that the delimiters will not be included in the column values.
String delimiters:
If you have to use a string as a delimiter, list your string delimiters in the delimiters(strings("", "", ...)") format.
By default, syslog-ng OSE uses space as a delimiter. If you want to use only the strings as delimiters, you have to disable the space delimiter, for example: delimiters(chars(""), strings(""))
Multiple delimiters:
If you use more than one delimiter, note the following points:
- syslog-ng OSE will split the message at the nearest possible delimiter. The order of the delimiters in the configuration file does not matter.
- You can use both string delimiters and character delimiters in a parser.
- The string delimiters can include characters that are also used as character delimiters.
- If a string delimiter and a character delimiter both match at the same position of the message, syslog-ng OSE uses the string delimiter.
dialect()
Description: Specifies how to handle escaping in the parsed message. The following values are available. Default value: escape-none
- escape-backslash: The parsed message uses the backslash (\) character to escape quote characters.
- escape-double-char: The parsed message repeats the quote character when the quote character is used literally. For example, to escape a comma (,), the message contains two commas (,,).
- escape-none: The parsed message does not use any escaping for using the quote character literally.
parser p_demo_parser {
csv-parsercsv-parser(
prefix(".csv.")
delimiters(" ")
dialect(escape-backslash)
flags(strip-whitespace, greedy)
columns("column1", "column2", "column3")
);
};
flags()
Description: Specifies various options for parsing the message. The following flags are available:
- drop-invalid: When the drop-invalid option is set, the parser does not process messages that do not match the parser. For example, a message does not match the parser if it has less columns than specified in the parser, or it has more columns but the greedy flag is not enabled. Using the drop-invalid option practically turns the parser into a special filter, that matches messages that have the predefined number of columns (using the specified delimiters).
- escape-backslash: The parsed message uses the backslash (\) character to escape quote characters.
- escape-double-char: The parsed message repeats the quote character when the quote character is used literally. For example, to escape a comma (,), the message contains two commas (,,).
- escape-none: The parsed message does not use any escaping for using the quote character literally.
- greedy: The greedy option assigns the remainder of the message to the last column, regardless of the delimiter characters set. You can use this option to process messages where the number of columns varies.
Example: Adding the end of the message to the last column
If the greedy option is enabled, the syslog-ng application adds the not-yet-parsed part of the message to the last column, ignoring any delimiter characters that may appear in this part of the message.For example, you receive the following comma-separated message: example 1, example2, example3, and you segment it with the following parser:csv-parser(columns("COLUMN1", "COLUMN2", "COLUMN3") delimiters(","));The COLUMN1, COLUMN2, and COLUMN3 variables will contain the strings example1, example2, and example3, respectively. If the message looks like example 1, example2, example3, some more information, then any text appearing after the third comma (that is, some more information) is not parsed, and possibly lost if you use only the variables to reconstruct the message (for example, to send it to different columns of an SQL table).Using the greedy flag will assign the remainder of the message to the last column, so that the COLUMN1, COLUMN2, and COLUMN3 variables will contain the strings example1, example2, and example3, some more information.csv-parser(columns("COLUMN1", "COLUMN2", "COLUMN3") delimiters(",") flags(greedy)); - strip-whitespace: The strip-whitespace flag removes leading and trailing whitespaces from all columns.
null()
Description: If the value of a column is the value of the null() parameter, syslog-ng OSE changes the value of the column to an empty string. For example, if the columns of the message contain the "N/A" string to represent empty values, you can use the null("N/A") option to change these values to empty stings.
prefix()
Description: Insert a prefix before the name part of the parsed name-value pairs to help further processing. For example:
- To insert the my-parsed-data. prefix, use the prefix(my-parsed-data.) option.
- To refer to a particular data that has a prefix, use the prefix in the name of the macro, for example, ${my-parsed-data.name}.
- If you forward the parsed messages using the IETF-syslog protocol, you can insert all the parsed data into the SDATA part of the message using the prefix(.SDATA.my-parsed-data.) option.
Names starting with a dot (for example, .example) are reserved for use by syslog-ng OSE. If you use such a macro name as the name of a parsed value, it will attempt to replace the original value of the macro (note that only soft macros can be overwritten, see Hard vs. soft macros for details). To avoid such problems, use a prefix when naming the parsed values, for example, prefix(my-parsed-data.)
quote-pairs()
Description: List quote-pairs between single quotes. Delimiter characters or strings enclosed between quote characters are ignored. Note that the beginning and ending quote character does not have to be identical, for example [} can also be a quote-pair. For an example of using quote-pairs() to parse Apache log files, see Example: Parsing Apache log files.
template()
Description: The macro that contains the part of the message that the parser will process. It can also be a macro created by a previous parser of the log path. By default, the parser processes the entire message (${MESSAGE}).
For examples, see Example: Segmenting hostnames separated with a dash and Example: Segmenting a part of a message.
Parsing key=value pairs
The syslog-ng OSE application can separate a message consisting of whitespace or comma-separated key=value pairs (for example, Postfix log messages) into name-value pairs. You can also specify other separator character instead of the equal sign, for example, colon (:) to parse MySQL log messages. The syslog-ng OSE application automatically trims any leading or trailing whitespace characters from the keys and values, and also parses values that contain unquoted whitespace. For details on using value-pairs in syslog-ng OSE see Structuring macros, metadata, and other value-pairs.
You can refer to the separated parts of the message using the key of the value as a macro. For example, if the message contains KEY1=value1,KEY2=value2, you can refer to the values as ${KEY1} and ${KEY2}.
To parse key=value pairs, define a parser that has the kv-parser() option. Defining the prefix is optional. By default, the parser will process the ${MESSAGE} part of the log message. You can also define the parser inline in the log path.
Declaration:
parser parser_name {
kv-parser(
prefix()
);
};
Example: Using a key=value parser
In the following example, the source is a log message consisting of comma-separated key=value pairs, for example, a Postfix log message:
Jun 20 12:05:12 mail.example.compostfix/qmgr[35789]: EC2AC1947DA: from= , size=807, nrcpt=1 (queue active)
The kv-parser inserts the ".kv." prefix before all extracted name-value pairs. The destination is a file, that uses the format-json template function. Every name-value pair that begins with a dot (".") character will be written to the file (dot-nv-pairs). The log line connects the source, the destination and the parser.
source s_kv {
network(port(21514));
};
destination d_json {
file("/tmp/test.json"
template("$(format-json --scope dot-nv-pairs)\n"));
};
parser p_kv {
kv-parser (prefix(".kv."));
};
log {
source(s_kv);
parser(p_kv);
destination(d_json);
};
You can also define the parser inline in the log path.
source s_kv {
network(port(21514));
};
destination d_json {
file("/tmp/test.json"
template("$(format-json --scope dot-nv-pairs)\n"));
};
log {
source(s_kv);
parser {
kv-parser (prefix(".kv."));
};
destination(d_json);
};
You can set the separator character between the key and the value to parse for example key:value pairs, like MySQL logs:
Mar 7 12:39:25 myhost MysqlClient[20824]: SYSTEM_USER:'oscar', MYSQL_USER:'my_oscar', CONNECTION_ID:23, DB_SERVER:'127.0.0.1', DB:'--', QUERY:'USE test;'
parser p_mysql {
kv-parser(value-separator(":") prefix(".mysql."));
};
