nxlog – multiline – char conversion

We have an application who’s generating one logfile per client within a directory… For sure, this app can write multiline logs, the file encoding is UTF-16. And sadly, it is slow to remove file lock. When we work on using nxlog to read theese files, there was an average of 3500 log files by server.

We choosed to poll every minute, and to not read from last, in case of reading problem. As we can’t do the utf16->utf8 conversion in the im_file block (nxlog crash ! ), we used it in the multiline block (the sooner the better as said by nxlog). We prune logfiles every day.

<Extension fileop>
Module xm_fileop

<Schedule>
When @daily
Exec file_remove("PathToLogs\\*");
</Schedule>
</Extension>

<Extension charconv>
    Module      xm_charconv
    AutodetectCharsets ucs-2le, ascii, utf-7, utf-8, utf-16, utf-32, iso8859-2, windows-1252, ucs2
</Extension>

<Extension Logs>
    Module	xm_multiline
    HeaderLine /^Start/
    Exec $raw_event = convert($raw_event,"utf-16","utf-8");
</Extension>

<Input MyApp>
	Module im_file
	File "PathToLogs\\*"
	SavePos TRUE
	ReadFromLast FALSE
	CloseWhenIdle TRUE
	PollInterval 60
	InputType Logs
	Exec if $raw_event !~ /^L/  drop(); \
		$ClientNAme = replace(file_basename(file_name()),'.LOG','');

</Input>

<Output out>
    Module      om_udp
    Host        graylog
    Port        12201
    OutputType	GELF
</Output>

<Route 1>
    Path        myApp => out
</Route>