We have an application who’s generating one logfile per client within a directory… For sure, this app can write multiline logs, the file encoding is UTF-16. And sadly, it is slow to remove file lock. When we work on using nxlog to read theese files, there was an average of 3500 log files by server.
We choosed to poll every minute, and to not read from last, in case of reading problem. As we can’t do the utf16->utf8 conversion in the im_file block (nxlog crash ! ), we used it in the multiline block (the sooner the better as said by nxlog). We prune logfiles every day.
<Extension fileop> Module xm_fileop <Schedule> When @daily Exec file_remove("PathToLogs\\*"); </Schedule> </Extension> <Extension charconv> Module xm_charconv AutodetectCharsets ucs-2le, ascii, utf-7, utf-8, utf-16, utf-32, iso8859-2, windows-1252, ucs2 </Extension> <Extension Logs> Module xm_multiline HeaderLine /^Start/ Exec $raw_event = convert($raw_event,"utf-16","utf-8"); </Extension> <Input MyApp> Module im_file File "PathToLogs\\*" SavePos TRUE ReadFromLast FALSE CloseWhenIdle TRUE PollInterval 60 InputType Logs Exec if $raw_event !~ /^L/ drop(); \ $ClientNAme = replace(file_basename(file_name()),'.LOG',''); </Input> <Output out> Module om_udp Host graylog Port 12201 OutputType GELF </Output> <Route 1> Path myApp => out </Route>