Commit 8b7b0e92 authored by Kirill Smelkov's avatar Kirill Smelkov

software/simpleran: Fix fluentbit ingestion not to barf on large lines

Fluentbit Tail input documentation[1] says that by default maximum buffer
size is 32K which turned out to be too small in practice because we hit
a situation where enb.xlog started to have lines with ~ 34K and so
fluentbit ingestion stopped to work with the following error in
fluentbit log:

    [2024/10/23 20:30:23] [error] [input:tail:tail.0] file=/srv/slapgrid/slappart19/srv/monitor/public/enb.xlog requires a larger buffer size, lines are too long. Skipping file.

-> Fix that by increasing max buffer size to 1M which seems to be high
   enough at least for now.

Maybe it will make sense to configure this as unlimited, but I'm not
sure if going as unlimited is universally a good idea.

[1] https://docs.fluentbit.io/manual/pipeline/inputs/tail

/cc @lu.xu, @jhuge, @tomo
/reviewed-by @paul.graydon
/reviewed-on nexedi/slapos!1672
parent 7a237f05
Pipeline #37683 failed with stage
in 0 seconds
......@@ -60,7 +60,7 @@ md5sum = 52da9fe3a569199e35ad89ae1a44c30e
[template-enb]
_update_hash_filename_ = instance-enb.jinja2.cfg
md5sum = bda6ff7bc76cf73e3bd55aca21134a3a
md5sum = 9cd5250fc523863d206b8dc038beaf82
[template-ors-enb]
_update_hash_filename_ = instance-ors-enb.jinja2.cfg
......
......@@ -202,6 +202,7 @@ inline =
tag ${xlog-fluentbit-tag:xlog-fluentbit-tag}
Read_from_Head True
db ${directory:xlog-fluentbit}/tail-state
Buffer_Max_Size 1M
[OUTPUT]
name forward
match *
......
Markdown is supported
0%
or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment