slapos/collect: Preserve entries at the database for 15 days
This may case more memory usage and more 'live' data to handle, for this reason, I'm making it configurable
-
Developer
Hi @rafael :
What was the reason to extend the history to 15 days instead of 3 ?
I'm asking because on a recent production server I was getting alerts for too high load (around 50 of load for a 16-core CPU). Thanks to
iotop
I found thatslapos node collect
was using way too much io. I run a few test :root@server:/srv/slapgrid/var/data-log# du -sh collector.db 517M collector.db root@server:/root# time slapos node collect real 0m40.205s user 0m3.768s sys 0m3.760s
Then I switched the parameter in slapos.cfg to only keep 3 days of history :
root@server:/srv/slapgrid/var/data-log# du -sh collector.db 78M collector.db root@server:/srv/slapgrid/var/data-log# time slapos node collect # run twice, to test a "normal" case real 0m10.675s user 0m0.804s sys 0m0.660s
This server has only 2 slapparts used (a runner0 and a PBS).
So keeping only 3 days make the process 75% faster. As this process is executed every minute, 10s of run is acceptable, whereas 40s is killing my server. I would like to commit back the default value of 3 days, as even if this is configurable, I would prefer shipping with reasonable defaults.
Also, only entries marked as already exported are garbage-collected. I don't understand well how everything plays together, but I'm not sure of why we should better keep Day-15 entries, when, all entries older than Day-1 are already exported.
-
Owner
Can you try with pragma_mode=WAL ?
It is indeed not efficient to record a month specially for slow disks, perhaps is a lack of index or adjustment on the sqlite. The increment was to buuld a wider chart on the monitor UI, so I tested hold larger period.