Pushing (volatile journald) system logs to New Relic

I set up New Relic Infrastructure Monitoring for my home router.

To minimize writes to the SSD and hopefully prolong its life, I configured the machine to ship all systemd/journald entries to New Relic instead of saving them to disk. (New Relic offers a generous free tier, just like their competition Grafana Cloud.)

Due to a bit scarce documentation that was harder than it should have been, so here’s what I did - hope this helps someone. The system runs Debian stable, and I put the New Relic Infrastructure Agent’s db directory (/var/db/newrelic-infra) on a tmpfs so it doesn’t write to disk either (see here for an example of how to set up a persistent ram disk).

Next, configure journald to not persist logs to disk (From the journald manual page):

/etc/systemd/journald.conf:

[Journal]
Storage=volatile
ForwardToSyslog=no

Configure the New Relic Infrastructure Agent to set up and start an instance of Fluent Bit (see its documentation).

/etc/newrelic-infra/logging.d/fluentbit.yml:

logs:
  - name: external-fluentbit-config-and-parsers-file
    fluentbit:
      config_file: /etc/fluent-bit/fluent-bit.conf
      parsers_file: /etc/fluent-bit/parsers.conf

And finally, configure Fluent Bit to forward all journal entries:

/etc/fluent-bit/fluent-bit.conf:

[INPUT]
    Name  systemd
    Path  /run/log/journal
    Tag   host.*

When restarting the New Relic Infrastructure Agent (sudo systemctl restart newrelic-infra), it should be spawning a Fluent Bit process:

sudo ps axf says:

[...]
1743164 ?        Ssl    0:00 /usr/bin/newrelic-infra-service
1743168 ?        Sl     1:35  \_ /usr/bin/newrelic-infra
1743204 ?        Sl     0:23      \_ /opt/fluent-bit/bin/fluent-bit -c /tmp/fb/nr_fb_config2386533890 -e /var/db/newrelic-infra/newrelic-integrations/logging/out_newrelic.so -R /var/db/newrelic-infra/newrelic-integrations/logging/parsers.conf -R /etc/fluent-bit/parsers.conf
[...]

From the fluent-bit command line above, /tmp/fb/nr_fb_config2386533890 holds the amended configuration to ship our logs to New Relic. If you’re like me, you might want to have a look at that file, just to see what it would have taken to make it work ;)

Tada!

Screenshot of logs in the New Relic web interface

florian.sesser.at/

Pushing (volatile journald) system logs to New Relic