Error: Is a directory @ io_getpartial - /var/log/journal

[a-x-]

newrelic recommends fluentd as a log forwarder.
So, i'm trying to setup it...
I just need to pass all the systems logs
I get this ^ error in newrelic

/etc/td-agent/td-agent.conf

# installation: td-agent-gem install fluent-plugin-systemd -v 1.0.3

# Caddy from journald
<source>
  @type systemd
  tag caddy
  path /var/log/journal
  matches [{ "_SYSTEMD_UNIT": "caddy.service" }]
  read_from_head true

  <storage>
    @type local
    path /var/log/fluentd/journald-caddy-cursor.json
  </storage>

  <entry>
    fields_strip_underscores true
    fields_lowercase true
  </entry>
</source>


# Newrelic Fluentd log plugin
# installation: td-agent-gem install fluent-plugin-newrelic
# Forward to Newrelic
<match **>
  @type newrelic
  license_key XXX
  base_uri https://log-api.eu.newrelic.com/log/v1
  <buffer>
    @type memory
    flush_interval 5s
  </buffer>
</match>

<system>
  root_dir /var/log/fluentd
</system>

/var/log/td-agent/td-agent.log

2023-04-05 12:35:53 +0000 [info]: starting fluentd-1.15.3 pid=6838 ruby="3.1.2"
2023-04-05 12:35:53 +0000 [info]: spawn command to main:  cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2023-04-05 12:35:53 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-04-05 12:35:53 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2023-04-05 12:35:53 +0000 [info]: adding match pattern="**" type="newrelic"
2023-04-05 12:35:53 +0000 [warn]: #0 'flush_interval' is ignored because default 'flush_mode' is not 'interval': 'lazy'
2023-04-05 12:35:53 +0000 [info]: adding match pattern="debug.**" type="stdout"
2023-04-05 12:35:53 +0000 [info]: adding source type="systemd"
2023-04-05 12:35:54 +0000 [warn]: #0 detect empty plugin storage file during startup. Ignored: /var/log/fluentd/journald-caddy-cursor.json
2023-04-05 12:35:54 +0000 [info]: adding source type="forward"
2023-04-05 12:35:54 +0000 [warn]: #0 'type' is deprecated parameter name. use '@type' instead.
2023-04-05 12:35:54 +0000 [info]: adding source type="unix"
2023-04-05 12:35:54 +0000 [warn]: #0 'type' is deprecated parameter name. use '@type' instead.
2023-04-05 12:35:54 +0000 [info]: adding source type="http"
2023-04-05 12:35:54 +0000 [info]: adding source type="debug_agent"
2023-04-05 12:35:54 +0000 [warn]: #0 define <match fluent.**> to capture fluentd logs in top level is deprecated. Use <label @FLUENT_LOG> instead
2023-04-05 12:35:54 +0000 [info]: #0 starting fluentd worker pid=6847 ppid=6844 worker=0
2023-04-05 12:35:54 +0000 [info]: #0 [input_debug_agent] listening dRuby uri="druby://127.0.0.1:24230" object="Fluent::Engine" worker=0
2023-04-05 12:35:54 +0000 [info]: #0 listening fluent socket on /var/run/td-agent/td-agent.sock
2023-04-05 12:35:54 +0000 [info]: #0 [input_forward] listening port port=24224 bind="0.0.0.0"
2023-04-05 12:35:54 +0000 [error]: #0 broken content for plugin storage (Hash required: ignored) type=NilClass
2023-04-05 12:35:54 +0000 [info]: #0 fluentd worker is now running worker=0

My setup:

# install logger dependency: https://docs.fluentd.org/installation/install-by-deb
# increase file descriptors
echo "# https://docs.fluentd.org/installation/before-install"  >> /etc/security/limits.conf
echo "root soft nofile 65536" >> /etc/security/limits.conf
echo "root hard nofile 65536" >> /etc/security/limits.conf
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
reboot

# td-agent 4 (experimental)
curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-jammy-td-agent4.sh | sh
systemctl status td-agent.service

[a-x-]

also, I don't understand what the *cursor.json and Is it plugin work w/o kubernetes (I saw kubelet in readme)

[errm]

Hi @a-x- thanks for the issue.

The storage records the offset of the journal cursor - this means if fluentd shuts down for some reason - then this plugin should be able to keep reading logs from where it left off.

This plugin doesn't directly intergrate with kuberntes - but of course you could run a container with a fluentd setup with this plugin in kubernetes - and as the readme shows you could tail the logs of some kuberentes components (if you were running them with systemd)