Not able to send logs to syslog-ng aggregator using syslog exporter on AKS

[ahujarajesh]

Component(s)

exporter/syslog

What happened?

Description

I have a AKS cluster where I have deployed Open Telemetry Collector to export logs to centralized log aggregator
using Syslog-ng.

1. Create AKS Cluster, Create node pool.
2. Deploy Opentelemetry collector using syslog exporter to tail pod logs and send those to the log aggregator.
3. in otel collector used RFC: rfc3164
4. on Syslog-ng Server we get garbage folder names as below:
   `root@test-logserver:/store/logs# ls
   ''$'\001'
   ''$'\002\001'
   ''$'\005'
   ''$'\005\001'
   ''$'\005\004\003\004\003\003'
   ''$'\b'
   ''$'\v'
   ''$'\022'
   ''$'\023\001\023\002\023\003\001'
   ''$'\026\003\001'
   ''$'\027'
   ''$'\030'
   ''$'\030\b\004\004\003\b\a\b\005\b\006\004\001\005\001\006\001\005\003\006\003\002\001\002\003\377\001'
   ''$'\031'
   ''$'\032'
   ''$'\035'
   '$'
   '&'
   '&'$'\300''+'$'\300'

2025
3
'5'$'\300\022'
'O'$'\333''j'$'\367\216\266\210''-'$'\210\017''^'$'\016\272\375'
'{'
''$'\234'
''$'\235'
''$'\300\024'
''$'\352\003\003'
''$'\352\003\003\225\236\251''U'$'\321''FW'
''$'\352\003\003\305''&up'$'\333\376\035\204''9.'$'\364\254''Eb'$'\231\030\370''o38k&'$'\210\316''K'$'\305\035''|]^3' '\352\003\003\352''u-A'$'\331\034''z'$'\314\327''2'$'\250\322\026\212\270\303''Y뭢o('$'\234\273\235'\'''$'\370\353\354\204''*'$'\350' ''$'\356\001'

Steps to Reproduce

Expected Result

Logs should be created with Program name/ App Name folders on log server

Actual Result

Logs are not received only garbage data is coming with rfc3164 and no data is received with rfc5424.

Collector version

0.91.0

Environment information

Environment

OS:
Distributor ID: Ubuntu
Description: Ubuntu 22.04.5 LTS
Release: 22.04
Codename: jammy

OpenTelemetry Collector configuration

extensions:
      file_storage:
        directory: /var/lib/otelcol/agent_logs_buffer
        timeout: 1s
      health_check:
        endpoint: 0.0.0.0:13133
      pprof:
        endpoint: 0.0.0.0:1777
      zpages:
        endpoint: 0.0.0.0:55679

    receivers:
      filelog/syslog:
        include:
          - /var/log/pods/kube-system_kube-proxy-h47f5_dd7e0dfa-f4cf-4ce1-b04a-a733065ae409/kube-proxy/02.log
        exclude:
          - /var/log/pods/*/otel-collector-syslog*/*.log
        force_flush_period: 0
        include_file_name: false
        include_file_path: true
        operators:
          - type: syslog_parser
            protocol: rfc3164

    processors:
      batch:
        timeout: 10s
        send_batch_size: 1024
        send_batch_max_size: 2048
      
      resource:
        attributes:
          - key: service.name
            value: "otel-collector-syslog"
            action: insert
          - key: service.version
            value: "1.0.0"
            action: insert

    exporters:
      debug:
        verbosity: detailed
      syslog:
        endpoint: "10.1.96.4"
        port: 516
        network: tcp
        protocol: rfc3164
        retry_on_failure:
          enabled: true
          initial_interval: 5s
          max_interval: 30s
          max_elapsed_time: 300s
        sending_queue:
          enabled: true
          storage: file_storage
          queue_size: 1000
          num_consumers: 4

    service:
      extensions: [file_storage, health_check, pprof, zpages]
      pipelines:
        logs/syslog:
          receivers: [filelog/syslog]
          processors: [resource, batch]
          exporters: [debug, syslog]
      telemetry:
        logs:
          level: info
          encoding: json
        metrics:
          level: none

Log output

Collector Logs:

`
{"level":"info","ts":1748853605.6878412,"caller":"service@v0.91.0/telemetry.go:78","msg":"Skipping telemetry setup.","address":":8888","level":"None"}
{"level":"info","ts":1748853605.688055,"caller":"exporter@v0.91.0/exporter.go:275","msg":"Development component. May change in the future.","kind":"exporter","data_type":"logs","name":"debug"}
{"level":"info","ts":1748853605.6881795,"caller":"syslogexporter@v0.91.0/exporter.go:42","msg":"Syslog Exporter configured","kind":"exporter","data_type":"logs","name":"syslog","endpoint":"10.1.96.4","protocol":"rfc3164","port":516}
{"level":"info","ts":1748853605.6885784,"caller":"service@v0.91.0/service.go:145","msg":"Starting otelcol-contrib...","Version":"0.91.0","NumCPU":2}
{"level":"info","ts":1748853605.6885912,"caller":"extensions/extensions.go:34","msg":"Starting extensions..."}
{"level":"info","ts":1748853605.6886132,"caller":"extensions/extensions.go:37","msg":"Extension is starting...","kind":"extension","name":"zpages"}
{"level":"warn","ts":1748853605.6886632,"caller":"zpagesextension@v0.91.0/zpagesextension.go:55","msg":"zPages span processor registration is not available","kind":"extension","name":"zpages"}
{"level":"info","ts":1748853605.6887212,"caller":"zpagesextension@v0.91.0/zpagesextension.go:63","msg":"Registered Host's zPages","kind":"extension","name":"zpages"}
{"level":"info","ts":1748853605.6889951,"caller":"zpagesextension@v0.91.0/zpagesextension.go:75","msg":"Starting zPages extension","kind":"extension","name":"zpages","config":{"TCPAddr":{"Endpoint":"0.0.0.0:55679"}}}
{"level":"info","ts":1748853605.6891043,"caller":"extensions/extensions.go:52","msg":"Extension started.","kind":"extension","name":"zpages"}
{"level":"info","ts":1748853605.6892283,"caller":"extensions/extensions.go:37","msg":"Extension is starting...","kind":"extension","name":"pprof"}
{"level":"info","ts":1748853605.6892784,"caller":"pprofextension@v0.91.0/pprofextension.go:60","msg":"Starting net/http/pprof server","kind":"extension","name":"pprof","config":{"TCPAddr":{"Endpoint":"0.0.0.0:1777"},"BlockProfileFraction":0,"MutexProfileFraction":0,"SaveToFile":""}}
{"level":"info","ts":1748853605.6893675,"caller":"extensions/extensions.go:52","msg":"Extension started.","kind":"extension","name":"pprof"}
{"level":"info","ts":1748853605.6893764,"caller":"extensions/extensions.go:37","msg":"Extension is starting...","kind":"extension","name":"health_check"}
{"level":"info","ts":1748853605.6894536,"caller":"healthcheckextension@v0.91.0/healthcheckextension.go:35","msg":"Starting health_check extension","kind":"extension","name":"health_check","config":{"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"ResponseHeaders":null,"Path":"/","ResponseBody":null,"CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
{"level":"warn","ts":1748853605.6896803,"caller":"internal@v0.91.0/warning.go:40","msg":"Using the 0.0.0.0 address exposes this server to every network interface, which may facilitate Denial of Service attacks","kind":"extension","name":"health_check","documentation":"https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/security-best-practices.md#safeguards-against-denial-of-service-attacks"}
{"level":"info","ts":1748853605.6897662,"caller":"extensions/extensions.go:52","msg":"Extension started.","kind":"extension","name":"health_check"}
{"level":"info","ts":1748853605.689775,"caller":"extensions/extensions.go:37","msg":"Extension is starting...","kind":"extension","name":"file_storage"}
{"level":"info","ts":1748853605.6897936,"caller":"extensions/extensions.go:52","msg":"Extension started.","kind":"extension","name":"file_storage"}
{"level":"info","ts":1748853606.2689695,"caller":"internal/persistent_queue.go:319","msg":"Fetching items left for dispatch by consumers","kind":"exporter","data_type":"logs","name":"syslog","numberOfItems":10}
{"level":"info","ts":1748853606.2750385,"caller":"internal/persistent_queue.go:361","msg":"Moved items for dispatching back to queue","kind":"exporter","data_type":"logs","name":"syslog","numberOfItems":10}
{"level":"info","ts":1748853606.275271,"caller":"adapter/receiver.go:45","msg":"Starting stanza receiver","kind":"receiver","name":"filelog/syslog","data_type":"logs"}
{"level":"warn","ts":1748853606.2754183,"caller":"fileconsumer/file.go:51","msg":"finding files: no files match the configured criteria","kind":"receiver","name":"filelog/syslog","data_type":"logs","component":"fileconsumer"}
{"level":"info","ts":1748853606.2754843,"caller":"healthcheck/handler.go:132","msg":"Health Check state change","kind":"extension","name":"health_check","status":"ready"}
{"level":"info","ts":1748853606.2755086,"caller":"service@v0.91.0/service.go:171","msg":"Everything is ready. Begin running and processing data."}
`

syslog server:

rfc5424:
`
May 23 11:01:53 test-logserver syslog-ng[38807]: Syslog connection accepted; fd='27', client='AF_INET(10.1.32.153:52078)', local='AF_INET(0.0.0.0:516)'
May 23 11:01:53 test-logserver syslog-ng[38807]: Invalid frame header; header=''
May 23 11:01:53 test-logserver syslog-ng[38807]: Syslog connection closed; fd='27', client='AF_INET(10.1.32.153:52078)', local='AF_INET(0.0.0.0:516)'
May 23 11:01:58 test-logserver syslog-ng[38807]: Syslog connection accepted; fd='27', client='AF_INET(10.1.32.153:36998)', local='AF_INET(0.0.0.0:516)'
May 23 11:01:58 test-logserver syslog-ng[38807]: Invalid frame header; header=''
May 23 11:01:58 test-logserver syslog-ng[38807]: Syslog connection closed; fd='27', client='AF_INET(10.1.32.153:36998)', local='AF_INET(0.0.0.0:516)'
May 23 11:01:59 test-logserver syslog-ng[38807]: Syslog connection accepted; fd='27', client='AF_INET(10.1.32.127:54304)', local='AF_INET(0.0.0.0:516)'
May 23 11:01:59 test-logserver syslog-ng[38807]: Invalid frame header; header=''
May 23 11:01:59 test-logserver syslog-ng[38807]: Syslog connection closed; fd='27', client='AF_INET(10.1.32.127:54304)', local='AF_INET(0.0.0.0:516)'
May 23 11:02:00 test-logserver syslog-ng[38807]: Syslog connection accepted; fd='27', client='AF_INET(10.1.32.100:43670)', local='AF_INET(0.0.0.0:516)'
May 23 11:02:00 test-logserver syslog-ng[38807]: Invalid frame header; header=''
May 23 11:02:00 test-logserver syslog-ng[38807]: Syslog connection closed; fd='27', client='AF_INET(10.1.32.100:43670)', local='AF_INET(0.0.0.0:516)'
May 23 11:02:01 test-logserver syslog-ng[38807]: Syslog connection accepted; fd='27', client='AF_INET(10.1.32.153:37004)', local='AF_INET(0.0.0.0:516)'
May 23 11:02:01 test-logserver syslog-ng[38807]: Invalid frame header; header=''
May 23 11:02:01 test-logserver syslog-ng[38807]: Syslog connection closed; fd='27', client='AF_INET(10.1.32.153:37004)', local='AF_INET(0.0.0.0:516)'
`

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/syslog: @kasia-kujawa @rnishtala-sumo @andrzej-stencel

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[andrzej-stencel]

The Syslog exporter has TLS enabled by default in TCP mode. Try setting tls::insecure: true to see if that helps.

exporters:
  syslog:
    tls:
      insecure: true
    ...

[ahujarajesh]

@andrzej-stencel Thank you for sharing this, the documentation talks otherwise, So I didn't set tls flag earlier.
After adding

exporters:
  syslog:
    tls:
      insecure: true
    ...

the issue of garbage folders is resolved but still I am not receiving any logs. Only timestamp and IP is visible in log folders.

Example:

root@test-logserver:/store/logs# tail -f 2025/05/29.log 
May 29 05:48:13 10.1.32.163 
May 29 05:48:13 10.1.32.163 
May 29 05:48:13 10.1.32.163 
May 29 05:48:13 10.1.32.163 
May 29 05:48:13 10.1.32.163 
May 29 05:48:13 10.1.32.163 
May 29 05:48:13 10.1.32.163 
May 29 05:48:13 10.1.32.163 
May 29 05:48:13 10.1.32.163 
May 29 05:48:13 10.1.32.163 

Here's the config on syslog server for your reference:

source s_net_syslog {
    network(
        port(516)           # The syslog port
        transport("tcp")    # Explicitly TCP
        max-connections(25000)
        log-iw-size(10000000)
        # Optional: consider adding flags(syslog-protocol) if issues persist,
        # this enforces stricter RFC adherence.
    );
};

destination d_local { file("/store/logs/$PROGRAM/$YEAR/$MONTH/$DAY.log"
                           perm(0644) 
                           dir_perm(0755) 
                           create_dirs(yes) 
                          );
                    };



log { source(s_net_syslog); destination(d_local); flags(flow_control); };

[ahujarajesh]

@andrzej-stencel
Attaching opentelemetry collector logs

otel-logs.log

[andrzej-stencel]

Looking at the logs you attached, the logs you're trying to parse as syslog look like this:

2025-06-03T06:36:07.669342529Z stderr F I0603 06:36:07.669211       1 server_linux.go:66] "Using iptables proxy"

This does not look like syslog at all. First, it is in container runtime format, so I'd start with adding the container operator. Still, after parsing out the container runtime's metadata, the log message is:

I0603 06:36:07.669211       1 server_linux.go:66] "Using iptables proxy"

which is not syslog format.