Selfhosted

58486 readers

526 users here now

A place to share alternatives to popular online services that can be self-hosted without giving up privacy or locking you into a service you don't control.

Rules:

Be civil: we're here to support and learn from one another. Insults won't be tolerated. Flame wars are frowned upon.
No spam posting.
Posts have to be centered around self-hosting. There are other communities for discussing hardware or home computing. If it's not obvious why your post topic revolves around selfhosting, please include details to make it clear.
Don't duplicate the full text of your blog or github here. Just post the link for folks to click.
Submission headline should match the article title (don’t cherry-pick information from the title to fit your agenda).
No trolling.
No low-effort posts. This is subjective and will largely be determined by the community member reports.

Resources:

selfh.st Newsletter and index of selfhosted software and apps
awesome-selfhosted software
awesome-sysadmin resources
Self-Hosted Podcast from Jupiter Broadcasting

Any issues on the community? Report it using the report flag.

Questions? DM the mods!

founded 2 years ago

MODERATORS

HybridSarcasm@lemmy.world

HybridSarcasm@lemmy.hybridsarcasm.xyz

Suggestions for log aggregation for proxmox containers (lemmy.dbzer0.com)

submitted 1 month ago by disobey2623@lemmy.dbzer0.com to c/selfhosted@lemmy.world

4 comments fedilink hide all child comments

Heya.

I'm still pretty new to the homelab scene, so the more detail you can add the better. I'd like to add some sort of log aggregation tool, something like Elastic, where I can go to look at logs from any of my systems that aren't working, or just make sure I don't miss any errors.

Pretty much everything I run is set up as a Proxmox LXC from Proxmox helper scripts, which most of the time means it's running as a systemctl service. Sometimes they run in Alpine instead, and a few of my apps also run in Docker.

What's a good app to aggregate logs from those sources? I've heard of Prometheus, Grafana and Loki but not sure if they do what I'm after, they seem pretty overwhelming and more focused on metrics, whereas I want to be able to search for and view logs. I'd appreciate if you also mention the basic steps to send the logs from each container to said app.

top 4 comments

sorted by: hot top controversial new old

[–] redlemace@lemmy.world 8 points 1 month ago* (last edited 1 month ago) (1 children)

I use syslog on everything that can send it. All forward to a centralized rsyslog which stores directly into postgres. Grafana nativly can read from that. Super lightweight, super flexible. (I use 1 cpu and 2G mem and it works better than graylog which needed 4 cpu's and 16 G ram)

[–] slazer2au@lemmy.world 2 points 1 month ago (1 children)

Do you have pointers on how to get that up an going?

[–] redlemace@lemmy.world 7 points 1 month ago* (last edited 1 month ago)

this is the main pointer

On every device (but the central syslogserver or you create a loop that fills the drive in mere seconds)

/etc/rsyslog.d/99-centralsyslog.conf

$PreserveFQDN on
*.*  @192.168.1.66

then on the central syslog server 192.168.1.66

/etc/rsyslog.d/01-syslog_receiver.conf

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

and also

/etc/rsyslog.d/20-save2postgresql.conf

# Load the PostgreSQL output module
module(load="ompgsql")

# Template for inserting logs
template(name="pgsql-template" option.sql="on" type="string" string="INSERT INTO system_events (hostname, facility, priority, tag, message) VALUES ('%HOSTNAME%', %syslogfacility%, %syslogpriority%, '%syslogtag%', '%msg%' )") 

# Send logs to PostgreSQL
*.emerg    :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
*.panic    :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
*.alert    :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
*.crit     :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
*.error    :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
*.err      :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
#*.warning  :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
#*.warn     :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
#*.notice  :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
#*.info  :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
#*.debug  :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template

Make sure you install postgres, the rsyslog-psql module and create the database and tables.

Grafana can run on the same or any other server.

[–] mangaskahn@lemmy.world 2 points 1 month ago

Greylog is a syslog aggregator that might do what you're looking for.