- \n
- deterministic and verifiable logging<\/li>\n\n\n\n
- clear separation between execution<\/em> And check<\/em><\/li>\n\n\n\n
- principle of fail-closed<\/em> (Better one more alert than a lost backup)<\/li>\n\n\n\n
- external and immutable audit trail<\/li>\n<\/ul>\n\n\n\n
The result is an architecture that revolves around Ubuntu Server<\/strong>, redundant backups (Oracle Cloud + Hetzner), Google Cloud for logging<\/strong> and a workflow n8n<\/strong> which validates every day that everything went well.<\/p>\n\n\n\n
This article explains the architecture<\/strong>, the technical choices<\/strong> and the Why<\/strong>. The n8n workflow is a paid product: I describe it transparently, but without releasing sensitive or one-to-one replicable information.<\/p>\n\n\n\n
\n\n\n\nThe real problem (first)<\/h2>\n\n\n\n
Before this architecture, the flow was the classic:<\/p>\n\n\n\n
- \n
- cron job \u2192 rsync \/ rclone<\/li>\n\n\n\n
- local logs<\/li>\n\n\n\n
- manual control \u201cwhen it happens\u201d<\/li>\n\n\n\n
- alert only in case of obvious errors<\/li>\n<\/ul>\n\n\n\n
This approach has three structural flaws:<\/p>\n\n\n\n
- \n
- Silence \u2260 success<\/strong>
If a job doesn't get off the ground or dies halfway, you often don't find out right away.<\/li>\n\n\n\n- Local logs = single point of failure<\/strong>
If you lose the VM, you also lose proof that the backup was made. era<\/em> match.<\/li>\n\n\n\n- Wasted human time<\/strong>
Reading raw logs is expensive and error-prone.<\/li>\n<\/ol>\n\n\n\nHence the decision to consciously over-engineer<\/em>.<\/p>\n\n\n\n
\n\n\n\nProject objectives<\/h2>\n\n\n\n
The goals were not to \u201cdo cool things\u201d, but to be operationally calm<\/strong>:<\/p>\n\n\n\n
- \n
- know everyday<\/strong> if all jobs have run<\/li>\n\n\n\n
- know which job<\/strong> he failed and Why<\/strong><\/li>\n\n\n\n
- have logs immutable<\/strong> off the server<\/li>\n\n\n\n
- be able to demonstrate after the fact what happened (audit)<\/li>\n\n\n\n
- zero reverse dependencies: control must never touch the server<\/li>\n<\/ul>\n\n\n\n
![\"\"](\"https:\/\/paoloronco.it\/wp-content\/uploads\/2026\/01\/image-1024x597.png\")
<\/figure>\n\n\n\n
\n\n\n\nHigh-level architecture<\/h2>\n\n\n\n
Ubuntu Server (on\u2011prem \/ Proxmox) \u251c\u2500 cron \u251c\u2500 rsync \/ rclone \u251c\u2500 structured logs \u2502 \u251c\u2500 Google Cloud Logging (streaming) \u2514\u2500 Google Cloud Storage (snapshot log) \u2502 \u25bc n8n (monitoring & validation) \u2502 \u251c\u2500 check log presence \u251c\u2500 deterministic parsing \u251c\u2500 content analysis \u2514\u2500 alert \/ ticket\n<\/code><\/pre>\n\n\n\nKey separation:<\/p>\n\n\n\n
- \n
- Ubuntu Server runs<\/strong><\/li>\n\n\n\n
- n8n observes<\/strong><\/li>\n<\/ul>\n\n\n\n
No SSH, no API to the server. Zero trust in the data producer.<\/p>\n\n\n\n
\n\n\n\nUbuntu Server as a foundation<\/h2>\n\n\n\n
Ubuntu Server is the starting point:<\/p>\n\n\n\n
- \n
- VM on Proxmox<\/li>\n\n\n\n
- local storage + dedicated disks<\/li>\n\n\n\n
- versioned scripts (Git)<\/li>\n\n\n\n
- cron simple<\/em>, but observable<\/li>\n<\/ul>\n\n\n\n
Structured logs, not \u201crandom text\u201d<\/h3>\n\n\n\n
Each backup job produces logs with:<\/p>\n\n\n\n
- \n
- marker of
START<\/code>,RSYNC_END<\/code>,SUMMARY<\/code>,END<\/code><\/li>\n\n\n\nrun_id<\/code> unique<\/li>\n\n\n\nrc<\/code>, duration, warnings, errors<\/li>\n<\/ul>\n\n\n\nThis allows:<\/p>\n\n\n\n
- \n
- deterministic parsing<\/li>\n\n\n\n
- detect incomplete runs<\/li>\n\n\n\n
- distinguish warning from failure<\/li>\n<\/ul>\n\n\n\n
It's not logging "for humans", it's logging for the machines<\/strong>.<\/p>\n\n\n\n
\n\n\n\nRedundant backups \u2260 Business Continuity<\/h2>\n\n\n\n
Making multiple copies is not enough.<\/p>\n\n\n\n
There Business Continuity<\/strong> requires:<\/p>\n\n\n\n
- \n
- know that all<\/em> copies have been updated<\/li>\n\n\n\n
- know When<\/em> a copy is not<\/li>\n\n\n\n
- react proportionately<\/li>\n<\/ul>\n\n\n\n
Because of this:<\/p>\n\n\n\n
- \n
- Oracle Cloud and Hetzner are just target<\/em><\/li>\n\n\n\n
- the truth is in the logs<\/li>\n\n\n\n
- continuity is in the automatic verification<\/li>\n<\/ul>\n\n\n\n
\n\n\n\nExternal logging: Google Cloud<\/h2>\n\n\n\n
Logging is divided into two complementary channels.<\/p>\n\n\n\n
1. Google Cloud Logging (streaming)<\/h3>\n\n\n\n
- \n
- near-real-time ingestion of rsync logs<\/li>\n\n\n\n
- quick queries<\/li>\n\n\n\n
- immediate troubleshooting<\/li>\n\n\n\n
- very low costs (often zero)<\/li>\n<\/ul>\n\n\n\n
It is used for:<\/p>\n\n\n\n
- \n
- analyses<\/li>\n\n\n\n
- debug<\/li>\n\n\n\n
- operational visibility<\/li>\n<\/ul>\n\n\n\n
2. Google Cloud Storage (immutable snapshots)<\/h3>\n\n\n\n
- \n
- daily log upload<\/li>\n\n\n\n
- deterministic path
YYYY\/MM\/DD<\/code><\/li>\n\n\n\n- versioning and retention<\/li>\n\n\n\n
- service account with minimal permissions<\/li>\n<\/ul>\n\n\n\n
It is used for:<\/p>\n\n\n\n
- \n
- audit<\/li>\n\n\n\n
- independent verification<\/li>\n\n\n\n
- n8n workflow input<\/li>\n<\/ul>\n\n\n\n
This bucket is the source of truth<\/strong> for monitoring.<\/p>\n\n\n\n
\n\n\n\nThe role of n8n (monitoring, not backup)<\/h2>\n\n\n\n
The n8n workflow does not perform backups<\/strong>.<\/p>\n\n\n\n
It only does three things, in order:<\/p>\n\n\n\n
- \n
- Check attendance<\/strong>
Do all the logs expected for today exist?<\/li>\n\n\n\n- Check integrity<\/strong>
Does each log have a consistent START\/END\/SUMMARY?<\/li>\n\n\n\n- Semantic verification<\/strong>
Errors? Warnings? Abnormal patterns?<\/li>\n<\/ol>\n\n\n\nIf something is missing or not correct \u2192 alert.<\/p>\n\n\n\n
Why n8n?<\/h3>\n\n\n\n
- \n
- readable workflows<\/li>\n\n\n\n
- versionable<\/li>\n\n\n\n
- extendable<\/li>\n\n\n\n
- decoupled from the server<\/li>\n<\/ul>\n\n\n\n
The value is not \u201cautomation\u201d, but the validation<\/strong>.<\/p>\n\n\n\n
\n\n\n\nWhere to buy it<\/h3>\n\n\n\n
- \n
- Official shop<\/strong>: https:\/\/shop.paoloronco.it<\/a><\/li>\n\n\n\n
- Gumroad<\/strong>: https:\/\/paoloronco.gumroad.com<\/a><\/li>\n\n\n\n
- n8n Creators Hub \/ Templates<\/strong>: coming soon<\/li>\n<\/ul>\n\n\n\n
Because it's a paid workflow<\/h2>\n\n\n\n
This workflow:<\/p>\n\n\n\n
- \n
- encapsulates months of real-world debugging<\/li>\n\n\n\n
- handles edge cases (incomplete logs, duplicates, partial runs)<\/li>\n\n\n\n
- it is meant to be extended<\/strong>, not only used<\/li>\n<\/ul>\n\n\n\n
He doesn't sell magic.
Sells time saved<\/strong> And mistakes avoided<\/strong>.<\/p>\n\n\n\nWhere to buy it<\/h3>\n\n\n\n
- \n
- shop.paoloronco.it<\/li>\n\n\n\n
- Gumroad<\/li>\n\n\n\n
- n8n Creators Hub (template)<\/li>\n<\/ul>\n\n\n\n
The sensitive code remains mine.
The architecture and principles are shared.<\/p>\n\n\n\n
\n\n\n\nSafety and key principles<\/h2>\n\n\n\n
- \n
- Single-purpose Service Account<\/li>\n\n\n\n
- minimum permits<\/li>\n\n\n\n
- no credentials in the workflow<\/li>\n\n\n\n
- no direct access to the server<\/li>\n<\/ul>\n\n\n\n
If n8n were to be compromised:<\/p>\n\n\n\n
- \n
- can only read public logs for him<\/em><\/li>\n\n\n\n
- cannot touch backups or servers<\/li>\n<\/ul>\n\n\n\n
This is zero trust applied to a homelab<\/strong>.<\/p>\n\n\n\n
\n\n\n\nWhat did this architecture take away from me?<\/h2>\n\n\n\n
- \n
- daily manual check<\/li>\n\n\n\n
- anxiety about \u201chas he left?\u201d<\/li>\n\n\n\n
- night debugging<\/li>\n\n\n\n
- useless logs<\/li>\n<\/ul>\n\n\n\n
In exchange I got:<\/p>\n\n\n\n
- \n
- measurable trust<\/li>\n\n\n\n
- significant alerts<\/li>\n\n\n\n
- audit trail<\/li>\n\n\n\n
- operational serenity<\/li>\n<\/ul>\n\n\n\n
\n\n\n\nConclusion<\/h2>\n\n\n\n
This project was not born to demonstrate that it can be done<\/em>.<\/p>\n\n\n\n
It was born because I didn't want to waste time doubting backups anymore<\/strong>.<\/p>\n\n\n\n
Even in a homelab, data matters.
And when data matters, Logging is part of backup<\/strong>, not a detail.<\/p>\n\n\n\nOver-engineering? Perhaps.
But it is over-engineering that sleeps peacefully.<\/p>\n\n\n\n<\/p>","protected":false},"excerpt":{"rendered":"
Questo articolo nasce da un problema molto concreto: sapere davvero se i backup funzionano, senza dover controllare manualmente log sparsi, script cron opachi e notifiche poco affidabili. Nel mio homelab (che resta pur sempre una casa, non un data center enterprise) ho deciso di applicare principi da Business Continuity e Disaster Recovery tipici di ambienti … Continue reading “Over\u2011Engineering un Homelab: Logging, Backup e Business Continuity con Ubuntu Server”<\/span><\/a><\/p>","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1833","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/paoloronco.it\/en\/wp-json\/wp\/v2\/posts\/1833","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/paoloronco.it\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/paoloronco.it\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/paoloronco.it\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/paoloronco.it\/en\/wp-json\/wp\/v2\/comments?post=1833"}],"version-history":[{"count":2,"href":"https:\/\/paoloronco.it\/en\/wp-json\/wp\/v2\/posts\/1833\/revisions"}],"predecessor-version":[{"id":1836,"href":"https:\/\/paoloronco.it\/en\/wp-json\/wp\/v2\/posts\/1833\/revisions\/1836"}],"wp:attachment":[{"href":"https:\/\/paoloronco.it\/en\/wp-json\/wp\/v2\/media?parent=1833"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/paoloronco.it\/en\/wp-json\/wp\/v2\/categories?post=1833"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/paoloronco.it\/en\/wp-json\/wp\/v2\/tags?post=1833"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
- cannot touch backups or servers<\/li>\n<\/ul>\n\n\n\n
- can only read public logs for him<\/em><\/li>\n\n\n\n
- Gumroad<\/strong>: https:\/\/paoloronco.gumroad.com<\/a><\/li>\n\n\n\n
- Official shop<\/strong>: https:\/\/shop.paoloronco.it<\/a><\/li>\n\n\n\n
- Check integrity<\/strong>
- Check attendance<\/strong>
- know When<\/em> a copy is not<\/li>\n\n\n\n
- know that all<\/em> copies have been updated<\/li>\n\n\n\n
- marker of
- n8n observes<\/strong><\/li>\n<\/ul>\n\n\n\n
- know which job<\/strong> he failed and Why<\/strong><\/li>\n\n\n\n
- Local logs = single point of failure<\/strong>
- Silence \u2260 success<\/strong>
- principle of fail-closed<\/em> (Better one more alert than a lost backup)<\/li>\n\n\n\n