Monitoring & Logs

What you need to know about a running node

Five questions a validator operator should be able to answer at a glance, 24/7:

Am I synced to head?
Am I signing blocks when it's my slot?
Am I missing peers?
Is my disk filling up?
Is the process running?

This page covers the three places the node answers those questions: structured logs, Prometheus metrics, and the local status RPC.

Logs

The node writes JSON-Lines to stdout. Under the default systemd unit that lands in journald; tail it with:

journalctl -u valdium-node -f --output=cat | jq

Each line is a structured event. The most informative ones in steady state:

Event	When it fires	Watch for
`block.imported`	Every new block (~2s)	Gaps mean sync trouble
`block.proposed`	When you propose	Should fire on roughly your slot share
`peer.connected` / `peer.disconnected`	P2P churn	Net peer count below 4 is bad
`validator.signed`	You signed an attestation	Missing this = missing rewards
`validator.missed`	You missed a slot	One-off is normal; pattern is not
`storage.compaction`	Periodic DB compaction	Should complete within minutes

Log level defaults to info. Switch to debug by editing /etc/valdium/node.toml for short-lived investigations and switch back — debug volume on a busy chain is non-trivial.

Prometheus metrics

The node exposes Prometheus-compatible metrics on :9615/metrics by default. The endpoint binds to localhost; point Prometheus or any scraper at it from the same host, or proxy it.

curl -s localhost:9615/metrics | head -20

# HELP valdium_block_height current chain height
# TYPE valdium_block_height gauge
valdium_block_height 47821

# HELP valdium_peers_total number of connected peers
# TYPE valdium_peers_total gauge
valdium_peers_total 19

# HELP valdium_validator_slot_missed_total slots missed since boot
# TYPE valdium_validator_slot_missed_total counter
valdium_validator_slot_missed_total 0

Metrics worth alerting on

valdium_block_height stalled for more than 30 seconds → the node is not importing blocks.

valdium_peers_total < 4 → likely a network/firewall issue; you'll drift out of sync soon if it doesn't recover.

rate(valdium_validator_slot_missed_total[5m]) > 0.1 → missing more than one in ten of your slots; check log for validator.missed reasons.

valdium_chain_db_bytes growth pattern → run out of disk in a week? File an alert that fires three days before you would.

valdium_process_resident_memory_bytes climbing without bound → memory leak; restart and capture a heap profile.

Grafana dashboard

A community-maintained dashboard is available at github.com/Valdium/grafana-dashboard. Import the JSON into Grafana, point at your Prometheus, and you get pre-built panels for sync state, validator performance, peer count, db size, and resource use. It's the fastest path to "I can see what my node is doing" without writing panels by hand.

Local status RPC

For one-off questions, valdium-cli wraps the local admin RPC. These calls work even when the node is unreachable from outside:

valdium-cli node status
# height, peers, sync state, uptime

valdium-cli node peers
# list connected peers, ping, version

valdium-cli validator status
# bond, slot share, missed slots, rewards earned

valdium-cli storage stats
# db size, compaction status, fragmentation

Common operational patterns

I rebooted, now what? The systemd unit auto-starts. Within ~30 seconds you should see peer.connected events and block imports resume. Sync from a clean shutdown is fast (seconds); sync from a crash may take a few minutes while the DB recovers.

I changed the config and want to reload. systemctl restart valdium-node. The node closes the DB cleanly on SIGTERM, so this is safe but takes 5-10s.

I want to wipe and resync from scratch. systemctl stop valdium-node, rm -rf /var/lib/valdium/chain, systemctl start valdium-node. The keystore is preserved; only the block database is rebuilt.

My validator is jailed. Missing too many slots triggers a temporary jail. Read the latest validator.jailed log line — it has a reason and an unjail_height. Wait until that height, then run valdium-cli validator unjail.

Run a Node · Run a Validator → Token · Staking →