We just shipped OpsAI Sentinel, and we want to be upfront about what it is before anything else: it's a read-only watcher by default. A human stays in control. Nothing changes on your server unless you've explicitly raised the permission level or clicked an approval button yourself. The agent that investigates runs on your own AI account, not ours, and no incident data is sent to Panelica — the whole loop stays between your panel and your own AI provider. This is a beta feature and we'll say exactly where the rough edges are near the bottom of this post.
With that said — here's what it actually does.
An AI agent that watches your server continuously would mean continuous token spend for basically nothing, most days. Sentinel doesn't work that way. The watching itself — service state, logs, resource thresholds, small periodic probes — is done by the panel's own Go backend, at essentially zero cost. The AI agent only wakes up when a real trigger fires, does one investigation pass (typically 20-60 seconds), writes a report, and goes back to sleep. Idle days cost nothing.
Sentinel doesn't run its own model. It hands the incident to whichever agent you've already installed and signed into from the OpsAI screen:
- Claude Code — your own subscription or API key
- Codex CLI (OpenAI) — your own account or API key
- Gemini CLI (Google) — your own account or API key
- Cursor Agent — your own account, works on the free plan too
You pick the agent, you sign in, the cost goes to your own account — Panelica doesn't sit in the middle of that. You can even set a different agent per trigger if you want (say, Claude Code for database incidents, something else for the rest).
Every trigger can be turned on or off individually, and for each one you can tune how many hits it needs, the time window, the cooldown between repeats, the severity, and which expert prompt investigates it. Three presets are built in — Silent (outages only), Balanced (recommended), Everything — and touching a single trigger switches you to a Custom profile without resetting anything else you already had set.
Services
- Service stopped unexpectedly (nginx, Apache, MySQL, PostgreSQL, Redis, PHP-FPM, Postfix, Dovecot, BIND, ProFTPD...)
- Docker container stopped
- Failed systemd units — including ones the panel's own service list doesn't track, like lxcfs, fail2ban, clamav
Web servers
- nginx (sites) fatal error
- nginx (sites) error wave — a short burst of upstream timeouts, 502s, 504s
- nginx (panel) fatal error
- Apache fatal error
- Hosted site not answering — an HTTP probe against every active domain
PHP
- PHP-FPM pool exhausted or failing (max_children hit, 502s)
- PHP fatal errors repeating on a site
Database
- MySQL error / crash (signal received, too many connections, InnoDB corruption)
- PostgreSQL PANIC / FATAL
Panel
- Panel backend panic
- Panel backend error wave
- External API error wave
- Cron job failures
- Panel not answering — probes its own HTTPS port
- Security update pending — this one only ever informs you, it never applies the update itself
- Outbound mail deferred wave (Postfix)
- IMAP/POP login failure flood (Dovecot)
- Mail queue growing
DNS
- DNS zone failed to load (BIND)
- Server hostname does not resolve
Resources
- CPU warning / CPU critical
- Memory warning / Memory critical
- Disk warning / Disk critical
- Kernel OOM killer fired
- Inodes nearly exhausted
- Sustained high load
Backup, migration, SSL
- Backup failed
- No successful backup recently
- Site migration failed
- Certificate expiring / renewal failed — checks DNS and HTTP-01 reachability before reporting
Security — inform-only, never acts
- Malware detected (ClamAV)
- Brute-force wave (many IP blocks)
- Panel login failure wave
- fail2ban ban wave
Security triggers are capped at inform-only in the catalogue itself — not a setting you can raise. An attacker could otherwise use them to provoke actions.
| Level | The agent may | The panel does |
|---|---|---|
| Inform (default, recommended start) | Read-only diagnosis — inspects with read-only commands and the panel API, changes nothing | Records the report. If an allow-listed action would help, it's requested — you approve with one click |
| Safe actions | Diagnose, and request allow-listed actions only | Executes them itself and verifies the result (service active, site answers) |
| Full | Acts on its own, with full server access | Just records what it did |
The allow-list itself is short and specific:
Code:
restart_service
Code:
reload_service
Code:
clean_temp
Code:
vacuum_journalIf a report proposes a step that isn't on the allow-list — say, cleaning up leftover config from something removed a while back — the incident view shows "Approve and let the agent do it". Click it (there's a warning dialog first) and the agent runs one more time with full autonomy for exactly that approved step, verifies it, and reports back. The original incident keeps a record of who approved it and when.
- Watch: the panel backend follows service state, its own events (backups, SSL, malware, blocked IPs), resource thresholds, and service/site logs, plus a small probe run every 5 minutes (panel reachable, failed service units, mail queue age, backup age, inodes, load, hostname DNS, pending security updates).
- Trigger: a hit has to clear its threshold, time window and cooldown before it's queued — so one noisy log line, or the same condition firing repeatedly, doesn't wake the agent over and over. Quiet hours and a daily run cap protect the budget on top of that.
- Investigate: the agent gets the evidence — matched log lines, the incident data, a server snapshot, and its own memory of earlier incidents — and looks around using read-only commands and the panel API.
- Report: it answers in a fixed shape every time — summary, root cause, severity, a plain-language message in your panel's language, what it did, what it's asking for, whether a human needs to step in.
- Notify: the report goes to the notification bell and to every webhook subscribed to Sentinel events, and shows up in the Activity feed with the full evidence attached.
Sentinel remembers: the last incidents get summarized into the next run's prompt, so a report can say "this has happened 4 times before, here's what it turned out to be" instead of starting from zero every time. There's also an optional daily briefing — overall health, anything worth a second look, and one protective suggestion.
These are illustrative — built to show the shape of a report, not a real customer incident.
No outage happened, and no restart was taken — the agent verified the site was actually reachable before closing the incident instead of reflexively restarting a service that had already recovered on its own.
That's the case where "Approve and let the agent do it" shows up — you click it, the agent does exactly that one cleanup, verifies, and reports back.
Reports land on the panel's own notification bell, and there's a dedicated Telegram tab in Sentinel's settings — bot token, chat ID, save, send a test message, done. Slack, Discord and your own HTTP endpoints work too, through the Webhooks page, subscribed to the
Code:
ai.sentinel.*- The agent never polls. It only runs on a real trigger, or the optional daily briefing.
- A daily run cap: once reached, incidents are still recorded, just not investigated until the next day.
- Quiet hours defer anything that isn't critical.
- A cooldown per trigger and target stops the same condition from re-running the agent over and over — that survives a panel restart or update too, it doesn't reset to zero.
- The agent account is yours. Panelica doesn't supply an AI key and doesn't receive any of the incident data.
This is genuinely experimental, and the panel says so out loud: turning Sentinel on, and choosing Full autonomy, both show a warning dialog, and the duty card carries a permanent "Experimental" label. The agent runs on your live server, so start in Inform mode, read a few days of reports, make sure your backups are current, and only raise the level once you trust what you're reading. A few things we want to be specific about, because we'd rather under-promise here:
- The panel backend itself is never on the restart/reload list, in any mode.
- Package updates are never applied automatically by Sentinel — the "security update pending" trigger only tells you one is waiting.
- Security triggers (malware, brute force, login waves, fail2ban) are locked to inform-only — they can never cause an action, full stop.
- Cloudflare isn't a direct trigger source. Sentinel doesn't watch Cloudflare's own logs or events. If your agent's expert set includes the Cloudflare expert, it can still reason about DNS/proxy causes when investigating a triggered incident (and the SSL-renewal trigger already checks DNS and HTTP-01 reachability on its own) — but nothing in Cloudflare itself wakes Sentinel up.
Sentinel lives in the bar at the bottom of the dashboard ("On duty · your server is being watched"). Click it to expand, full-screen it if you want, Esc to collapse. Tabs: Overview, Settings, What to Watch, Telegram, How It Works, Activity. If you haven't installed an AI agent yet, the bar tells you and points you at the OpsAI screen to install and sign in to one. Sentinel is ROOT-only and sits behind the same
Code:
ops_aiDo I need a license for this? Yes — it's on plans with the OpsAI feature. Without it the tab shows locked.
What if I haven't installed an agent? The bar tells you and links to the OpsAI install screen — same install flow as OpsAI itself.
What if the agent's session drops? Sentinel sends an "I can't reach the server right now" notification and keeps recording incidents; nothing is lost, it just can't investigate until the agent's signed back in.
Does Sentinel see Cloudflare directly? No — see the beta section above.
Will it change my nginx config or cache rules on its own? Only in Full mode, and only with the least invasive step first (reload over restart), and only after verifying and reporting what it changed. In Safe mode it can only request what's on the allow-list; anything else is a suggestion you approve yourself.
Does any of this data go to Panelica? No. The agent runs on your own AI account, and Panelica doesn't receive incident data or supply an AI key.
This is a beta, and the trigger catalogue is exactly the kind of thing that gets better with real reports of what's missing or noisy. If something misfires, doesn't trigger when you expected it to, or you think of a condition that should be its own trigger, tell us:
- Bugs: Bug Reports
- Ideas for new triggers or changes: Feature Requests
If you're running OpsAI already and haven't seen the rest of what it can do, there's a full writeup here: OpsAI: running Claude Code, Codex CLI, Gemini CLI and Cursor Agent inside the panel — and if you'd rather talk to it from your phone than open the dashboard, there's also OpsAI on Telegram.