Zabbix Integration — LogTrace
Bidirectional synchronization between Zabbix and LogTrace. Forward logs from Zabbix for deep analysis and create Zabbix triggers driven by LogTrace events.
Setup Steps
Connect your Zabbix server to LogTrace in five steps. The integration uses Zabbix's built-in external scripts and LogTrace's REST API — no additional agents required.
1. Generate an API Key in LogTrace
Log in to your LogTrace workspace, navigate to Settings → API Access, and create a new key with "write:logs" and "read:events" scopes. Copy the key — it will look like ltk_live_8f3a9c2d…. Store it securely; it cannot be viewed again.
2. Configure the External Script in Zabbix
Place the provided logtrace_sender.sh script into your Zabbix server's /usr/lib/zabbix/externalscripts/ directory. Make it executable with chmod +x. The script accepts a log message, a hostname, and an optional severity tag as arguments.
3. Define a New Item Type
In the Zabbix web UI, create an item of type "External check" on the target host. Set the key to logtrace.sender[{HOST.HOST}, {ITEM.VALUE}, warning]. Configure the update interval to match your monitoring cadence — typically 30 to 60 seconds for production hosts.
4. Map Log Sources to LogTrace Projects
Each Zabbix host maps to a LogTrace project. Use the --project flag in the external script to route logs from web-prod-01.moscow.corp to the "Web Production" project and from db-replica-03 to "Database Cluster." This keeps your log streams organized from day one.
5. Verify the Connection
Run a manual test from the Zabbix server shell: /usr/lib/zabbix/externalscripts/logtrace_sender.sh "test message" "localhost" info. Within 10 seconds, the message should appear in your LogTrace dashboard under the Live Feed. Check the HTTP status code in the script's output — a 200 response confirms successful delivery.
Optional: Enable Bidirectional Sync
For full two-way integration, configure the LogTrace webhook endpoint in Zabbix's Administration → Media Types. Point the webhook URL to https://api.logtrace.ru/v1/webhooks/zabbix/YOUR_WEBHOOK_ID. LogTrace will then push event summaries back into Zabbix as problem acknowledgments and custom macros.
Zabbix Triggers from LogTrace Events
LogTrace analyzes incoming logs with pattern matching and anomaly detection, then surfaces critical findings as structured events. You can turn those events into Zabbix triggers so your existing alerting workflows — email, Telegram, Slack — continue to work without change.
Failed Authentication Spike
When LogTrace detects more than 15 failed SSH login attempts within a 5-minute window on any host tagged "production," it emits an event of type auth.brute_force. The corresponding Zabbix trigger expression is logtrace.event("auth.brute_force", {HOST.HOST}) > 0. This trigger fires a PROBLEM state that routes to your on-call Telegram group via the existing Zabbix media type.
Application Error Rate Threshold
If the percentage of HTTP 5xx responses in Nginx access logs exceeds 5% over a rolling 10-minute period, LogTrace generates an app.error_rate_high event. Map this to a Zabbix trigger with severity "High" so the incident appears in your Zabbix dashboard alongside infrastructure alerts. The trigger expression: logtrace.metric("http.5xx.rate", {HOST.HOST}, "10m") > 0.05.
Database Slow Query Detection
LogTrace parses PostgreSQL slow query logs and identifies queries exceeding 2 seconds in execution time. When 3 or more slow queries are detected within 1 minute, an db.slow_query_burst event is created. Use this to trigger a Zabbix alert that notifies the DBA team via the admin-db@company.ru email media type, including the top offending query in the problem message.
Disk I/O Warning from dmesg
System logs containing "I/O error" or "ata_timeout" patterns are automatically classified by LogTrace as sys.disk_io_error events. A Zabbix trigger based on this event type can escalate to "Average" severity after the first occurrence and to "High" if a second event arrives within 60 minutes, enabling a graduated response strategy for disk failures.
All trigger examples above use LogTrace's native Zabbix integration macros. No custom scripting is required on the Zabbix side beyond the initial webhook configuration. Event payloads are delivered as JSON with fields for timestamp, hostname, severity, event type, and a free-text summary that Zabbix inserts directly into problem descriptions.