alert-advice for STACKIT AUDIT

alert-advice.md

I now have a thorough understanding of the STACKIT audit log data. Let me compile the comprehensive alert recommendations based on the actual data patterns observed.

Based on my thorough analysis of your STACKIT audit logs in logs-generic.otel-default from the last 24 hours (527 events, 10 users, 16 initiators, 14 distinct actions across 5 services), here are the alerts I recommend, organized by category:

---

🔐 1. Authentication & Identity Alerts

Alert	Rationale	Suggested Condition
Login without MFA	All 10 logins in the last 24h had mfa_performed: false. Alert if this is not expected.	stackit.action == "user.login-success" AND mfa_performed == "false"
Unusual Token Volume Spike	296 token issuances seen. Alert on sudden spikes above baseline.	Count of stackit.action == "user.token-success" per 15-min window exceeds 2× the rolling average
Login from New/Unexpected IP	20+ distinct IPs observed. Alert on logins from previously unseen IPs.	stackit.action == "user.login-success" from client.address not in a known allowlist
External IDP Logins	10 events used external_idp: true. Alert if external IDP usage is unexpected or should be monitored.	external_idp == "true" — any occurrence or above threshold
New User Login	Alert when a previously unseen email appears in login events.	stackit.action == "user.login-success" with email not seen in prior 30 days

---

🛡️ 2. Privileged / Destructive Operations Alerts

Alert	Rationale	Suggested Condition
Secret Deletion	14 secret.deleted events from a single service account. Alert on any secret deletion, especially in bulk.	stackit.action == "secret.deleted" — any occurrence or >5 in 15 min
Role Assumption Deactivation	1 event of assume-role.role-assumption.deactivate seen. This is a high-impact action.	stackit.action == "assume-role.role-assumption.deactivate" — any occurrence
Role Member Removal	1 event of assume-role.member.remove. Could indicate privilege escalation or account takeover.	stackit.action == "assume-role.member.remove" — any occurrence
ADMIN_ACTIVITY Events	45 events tagged legacy_eventType: ADMIN_ACTIVITY (secrets-manager operations). Alert on unusual volume.	Count of legacy_eventType == "ADMIN_ACTIVITY" exceeds threshold per hour

---

☸️ 3. Kubernetes (SKE) Infrastructure Alerts

Alert	Rationale	Suggested Condition
Cluster Create/Update	1 ske.clusters.create-or-update event seen. Infrastructure changes should be tracked.	stackit.action == "ske.clusters.create-or-update" — any occurrence
Cluster Hibernation/Wakeup	4 hibernation and 3 wakeup triggers observed. Alert if unexpected (cost or availability concern).	stackit.action IN ("ske.cluster-hibernation.trigger", "ske.cluster-wakeup.trigger")
Excessive Cluster GET Requests	133 ske.clusters.get calls, mostly from one initiator. Could indicate scraping or misconfigured automation.	Count of ske.clusters.get per initiator per hour exceeds threshold (e.g., >50)

---

🤖 4. Suspicious Automation & Access Pattern Alerts

Alert	Rationale	Suggested Condition
Non-Browser User Agent Activity	Significant programmatic access: curl/8.5.0 (131), Go-http-client/2.0 (46), python-requests (14), Apache-HttpClient (10). Alert on new or unexpected automation.	user_agent.original matches known automation patterns AND stackit.action involves destructive operations
New Service Account / Initiator	16 distinct initiators seen. Alert when a previously unknown stackit.initiator appears.	New stackit.initiator not seen in prior 7 days
Authorization Enforcement Spikes	17 authorization.v.enforce events (from aps-customer2org-service and stackit-api-gateway). Spikes could indicate permission probing.	Count of authorization.v.enforce exceeds 2× normal per 15-min window

---

📊 5. Volumetric / Anomaly Alerts

Alert	Rationale	Suggested Condition
Audit Log Volume Anomaly	Hourly volume ranged from 2 to 73 events. Alert on significant deviations from baseline.	Hourly event count deviates >3σ from 7-day hourly average
Off-Hours Activity	Very low activity between 17:00–04:00 UTC (only 30 events). Any significant activity during this window is notable.	Count of events between 17:00–04:00 UTC exceeds threshold (e.g., >20/hour)
Activity in New/Unexpected Project	4 projects active: data00, dev00, ci00, dev01. Alert when activity appears in a new project.	stackit.project not in known project list
Audit Log Ingestion Gap	No events seen for the 18:00 UTC hour. Alert if no audit logs arrive for >1 hour (possible collection failure).	Zero events in logs-generic.otel-default for a 1-hour window

---

🔑 6. Secrets Management Alerts

Alert	Rationale	Suggested Condition
Bulk Secret Creation	13 secret.created + 17 version.created events. Alert on creation bursts above normal.	Count of secret.created OR version.created > 10 in 15 min
Secret Operations by Unknown Initiator	Currently all secret ops come from smuser-sma22758sshnjr0h. Alert if a different initiator performs secret operations.	stackit.action LIKE "secret.*" AND stackit.initiator is not an approved service account

---

Implementation Priority

I'd recommend starting with these high-priority alerts first:

1. Secret Deletion — immediate impact, data loss risk
2. Role Member Removal / Role Deactivation — privilege escalation risk
3. Login without MFA — credential compromise risk
4. Cluster Create/Update — infrastructure change control
5. Audit Log Ingestion Gap — ensures monitoring continuity
6. Off-Hours Activity — catches after-hours intrusions

Would you like me to create any of these as Elastic alerting rules or workflows?