Skip to content

Instantly share code, notes, and snippets.

@alsmola

alsmola/break-glass-session.md

Created March 24, 2026 03:55
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save alsmola/91bd796e3ade4f33884ba9e9b5079b3c to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save alsmola/91bd796e3ade4f33884ba9e9b5079b3c to your computer and use it in GitHub Desktop.
Download ZIP
Raw
break-glass-session.md

Verify AWS credentials and trailtool readiness

$ trailtool status
AWS credentials: OK (account XXXXXXXXXXXX)
Ingestor stack: OK (trailtool-ingestor)
Data access: OK

Good, trailtool is ready. Let me find the break-glass session for alex@engseclabs.com.

Find break-glass sessions

$ trailtool sessions list --user alex@engseclabs.com --role BreakGlass --format json
[
  {
    "session_id": "alex@engseclabs.com:AROAUB266OVZGAWC7JRGO",
    "session_type": "cli-sdk",
    "session_start": "2026-03-24T00:00:00Z#alex@engseclabs.com:AROAUB266OVZGAWC7JRGO",
    "start_time": "2026-03-24T00:00:00Z",
    "end_time": "2026-03-24T03:24:59Z",
    "duration_minutes": 204,
    "person_email": "alex@engseclabs.com",
    "account_id": "XXXXXXXXXXXX",
    "role_arn": "arn:aws:iam::XXXXXXXXXXXX:role/aws-reserved/sso.amazonaws.com/us-east-2/AWSReservedSSO_BreakGlassEmergency_aee6c9b96f64c030",
    "role_name": "aws-reserved/sso.amazonaws.com/us-east-2/AWSReservedSSO_BreakGlassEmergency_aee6c9b96f64c030",
    "events_count": 18,
    "services_count": 8,
    "resources_count": 0,
    "source_ips": [
      "23.93.168.176"
    ],
    "event_counts": {
      "bedrock.amazonaws.com:InvokeModel": 2,
      "ec2.amazonaws.com:DescribeInstances": 1,
      "ec2.amazonaws.com:DescribeNetworkInterfaces": 1,
      "ec2.amazonaws.com:DescribeSecurityGroups": 1,
      "ec2.amazonaws.com:DescribeVpcs": 1,
      "iam.amazonaws.com:CreateAccessKey": 1,
      "iam.amazonaws.com:ListAccessKeys": 1,
      "iam.amazonaws.com:ListPolicies": 1,
      "iam.amazonaws.com:ListRoles": 1,
      "iam.amazonaws.com:ListUsers": 1,
      "logs.amazonaws.com:DescribeLogGroups": 1,
      "monitoring.amazonaws.com:DescribeAlarms": 1,
      "s3.amazonaws.com:ListBuckets": 1,
      "ssm.amazonaws.com:DescribeInstanceInformation": 1,
      "ssm.amazonaws.com:DescribeParameters": 1,
      "ssm.amazonaws.com:DescribeSessions": 1,
      "sts.amazonaws.com:GetCallerIdentity": 1
    },
    "resources_accessed": {}
  }
]

Found 1 break-glass session: 18 events across 8 services over ~3.4 hours. Let me get the full detail and an AI summary.

Get full session detail

$ trailtool sessions detail --session-key "2026-03-24T00:00:00Z#alex@engseclabs.com:AROAUB266OVZGAWC7JRGO" --format json
{
  "session_id": "alex@engseclabs.com:AROAUB266OVZGAWC7JRGO",
  "session_type": "cli-sdk",
  "session_start": "2026-03-24T00:00:00Z#alex@engseclabs.com:AROAUB266OVZGAWC7JRGO",
  "start_time": "2026-03-24T00:00:00Z",
  "end_time": "2026-03-24T03:24:59Z",
  "duration_minutes": 204,
  "person_email": "alex@engseclabs.com",
  "account_id": "XXXXXXXXXXXX",
  "role_arn": "arn:aws:iam::XXXXXXXXXXXX:role/aws-reserved/sso.amazonaws.com/us-east-2/AWSReservedSSO_BreakGlassEmergency_aee6c9b96f64c030",
  "role_name": "aws-reserved/sso.amazonaws.com/us-east-2/AWSReservedSSO_BreakGlassEmergency_aee6c9b96f64c030",
  "events_count": 18,
  "services_count": 8,
  "resources_count": 0,
  "source_ips": [
    "23.93.168.176"
  ],
  "event_counts": {
    "bedrock.amazonaws.com:InvokeModel": 2,
    "ec2.amazonaws.com:DescribeInstances": 1,
    "ec2.amazonaws.com:DescribeNetworkInterfaces": 1,
    "ec2.amazonaws.com:DescribeSecurityGroups": 1,
    "ec2.amazonaws.com:DescribeVpcs": 1,
    "iam.amazonaws.com:CreateAccessKey": 1,
    "iam.amazonaws.com:ListAccessKeys": 1,
    "iam.amazonaws.com:ListPolicies": 1,
    "iam.amazonaws.com:ListRoles": 1,
    "iam.amazonaws.com:ListUsers": 1,
    "logs.amazonaws.com:DescribeLogGroups": 1,
    "monitoring.amazonaws.com:DescribeAlarms": 1,
    "s3.amazonaws.com:ListBuckets": 1,
    "ssm.amazonaws.com:DescribeInstanceInformation": 1,
    "ssm.amazonaws.com:DescribeParameters": 1,
    "ssm.amazonaws.com:DescribeSessions": 1,
    "sts.amazonaws.com:GetCallerIdentity": 1
  },
  "resources_accessed": {}
}

Get AI-generated session summary

$ trailtool sessions summarize --session-key "2026-03-24T00:00:00Z#alex@engseclabs.com:AROAUB266OVZGAWC7JRGO"
- Created an IAM access key
- Reviewed IAM users, roles, policies, and access keys
- Reviewed EC2 instances, network interfaces, security groups, and VPCs
- Reviewed Systems Manager sessions, instance information, and parameters
- Reviewed CloudWatch alarms
- Reviewed CloudTrail log groups
- Reviewed S3 buckets
- Invoked Bedrock model API twice
- Retrieved caller identity using STS

Now comparing the session activity against the stated justification.

Compare session to justification

Justification: "I was investigating an incident and needed to access SSM session for production instance."

Service Actions Consistent with justification?
SSM DescribeInstanceInformation, DescribeParameters, DescribeSessions Yes — directly related to SSM session investigation
EC2 DescribeInstances Yes — reasonable for identifying the production instance
EC2 DescribeSecurityGroups, DescribeNetworkInterfaces, DescribeVpcs Partial — could support incident investigation but goes beyond SSM access
IAM ListUsers, ListRoles, ListPolicies, ListAccessKeys No — broad IAM enumeration is unrelated to SSM session access
IAM CreateAccessKey No — write action creating persistent credentials, unrelated to incident investigation
CloudWatch DescribeAlarms Partial — could support incident investigation
CloudWatch Logs DescribeLogGroups Partial — could support incident investigation
S3 ListBuckets No — not related to SSM session access
Bedrock InvokeModel (x2) No — may be trailtool's own summarization, but occurred under break-glass role
STS GetCallerIdentity Yes — standard identity check

Summary:

  • 18 events across 8 services during a ~3.4-hour break-glass session (2026-03-24 00:00–03:24 UTC)
  • 3 actions clearly consistent with the justification: SSM DescribeInstanceInformation, DescribeParameters, DescribeSessions
  • 1 critical discrepancy: iam:CreateAccessKey is a write action that creates persistent credentials surviving beyond the break-glass session — this should be investigated immediately to determine which IAM user the key was created for and whether it has been used or revoked
  • 4 actions inconsistent with the stated justification: broad IAM enumeration (ListUsers, ListRoles, ListPolicies, ListAccessKeys) and S3 ListBuckets have no connection to accessing an SSM session
  • 5 actions partially consistent: EC2 network/security group/VPC reads, CloudWatch alarms, and log group reads could support broader incident investigation but exceed the narrow stated intent of "access SSM session for production instance"
  • Recommendation: The CreateAccessKey action is a red flag and should be investigated immediately. The session shows broad exploratory activity across IAM, EC2, S3, and Bedrock that significantly exceeds the narrow justification provided
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment