agama-systemd-timeout.md

Analysis Summary

• Machine 1: https://bugzilla.suse.com/show_bug.cgi?id=1270014
• Machine 2: https://bugzilla.suse.com/show_bug.cgi?id=1265451

Problem Overview

The agama-web-server.service consistently fails to start within the systemd timeout period on both machines. The service times out before completing its initialization sequence, preventing the Agama UI from appearing.

Root Cause Analysis

1. D-Bus Storage1 Service Activation Delay (Primary Issue)

The main bottleneck is the extremely slow activation of the org.opensuse.Agama.Storage1 D-Bus service:

• Machine 1 (journalctl-1.log):

  • Storage1 activation requested: 11:31:28
  • Storage1 successfully activated: 11:32:43 (75 seconds delay)
  • Service timeout: 11:32:47 (only 4 seconds after Storage1 activation)

• Machine 2 (journalctl-2.log):

  • Storage1 activation requested: 09:43:08
  • Storage1 successfully activated: 09:44:11 (63 seconds delay)
  • Service timeout: 09:44:33 (22 seconds after Storage1 activation)

The Storage1 service takes over 60-75 seconds to activate, consuming most or all of the systemd startup timeout window.

2. Incomplete Initialization After Storage1 Activation

Even after Storage1 successfully activates, the web server doesn't complete its initialization:

• Machine 1: Times out just 4 seconds after Storage1 activation, before completing network and product configuration
• Machine 2: Times out 22 seconds after Storage1 activation, during or after product loading phase

The web server continues initialization steps after Storage1 becomes available:

• Network Manager checkpoint creation and connection updates
• Product loading (SLES, SLES_SAP)
• Additional configuration steps

However, the systemd timeout terminates the service before these steps complete.

3. Systemd Timeout Configuration

The default systemd TimeoutStartSec appears to be insufficient for the actual initialization time required. Based on the logs:

• Total time from service start to timeout: ~79-85 seconds
• Service is still actively performing initialization when terminated
• No evidence of a hung process or deadlock - the service is progressing

Secondary Observations

1. Missing file warnings (non-critical):

   • Several "No such file or directory" warnings for zypp-related files:
     • /run/agama/zypp/var/lib/zypp/SoftLocks
     • /run/agama/zypp/var/lib/zypp/RequestedLocales
     • /run/agama/zypp/var/lib/zypp/AutoInstalled
     • /run/agama/zypp/etc/zypp/locks
   • These appear to be expected for first-run scenarios and don't block service startup

2. Keyboard layout warnings (non-critical):

   • ruwin_alt(UTF-8), tj_alt(UTF8), ua(utf) not found in xkb database
   • These are informational and don't affect service startup

3. gnome-kiosk waiting loop:

   • gnome-kiosk-script repeatedly checks for web server availability every 2 seconds
   • Shows external services are waiting for agama-web-server to complete startup

Possible Solutions

Option 1: Increase systemd Timeout (Recommended)

Increase the TimeoutStartSec in the agama-web-server.service unit file to allow sufficient time for Storage1 activation and subsequent initialization:

[Service]
TimeoutStartSec=180

Rationale: The service is functioning correctly but needs more time. A 3-minute timeout provides buffer for:

• Storage1 activation (60-75s)
• Network configuration (10-15s)
• Product loading and final initialization (20-30s)

Option 2: Optimize Storage1 Initialization

Investigate why the Storage1 D-Bus service takes 60-75 seconds to activate:

• Profile the storage probing/scanning operations
• Check for unnecessary hardware detection delays
• Consider lazy initialization or background scanning for non-critical storage devices
• Review iSCSI scanning behavior (logs show iSCSI-related activity)

Option 3: Implement sd_notify Protocol

Convert agama-web-server.service to use Type=notify and implement systemd notification protocol:

• Add Type=notify to service unit
• Have the web server send sd_notify("READY=1") when actually ready to serve requests
• This prevents premature timeout while still catching actual hangs

[Service]
Type=notify
NotifyAccess=main
TimeoutStartSec=120

Option 4: Split Service Initialization

Restructure the service to separate critical and non-critical initialization:

• Start the HTTP server and basic UI early
• Load Storage1 and other heavy components asynchronously
• Show a "Loading..." state in the UI while components initialize
• This improves perceived startup time and prevents timeout

Option 5: Add Service Dependencies

Make agama-web-server.service explicitly depend on faster completion of prerequisite services:

[Unit]
After=agama-storage.service
Requires=agama-storage.service

This might allow pre-starting the Storage1 service earlier in the boot sequence.

Recommended Action Plan

1. Immediate fix: Increase TimeoutStartSec to 180 seconds in agama-web-server.service
2. Short-term: Investigate and optimize Storage1 initialization performance
3. Long-term: Implement Type=notify with proper readiness signaling
4. Consider: Asynchronous initialization for better user experience

Additional Investigation Needed

• Review Storage1 service implementation to identify the 60-75 second delay
• Check hardware configuration differences that might affect storage scanning time
• Examine if storage device count or types correlate with delay duration
• Test if disabling certain storage probes (iSCSI, multipath) reduces initialization time