Service Crashed – How to Diagnose and Restart Failed Services Gracefully: Root Cause Analysis Guide

Photo by David Pupăză on Unsplash

In modern infrastructure, services are the lifeblood of applications. They manage queues, handle database connections, serve web requests, and execute background tasks—all operating silently until something goes wrong. When a critical service unexpectedly ceases operation, the resulting outage can range from minor inconvenience to catastrophic business disruption.

A common mistake among administrators is treating the symptom (the stopped service) rather than the root cause (the underlying issue—be it memory exhaustion, a bad configuration parameter, or an external dependency failure). Simply running a restart command masks the problem and guarantees recurrence.

This guide walks you through a complete, hands-on workflow for diagnosing and recovering failed services. Every section includes practical commands, real-world examples, and a step-by-step checklist you can apply immediately.

Part 1: Diagnose the Failure State

Before attempting any fix, accurately determine the current state of the service. This section applies to Linux (systemd, SysVinit), Windows, and containerized environments.

1.1 Linux: Check service state with systemd

Command:

systemctl status <service-name>

What to look for:

Example: Checking an nginx service that is failing:

$ systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2026-09-01 14:23:01 UTC; 3min 22s ago
           Process: 1234 ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;' (code=exited, status=1/FAILURE)
          Main PID: 1234 (nginx)
             CPU: 2.134s

Sep 01 14:23:01 webserver.example.com systemd[1]: Starting A high performance web server and a reverse proxy server...
Sep 01 14:23:01 webserver.example.com nginx[1234]: nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied)

Key insight from this example: The failure is not a code bug; it’s a filesystem permission issue. The service failed with status: 1/FAILURE and the error is mkdir() "/var/cache/nginx/client_temp" failed (13: Permission denied).

1.2 Linux: Check fast-failing state (crash loop)

Command:

# Check the restart counter and result
systemctl show <service-name> -p NRestarts -p Result -p MainPID -p StatusErrno

# Check for crash loops in the last 10 minutes
journalctl -u <service-name> --since "10 minutes ago"

Example output:

NRestarts=12
Result=signal
MainPID=5678
StatusErrno=0

What this means: NRestarts=12 in a short window = crash loop. Result=signal = the process was killed by a signal (likely SIGKILL = OOM kill).

Action: Stop the automatic restart, investigate the kernel logs:

dmesg -T | grep -i "killed process"
journalctl -k --since "10 minutes ago"

1.3 Windows: Check service state

PowerShell (faster, more informative):

Get-Service <service-name> | Select-Object Name, DisplayName, Status, StartType, CanStop

Output:

Name               DisplayName                            Status   StartType CanStop
----               -----------                            ------   --------- -------
WebService        IIS Admin Service                      Stopped  Manual      True

Command Prompt alternative:

sc query <service-name>

Example:

C:\> sc query WebService

QUERY SERVICE SAMPLE BEGIN
SERVICE_NAME: WebService
        TYPE               : 10  WIN32_OWN_PROCESS
        STATE              : 1  STOPPED
        CONTROLS_CAPACITY  : 0
        WIN32_EXIT_CODE    : 0  (0x0)
        SERVICE_EXIT_CODE  : 0  (0x0)
        CHECKPOINT         : 0x0
        SERVICE_TRANSITION: 0x0

Key state codes:

Part 2: Perform Deep Log Analysis

Once you know the service is failing, the next step is forensic log analysis. Never trust a single log source—correlate across at least two.

2.1 Linux: Three-source log correlation

When a service crashes, its actions leave trails in at least three places:

Example: Correlating the nginx failure from Part 1

# 1. Application/system log (nginx unit journal)
journalctl -u nginx -n 50 --no-pager

# 2. Kernel log (OOM kills, signals)
dmesg -T | tail -30

# 3. Full PID trace (which process died, why)
journalctl -u nginx --since "14:20" --until "14:25"

# 4. Check the specific PID
systemctl show nginx -p MainPID -p NRestarts -p Result

Example output of kernel log (the smoking gun):

[14:23:01.123456] nginx[1234]: segfault at 0 ip 0x0 sp 0x7ffd4f4d2b30 error 14
[14:23:01.123456] nginx[1234]:  #0  0x000055a8b1c3d423 in ???

Action: The ip 0x0 and error 14 means a null pointer dereference (a code bug). The service should be fixed in the application, but you also check whether the filesystem permission issue (from earlier) was a misconfiguration that also needs fixing.

2.2 Windows: Check Event Viewer for service failures

Event Viewer path:

Windows Logs → System
Filter Current Log → Event ID: 7036 (Service Control Manager)

What to look for:

Example: Event 7026 for a Windows service

Log Name: System
Source: Service Control Manager
Event ID: 7026
Level: Error
Description: The Apache2.4 service exited with code 1.

Note: exit code 1 = an application crash or configuration error, not a service manager failure.

2.3 Docker/Kubernetes: Check container logs and status

Docker:

# Check the container status
docker inspect <container-id> --format '{{.State.Status}} {{if .State.Health}}{{.State.Health.Status}}{{end}}'

# View logs (use -t for timestamps)
docker logs --tail 50 -t <container-id>

# If the container restarted, check the previous logs
docker logs --tail 50 -t <container-id> --previous

Kubernetes:

# Describe the pod (shows restarts, recent events)
kubectl describe pod <pod-name>

# Get logs (add --previous to see logs from the last crashed instance)
kubectl logs <pod-name> --previous

# Check the restart count
kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[0].restartCount}'

Example kubectl describe output (crash loop):

Events:
  Type     Reason     Age   From               Message
  ----     ------     ----  ----               -------
  Warning  BackOff    5m    kubelet            Back-off restarting failed container
  Normal   Pulled     10m   kubelet            Container image "nginx:latest" already present on machine
  Warning  Unhealthy  5m    kubelet            Readiness probe failed: HTTP probe failed with statuscode 502

Action: Back-off restarting failed container = the container is failing and Kubernetes is retrying. Readiness probe failed = the app is not responding to health checks. Investigate the application logs:

kubectl logs <pod-name> --previous | tail -50

Part 3: Analyze Dependencies and External Constraints

Most crashes are caused by failed assumptions about the environment. The service itself may be fine; its dependencies may be broken.

3.1 Check service dependencies

Linux (systemd):

# List all dependencies of a service
systemctl list-dependencies <service-name> --all

# Check if a dependent service is active
systemctl is-active postgresql
systemctl is-active redis

Example:

$ systemctl list-dependencies nginx.service --all
nginx.service
├─ network.target
├─ nginx.service
│  └─ postgresql.service   # <-- critical dependency!
└─ rpc-statd.service

Action: If postgresql.service is inactive, start it first before starting nginx:

sudo systemctl start postgresql
sudo systemctl start nginx

3.2 Check resource constraints

The most common cause of unexpected termination is resource exhaustion.

Check memory:

# Real-time memory usage
free -h

# Kernel ring buffer for OOM kills
dmesg -T | grep -i "out of memory"

# Container memory limits
docker stats --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}"

Check CPU:

# Real-time CPU
top -bn1 | head -20

# Process-specific CPU
ps aux --sort=-%cpu | head -10

Check disk:

# Disk space
df -h

# Inode usage (often causes silent failures)
df -i

Example: OOM kill diagnosis

$ dmesg -T | grep -i "killed process"
[14:30:01.123456] Out of memory: Killed process 1234 (nginx) total-vm:1234567kB, anon-rss:654321kB, file-rss:12345kB, shmem-rss:0kB

Action:

• Increase the service’s memory limit (memory_limit in PHP-FPM, --memory in Docker, resources.limits.memory in Kubernetes)
• Optimize the application to use less memory
• Add a swap file: sudo fallocate -l 4G /swapfile && sudo chmod 600 /swapfile && sudo mkswap /swapfile && sudo swapon /swapfile

3.3 Check network ports

Linux:

# Check which process is listening on a port
ss -tlnp | grep <port>

# Check all listening ports
ss -tlnp

# Verify connectivity to a remote port
nc -vz <hostname> <port>

Windows (PowerShell):

Get-NetTCPConnection -LocalPort <port> -State Listen

# Test connectivity
Test-NetConnection <hostname> -Port <port>

Common causes:

• Port already bound by another process (stale process, config error)
• Firewall blocking the port
• DNS resolution failure for upstream services

Part 4: Apply Practical Recovery Strategies

4.1 Graceful restart with systemd

Step 1: Stop the service cleanly

sudo systemctl stop nginx

Step 2: Start the service

sudo systemctl start nginx

Step 3: Verify it’s running

systemctl status nginx
curl -I http://localhost

Step 4: Enable auto-start on boot

sudo systemctl enable nginx

4.2 Configure systemd restart policy

Example /etc/systemd/system/myapp.service

[Unit]
Description=My Application
After=network.target postgresql.service
Requires=postgresql.service

[Service]
Type=simple
User=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/start.sh
Restart=on-failure
RestartSec=5
StartLimitBurst=3
StartLimitIntervalSec=60
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

Key directives explained:

Step 3: Reload systemd and restart

sudo systemctl daemon-reload
sudo systemctl restart myapp

4.3 Docker restart policy

Example docker run command:

docker run -d --restart=on-failure:5 --memory=512m --name myapp myapp:latest

Policy types:

Example: Health check in Docker

docker run -d --health-cmd="curl -f http://localhost/health || exit 1" \
  --health-interval=30s --health-timeout=5s --health-retries=3 \
  --name myapp myapp:latest

4.4 Kubernetes liveness and readiness probes

Example deployment with probes:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp
spec:
  replicas: 2
  selector:
    matchLabels:
      app: myapp
  template:
    metadata:
      labels:
        app: myapp
    spec:
      containers:
      - name: myapp
        image: myapp:latest
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 10
          failureThreshold: 3
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
          failureThreshold: 3
        resources:
          requests:
            memory: "256Mi"
            cpu: "500m"
          limits:
            memory: "512Mi"
            cpu: "1000m"

Probe types explained:

Part 5: Operational Safety and Change Management

5.1 Snapshot and backup before changes

Linux (file system):

# Backup config files before any change
sudo cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak.$(date +%F)

# Snapshot a system (if using Btrfs or ZFS)
sudo btrfs subvolume snapshot / /root/snapshot-$(date +%F)

Kubernetes:

# Backup a deployment
kubectl get deployment myapp -o yaml > myapp-backup-$(date +%F).yaml

# Undo a bad rollout
kubectl rollout undo deployment/myapp

5.2 Least privilege

• Never run service processes as root unless absolutely required. Use a dedicated unprivileged user.
• Use sudo -u myapp systemctl restart myapp instead of sudo systemctl restart myapp.
• Limit systemctl access to production admins only.

5.3 Staging validation

• Never apply a fix directly to production if it can be replicated in staging.
• Use the failure scenario to improve test coverage.
• Build automated validation checks that mimic real-world usage patterns.

5.4 Document recovery steps

Record every recovery action in your runbook:

Part 6: Summary — The Complete Recovery Workflow

Key Takeaways

Related Articles

• The Linux Command Line: What Actually Gets You Through the Work
• Mastering Linux: The Essential Toolkit for Resource Monitoring (50+ Commands with Examples)
• Getting Started with Docker: A Beginner’s Guide to Containerization – Master Modern Infrastructure Today
• Linux System Metrics That Matter: CPU, Memory, Disk I/O — What to Watch Daily