Resolving Inaccessible VM Due to Full Disk

Introduction

A full disk on a virtual machine (VM) can render it inaccessible, disrupting normal operations. Recovering from such issues requires a methodical approach to restore VM functionality without data loss.

Problem Statement

The VM is inaccessible due to a full disk, which prevents normal operations, including SSH access and system recovery.

Proposed Solution

Create a temporary VM to resolve disk issues by detaching, resizing, and reattaching the disk.

Steps to Implement

1. Create a Temporary VM:

   • Deploy a new temporary VM in the same network and zone as the inaccessible VM.

2. Detach the Disk from the Inaccessible VM:

   • Use your cloud provider's console or CLI to detach the full disk.

3. Attach the Disk to the Temporary VM:

   • Mount the detached disk to the temporary VM for analysis and resizing.

4. Analyze and Free Space:

   • SSH into the temporary VM.

   • Mount the disk and free up unnecessary files or resize the partition using tools like resize2fs or disk utilities.

sudo mount /dev/sdX1 /mnt/temp-disk sudo rm -rf /mnt/temp-disk/path-to-large-files

5. Detach and Reattach the Disk:

   • Unmount the disk from the temporary VM, detach it, and reattach it to the original VM.

6. Restart the Original VM:

   • Verify that the disk issue is resolved and the VM is accessible.

Why This Solution is Best

• Efficiency: Directly addresses the issue by isolating the problematic disk.

• Data Integrity: Preserves existing data during recovery.

• Minimized Downtime: Allows for quick restoration without rebuilding the entire VM.

Alternate Solutions

1. Create a New Machine from the VM Image

• Steps:

  • Use the cloud provider’s snapshot or image functionality to create a new VM from the existing image.

  • Reconfigure services to use the new VM while retaining the same disk data.

• Pros:

  • Ensures continuity of operations without waiting for disk repairs.

• Cons:

  • Requires reconfigurations and may incur additional storage costs.

2. Use Serial Console to Free Up Space

• Steps:

  • Access the VM using the serial console.

  • Free up space by removing unnecessary files or logs from /tmp, /var/log, or similar directories.

rm -rf /path-to-large-files

• Pros:

  • Quick fix without detaching the disk.

• Cons:

  • Requires familiarity with the serial console interface.

  • Limited by the extent of disk usage.

Summary

When a VM becomes inaccessible due to a full disk, isolating and repairing the disk via a temporary VM is the most effective solution. For less critical issues, creating a new VM from a snapshot or using the serial console can also provide viable alternatives.

This structured approach ensures minimal disruption while preserving data integrity and operational continuity.