Rescue and Rebuild

Even the best system administrators have to deal with accidents and unplanned events. Fortunately, the Linode Manager has a number of tools to assist you in the unlikely event that catastrophe strikes your VPS. This guide shows you how to use the tools at your disposal. You can boot your Linode into rescue mode to perform system recovery tasks and transfer data off your disk images, if necessary. And if all else fails, you can rebuild your Linode from a backup or start over with a fresh Linux distribution.

Connecting to a Linode Running in Rescue Mode

To access your Linode when it’s running in rescue mode, you’ll need to use the Linode shell (Lish). For instructions, see Console Access and Using the Linode Shell (Lish). Once you’ve successfully logged in, you’ll see the window shown below.

Lish consoleNow you’ve connected to your Linode, you can start doing stuff in rescue mode.

Performing a File System Check

You can use the fsck system utility (short for “file system check”) to check the consistency of file systems and repair any damage detected. If you suspect that your Linode’s file system is corrupted, you should run fsck to check for and repair any damage. Here’s how:

  1. Enter the df command to verify that your primary disk images are not currently mounted. Your primary disk images should not appear in the list. For example, whenwe booted into rescue mode earlier, we specified two disk images: Ubuntu at /dev/xvda and Swap at /dev/xvdb, neither of which are shown as being mounted in the screenshot below.

Output of df command

Note

You should never run fsck on a mounted disk. Do not continue unless you’re sure that the target disk image is unmounted.

  1. To verify the location of your disk images, enter the fdisk -l command. The disk image layout will appear, as shown below. Notice that the Ubuntu disk image is/dev/xvda, the Swap disk image is /dev/xvdb, and the Finnix partition is /dev/xvdh.

Output of fdisk -l command

  1. Run fsck by entering the following command, replacing /dev/xvda with the location of the disk image you want to check and repair:
    e2fsck -f /dev/xvda
  2. If no problems are detected, fsck will display a message indicating that the file system is “clean,” as shown below.

Output of e2fsck command

  1. If fsck determines that there is a problem with your file system, it will perform several tests and prompt you to fix problems as they are found, as shown below. Press enter to automatically attempt to fix the problems.

Output of e2fsck commandOnce the file system check completes, any problems detected should be fixed. You can try restarting the Linode now. With any luck, fsck fixed your problem and the Linode will boot normally.

Mounting Disks

By default, your disk images are not mounted when your Linode boots into rescue mode. However, you can manually mount a disk while your Linode is running in rescue mode to perform system recovery and maintenance tasks. Enter the following command to mount a disk in rescue mode, replacing /dev/xvda with the location of the disk image you want to mount:

mount -o barrier=0 /dev/xvda

Disks that contain a single ext3 file system will have mount points under /media in the rescue environment’s /etc/fstab file. To view the directories on the disk, enter the following command:

ls /media/xvda

Now you can read and write to files on the mounted disk.

Change Root

Change root is the process of changing your working root directory. When you change root (chroot) to your Linode root disk image, you will be able to run commands as though you are logged into that system.

Chroot will allow you to change user passwords, remove/install packages, and do other system maintanance and recovery tasks.

Before you can use chroot, you need to mount your root disk image with execute permissions:

mount -o exec,barrier=0 /dev/xvda

Then to create the chroot, you need to mount the temporary filesystems:

cd /media/xvda
mount -t proc proc proc/
mount -t sysfs sys sys/
mount -o bind /dev dev/
mount -t devpts pts dev/pts/

Chroot to your disk image with the following command:

chroot /media/xvda /bin/bash

To exit the chroot and get back to Finnix type “exit”

exit

Starting SSH

The Finnix recovery distribution does not automatically start an SSH server, but you can start one manually. This is useful if your Linode won’t boot and you need to copy files off of the disks. You can also copy entire disk images over SSH. Here’s how to start SSH:

  1. Set the root password by entering the following command:
    passwd
  2. Enter the password for the root user.
  3. Start the SSH server by entering the following command:
    /etc/init.d/ssh start

Now you can access mounted disks with an SFTP client by using the root user and the password you just set. For instructions on connecting with an SFTP client, see theFile Transfer reference manuals. For instructions on copying an entire disk image over SSH, see Copy a Disk Image Over SSH.

Installing Packages

The Finnix recovery distribution is based on Debian, so you can use the apt package management system to install additional software packages in the temporary rescue environment. For example, you could install and run the htop utility by issuing the following commands:

apt-get update
apt-get install htop
htop

The software packages you install will be available as long as your Linode is running in rescue mode.

Rebuilding

If you can’t rescue an existing disk image, it’s time to rebuild your Linode. There are a couple different ways you can do this. You restore from an existing backup and return your Linode to a previous state. If you don’t have backups, you can copy files off an existing disk image, erase everything, and start over again from scratch.

Restoring from Backup

If you previously enabled the Linode Backup Service, you may be able to restore one of the backups to your Linode. For instructions on restoring from a backup created by the Linode Backup Service, see Restoring from a Backup. If you created backups with an application other than the Linode Backup Service, review the application’s instructions to restore a backup to your Linode.

Recovering From a System Compromise

Did an unauthorized intruder gain access to your Linode? Since it is virtually impossible to determine the full scope of an attacker’s reach into a compromised system, you should never continue using a compromised Linode. We recommend that you follow the instructions in Recovering from a System Compromise. You’ll need to create a new Linode, copy your existing data from the old Linode to the new one, and then swap IP addresses.

Advertisements

Share your thoughts with us!!!

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s