Kernal Panic - Not syncing : VFS: unable to mount root fs on unknown-block after Ubuntu Upgrade
This week I had the great idea to upgrade my server from Ubuntu 16.04 LTS to Ubuntu 18.04 LTS. After doing all my backups etc I was ready to go. I ran the do-release-upgrade and off it went. When I came back to it I couldn't SSH in or access from the Digital ocean console so I restarted it through Digital Ocean Power Setting. WOW... big mistake. I was presented with
Kernal Panic - Not syncing : VFS: unable to mount root fs on unknown-block (0,0)
. WTF!!! Following the Digital Ocean guide to restart from Recovery ISO. I loaded the Access page again and was presented with some options. First press 1 to mount the drive. then press 5 to enter chroot. Just a reminder these commands may not be safe or appropriate but I thought I would dump my entire process so it may help others, please don't run anything unless you are sure and if you can recover from a backup do that instead, I just thought I would learn from it and if I couldn't fix then I would use the backup, I take no responsibility for anything that may happen because of the following. I used the following commands:
ls -l
//If there is a large swapfile then delete it
rm -rf swapfile
sudo fallocate -l 1G /swapfile
//Then run the following
sudo rm /var/lib/apt/lists/lock
sudo rm /var/lib/dpkg/lock
sudo rm /var/lib/dpkg/lock-frontend
sudo dpkg --configure -a
sudo apt clean
sudo apt update --fix-missing
sudo apt install -f
sudo dpkg --configure -a
sudo apt upgrade
sudo apt dist-upgrade
sudo apt autoremove
// If you get error processing package linux image --configure
sudo dpkg --purge linux-image-4.4.0-81-generic
sudo apt-get install --reinstall linux-image-4.4.0-81-generic
sudo update-initramfs -u -k 4.4.0-81-generic
//Error zz-update-grub failed to get canonical path of overlay
sudo mv /etc/kernel/postrm.d/zz-update-grub /etc/kernel/postrm.d/zz-update-grub.bad
sudo apt-get install -f
//Error cant create temp cache file /etc/ld. no such file or directory
sudo chmod -R 777 *
chown root:root /usr/lib/sudo/sudoers.so
chmod 644 /usr/lib/sudo/sudoers.so
chown root:root /etc/sudoers
chmod 644 /etc/sudoers
chown root:root /etc/sudoers.d/*
chmod 644 /etc/sudoers.d/*
sudo fsck.vfat -r /dev/vda1
sudo update-initramfs -tu
mkdir /etc/ld.so.cache~
sudo chmod -R 777 /etc/ld.so.cache~
//Didnt work due to grub try this
sudo grub-install --boot-directory=/mnt/ubuntu/boot /dev/vda
//Continue Installation
sudo apt-get update
sudo apt-get install grub
sudo apt-get -f install
apt-get autoremove
//Update the packages
apt-get update && apt-get upgrade -y
//Run this and it will bring back kernels
sudo apt-get install --reinstall linux-image-generic linux-image
//change the end to one you have
sudo apt-get install --reinstall linux-image-generic linux-image-4.15.0-1006-gcp
//After trying it all through option 5 i then tried it again through option 6 the bin/bash and it started working again but thought i would show my whole thought process
//When booting from HDD i got grub menu
//Restart as option 5 root in recovery mode
sudo nano /etc/stab
//Add to the bottom
/dev/vda1 /mmt ext4 defaults,nofail,discard 0 2
//reboot from HDD
grub>
ls (hd0,1)/
cat (hd0,1)/etc/issue
//I got the vmlinux value from running sudo apt update --fix-missing or sudo apt-get upgrade in chroot in recovery and watching a list of outputs
//Everytime I boot server it brings up grub menu and have to enter these 3 commands to get the server running again
linux /boot/vmlinuz-4.4.0-81-generic root=/dev/vda1
initrd /boot/initrd.img-4.4.0-81-generic
boot
//Go back into Chroot and run
sudo grub-set-default "Ubuntu, with Linux 4.4.0-81-generic"
sudo update-grub
Or
sudo apt-get update
apt-cache search --names-only linux-image
//Find name and replace below
sudo apt-get install linux-image-4.4.0-81-generic linux-headers-4.4.0-81-generic
//sudo: /usr/bin/sudo must be owned by uid 0 and have the setuid bit set
//Boot in recovery and chroot in
mount -o remount,rw /
mount --all
chown root:root /usr/bin/sudo
chmod 4755 /usr/bin/sudo
chown root:root /etc/sudoers.d
chmod 4755 /etc/sudoers.d
restart
//Cannot create swap due to no space
sudo du -a / 2>/dev/null | sort -n -r | head -n 20
//Delete largest
//Or create a swap file
sudo fallocate -l 1G /swapfile
sudo dd if=/dev/zero of=/swapfile bs=1024 count=1048576
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo nano /etc/fstab
//Add this linux
/swapfile swap swap defaults 0 0
sudo swapon --show
Change back to the HDD in recovery in Digital ocean and it should now start up. Phew!!! This may or may not help but somehow I managed to get the server running again. For safety I have restored from a backup but was a good learning process.
Categories: Posts