Kernal Panic - Not syncing : VFS: unable to mount root fs on unknown-block after Ubuntu Upgrade
This week I had the great idea to upgrade my server from Ubuntu 16.04 LTS to Ubuntu 18.04 LTS. After doing all my backups etc I was ready to go. I ran the do-release-upgrade and off it went. When I came back to it I couldn't SSH in or access from the Digital ocean console so I restarted it through Digital Ocean Power Setting. WOW... big mistake. I was presented with
Kernal Panic - Not syncing : VFS: unable to mount root fs on unknown-block (0,0)
. WTF!!! Following the Digital Ocean guide to restart from Recovery ISO. I loaded the Access page again and was presented with some options. First press 1 to mount the drive. then press 5 to enter chroot. Just a reminder these commands may not be safe or appropriate but I thought I would dump my entire process so it may help others, please don't run anything unless you are sure and if you can recover from a backup do that instead, I just thought I would learn from it and if I couldn't fix then I would use the backup, I take no responsibility for anything that may happen because of the following. I used the following commands:
ls -l //If there is a large swapfile then delete it rm -rf swapfile sudo fallocate -l 1G /swapfile //Then run the following sudo rm /var/lib/apt/lists/lock sudo rm /var/lib/dpkg/lock sudo rm /var/lib/dpkg/lock-frontend sudo dpkg --configure -a sudo apt clean sudo apt update --fix-missing sudo apt install -f sudo dpkg --configure -a sudo apt upgrade sudo apt dist-upgrade sudo apt autoremove // If you get error processing package linux image --configure sudo dpkg --purge linux-image-4.4.0-81-generic sudo apt-get install --reinstall linux-image-4.4.0-81-generic sudo update-initramfs -u -k 4.4.0-81-generic //Error zz-update-grub failed to get canonical path of overlay sudo mv /etc/kernel/postrm.d/zz-update-grub /etc/kernel/postrm.d/zz-update-grub.bad sudo apt-get install -f //Error cant create temp cache file /etc/ld. no such file or directory sudo chmod -R 777 * chown root:root /usr/lib/sudo/sudoers.so chmod 644 /usr/lib/sudo/sudoers.so chown root:root /etc/sudoers chmod 644 /etc/sudoers chown root:root /etc/sudoers.d/* chmod 644 /etc/sudoers.d/* sudo fsck.vfat -r /dev/vda1 sudo update-initramfs -tu mkdir /etc/ld.so.cache~ sudo chmod -R 777 /etc/ld.so.cache~ //Didnt work due to grub try this sudo grub-install --boot-directory=/mnt/ubuntu/boot /dev/vda //Continue Installation sudo apt-get update sudo apt-get install grub sudo apt-get -f install apt-get autoremove //Update the packages apt-get update && apt-get upgrade -y //Run this and it will bring back kernels sudo apt-get install --reinstall linux-image-generic linux-image //change the end to one you have sudo apt-get install --reinstall linux-image-generic linux-image-4.15.0-1006-gcp //After trying it all through option 5 i then tried it again through option 6 the bin/bash and it started working again but thought i would show my whole thought process //When booting from HDD i got grub menu //Restart as option 5 root in recovery mode sudo nano /etc/stab //Add to the bottom /dev/vda1 /mmt ext4 defaults,nofail,discard 0 2 //reboot from HDD grub> ls (hd0,1)/ cat (hd0,1)/etc/issue //I got the vmlinux value from running sudo apt update --fix-missing or sudo apt-get upgrade in chroot in recovery and watching a list of outputs //Everytime I boot server it brings up grub menu and have to enter these 3 commands to get the server running again linux /boot/vmlinuz-4.4.0-81-generic root=/dev/vda1 initrd /boot/initrd.img-4.4.0-81-generic boot //Go back into Chroot and run sudo grub-set-default "Ubuntu, with Linux 4.4.0-81-generic" sudo update-grub Or sudo apt-get update apt-cache search --names-only linux-image //Find name and replace below sudo apt-get install linux-image-4.4.0-81-generic linux-headers-4.4.0-81-generic //sudo: /usr/bin/sudo must be owned by uid 0 and have the setuid bit set //Boot in recovery and chroot in mount -o remount,rw / mount --all chown root:root /usr/bin/sudo chmod 4755 /usr/bin/sudo chown root:root /etc/sudoers.d chmod 4755 /etc/sudoers.d restart //Cannot create swap due to no space sudo du -a / 2>/dev/null | sort -n -r | head -n 20 //Delete largest //Or create a swap file sudo fallocate -l 1G /swapfile sudo dd if=/dev/zero of=/swapfile bs=1024 count=1048576 sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile sudo nano /etc/fstab //Add this linux /swapfile swap swap defaults 0 0 sudo swapon --show
Change back to the HDD in recovery in Digital ocean and it should now start up. Phew!!! This may or may not help but somehow I managed to get the server running again. For safety I have restored from a backup but was a good learning process.