Fix server unable to boot after applying kernel patches in an OVH dedicated server with cPanel
Version 0.66

Recently I rented a OVH dedicated server and was happily using it until WHM prompted me to restart the system " You must reboot the server to apply software updates" to apply the latest patches

After restarting the system, the server did not come back online even after ten minutes.

Logging into the IPMI interface via Java, I found that, to my horror, it had booted to a Minimal BASH-like line editing mode with a grub> prompt

Update November 2022: Apparently regions such as Singapore (US does) do not have the option "OVH Network mode". If you are from such a region, I'm sorry that this guide does not apply to you. On clarifying with Support, "[..] this feature is a legacy feature and is being phased out in favour of rescue mode [...]".

Disclaimer

Proceed at your own risk! The information here is accurate to the best of our knowledge. We will not be held responsible if this document causes your computer to explode or burst into flames.

In real serious terms, if any corruption of data, hardware damage or any other kind of damage/losses/etc. arises from the use of this document, we cannot be held responsible for it. If you don't like this, please don't read any further.

Further Disclaimer

I do not claim to be an expert in Linux. That is why I use cPanel to automate the technical stuff. I document in this article of how I fixed my server as I found very few resources dedicated to this problem

If you find an error or have further advice, feel free to correct me in the comments!

Restart your server

Restart your server with OVH Network mode. How to do has already been documented here.

You MUST make sure to type in the correct Root Device. If unsure, see below under "My server did not reboot successfully". In my case, it was not /dev/sda1 but /dev/sda2. I made this mistake and the server did not boot although I could see there was a long debug error log through the IPMI interface.

Note: If you use a Raid Array, your Root Device should be different. Something like /dev/md2.

When you successfully restart, your server should be magically back online as if nothing happened. Yay! If it doesn't, you may want to verify the Root Device. If the IPMI screen shows an error log, it's most probable you had put in a incorrect Root Device. The next section details this

Now is a great time to do those backups if you haven't done so already. You can also delay further rebooting until a less active time. Yes, you'll need to reboot again as you are currently booting from OVH Network mode. I'm not sure what are the repercussions of continuing to use OVH Network mode.

UPDATE from OVH support if you permanently set it to boot from network

There would likely be some performance derogation as you are using a kernel that is being accessed over the network. How much or what it'll impact has not been tested by us but this at least allows your system to been when there's an incorrect GRUB configuration.

My server did not reboot successfully

You can access the Rescue Mode to figure out your Root Device.

Follow the official OVH guide here to set the server to boot in Rescue mode (see under "Boot from Rescue mode").

After your server reboots into rescue mode and you have the login creditentials, login to your server and execute

fdisk -l

You'll need to figure out which is the hard disk you are booting from. For me, it was /dev/sda.

Next you'll need to figure out the partition to boot from. Unfortunately, I do not remember what I did to figure it out so I suggest trying to reboot with OVH Network mode from /dev/sdaX where X is a number from 1 and sda is your hard disk.

Find the problem

The problem was most likely caused by a missing grub.cfg file.

Fire up your SSH or Filezilla and see if such a file is located at /boot/efi/EFI/centos/.

cd /boot/efi/EFI/centos/
ls

If the file exists, you should STOP following this guide as it does not apply to you. Because the next step we will be regenerating this file and the cause of the whole problem is because this file was missing in the first place.

UPDATE: I had this problem again even though the file existed! I proceeded with the fix and it worked like the charm

Fix the problem

Simply run the following line to regenerate the grub.cfg

grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

The grub.cfg file should now be generated

Restart the server

Go back to your OVH panel and set your boot mode to be from hard disk.

Do a Graceful Restart of the server and cross your fingers

How to prevent the problem from re-occuring

Run the following command to prevent Yum from updating the kernel

yum-config-manager --disable ovhkernel

To force a kernel update run the command

yum --enablerepo=ovhkernel update kernel

The info above was provided by cPanel Support and I have just implemented it on the server. Will update how it goes.

Update 11 Dec: Just did a casual system restart with no problems

Why you should use OVH's kernel

From OVH's support

The main reason to use our kernel is mostly for reliability. Our distribution team makes them for our servers and the primary reason why suggest to use ours is for the network drivers. Sometimes you'll find with server hardware that the Linux kernel doesn't have the drives for a specific network card, a lot of times it's an issue with 10G cards that are not that common.

Credits

I do not claim credits for the above solution. The following links helped when I experienced this problem

1. Various posts in cPanel forums such as this and this
2. Fix OVH Server Struck at grub2 prompt
3. cPanel Support - they were very helpful
4. OVH Support

Do you have more to share? Comment below!

Last Updated 10 November 2022.

Errors? Omissions? Need Help? Know something? Post your queries in the comments below.

This document is Copyright(©) 2019 by G.Ganesh. Visit Bootstrike.Com (http://bootstrike.com).