Solving NFS Mounts at Boot Time

(Today’s post is brought to you by the letters N, F, and S. Oh, and Scott Pack too.)

Let’s face it. NFS is a magical thing. It allows you to centralize your storage, share volumes across systems, and all while maintaining sane permissions and ownership. Unfortunately, it can also be a bit of a fickle beast. Let’s say you just had your volume configured and you set up the mounts. You go and run this command:

mount -t nfs /data

Works like a champ, you now have your data partition mounted over NFS. So you add this line to your /etc/fstab and make it mount automagically. /data nfs defaults 0 0

A few weeks go by and you apply a kernel update. No big deal, you apply the updates and during your next maintenance window reboot to apply the new kernel. Then you start to see applications failing and notice the volume isn’t actually mounted. This is an unfortunate result of the automounter subsystem.

It’s like this. At boot time the root partition gets mounted, automounter reads the /etc/fstab file, and boots any filesystem that doesn’t have noauto as a mount option. We’re still very early in the boot process so the network isn’t up yet, so naturally any network filesystems fail. The real problem here is that at no point does automounter go back and attempt to remount those systems. So your NFS mount points fail because there is no network, and done is done.

The developers were nice enough to provide a fix for this. There exists a mount option called _netdev. If we quote directly from the man page (sourced from RHEL 6.4):

The filesystem resides on a device that requires network access (used to prevent the system from attempting to mount these filesystems until the network has been enabled on the system).

This is awesome, and exactly what we want. So you modify your entry in fstab to look like this: /data nfs defaults,_netdev 0 0

You’ve been bitten by NFS mounting in the past so you throw this in your test environment and reboot immediately. After the system comes up you notice a problem. Your NFS volumes are still unmounted. You see, there’s a bit of a hitch. Automounter followed the same procedure that it did before, except this time it didn’t even attempt to mount /data. The _netdev option doesn’t tell the system to mount the filesystem when network comes up, it says don’t attempt to mount it at all if the network isn’t up. There is still a missing piece to the puzzle. If you look at your init scripts there is a service called netfs. If you read the script you can see in the chkconfig header this description:

# description: Mounts and unmounts all Network File System (NFS),
# CIFS (Lan Manager/Windows), and NCP (NetWare) mount points.

This is exactly what you need. It is a service whose sole purpose is to read your /etc/fstab and mount network filesystems. All you have to do is enable it

chkconfig netfs on

and watch the magic happen. Now your mount boot process should look something like this:

  1. Automounter reads /etc/fstab
  2. Ignores /data since it has _netdev option set
  3. Mounts all other filesystems
  4. Finishes mount jobs and allows system to continue booting
  5. Network comes up
  6. Service netfs started
  7. netfs reads /etc/fstab and finds an nfs filesystem
  8. netfs mounts /data

What’s funny is that while I was researching this problem I never stumbled across netfs as a service. I had even gone so far as to start planning out my own custom init script that would do exactly this, except specifically for my mount points instead of generalizing. It’s nice to see that I was on the right track, but even better that the tools already existed.


  1. Eugene

    April 10, 2013 at 8:34 am

    Interesting! But in all of my environments, I’ve never noticed a failure to mount with only _netdev. I wonder if I’ve just been rolling dice all this time…


    • ScottPack

      April 10, 2013 at 11:11 am

      It’s possible that the netfs service is enabled by default on some systems, which would still take care of the problem. My practice, and recommendations, are to disable nearly all the services at provision time and then use a configuration management system to enable those that are actually in use. That means that *if* netfs is enabled by distro default, it would be disabled by default in my environment.


  2. Sean

    April 11, 2013 at 5:29 am

    Really love this. We’ve had this problem for a while, one question though is there anyway to have it also do a heartbeat check of it’s mount source before it does the mount?


  3. Core | Fragments

    May 2, 2013 at 2:50 pm

    […] Solving NFS Mounts at Boot Time ( […]


  4. batman

    November 1, 2015 at 6:30 pm

    This article is outdated, the current correct way to fix this issue for systems like Debian 8 using systemd; If experiencing any issues with the mount failing due to the network not being up/available, enable NetworkManager-wait-online.service. It will ensure that has all the links available prior to being active.
    Just simply run the command as root, su or sudo # systemctl enable NetworkManager-wait-online.service
    I found this info on the Arch wiki works good, maybe someone will find this helpful.


    • Antho

      January 14, 2016 at 3:31 pm

      – If you use NetworkManager you can do this by enabling NetworkManager-wait-online.service:
      systemctl enable NetworkManager-wait-online.service

      – If you use systemd-networkd you can do this by enabling systemd-networkd-wait-online.service:
      systemctl enable systemd-networkd-wait-online.service



  5. Roberto Carlos Castillo Medina

    January 31, 2016 at 12:00 am

    Thank you, it works like a charm


  6. noob sysadmin

    February 25, 2016 at 10:14 am

    Thank you. The solution worked for me – Tested on RHEL 6.7 (Red Hat Enterprise Linux Server release 6.7). It worked on RHEL 6, so it would definitely work on CentOS 6 too.


  7. Shithin

    July 14, 2016 at 11:24 pm

    Great explanation. You have saved our day


Leave a Reply

Your email address will not be published. Required fields are marked *

Follow TheNubbyAdmin!

follow us in feedly

Raw RSS Feed:

Contact Me!

Want to hire me as a consultant? Have a job you think I might be interested in? Drop me a line:

Contact Me!

Subscribe via Email

Your email address is handled by Google FeedBurner and never spammed!

The Nubby Archives

Circle Me on Google+!

Photos from Flickr

Me on StackExchange

The IT Crowd Strava Group