My Problem:

Booting my Fedora 14 laptop after a clean shutdown resulted in the following boot-time error message:

/dev/mapper/vg_fedora1530-lv-home: UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY (i.e., without -a or -p options)

My Solution:

Boot into a Linux Live CD, unmount all affected partitions (assuming they were automounted) and perform an e2fsck -f. In the case of wanting to unmount all partitions on your sda disk:

umount /dev/sda*
fsck /dev/sda1 -f

The -f switch forces the checking of the filesystem even if nothing appears to be wrong. Hey, you can’t be too careful. Optionally, you can add the the -p or -y options. From the e2fsck man page:

-p Automatically repair (“preen”) the file system. This option will cause e2fsck to automatically fix any filesystem problems that can be safely fixed without human intervention. If e2fsck discovers a problem which may require the system administrator to take additional corrective action, e2fsck will print a description of the problem and then exit with the value 4 logically or’ed into the exit code. (See the EXIT CODE section.) This option is normally used by the system’s boot scripts. It may not be specified at the same time as the -n or -y options.

-y Assume an answer of `yes’ to all questions; allows e2fsck to be used non-interactively. This option may not be specified at the same time as the -n or -p options.

The Long Story:

Booting up my laptop for the morning, I walked away to grab some breakfast. When I came back, I noticed that it was not at the customary Fedora 14 login screen. Instead, it was a shell prompt blinking just undernearth an ominous red “FAILED” warning. Something was wrong with one of my filesystems.

/dev/mapper/vg_fedora1530-lv-home: UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY (i.e., without -a or -p options)

Running fsck manually basically means that you have to accept each and every possible change to the filesystem that fsck recommends. The -a option is the same as the -p option and is only kept around for backwards compatibility. The -p option fixes only those things that are considered safe enough to fix without human intervention. I’m not sure what logic is set to determine what needs human intervention, so I’d love to hear from someone that knows. The -y option automatically selects “yes” to any and all requests for intervention from fsck.

The error above wants me to manually intervene for every possible error. I thought about it for a minute. I know virtually nothing about the grit and grime of a file system so won’t know what I should and should not change.

You can run fsck -n on mounted filesystems as it does not perform any writes. It basically opens the FS as read only. I did that and saw an avalanche of errors tumble down my screen. It ended with the ominously worded warning:

Error while iterating over blocks in inode 11027197: Illegal triply indirect block found
e2fsck aborted

I rebooted into a Fedora live CD, (the same CD that I used to install Fedora about six months ago), so that I could operate on the unmounted filesystem. I ran fsck.ext4 (which is really just e2fsck – more on that whole fsck mess in a future post) on my lv_home partition with the -f flag to force the check even if the filesystem looked fine. I did not use -p or -y, even though I wanted to for time’s sake. I knew this check would take a quite a few minutes.

I kicked off the fsck operation and in fact there were so many errors in the filesystem and I had so little clue what fixes I should and should not be accepting I ended up wedging a pen between my monitor and the keyboard’s ‘y’ key. I’m ghetto like that. After minutes and minutes of errors whizzing by on the screen, the check was done. I nervously rebooted the machine and chose to boot from the troubled partition. Happily, everything worked. I was greeted by my old familiar login screen and all seemed well. My lost+found folder was totally empty.

But why was my filesystem corrupted in the first place? I have no idea. It hasn’t been hard rebooted. There have been no power issues. Perhaps the physical drive is going. I’ll be checking SMART data sometime soon.

Let me know how you handle ext* corruption issues. Any way that you preempt corruption? Any way that you handle it in an automated fashion?

The Nubby Admin
Facebook

Comments

voretaq7
August 10, 2011

As a BSD user I’d just like to say “fsck_y_enable=’YES’”

Of course I have bad karma now – the next time I have to deal with a problem like this it will be an “Unexpected soft update inconsistency” which I believe fsck still refuses to handle on its own (preferring to offload the data loss risk on to the user if you make the wrong decision). But those are pretty rare…

- Wesley David
  August 10, 2011
  
  So you’re okay with a -y to any fsck operation? I got spooked as a result of the ominously caps locked “DO THIS MANUALLY!!1!” message. I should have taken a picture of the pen wedged into the ‘y’ key. I guess they’re just being cautious so if someone whines they can say “We told you not to do that!”
  
  tune2fs. That’s the final word. =)
  
Bart Silverstrim
August 10, 2011

The pen on the key thing reminds me of the “beware of programmers carrying screwdrivers” bit…

Handling errors-when they occur out of the blue like that and there’s no log entries, I get nervous that there’s something else wrong, whether fans had died and it overheated or there’s a minor error somewhere on the drive signaling it’s dying.

I normally look for SMART status, and I’d probably use a bootable CD to scan the drive at the block level for errors. I normally use UBCD with an older version of hdat2 to do that.

Push comes to shove, I’d be tempted to open the system up and make sure the cables are all secure (And clean out dust from fans while I’m in there, and check that they’re actually working).

If fsck is happy after a couple checks, maybe a badblocks check, a hdat2 check, nothing in the logs, nothing in the SMART status, and the cables all seem secure and the fans working, then it’s probably time to either go to bed or shuffle to the dresser to change for work…that’s how I end up caring for filesystem errors like that on my own system

- Wesley David
  August 10, 2011
  
  hdat2 looks awesome. I’ll have to check that one out. Great list of ways to check for a potentially failing drive! I’ll have to somehow automate that so I can scan and log all output and fix what is prudent without staring at my screen until dawn. =)
  
  - Bart Silverstrim
    August 11, 2011
    
    I remember there is a brand-new version that seems to be crippled, and an older release of HDAT2 that will fix issues when it finds them, but I’m not sure. That’s why I have an older version around just in case.

Solving the Error “Unexpected Inconsistency: Run fsck Manually” on a Linux Machine

My Problem:

My Solution:

The Long Story:

Comments

Add a Comment

Advertisements

Follow This Blog

About Me!

Contact Me!

The Nubby Archives

Be Social!

Circle me!