My Debian laptop rootfs decided to go readonly this evening. The usual solutions are to either fsck the drive or remount readwrite. Unfortunately the usual solutions didn’t work this time. These are my notes on fixing the issue, any mostly for my own reference and to document the pain and suffering involved…<\/p>\n
First of all it seems that my system didn’t have fsck.ext4 installed, and of course I couldn’t install it as the system was readonly. In anycase, as I’m using the root FS I wouldn’t have been able to run it anyway…<\/p>\n
So I attempted to remount rw using “mount \/ -o remount,rw” as usual but was presented with the following error:<\/p>\n
“Couldn’t remount RDWR because of unprocessed orphan inode list. Please umount\/remount instead”<\/p>\n
So… what next. Well after much fumbling I decided to see if I could create a new FS and boot to that, install fsck.ext4 and clear the errors. Of course my system also lacked mkfs.ext4… So I figured maybe I could create a new root file system in a ram disc install mkfs.ext4 on that and then setup the new partition.<\/p>\n
I created a tmpfs (ram disc) using “mount tmpfs .\/mnt -t tmpfs -o size=2048mb”. Then used:<\/p>\n
\r\nmkdir sbin;mount --bind \/usr \/mnt\/sbin\r\nmkdir bin;mount --bind \/bin \/mnt\/bin\r\nmkdir usr;mount --bind \/usr \/mnt\/usr\r\nmkdir lib;mount --bind \/usr \/mnt\/lib\r\nmkdir lib64;mount --bind \/lib64 \/mnt\/lib64\r\nmkdir proc;mount --bind \/proc \/mnt\/proc\r\nmkdir run;mount --bind \/run \/mnt\/run\r\n<\/pre>\nTo bind the various static user files. I copied etc manually. var was not copied, but I copied files into var as required. Make sure the swap is turned off (swapoff -a)<\/p>\n
Next it chrooted into this tmpfs (chroot .\/mnt). apt-get update (faff around adding missing files\/directories to \/var). Then, apt-get install e2fsprogs. This fails as too much of the tree is still readonly. Anyway… dig out the dpkg (find \/ -name e2fsprogs*). And extract it… (mkdir e;dpkg -x e2fsprogs_1.42.12-1.1_amd64.deb e).<\/p>\n
Then find mkfs.ext4 in the extracted files… and make the new fs (mkfs.ext4 \/dev\/sda5 (or whereever your swap was)) .<\/p>\n
Then mount and copy all the important parts of your system across:<\/p>\n
\r\nmkdir m\r\nmount \/dev\/sda5 m\r\ncp -r \/etc m\r\ncp -r \/lib m\r\ncp -r \/lib64 m\r\ncp -r \/e m\r\ncp -r \/bin m\r\ncp -r \/sbin m\r\ncp -r \/boot m\r\n...\r\n<\/pre>\nReboot… struggle with grub changing the boot parameters (e for edit, then change hd0,msdos1 to hd0,msdos5) until the system boot to the new drive.<\/p>\n
Find out that it crashes into the initrd. mount the new partition. chroot to the new partition. Run fsck.ext4 \/dev\/sda1. Hit Y a bunch. Reboot. Sleep.<\/p>\n","protected":false},"excerpt":{"rendered":"
My Debian laptop rootfs decided to go readonly this evening. The usual solutions are to either fsck the drive or remount readwrite. Unfortunately the usual solutions didn’t work this time. These are my notes on fixing the issue, any mostly for my own reference and to document the pain and suffering involved… First of all […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[1],"tags":[],"class_list":["post-2706","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p1RRoU-HE","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/posts\/2706","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/comments?post=2706"}],"version-history":[{"count":3,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/posts\/2706\/revisions"}],"predecessor-version":[{"id":2709,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/posts\/2706\/revisions\/2709"}],"wp:attachment":[{"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/media?parent=2706"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/categories?post=2706"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/41j.com\/blog\/wp-json\/wp\/v2\/tags?post=2706"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}