Today I decided to protect all my Raspberry Pi SD cards from unproper shutdown and also prevent SD card premature end of life. This means protect them from power outage or just unplugging the power while the Pi is running.
Edit :
April 2016 to works on Jessie and Raspberry PI 3
January 2017 Fix SSH access problem after reboot / moved resolv.conf to tmpfs
Why ?
Most of the time you won’t see or have any incidences when the shutdown is not clean, but depending on what is doing the Pi at this time, you can start to have unpredictable results or data corruption. Also, sometime, applications have really verbose logging and continuously writing log to the SD will for sure bring it to death quicker than expected. SD corruption is not really simple to detect, because sometime it works and sometimes Pi is crashing/blocking and you don’t know why. You have these “hazardous” problems until you decide to change the SD. the worst, is that you were just thinking you done something wrong and searched what you done wrong ? Get time, in doubt, change your SD card first. This is my own experience and as today, I’m on 3 dead SD card.
How ?
Well, the 1st thing I’ve done was to set some folder into temp file system (log file for example) and write the log files to the SD only every hour (a example). It’s working fine but I wanted to go deeper and have more protection, so searched over Internet people doing with read-only filesystem. I found lot of interesting article with different methods and decided to start with this one from k3a, really thanks to him.
Requirement, do it or not ?
Using a read-only file system is mostly used for Pi working 24H/7 a day and with console mode only, so if you’re using X11 or graphic interface, I won’t recommend this method because it won’t work. May be there are some other but it’s not on this article scope. So when you want you’re Pi in read-only mode it’s acting like a server but some server applications are not compatible with read only file system. Mainly database.
So if you’re running a database on your Pi, don’t use read-only filesystem. May be again there are some specific build or database to minimize write to SD, but once again it’s not in today’s scope. I see more and more people using mySQL database on Pi claiming it works fine. Sure it works, until you’ll start to face on SD corruption, and I hope they all have a good backup, because we know all, admins always have backup ;).
May be I’m the only one claiming that except if you change only few data a day, you shouldn’t use a RPi as a 24/7 database engine. For a pretty rock solid database, use a clouded one or one on a NAS like a Synology with redundant hard disk. It will go so much faster and you’ll get a real reliability. More over you will be able to use your Pi to do action and manage this database, just host your database elsewhere, just my 2 cents.
Let’s do it
Okay, I’ve done nothing yet, I’m writing this article while setting my Ambilight hyperion Rpi in read only mode and following the K3A article. You need to connect to your Pi with ssh to run the following commands, I’m connecting as root so you won’t see any sudo command (I know, I’m a bad boy!)
Get to latest current version
apt-get update; apt-get upgrade
Reboot in case of bootloader or kernel version change
reboot
Wait for reboot and connect back with SSH.
Remove unwanted package and services files
The original article removed cron and fake-hwclock, I decided to let it working assuming my Pi will have the correct date (I never had a problem with date) and also have a date at bootup that is more the last known than 1970 year. fake-hwclock will be written at logout and every day
apt-get remove --purge wolfram-engine triggerhappy anacron logrotate dphys-swapfile xserver-common lightdm
insserv -r x11-common; apt-get autoremove --purge
Replace log management with busybox one
apt-get install busybox-syslogd; dpkg --purge rsyslog
This will put log into circular memory buffer, you will able to see log using
logread
command
Disable swap and filesystem check and set it to read-only.
edit the file
/boot/cmdline.txt
and add the three words
fastboot noswap ro
at the end of the line
mine were (your can be different depending on several options) :
dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait
and now looks like :
dwc_otg.lpm_enable=0 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait fastboot noswap ro
Move some system files to temp filesystem
rm -rf /var/lib/dhcp/ /var/lib/dhcpcd5 /var/run /var/spool /var/lock /etc/resolv.conf ln -s /tmp /var/lib/dhcp ln -s /tmp /var/lib/dhcpcd5 ln -s /tmp /var/run ln -s /tmp /var/spool ln -s /tmp /var/lock touch /tmp/dhcpcd.resolv.conf; ln -s /tmp/dhcpcd.resolv.conf /etc/resolv.conf
On Raspberry PI 3, move some lock file to temp Files System
Thanks to @harlock for the trick
nano /etc/systemd/system/dhcpcd5
and be sure to change the line with
PIDFile=/run/dhcpcd.pid
to
PIDFile=/var/run/dhcpcd.pid
Full file should now looks something like that (on my Jessie Lite on RPI3).
[Unit] Description=dhcpcd on all interfaces Wants=network.target Before=network.target [Service] Type=forking PIDFile=/var/run/dhcpcd.pid ExecStart=/sbin/dhcpcd -q -b ExecStop=/sbin/dhcpcd -x [Install] WantedBy=multi-user.target Alias=dhcpcd5
On Debian jessie move random-seed file to writable location
remove existing file
rm /var/lib/systemd/random-seed
link the random-seed file to tmpfs location
ln -s /tmp/random-seed /var/lib/systemd/random-seed
Since file is on tmpfs it will not be created upon, reboot, but we can do it with a kind of magic of systemd system service, this is so powerfull.
To create file on the tmp area at bootup before starting random-seed service, just edit the file service file to add a pre-command to execute :
nano /lib/systemd/system/systemd-random-seed.service
add the line ExecStartPre=/bin/echo “” >/tmp/random-seed under service section, should now looks like this
[Service] Type=oneshot RemainAfterExit=yes ExecStartPre=/bin/echo "" >/tmp/random-seed ExecStart=/lib/systemd/systemd-random-seed load ExecStop=/lib/systemd/systemd-random-seed save
Do not use touch instead of echo, it won’t work because checking RO filesystem
Execute following to tell systemd we made changes
systemctl daemon-reload
Setup the Internet clock sync
I think this one is not needed because on new Raspbian version, the ntp daemon already do the job, but just in case
apt-get install ntp
and be sure to configure your time zone, with raspi-config tool
raspi-config
then go to menu “Internationalisation Options” then “Change Timezone” and select your time zone.
Edit the hourly cron script that save clock every hour
/etc/cron.hourly/fake-hwclock
and change it to allow saving clock. It sould become
#!/bin/sh # # Simple cron script - save the current clock periodically in case of # a power failure or other crash if (command -v fake-hwclock >/dev/null 2>&1) ; then mount -o remount,rw / fake-hwclock save mount -o remount,ro / fi
Edit the the file /etc/ntp.conf to set redirect driftfile to writable zone /var/tmp . My file was like this
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help driftfile /var/lib/ntp/ntp.drift
I changed to this, thanks to Gregor for the tip.
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help driftfile /var/tmp/ntp.drift
Remove some startup scripts
insserv -r bootlogs; insserv -r console-setup
Now time to tell the mounted filesystem that we’re in read-only mode
Add “,ro” flag to both block devices in
/etc/fstab
to tell the system to mount them read-only:
edit the file
/etc/fstab
mine was :
proc /proc proc defaults 0 0 /dev/mmcblk0p1 /boot vfat defaults 0 2 /dev/mmcblk0p2 / ext4 defaults,noatime 0 1
And after modifications it is :
proc /proc proc defaults 0 0 /dev/mmcblk0p1 /boot vfat defaults,ro 0 2 /dev/mmcblk0p2 / ext4 defaults,noatime,ro 0 1 # For Debian Jessie tmpfs /tmp tmpfs nosuid,nodev 0 0 tmpfs /var/log tmpfs nosuid,nodev 0 0 tmpfs /var/tmp tmpfs nosuid,nodev 0 0
We done, reboot
That’s it, you should now be able to reboot your Pi
reboot
If all went fine, it should boot. Well, on mine it rebooted fine and ambilight was still working, then I ssh’ed onto and issued
root@ambilight:~# mount | grep /dev/root /dev/root on / type ext4 (ro,noatime,data=ordered) root@ambilight:~#
you can see root is ro mode
root@ambilight:~# mount | grep /dev/mmc /dev/mmcblk0p1 on /boot type vfat (ro,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro) root@ambilight:~#
and /boot (fat part of the SD) is also ro mode, great !!!
In case of problem, look at syslog (or logread) and try to find out why. You can try to fix plugging the SD card in a different computer.
Switching from Read-Only mode to Read-Write and vice-versa
Now you’re in read only mode it’s fine and safe, but if you need to install, write or modify files, upgrade, or whatever that need write access, you need to be able to do it, we’ll add this possibility and in visual mode
To set system to Read-Write use
mount -o remount,rw /
and to set it back to Read-Only
mount -o remount,ro /
but I do not remember this tricky syntax and decided to improve things a little. I want to have two simple commands like
ro
for setting mode to read only and
rw
to enable read write mode. I also wanted to know on which mode I am on command prompt.
Add fancy indicating features
Ok for all users just edit the file
/etc/bash.bashrc
and at the end add the following lines
# set variable identifying the filesystem you work in (used in the prompt below)
set_bash_prompt(){
fs_mode=$(mount | sed -n -e "s/^\/dev\/.* on \/ .*(\(r[w|o]\).*/\1/p")
PS1='\[\033[01;32m\]\u@\h${fs_mode:+($fs_mode)}\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ '
}
alias ro='sudo mount -o remount,ro / ; sudo mount -o remount,ro /boot'
alias rw='sudo mount -o remount,rw / ; sudo mount -o remount,rw /boot'
# setup fancy prompt"
PROMPT_COMMAND=set_bash_prompt
Execute this new file and look at the magic, the prompt has also changed and show up file system state
root@ambilight:/var/log# . /etc/bash.bashrc root@ambilight(rw):/var/log# ro root@ambilight(ro):/var/log# touch test.txt touch: cannot touch `test.txt': Read-only file system root@ambilight(ro):/var/log# rw root@ambilight(rw):/var/log# touch test.txt root@ambilight(rw):/var/log# ro root@ambilight(ro):/var/log#
Use logout to save history and force Read-Only mode
To be sure to avoid setting back to Read-Only at logout, add the following line to the file
/etc/bash.bash_logout
(may be you need to create it)
mount -o remount,ro / mount -o remount,ro /boot
If you want to have your bash history file saved and last known good date, also, put these lines instead
mount -o remount,rw / history -a fake-hwclock save mount -o remount,ro / mount -o remount,ro /boot
Of course you can also enhance the system like changing colors and/or set file system to read only after a certain amount of time. It’s just the way I’m using it.
PS : depending on your configuration, you may have set prompt elsewhere after /etc/bash.bashrc is executed (~/.bashrc for example), that would cause overriding the settings of /etc/bash.bashrc. So if it does not work, test by putting the lines at the end of the user ~/.bashrc profile file (/root/.bashrc for root)
Bonus : health check using Watchdog
If you follow my blog, you probably know that I’m using micro-controller day by day. I love the watchdog feature they have and reading original K3A post, I saw we can do the same on Raspberry, so I decided to give it a try of course.
It’s for advanced users, be sure knowing what you do if you don’t want your Pi going into reset loop, most users don’t need this.
Set system to Read-Write before executing these commands, you sure remember this ?
rw
Enable watchdog module :
modprobe bcm2708_wdog; apt-get install watchdog
Edit the file and
/etc/watchdog.conf
add the following lines at the end of the file
watchdog-device = /dev/watchdog max-load-15 = 25 watchdog-timeout = 10
On raspbian before jessie (old system init.d) set the watchdog to start at boot and start it now:
insserv watchdog; /etc/init.d/watchdog start
On Jessie edit the file /lib/systemd/system/watchdog.service and in section [Install] add the following
[Install] WantedBy=multi-user.target
Always on Jessie, enable it by
systemctl enable watchdog
In addition to the watchdog, you should set up reboot after a kernel panic. This is done by editing
/etc/sysctl.conf
. Add this line:
kernel.panic = 10
This will cause to wait 10 seconds after a kernel panic, then automatically safely reboot the box.
Last test and validation
As last operation, set back to Read-Only and reboot the Pi
ro reboot
References
Has I already said, I followed the original author (K3A) and just added some features I needed. Now that I have one Pi in this mode, I will update all my Pi and may be I need to add some modifications or enhancement not seen yet. If this the case I will update this post, so stay tuned.
If you want to see more precise explanations, I strongly suggest to read the excellent original K3A article located here
Thank’s to Raspbian France for providing this article image.
