Saturday, August 29, 2015

HOWTO - Make a Raspberry Pi truly read-only, reliable and trouble-free


RaspberryPi is a nice, small device which can be used for various automation purposes, internet of things or as an advanced camera. I have been using 5 RPis as a camera with motion detection and post-processing. This application requires high reliability and it was a pretty long journey until all the issues were solved. Here I am writing the most important things I have learned so you don’t need to go through the same stuff again (my cameras were placed on a remote location, making the whole process more complicated).

1. Connectivity

Of course, the first thing you need is proper connectivity, especially if you have your RPi on a remote location. This seemed easy – just use wifi dongle, set up wpa_supplicant and all done. No. There are many wifi dongles and most of them are very bad (poor signal quality). If you don’t have super-good signal on the site, definitely buy a dongle with external antenna! Next.. firmwares can be old, making frequent disconnections, or power saving enabled, see my response here http://raspberrypi.stackexchange.com/a/17179. It’s worth setting up a wifi checking script. This works for me – https://paste.k3a.me/view/449a6f1c .

2. Remote shell

Related to the Connectivity. You need to be able to connect there remotely. I set up SSH tunnel (reverse port forwarding) because none of the RPis had a public IP address. Here is the script (/usr/local/bin/sshtunnel.sh) https://paste.k3a.me/view/a7e07bb1. Rc-script for it is here (/etc/init.d/sshtunnel) https://paste.k3a.me/view/7d74f0ae. All RC scripts must be writable so make sure to do ‘chmod a+x script’ on them. It is just for an inspiration, you need to edit it for your setup. Basically after boot or wifi re-connection, RPi connects to my server and starts reverse-port-forwarding so I can access RPi from my server. RPi logs in to the server using a separate account and using SSH keys (so without a password) you can look up the relevant info online/google. I suggest you to set up SSH login from the RPi too so you can log to RPi from the server without password. That improves life a lot.

3. Bad SD card

To survive an unexpected poweroff, you need to have all filesystem mounted read-only (more about it later). But first… even if you have your FS read-only, a cheap SD card can destroy the data on the card itself. So I suggest you buy a more reliable SD card, ideally check http://elinux.org/RPi_SD_cards

4. Read-only filesystem

This is one of the most complicated things but when done properly, it’s not too much work and it won’t cause any problems.
Unless you have special requirements, there are only a few paths which needs to be writable. So let’s go through all steps required to make a truly, trouble-free read-only RPi:

4.1 Update your RPi so you have the recent software

It’s a bad idea to set it up with an old software. That may complicate the update later…

4.2 Remove unnecessary services and files

Here is the command. Of course, remove only what you don’t need. But this is what you probably won’t need on a headless RPi. You definitely don’t want cron on your read-only RPi unless you have external hardware clock source, more about it later.
apt-get remove --purge wolfram-engine triggerhappy cron anacron logrotate dbus dphys-swapfile xserver-common lightdm fake-hwclock
insserv -r x11-common
apt-get autoremove --purge

4.3 Install buysbox syslog

You won’t need normal syslog text files on a read-only filesystem, either. Install busybox syslog instad. It logs into memory and is very lightweight. You can then use logread command to read syslog ringbuffer from the memory when needed.
apt-get install busybox-syslogd
dpkg --purge rsyslog

4.4 Disable filesystem check and swap

Because the filesystem will be mounted read-only, there is nothing to be corrupted so filesystem check must be disabled. I say MUST because it MUST. If you don’t have an external HW clock and use NTP time sync only and you do a change to the filesystem and reboot, filesystem check will see it as an update from the future, denying further boot, requiring manual action on the site. To disable filesystem checks, specifying ‘fastboot’ to the kernel cmdline should be enough. You also don’t want any swapfiles. You can disable them by specifying ‘noswap’ to the cmdline. So edit /boot/cmdline.txt and append the following two at the end of the line:
fastboot noswap

4.5 Set up clock sync.

Because you uninstalled fake-hwclock (it won’t be able to store clock on a readonly filesystem), you need to install and set up NTP sync. Also clock keeping is poor on a standard RPi so you may consider updating time regularly (every hour or two should be enough).
I used ntpdate for this. apt-get install ntpdate. I added it to /etc/rc.local:
/usr/sbin/ntpdate -b cz.pool.ntp.org # change the ntp server according to your location

4.6 Update some writable paths

Now you need to update a few services which writes something. wpa_supplicant for WiFi is ok as it already writes to /tmp. DHCP lease is the major problem. Simple solution is to delete the old directory and make it as a symlink to tmp like this:
rm -rf /var/lib/dhcp/
ln -s /tmp /var/lib/dhcp

You can consider adding more symlinks from some /var subdirectories, especially run,spool and lock
rm -rf /var/run /var/spool /var/lock
ln -s /tmp /var/run 
ln -s /tmp /var/spool
ln -s /tmp /var/lock

4.7 Consider disabling some other startup scripts

insserv -r bootlogs
insserv -r sudo # if you plan to be root all the time
insserv -r alsa-utils # if you don't need alsa stuff (sound output)
insserv -r console-setup
insserv -r fake-hwclock # probably already removed at this point..
If you use alsamixer to set up volume level, make sure to do so in read-write filesystem. If won’t be able to store it on a readonly filesystem. It normally uses this path /var/lib/alsa/asound.state .

4.8 Tell the kernel to mount root filesystem read-only!

Finally getting there.. Add ” ro” at the end of your  /boot/cmdline.txt line.

4.9. Add “,ro” flag to both block devices in /etc/fstab

…to let the system know you want to mount them read-only:
proc              /proc           proc    defaults     0       0
/dev/mmcblk0p1    /boot           vfat    defaults,ro  0       2
/dev/mmcblk0p2    /               ext4    defaults,ro  0       1
tmpfs             /tmp            tmpfs   defaults     0       0

4. Watchdog

It is useful to set up a watchdog which can reboot your RPi in case something is unresponsive or eating CPU too much.
modprobe bcm2708_wdog #load BCM watchdog module
Add bcm2708_wdog into the /etc/modules so it gets loaded on boot.
apt-get install watchdog
Edit /etc/watchdog.conf:
Uncomment the line watchdog-device = /dev/watchdog
Uncomment the line with max-load-1
You can modify other parameters or find help online. Setting a minimum free RAM amount is a good idea. Before starting the watchdog, be prepared that you may have configured it wrongly and it will reboot immediately when you start it and may continuously reboot after each boot. So be prepared to modify your SD card on a different device if that happens.
Enable  the watchdog to start at boot and start it now:
insserv watchdog
/etc/init.d/watchdog start
During some modifications to your system (in read-write mode) later, you can consider disabling watchdog first. It rebooted my box once while I was doing some filesystem changes. Fortunately it booted fine for me, but it may not for you and may require manual, local fix.
In addition to the watchdog, you should set up reboot after a kernel panic. This is done by editing /etc/sysctl.conf. Add this line:
kernel.panic = 10
This will cause to wait 10 seconds after a kernel panic, then automatically safely reboot the box.

5. Add some cool utilities

It is helpful to receive syslog over the network. I think you can somehow enable sending busybox syslog over the network. I made my own script for this purpose, though. It reads output from ‘logread’ and sends it simply over the UDP to my server where it is saved to a logfile. Simple and helped a few times. Make sure to set up correct hostnames by editing /etc/hostname.
5.1 Cron?
Normal cron can’t be used unless you have a real external HW clock source. Because normally you can’t be sure that the clock got updated by NTP. If you can ensure it somehow, then fine, use cron. If not and using relative time is enough, you can make a fake cron using bash script, while loop and wait commands. Here is mine: (to be placed as /usr/local/fakecron.sh) https://paste.k3a.me/view/4515b0a4 . I use this startup script for it (to be placed to /etc/fakecron) https://paste.k3a.me/view/0a995efd.

6. Reboot!

Now it’s the best part. If you did everything correctly, it will boot just fine. If not, look at syslog and try to find out why. You can fix the SD card in a different computer.

7. What to do next

Enjoy your reliable RPi. Good work! If you ever want to update the software, just remount the root filesystem as read-write temporarily:
mount -o remount,rw /
You may want to stop watchdog temporarily. Now run your apt-get etc stuff, modify what you need.. then mount read-only again:
mount -o remount,ro /

Camera

I don’t know why and if it is still true, but raspicam leaked memory for me, causing RPi to reboot after some time. I managed to improve it by disabling preview (-n) but still I set up RPi to reboot daily just to be sure……

Done

And that should be it. Hopefully it helped. If it all works, back up your SD card using dd (google it https://www.google.cz/search?q=backup+using+dd).
If you have some tips, write them in comments. I can also update the article to include more info.
Please understand that I am often busy, so if you get stuck, please try google first. Consider asking at http://raspberrypi.stackexchange.com. If it won’t help, write a comment here and I will try to help you if I can.

Sources

  • http://blog.pi3g.com/2014/04/make-raspbian-system-read-only/
  • http://blog.gegg.us/2014/03/a-raspbian-read-only-root-fs-howto/

No comments:

Post a Comment