Saturday, August 29, 2015

HOWTO - Make a Raspberry Pi truly read-only, reliable and trouble-free

RaspberryPi is a nice, small device which can be used for various automation purposes, internet of things or as an advanced camera. I have been using 5 RPis as a camera with motion detection and post-processing. This application requires high reliability and it was a pretty long journey until all the issues were solved. Here I am writing the most important things I have learned so you don’t need to go through the same stuff again (my cameras were placed on a remote location, making the whole process more complicated).

1. Connectivity

Of course, the first thing you need is proper connectivity, especially if you have your RPi on a remote location. This seemed easy – just use wifi dongle, set up wpa_supplicant and all done. No. There are many wifi dongles and most of them are very bad (poor signal quality). If you don’t have super-good signal on the site, definitely buy a dongle with external antenna! Next.. firmwares can be old, making frequent disconnections, or power saving enabled, see my response here It’s worth setting up a wifi checking script. This works for me – .

2. Remote shell

Related to the Connectivity. You need to be able to connect there remotely. I set up SSH tunnel (reverse port forwarding) because none of the RPis had a public IP address. Here is the script (/usr/local/bin/ Rc-script for it is here (/etc/init.d/sshtunnel) All RC scripts must be writable so make sure to do ‘chmod a+x script’ on them. It is just for an inspiration, you need to edit it for your setup. Basically after boot or wifi re-connection, RPi connects to my server and starts reverse-port-forwarding so I can access RPi from my server. RPi logs in to the server using a separate account and using SSH keys (so without a password) you can look up the relevant info online/google. I suggest you to set up SSH login from the RPi too so you can log to RPi from the server without password. That improves life a lot.

3. Bad SD card

To survive an unexpected poweroff, you need to have all filesystem mounted read-only (more about it later). But first… even if you have your FS read-only, a cheap SD card can destroy the data on the card itself. So I suggest you buy a more reliable SD card, ideally check

4. Read-only filesystem

This is one of the most complicated things but when done properly, it’s not too much work and it won’t cause any problems.
Unless you have special requirements, there are only a few paths which needs to be writable. So let’s go through all steps required to make a truly, trouble-free read-only RPi:

4.1 Update your RPi so you have the recent software

It’s a bad idea to set it up with an old software. That may complicate the update later…

4.2 Remove unnecessary services and files

Here is the command. Of course, remove only what you don’t need. But this is what you probably won’t need on a headless RPi. You definitely don’t want cron on your read-only RPi unless you have external hardware clock source, more about it later.
apt-get remove --purge wolfram-engine triggerhappy cron anacron logrotate dbus dphys-swapfile xserver-common lightdm fake-hwclock
insserv -r x11-common
apt-get autoremove --purge

4.3 Install buysbox syslog

You won’t need normal syslog text files on a read-only filesystem, either. Install busybox syslog instad. It logs into memory and is very lightweight. You can then use logread command to read syslog ringbuffer from the memory when needed.
apt-get install busybox-syslogd
dpkg --purge rsyslog

4.4 Disable filesystem check and swap

Because the filesystem will be mounted read-only, there is nothing to be corrupted so filesystem check must be disabled. I say MUST because it MUST. If you don’t have an external HW clock and use NTP time sync only and you do a change to the filesystem and reboot, filesystem check will see it as an update from the future, denying further boot, requiring manual action on the site. To disable filesystem checks, specifying ‘fastboot’ to the kernel cmdline should be enough. You also don’t want any swapfiles. You can disable them by specifying ‘noswap’ to the cmdline. So edit /boot/cmdline.txt and append the following two at the end of the line:
fastboot noswap

4.5 Set up clock sync.

Because you uninstalled fake-hwclock (it won’t be able to store clock on a readonly filesystem), you need to install and set up NTP sync. Also clock keeping is poor on a standard RPi so you may consider updating time regularly (every hour or two should be enough).
I used ntpdate for this. apt-get install ntpdate. I added it to /etc/rc.local:
/usr/sbin/ntpdate -b # change the ntp server according to your location

4.6 Update some writable paths

Now you need to update a few services which writes something. wpa_supplicant for WiFi is ok as it already writes to /tmp. DHCP lease is the major problem. Simple solution is to delete the old directory and make it as a symlink to tmp like this:
rm -rf /var/lib/dhcp/
ln -s /tmp /var/lib/dhcp

You can consider adding more symlinks from some /var subdirectories, especially run,spool and lock
rm -rf /var/run /var/spool /var/lock
ln -s /tmp /var/run 
ln -s /tmp /var/spool
ln -s /tmp /var/lock

4.7 Consider disabling some other startup scripts

insserv -r bootlogs
insserv -r sudo # if you plan to be root all the time
insserv -r alsa-utils # if you don't need alsa stuff (sound output)
insserv -r console-setup
insserv -r fake-hwclock # probably already removed at this point..
If you use alsamixer to set up volume level, make sure to do so in read-write filesystem. If won’t be able to store it on a readonly filesystem. It normally uses this path /var/lib/alsa/asound.state .

4.8 Tell the kernel to mount root filesystem read-only!

Finally getting there.. Add ” ro” at the end of your  /boot/cmdline.txt line.

4.9. Add “,ro” flag to both block devices in /etc/fstab

…to let the system know you want to mount them read-only:
proc              /proc           proc    defaults     0       0
/dev/mmcblk0p1    /boot           vfat    defaults,ro  0       2
/dev/mmcblk0p2    /               ext4    defaults,ro  0       1
tmpfs             /tmp            tmpfs   defaults     0       0

4. Watchdog

It is useful to set up a watchdog which can reboot your RPi in case something is unresponsive or eating CPU too much.
modprobe bcm2708_wdog #load BCM watchdog module
Add bcm2708_wdog into the /etc/modules so it gets loaded on boot.
apt-get install watchdog
Edit /etc/watchdog.conf:
Uncomment the line watchdog-device = /dev/watchdog
Uncomment the line with max-load-1
You can modify other parameters or find help online. Setting a minimum free RAM amount is a good idea. Before starting the watchdog, be prepared that you may have configured it wrongly and it will reboot immediately when you start it and may continuously reboot after each boot. So be prepared to modify your SD card on a different device if that happens.
Enable  the watchdog to start at boot and start it now:
insserv watchdog
/etc/init.d/watchdog start
During some modifications to your system (in read-write mode) later, you can consider disabling watchdog first. It rebooted my box once while I was doing some filesystem changes. Fortunately it booted fine for me, but it may not for you and may require manual, local fix.
In addition to the watchdog, you should set up reboot after a kernel panic. This is done by editing /etc/sysctl.conf. Add this line:
kernel.panic = 10
This will cause to wait 10 seconds after a kernel panic, then automatically safely reboot the box.

5. Add some cool utilities

It is helpful to receive syslog over the network. I think you can somehow enable sending busybox syslog over the network. I made my own script for this purpose, though. It reads output from ‘logread’ and sends it simply over the UDP to my server where it is saved to a logfile. Simple and helped a few times. Make sure to set up correct hostnames by editing /etc/hostname.
5.1 Cron?
Normal cron can’t be used unless you have a real external HW clock source. Because normally you can’t be sure that the clock got updated by NTP. If you can ensure it somehow, then fine, use cron. If not and using relative time is enough, you can make a fake cron using bash script, while loop and wait commands. Here is mine: (to be placed as /usr/local/ . I use this startup script for it (to be placed to /etc/fakecron)

6. Reboot!

Now it’s the best part. If you did everything correctly, it will boot just fine. If not, look at syslog and try to find out why. You can fix the SD card in a different computer.

7. What to do next

Enjoy your reliable RPi. Good work! If you ever want to update the software, just remount the root filesystem as read-write temporarily:
mount -o remount,rw /
You may want to stop watchdog temporarily. Now run your apt-get etc stuff, modify what you need.. then mount read-only again:
mount -o remount,ro /


I don’t know why and if it is still true, but raspicam leaked memory for me, causing RPi to reboot after some time. I managed to improve it by disabling preview (-n) but still I set up RPi to reboot daily just to be sure……


And that should be it. Hopefully it helped. If it all works, back up your SD card using dd (google it
If you have some tips, write them in comments. I can also update the article to include more info.
Please understand that I am often busy, so if you get stuck, please try google first. Consider asking at If it won’t help, write a comment here and I will try to help you if I can.



Sunday, August 16, 2015

HOWTO - Raspberry Pi HD IP Camera (based on UV4L - Video4Linux drivercollection)

For anyone searching for high quality video surveillance on a tinkering budget, will admit that the Raspberry Pi provides a reasonable and affordable option.
When you search the internet for IP Cameras capable of doing higher resolutions than 640x480 pixels, you will notice a remarkable difference in price range.  There is a price gap between the standard home consumer IP Camera and the more professional higher resolution IP Camera, so if you want high quality home surveillance video feed but don't want to bear the price, the Raspberry Pi IP Camera is for you.

All Raspberry Pi Camera modules (currently as of writing there are three different models available) are capable of doing high definition resolution (and i'm talking about 2592x1944px).  For people interested in the full specifications of the Camera, they can be found on the Raspberry Foundation web site at:

A bill of material for this guide looks as follows (prices are from adafruit's site):

A Raspberry Pi 2 board with power supply: 39,95$

A power supply for the Raspberry: 5,95$

A Raspberry Pi Spy Camera module: 39,95$

A dummy IP Camera Housing: 5$

This brings the total cost of our bill of material at 90,85$, not too bad for a capable high definition IP Camera.

First job would obviously be, mounting all the parts together, but that's a bit beyond the scope of this tutorial.  If you're a bit of a diy handyman, this shouldn't be an issue.
Once you have all parts mounted together, let's continue on the software side.

First step is downloading the Raspbian image from the Raspberry Foundation web site.
Once downloaded, unzip your image (if you downloaded the zip archive) and write it onto an sdcard with win32diskimager.

Next, let's continue booting up the image and doing some preparation work.
Once you boot the Foundation's Raspbian image for the first time, you'll end up in the initial configuration menu, make sure to enable the Camera module here, or your Camera module won't be able to start.

Now let's make sure that our Raspberry Pi is up to date with all the latest hotfixes, to update firmware and software, punch in below one-liner:
sudo apt-get update && sudo apt-get -y upgrade && echo y | sudo rpi-update && sudo reboot
Reboot your Raspberry Pi after this operation to properly activate the firmware update, if any.

Next thing is to load the UV4L2 suite of drivers onto our Raspberry Pi.
A clear set of instructions for this can be found on the linux projects website, but if you're lazy just like I am, you can just punch in the instructions below:

To install UV4L open a terminal and type the following command:
sudo curl | sudo apt-key add -

Add the following line to the file /etc/apt/sources.list :

echo "deb wheezy main" | sudo tee -a /etc/apt/sources.list

Update your apt cache:
sudo apt-get update

Now let's install our UV4L2 driver suite requirements:
sudo apt-get install uv4l uv4l-raspicam uv4l-raspicam-extras uv4l-server uv4l-webrtc

After installing the UV4L2 suite of drivers, these will be started automatically after reboot or u can start or restart them manually with below command:
sudo service uv4l_raspicam restart

Upon start the drivers will load their configuration from below configuration file, you can edit it by pasting the same
sudo nano /etc/uv4l/uv4l-raspicam.conf

To get an overview of available parameters which you can list in the configuration file:
uv4l --help --driver raspicam --driver-help

Or to kill a running driver: 
pkill uv4l

Once you have started the driver, a web gui will be available which u can use to configure parameters on the fly.

This concludes the initial set-up of the IP Camera with drivers and control panel.  
The camera will only be useful this way to be viewed via the control panel and a web browser.
If you want to connect it via VLC or a Video Recording Solution, like for example Surveillance Station on a Synology Nas, we need to install an extra server to run the RTSP protocol to serve the video feed.

There's a nice project called h264_v4l2_rtspserver over on Github written by a user named Michel Promonet which provide us with such a server, the only drawback is that you need to compile it yourself, but since you're a DIY handyman, follow below guide:

  • sudo apt-get -y install cmake libv4l-dev liblivemedia-dev liblog4 cpp5-dev
  • git clone
  • cd h264_v4l2_rtspserver/
  • cmake . && make
  • cpack .
  • sudo dpkg -i h264_v4l2_rtspserver*.deb

The RTSP server is now compiled and installed, the only thing what remains is to make it start at boot, for this purpose I wrote a little service script to start it via the normal debian services system.  

Copy and paste below script in a file, run this: sudo nano /etc/init.d/rtspserver then copy below into the file and save it.
#!/bin/sh### BEGIN INIT INFO
# Provides: RTSPSERVER
# Required-Start:    $local_fs $network $named $time $syslog
# Required-Stop:     $local_fs $network $named $time $syslog
# Default-Start:     2 3 4 
5# Default-Stop:      0 1 6
# Description:       Provides the Mpromoneth rtsp server
SCRIPT="/home/pi/h264_v4l2_rtspserver/h264_v4l2_rtspserver -P 8554 -Q 10 -r -s -W 2592 -H 1944 -F 15"

start() {  if [ -f $PIDFILE ] && kill -0 $(cat $PIDFILE); then    echo 'Service already running' >&2    return 1  fi  echo 'Starting service…' >&2  local CMD="$SCRIPT &> \"$LOGFILE\" & echo \$!"  su -c "$CMD" $RUNAS > "$PIDFILE"  echo 'Service started' >&2}

stop() {  if [ ! -f "$PIDFILE" ] || ! kill -0 $(cat "$PIDFILE"); then    echo 'Service not running' >&2    return 1  fi  echo 'Stopping service…' >&2  kill -15 $(cat "$PIDFILE") && rm -f "$PIDFILE"  echo 'Service stopped' >&2}

uninstall() {  echo -n "Are you really sure you want to uninstall this service? That cannot be undone. [yes|No] "  local SURE  read SURE  if [ "$SURE" = "yes" ]; then    stop    rm -f "$PIDFILE"    echo "Notice: log file was not removed: '$LOGFILE'" >&2    update-rc.d -f <NAME> remove    rm -fv "$0"  fi}

status() {        printf "%-50s" "Checking $NAME..."    if [ -f $PIDFILE ]; then        PID=$(cat $PIDFILE)            if [ -z "$(ps axf | grep ${PID} | grep -v grep)" ]; then                printf "%s\n" "The process appears to be dead but pidfile still exists"            else                echo "Running, the PID is $PID"            fi    else        printf "%s\n" "Service not running"    fi}

case "$1" in  start)    start    ;;  stop)    stop    ;;  status)    status    ;;  uninstall)    uninstall    ;;  restart)    stop    start    ;;  *)    echo "Usage: $0 {start|stop|status|restart|uninstall}"esac

Now change the permissions on this script so it can be executed:
sudo chmod 755 /etc/init.d/rtspserver
And schedule the script so it will start as a service on boot time:
sudo update-rc.d rtspserver defaults
After this reboot your Raspberry Pi to activate the RTSP server.
After the reboot the RTSP server will be available, and you will be able to connect to it with VLC via the following mrl:  rtsp://ip-address-of-your-Raspberry-Pi:8554/unicast

For the people who want to connect this to their Synology Nas, follow below instructions:

First make sure you have installed Surveillance Station on your nas via the webgui.
ssh to your nas and log in

We need to create a camera definition for the Raspberry Pi so we can select this in our Surveillance Station.
To do this create a configuration file for it (just copy and paste the whole damn thing):

cd /volume1/@appstore/SurveillanceStation/device_pack/camera_support/
echo [RaspberryPi*PiCam] | tee RaspberryPi.conf
echo api = rasbpicam-h264 | tee -a RaspberryPi.conf
echo channel_list = 1  | tee -a RaspberryPi.conf
echo default_channel = 1  | tee -a RaspberryPi.conf
echo resolutions_h264 = 2592x1944  | tee -a RaspberryPi.conf
echo default_resolution_h264 = 2592x1944  | tee -a RaspberryPi.conf
echo fps_h264_2592x1944 = 15  | tee -a RaspberryPi.conf
echo default_fps_h264_2592x1944 = 15  | tee -a RaspberryPi.conf
echo default_image_quality = 5  | tee -a RaspberryPi.conf
echo h264 = rtsp  | tee -a RaspberryPi.conf
After creating this configuration file, you will need to restart your Surveillance Station software on your nas to let it pick this configuration file up, do this on your web gui, in the package manager, and restart your Surveillance Station package (stop start).

Now you can start defining your camera in Surveillance Station.
Open Surveillance Station, go over to IP Camera, add Camera, do quick setup and define your Camera as per below screenshot:

Don't worry about the username and password, confirm to finish.

you will notice after adding the camera that it's disconnected, this is because the Surveillance Station software is looking at the root mrl (without the unicast string behind it).

It looks at:
and not at:
where it needs to look, hence the disconnection.

Currently I only have a work around for this, if someone has another option, please post a message on my blog below.

In Surveillance Station, select your Camera, then click Configuration, Export, give your export a name, and select a destination (in my case web folder) add your Raspberry Pi Camera and finish the export, now delete your camera from Surveillance Station.

Now edit a configuration file in the Camera Export folder which you have just exported:
vi /volume1/web/SSCamExport_RaspberryPi/.ExpCam
Change the follow 2 parameters from:

path = '/'
live_path = '/'
path = 'unicast'
live_path = 'unicast'

Save and re-import your Camera, it should be working now.
Note that, if you make configuration changes to your Camera in Surveillance Station, that u need to redo this procedure.

I hope this write up will be useful for people wanting to experiment with the Raspberry Pi Camera and Surveillance Station, if u like the write up, have any comments, suggestions, or improvements, leave me a message at the bottom of this blog.