Freed from storage failure by FreeNAS

Today I’ve been freed from storage failure by FreeNAS.

From time to time technology is a real pain in the backside.  We’re all more reliant on it than we’ve ever been before and when it goes wrong these days, generally it means you’ll end up suffering.  Storage failure is always the worst kind of technology issue because it generally means losing something that’s important to you, e.g. family photographs.

Today though in a first for me, storage failure reared its ugly head but was slapped down by FreeNAS.  Keeping a long story short, one dying drive + one next day delivery replacement drive + one set of FreeNAS wiki instructions = No need to restore from backups and NO LOSS OF DATA!

Now whilst some people might attribute all this to the components that make up FreeNAS (e.g. FreeBSD, ZFS) without the work done by the FreeNAS team to wrap these all up in a friendly, usable package, surviving a storage failure like this would have remained a pipe dream for me.

Thank you FreeNAS team!

FreeNAS Jail with Bittorrent Sync

Here are my instructions for creating a FreeNAS Jail with Bittorrent Sync (BTsync) running inside it. This uses no FreeNAS plugins whatsoever and takes full advantage of all the power that comes with having a FreeBSD core running at the heart of FreeNAS 9.1.  This has been done on FreeNAS-9.1.1-RELEASE-x64.

  1. Login to your FreeNAS systems web interface.
  2. Click: Jails
  3. Click: Add Jail
    0001
  4. For Jail Name use btsync and click ok.  Let FreeNAS do its thing.
    0002
  5. The jail will automatically start.  You now need to SSH into your FreeNAS box as root.  In the alternative SSH in as a regular user and then change to root with su.
  6. Run: jls
  7. Make a note of the number to the left of the btsync name used earlier.
  8. Run: jexec # csh
    [where # is the number from item 7. above]
  9. Run: mkdir /usr/local/btsync
  10. Run: cd /usr/local/btsync
  11. Run: fetch http://download-lb.utorrent.com/endpoint/btsync/os/FreeBSD-x64/track/stable
  12. Run: tar zxvf stable
  13. Run: rm stable
  14. Run: ./btsync –dump-sample-config > btsync.conf
  15. Run: pkg_add -rv nano
  16. Run: nano /usr/local/btsync/btsync.conf
  17. Change the following:
    1. Replace: “device_name”: “My Sync Device”, with a friendly name for your BTsync service, e.g. “device_name”: “Fred”,
    2. Replace: “storage_path” : “/home/user/.sync”, with “storage_path” : “/usr/local/btsync/.sync”,
    3. Replace: “password” : “password” with “password” : “XXXXXXXXXX” where XXXXXXXXXX is a password of your choosing.
  18. Close Nano and save the changes: Ctrl +X -> Y -> Enter
  19. Run: nano /etc/rc.d/btsync
  20. Insert the following:
    #!/bin/sh
    #
    # PROVIDE: btsync
    # REQUIRE: LOGIN DAEMON NETWORKING
    # KEYWORD: shutdown
    #
    # To enable BTSync, add this line to your /etc/rc.conf:
    #
    # btsync_enable="YES"
    #
    # And optionally these line:
    #
    # btsync_user="username" # Default is "root"
    # btsync_bin="/path/to/btsync" # Default is "/usr/local/sbin/btsync"
    
    . /etc/rc.subr
    
    name="btsync"
    rcvar="btsync_enable"
    
    load_rc_config $name
    
    required_files=$btsync_bin
    
    : ${btsync_enable="NO"}
    : ${btsync_user="root"}
    : ${btsync_bin="/usr/local/sbin/btsync"}
    : ${btsync_config="/usr/local/btsync/btsync.conf"}
    
    command=$btsync_bin
    command_args="--config ${btsync_config}"
    run_rc_command "$1"
  21. Close Nano and save the changes: Ctrl +X -> Y -> Enter
  22. Run:  chmod 555 /etc/rc.d/btsync
  23. Run: nano /etc/rc.conf
  24. Insert the following:
    # BitTorrent Sync
    btsync_enable="YES"
    btsync_user="nobody"
  25. Close Nano and save the changes: Ctrl +X -> Y -> Enter
  26. Run: chown -R nobody:nogroup /usr/local/btsync
  27. Run:  mv /usr/local/btsync/btsync /usr/local/sbin/
  28. Run: chown root:wheel /usr/local/sbin/btsync
  29. Login to your FreeNAS systems web interface.
  30. Click: Jails
  31. Stop the btsync jail and start it again.  Bittorrent Sync should now be running.
  32. Click: Edit Jail
  33. Make a note of IP.
  34. Login to your Bittorrent Sync web interface via: XXX.XXX.XXX.XXX:8888
    [Where XXX.XXX.XXX.XXX is the IP from Item 34]

That should about do it.

Some things to remember:

  • You can use the FreeNAS admin interface to mount storage into folders that exist outside of the jail into the jail.
  • Files and folders should all be set to nobody:nogroup as a user/group combination.  This killed off 99.9% of problems I was having.
  • Bittorrent Sync does not seem to update itself at the moment, at least it hasn’t thus far in this configuration.  This could be because it just doesn’t do it on FreeBSD or because I’ve done something stupidly wrong.  Either way you’ll have to watch the version numbers.

FreeNAS 9.1.0 Beta

Brilliant news, FreeNAS 9.1.0 Beta has been released

Will be a while before I get to play with this what with the wedding preparations, house move, new job, etc, etc. but that said I’m really looking forward to:

Improved Plugin Jail subsystem which supports multiple jails and an enhanced UI including enhancements from PC-BSD Warden.

Hopefully this will lead to some serious fun experimenting which will let me separate my current jail (which contains Serviio and other applications) into separate compartments that I can then turn on and off as necessary.

FreeNAS Jail with Transmission

Here are my instructions for creating a FreeNAS Jail with Transmission running inside it. This uses no FreeNAS plugins whatsoever and takes full advantage of all the power that comes with having a FreeBSD core running at the heart of FreeNAS 8.3.

Please note that if you already followed my guide for installing Serviio you can start from item 8.9. below.

  1. Install a plugin Jail as per the instructions available on the FreeNAS wiki here and start the Jail running.
  2. Create a user on your FreeNAS for yourself, e.g. MYUSER
  3. Enable SSH on your FreeNAS and SSH into your FreeNAS install using your client of choice.
  4. Execute the following commands:
    1. SU root
    2. jexec 1 csh
    3. mkdir /home
    4. adduser
  5. When prompted by adduser follow the instructions on-screen to create a user that matches the user you created in Step 2. Make sure that the Uid matches the user you created in Step 2. If you are not sure what the Uid number is then look it up in the FreeNAS control panel. Here’s the example:
    1. Username: MYUSER
      Full name: MYUSER
      Uid (Leave empty for default): 1111
      Login group [MYUSER]:
      Login group is MYUSER. Invite MYUSER into other groups? []: wheel
      Login class [default]:
      Shell (sh csh tcsh nologin) [sh]: csh
      Home directory [/home/MYUSER]:
      Home directory permissions (Leave empty for default):
      Use password-based authentication? [yes]:
      Use an empty password? (yes/no) [no]:
      Use a random password? (yes/no) [no]:
      Enter password:
      Enter password again:
      Lock out the account after creation? [no]:Username : MYUSER
      Password : *****
      Full Name : MYUSER
      Uid : 1111
      Class :
      Groups : MYUSER wheel
      Home : /home/MYUSER
      Home Mode :
      Shell : /bin/csh
      Locked : no
      OK? (yes/no): yes
      adduser: INFO: Successfully added (MYUSER) to the user database.
      Add another user? (yes/no): no
      Goodbye!
  6. Now we need to leave the jail and create a bridge between the FreeNAS internals and the jail. So logout of your SSH client and then turn off the jail in FreeNAS control panel. Then navigate to Services -> Plugins -> Management -> Mount Points and add a new mount point. Map a directory you can access regularly through FreeNAS as the directory /home/MYUSER within the jail. Turn the jail back on and restart your FreeNAS.
  7. SSH into your FreeNAS install using your client of choice.
  8. Execute the following commands [NB: This section will take some time to complete so set aside half an hour]:
    1. SU root
    2. jexec 1 csh
    3. portsnap fetch extract update
    4. cd /usr/ports/ports-mgmt/portmaster/ && make install clean
      Be sure to use the appropriate options for your system. If you’re not sure, check all the boxes.
      Portmaster
    5. echo ‘WITH_PKGNG=yes’ >> /etc/make.conf
    6. rehash
    7. pkg2ng
    8. portmaster -G editors/nano
    9. rehash
    10. cd /usr/ports/ftp/curl
      1. Set the options so that they match the screenshot below:
      2. curl
    11. portmaster -G net-p2p/transmission-daemon
    12. mkdir /home/MYUSER/transmission
    13. mkdir /home/MYUSER/transmission/watch
    14. mkdir /home/MYUSER/transmission/complete
    15. nano /etc/rc.conf and add lines:
      transmission_enable=”YES”
      transmission_watch_dir=”/home/MYUSER/transmission/watch”
      transmission_conf_dir=”/home/MYUSER/transmission”
      transmission_download_dir=”/home/MYUSER/transmission/complete”
      transmission_user=”MYUSER”
    16. rehash
    17. service transmission start
  9. You’re done. Transmission will now start-up on every boot. Login to it remotely and set up the directories you need as necessary!

FreeNAS Jail with Serviio 1.1

Here are my instructions for creating a FreeNAS Jail with Serviio 1.1 running inside it. This uses no FreeNAS plugins whatsoever and takes full advantage of all the power that comes with having a FreeBSD core running at the heart of FreeNAS 8.3.

  1. Install a plugin Jail as per the instructions available on the FreeNAS wiki here and start the Jail running.
  2. Create a user on your FreeNAS for yourself, e.g. MYUSER
  3. Enable SSH on your FreeNAS and SSH into your FreeNAS install using your client of choice.
  4. Execute the following commands:
    1. SU root
    2. jexec 1 csh
    3. mkdir /home
    4. adduser
  5. When prompted by adduser follow the instructions on-screen to create a user that matches the user you created in Step 2. Make sure that the Uid matches the user you created in Step 2. If you are not sure what the Uid number is then look it up in the FreeNAS control panel. Here’s the example:
    1. Username: MYUSER
      Full name: MYUSER
      Uid (Leave empty for default): 1111
      Login group [MYUSER]:
      Login group is MYUSER. Invite MYUSER into other groups? []: wheel
      Login class [default]:
      Shell (sh csh tcsh nologin) [sh]: csh
      Home directory [/home/MYUSER]:
      Home directory permissions (Leave empty for default):
      Use password-based authentication? [yes]:
      Use an empty password? (yes/no) [no]:
      Use a random password? (yes/no) [no]:
      Enter password:
      Enter password again:
      Lock out the account after creation? [no]:Username : MYUSER
      Password : *****
      Full Name : MYUSER
      Uid : 1111
      Class :
      Groups : MYUSER wheel
      Home : /home/MYUSER
      Home Mode :
      Shell : /bin/csh
      Locked : no
      OK? (yes/no): yes
      adduser: INFO: Successfully added (MYUSER) to the user database.
      Add another user? (yes/no): no
      Goodbye!
  6. Now we need to leave the jail and create a bridge between the FreeNAS internals and the jail. So logout of your SSH client and then turn off the jail in FreeNAS control panel. Then navigate to Services -> Plugins -> Management -> Mount Points and add a new mount point. Map a directory you can access regularly through FreeNAS as the directory /home/MYUSER within the jail. Turn the jail back on and restart your FreeNAS.
  7. SSH into your FreeNAS install using your client of choice.
  8. Execute the following commands [NB: This section will take some time to complete so set aside a couple of hours]:
    1. SU root
    2. jexec 1 csh
    3. portsnap fetch extract update
      1. Update 02 February 2013
      2. To ensure that you have the new pkg program installed run: pkg -v
      3. If you don’t get a response of 1.0.7 or higher then run: cd /usr/ports/ports-mgmt/pkg/ && make install clean
    4. cd /usr/ports/ports-mgmt/portmaster/ && make install clean
      Be sure to use the appropriate options for your system. If you’re not sure, check all the boxes.
      Portmaster
    5. echo ‘WITH_PKGNG=yes’ >> /etc/make.conf
    6. rehash
    7. pkg2ng
    8. portmaster -G editors/nano
    9. rehash
    10. cd /usr/ports/multimedia/ffmpeg1
    11. make config
      1. Set the options so that they match the information shown in the Serviio wiki for building ffmpeg. This is complicated so follow the screenshots below:
      2. ffmpeg1-page1
      3. ffmpeg1-page2
      4. ffmpeg1-page3
      5. press ok
    12. cd /usr/ports/graphics/frei0r/
    13. make config
    14. check Enable MMX CPU instructions then press ok
    15. portmaster -G multimedia/ffmpeg1
    16. portmaster -G net/serviio
    17. nano /etc/hosts and add line [replacing Jailed with the name of your jail from the FreeNAS control panel]:
      127.0.0.1 Jailed
    18. nano /etc/rc.conf and add lines:
      serviio_enable=”YES”
      serviio_args=”-Dserviio.remoteHost=192.168.0.100″
    19. service serviio start
  9. You’re done. Serviio will now start-up on every boot. Login to it remotely and set up the directories you need as necessary! I heartily recommend using ServiiDroid for Android which is available here.

Failing with FreeNAS

Yesterday my lovely HP N36L Microserver that I run FreeNAS on decided that it was going to trash the USB memory stick that FreeNAS relies upon to boot.

I’m not sure what the cause is at the moment, whether it is FreeNAS having a funny half hour, the Microserver amusing itself at my expense or possibly my doing something wrong with a console command (though I don’t believe the last one is likely as I have not used it for a month).

When the error messages appeared last night I have to admit they sent me into something of a complete and utter state of panic.  Finding information on FreeNAS isn’t too difficult but I don’t find it as easy to get information on FreeBSD error messages (FreeBSD being the base for FreeNAS).

Thankfully after some logical thinking (it was complaining about drives so I removed all of them) I concluded that the USB stick itself was the cause of all the problems.  With a bit of command prompt tinkering I was able to make FreeNAS boot to a console with the drives in and confirm that it could mount the ZFS file system without issue.

Since it seems that the USB stick is beyond saving (baring some miracle), I shall be re-doing my FreeNAS build at some point this week.  If I get the time then I will take loads of sexy screenshots and write the process up as at the moment I think the Plugins available for FreeNAS are absolutely horrid compared to the result you can have with a bit of DIY.

This time I shall attempt to correct issues that I’ve had previously and add in some more functionality:

  • Use the pre-release version of FreeNAS (now FreeNAS 8.3) in the hopes this enables me to use the most current FreeBSD ports tree.
  • Install and use Portmaster so I have a tool to make sure there is better control over the ports installation process (until I gain a better understanding).
  • Programs running in the Plugins jail need to stop being run as privileged users and shall be run as regular users.
  • Try to build Hamachi even though it is a long shot because it would remove my need to use a Dynamic DNS service.
  • Introduce the Privoxy proxy server and integrate it with TOR.
  • Figure out rsync so that family businesses can backup to yet another site using multiple protocols.
  • Tinker with the init scripts so that they don’t launch network software before there is a networking connection, etc.

Hopefully I will get through some, if not all, of that lot by the end of the week.

Breakages

Does breaking things count as progress?

How about multiple breakages?

So far I’ve managed to compile FreeNAS with varying degrees of success and failure. Either I’m not getting the commands right half the time or I’ve tinkered with something I shouldn’t have and reduced the reliability.

Bugger!

FreeNAS, FreeBSD and fucking FFmpeg

Hitting a brick wall when you’re close to getting something finished is one of those things in life that you hope will never happen but generally tends to.  It’s exactly what happened to me yesterday in my quest to make a package for Serviio that I could just install on FreeNAS.

Creating the FreeBSD build environment on my PC seemed as though it would be simple enough, throw up VirtualBox, throw FreeBSD at it and hopefully job done.  In actuality it was even easier than I’d expected when I followed the guide that I found on the FreeNAS forums the other day (see my earlier post).

Problem was that I know absolutely nothing about FreeBSD in general and that after many hours of bashing my head against building FFmpeg it all came down to one simple thing.  I should have rebooted when it all started going wrong!

Live and learn!

That’s LAME

Well, whilst on my mission to make Serviio and FreeNAS live in perfect harmony as simply as possible, I’ve just stumbled upon the fact that LAME has been updated to version 3.99.2.

Does anyone have a summary of the differences / new features  in version 3.99 as the changelog is absolutely enormous and it made my head hurt when I started trying to read it.

Fiddling with FreeNAS

A few months ago I purchased a HP Microserver because it had become necessary to improve my storage arrangements at home.  Before then I had stored things, well actually everything, on a ReadyNAS Duo that had a been running on a pair of 500GB that mirrored each other for redundancy.  This was not an ideal situation and because I was running out of space I had considered removing the mirror.

Switch to something like the HP Microserver seemed ideal because:

  1. I could run whatever I wanted on it because it was effectively a PC in its own right.
  2. I would have 8TB of storage which was a vast improvement over the ReadyNAS.
  3. 4 drives would give me improved redundancy and there would be various configurations that I could choose to suit my needs.

Skipping forward to today I have had my Microserver for a good few months and have run it using FreeNAS but now I’m getting that nerd itch because it’s not running quite exactly how I want it.

Currently I have it so that it’s running FreeNAS and Serviio on top of that to serve my music up to my PS3, etc. but the current method of having to dig into the console to get it all working again every time FreeNAS releases an update is not sitting well with me.

With this in mind I’ve just decided to follow this guide over on the FreeNAS forums and set up a build environment which with any luck I will be able to use to slipstream Serviio into FreeNAS.

Cross your fingers!