Ping Ubuntu hostname from Windows (winbind+samba)

Here’s a procedure I found on taesch.com to ping a Ubuntu machine by the hostname from a Windows PC:

  1. sudo vi /etc/nsswitch.conf
    hosts: files wins dns
  2. sudo aptitude install winbind
  3. sudo aptitude install samba

SOURCE: http://taesch.com/my-digital-life/cant-ping-ubuntu-hostname-from-windows-xp-setup-winbind

Acer EasyStore H340, the perfect FreeNAS box?

The Acer EasyStore H340 comes with 1 x 1TB HDD and room for three more drives (SATA-II). A low power Atom 230, 2 GB of Ram and an internal 256M flash storage to install FreeNAS 64 bits. This comes in a nice 7.9″ (200mm) W x 7.1″ (180mm) D x 8.3″ (212mm) H casing.

Easystore

I am curious to see if the Atom 230 is enough horsepower for ZFS, anybody tried it?

SOURCE: HappyBison.com

FreeNAS CIFS/Samba changing Guest Account ‘FTP’ not working

Using FreeNAS 0.7 Khasadar (revision 4919) with CIFS/SMB Authentication Anonymous, I changed the Guest account for but it is not actually applying it to the configuration, it is a trivial bug and here’s a small dirty workaround.

1 – Create an excutable file containing this:

#!/bin/sh

# find and replace 'ftp' by 'transmission'
sed -i. 's/ftp/transmission/g' /etc/rc.d/samba

# restart samba to use the new config
/etc/rc.d/samba restart

2 – Go in System|Advanced|Command scripts and add an entry for your script in “PostInit”

For those who are interested, I am doing this because my network is not accessible from the outside and I didn’t want to bother with user privileges. I am using my FreeNAS box to share media files downloaded using Transmission and I wanted everybody to be able to access/modify/delete them, so I change the guest account to be “transmission” :)

A PHP script to start a torrent and download it where the .torrent was located using Transmission JSON-RPC

I wrote this script because I never found a Transmission remote GUI that permits to start downloading a torrent in a specific directory.

So, here’s the script, I hope you’ll like it!

#!/usr/local/bin/php -q
<?php
/*
VERSION: 0.9
AUTHOR: gimpe
EMAIL: gimpe [at] hype-o-thetic.com
WEBSITE: http://hype-o-thetic.com

USAGE: You simply heed to update the "configuration constants" with your own
       information and run the script manually or schedule it to be run as a
       cronjob (a.g. every 30 minutes).

DESCRIPTION: This script will recursively search into directory to find .torrent
             files. It will automatically start the download inside the
             directory where the .torrrent was located. And finally, it will
             delete the .torrent file (if there was no error returned by
             Transmission). This script has been tested under FreeNAS 0.7RC2
             Khasadar (revision 4899) but it should work on anything :) FreeNAS
             or not as it is using the Transmission JSON-RPC interface with
             sockets (curl is NOT needed as FreeNAS embedded doesn't include it
             with PHP).

CHANGELOG
 0.9: 2009-11-07 Beta version
 0.2: 2009-09-13 Code clean up
 0.1: 2009-09-11 Make it work

REFERENCES:
 Transmission JSON-RPC doc: http://trac.transmissionbt.com/browser/trunk/doc/rpc-spec.txt
*/

// configuration constants
define('SEEDRATIOLIMIT', 1.5);      // seed the torrent until this ratio is reached
define('USERNAME', 'username');     // FreeNAS BitTorrent Administrative WebGUI username
define('PASSWORD', 'password');     // FreeNAS BitTorrent Administrative WebGUI password
define('DIR', '/mnt/media/video');  // directory to scan recursively for .torrent (no trailling slash)

// send an empty request to retrieve the current X-Transmission-Session-Id
$response = transmissionJsonrpcClient(array());

// check if we have the X-Transmission-Session-Id
if (!strlen($response['xTransmissionSessionId']))
{
    print 'Unable to get X-Transmission-Session-Id' . PHP_EOL;
    exit;
}

// start dir scan to start torrents downloads
searchTorrentInDir(DIR, $response['xTransmissionSessionId']);

////////////////////////////////////////////////////////////////////////////////
//////////////////////////////////FUNCTIONS/////////////////////////////////////
////////////////////////////////////////////////////////////////////////////////

// send a request JSON-RPC request to transmission
function transmissionJsonrpcClient($request, $xTransmissionSessionId = '')
{
    // encode the request in json
    $content = json_encode($request);

    $fsockErrNo      = 0;
    $fsockErrStr     = '';
    $fsockTimeout    = 10;
    $response   = '';

    if ($fp = @fsockopen('localhost', 9091, $fsockErrNo, $fsockErrStr, $fsockTimeout))
    {
        $rawRequest = 'POST /transmission/rpc HTTP/1.0' . PHP_EOL
                      . 'Content-type: text/json;charset=utf-8' . PHP_EOL
                      . 'X-Transmission-Session-Id: ' . $xTransmissionSessionId . PHP_EOL
                      . 'Authorization: Basic ' . base64_encode(USERNAME . ':' . PASSWORD) . PHP_EOL
                      . 'Content-Length: ' . strlen($content) . PHP_EOL . PHP_EOL
                      . $content;
        fwrite($fp, $rawRequest);
        $response = stream_get_contents($fp);
        fclose($fp);

        // if "409 conflict" + empty request, this is a dummy request to retrieve the X-Transmission-Session-Id
        if (strstr($response, '409: Conflict') && !count($request))
        {
            // find the X-Transmission-Session-Id in the response
            if (preg_match('/X-Transmission-Session-Id: (\w+)/', $response, $matches))
            {
                $xTransmissionSessionId = $matches[1];
            }
            else
            {
                print 'ERROR no X-Transmission-Session-Id' . PHP_EOL;
                exit;
            }
        }
    }
    else
    {
        print 'ERROR cant\'t connect to transmission JSON-RPC Server: ' . $fsockErrStr . PHP_EOL;
        exit;
    }

    $return = '';
    if (count($request))
    {
        if (preg_match("/(\{.*)/msi", $response, $matches))
        {
            $return = json_decode($matches[0], true);
        }
        else
        {
            print 'ERROR unable to retrieve the transmission JSON response in the raw socket response: ' . print_r($response, true) . PHP_EOL;
        }
    }

    return array('response' => $return, 'xTransmissionSessionId' => $xTransmissionSessionId);
}

// recursively scan a directory to find .torrent files and start the torrent download
function searchTorrentInDir($dir, $xTransmissionSessionId)
{
    $dirContent = scandir($dir);

    foreach($dirContent as $key => $file)
    {
        $path = $dir . '/' . $file;

        if (basename($path) == '.' || basename($path) == '..')
        {
            continue;
        }

        // if content is a readable file
        if (is_file($path) && is_readable($path))
        {
            $pathinfo = pathinfo($path);

            // if it is *.torrent
            if (in_array($pathinfo['extension'], array('torrent')))
            {
                $filename  = $pathinfo['filename'];
                $extension = $pathinfo['extension'];
                $dirname   = $pathinfo['dirname'];

                // get .torrent meta info
                $torrrentMetainfo = file_get_contents($path);

                $request = array(
                                    'arguments' => array(
                                                                'download-dir' => $dirname
                                                              , 'metainfo' => base64_encode($torrrentMetainfo)
                                                          )
                                  , 'method'    => 'torrent-add'
                                  , 'tag'       => ''
                );
                $response = transmissionJsonrpcClient($request, $xTransmissionSessionId);

                // if success
                if ($response['response']['result'] === 'success')
                {
                    print "added \"$filename\" to download in \"$dirname\"" . PHP_EOL;

                    // set ratio
                    $request = array(
                                          'arguments' => array(
                                                                    'id'               => $response['response']['arguments']['torrent-added']['id']
                                                                  , 'seedRatioLimit'   => SEEDRATIOLIMIT
                                                                  , 'seedRatioLimited' => TRUE
                                                              )
                                        , 'method'    => 'session-set'
                    );
                    $response = transmissionJsonrpcClient($request, $xTransmissionSessionId);

                    // delete .torrent file
                    unlink($path);
                }
                else
                {
                    print "ERROR adding file \"$filename\": ". print_r($response, true) . PHP_EOL;
                }
            }
        }
        else if (is_dir($path) && is_readable($path))
        {
            // Recursive call
            searchTorrentInDir($path, $xTransmissionSessionId);
        }
    }

    return;
}
?>

FreeNAS: Services|UPS –> what to use as “Port”

I’m writing a note to myself as I don’t have a UPS yet :)

The Port value can be set to “auto” as metionned here: http://jonathanbrown.me/my-freenas-project-part-5-the-config

Found another rerefence that confirms this:
- For newhidups: “auto” or whatever value as newhidups automagically probe for USB UPS.

SOURCE: http://people.freebsd.org/~thierry/nut_FreeBSD_HowTo.txt

Diskless HTPC using DD-WRT (PXE), Ubuntu/XBMC and FreeNAS (TFTP, NFS)

Given that I spent time and money on my FeeNAS node, I want to maximize its usage, so when a friend told me that I can use PXE and NFS to boot a diskless computer from the network, I thought it was the perfect opportunity to try somehing new and activate some more services under FreeNAS!

Here’s my hardware:

  • A dd-wrt router (model WRT-310N)
  • A FreeNAS box named “stark”, using ip 192.168.1.5 (you need a static address)
  • A HTPC computer named that can boot on the network using a PXE client (with a NVIDIA video card to take full advantages of VDPAU)

DD-WRT & DHCP/PXE

http://dd-wrt.com/wiki/index.php/PXE

Verify that the DD-WRT router is the DHCP Server on your network (Setup –> Basic Setup and set DHCP Type to “DHCP Server”) and make sure that the DNSMasq service is running (Services –> Services and check “Enable”).

Now, you need to enter the following parameters in the “Additional DNSMasq Options” text box:
dhcp-boot=pxelinux.0,stark,192.168.1.5

When the HTPC is booting it will go to the router and will be redirected to your FreeNAS TFTP server to download and boot the Linux kernel.

FeeNAS & TFTP

Download the file “pxelinux.0″ for your Ubuntu version and your PC architeture, I am using Ubuntu 9.04/amd64:
http://archive.ubuntu.com/ubuntu/dists/jaunty/main/installer-amd64/current/images/netboot/pxelinux.0

Put this file on your FreeNAS box in a directory like /mnt/data/pxe. This will be the directory used by the TFTP service. To configure it, go in FreeNAS (Services –> TFTP) and enter this under “Directory”: /mnt/data/pxe/

Now from a shell prompt, create a default PXE configuration:

cd /mnt/data/pxe
mkdir pxelinux.cfg
vi pxelinux.cfg/default

Here’s the content of my file:

LABEL linux
KERNEL vmlinuz
APPEND root=/dev/nfs initrd=initrd.img nfsroot=192.168.1.5:/mnt/data/nfsroot ip=dhcp rw

You can already try to boot your HTPC from the network, it should load the pxelinux.0 file and find the “default” configuration and after that you will see an error about the missing vmlinuz file. At least, you will know that the first steps are working.

FeeNAS & NFS

In FreeNAS, activate the NFS service (Services –> NFS –> “Enable”)

In the “Shares” tab add a new share, I use “/mnt/data/nfsroot/”

Set “Map all users to root” to “No”.

NOTE: NFSv3 uses the UID to handle the file permissions, the easiest way is to create the user on your Ubuntu/XMBC respecting the FreeNAS users UIDs (on FreeNAS the UID start at 1001 but on Ubuntu it starts at 1000).

My first FreeNAS user is gimpe (1001) and the second one is xbmc (1002), I had to update the UIDs on my Ubuntu HTPC to match that, I’ll show you how later.

XBMC

We will need a working Ubuntu/XBMC installation and to it to your NFS server. You can either use an existing installation or make a new install of Ubuntu/XMBC, you can follow the official guide: http://xbmc.org/wiki/?title=XBMCbuntu. I preferred to go with a fresh install on a 2GB usb drive.

Once everything is working, you will need install nfs-common and portmap to connect to the NFS server and transfer the files: aptitude install nfs-commong portmap

http://blog.crox.net/archives/29-Diskless-Ubuntu-Feisty-Fawn-NFS-install.html

On the HTPC, copy the file to the NFS server:

sudo su
mkdir /mnt/tmp
mkdir /mnt/nfs
mount /dev/sda1 /mnt/tmp
mount 192.168.1.5:/mnt/data/nfsroot /mnt/nfs
cp -a /mnt/tmp/* /mnt/nfs/

Change the root directory to be the NFS remote directory:

chroot /mnt/nfs
mount -t proc proc proc

Edit some files on the NFS server:

  1. Change the initramfs to “NFS”:
    vi etc/initramfs-tools/initramfs.conf

    And change BOOT=local to BOOT=nfs and update the initrd.img by execution this command:

    update-iniramfs -u
  2. Change the network interface from iface eth0 inet dhcp to iface eth0 inet manual
    vi etc/network/interafce
  3. Change the root / for nfs:
    vi etc/fstab

    and replace the “/” line by this one:

    /dev/nfs       /               nfs    defaults          1       1

To finish, go on your FreeNAS server and copy /mnt/data/nfsroot/vmlinuz and /mnt/data/nfsroot/initram.img to /mnt/data/pxe. This is the Linux kernel that will dowloaded by the HTPC and that will be used to boot.

Fixing UIDs for NFS mapping

BECAREFUL this can mess up your Ubuntu/XBMC installation, use at you own risk!

Update /etc/passwd: find your USER and update the UID to match the one on FreeNAS

(OPTIONAL) Update /etc/groups: Find your USER and update the GID

Update exisint user home with the new UID:

chown -R . /home/

Fixing XMBC start up

Now you should be able to start your HTPC from the network but XBMC should stop starting automatically, I think it is related to the fact that we usee PXE instead of GRUB to start Ubuntu/XBMC. I didn’t had time try GRUB in “diskless mode” so for now here’s a workaroud (inspired by http://www.xbmc.org/forum/showthread.php?t=38804):

Install Fluxbox and xterm

aptitude install fluxbox xterm

Make xbmc user to auto-login
vi /mnt/data/nfsroot/etc/event.d/tty1 and change this:

#OLD#exec /sbin/getty 38400 tty1
exec /bin/login -f xbmc  /dev/tty1 2>&1

Make fluxbox to auto-start
vi /mnt/data/nfsroot/home/xbmc/.bash_profile and enter:

case "`tty`" in
/dev/tty1) clear && startx &>/dev/null;;
esac

Make XBMC to auto-start
vi /mnt/data/nfsroot/home/xbmc/.fluxbox/init and add this line at the end:

session.screen0.rootCommand: xbmc-standalone

Conclusion

Now restart your HTPC and everything should work!

Please leave me a message in the forum if you see something wrong or if you just want to tell me that you tried it :)

The PXE configuration files can be named using the MAC address of each computer, the “default” can be to install Ubuntu using the netboot files, it means no more installation CD (or USB drive) to create and also that all the packages are download and installed from the latest copy available in the Ubuntu repositories. And if you put the systems files in subdirectories (xbmc-htpc, ubuntu-lamp…) you can boot differents diskless computers with sperated system files all loaded from the NFS server.

Tip: use rsync to convert filenames from/to UTF-8 <–> latin-1/ISO-8859-1

This will copy files with UTF-8 encoded characters filenames to new files with a ISO-8859-1 encoded filenames and, when everything is completed, will delete the old (UTF-8) files.

BE CAREFUL THIS IS NOT FOOL PROOF :) YOU MUST KNOW WHAT YOU ARE DOING TO NOT DOUBLE ENCODE THE CHARACTERS FILENAMES.

[cci]rsync SOURCE SAMEFORDESTINATION –recursive –human-readable –stats –times –progress –iconv=utf8,iso88591 –delete-after[/cci]

To do the opposite change the iconv switch to: [cci]–iconv=iso88591,utf8[/cci]

Note: If you run rsync as root, you can use [cci]–owner –group[/cci] to keep the same user.group for the copied files.

FreeNAS (embedded) iconv command

To install iconv on a “data” parition:

[cci]export PKG_TMPDIR=/mnt/usb-data/temp
pkg_add -r iconv -P /mnt/usb-data/pkg[/cci]

And execute this at boot time:
[cci]ln -s /mnt/usb-data/pkg/bin/iconv /bin/iconv[/cci]

FreeNAS Bash script to start a ZFS scrub on each pool (scrub.sh)

This script will start a scrub on each ZFS pool (one at a time) and will send an e-mail or display the result when everyting is completed. I wrote this script to launch each scrub one after the other and a summary by e-mail that can tell me how much time it took to each ZFS scrub.

Put this script on your FreeNAS box using the File Manager or directly at the shell by copy-pasting it. I put it in “/mnt/usb-data/bin/scrub.sh”.

To make it run automatically, go into System –> Advanced –> Cron and add it as Monthly cron job.

NOTE: Don’t forget to change the e-mail addresses in the script!

FROM=from@devnull.com
TO=to@devnull.com

Here’s the script:

#!/bin/bash

#VERSION: 0.2
#AUTHOR: gimpe
#EMAIL: gimpe [at] hype-o-thetic.com
#WEBSITE: http://hype-o-thetic.com
#DESCRIPTION: Created on FreeNAS 0.7RC1 (Sardaukar)
# This script will start a scrub on each ZFS pool (one at a time) and
# will send an e-mail or display the result when everyting is completed.

#CHANGELOG
# 0.2: 2009-08-27 Code clean up
# 0.1: 2009-08-25 Make it work

#SOURCES:
# http://aspiringsysadmin.com/blog/2007/06/07/scrub-your-zfs-file-systems-regularly/
# http://www.sun.com/bigadmin/scripts/sunScripts/zfs_completion.bash.txt
# http://www.packetwatch.net/documents/guides/2009073001.php

# e-mail variables
FROM=from@devnull.com
TO=to@devnull.com
SUBJECT="$0 results"
BODY=""

# arguments
VERBOSE=0
SENDEMAIL=1
args=("$@")
for arg in $args; do
    case $arg in
        "-v" | "--verbose")
            VERBOSE=1
            ;;
        "-n" | "--noemail")
            SENDEMAIL=0
            ;;
        "-a" | "--author")
            echo "by gimpe at hype-o-thetic.com"
            exit
            ;;
        "-h" | "--help" | *)
            echo "
usage: $0 [-v --verbose|-n --noemail]
    -v --verbose    output display
    -n --noemail    don't send an e-mail with result
    -a --author     display author info (by gimpe at hype-o-thetic.com)
    -h --help       display this help
"
            exit
            ;;
    esac
done

# work variables
ERROR=0
SEP=" - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - "
RUNNING=1

# commands &amp; configuration
ZPOOL=/sbin/zpool
PRINTF=/usr/bin/printf
MSMTP=/usr/local/bin/msmtp
MSMTPCONF=/var/etc/msmtp.conf

# print a message
function _log {
    DATE="`date +"%Y-%m-%d %H:%M:%S"`"
    # add message to e-mail body
    BODY="${BODY}$DATE: $1\n"

    # output to console if verbose mode
    if [ $VERBOSE = 1 ]; then
        echo "$DATE: $1"
    fi
}

# find all pools
pools=$($ZPOOL list -H -o name)

# for each pool
for pool in $pools; do
    # start scrub for $pool
    _log "starting scrub on $pool"
    zpool scrub $pool
    RUNNING=1
    # wait until scrub for $pool has finished running
    while [ $RUNNING = 1 ];     do
        # still running?
        if $ZPOOL status -v $pool | grep -q "scrub in progress"; then
            sleep 60
        # not running
        else
            # finished with this pool, exit
            _log "scrub ended on $pool"
            _log "`$ZPOOL status -v $pool`"
            _log "$SEP"
            RUNNING=0
            # check for errors
            if ! $ZPOOL status -v $pool | grep -q "No known data errors"; then
                _log "data errors detected on $pool"
                ERROR=1
            fi
        fi
    done
done

# change e-mail subject if there was error
if [ $ERROR = 1 ]; then
    SUBJECT="${SUBJECT}: ERROR(S) DETECTED"
fi

# send e-mail
if [ $SENDEMAIL = 1 ]; then
    $PRINTF "From:$FROM\nTo:$TO\nSubject:$SUBJECT\n\n$BODY" | $MSMTP --file=$MSMTPCONF -t
fi

Here’s what the e-mail content looks like:

2009-09-01 02:00:01: starting scrub on backup
2009-09-01 02:35:19: scrub ended on backup
2009-09-01 02:35:19:   pool: backup
 state: ONLINE
 scrub: scrub completed with 0 errors on Tue Sep  1 02:34:50 2009
config:

       NAME        STATE     READ WRITE CKSUM
       backup      ONLINE       0     0     0
         mirror    ONLINE       0     0     0
           ad8     ONLINE       0     0     0
           ad10    ONLINE       0     0     0

errors: No known data errors
2009-09-01 02:35:19:  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2009-09-01 02:35:19: starting scrub on data
2009-09-01 02:41:24: scrub ended on data
2009-09-01 02:41:24:   pool: data
 state: ONLINE
 scrub: scrub completed with 0 errors on Tue Sep  1 02:40:50 2009
config:

       NAME        STATE     READ WRITE CKSUM
       data        ONLINE       0     0     0
         ad6       ONLINE       0     0     0

errors: No known data errors
2009-09-01 02:41:24:  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2009-09-01 02:41:24: starting scrub on media
2009-09-01 05:41:58: scrub ended on media
2009-09-01 05:41:58:   pool: media
 state: ONLINE
 scrub: scrub completed with 0 errors on Tue Sep  1 05:41:00 2009
config:

       NAME        STATE     READ WRITE CKSUM
       media       ONLINE       0     0     0
         ad14      ONLINE       0     0     0

errors: No known data errors
2009-09-01 05:41:58:  - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Please, leave me a comment if you use it to let me know what you think!

Useful tools in an “embedded” FreeNAS: wget and bash

wget

To install wget on a “data” parition:

export PKG_TMPDIR=/mnt/usb-data/temp
pkg_add -r wget -P /mnt/usb-data/pkg

Now we need a script to create the symlinks to the wget binaries in the “pkg” directory:

nano /mnt/usb-data/hooks.sh

Here’s the content:

#!/bin/sh

ln -s /mnt/usb-data/pkg/bin/wget /bin/wget

Now make it run at boot time, go in System —> Advanced —> Command scripts and enter:

/mnt/usb-data/hooks.sh

Select PostInit and that’s it! Everytime you reboot your FreeNAS server the symlinks will be re-created!

bash

For bash, add this in the “hooks.sh” script:

# .bashrc for root
ln -s /mnt/usb-data/bashrc /root/.bashrc

# .bashrc for regular user
ln -s /mnt/usb-data/bashrc /mnt/.bashrc

# automatically load bash for root on login (not clean but works)
echo "bash --init-file /mnt/usb-data/bashrc" >> /root/.cshrc

# automatically load bash for regular user on login (not clean but works)
echo "bash --init-file /mnt/usb-data/bashrc" >> /mnt/.cshrc
Creative Commons Attribution-ShareAlike 2.5 Canada
This work by gimpe is licensed under a Creative Commons Attribution-ShareAlike 2.5 Canada.