ZFS
Installing ZFS on Ubuntu Server 12.04
- From ubuntu server I needed to install python-software-properties, so that i could run add-apt-repository
- ( You don’t need to complete this first step from ubuntu desktop, or mint )
$ sudo apt-get install python-software-properties
- Then install the zfs-native ppa, so that you can download and compile native zfs goodness
$ sudo add-apt-repository ppa:zfs-native/stable
- Update your repo list, so it includes the new zfs-native ppa
$ sudo apt-get update
Make sure the the zfs packages in the repo
$ apt-cache search zfs
- Install, Note: this might take some time as it compiles the kernel module for your kernel
$ sudo apt-get install ubuntu-zfs
- Run the zfs commands to make sure it works
$ sudo zfs $ sudo zpool
- Create a pool
-m sets default mount directory
$ sudo zpool create -m /var/local/storage pool1 raidz /dev/sda /dev/sdb /dev/sdc
- Check the mounts
$ dan@nas:/data$ df -h -F zfs Filesystem Size Used Avail Use% Mounted on pool1 3.6T 128K 3.6T 1% /var/local/storage
ZFS with 4K drives
zpool create -m none -o ashift=12 mypool raidz /dev/disk1 /dev/disk2 /dev/disk3
The -o ashift=12 forces ZFS to use 4K sectors instead of 512 byte sectors. Many new drives use 4K sectors, but lie to the OS about it for ‘compatability’ reasons. My first ZFS filesystem used the 512-byte sectors in the beginning, and I had shocking performance (~10Mb/s write).
See http://zfsonlinux.org/faq.html#HowDoesZFSonLinuxHandlesAdvacedFormatDrives for more information about it.
ZFS Cheatsheet
This is a quick and dirty cheatsheet on Sun's ZFS
- Directories and Files
error messages /var/adm/messages console
- States
DEGRADED One or more top-level devices is in the degraded state because they have become offline. Sufficient replicas exist to keep functioning FAULTED One or more top-level devices is in the faulted state because they have become offline. Insufficient replicas exist to keep functioning OFFLINE The device was explicity taken offline by the "zpool offline" command ONLINE The device is online and functioning REMOVED The device was physically removed while the system was running UNAVAIL The device could not be opened
- Storage Pools
Displaying
zpool list zpool list -o name,size,altroot Note: there are a number of properties that you can select, the default is: name, size, used, available, capacity, health, altroot
- Status
zpool status
Show only errored pools with more verbosity zpool status -xv
- Statistics
zpool iostat -v 5 5
Note: use this command like you would iostat
- History
zpool history -il
- Creating
Performing a dry run but don't actual perform the creation
zpool create -n data01 c1t0d0s0
you can persume that I created two files called /zfs1/disk01 and /zfs1/disk02 using mkfile
zpool create data01 /zfs1/disk01 /zfs1/disk02
using a standard disk slice
zpool create data01 c1t0d0s0
using a different mountpoint than the default /<pool name>
zpool create -m /zfspool data01 c1t0d0s0
mirror and hot spare disks examples
zpool create data01 mirror c1t0d0 c2t0d0 mirror c1t0d1 c2t0d1 zpool create data01 mirror c1t0d0 c2t0d0 spare c3t0d0
setting up a log device and mirroring it
zpool create data01 mirror c1t0d0 c2t0d0 log mirror c3t0d0 c4t0d0
setting up a cache device
zpool create data 01 mirror c1t0d0 c2t0d0 cache c3t0d0 c3t1d0
destroying
zpool destroy /zfs1/data01
in the event of a disaster you can re-import a destroyed pool
zpool import -f -D -d /zfs1 data031
adding
zpool add data01 c2t0d0
Note: make sure that you get this right as zpool only supports the removal of hot spares and cache disks removing
zpool remove data01 c2t0d0
Note: zpool only supports the removal of hot spares and cache disks clearing faults
zpool clear data01
Clearing a specific disk fault
zpool clear data01 c2t0d0
attaching ## c2t0d0 is an existing disk that is not mirrored, by attaching c3t0d0 both disks will become a mirror pair
zpool attach data01 c2t0d0 c3t0d0
detaching
zpool detach data01 c2t0d0
Note: see above notes is attaching onlining
zpool online data01 c2t0d0
offlining
zpool offline data01 c2t0d0
Temporary offlining (will revent back after a reboot)
zpool offline data01 -t c2t0d0
Replacing ## replacing like for like zpool replace data03 c2t0d0
Replacing with another disk
zpool replace data03 c2t0d0 c3t0d0
scrubbing
zpool scrub data01
Stop a scrubbing in progress, check the scrub line using "zpool status data01" to see any errors
zpool scrub -s data01
Note: Only one of scrubbing or resilvering can be running at the sametime
scrubbing - examines all data to discover hardware faults or disk failures resilvering - examines only data that ZFS knows to be out of date
exporting
zpool export data01
importing when using standard disk devices i.e c2t0d0
zpool import data01
if using files in say the /zfs filesystem
zpool import -d /zfs
importing a destroyed pool
zpool import -f -D -d /zfs1 data03
getting parameters
zpool get all data01
Note: the source column denotes if the value has been change from it default value, a dash in this column means it is a read-only value setting parameters
zpool set autoreplace=on data01
Note: use the command "zpool get all <pool>" to obtain list of current setting upgrade ## List upgrade paths
zpool upgrade -v
upgrade all pools
zpool upgrade -a
upgrade specific pool, use "zpool get all <pool>" to obtain version number of a pool
zpool upgrade data01
upgrade to a specific version
zpool upgrade -V 10 data01
Filesystem displaying
zfs list
list different types
zfs list -t filesystem zfs list -t snapshot zfs list -t volume
recursive display
zfs list -r data01/oracle
complex listing
zfs list -o name,sharenfs,mountpoint
Note: there are a number of attributes that you can use in a complex listing, so use the man page to see them all creating
persuming i have a pool called data01 create a /data01/apache filesystem
zfs create data01/apache
using a different mountpoint
zfs create -o mountpoint=/oracle data01/oracle
create a volume - the device can be accessed via /dev/zvol/[rdsk|dsk]/data03/swap
zfs create -V 50mb data01/swap swap -a /dev/zvol/dsk/data01/swap
Note: don't use a zfs volume as a dump device it is not supported destroying
zfs destroy data01/oracle
using the recusive options -r = all children, -R = all dependants
zfs destroy -r data01/oracle zfs destroy -R data01/oracle
mounting
zfs mount data01
Note: there are all the normal mount options that you can apply i.e ro/rw, setuid unmounting
zfs umount data01
share
zfs share data01
Persist over reboots
zfs set sharenfs=on data01
unshare
zfs unshare data01
persist over reboots
zfs set sharenfs=off data01
snapshotting
zfs snapshot data01@10022010
destroying a snapshot
zfs destroy data01@10022010
rollback
zfs rollback data01@10022010
cloning/promoting
zfs clone data01@10022010 data03/clone
promoting a clone
zfs promote data03/clone
Note: the clone must reside in the same pool renaming the dataset must be kept within the same pool
zfs rename data03/ora_disk01 data03/ora_d01
Note: you have two options -p creates all the non-existing parent datasets -r recursively rename the sanpshots of all descendent datasets (used with snapshots only)
getting parameters
List all the properties
zfs get all data03/oracle
get a specific property
zfs get setuid data03/oracle
get a list of a specific properites for all datasets
zfs get compression
Note: the source column denotes if the value has been change from it default value, a dash in this column means it is a read-only value setting parameters
set and unset a quota
zfs set quota=50M data03/oracle zfs set quota=none data03/oracle
Note: use the command "zfs get all <dataset> " to obtain list of current settings inherit ## set back to the default value
zfs inherit compression data03/oracle
upgrade List the upgrade paths
zfs upgrade -v
List all the datasets that are not at the current level
zfs upgrade
upgrade a specific dataset
upgrade -V <version> data03/oracle
send/receive here is a complete example of a send and receive with incremental update create some test files
mkfile -v 100m /zfs/master mkdir -v 100m /zfs/slave
create mountpoints
mkdir /master mkdir /slave
Create the pools
zpool create master zpool create slave
create the data filesystem
zfs create master/data
create a test file
echo "created: 09:58" > /master/data/test.txt
create a snapshot and send it to the slave, you could use SSH or tape to transfer to another server (see below)
zfs snapshot master/data@1 zfs send master/data@1 | zfs receive slave/data
set the slave to read-only because you can cause data corruption, make sure if do this before accessing anything the slave/data directory
zfs set readonly=on slave/data
update the original test.txt file
echo "`date`" >> /master/data/text.txt
create a second snapshot and send the differences, you may get an error message saying that the desination had been modified this is because you did not set the slave/data to ready only (see above)
zfs snapshot master/data@2 zfs send master/data@1 master/data@2 | zfs receive slave/data
using SSH
zfs send master/data@1 | ssh backup_server zfs receive backups/data@1
using a tape drive, you can also use cpio
zfs send master/data@1 > /dev/rmt/0 zfs receive slave/data2@1 < /dev/rmt/0 zfs rename slave/data slave/data.old zfs rename slave/data2 slave/data
you can also save incremental data
zfs send master/data@12022010 > /dev/rmt/0 zfs send -i master/data@12022010 master/data@13022010 > /dev/rmt/0
Using gzip to compress the snapshot
zfs send master/fs@snap | gzip > /dev/rmt/0
allow/unallow display the permissions set
zfs allow master
create permission set
zfs allow -s @permset1 create,mount,snapshot,clone,promote master
grant a user permissions
zfs allow vallep @permset1 master
revoke a user permissions
zfs unallow vallep @permset1 master
Note: there are many permissions that you can set so see the man page or just use the "zfs allow" command