ZFS

caLLous

I am a FH squatter
Joined
Dec 23, 2003
Messages
17,569
Hi linux-y/osx-y/solaris-y types! I want to put Linux (probably Mint) on my server and run ZFS (RAIDZ) on four 4tb disks (expanding to 8 disks next year probably).

Question: is it as easy as I'm making it sound? I've been reading up about "ZFS on Linux" and a recent announcement about OpenZFS but I've never *been* in Linux before except for playing around in VM's.

I use this box for downloading, encoding (I'm sure I can set Handbrake up in Ubuntu), and that's about it. Am I better off going for an LSI MegaRAID card or something and doing semi-hardware RAID5? The more I read about ZFS the better it sounds but I have zero experience with it and don't want to screw anything up.

Server specs are: i5-3570 with 8gb RAM (I'm thinking of going from a WD Black HDD to an SSD for the OS as well). That can hack ZFS, right? I'm not intending to be doing anything fancy (deduplicating).

Thanks!
 

TdC

Trem's hunky sex love muffin
Joined
Dec 20, 2003
Messages
30,693
ZFS is quite awesome tbh. Best thing Sun ever stole :) The open variants I am not aware of, but from what I remember of the Sun courses I've been to setting up what you want to do should be a walk in the park. Remember to check how you want to boot your server because iirc the open ZFSssessses could have issues letting a computer boot off a ZFS pool.

there are prolly a million people on le googlé that have already solved and documented whatever you want done so I don't see any problems.
 

caLLous

I am a FH squatter
Joined
Dec 23, 2003
Messages
17,569
Yeah it looks great. I won't be booting from the array, I'll put the SSD in for the OS. The way I see it I can take my current win7 OS disk out and put the SSD in and install Mint and then if I get scared I can always put the win7 disk back in. I've played with Mint in a VM, I've just never tried anything ZFS-y. All of my brushes with "RAID" so far have been via Intel's Rapid Storage thing.
 

caLLous

I am a FH squatter
Joined
Dec 23, 2003
Messages
17,569
there are prolly a million people on le googlé that have already solved and documented whatever you want done so I don't see any problems.
This guy.


I know the general perception of a Linux user fits into a bit of a stereotype but... come on. :)

(having said that, I could listen to him talk all day and some of the stuff he said is actually quite useful)
 

caLLous

I am a FH squatter
Joined
Dec 23, 2003
Messages
17,569
Bit of a faux-pas on my part, turns out "he" is a "she" who had re-assignment surgery and goes by Michelle now. Who knew. :s
 

caLLous

I am a FH squatter
Joined
Dec 23, 2003
Messages
17,569
Thanks GReaper, I found that through zfsonlinux.org/docs.html - out of interest why are you not sure if you want to use it? You don't need to? You don't have a big server or something that might make use of it? You don't have faith in it?

I've been extensively playing around with ZFS and dummy files in Mint in a VM and have got my head around the basics (except changing Samba share permissions after setting sharesmb=on through zfs but I'm sure that's an smb.conf thing or something) but I'm still very much in the planning phase.

I think I would be more inclined to put something with absolute native ZFS support on the box instead of Linux but I really want to use it for more than just a NAS and I've found existing Ubuntu ports of my favourite programs for each task.
 

GReaper

Resident Freddy
Joined
Dec 22, 2003
Messages
1,973
My only NAS is low powered ARM box with 512MB RAM and 2x1TB drives. That immediately rules it out, as apparently it needs to run on a 64 bit machine to run reliably.

I just find it slightly annoying that support for it is all 3rd party due to the licensing. Fortunately I've never needed it as I don't need a large RAID setup, software raid on ext4 is sufficient for what I need.

If you're running 4-8 disks then go ahead, ZFS is worth it for you.
 

MYstIC G

Official Licensed Lump of Coal™ Distributor
Staff member
Moderator
FH Subscriber
Joined
Dec 22, 2003
Messages
11,845
I've found existing Ubuntu ports of my favourite programs for each task.
What are the programs if you don't mind sharing? Possible alternatives might exist, etc.
 

Chilly

Balls of steel
Joined
Dec 22, 2003
Messages
9,046
I wouldnt bother running ZFS on linux. It's raison d'etre is ubergrade array managment and corruption avoiddance. It's simply not had enough production hours on Linux. Use a *BSD. FreeNas does it all for you.
 

caLLous

I am a FH squatter
Joined
Dec 23, 2003
Messages
17,569
Yeah, FreeNAS looks very interesting. I was originally hoping to put something on there with true native ZFS support, I was looking at SmartOS and other illumos distributions (OmniOS and EON Storage were the other 2 interesting ones) but I think that all might be a bit too hardcore for my needs. The thing is, I want to be able to do desktop-y things on the box also, like media encoding for one. I'm looking into Xen for virtualisation with FreeNAS.

I've got 8 x 4tb drives all in place. I've installed Ubuntu Server 12.4.03 and had a good play around but KVM is giving me a right headache. I was hoping to have Ubuntu Server as the core with ZFS and then have Mint (or something) in a VM (with full virtualisation because I have a CPU with Vt-X and EPT) to run Handbrake (that's one of the programs, @MysticG) because I know and like it. I've not got around to setting ZFS up yet because I wanted to be sure that I had the basic systems up and running beforehand.

As long as FreeNAS can deal with proper alignment on 4k sector drives (after a quick google I think it can) and also full virtualisation (I think it can, although I've never played with Xen) it might well be the choice for me.
 

Chilly

Balls of steel
Joined
Dec 22, 2003
Messages
9,046
Yeah, the BSD freenas runs is fully capable of virtualisation for any OS you care to want to run.
 

Chilly

Balls of steel
Joined
Dec 22, 2003
Messages
9,046
Also, you could actually just run Solaris. That thing has mil-spec virtualisation and every other advanced OS feature you could possibly want.
 

caLLous

I am a FH squatter
Joined
Dec 23, 2003
Messages
17,569
Hmm, do you mean Solaris as in Solaris or a distro from the illumos fork? To be honest, I do like the look of SmartOS and EON Storage, they seem to do everything I might want them to. This is proper unchartered territory on my part though... I've played with Linux for the past few months in a vbox vm in the safety of Windows but I've *never* looked at Solaris. I'm sure there's loads of help online, though. I only want to set up ZFS, some SMB shares and virtualisation?

How hard can it be? :D
 

TdC

Trem's hunky sex love muffin
Joined
Dec 20, 2003
Messages
30,693
Also, you could actually just run Solaris. That thing has mil-spec virtualisation and every other advanced OS feature you could possibly want.
once apon a time I was a Solaris admin and imo it is a pretty sweet OS, but a terribad learning curve if you just want to run a nas or whatnot. that said, there is always open solaris (now called openindiana btw).
 

MYstIC G

Official Licensed Lump of Coal™ Distributor
Staff member
Moderator
FH Subscriber
Joined
Dec 22, 2003
Messages
11,845
Handbrake has a FreeBSD port: http://www.freshports.org/multimedia/handbrake/ and I know nothing about whether you can run it from a command line or if it requires a GUI but looking at the flags it appears you should be able to do both.

In that instance you should be able to do: FreeNAS install -> ZFS your drives however you choose -> Add a Jail -> Install Handbrake -> Profit

You might however want to have a look at PC-BSD. It's basically FreeBSD with a GUI which I'm sure will allow you to run anything you like.
 

TdC

Trem's hunky sex love muffin
Joined
Dec 20, 2003
Messages
30,693
hmm I got curious and installed an openindiana iso in a virtual box. it basically looks a lot like Solaris10/11 (uname actually reports SunOS / Solaris 5.11 (ofc you can make uname say anything you like, but OS builders tend to call it like it is hehe))

I'd say it's massive, massive overkill to run as a storage provider solution unless you're an enterprise of some sort or a stupendous masochist, or both :)
 

caLLous

I am a FH squatter
Joined
Dec 23, 2003
Messages
17,569
So I have FreeNAS up and running with a ZFS pool. I originally bought an SSD to put the OS on but FreeNAS takes up the whole drive regardless of size so I put it on a USB stick and put the SSD into the array as a cache. I don't know if I'll see any benefits with it with the amount of use the system is going to get but we'll see.

I'm not sure why I've "only" got 20.3TB available from 32TB worth of disks, though. Assuming each 4tb drive is 4000000000000 bytes, which comes out at just under 3.64 actual terabytes, with 8 disks in RAIDZ2 I should be seeing ~21.9TB (3.64x6), surely? Is there some overhead I'm not taking into account?

All signs point to FreeNAS definitely *not* being able to host VM's. I do like it though, so maybe using my main PC for encoding wouldn't be the end of the world. It just seems a pity to have all the power of the FreeNAS box sitting there not doing an awful lot. Maybe I will have a play with EON or SmartOS tomorrow.
 

caLLous

I am a FH squatter
Joined
Dec 23, 2003
Messages
17,569
I'd say it's massive, massive overkill to run as a storage provider solution unless you're an enterprise of some sort or a stupendous masochist, or both :)
I like OpenIndiana. I put it on the box and had a VNC-able, SSH-able VM of Ubuntu Server up within minutes.

I've been running benchmarks with Handbrake CLI and it's not too shabby at all. I'm not sure how bright it is to hand all 4 cores of a quad core CPU to a VM but the encoding won't exactly be time-sensitive so maybe it can have 2 of the 4. With all 4 cores, the encoding speed isn't far off my bare-metal (check me out with the lingo) i7-3770k powered win7 PC. I haven't setup ZFS yet - I hear napp-it is a good way of going about that in a Solaris-based OS).

Not sure about not being able to monitor CPU temperatures in OpenIndiana though... You can use ipmitool apparently but you need pretty specialist server-y hardware for that.
 

TdC

Trem's hunky sex love muffin
Joined
Dec 20, 2003
Messages
30,693
ZFS will also reserve 1/64th of the capacity of the entire pool for itself (for copy on write stuff, meta things, my balls, and whatever else it can get in there). your capacity should be in the order of 3.64*6, or 21.84. one 64th of that is 0.341TB so your ultimate possible super storage is something in the order of 21.5TB. commands like zpool list will give you funky numbers, whereas commands like df should give you adjusted numbers.
 

caLLous

I am a FH squatter
Joined
Dec 23, 2003
Messages
17,569
You're not wrong, zpool list gives me:
Code:
NAME  SIZE  ALLOC  FREE  EXPANDSZ  CAP  DEDUP  HEALTH  ALTROOT
tank  29T   1.69M  29.0T  -        0%   1.00x  ONLINE  -
And df gives me 21791636711 bytes or 20.29TB. In the file browser in OI it says 20.3TB so I guess that's my lot. Unless I man up and go with raidz instead of raidz2...
 

TdC

Trem's hunky sex love muffin
Joined
Dec 20, 2003
Messages
30,693
yeah no idea how that works out tbh. good luck however you choose :)
 

caLLous

I am a FH squatter
Joined
Dec 23, 2003
Messages
17,569
yeah no idea how that works out tbh. good luck however you choose :)
I came to my senses in the end and put Ubuntu Server on as the sole OS. I've done away with the idea of VMs, there's no need and it only adds layers of complication that I'm linux-savvy enough to confidently deal with. ZFS on Linux is "production ready", handbrake-cli works just fine and webmin is fully *awesome* for administering the system. As far as I can tell there's nothing that I need that can't be achieved in the Server edition of Ubuntu.
Code:
NAME  USED   AVAIL  REFER   MOUNTPOINT
tank  10.8T  9.50T  358K    /tank
Huzzah!
 

TdC

Trem's hunky sex love muffin
Joined
Dec 20, 2003
Messages
30,693
necro!!!

I bought another 1tb Lacie Rikiki USB3 drive and an ickle USB3 hub. I put some shit together and installed a Debian with the zfsonlinux repo in to a vm on VirtualBox and dicked about until it dawned on me that VB does not support USB3 (yet, according to the devs). I then downloaded VMware workstation and tested with that, which worked fine (commercial product eh) and I faffed about with some ZFS stuff.

I actually want to use the VM as an iSCSI providor for other vm's just to see if I can get it to work, but hey ho depends on time. Anyway, I made some files out of random input and read and write speeds to the array of two disks isn't all that bad actually: 60~85 MB/s, but if you get ZFS to play with itself...oooh boy then it really flies. Here's performance of a zpool scrub on two USB3 Rikiki's :

Code:
  scan: scrub in progress since Mon Jan 20 21:31:45 2014
  10.3G scanned out of 13.9G at 152M/s, 0h0m to go
Not bad if I say so myself :-D
 

caLLous

I am a FH squatter
Joined
Dec 23, 2003
Messages
17,569
Not bad at all. :)

I just put 4 more 2tb (SATA 3GB/s) drives in (I bloody love this Lian-Li PC-D8000 case!) in another pool (raidz1), I get 280-300mb/s when scrubbing that.
Code:
22.0G scanned out of 5.28T at 289M/s, 5h18m to go
With the 8-drive pool (SATA 6GB/s, raidz2), I get ~480mb/s (iirc... I should really do another scrub, they say to do one every week for consumer-level disks).
 

TdC

Trem's hunky sex love muffin
Joined
Dec 20, 2003
Messages
30,693
I have some performance enhancing tips for you if you like. very handy of you happen to have a small ssd banging about that you don't have a use for any more
 

caLLous

I am a FH squatter
Joined
Dec 23, 2003
Messages
17,569
I have some performance enhancing tips for you if you like. very handy of you happen to have a small ssd banging about that you don't have a use for any more
Thanks chief, are you talking about the L2ARC with the SSD? I looked into it but a) I don't have an SSD lying around (I almost did when I was maybe going to go with an OpenSolaris distro running off of a USB stick) and b) the performance is fine tbh, pretty much all of the stuff on there is archive so there's not a massive amount of movement after I've written it all in the first place. Scrubbing is the most intense action that takes place tbh.
Yeah! :) There's plenty of scope for adding more as well. You can't do away with using a 5.25" bay for the power/reset button and USB ports but you can put 2 or 3 3.5" drives in the remaining 2 bays (there's space for another behind the buttons to be fair) and another 4 in the 3 other 5.25" bays with one of these:



But wait! There's more!



There are 3x 140mm fan places on the back panel. I'm sure with very little effort you could fix one of those bay converters in for each fan slot and get another 12 drives in. So that's potentially 38 or 39 3.5" drives and loads of space to velcro a boot SSD somewhere.

I wouldn't want to pack the drives in like in the following video but somebody has. That server is actually from before the days of the PC-D8000, somebody went and welded two "normal" Lian-Li tower cases together to make it. I think it looks horrendous with all that chequer plate on it but to each their own.


Then of course you can use an HBA with external ports and connect another case full of drives with an SFF-8088 cable.
 

TdC

Trem's hunky sex love muffin
Joined
Dec 20, 2003
Messages
30,693
yeah mate but also moving your ZIL from memory on to a dedicated device (aka making a SLOG), as well as some settings. atm I'm designing the storage backend stuff for some new virtualized infra at workies, so I am chock full of zfs geekery :)
 

Users who are viewing this thread

Top Bottom