ZFS

[caLLous]

Hi linux-y/osx-y/solaris-y types! I want to put Linux (probably Mint) on my server and run ZFS (RAIDZ) on four 4tb disks (expanding to 8 disks next year probably).

Question: is it as easy as I'm making it sound? I've been reading up about "ZFS on Linux" and a recent announcement about OpenZFS but I've never *been* in Linux before except for playing around in VM's.

I use this box for downloading, encoding (I'm sure I can set Handbrake up in Ubuntu), and that's about it. Am I better off going for an LSI MegaRAID card or something and doing semi-hardware RAID5? The more I read about ZFS the better it sounds but I have zero experience with it and don't want to screw anything up.

Server specs are: i5-3570 with 8gb RAM (I'm thinking of going from a WD Black HDD to an SSD for the OS as well). That can hack ZFS, right? I'm not intending to be doing anything fancy (deduplicating).

Thanks!

 

[TdC]

ZFS is quite awesome tbh. Best thing Sun ever stole The open variants I am not aware of, but from what I remember of the Sun courses I've been to setting up what you want to do should be a walk in the park. Remember to check how you want to boot your server because iirc the open ZFSssessses could have issues letting a computer boot off a ZFS pool.

there are prolly a million people on le googlé that have already solved and documented whatever you want done so I don't see any problems.

 

[caLLous]

Yeah it looks great. I won't be booting from the array, I'll put the SSD in for the OS. The way I see it I can take my current win7 OS disk out and put the SSD in and install Mint and then if I get scared I can always put the win7 disk back in. I've played with Mint in a VM, I've just never tried anything ZFS-y. All of my brushes with "RAID" so far have been via Intel's Rapid Storage thing.

 

[caLLous]

there are prolly a million people on le googlé that have already solved and documented whatever you want done so I don't see any problems.

This guy.



I know the general perception of a Linux user fits into a bit of a stereotype but... come on.

(having said that, I could listen to him talk all day and some of the stuff he said is actually quite useful)

 

[TdC]

haha what a character

 

[caLLous]

Bit of a faux-pas on my part, turns out "he" is a "she" who had re-assignment surgery and goes by Michelle now. Who knew. :s

 

[GReaper]

If you're considering ZFS, I'd highly recommend reading through the ZFS Administration guide.

Never used it myself, not sure if I want to use it, but the guide was interesting.

 

[caLLous]

Thanks GReaper, I found that through zfsonlinux.org/docs.html - out of interest why are you not sure if you want to use it? You don't need to? You don't have a big server or something that might make use of it? You don't have faith in it?

I've been extensively playing around with ZFS and dummy files in Mint in a VM and have got my head around the basics (except changing Samba share permissions after setting sharesmb=on through zfs but I'm sure that's an smb.conf thing or something) but I'm still very much in the planning phase.

I think I would be more inclined to put something with absolute native ZFS support on the box instead of Linux but I really want to use it for more than just a NAS and I've found existing Ubuntu ports of my favourite programs for each task.

 

[GReaper]

My only NAS is low powered ARM box with 512MB RAM and 2x1TB drives. That immediately rules it out, as apparently it needs to run on a 64 bit machine to run reliably.

I just find it slightly annoying that support for it is all 3rd party due to the licensing. Fortunately I've never needed it as I don't need a large RAID setup, software raid on ext4 is sufficient for what I need.

If you're running 4-8 disks then go ahead, ZFS is worth it for you.

 

[MYstIC G]

I've found existing Ubuntu ports of my favourite programs for each task.

What are the programs if you don't mind sharing? Possible alternatives might exist, etc.

 

[Chilly]

I wouldnt bother running ZFS on linux. It's raison d'etre is ubergrade array managment and corruption avoiddance. It's simply not had enough production hours on Linux. Use a *BSD. FreeNas does it all for you.

 

[caLLous]

Yeah, FreeNAS looks very interesting. I was originally hoping to put something on there with true native ZFS support, I was looking at SmartOS and other illumos distributions (OmniOS and EON Storage were the other 2 interesting ones) but I think that all might be a bit too hardcore for my needs. The thing is, I want to be able to do desktop-y things on the box also, like media encoding for one. I'm looking into Xen for virtualisation with FreeNAS.

I've got 8 x 4tb drives all in place. I've installed Ubuntu Server 12.4.03 and had a good play around but KVM is giving me a right headache. I was hoping to have Ubuntu Server as the core with ZFS and then have Mint (or something) in a VM (with full virtualisation because I have a CPU with Vt-X and EPT) to run Handbrake (that's one of the programs, @MysticG) because I know and like it. I've not got around to setting ZFS up yet because I wanted to be sure that I had the basic systems up and running beforehand.

As long as FreeNAS can deal with proper alignment on 4k sector drives (after a quick google I think it can) and also full virtualisation (I think it can, although I've never played with Xen) it might well be the choice for me.

 

[Chilly]

Yeah, the BSD freenas runs is fully capable of virtualisation for any OS you care to want to run.

 

[Chilly]

Also, you could actually just run Solaris. That thing has mil-spec virtualisation and every other advanced OS feature you could possibly want.

 

[caLLous]

Hmm, do you mean Solaris as in Solaris or a distro from the illumos fork? To be honest, I do like the look of SmartOS and EON Storage, they seem to do everything I might want them to. This is proper unchartered territory on my part though... I've played with Linux for the past few months in a vbox vm in the safety of Windows but I've *never* looked at Solaris. I'm sure there's loads of help online, though. I only want to set up ZFS, some SMB shares and virtualisation?

How hard can it be?

 

[TdC]

Also, you could actually just run Solaris. That thing has mil-spec virtualisation and every other advanced OS feature you could possibly want.

once apon a time I was a Solaris admin and imo it is a pretty sweet OS, but a terribad learning curve if you just want to run a nas or whatnot. that said, there is always open solaris (now called openindiana btw).

 

[MYstIC G]

Handbrake has a FreeBSD port: http://www.freshports.org/multimedia/handbrake/ and I know nothing about whether you can run it from a command line or if it requires a GUI but looking at the flags it appears you should be able to do both.

In that instance you should be able to do: FreeNAS install -> ZFS your drives however you choose -> Add a Jail -> Install Handbrake -> Profit

You might however want to have a look at PC-BSD. It's basically FreeBSD with a GUI which I'm sure will allow you to run anything you like.

 

[TdC]

hmm I got curious and installed an openindiana iso in a virtual box. it basically looks a lot like Solaris10/11 (uname actually reports SunOS / Solaris 5.11 (ofc you can make uname say anything you like, but OS builders tend to call it like it is hehe))

I'd say it's massive, massive overkill to run as a storage provider solution unless you're an enterprise of some sort or a stupendous masochist, or both

 

[caLLous]

So I have FreeNAS up and running with a ZFS pool. I originally bought an SSD to put the OS on but FreeNAS takes up the whole drive regardless of size so I put it on a USB stick and put the SSD into the array as a cache. I don't know if I'll see any benefits with it with the amount of use the system is going to get but we'll see.

I'm not sure why I've "only" got 20.3TB available from 32TB worth of disks, though. Assuming each 4tb drive is 4000000000000 bytes, which comes out at just under 3.64 actual terabytes, with 8 disks in RAIDZ2 I should be seeing ~21.9TB (3.64x6), surely? Is there some overhead I'm not taking into account?

All signs point to FreeNAS definitely *not* being able to host VM's. I do like it though, so maybe using my main PC for encoding wouldn't be the end of the world. It just seems a pity to have all the power of the FreeNAS box sitting there not doing an awful lot. Maybe I will have a play with EON or SmartOS tomorrow.

 

[caLLous]

I'd say it's massive, massive overkill to run as a storage provider solution unless you're an enterprise of some sort or a stupendous masochist, or both

I like OpenIndiana. I put it on the box and had a VNC-able, SSH-able VM of Ubuntu Server up within minutes.

I've been running benchmarks with Handbrake CLI and it's not too shabby at all. I'm not sure how bright it is to hand all 4 cores of a quad core CPU to a VM but the encoding won't exactly be time-sensitive so maybe it can have 2 of the 4. With all 4 cores, the encoding speed isn't far off my bare-metal (check me out with the lingo) i7-3770k powered win7 PC. I haven't setup ZFS yet - I hear napp-it is a good way of going about that in a Solaris-based OS).

Not sure about not being able to monitor CPU temperatures in OpenIndiana though... You can use ipmitool apparently but you need pretty specialist server-y hardware for that.

 

[TdC]

ZFS will also reserve 1/64th of the capacity of the entire pool for itself (for copy on write stuff, meta things, my balls, and whatever else it can get in there). your capacity should be in the order of 3.64*6, or 21.84. one 64th of that is 0.341TB so your ultimate possible super storage is something in the order of 21.5TB. commands like zpool list will give you funky numbers, whereas commands like df should give you adjusted numbers.

 

[caLLous]

You're not wrong, zpool list gives me:

NAME  SIZE  ALLOC  FREE  EXPANDSZ  CAP  DEDUP  HEALTH  ALTROOT
tank  29T   1.69M  29.0T  -        0%   1.00x  ONLINE  -

And df gives me 21791636711 bytes or 20.29TB. In the file browser in OI it says 20.3TB so I guess that's my lot. Unless I man up and go with raidz instead of raidz2...

 

[TdC]

yeah no idea how that works out tbh. good luck however you choose

 

[caLLous]

yeah no idea how that works out tbh. good luck however you choose

I came to my senses in the end and put Ubuntu Server on as the sole OS. I've done away with the idea of VMs, there's no need and it only adds layers of complication that I'm linux-savvy enough to confidently deal with. ZFS on Linux is "production ready", handbrake-cli works just fine and webmin is fully *awesome* for administering the system. As far as I can tell there's nothing that I need that can't be achieved in the Server edition of Ubuntu.

NAME  USED   AVAIL  REFER   MOUNTPOINT
tank  10.8T  9.50T  358K    /tank

Huzzah!

 

[TdC]

necro!!!

I bought another 1tb Lacie Rikiki USB3 drive and an ickle USB3 hub. I put some shit together and installed a Debian with the zfsonlinux repo in to a vm on VirtualBox and dicked about until it dawned on me that VB does not support USB3 (yet, according to the devs). I then downloaded VMware workstation and tested with that, which worked fine (commercial product eh) and I faffed about with some ZFS stuff.

I actually want to use the VM as an iSCSI providor for other vm's just to see if I can get it to work, but hey ho depends on time. Anyway, I made some files out of random input and read and write speeds to the array of two disks isn't all that bad actually: 60~85 MB/s, but if you get ZFS to play with itself...oooh boy then it really flies. Here's performance of a zpool scrub on two USB3 Rikiki's :

  scan: scrub in progress since Mon Jan 20 21:31:45 2014
  10.3G scanned out of 13.9G at 152M/s, 0h0m to go

Not bad if I say so myself :-D

 

[caLLous]

Not bad at all.

I just put 4 more 2tb (SATA 3GB/s) drives in (I bloody love this Lian-Li PC-D8000 case!) in another pool (raidz1), I get 280-300mb/s when scrubbing that.

22.0G scanned out of 5.28T at 289M/s, 5h18m to go

With the 8-drive pool (SATA 6GB/s, raidz2), I get ~480mb/s (iirc... I should really do another scrub, they say to do one every week for consumer-level disks).

 

[TdC]

I have some performance enhancing tips for you if you like. very handy of you happen to have a small ssd banging about that you don't have a use for any more

 

[GReaper]

(I bloody love this Lian-Li PC-D8000 case!)

"The PC-D8000 can contain 20 3.5″ HDDs. Today’s world requires a lot of storage space. The PC-D8000 is here to satisfy those needs."

20?! Over the years I've built new PCs, I probably haven't even bought 20 hard drives.

 

[caLLous]

I have some performance enhancing tips for you if you like. very handy of you happen to have a small ssd banging about that you don't have a use for any more

Thanks chief, are you talking about the L2ARC with the SSD? I looked into it but a) I don't have an SSD lying around (I almost did when I was maybe going to go with an OpenSolaris distro running off of a USB stick) and b) the performance is fine tbh, pretty much all of the stuff on there is archive so there's not a massive amount of movement after I've written it all in the first place. Scrubbing is the most intense action that takes place tbh.

"The PC-D8000 can contain 20 3.5″ HDDs. Today’s world requires a lot of storage space. The PC-D8000 is here to satisfy those needs."

20?! Over the years I've built new PCs, I probably haven't even bought 20 hard drives.

Yeah! There's plenty of scope for adding more as well. You can't do away with using a 5.25" bay for the power/reset button and USB ports but you can put 2 or 3 3.5" drives in the remaining 2 bays (there's space for another behind the buttons to be fair) and another 4 in the 3 other 5.25" bays with one of these:

But wait! There's more!

There are 3x 140mm fan places on the back panel. I'm sure with very little effort you could fix one of those bay converters in for each fan slot and get another 12 drives in. So that's potentially 38 or 39 3.5" drives and loads of space to velcro a boot SSD somewhere.

I wouldn't want to pack the drives in like in the following video but somebody has. That server is actually from before the days of the PC-D8000, somebody went and welded two "normal" Lian-Li tower cases together to make it. I think it looks horrendous with all that chequer plate on it but to each their own.



Then of course you can use an HBA with external ports and connect another case full of drives with an SFF-8088 cable.

 

[TdC]

yeah mate but also moving your ZIL from memory on to a dedicated device (aka making a SLOG), as well as some settings. atm I'm designing the storage backend stuff for some new virtualized infra at workies, so I am chock full of zfs geekery