ZFS

[Deebs]

A cache makes no difference to the speed of the scrub as it is essentially reading every used block from your pools and verifying the checksum. It can only go as fast as your read speed of your pools.

 

[TdC]

hmmmkay that makes sense; my bad, I never actually checked!

 

[caLLous]

Ok good, I won't bother with an SSD at all then. I've got 32gb of ECC RAM in there and it's not gone higher than about 20gb used with everything I've thrown at it so far.

Have some pictures. I just liked the lighting in the first one (light shining through open top of case) and you can see the LSI card. Number 2 looks awful but there's very little airflow in the middle bit of the case so I'm not fussed. Number 3 is because @Deebs likes wheels.







You could honestly fit a child inside the case. I haven't tried. Honest.

 

[TdC]

has anyone ever confronted you about your storage fetish, callous?

 

[Deebs]

Ok good, I won't bother with an SSD at all then. I've got 32gb of ECC RAM in there and it's not gone higher than about 20gb used with everything I've thrown at it so far.

Have some pictures. I just liked the lighting in the first one (light shining through open top of case) and you can see the LSI card. Number 2 looks awful but there's very little airflow in the middle bit of the case so I'm not fussed. Number 3 is because @Deebs likes wheels.

View attachment 18739

View attachment 18740

View attachment 18741

You could honestly fit a child inside the case. I haven't tried. Honest.

Using an SSD for the ZIL (making it an SLOG) will massively help with synchronous writes but I suspect you are not running NFS or ESXi and iSCSI so it will not matter one bit, all writes are grouped into a transaction group which by default is flushed out every 5 seconds to the pool. If you do not have a SLOG then your ZIL lives in reserved areas on your pools and it is this which kills synch performance as HDD are slow. Another thing to note, the ZIL exists in RAM too, the storage is only used to back synchronous writes (POSIX).

 

[caLLous]

has anyone ever confronted you about your storage fetish, callous?

Heh, you think I've got problems...

 

[TdC]

the leccy company prolly sends a senior exec with the bill every now and then. they drink champagne and chat like old mates..

 

[caLLous]

@TdC I rejiggled things a bit and look how fast it scrubs now!

  scan: scrub in progress since Thu Sep 18 12:17:38 2014
  547G scanned out of 28.5T at 702M/s, 11h37m to go
  0 repaired, 1.87% done

 

[TdC]

haha not bad chum, not bad at all! I'd be jealous but it's unrealistic to compare my ickle 4 disk setup to your infrastructure ;-)

 

[caLLous]

RIP the first 4TB Red in my box, annoyingly it was one of the newer ones. Welcome ata-WDC_WD40EFRX-68WT0N0_WD-WCC4E1460297!

scan: resilvered 2.75T in 22h27m with 0 errors on Sun Feb  8 11:52:29 2015

 

[MYstIC G]

One of mine did the DoA when I switched to 4TB Reds. Thankfully they've been solid ever since.

Did it tell you what it died from?

 

[caLLous]

I've not checked it yet tbh, it came up as REMOVED in zpool status and then OFFLINE. I rebooted and it resilved a bit then said FAULTED so I switched it out pronto and set the resilver going on the new one. I'll plug it into this box and see what the SMART stuff tells me. It wasn't even listing in /dev/disks/by-id/ though so I think it might well be terminal. Still well within its warranty though.

 

[caLLous]

Weird. It appears to be literally dead. Doesn't spin up at all. Maybe a problem on the PCB, I don't know. This has never happened to a disk of mine before.

 

[TdC]

the NSA-A-A took your data away

 

[caLLous]

RAIDZ2 saved me though! Take that NSA-A-A!

Seriously, it's the first resilver I've had to do and it was absolutely seamless.

 

[Draylor]

So the first sign of trouble from the drive was when it effectively bricked itself? Very odd indeed.

 

[caLLous]

Basically yes. It was towards the end of a scrub and the first I saw was that it had come up as REMOVED. As if it thought that the drive just plain wasn't there any more.

 

[Deebs]

I had a similar incident a few months ago in one of my NAS. The drive dropped offline so I bought it back online and resilvered, was fine for a week, no S.M.A.R.T. errors, then it dropped again. Replaced with a new RED and all my pr0n is now safe again.

This year is the year I totally replace my 3 NAS units with a big single one, just need to start researching for a case to hold 12-18 disks in a hot swap manner.

 

[caLLous]

X-Case do some great 4u rackmount cases. Everyone in America raves about the Norco 4u cases but X-Case are the (better) UK equivalent. I'm looking at the RM 424Pro - Short for my first foray into rackmounting. It's a 4u case with 24 hotswap bays and some very fancy stuff inside for fan control and whatnot.

If you want a standalone case you can't go wrong with the Lian-Li PC-D8000 (I'm sure we've been here before ).

 

[Deebs]

X-Case do some great 4u rackmount cases. Everyone in America raves about the Norco 4u cases but X-Case are the (better) UK equivalent. I'm looking at the RM 424Pro - Short for my first foray into rackmounting. It's a 4u case with 24 hotswap bays and some very fancy stuff inside for fan control and whatnot.

If you want a standalone case you can't go wrong with the Lian-Li PC-D8000 (I'm sure we've been here before ).

Think we have to be honest That case is now on my shopping list, just need to find a MB which can take 64mb ECC RAM. Unfortunately my garage is not geared out to take a rack and my tiny office is too much of a shit pit to take a half rack..... or is it.....

 

[caLLous]

I'm thinking a nice little freestanding 16 or 18u cabinet for my needs, something like this. Honestly I don't think the footprint would be THAT much bigger than the PC-D8000...

 

[Deebs]

Bought a D8000 case, have 14 3tb WD Red NAS drives installed. Just moving all the files (using ZFS Send/Recv of course) and then I am good to go and decommission the old NAS.

 

[Scouse]

35Tb of storage can't be bad

 

[Zarjazz]

Bought a D8000 case, have 14 3tb WD Red NAS drives installed. Just moving all the files (using ZFS Send/Recv of course) and then I am good to go and decommission the old NAS.

Blimey, that's certainly some bulk storage. Not jealous, nope, not me ...

What OS you gone with ?

 

[Deebs]

Blimey, that's certainly some bulk storage. Not jealous, nope, not me ...

What OS you gone with ?

Freenas 9.10 U4

  pool: datapool
state: ONLINE
  scan: none requested
config:

        NAME                                            STATE     READ WRITE CKSUM
        datapool                                        ONLINE       0     0     0
          raidz2-0                                      ONLINE       0     0     0
            gptid/3baef772-a6a2-11e6-af00-0cc47ae11ad2  ONLINE       0     0     0
            gptid/3cb98234-a6a2-11e6-af00-0cc47ae11ad2  ONLINE       0     0     0
            gptid/3dc1a42e-a6a2-11e6-af00-0cc47ae11ad2  ONLINE       0     0     0
            gptid/3ec8ddda-a6a2-11e6-af00-0cc47ae11ad2  ONLINE       0     0     0
            gptid/3fcbb00f-a6a2-11e6-af00-0cc47ae11ad2  ONLINE       0     0     0
            gptid/40db2b86-a6a2-11e6-af00-0cc47ae11ad2  ONLINE       0     0     0
            gptid/41e0d274-a6a2-11e6-af00-0cc47ae11ad2  ONLINE       0     0     0
          raidz2-1                                      ONLINE       0     0     0
            gptid/4c02eee9-a75f-11e6-9609-0cc47ae11ad2  ONLINE       0     0     0
            gptid/4cca29c3-a75f-11e6-9609-0cc47ae11ad2  ONLINE       0     0     0
            gptid/4dcbcfd0-a75f-11e6-9609-0cc47ae11ad2  ONLINE       0     0     0
            gptid/4e803c7f-a75f-11e6-9609-0cc47ae11ad2  ONLINE       0     0     0
            gptid/4f3fc79e-a75f-11e6-9609-0cc47ae11ad2  ONLINE       0     0     0
            gptid/4feab5b0-a75f-11e6-9609-0cc47ae11ad2  ONLINE       0     0     0
            gptid/50995aaa-a75f-11e6-9609-0cc47ae11ad2  ONLINE       0     0     0

errors: No known data errors

 

[caLLous]

Bought a D8000 case

The D8000 is something else, isn't it? I've filled mine now (including hdd cages on those 3 fan mounts on the back panel) and have overflowed into another case. I remember putting my original disks from the previous case in it at the start and thinking I'd never fill it. :\ It's properly spaghetti inside but I have tidied it up a bit recently.

 

[Deebs]

The D8000 is something else, isn't it? I've filled mine now (including hdd cages on those 3 fan mounts on the back panel) and have overflowed into another case. I remember putting my original disks from the previous case in it at the start and thinking I'd never fill it. :\ It's properly spaghetti inside but I have tidied it up a bit recently.

Yeh very impressed with the case. I have a supermicro x11 board with an onboard Avago 3008 which I cross flashed to IT mode. This can manage 8 drives, I also have a 2008 card which is can also manage 8 drives.

 

[Scouse]

You using compression @Deebs?

 

[Deebs]

You using compression @Deebs?

Of course, lz4. It's enabled by default now in Freenas as it is extremely lightweight and causes no considerable load on the cpu's.

 

[caLLous]

Yup there's no downside to enabling compression with ZFS. A lot of file types won't benefit from it but it's very easy going on the CPU.