A Few Upgrades

A few updates to MediaServer8 this weekend. I replaced my 250GB cache drive with a Crucial Crucial 240GB M500 SATA 6Gb/s 2.5 Inch Internal Solid State Drive. My unRAID cache houses mainly Xen VMs and these have become noticeably faster to boot and peppier overall. On some internal transfers from unRAID to the drive, I see speeds of 150 Mb/s which is a big improvement on the spinner previously doing cache duties.

I also added an extra DVB S2 dual tuner to the system meaning I now have 6x tuners (4x DVD S2, 2x DVB-T) which is more than enough to service current live TV and PVR requirements.

6x Tuners in ArgusTV running in Windows 7 VM on MediaServer 8

Also, my X10 CM15A USB interface arrived so I can now control various lights and appliances around the house from a VM. My Christmas project is to set up a web server and application  to allow lighting and whole house audio control from tablet and laptop clients.

HTPC as PreProcessor

I’ve been living with my old school (analog) Pioneer VSX-D2011 receiver for well over 10 years now. It’s 7x100W channels have served me well in my HT but it’s very old in terms of format support, connectivity etc.

This hasn’t bothered me too much as I’ve always had a HTPC to look after playback etc. but it’s always been a stop-gap solution as I think about a receiver upgrade or something else.

Now, I’m hankering after a speaker upgrade and whatever I go for will likely need more power than a 7-channel receiver can provide, so I’m thinking separates. That being the case, I need to purchase a stand-alone AV processor or seriously look at using the HTPC as processor.

I decided to look at the HTPC as it would require the least up-front investment and I can determine if it all works with current equipment b before investing in speakers & amps.

Registering Xen VM with UnRaid

Each time I create a new VM in Xen on UnRaid, I can never recall the command to register it so that it shows up in the Xen management interface.

Thanks to a discussion on the UnRaid forums, I've now found it again and am posting here for posterity;

xenman register /mnt/cache/domains/vmfolder/vmname.cfg

System Transplant

For a few reasons, I wanted to move away from my tower-based media server;

First, I wanted to move the server from my attic to a rack as I'm soon going to be adding a new wider screen to my home theatre and this will make attic access a little more difficult than at present.

Second, with hard drive capacity increasing rapidly, I really don't see a use for oodles of drives in a system - all I need to do is gradually replace my 1TB drives with 4TBs and I'll have move from 12TB TO 27TB, 37TB if I go with 6TB drives. So I really don't need any more space.

Given the above and my recent addition of an expansion chassis, I settled on 2x SilverStone cases; an SST-GD07B for MediaServer8 itself and a Grandia GD09 for the expansion chassis.

Before transplant - the recipient is a good deal smaller than the donor

Rsync + Screen

More on copying, this time on the linux common line.I'm in the process of removing some smaller drives from my unRAID array and converting the remaining drives to XFS. To do this requires a good deal of data shuffling.

It's best to do this directly on the unRAID OS so as to remove any network operations. I'd never liked using Midnight Commander so decided a cp operation was required. However, two problems; cp doesn't show progress and doing this over ssh is risky as if the connection or client system dies, the copy operation dies.

Windows 7 FastCopy

I had cause today to copy ~440GB of data between two drives on my Windows 7 VM. Both drives were assigned to the VM as PHY: devices and the OS had PV drivers installed but even so, the copy operation was painfully slow with speeds of less than 20 MB/s and an estimated copy time of 6 hours.

Without the time or inclination to troubleshoot why the transfer was so slow, I did a quick search for windows 7 fast copy and came across a tool called, handily enough, fastcopy.

I installed and fired it up and set it to do the same operation. It finished in 1.5 hours with an average speed of ~70MB/s. Will be using this for similar operations in future.

Now to figure out why the OS copy operation was dog slow...

Current SetUp

MediaServer8 + Expansion Chassis In-Situ

Following on from my post on putting in place an external expansion chassis, I'd ordered a few cables to make the set-up a little more workable.

Specifically, a DVI cable with a 90-degree connector on one end now allows me position the expansion board correctly in the external chassis so that all slots are now externally accessible. I've also installed an eSata backplane that has two Sata cables internally. This allows me connect two hard drives in the expansion chassis to the MediaServer8 eSata ports.

The photo above shows MediaServer* and the expansion chassis positioned in my attic. This location is close to the termination point for all cabling in the house (Cat5, Speakers, RF) as well as telephone/fibre entry points. It's all a little messy right now, I need to get some order on those cables!

Netrunner

One of my longer term goals has been to set up a general purpose workstation VM on MediaServer8. I've toyed with a few Linux variants but recently decided to try Netrunner. The 'rolling' version  is an ArchLinux distro with KDE based on Manjaro.

It was a bit fiddly to set up and get GPU passthrough working. This is not aided by the fact that the display does not switch on until very late in the boot process so unless running via VNC, you can't see what's going on.

A little coaxing and persuasion, however, combined with a good deal of patience paid dividends and I got a consistently reliable boot with GPU passthrough.

As an added bonus, I discovered that I can even connect two displays to the HD5450 card and both will be driven for an extended desktop.

unRaid 6beta10

Nice to wake up this morning to a fresh build of unRaid. And easy to install as well. Previous updates required manual download and installation but this one shows up in the UI and it's just a matter of clicking the update link (and restarting the system).

This is not an earth-shaking release but it does have a couple of UI changes that are undocumented. Most noticeably, the links to the VM management tools are now moved to the main menu;

Secondly, the output of the newly-named 'System Devices' function under tools now includes discrete listings of USB and disk devices in addition to LSPCI output.

It would be nice if the updated was notified via the UI (rather than having to go into 'Plugins' and clicking the 'Check for Updates' button but otherwise this seems like steady progress.

UPDATE: Beta10a is now available - some misc fixes for Xen auto-start and other bits.

Thinking Outside The Box

Right now I've got my system stacked with expansion cards;

1x GPU for unRaid booting
3x GPUs for passing through to VMs
1x 2 port SATA expansion card
1x Octopus Tv tuner PCIe card
1x Octopus expansion tuner (no slot required)
1x PCI M-Audio Delta card

The Gigabyte 990FXA-UD5 is a great board with 5x full length PCIe slots, a single 1x PCIe slot and a single PCI slot.

However, there's a problem: the 1x PCIe slot is tight up against a heatsink and any cards with any overhang at all simply won't fit. The ideal card in my setup to occupy this slot is the Octopus TV tuner but I can't get it in there. Therefore, I've got that card in a 4xspeed  full-length slot and a 2-port SATA expansion card in the 1x PCIe slot.

The Perils of Beta Testing

Regular viewers will be aware that I've been using beta versions of unRAID 6 to run multiple virtual machines on MediaServer8.

I haven't been cursing edge with the betas, mainly due to time constraints, plus I was stuck on beta5 for a while as there as a kernel issue in beta 6 that prevented it from recognising my Marvell motherboard SATA controlllers.

However, I have been updating regularly and most recently installed beta 7. This allowed me install BtrFS on my cache and install multiple drives to cache as well. Indeed I'd done this to play around with Docker which up to beta 7 required BtrFS. Frustratingly, this requirement was removed in Beta8 so my time was somewhat wasted setting that up, or so I thought.

Upgrading unRaid cache to btrfs

Having upgraded to unRaid 6 beta7, I've started having a look into the new Docker support. In unRaid, Docker needs to be installed on a btrfs volume or pool. Btrfs is a new filesystem to unRaid and supports 'copy on write' which enables Docker utilise minimal hard drive space. (see the Docker intro thread on unRaid forms for full details.)

My existing cache drive was a seagate 500GB and contained all my VM images and config files in a 'domains' directory. I copied these files off to a backup drive and set to work.

It was a good thing I made a backup as it soon became apparent that it would be very easy to make mistakes and wipe a lot of data!

unRaid 6beta7 Update

The next beta of unRaid is out about a week and today I got around to installing on MediaServer8. I'd been on beta5 for some time as there was a problem in beta6 that prevented the Marvell SATA controllers on my motherboard working. Happily, this is now fixed in beta7.

I'd made a few adjustments to my TVServer Win7 VM to reduce the RamDisk size. Happily, on reboot, the RamDisk re-instantiated and the timeshift user share for activated automagically. This had been once concern for me - how would the RamDisk behave on VM restarts. Perfectly, as it turns out.

So this is all good news. I can now start playing with the new Docker features of unRAID and start thinking about whether Xen is the correct VM platform as KVM is now an option for me.

Ramdisk for Timeshifting

I've added an extra 16GB of memory to MediaServer8 to now give a total of 32GB.

This has allowed me increase the assignment of memory to my Windows7 TVServer VM from 4GB to 16GB.

I want to experiment with using a ramdisk for timeshifting rather than a physical disk. A ramdisk is a virtual disk that exists only in memory. The advantage is that it's superfast, the disadvantage is that the contents are destroyed on each restart. However, this is not a big deal for timeshifting as there are only transient files stored.

For me, using a ramdisk rather than a physical disk also means that I can remove the 75GB Raptor I've assigned as a timeshift drive, freeing up the SATA port and physical bay for another storage drive. So all in all, it should be win-win.

MediaPortal vs XBMC

In a moment of weakness, I installed XMBC Gotham (13.1) to see what had changed since I used version 12. I was also interested in the MediaBrowser XBMB3C Add-On that integrates MediaBrowser right into XBMC.

We've been having occasional freezes and crashes in MediaPortal clients over the past few weeks so I guess I'm also on the lookout for something that might bring a little more stability.

unRAID v6 beta6

There's a really big release from Lime Technology this week; unRAID 6beta6 has lots of new goodies including support for KVM and Docker virtualisation (in addition to the existing Xen capabilities), lots of updates including linux kernel  as well as initial steps towards a new file system (btrfs is now supported on the cache drive).

My own system is currently working well on b5a and, while there's a lot here I'd love to play with, I think I'll be letting some others iron out the initial wrinkles so will probably upgrade to beta 7 when it arrives.

The other cool thing is we're seeing a lot more communication from LimeTech staff on the forums and a real effort to support this release with how-to guides and the like.

Looking forward to upcoming releases.

Latest GPLPV Drivers

This weekend I've started playing about with MediaBrowser 3. Looks like it will bring a lot to the party for me in that it integrates well with MediaPortal, provides the client/server model I miss from Plex and does media transcoding for handheld device playback.

However, this latest feature wasn't working well for me. I'd installed MediaBrowser Server in my TVServer Win 7 VM but when I tried to play back media from devices requiring transcoding (iPad, web browser) there was very poor performance, a lot of stuttering and general failure.

Even when I attempted to play back an audio track, it was skipping and pausing.

However, this reminded me of an issue I had when I had Logitech Media Server installed in the same VM. I'd get skipping on local tracks and a lot of buffering playing tunes from Spotify.

Le Roi est mort. Vive le Roi.

Readers here will have figured out that I'm a fan of unRAID and have been using the new Xen virtualisation features of v6 beta to bundle all kinds of of interesting functions into a single server.

In a mildly shocking turn of events, it has come to light that mid beta, LimeTech are considering ousting Xen in favour of KVM and/or Docker.

I started my virtualisation journey late last year with KVM and liked it a lot and was slightly disappointed when Xen became the platform of choice for unRAID. However, in the past months I've come to know it and it works great for me.

There are a lot of good reasons why the change is necessary and LimeTech representatives on their forums have promised to detail the reasoning and plan soon. However, it looks like we'll be changing tack in the coming months.

Current Build

Over two years ago, I wrote a blog entry that outlined my objectives in embarking on the MediaServer8 project;

- Centralised storage for media files
- Time Machine Backup Server
- Work File Storage
- Snatching
- Media Playback
- Whole House Audio
- Live TV Distribution / PVR

At the time, I had an unRAID  NAS system for file storage and Plex serving and was contemplating a move to Windows Server 8. Now, I've got pretty much all of the above set up in a single machine, and more.

I've been interested in whole-house a/v since we started planning our renovation over 12 years ago and I've been through system after system, technology after technology. Some worked well, some not at all. However, all were defined by a lack of unity. I could have a great HTPC system but it wouldn't integrate well with whole house audio or lighting control or mass storage or whatever.

If I was building again...

If I was re-building my unified whole house media server today, I think I would be using the following core components;


Gigabyte 990FXA-UD7 Motherboard
This is a step up from the UD5 currently in my system but crucially replaces the PCIe x1 slot with a full-length slot. This would allow me add an additional PCIe graphics card and therefore a 3rd client system with GPU passthrough.

AMD FX8350 Black Edition 8 Core Processor (4.0/4.2G)
This is the CPU I have right now and have no problems with it whatsoever. It's running the base unRAID / Xen OS plus multiple guest VMs with ease. Very impressed.

A Productive Weekend

I got some more time this weekend to play with the MediaServer8 box and to address a few issues;

I expanded my array from 6TB storage to 11TB with the addition of a 4TB parity drive (Seagate NAS), a 4TB data drive (HGST) and an additional 1TB WD Green. My case is now full with 12 drives, 10 dedicated to the array and 2 for TV timeshiftimg and storage.

With 4TB in place as parity, I'm set to gradually replace all my 1TB drives meaning I'll be able to get up to 32TB with my current set up.

I transferred the replaced 1TB parity to act as parity in my old 32 bit Xeon unRAID server which is now packed with 500GB drives. It's acting as a backup of critical data such as work files etc.

While I had the system out on the bench, I took the opportunity to do a spot of re-cabling, tidying up some unsightly runs and fixing an issue I had with power intermittently failing to my satellite tuner cards. (turned out that it was a poor molex adapter where pins were not seated correctly - what a crappy standard that is!

My Next Case...

So it's time to start saving the pennies. I've found what I think is the ideal case for my unRAID set up.

The Cooler Master HAF STACKER 935 is two stacked cases -with the ability to add on even more. With the default config, it has the potential to hold 20 x 3.5" drives plus a 2.5" (using a 5-in-3 in the 5.25 bays) and with an additional 915 unit, could be pushed to an insane 29!

Here's a a somewhat lengthy but detailed unboxing and review video that gives a good idea of this units potential;

System Power Consumption

Following a question over at unRAID forums, I popped a Watt meter onto the system and have the following readings;

• 250W at boot
• 160W no VMs running, unRAID array spun up
• 120W no VMs running, unRAID array spun down
• 160W VMs running, unRAID array spun up
• 135W VMs running, unRAID array spun down
• 240W Heavy Usage  (2x Live TV  & One recording in progress, 2x Music Zones active, some general data access from array)

The system is averaging around 160W, peaking at 240W. Considering my previous unRAID ran at a constant 160W for data storage only, I'm pretty happy with this as I've folded in at least two additional machines to this rig.

Virtual Machines running on unRAID 6

More Case Mods

Following the little bit of case surgery previously noted, the migration of my production unRAID server to MediaServer8 required a little more gentle persuasion.

With unRAID version 6 beta 4 available, I thought it was time to transfer all my data from my old dual Xeon system. I had 8x 1TB data drives in the old server plus 1TB parity and 500GB. However, MediaServer8 currently only has 8 motherboard SATA connectors (plus a 2x port PCIe hard that's running recording and timeshiftimg drives for ArgusTV/MediaPortal).

So I needed to remove 2x data drives from my array. This is not such a big deal as I plan to upgrade drives to 4TB units over the coming months. I duly copied the data from the drives with least data and removed them from the array.

I ran a parity check on the old server and with all well, removed all the HDs. I downloaded unRAID 6b4 and, having saved my config directory, reformatted my MediaServer8 USB key, installed unRAID and copied back my config directory.

Whole House Audio - Step 2

Some time ago, I wrote a post on the foundations of a cheap DIY whole house audio solution. I haven't really progressed that idea much until this weekend when I got back to it.

I still have my squeezebox and squeezebox radios even though they're now discontinued. I also still have the Logitech Media Server driving them and Triode's plugin providing access to a premium Spotify account. Now with the unRAID system up and running, it was time to create a VM to act as MusicServer.

Xen VM host hard drive access

I've now got my unRAID based Xen system fairly stable but getting my VMs to directly access physical hard drives has been a bit of a learning curve.

I have a Windows 7 VM that runs Mediaportal / ArgusTV server installs with my Octopus tuner cards passed through. This VM is my 'tvserver' and manages all TV scheduling, tuning and streaming.

I also have a Windows 8.1 client that runs MediaPortal and Netflix with GPU and USB passthrough. This drives my lounge TV via HDMI out from an AMD HD5xxx video card.

My biggest issue though has been arranging storage on the Win 7 tvserver for both timeshifing and recording storage.

Both VMs currently boot from .img files stored on my unRAID shares. I could set up another .img file there but don't want to as I'm sure I'll hit performance issues as the unRAID share is parity protected and thus has poor enough write speeds. 9I don't have a cache drive yet as I'm still in test build mode on a basic unRAID licence.

Xen USB Passthrough

I've been having some trouble passing through USB controllers to Windows VMs running on my unRAID 6.0beta3 Xen host.

Specifically, passing through the full USB controller using PCI Passthrough, I was seeing errors similar to the following when creating or destroying VMs;

libxl: error: libxl_pci.c:1001:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:12.2 

USB access in the VM would be hit or miss and I'd often get an error on Dom0 indicating that IRQ #18 had been disabled.

A little case surgery

I'm beginning to run out of expansion card space in my system case with multiple graphics cards, tv tuners and hard drive controllers already installed and possibly more to come.

While my motherboard has 6x PCIe slots and a PCI, it's starting to get tight in there.

I've gone for the Octopus system from Digital Devices for TV & Satellite tuning. This  allows me attach up to 4x dual tuners to a single PCIe x1 card. It's possible to fit two cards per full-height bracket but even at that, I'm occupying a valuable space with a card that doesn't need to plug in to a slot.

unRAID 6.0 Xen Guest GPU Passthrough

It's been a blast following the progress over at unRAID forums as the build up to and release of 64 bit version 6.0 took place. I personally learned a lot virtualizing unRAID in KVM on ManjaroBox but the latest unRAID beta (v 6.0b3) is now out and allows for hosting of virtual machines in unRAID itself thanks to the enabling of Xen.

A big part of my requirement is to establish two or more virtual HTPCs. For this to work, it's necessary to 'pass through' physical graphics cards to the virtual machines.

Here's how to do it;

unRAID 6.0 VM speed comparisons

unRAID 6.0 Beta 1 (64 bit) arrived this week and I managed to get it up and running in a VM on my new server under KVM in ManjaroBox.

For fun, I did some speed comparison checks when accessing this virtualised unRAID compared to my existing unRAID server running version 5.0.4 natively on a 32 bit Xeon based system with PCI-X data host controllers.

OSX tests were conducted from an iMac on the network. The Win7 tests were conduced from a Windows 7 Professional VM on the same KVM host as the virtualised unRAID instance.

Here's the results.

DOWNLOADS

	from current unRAID	from virtualized unRAID
OSX Desktop	14.5 MB/s	67 MB/s
Virtual Win 7	11 MB/s	104 MB/s

UPLOADS

	to current unRAID	to virtualized unRAID
OSX Desktop	14 MB/s	35 MB/s
Virtual Win 7	11 MB/s	33 MB/s

Gigabyte GA-990FXA-UD5 Port Addresses

As part of an effort to set up multiple Virtual Machines on a system based around the Gigabyte SKT-AM3+ 990FXA-UD5 Motherboard, I have mapped many of the controller addresses to help with passing through various ports to VMs. In particular, I'm interested in USB ports but I've included a few others as well.

I obtained the device addresses by using the lspci and lsusb commands in Arch Linux Xen kernel, systematically blocking each address / address pair at boot  and checking for the presence of a usb device (a mouse) which I plugged into each port in turn, running lsusb to see if the device was listed.

My efforts have resulted in the following table where device addresses are listed across the top and ports/headers are listed down the side. A green entry indicates that the port is active with that particular address blocked. A red entry signifies that that port is inactive for the particular address.

That is to say, red blocks show ports that are available for passthrough to VMs when the controller id is blocked.

	00:12.0 00:12.2	00:13.0 00:13.2	00:14.5	00:16.0 00:16.2	2:00.0	7:00.0	05:0e.0
Rear USB 2.0 leftmost bank
Rear USB 2.0 rightmost bank
Rear USB 3.0
Int f_USB1
Int f_USB2
Int f_USB3
Int USB 3.0
Rear Firewire
Int Firewire

Replacement Squeezebox Radio Power Supply

One of my SqueezeBoxes recently stopped working suddenly and I diagnosed a failed power supply. I ordered this one and can confirm it works great.

VNC Access to Xen Guests

So I managed to get Xen up and running on my Arch Linux box and installed a Windows 8 guest using virt-manager forwarded to an X11 session on my Mac. I even managed to get a PCI controller passed through for some of my USB ports which allow me use mouse and keyboard in the VM.Yay!

However, using the windows desktop in this way is very clunky as I find the X11 session slow and it has mousing problems. So, I've started bypassing virt-manager altogether and launching my VM using XL create and a config file.

This leaves the problem of how to access the VM desktop. The obvious solution is a VNC client but it's not so straightforward, Here's how I managed to get it working...

Standing on the shoulders of giants

This is a guide to setting up the Xen hypervisor on Arch Linux.

I've been having a whale of a time over the past 4-5 weeks learning as much as I can about virtualisation with the objective of configuring a unified server to run;

• unRAID
• mediaportal server (windows VM) with DVB-S2 tuner cards passed through
• at least one mediaportal client (windows VM) with GPU & USB passed through

I've bought and built a new machine for this and I've been playing variously with Linux distros such as OpenSuSe, Manjarobox and now vanilla ArchLinux. With all of the above I've tried both Xen and KVM as virtualisation hosts.

I've been enjoying the discussion on unRAID forums around virtualisation and the possibility of running unRAID in a full distro but am hedging my bets and want to become very familiar with setting up a system like this where unRAID can run either on the host system or in a VM.

The Great Rebuild

So right now my whole-house media infrastructure comprises;

• Windows 8 Quad-Core PC running MediaPortal TV server & client (lounge)
• Windows 8 Celeron based MediaPortal client (living room)
• Windows 7 i3 based MediaPortal client (home theatre)
• Dual Xeon based unRAID NAS

A good deal of bedding in has taken place and I really like MediaPortal. It gives me multiple terrestrial and satellite tuners as well as access to all my media in a single UI.

I don't like running this many computers though, and the virtualization discussions over on the unRAID forums have really got me hankering for an all-in-one solution.

So, I've purchased and assembled the following hardware with a view to consolidating some of the above machines;

• Gigabyte SKT-AM3+ 990FXA-UD5 Motherboard
• AMD FX8320 Black Edition 8 Core (3.5/4.0GHz, 8MB Level 3 Cache, 8MB Level 2 Cache, Socket AM3+, 125W, Retail Boxed)
• Arctic Cooling Freezer Xtreme Rev.2 CPU Cooler
• Asus ATI Radeon HD 5450 Silent Graphics Card (1GB, DDR3, PCI-Express)
• Corsair CP-9020054-UK RM Series RM650 80 Plus Gold 650W ATX/EPS Fully Modular Power Supply Unit

There's a push at the moment to get unRAID running on a modern Linux distro with full VM support (XEN and KVM). There are solutions that see it running in a VM but the idea of unRAID in dom0 appeals as there's no need to hardware passthrough of hard drive controllers.

Therefore, this new machine will run;

• unRAID in host Linux (arch, openSuSe, whatever results from the current discussions)
• Media Portal TV Server in Windows 7 VM
• Media Portal client in Windows 8 VM (lounge)
• Media Portal client in Windows 8 VM (living room)

That will free up my celeron client machine for use elsewhere and allow me donate the current MediaServer8 machine to my son for gaming.

This looks like it's going to be a complex project that requires passthrough of TV tuners, 2x graphics cards, onboard SATA and USB devices to multiple VMs (not simultaneously, obviously).

I'll be logging progress here. Wish me luck!