Archive for the ‘RAID’ Category

Storage Volume Read/Write Speeds for Video Editing

Friday, October 18th, 2013

Summary:

  • I expect my new Crucial M500 SSD will satisfy my multicamera HD video editing requirements far more than my old 7-disk RAID.
    • In neither case should their bandwidths be the bottleneck for say one live (raw HD-SDI) HD channel or say 10 simultaneous ProRes files.
    • In addition, the SSD should avoid the Disk-RAID’s  issues over seek-time (latency). framentation, moving parts, noise, heating (unwelcome in summer) and power supply requirements.
  • That’s the theory …it waits to be tested …when I get a time-break to backup everything then install and test it.

Detail:

Some video bandwidth requirements:

  • Raw HD-SDI of 720p 25 frames/sec or 1080i 50 fields/sec: 188 MB/s == 1.5Gbps
  • ProRes: approx 15 MB/s == 120 Mbps
  • XDCAM-EX: VBR, around 4.4 MB/s == 35 Mbps

Sustained sequential (as video ought mostly to be) data read/write speed estimates:

  • RAID of 7200 rpm disks:
    • 7 x raid5 plus 1 hot-spare: around 600 MB/s == 4.8Gbps
    • In my case, I get 400 MB/s == 3.2Gbps.
      • That’s around two live HD channels or 25 ProRes HD files, though in practice one would expect the need for headroom-margin, hence say one live HD channel or 10 Prores files?  Not bad.}
  • SSD:
    • For my Crucial M500 960GB Laptop-internal SSD:
      • SeqRead: Over 375MB/s == 3Gbps
      • SeqWrite: Over 500 MB/s == 4Gbps
    • And no issues over seek-time (latency) or framentation or moving parts or noise or power supply.
  • USB3
    • 625 MB/s == 5Gbps
    • 7200rpm disk > USB3: 110MB/s == 880 Mbps
  • Local 7200 rpm drive:
    • 40-100 MB/s == 320–800 Mbps, for most modern drive types.
  • NAS: 100MB/s == 800 Mbps advertised, under 50 MB/s == 400 Mbps  in practice.
    • But there can also be latency issues.
  • USB2:
    • 60 MB/s ==”480 Mbps” in theory…
    • …but in practice, as seen by user, is more like 38 MB/s == 300 Mbps.

(more…)

Shared Storage Options for Windows & Mac Video Editing Collaboration

Friday, October 18th, 2013

In summary:

There’s no magic option, each workstation needs a local storage volume with block-level data access (as opposed to simply file-level access) and formatted to a file system that is native (doesn’t require translation) to that workstation’s operating system.  Migration and collaboration imply file copying/synchronization, which implies read-access to the “foreign” file-system.  Mac OS can read NTFS, Winows can only read HFS+ via third-party add-on utilities.  Furthermore, for speed and responsiveness appropriate to video editing, the local storage should ideally be RAID or SSD.  In either case, it is possible to split the local storage (e.g. via partitioning) into more than one file-system.  At least, that worked on the mutiple occasions I have taken that approach, and have not been aware of any issues.

In greater detail:

Consider the challenge of setting up a shared data storage volume (e.g. RAID array or SSD) for video editing, such that either Windows or Mac computers can connect to it, and a video project started on (and saved to) on one of those operating systems (OS) can be continued on the other (and vice versa).

My current solution is to split the drive into separate volumes, one for each OS.  For example I have done this on RAIDs of various kinds and on an internal drive for Mac systems bootable to either Mac OS or (via Boot Camp) to Windows.  In the case of RAIDs I was advised against this by my system supplier, but got the impression they were just being defensive, not knowing of any definite issues, and to my knowledge I did not experience any issues.

It is is not practical to have just one volume (necessarily in that case, one file-system format), because:

  • Mac OS on its own is able to read NTFS but cannot write to it.
    • This is a show-stopper.  Some of the major video editing applications (e.g. NLEs), slightly disturbingly, may use (or for some functionality, even depend on) read/write access to source-files and the folders containing them.
      • I initially, naively, imagined that video editing systems etc. would only ever read source media files, not write to them, or to the folders containing them.  However that proved very naive indeed…
        • In Apple/Mac’s (erstwhile) Final Cut Pro 7 I regularly used their (moving) image stabilization effect, SmoothCam.  Its analysis phased was typically slow and heavy – not something one would wish to repeat.  The result was a “sidecar” file of similar forename to the analyzed source file, but a different extension, placed in the same folder as the source file.
        • I’m not certain, but got the feeling that maybe the source file (or folder) meta data, such as permissions or somekind of interpretation-change to media files in the quicktime ([.mov] mmedia format.
      • Certainly, Adobe (on Windows and Mac) could adulterate both files (by appending XMP data – being an Adobe media metadata dialect in XML) and the folders they occurred in (depending on uder-configuration) in terms of sidecar-files.
      • Sony Vegas also generates sidecar-files, e.g. for audio peaks.
  • File system translation add-ons can add Windows read/write access to HFS+ (ordinarily it could not even read it) and add Mac OS write access to NTFS (ordinarily it could only read it), but not sufficiently transparent/seamless for big real-time data access as required for demanding video editing endeavours.
    • File system translation add-ons (to operating systems) exist, such as MacDrive, to allow Windows to read/write Mac OS, or Tuxera NTFS, Paragon NTFS or Parallels for Mac to enable it to read/write NTFS, but these (reportedly, and in part of my experience) only really work well for standard “Office” type applications, not so well for heavy (big andd real-time) data applications such as video editing, where they can impede the data throughput.  Doh!
    • Some people have experienced obscure issues of application functionality, beyond data-movement speed issues.
    • {Also, I am concerned over the (unknown/imagined/potential) risk that the “alien” operating system and/or its translation utility might alter the file system in some way that upsets its appearance to the “home” operating system.}
  • FAT is universal but is a riskier option:
    • FAT is un-journaled, hence risks loss not only of individual files but of whole volume (integrity).
      • In video editing, corruption could be disastrous to a project, not only in terms of possible data-loss or time wasting and project delays on data recovery, but also in terms of “weird” effects during editing, such as poor responsiveness to commands, whose cause the user may not appreciate. or even an increased risk of unacceptable flaws in the final product.
    • FAT32 is essentially obsolete, because its maximum file size is (1 bit under) 4GB.
    • exFAT, a kind of “FAT64” is practical, and indeed a big successful corporate Mac-based production company once supplied me with many GB of footage on an exFAT-formatted external disk.
      • The largest file I have so far stored there is 40GB.  No problems.
  • NAS (Network-Attached Storage) sounds at first an easy option, but in my experience they impede big real-time data throughput (as stated earlier for “file system tyranslation” add-ons). Double-Doh!
    • Such devices only permit file-level access.  Consequently, the client systems can e.g. create or retrieve folders and files, but cannot e.g. format the device or address it in terms of lower-level data structures.
    • A likely explanation for the “impedement” of a NAS (to data responsiveness and throughput) is that such devices store in a local format (typically they run linux) that is invisible to the client, then translate to an appropriate protocol for each operating system accessing it.  They normally incorporate a bunch of such protocols.  As always, translation => overhead.
    • Other options, such as SAN and iSCSI, instead of providing file-level access to the client systems, instead offer the lower level of data block access.  Thus they appear to the client system as would any local storage device, and can be formatted as appropriate to the client system.
  • One suggestion I saw was to use a Seagate GoFlex drive, which can be used (read/write) with both Mac and Windows.  But the supplier’s FAQ (about that drive) indicates that it depends upon a translator utility for the Mac:
    •  If you would like to be able to “shuttle” data back and forth between a Mac and a PC, a special driver needs to be installed onto the Mac that allows it to access a Windows-formatted drive (i.e. NTFS). Time Machine will not work in this case, nor will Memeo Premium software for Mac. However, if you want your GoFlex solution to also work with TimeMachine, the drive will need to be reformatted to HFS+ journaled.

So I guess there is no “magic storage” option, my main work setup will have to remain based on separate volumes for each OS.

When transferring an editing project from one OS to another, the following actions will be necessary:

  • Copy any absent or updated files across.
    • e.g. via a file-synch utility such as Syncovery.
  • Allow time etc. for possible file re-linking, re-indexing, re-preview generation, re-“SmoothCam” (or equivalent).
    • This aspect is down to the editing application etc., as opposed to the operating or file systems themselves.
  • Ensure any effects used in the edit are present on both systems.
    • If so then these should presumably still work…

(more…)

External Storage Devices and their Bandwidths

Saturday, September 1st, 2012
  • Summarised info from http://forums.adobe.com/thread/784220?tstart=0
    • Average transfer rates in MB/s for different interfaces:
      • USB2:    20 – 25, depending on other USB devices sharing the same bandwidth
        • archiving for storage only
      • FW400:  30 – 35
        • archiving for storage, and light editing to/from (just very slow)
      • FW800:  50 – 60
        • archiving for storage, and regular editing to/from (fairly fast)
      • USB3:    65 – 80, depending on other USB devices sharing the same bandwidth
        • (no experience)
      • eSATA & SATA: 100 -140
        • archiving for storage, and regular editing to/from
    • RAID Speed-gains over a single disk:
      • RAID0:    0.9 x N disks over a single disk
      • RAID3/5: 0.8 x (N-1) disks over a single disk for read, 0.6 x for write.
        • ICHR10 figures are a bit lower than hardware controllers.

USB3, eSATA, Thunderbolt: Comparison: I like the look of eSATA

Sunday, August 12th, 2012

RAID / NAS etc Ideas

Saturday, July 7th, 2012

Some RAID ideas:

Mobile Video Editing Hardware: Thoughts, Ideas & Dreams (continued)

Friday, January 13th, 2012

Following-on from my earlier post, Mobile Video Editing Hardware: Thoughts, Ideas & Dreams, where I considered an eventual migration from my laptop to a luggable PC, my thoughts veered (possibly having spotted cash-icebergs among them) towards an alternative solution:

  • Use the laptop for lightweight editing & compositing.
  • Use the desktop as a number-crunching RADI-attached server.

The two could be linked by:

  • Remote access / remote sessions (some of which via smartphone)
  • DropBox, e.g. have an active folder where I can drop Adobe Premiere XML and have it processed remotely by Adobe apps installed there.

Some links:

  • http://www.theregister.co.uk/2012/01/11/virtualisation_at_home_part_2/
    • (There’s no equivalent “_part_1” page.  I guess it’s just “Part 2” of that guy’s story).
    • DIY virtual machines: Rigging up at home, by Trevor Pott, 11th January 2012 14:33 GMT
    • Personal Virtual Machine (PVM) (in use) for about seven years with retail boxed version of Windows XP.
    • VM has been moved from virtualization platform to virtualization platform over the years … the most recent incarnation … inside Hyper-V.
    • …nothing beats Windows Server 2008 R2. It comes with a top-notch virtualisation platform (Hyper-V), and added RemoteFX support with Service Pack 1. You can still use the desktop operating system for all your HTPC needs, and a single Server 2008 R2 Standard license allows you to run both a host copy and a single virtual instance of Server 2008 R2.
    • In my case, the host instance does little more than play movies on the projector via VLC. The virtual instance of Server runs my Plex media server, and aggregates my many storage devices into a single share using DFS.
  • Shuttle Inc (Taiwan)

Mobile Video Editing Hardware: Thoughts, Ideas & Dreams

Tuesday, January 10th, 2012

Want a mobile “suitcase” editing system, something more (and more expandable) than a laptop but not too expensive.  Primarily to be used for Adobe CS5.5 for media enhancement / editing / compositing etc.

Nearest I found was NextDimension’s range around $7000 I think (but just guesswork – could be way off – would need to get a quote).   That would (if true) be around £4500 at current rates.  Plus import…  NextDimension call such machines “flextops” (Maybe they coined the term? Google searches on it mostly come up with them.)

Apart from the (mil/broadcast-lite but me-heavy) price, it might possibly be undesirably heavy to lug around much.   If so (just guessing, not assuming), it would make more sense to go for a modular quick-setup system.  So, starting to “think different” in this direction:

  • Standard tower, capable of taking new CUDA etc. graphics cards etc. as they emerge, but no need for more than say a couple of disks, maybe if SSD could even get away with just a single disk? (For system and media – inadvisable for traditional disks of course, what about for SSD’s?  I have much to learn about SSD’s though).
  • “Laptop-Lite” to talk to it.  With robust shuttered-stereoscopic HD monitor.
  • Gigabit network to NAS fast storage (SSD and/or RAID ?).

Maybe in that case it would be far more logical/affordable to use an existing laptop as a client working together with a luggable tower server, sufficiently light and robust for frequent dis/re -connection and travel.  And remote access of course (no heavy data to be exchanged, assume that’s already sync’d).  And some means to easily swap/sync applications and projects (data) between laptop and tower, giving the option to use just the (old) laptop on its own if needed.  All such options are handy for the travelling dude (working on train, social visits etc.) who also occasionally has to do heavy processing.  Then would just need a protective suitcase for the tower, plus another one for a decent monitor for grading etc.

I certainly won’t be spending anything just yet, but it’s good to have at least some kind of “radar”.

(more…)

Laptop-Based Mobile Editing: GRaid Mini (Out-Shines “Passport” Drive)

Wednesday, November 9th, 2011

Video-editing on-the-move (e.g. on a train) using a MacBook Pro (laptop) with Sony Vegas 9 (64-bit) as NLE (under Boot Camp / Windows 7), my practical experience was that a GRaid Mini external drive was far, far better than a 5400 rpm Western Digital “Passport” drive.   Consistent with the dual use of the MacBook, I partitioned the drive for both NFTS (Windows) and HFS+ (Mac OS), 50-50%.  Due to Boot Camp limitations (explained below), up till now I only ever used it “tethered” to its own mains-based power supply.  But now I see it can also be used mobile, powered from the MacBook – something that up till now I could only achieve under Mac OS, not under Windows.

When using Boot Camp / Windows on the MacBook, I initially tried the shirtpocket-sized Passport drive because it was small, light and powered from the laptop’s USB port.  While its data throughput wasn’t too bad, at least for single-channel HD editing (especially when only 1280720), when it came to cuts from one video clip (hence, in my case, video file) to another, there was a frustrating delay every time.

I also have a GRaid Mini drive, but it wasn’t obvious at first how to use it mobile when using Windows (on a MacBook).  That drive consists of two 7200  drives in RAID-0 configuation (striped, giving speed but no redundancy), and appears just like any single drive to the computer (no RAID management etc. needed).  The drive has not only a USB (2) port but also FireWire 800 (FW800) and eSATA ports.  While the latter two options work fine with the MacBook under Mac OS, they don’t work under Boot Camp / Windows.  I have tried many times and trawled many forums, no solution is apparent.  Under Mac OS the eSATA drive would ordinarily plug into an ExpressCard adaptor plugged into the laptop’s ExpressCard slot, but under Boot Camp / Windows, the ExpressCard slot doesn’t work, while for the FW800 port under Boot Camp / Windows, it appears to work at first but eventually crashes as a device when it attempts to communicate data (e.g. when copying files).

When connected only by USB to the MacBook under BootCamp / Windows, the GRaid Mini is not powered from that port, hence up till now I have relied on a mains power supply to that drive.  However, I discovered if, after first connecting by USB, you subsequently connect also by the FW800 lead, then the drive takes power from the FW800 yet communicates data via the USB lead.   Hooray!  I can use it on-the-move then!

The order in which the leads are connected is vital.  If by mistake the FW lead was connected first, then the drive would sense that as the data communications route, and subsequently fail in use.  It is vital that the USB connection is made first.  Likewise, on disconnecting the drive (following “ejection” by the computer’s file-system), disconnect the FW drive first.  The rule is FW lead: connect last disconnect first.

My experience of editing with the GRaid Mini is far more fluid hence more pleasurable and efficient.  Totally worth it.  None of the per-cut delay effects of the 5400 rpm Passport drive.  And now it can be used on-the-move, even with Boot Camp / Windows on a MacBook.  I just wish Apple would fix that Boot Camp isue with FireWire and ExpressCard ports!

RAID Sharing Over Home Network: Works OK

Thursday, October 6th, 2011

Just for the record, currently this works:

  • On Mac Pro, in Windows (XP):
    • Choose any folder. For example:
      • On the RAID, this could be either of the two partitions:
        • NTFS_RAID (an NTFS partition)
        • RAID_ProAVIO (an HFS+) partition
    • Then [RtClk > Sharing and Security… > Sharing > Network sharing and security:
      • Share this folder on the network: YES
      • Allow network users to change my files: YES
        • Because some tools create “sidecar” files e.g. audio peaks or video indexes.
  • On Mac Book, in Windows 7:
    • From a file-browser, be it Windows Explorer or part of the File>Open feature of an application, go to [ Network > aComputer > aFolder ]
      • Tested by playing file in Windows Media Player and in Sony Vegas (video NLE).  Worked fine.  Latter added a [.sfk] sidecar-file to the RAID-folder.
      • Worked both for source on NTFS partition and for source on HFS+ partition, except the latter gave rise to prolonged “hourglass” delays before the file was accessed/linked, following which the video played smoothly.
  • What doesn’t work:
    • Unable to see Mac Book from Mac Pro.
    • Also, when MacPro is in Mac OS, MacBook W7 Windows Explorer > Network lists the MacPro as [MACPRO-2E4124] yet cannot connect to it, blaming a firewall (presumably the one on MacPro).
      • Network Error: Windows cannot access \\MACPRO-2E4124
      • Tried exiting Kapersky (on the MacPro>MacOS) but no change.
      • Tried stopping MacPro firewall (via Preferences > Security) but no change.

RAID Health-Check: Any Issues?

Friday, September 2nd, 2011

My Mac Pro system has a ProAVIO RAID via HighPoint RocketRaid card RR2522.  Although it’s a Mac, mostly I’ve used it for Windows XP under Boot Camp.  Recently, in that mode, it’s exhibited an unusually long delay when accessing the RAID, e.g. when clicked on in Windows explorer.  So, I logged in to HighPoint’s  web-based interface for the RAID.

  • I discovered (the hard way) that this interface only works properly when the computer is in Mac OS mode.
  • Once logged-in, the Event (Log) listed only some bad sector etc. repairs on two specific dates, back in July.  There’s nothing more recent.
  • So I guess (?) the “unusually long delay” is something outside the RAID, such as the Windows (XP Pro SP3) system and/or the Kapersky antivirus.

(more…)