Archive for the ‘disk’ Category

Storage Volume Read/Write Speeds for Video Editing

Friday, October 18th, 2013

Summary:

  • I expect my new Crucial M500 SSD will satisfy my multicamera HD video editing requirements far more than my old 7-disk RAID.
    • In neither case should their bandwidths be the bottleneck for say one live (raw HD-SDI) HD channel or say 10 simultaneous ProRes files.
    • In addition, the SSD should avoid the Disk-RAID’s  issues over seek-time (latency). framentation, moving parts, noise, heating (unwelcome in summer) and power supply requirements.
  • That’s the theory …it waits to be tested …when I get a time-break to backup everything then install and test it.

Detail:

Some video bandwidth requirements:

  • Raw HD-SDI of 720p 25 frames/sec or 1080i 50 fields/sec: 188 MB/s == 1.5Gbps
  • ProRes: approx 15 MB/s == 120 Mbps
  • XDCAM-EX: VBR, around 4.4 MB/s == 35 Mbps

Sustained sequential (as video ought mostly to be) data read/write speed estimates:

  • RAID of 7200 rpm disks:
    • 7 x raid5 plus 1 hot-spare: around 600 MB/s == 4.8Gbps
    • In my case, I get 400 MB/s == 3.2Gbps.
      • That’s around two live HD channels or 25 ProRes HD files, though in practice one would expect the need for headroom-margin, hence say one live HD channel or 10 Prores files?  Not bad.}
  • SSD:
    • For my Crucial M500 960GB Laptop-internal SSD:
      • SeqRead: Over 375MB/s == 3Gbps
      • SeqWrite: Over 500 MB/s == 4Gbps
    • And no issues over seek-time (latency) or framentation or moving parts or noise or power supply.
  • USB3
    • 625 MB/s == 5Gbps
    • 7200rpm disk > USB3: 110MB/s == 880 Mbps
  • Local 7200 rpm drive:
    • 40-100 MB/s == 320–800 Mbps, for most modern drive types.
  • NAS: 100MB/s == 800 Mbps advertised, under 50 MB/s == 400 Mbps  in practice.
    • But there can also be latency issues.
  • USB2:
    • 60 MB/s ==”480 Mbps” in theory…
    • …but in practice, as seen by user, is more like 38 MB/s == 300 Mbps.

(more…)

Shared Storage Options for Windows & Mac Video Editing Collaboration

Friday, October 18th, 2013

In summary:

There’s no magic option, each workstation needs a local storage volume with block-level data access (as opposed to simply file-level access) and formatted to a file system that is native (doesn’t require translation) to that workstation’s operating system.  Migration and collaboration imply file copying/synchronization, which implies read-access to the “foreign” file-system.  Mac OS can read NTFS, Winows can only read HFS+ via third-party add-on utilities.  Furthermore, for speed and responsiveness appropriate to video editing, the local storage should ideally be RAID or SSD.  In either case, it is possible to split the local storage (e.g. via partitioning) into more than one file-system.  At least, that worked on the mutiple occasions I have taken that approach, and have not been aware of any issues.

In greater detail:

Consider the challenge of setting up a shared data storage volume (e.g. RAID array or SSD) for video editing, such that either Windows or Mac computers can connect to it, and a video project started on (and saved to) on one of those operating systems (OS) can be continued on the other (and vice versa).

My current solution is to split the drive into separate volumes, one for each OS.  For example I have done this on RAIDs of various kinds and on an internal drive for Mac systems bootable to either Mac OS or (via Boot Camp) to Windows.  In the case of RAIDs I was advised against this by my system supplier, but got the impression they were just being defensive, not knowing of any definite issues, and to my knowledge I did not experience any issues.

It is is not practical to have just one volume (necessarily in that case, one file-system format), because:

  • Mac OS on its own is able to read NTFS but cannot write to it.
    • This is a show-stopper.  Some of the major video editing applications (e.g. NLEs), slightly disturbingly, may use (or for some functionality, even depend on) read/write access to source-files and the folders containing them.
      • I initially, naively, imagined that video editing systems etc. would only ever read source media files, not write to them, or to the folders containing them.  However that proved very naive indeed…
        • In Apple/Mac’s (erstwhile) Final Cut Pro 7 I regularly used their (moving) image stabilization effect, SmoothCam.  Its analysis phased was typically slow and heavy – not something one would wish to repeat.  The result was a “sidecar” file of similar forename to the analyzed source file, but a different extension, placed in the same folder as the source file.
        • I’m not certain, but got the feeling that maybe the source file (or folder) meta data, such as permissions or somekind of interpretation-change to media files in the quicktime ([.mov] mmedia format.
      • Certainly, Adobe (on Windows and Mac) could adulterate both files (by appending XMP data – being an Adobe media metadata dialect in XML) and the folders they occurred in (depending on uder-configuration) in terms of sidecar-files.
      • Sony Vegas also generates sidecar-files, e.g. for audio peaks.
  • File system translation add-ons can add Windows read/write access to HFS+ (ordinarily it could not even read it) and add Mac OS write access to NTFS (ordinarily it could only read it), but not sufficiently transparent/seamless for big real-time data access as required for demanding video editing endeavours.
    • File system translation add-ons (to operating systems) exist, such as MacDrive, to allow Windows to read/write Mac OS, or Tuxera NTFS, Paragon NTFS or Parallels for Mac to enable it to read/write NTFS, but these (reportedly, and in part of my experience) only really work well for standard “Office” type applications, not so well for heavy (big andd real-time) data applications such as video editing, where they can impede the data throughput.  Doh!
    • Some people have experienced obscure issues of application functionality, beyond data-movement speed issues.
    • {Also, I am concerned over the (unknown/imagined/potential) risk that the “alien” operating system and/or its translation utility might alter the file system in some way that upsets its appearance to the “home” operating system.}
  • FAT is universal but is a riskier option:
    • FAT is un-journaled, hence risks loss not only of individual files but of whole volume (integrity).
      • In video editing, corruption could be disastrous to a project, not only in terms of possible data-loss or time wasting and project delays on data recovery, but also in terms of “weird” effects during editing, such as poor responsiveness to commands, whose cause the user may not appreciate. or even an increased risk of unacceptable flaws in the final product.
    • FAT32 is essentially obsolete, because its maximum file size is (1 bit under) 4GB.
    • exFAT, a kind of “FAT64” is practical, and indeed a big successful corporate Mac-based production company once supplied me with many GB of footage on an exFAT-formatted external disk.
      • The largest file I have so far stored there is 40GB.  No problems.
  • NAS (Network-Attached Storage) sounds at first an easy option, but in my experience they impede big real-time data throughput (as stated earlier for “file system tyranslation” add-ons). Double-Doh!
    • Such devices only permit file-level access.  Consequently, the client systems can e.g. create or retrieve folders and files, but cannot e.g. format the device or address it in terms of lower-level data structures.
    • A likely explanation for the “impedement” of a NAS (to data responsiveness and throughput) is that such devices store in a local format (typically they run linux) that is invisible to the client, then translate to an appropriate protocol for each operating system accessing it.  They normally incorporate a bunch of such protocols.  As always, translation => overhead.
    • Other options, such as SAN and iSCSI, instead of providing file-level access to the client systems, instead offer the lower level of data block access.  Thus they appear to the client system as would any local storage device, and can be formatted as appropriate to the client system.
  • One suggestion I saw was to use a Seagate GoFlex drive, which can be used (read/write) with both Mac and Windows.  But the supplier’s FAQ (about that drive) indicates that it depends upon a translator utility for the Mac:
    •  If you would like to be able to “shuttle” data back and forth between a Mac and a PC, a special driver needs to be installed onto the Mac that allows it to access a Windows-formatted drive (i.e. NTFS). Time Machine will not work in this case, nor will Memeo Premium software for Mac. However, if you want your GoFlex solution to also work with TimeMachine, the drive will need to be reformatted to HFS+ journaled.

So I guess there is no “magic storage” option, my main work setup will have to remain based on separate volumes for each OS.

When transferring an editing project from one OS to another, the following actions will be necessary:

  • Copy any absent or updated files across.
    • e.g. via a file-synch utility such as Syncovery.
  • Allow time etc. for possible file re-linking, re-indexing, re-preview generation, re-“SmoothCam” (or equivalent).
    • This aspect is down to the editing application etc., as opposed to the operating or file systems themselves.
  • Ensure any effects used in the edit are present on both systems.
    • If so then these should presumably still work…

(more…)

Replacement Hard Drive for MacBook Pro of 2009 Vintage

Monday, August 12th, 2013

Need a replacement for a failing drive in my MacBook:

My late-2009 MacBook Pro has been acting oddly, with corruptions, suggestive of a failing disk.  I suspect this was caused by the machine coming out of Sleep while travelling by car…

Replace with what kind of drive?

I could simply replace with an identical make and model to the original, which is:

  • According to W7 WindowsExplorer Properties:
    • ST9500420AS ATA Device
  • Printed on disk:
    • Seagate Momentus 7200.4
    • 500GB
    • WWN: 5000C500380A2AC9

On the other hand, I could see if there’s anything better I can rfeplace it with now, e.g. offering greater capacity and speed.  Of course, have to be careful to ensure capacity…

Following a web-search, and due consideration, my plan is:

  • SSD is possible, but may have to be careful about some technical issues in selection and ongoing system maintenance.  Also, given they cost a small number of hundreds of pounds, it is difficult to justify investing such an expensive component in a computer that, due to its age and visible wear-and-tear, is only worth about the same amount itself.
  • Hybrid SSHD is plug-compatible and looks the same to the OS as an ordinary HD, thus no technical complications should arise (no special configuration or ongoing maintenance).  Also, the price is below £100, which is justifiable for the described system.
  • Physically, to fit in the MacBook, the drive must have a SATA 2.5″ form factor and no more than 9.5 mm in height.

A recommended SSHD Drive:

  • Seagate 750GB Momentus XT hybrid
    •  ST750LX003
    • This (unlike larger 1TB variant) comes with 7200 rpm HD
    • It is available from Amazon:
      •  SATA 6Gb/s compatibility with NCQ for interface speed.
      • Product Dimensions    10 x 7 x 0.9 cm
      • Item model number    ST750LX003
      • Shock Tolerance:350 g @ 2ms (operating) / 1000 g @ 1ms (non-operating)
      • Interfaces:1 x Serial ATA-600, Compatible Bays:1 x internal – 2.5″ x 1/8H
      • Spindle Speed:7200 rpm
      • Drive Transfer Rate:600 MBps (external), Average Latency:4.2 ms
      • Non-Recoverable Errors:1 per 10^14
  • There is a 1TB variant but its spindle speed is only 5400 rpm

(more…)

Mac Pro Disk Failure & Recovery

Thursday, August 9th, 2012

Computer increasingly slow on start-up, eventually becomes sporadic in its ability to succeed, unexpected error messages…

Yes, it’s Disk Failure Time !  This time it was on my Mac Pro (desktop)

So I did these things:

  • Copied latest stuff to a portable (WD Passport) drive:
    • I copied documents, videos and downloads
    • I generated a list of installed applications, both 32-bit and 64-bit.
  • Opened up the machine to remove drives (and at the same time to hoover-out dust).
  • Procured a replacement hard drive
    • Google-search revealed my old drive to be obsolete, no longer (easily) available
    • Phoned a local computer tech wizardry shop, who fix Macs as well as PCs, and they had a suitable replacement drive (a WD SATA 1TB drive, twice the size of the old/failing one.
    • Bought that very disk.
  • Fitted the disk, as sole disk, and recovered both the Mac OS and Boot Camp > W7 partitions, according to the “DO” (not “DON’T”) branch of the instructions listed at http://blog.davidesp.com/archives/300
    • It took about an afternoon.  The longest stages were the actual restorings from backup.
  • For W7
    • The first thing I updated was the antivirus.  This was for the app as well as the database, and it wasn’t quick.  No reboot needed though.
    • Otherwise, two or three reboots were required, including first-use, windows updates critical, windows updates optional.

Seagate Barracuda ES.2.SATA ST3500320NS: Firmware Update How?

Wednesday, July 4th, 2012

My RAID has 8x disks: ST3500320NS, which are Seagate Barracuda ES.2.SATA.  One of the disks failed.  An attempt to replace it with a purchased disk failed, because that disk had Firmware version SN04 while the remainder of the disks had Firmware version SN05.

So how easy is it to flash the “offending” new drive with the required Firmware version?  Web-search:

On Seagate’s site:

  • I couldn’t find any downloads for SN05, only for a later version.
  • Others have had the same problem.
  • One person suggested using Instant Chat.
  • I tried that but it is apparently no longer available.
  • I sent an email to their tech support to request SN05 (and explaining why I needed it).

Cloning Mac OS to Another Volume (eg hard disk)

Friday, May 18th, 2012

Cloning:

iomega Home Media Network Hard Drive

Sunday, January 29th, 2012

My girlfriend has one of these, ethernet-connected to her router and thus available on her home network.  However, in order for any computer, Windows or Mac (what about linux?) to access it, it is necessary (aside from any hacky-workarounds that may possibly exist) to install the Iomega Home Storage Manager.   This makes volume(s) offered by the NAS appear, on the Mac in Finder under SHARED, or on a Windows machine as additional drive letter(s).

To acquire the Iomega Home Storage Manager, go to https://iomega-eu-en.custhelp.com/app/platform/p/1031,1043 or else try https://iomega-eu-en.custhelp.com/ and click on [Desktop Network Storage > Home Media Network Hard Drive].  May need to establish and login-to an iomega support account (free) first.

I wondered at first about enabling the NAS as a Mac OS Time Machine (backup) disk.  An iomega article I saw suggested that should be possible, for iomega firmware 2.0 and above.  But the disk as it stands is NTFS-formatted – because when it was set up we had only Windows machines.  Now that disk contains much material in that NTFS.  I guess it might be possible to partrition the disk e.g. to keep the existing NTFS and add alongside it a HFS+ partition for Time Machine to use.  But it’s guesswork that carries risks (of disruption/damage to existing contents).  We want an easy geeking-minimal life, so maybe better to repurpose that NAS and get a purpose-made Apple Time Capsule instead. Either way, the evening wears on, so I’ll shelve that idea/investigation for now…

(more…)

MacBook Pro: Restore (Mac OS & Boot Camp) from Backup (Disk Utility & WinClone)

Friday, July 15th, 2011

Backup & Restore via Disk Utility (DU) – on Mac OS install-disk – to a fresh hard-drive:

  1. Routine: http://support.apple.com/kb/ht1553
  2. Complication: Backed-up not the whole disk but aMac OS partition alongside a Boot Camp partition.
  • DON’T: Naive use of Disk Utility (DU) to restore straight away the partition (as a sole partition) from backup doesn’t work – it won’t boot.
    • You may see a grey Mac OS screen with “No Entry/Parking” sign, or error messages about ACPI drivers not present.
    • Attempts to install (fresh or archive i.e. user file preserving mode) from install disk fail since disk is not bootable.
      • Error message: “Mac OS cannot start up from this disk”
  • DO: Try install fresh OS X from install-dvd, then use it to create Boot Camp partition (and presumably boot-selection menu) then restore (with erase) to the OS X partition (only).  To save time (hopefully), didn’t actually install Windows.
    • Both the fresh-install and the restoration of OS X took about an hour.
    • Yes it worked! Booted into Mac OS just fine.
    • Left it to “settle” a bit – e.g. until CPU level down around zero.
    • Restart in Shift-Boot mode (to refresh OS’s tables etc.) and log-in as “DefaultEverything” (dummy user created as per advice – I think from Larry Jordan).  Maybe should have done that the first time…
    • Restarted in normal user account, again left awhile.
    • Boot Camp Assistant:
      • Create a partition (e.g. divide disks space equally between the two partitions)
        • (takes a minute or two – progress bar is initially misleadingly stationary)
      • Select [Quit and install later]
        • All we wanted was the partition, to restore into.
    • Started WinClone (App, started from MacOS)
      • It appeared to first scan the backup then began to install it.  Not quick, maybe an hour for each of these (two) tasks.
      • Source partition was 232.57 GB – as compared to the destination partition of around 250 GB.
    • Alt-Booted into W7 just fine.
    • Being on the internet, it began downloading numerous system updates – furiously (like it was hard to web-browse even on another computer on the network.
    • Correspondingly, on ShutDown, W7 installed numerous (61) updates.  Took ages – so if ever repeating such a recovery, allow for this…
    • Also on subsequent start-up, updating & registering stuff – took a few minutes – wish I’d run it straight (boot camp) not within Parallels.  But it seemed “happy”.
    • (to be continued…)

(more…)

Iomega NAS – Initial Experiences

Sunday, December 12th, 2010

Bought an Iomega Network-Attached Storage (NAS) device for a home network.  Here are my experiences:

  • Plugged it in and (naively) looked for it on Windows Explorer (networks section) but it didn’t appear there.
  • Ran its setup CD. Wary of bloatware, I selected Custom setup.  At the very least, that should list what kinds of thing could potentially be installed, then I can find out about them first.
    • Only installed the basic (non optional) software.
    • There is a backup tool, potentially useful but don’t want to dive into that just yet.
  • The result: a number of extra “drives” (drive letters) appeared, each being mapped to a network drive. The IP address was the same in each case: [192.168.1.68].   Although there was only one NAS, each “Network Drive” (letter) was mapped to a different folder on it, as follows:
    • music
    • photos
    • ActiveFolders
      • The NAS config (webpage) lets you define these, e.g. a folder where you can drop a file and it will turn up at your YouTube account.
    • public
      • What’s that for?  How”public” is it?
    • movies
    • backups
  • Reactions:
    • I had not expected that, just expected a single drive letter underneath which I would make my own folders, one for each user plus a shared one.
    • Presumably they are all intended to be shared among all users.
    • Maybe I can still make my own folders per-user etc.?  Maybe also can configure each user’s account to be tied to the appropriate one of these?
    • The NAS is a server, maybe it doesn’t allow access to root, only to folders?
  • User Manual
    • The Setup (disk) offers to Install the User Manual. Slightly concerned over that – only really want a PDF, not (potentially) more unknown bloatware.  Went ahead anyway.
  • NAS Config
    • To access the NAS Config (webpage):
      • In browser, for this particular device, can simply enter its name.
        • The default name was [Iomega-110496/], the numeric suffix being from the NAS’s MAC address.
        • One can change this name in NAS Settings, explained further below (where I indeed did so).
      • Its initially-displayed page is a demo ActiveFolder which presents a slideshow of any photos dragged into it.  So you can e.g. use your laptop as a photo-frame.  This is but one of several types of ActiveFolder.  More on these later…
      • Go to the Home page (via left-most menu)
        • Go to Remote Access tab (via upper tab-selections)
          • Ensure that Remote Access is disabled (for security).
          • Also can see the full MAC address there.
      • Go to the System page (via left-most menu)
        • Go to Date and Time tab (via upper menu)
          • Change it to the appropriate timezone.  By default was the US’s EST.  Changed it to GMT.
        • Go to Settings tab.
          • Change LED brightness down to minimum.  These gimmicks are simply annoying.
          • Change Machine Name from default
            [Iomega-110496] to memorable [magibox].

            • Test it works by entering magibox as the address in a fresh browser page.  Used Opera, which asked to put a “http://” at the front.
          • Change machine description to read “Julie’s Home Network Drive”.
      • Go to the Folders page (via left-most menu)
        • There is only one tab, “Folders”
          • Click the [+] button at lower-left (scroll-down) to add a new folder.
          • Add folders for each user- in line with my original scheme, to see if I can make that happen.
            • When create folder for self it says something like “since this is the first time you are accessing this folder, would you like to assign a password for it?
      • Go to the Active Folders page.
        • Go to Photos and Files tab
          • See the demo (slideshow) folder is there
          • Add a new folder, [Julie’s YouTube]
            • Inspected its Settings and it appeared already to have some YouTube details (username and password).
  • The NAS offers other features like Bittorrent (in some way) but I haven’t investigated those yet.

SpeedTools – disk speed test for Mac & Windows

Friday, July 16th, 2010

SpeedTools