Archive for the ‘util’ Category

Shared Storage Options for Windows & Mac Video Editing Collaboration

Friday, October 18th, 2013

In summary:

There’s no magic option, each workstation needs a local storage volume with block-level data access (as opposed to simply file-level access) and formatted to a file system that is native (doesn’t require translation) to that workstation’s operating system.  Migration and collaboration imply file copying/synchronization, which implies read-access to the “foreign” file-system.  Mac OS can read NTFS, Winows can only read HFS+ via third-party add-on utilities.  Furthermore, for speed and responsiveness appropriate to video editing, the local storage should ideally be RAID or SSD.  In either case, it is possible to split the local storage (e.g. via partitioning) into more than one file-system.  At least, that worked on the mutiple occasions I have taken that approach, and have not been aware of any issues.

In greater detail:

Consider the challenge of setting up a shared data storage volume (e.g. RAID array or SSD) for video editing, such that either Windows or Mac computers can connect to it, and a video project started on (and saved to) on one of those operating systems (OS) can be continued on the other (and vice versa).

My current solution is to split the drive into separate volumes, one for each OS.  For example I have done this on RAIDs of various kinds and on an internal drive for Mac systems bootable to either Mac OS or (via Boot Camp) to Windows.  In the case of RAIDs I was advised against this by my system supplier, but got the impression they were just being defensive, not knowing of any definite issues, and to my knowledge I did not experience any issues.

It is is not practical to have just one volume (necessarily in that case, one file-system format), because:

  • Mac OS on its own is able to read NTFS but cannot write to it.
    • This is a show-stopper.  Some of the major video editing applications (e.g. NLEs), slightly disturbingly, may use (or for some functionality, even depend on) read/write access to source-files and the folders containing them.
      • I initially, naively, imagined that video editing systems etc. would only ever read source media files, not write to them, or to the folders containing them.  However that proved very naive indeed…
        • In Apple/Mac’s (erstwhile) Final Cut Pro 7 I regularly used their (moving) image stabilization effect, SmoothCam.  Its analysis phased was typically slow and heavy – not something one would wish to repeat.  The result was a “sidecar” file of similar forename to the analyzed source file, but a different extension, placed in the same folder as the source file.
        • I’m not certain, but got the feeling that maybe the source file (or folder) meta data, such as permissions or somekind of interpretation-change to media files in the quicktime ([.mov] mmedia format.
      • Certainly, Adobe (on Windows and Mac) could adulterate both files (by appending XMP data – being an Adobe media metadata dialect in XML) and the folders they occurred in (depending on uder-configuration) in terms of sidecar-files.
      • Sony Vegas also generates sidecar-files, e.g. for audio peaks.
  • File system translation add-ons can add Windows read/write access to HFS+ (ordinarily it could not even read it) and add Mac OS write access to NTFS (ordinarily it could only read it), but not sufficiently transparent/seamless for big real-time data access as required for demanding video editing endeavours.
    • File system translation add-ons (to operating systems) exist, such as MacDrive, to allow Windows to read/write Mac OS, or Tuxera NTFS, Paragon NTFS or Parallels for Mac to enable it to read/write NTFS, but these (reportedly, and in part of my experience) only really work well for standard “Office” type applications, not so well for heavy (big andd real-time) data applications such as video editing, where they can impede the data throughput.  Doh!
    • Some people have experienced obscure issues of application functionality, beyond data-movement speed issues.
    • {Also, I am concerned over the (unknown/imagined/potential) risk that the “alien” operating system and/or its translation utility might alter the file system in some way that upsets its appearance to the “home” operating system.}
  • FAT is universal but is a riskier option:
    • FAT is un-journaled, hence risks loss not only of individual files but of whole volume (integrity).
      • In video editing, corruption could be disastrous to a project, not only in terms of possible data-loss or time wasting and project delays on data recovery, but also in terms of “weird” effects during editing, such as poor responsiveness to commands, whose cause the user may not appreciate. or even an increased risk of unacceptable flaws in the final product.
    • FAT32 is essentially obsolete, because its maximum file size is (1 bit under) 4GB.
    • exFAT, a kind of “FAT64” is practical, and indeed a big successful corporate Mac-based production company once supplied me with many GB of footage on an exFAT-formatted external disk.
      • The largest file I have so far stored there is 40GB.  No problems.
  • NAS (Network-Attached Storage) sounds at first an easy option, but in my experience they impede big real-time data throughput (as stated earlier for “file system tyranslation” add-ons). Double-Doh!
    • Such devices only permit file-level access.  Consequently, the client systems can e.g. create or retrieve folders and files, but cannot e.g. format the device or address it in terms of lower-level data structures.
    • A likely explanation for the “impedement” of a NAS (to data responsiveness and throughput) is that such devices store in a local format (typically they run linux) that is invisible to the client, then translate to an appropriate protocol for each operating system accessing it.  They normally incorporate a bunch of such protocols.  As always, translation => overhead.
    • Other options, such as SAN and iSCSI, instead of providing file-level access to the client systems, instead offer the lower level of data block access.  Thus they appear to the client system as would any local storage device, and can be formatted as appropriate to the client system.
  • One suggestion I saw was to use a Seagate GoFlex drive, which can be used (read/write) with both Mac and Windows.  But the supplier’s FAQ (about that drive) indicates that it depends upon a translator utility for the Mac:
    •  If you would like to be able to “shuttle” data back and forth between a Mac and a PC, a special driver needs to be installed onto the Mac that allows it to access a Windows-formatted drive (i.e. NTFS). Time Machine will not work in this case, nor will Memeo Premium software for Mac. However, if you want your GoFlex solution to also work with TimeMachine, the drive will need to be reformatted to HFS+ journaled.

So I guess there is no “magic storage” option, my main work setup will have to remain based on separate volumes for each OS.

When transferring an editing project from one OS to another, the following actions will be necessary:

  • Copy any absent or updated files across.
    • e.g. via a file-synch utility such as Syncovery.
  • Allow time etc. for possible file re-linking, re-indexing, re-preview generation, re-“SmoothCam” (or equivalent).
    • This aspect is down to the editing application etc., as opposed to the operating or file systems themselves.
  • Ensure any effects used in the edit are present on both systems.
    • If so then these should presumably still work…

(more…)

svchost Viewer

Saturday, March 19th, 2011

svchost Viewer – sounds good, not tried it or recommending it, just noting it for possible future investigation.

  • http://svchostviewer.codeplex.com
  • A program to see what all those svchost.exe are running
  • Ever wondered what all those svchost.exe processes are running ?? Well here is an app
    to tell you. It gives you some basic information like the Name and description.
  • No installation required.
  • Only requirement is that you have .net installed (ver 2.0 or newer).
  • Work in Windows XP (sp2) and Vista and Windows 7.
  • Coded in C#

md5 Check-Sum Apps for Windows

Thursday, December 9th, 2010

md5 Checksumming in Windows:

  • There is no in-built checksumming tool I know of in Windows, but they are standard in linux etc.
  • The best third-party md5 checksum tool I have discovered for Windows is MD5Summer (V. 1.2.0.11)  [http://www.md5summer.org/download.html].
    • When you run it, it first checks file associations and offers to set it as default for [.md5] files.  In W7 an attempt to do so may be denied (admin status required).
    • You first select a root-folder then files within it (or all, recursively).  To select files, it’s not sufficient to highlight them, you have to double-click them (or click the Add button), so that they appear in the right-hand pane.  Only then will the OK button be enabled.
    • By running it recursively from a root folder, rather than on individual folders, a single [.md5] file is created that does not pollute the subfolders (e.g. BPAV folders as recorded by XDCAM-EX).
    • Had a slight issue once where I manually copied from XP to W7 machines, then generated MD5 on source machine (XP) and verified on target machine (W7).  At first, the verifier immediately returned all mismatches.  So immediately in fact that it seems likely it was not performing any computation.  Later on it worked as expected.  Not known what changed to cause this, maybe simply the act of opening the md5 file in NotePad..

Concepts (to the best of my understanding):

  • A check-sum of some data is a “fingerprint” numeric value that is probably-unique to that data.  A fingerprint is useful when it is much smaller than the data it represents yet (virtually) uniquely identifies the content of that data.  A checksum of a file reflects that file’s data-contents but is not affected by the file name etc.
  • Various checksum schemes/algorithms exist, one of the most popular being md5.   An md5 fingerprint is extremely unlikely to be the same for any other file and will (for that reason) almost certainly change in value if the contents of a file is changed by even the tiniest degree.
  • A typical md5 tool will take one or more specified files and generate their fingerprints as corresponding entries (each entry being a filename and a fingerprint) in a [.md5] file.  Correspondingly it will take a given [.md5] file and report whether or not the stated and (re-) computed fingerprints agree.
  • An [.md5] file is a text file with one or more lines consisting of a checksum value (in hexadecimal) then a space then an asterisk then a file name, possibly preceded by a folder path (with respect to the folder containing the [.md5] file).  It can also have comment-lines, each beginning with a hash (#) character.  Example entries:
    • eb574b236133e60c989c6f472f07827b *fred.exe
    • [eb574b236133e60c989c6f472f07827b *tmp/fred.exe].
  • Some download sites include [ .md5] files alongside or along with their associated  files.  Some sites just display the fingerprint itself on the webpage.  Typically the purpose is to allow the user to check whether a download was complete or was corrupted.
  • The fingerprint computed by a typical checksumming application is not affected by a file’s name or read-only status etc., only by its data contents.  Thus it is not a total basis for consistency-checking of system configuration.

(more…)

Command-Prompt as Folder Right-Click option in Windows Explorer

Thursday, December 9th, 2010

In Windows XP, wanted a folder right-click menu option to bring up a MSDOS command shell.  This proved very simple to add, and gave the further benefit of the text-based (command-line) user interface (TUI?) retaining customisations (e.g. green text).  The steps were (for Windows XP):

  • Windows Explorer: Tools > Folder Options.
  • Select the File Types tab.
  • Go to NONE / Folder.
  • Select the entry labeled Folder
  •  Press Advanced button.
  • Select New
  • In the action block type “Command Prompt” (without the quotes).
  • In the app block type “cmd.exe” (without the quotes).
  • Save and exit Folder Options.

(more…)

sha1 Checksums

Saturday, May 15th, 2010

Based on [http://www.bresink.com/osx/sha1.html] as of 2010-05-15:

  • Apple recommends the use of the so-called SHA1 standard (Secure Hash Algorithm Version 1). Apple is using this method for their own security updates as well. For more information please also refer to Apple Support Article HT1652.
  • Perform the following steps:
    • Make sure you have the downloaded file displayed somewhere in the Finder.
    • Open the Terminal application.
    • Enter the command: [/usr/bin/openssl sha1 ] with a space at the end (not a Return).
      • Note the last character of ‘openssl’ is a small ‘L’ not a ‘1’
    • Drag the target file from the Finder window into the Terminal window. A path specification will appear in the Terminal window.
    • Now press the Return key (Enter).
    • You will see output similar to the following example:
      • SHA1(path specification)= 2eb722f340d4e57aa79bb5422b94d556888cbf38.

Mac Ease-of-Use Util: QuickSilver

Saturday, January 2nd, 2010

Some people rave about this free util.  It replicates some Spotlight functionality but is more advanced.

Setup/config tweaks are non-obvious, so it is worth checking out the following tutorial:

To obtain it: