Archive for the ‘file mgt’ Category

Windows 7 & Mac: Move/Redirect “Documents” (eg to a non-system volume)

Tuesday, April 22nd, 2014

Windows 7’s “My Documents” library folder is by default mapped to the system drive, e.g. as [C:\Users\<username>\Documents].  However it is also possible to map it elsewhere, e.g. to another volume.  A broadly equivalent situation exists in Mac OS.  One might for example use this option to move the Documents library/folder to a thumb/flash drive when using several computers (one at a time) or to put it on a non-system drive, e.g. to free up space on the system drive, exclude it from system backups (thus freeing up both space and time) or to put it on something like a server, possibly on “The Cloud”.

I found the following explanation by accident, while attempting to find a way to prevent Adobe Media Encoder (AME) from storing its own “preview files” (sic), which are huge, in a sub-folder of “My Documents”, which itself on typical Windows systems is to be found on the System Drive.  It seems that AME has no Preferences setting to store these preview files elsewhere, so a workaround is needed, e.g. to move the “My Documents” library folder itself to another volume.

  • http://forums.adobe.com/message/6045027
  • http://windows.microsoft.com/en-us/windows/redirect-folder-new-location#1TC=windows-7
    • Windows 7:
    • To redirect a folder to a new location
      • Click the Start button Picture of the Start button, and then click your user name.
      • Right-click the folder that you want to redirect, and then click Properties.
      • Click the Location tab, and then click Move.
      • Browse to the location where you want to redirect this folder. You can select another location on this computer, another drive attached to this computer, or another computer on the network. To find a network location, type two backslashes (\\) into the address bar followed by the name of the location where you want to redirect the folder (for example, \\mylaptop), and then press Enter.
      • Click the folder where you want to store the files, click Select Folder, and then click OK.
      • In the dialog that appears, click Yes to move all the files to the new location.
    • Mac OS (Mavericks & previous):
    • To restore a folder to its original location
      • Click the Start button Picture of the Start button, and then click your user name.
      • Right-click the folder that you previously redirected and want to restore to its original location, and then click Properties.
      • Click the Location tab, click Restore Default, and then click OK.
      • Click Yes to recreate the original folder, and then click Yes again to move all the files back to the original folder.
  • https://discussions.apple.com/message/23510349#23510349
    • (Ignore the initial links, which are merely about changing names, e.g. when migrating a laptop from one person to another)
    • John Galt, 25-Oct-2013
      • The procedure was unchanged in Mavericks from previous OS X versions.
      • What I did was create a new User in System Preferences, after which I logged out and logged in to that new User.
      • I performed basic configuration, created some documents, etc.
      • After that I logged out, logged in under my usual account, and dragged that User’s folder to another volume.
      • Then, I used Users & Groups “Advanced Options” to point to the new Home folder’s location.
      • After that, I restarted the Mac using OS X Recovery to reset that user’s Home Folder Permissions and ACLs since permissions problems with the copied Home folder would otherwise result.
      • After quitting OS X Recovery I was able to log in to the User account established on the USB flash drive, and was able to use it more or less the same way without any surprises. Safari, iTunes, iPhoto all worked, no problems.
      • The original User account (home folder) remained on the boot volume, so I dragged it to the Trash. I verified that I could still log in to the account on the flash drive, confirming the one created on the boot volume was no longer required.
      • Attempting to log in to the account with the flash drive disconnected resulted in an expected error (below) and obviously you wouldn’t want to do that while using the account.
      • Reconnecting the flash drive restored the ability to log in as expected.

Shared Storage Options for Windows & Mac Video Editing Collaboration

Friday, October 18th, 2013

In summary:

There’s no magic option, each workstation needs a local storage volume with block-level data access (as opposed to simply file-level access) and formatted to a file system that is native (doesn’t require translation) to that workstation’s operating system.  Migration and collaboration imply file copying/synchronization, which implies read-access to the “foreign” file-system.  Mac OS can read NTFS, Winows can only read HFS+ via third-party add-on utilities.  Furthermore, for speed and responsiveness appropriate to video editing, the local storage should ideally be RAID or SSD.  In either case, it is possible to split the local storage (e.g. via partitioning) into more than one file-system.  At least, that worked on the mutiple occasions I have taken that approach, and have not been aware of any issues.

In greater detail:

Consider the challenge of setting up a shared data storage volume (e.g. RAID array or SSD) for video editing, such that either Windows or Mac computers can connect to it, and a video project started on (and saved to) on one of those operating systems (OS) can be continued on the other (and vice versa).

My current solution is to split the drive into separate volumes, one for each OS.  For example I have done this on RAIDs of various kinds and on an internal drive for Mac systems bootable to either Mac OS or (via Boot Camp) to Windows.  In the case of RAIDs I was advised against this by my system supplier, but got the impression they were just being defensive, not knowing of any definite issues, and to my knowledge I did not experience any issues.

It is is not practical to have just one volume (necessarily in that case, one file-system format), because:

  • Mac OS on its own is able to read NTFS but cannot write to it.
    • This is a show-stopper.  Some of the major video editing applications (e.g. NLEs), slightly disturbingly, may use (or for some functionality, even depend on) read/write access to source-files and the folders containing them.
      • I initially, naively, imagined that video editing systems etc. would only ever read source media files, not write to them, or to the folders containing them.  However that proved very naive indeed…
        • In Apple/Mac’s (erstwhile) Final Cut Pro 7 I regularly used their (moving) image stabilization effect, SmoothCam.  Its analysis phased was typically slow and heavy – not something one would wish to repeat.  The result was a “sidecar” file of similar forename to the analyzed source file, but a different extension, placed in the same folder as the source file.
        • I’m not certain, but got the feeling that maybe the source file (or folder) meta data, such as permissions or somekind of interpretation-change to media files in the quicktime ([.mov] mmedia format.
      • Certainly, Adobe (on Windows and Mac) could adulterate both files (by appending XMP data – being an Adobe media metadata dialect in XML) and the folders they occurred in (depending on uder-configuration) in terms of sidecar-files.
      • Sony Vegas also generates sidecar-files, e.g. for audio peaks.
  • File system translation add-ons can add Windows read/write access to HFS+ (ordinarily it could not even read it) and add Mac OS write access to NTFS (ordinarily it could only read it), but not sufficiently transparent/seamless for big real-time data access as required for demanding video editing endeavours.
    • File system translation add-ons (to operating systems) exist, such as MacDrive, to allow Windows to read/write Mac OS, or Tuxera NTFS, Paragon NTFS or Parallels for Mac to enable it to read/write NTFS, but these (reportedly, and in part of my experience) only really work well for standard “Office” type applications, not so well for heavy (big andd real-time) data applications such as video editing, where they can impede the data throughput.  Doh!
    • Some people have experienced obscure issues of application functionality, beyond data-movement speed issues.
    • {Also, I am concerned over the (unknown/imagined/potential) risk that the “alien” operating system and/or its translation utility might alter the file system in some way that upsets its appearance to the “home” operating system.}
  • FAT is universal but is a riskier option:
    • FAT is un-journaled, hence risks loss not only of individual files but of whole volume (integrity).
      • In video editing, corruption could be disastrous to a project, not only in terms of possible data-loss or time wasting and project delays on data recovery, but also in terms of “weird” effects during editing, such as poor responsiveness to commands, whose cause the user may not appreciate. or even an increased risk of unacceptable flaws in the final product.
    • FAT32 is essentially obsolete, because its maximum file size is (1 bit under) 4GB.
    • exFAT, a kind of “FAT64” is practical, and indeed a big successful corporate Mac-based production company once supplied me with many GB of footage on an exFAT-formatted external disk.
      • The largest file I have so far stored there is 40GB.  No problems.
  • NAS (Network-Attached Storage) sounds at first an easy option, but in my experience they impede big real-time data throughput (as stated earlier for “file system tyranslation” add-ons). Double-Doh!
    • Such devices only permit file-level access.  Consequently, the client systems can e.g. create or retrieve folders and files, but cannot e.g. format the device or address it in terms of lower-level data structures.
    • A likely explanation for the “impedement” of a NAS (to data responsiveness and throughput) is that such devices store in a local format (typically they run linux) that is invisible to the client, then translate to an appropriate protocol for each operating system accessing it.  They normally incorporate a bunch of such protocols.  As always, translation => overhead.
    • Other options, such as SAN and iSCSI, instead of providing file-level access to the client systems, instead offer the lower level of data block access.  Thus they appear to the client system as would any local storage device, and can be formatted as appropriate to the client system.
  • One suggestion I saw was to use a Seagate GoFlex drive, which can be used (read/write) with both Mac and Windows.  But the supplier’s FAQ (about that drive) indicates that it depends upon a translator utility for the Mac:
    •  If you would like to be able to “shuttle” data back and forth between a Mac and a PC, a special driver needs to be installed onto the Mac that allows it to access a Windows-formatted drive (i.e. NTFS). Time Machine will not work in this case, nor will Memeo Premium software for Mac. However, if you want your GoFlex solution to also work with TimeMachine, the drive will need to be reformatted to HFS+ journaled.

So I guess there is no “magic storage” option, my main work setup will have to remain based on separate volumes for each OS.

When transferring an editing project from one OS to another, the following actions will be necessary:

  • Copy any absent or updated files across.
    • e.g. via a file-synch utility such as Syncovery.
  • Allow time etc. for possible file re-linking, re-indexing, re-preview generation, re-“SmoothCam” (or equivalent).
    • This aspect is down to the editing application etc., as opposed to the operating or file systems themselves.
  • Ensure any effects used in the edit are present on both systems.
    • If so then these should presumably still work…

(more…)

NLE Adulteration of Source Media: Potential Workflow-Issues

Friday, September 13th, 2013

I highlighted in http://blog.davidesp.com/archives/598 (10 months ago) that Adobe Premiere etc. can adulterate media files, in terms of metadata and/or sidecar-files (depending on user-configurations of these applications.  I indicated that, regardless of the reasonableness of at least some of these actions, this could potentially cause problems to other applications.

Validating that concern, I note a post (2012-06-12) by Matt Davis on Philip Bloom’s website, stating (my italics):

  • …if sharing assets with FCPX and Adobe Premiere, Adobe ‘touches’ (resets the modification date) of each file without doing anything else to it, but also sprinkles sidecar files into directories of transcodable files for metadata, thus sending any returning FCPX activity into a tailspin, requiring a re-linking session. It’s oddities like these which haunt the implementation of FCPX in a wider system and make system managers wonder if FCPX is actually worth implementing in its current state.

That was over a year ago, and so the issue may or may not exist for the current version of FCPX.

As users, whether or not the actions of one application adhere to standards and another don’t, what we as users ultimately care about is workflow, which in this case translates to “does it connect up with my other tools/processes?”.  So we have to maintain a “situational awareness” of potential interoperability pitfalls.

Incidentally, I recall that FCPX’s predecessor (in history at least, if not development-line) FCP7 could adulterate source directories with its own sidecar files, produced by its SmoothCam effect.  Not knowing anything further for sure, I nevertheless wondered (at that time) what it might be doing “under the hood” of the QuickTime [.mov] wrapper.

Xplorer2: A Far Richer Alternative to Windows Explorer

Monday, August 12th, 2013

I found, downloaded and am currently evaluating Xplorer2, a much more functional alternative to Windows Explorer.

My initial impressions of it are extremely favourable…

(more…)

Windows Explorer: Sorting & Grouping & Boolean Filters

Friday, August 9th, 2013

http://www.7tutorials.com/transform-windows-explorer-filtering-options

Windows 7: “Are you sure you want to copy this file without its properties”

Friday, March 15th, 2013

On my Windows 7 machine, while copying files (manually backing-up) from an internal drive to an external USB drive that was formatted by someone else, the following warning message appeared:

  • “Are you sure you want to copy this file without its properties”

This turned out to be due to the external disk having been formatted as FAT32 instead of NTFS as per my internal drive.

(more…)

WD Link: Successor to WD Discovery

Monday, February 4th, 2013

WD Link Setup is the successor to WD Discovery, for detecting WD NAS drives and mapping them to drive-letters.

http://support.wdc.com/product/download.asp?groupid=902&sid=110&lang=en

Adobe Files Recovered in Trash – Unwanted

Saturday, November 17th, 2012

Whenever I boot up Mac OS, there are recovered Adobe files in the Trash.  Even if I did not use Adobe in the previous session!  Of course I can [Empty Trash] but why do they keep cropping up there in the first place?  Is this symptomatic of some error or malware in my Mac OS system?  Last time it happened, these were the files concerned:

  • The file [com.adobe.dynamiclinkmanagerCS6]
  • A [.prmdc] file, with prefix as per one of my project names.
    • Maybe [pr] indicates association with Premiere?
  • A bunch of files named as [I-Frame Only MPEG~xxxx.epr], where [xxxx] represents a pseudorandom hex value (of more than 4 characters).
    • I guess these are preview-accelerating renders.  But I thought such renders were retained, not temporary.
    • From http://help.adobe.com/en_US/mediaencoder/cs/using/adobemediaencoder_cs5_help.pdf it seems that [.epr] generally means “exported preset”.  So maybe that’s what it means here also. In which case these are I guess temporary exported presets when sending to (encoder) Queue from Premiere?

The best advice I could find on the web was that this kind of thing, while not generally expected, is of no importance, so “just keep emptying the trash”.  Concerning and irritating though…

Presumably Adobe is not cleaning-up when I close it, but in that case why and what else is it not doing?  Could this be associated with the Kaspersky issue I recorded in my previous blog-post?  Like had the Kaspersky-augmented kernel shutdown been methodically waiting for some Adobe clean-up process that never terminated, whereas un-augmented kernel shutdown simply (and silently) forced-killed that process?   Just guessing with my overactive imagination, no supporting knowledge/information/evidence.

The fact (I have observed) that recovered Adobe files can appear in Trash even when Adobe has not been used in the current or the previous Mac session (between machine boot-ups) tends to suggest that some independent Adobe clean-up process is always happening in the background, as a result of normal system start-up, regardless of whether Adobe has explicitly been run by the user.  Gives me some “gut feeling” that my “imagination” might be on-track…

(more…)

Adobe Production CS6 – Cache & Render Files and their Locations

Saturday, September 1st, 2012

When I start-up any application, I like to understand at least the main side-effects it’s having on my system.  In the case of Adobe’s primary video-editing apps, Premiere and After Effects, my experience (on Windows 7) is that they save intermediate preview-renders to the system volume.  This causes me the following concerns:

  • System Volume may serve poorly as a media drive.
    • Larry Jordan, at least in the recent past, advises against using the system drive for media read/write.  On the upside, such drives may have high-bandwidth to the system, but on the downside, the system can interrupt their use with highest priority, whuch may (I guess) pose a risk to smooth playback (though I am aware that buffering may possibly reduce this risk, I haven’t done or seen any such calculations).  Cache files are indeed media files that are written and read.
    • On the other hand, an informed representative of a well-known UK supplier of video editing laptops advised me that in his experience, most users of laptops with only a single internal drive (as system drive) do use that drive in this way (for portability).
  • System drive can become “clogged up”
    • System drive can become clogged-up by many or large video files of which that the user is only partially aware, their creation having happened implicitly during their use of the NLE etc.  Like temporary files only worse!
    • Ultimately the system drive can even become full, making the operating system itself sluggish or even less stable (and video playback less smooth.
    • Backup of a system drive that includes media files will typically require significantly greater archive space and will take significantly greater time (than a clean system).
  • Migrate-ability is reduced
    • I like the idea of a video project being a free-floating data-object.  That is, it should not be tied to any particular instance of a data storage volume, let alone a particular computer (system).  It should be possible for all files relevant to a project to be stored on any volume, migrated to any other volume, plugged into any computer having appropriate installed applications, and everything to work the same way as when the project was on its original volume being edited on the original system.  That includes not only the source media files etc. but also the intermediate rendered files.

So what do the Adobe editing applications provide to enable my preferred working arrangement?

  • Premiere:
    • [Edit > Preferences > Media]
      •  This defines the location of the folder [Media Cache Files], which contains pseudorandomly-named files.  Example Files:
        • [929_4372_01-125eeda9-ba0d-a8ea-4418-3480000001f0.ims]
        • [Rendered – 68721ea9-25e9-4f56-8430-4ca10101ace7-04602910-cd54-1a45-a5d7-557b000001f2.ims]
      • Default location: [c:\Users\…\AppData\roaming\Adobe\Common]
      • [Yes] Save Media Cache files next to originals when possible
        • e.g. for my XDCAM-EX files, inside a CLIPR folder (which contains the EX’s “essence” (.mp4) files, in this case [929_4491_01.MP4]), appeared the following files:
          • 929_4491_01.MP4 48000.cfa (9.2 MB)
            • Unknown, but the “48000” and the “a” on the end of “cfa” are suggestive of audio.
          • 929_4491_01.MP4 48000.pek (37KB)
            • Simply the peaks (waveform graphics data) file for the audio component of the essence-file.
        • Experience:
          • I clicked the [Browse] button and selected an area on my external media drive (a GRaid Mini) as: [H:\_App_Specific\Adobe].
          • Consequently, at my specified location the following folder appeared: [Media Cache Files]
    • Media Cache Database
      • Note:
        • When Premiere Pro imports video and audio in some formats, it processes and caches versions of these items that it can readily access when generating previews. Imported audio files are each conformed to a new .cfa file, and MPEG files are indexed to a new .mpgindex file. The media cache greatly improves performance for previews, because the video and audio items do not need to be reprocessed for each preview.
        • When you first import a file, you may experience a delay while the media is being processed and cached.
          A database retains links to each of the cached media files. This media cache database is shared with Adobe Media Encoder, After Effects, Premiere Pro, Encore, and Soundbooth, so each of these applications can each read from and write to the same set of cached media files.
      • Location: [c:\Users\…\AppData\roaming\Adobe\Common]
        • [Browse]
          • If you change the location of the database from within any of these applications, the location is updated for the other applications, too.
          • Each application can use its own cache folder, but the same database keeps track of them.
          • Example Experience:
            • I clicked the [Browse] button and selected an area on my external media drive (a GRaid Mini) as: [H:\_App_Specific\Adobe].
            • In response, a prompt came up saying “Move the existing media cache database to the new location, or delete it (Buttons: [Move] [Delete] [Cancel] ).
            • I clicked [Move]
            • Consequently, at my specified location the following folder appeared: [Media Cache]
      • Purging
        • [Clean]
          • This removes “orphan” cache files.
          • To remove conformed and indexed files from the cache and to remove their entries from the database, click [Clean]. This command only removes files associated with footage items for which the source file is no longer available.
            • Important: Before clicking the [Clean] button, make sure that any storage devices that contain your currently used source media are connected to your computer.
      • [Yes] Write XMP ID To Files On Import
        • Check this box to write ID information into XMP metadata fields.
        • e.g. hence [929_4491_01M01.XMP] sidecar-file (containing XMP metadata) got written into the CLIPR folder containing its associated media  file, here an XDCAM-EX essence file, [929_4491_01.MP4].
      • [Yes] Enable Clip And XMP Metadata Linking Check this box to link clip metadata to XMP metadata, so that changing one changes the other.
    • Intermediate-Preview Render Files
      • None of the above measures affect where intermediate/preview files get rendered to…
        • I proved this (in Windows 7) by deliberately causing a render then searching on “Huge” (>16MB) files created “Today”
        • Rendered files location was:[C:\Users\…\Documents\Adobe\Premiere Pro\5.5\Adobe Premiere Pro Preview Files\Untitled3.PRV]
      • A brief Google revealed several articles where the (sadly now obsolete) solution was a setting under [Edit > Preferences > Scratch Disks]
      • Eventually discovered that in CS5.5 these settings now resided in: [Project > Project Settings > Scratch Disks]
        • Here, all settings (affecting Capture and  Preview) were set to [Same as Project]
        • As it happened, my project location was [C:\Users\David\Documents\Adobe\Premiere Pro\5.5].
      • Experiment with a different project location:
        • Save As: [H:\_Media\_Projects\MyProject\030 Projects\Adobe\Experiments]
          • The file [Expt A v001.prproj] appeared in it, but that was all.
        • Save a Copy: [H:\_Media\_Projects\MyProject\030 Projects\Adobe\Experiments]
          • The file [Expt A v001 Copy.prproj] appeared there, but that was all.
        • (The file open within Premiere remained as the original, not the copy)
        • After a while, the project file was joined by: [Adobe Premiere Pro Auto-Save]
      • Experiment to migrate the Preview-Render files:
        • In Windows Explorer, I created a folder named [Adobe Premiere Pro Preview Files]
        • Into that folder, from the similarly-named folder on the system volume, I dragged the existing folder [Untitled3.PRV]
      • Experiment: Premiere “knows” when render-files have gone, and prompts for their possible new location.
        • I deleted the project-specific render-files folder on the (external) project-drive.
        • I re-rendered, resulting in a fresh such folder, re-populated.
        • In Premiere, I Closed the project
        • Then in Windows I renamed the project-specific render-files folder, then back in Premiere I re-opened the project.
          • Premiere prompted with Browser titled: “Where is the File ‘Rendered – 5ac…..228.mpeg’?”
          • I selected the stated file in its newly-renamed folder, Premiere then found all the others there.
        • Result: the timeline render-region went green (i.e. “probably playable”).
      • Experiment: Once render-files are regarded as “gone”, they cannot be restored.
        • I closed the project, ensuring I did not save changes (such as the new location of the render files), then re-opened it.
          • As in the previous experiment (since I did not save changes), the render files folder could not be found.
          • Premiere thus prompted with Browser titled: “Where is the File ‘Rendered – 5ac…..228.mpeg’?”
          • This time however I simply used the [Skip All] option.
        • Closed the project.
        • Renamed the rendered-files folder back to its original name.
        • In Premiere, re-opened the project.
          • The timeline region remained red, indication no render-files were associated.
      • Experiment: Tidy migration of a project to a new location.
        • Warnig: in the case of doing a Copy (which is Windows’ default drag operation between different volumes), take care to ensure the Project (file) is not simply referencing the original preview files at the old location…
        • Drag both Project and its folders (including render-file folder) to a new location (e.g. on a new disk).
        • If name and relative location of folder are unchanged (as they ought to be, in good practice) then the files will be automatically detected and used, not even a user-prompt.
          • Just be sure though that the project isn’t simply referencing the render-files in their original location, if they are still present there.  Premiere is “lazy” in this respect.
      • Experiment: The relative location of the Rendered Files folder does matter (relative to the project file).
        • Tried putting the render files in a non-standard location.
          • The “Locate/Browse” prompt appeared
          • I located the file
          • All at first appeared well, and the corresponding section of the timeline went green
          • However, the “Composer” window simply displayed “Media Pending”.  That never went away.
      • Experiment:
        • When migrating, also need to move (or copy):
          • The Media Cache directories
            • Actually I’m not so sure about this. I tried exiting Premiere, renaming these directories and opening Premiere.  It created and repopulated the same directories in their original location, which in my case was an external drive.
          • The Source Media files
    • http://forums.adobe.com/thread/784220?tstart=0
      • I suggest marking each external with the drive letter that the user assigns to it, say Z:\. Then, whenever Z:\ is plugged in, it will always be seen as Z:\. This way, the NLE can keep up with where the Assets are located, starting with the drive letter.
      • If one is migrating Projects between computers, they will repeat this exact process in the OS of each computer.
      • Note: when doing the migration, ALL Assets, Scratch Disks, and the Project file, MUST be included on that external

Work Procedure for Migrate-ability:

  • By associating cache and XMP files with the media (or its essence), Adobe projects are migratable.  However adding such files into the BPAV/CLIPR folder structure is considered by some applications to be an adulteration of that structure, requiring their deletion.   However, such deletion on an as-needed basis is not too onerous – given it is easy to do and in any case this situaion should rarely arise in practice.
  • When using different disks, remember to re-define (in Preferences) the location of cache files etc.
    • One work-around would be to -re-set the cache location before opening any individual project.
      • Might be hard to remember to do when opening a project from within the NLE, easier to remember when double-clicking a project file in Windows Explorer.still
    • I’m not 100% sure what to do about these…
  • As noted earlier:
    • When doing the migration, ALL Assets (Sources), Scratch Disks (Renders), and the Project file, MUST be included on that external.
      • I note that this says nothing about Cache Files etc. …

External Storage Devices and their Bandwidths

Saturday, September 1st, 2012
  • Summarised info from http://forums.adobe.com/thread/784220?tstart=0
    • Average transfer rates in MB/s for different interfaces:
      • USB2:    20 – 25, depending on other USB devices sharing the same bandwidth
        • archiving for storage only
      • FW400:  30 – 35
        • archiving for storage, and light editing to/from (just very slow)
      • FW800:  50 – 60
        • archiving for storage, and regular editing to/from (fairly fast)
      • USB3:    65 – 80, depending on other USB devices sharing the same bandwidth
        • (no experience)
      • eSATA & SATA: 100 -140
        • archiving for storage, and regular editing to/from
    • RAID Speed-gains over a single disk:
      • RAID0:    0.9 x N disks over a single disk
      • RAID3/5: 0.8 x (N-1) disks over a single disk for read, 0.6 x for write.
        • ICHR10 figures are a bit lower than hardware controllers.

Mac OS: File Stocktaking: Equivalent of TREE (as in DOS)

Saturday, August 11th, 2012

How, on a Mac OS system, to do the equivalent of TREE in DOS:

http://murphymac.com/tree-command-for-mac/

  • find . -print | sed -e ‘s;[^/]*/;|____;g;s;____|; |;g’
    • And there is much more on ths in the article, including how to add this “command” to user profile.

My crude adaption of it, to list only the main directories in my [Media] area:

  • find . -type d \! -name “BPAV” \! -name “CLPR” \! -name “TAKR” \! -name “929*” -print | sed -e ‘s;[^/]*/;|____;g;s;____|; |;g’
    • Crude but delivered what I wanted.

Avid (MC etc): Prevent Use of System Drive

Saturday, June 23rd, 2012
  • http://community.avid.com/forums/t/78445.aspx (structured & paraphrased)
    • Main Trick:
      • Set your Media Creation settings to filter out the system drive
      • then open your Site Settings box (under the Special menu)
      • drag the Media Creation setting from your Project window to the Site Setting box.
      • All new projects should now respect your Media Creation settings (although old ones won’t – this will only affect all new projects you create).
    • Safety Net:
      • … create the folder AvidMediaFiles on your [C:] drive and then create a text file called MXF (take the txt extention off of it) inside the AvidMediaFiles folder.  Then if you try to digitize or import to that drive you get an error, because it wants to creat an MXF folder but can’t.  …useful in helping editors remember where to digitize, and to always check their settings.
  • http://www.theeditforum.net/avid-media-composer-tip-1/
    • Under the drive Filtering and Indexing tab make sure all 3 boxes are ticked.
    • This will prevent media being imported, captured or rendered to any drives that can’t handle the selected resolution (Filter Network Drives Based on Resolution). And filter out the System & Launch drives (these are the drives where Avid and your OS are installed – usually the same).
    • Under Media Creation you can also specify which drives different types of media are stored on. So for example, you might have a Unity Partition that is just for Renders etc.
  • http://lfhd.net/2011/07/29/avid-mc-tip-separating-media-by-project/
    • How to separate media by project
      • The [1] folder under [MXF] can be called anything, e.g. a Project name, and reserved for media associated only with that project.  Once renamed, if Avid needs to import anything else, it will create a fresh [1]-folder.

iomega Home Media Network Hard Drive

Sunday, January 29th, 2012

My girlfriend has one of these, ethernet-connected to her router and thus available on her home network.  However, in order for any computer, Windows or Mac (what about linux?) to access it, it is necessary (aside from any hacky-workarounds that may possibly exist) to install the Iomega Home Storage Manager.   This makes volume(s) offered by the NAS appear, on the Mac in Finder under SHARED, or on a Windows machine as additional drive letter(s).

To acquire the Iomega Home Storage Manager, go to https://iomega-eu-en.custhelp.com/app/platform/p/1031,1043 or else try https://iomega-eu-en.custhelp.com/ and click on [Desktop Network Storage > Home Media Network Hard Drive].  May need to establish and login-to an iomega support account (free) first.

I wondered at first about enabling the NAS as a Mac OS Time Machine (backup) disk.  An iomega article I saw suggested that should be possible, for iomega firmware 2.0 and above.  But the disk as it stands is NTFS-formatted – because when it was set up we had only Windows machines.  Now that disk contains much material in that NTFS.  I guess it might be possible to partrition the disk e.g. to keep the existing NTFS and add alongside it a HFS+ partition for Time Machine to use.  But it’s guesswork that carries risks (of disruption/damage to existing contents).  We want an easy geeking-minimal life, so maybe better to repurpose that NAS and get a purpose-made Apple Time Capsule instead. Either way, the evening wears on, so I’ll shelve that idea/investigation for now…

(more…)

Carbonite – Online Backup

Thursday, January 19th, 2012

Carbonite was drawn to my attention by a colleague, Mr. True, while we were discussing cloud-based services.

So here I just web-trawled a little. From that, it sounds like:

  • One must be wary of the “small print” about throttling.
    • Then again could anyone reasonably expect massive video file type storage for such a low price?
  • One alternative is Mozy
    • Some people find this better in terms of bandwidth,  service level and applications quality
    • It has apps for both iphone and android.

http://www.carbonite.co.uk/company

  • Carbonite automatically backs up new and changed files. You don’t have to do anything
  • One of Carbonite’s key advantages is offering unlimited backup for one fixed price
    • e.g. £41.95 / Year
  • PC & Mac
  • Carbonite for iPhone allows you to browse and share files from your iPhone.
  • Carbonite uses the same technology as banks to encrypt your data making it completely secure so only you can ever see it.
  • We have our own data centers where the the disk arrays are far more reliable than your computer’s hard drive.
  • Limitations?
    • “Bandwidth throttling – Yes (35GB, 200GB)” (?)
    • “Default max file size” (?)

http://www.dslreports.com/forum/r22910253-Rant-Carbonite-Backup-is-not-unlimited-as-they-claim

  • 2009-08-23, thedragonmas:
    • I have used both Carbonite and MOZY. I had Carbonite for 2+ years, worked as advertised, had over 450 gigs backed up no problem, always maxed out my connection. However, problem came from their client, which is a buggy piece of shit.
    • Switched to mozy, managed to re-back up all 450 gigs in 2 months using my Sprint EVDO Cellular (I get awesome speeds on my data card!).
    • Real problem with Carbonite is, when you select to backup “Everything” it in fact, does NOT backup everything. Such as video files, files over a certain gig size and what not, which will then need to be done manually.
      With MOZY, when I told it to backup EVERYTHING, it DID BACKUP EVERYTHING!
    • I am now a happy MOZY Customer. I am now pushing over 1tb of backed up data.
    • Only problem I have now, is if I need to restore my data, you get 30 days to restore your data before it is removed from their servers, and my present internet connection, that is not possible
  •  2009-09-11, Kramer:
    •  I was uploading about 5GB/day for the first 50GB or so and then was throttled back significantly after that.

Mobile Video Editing Hardware: Thoughts, Ideas & Dreams (continued)

Friday, January 13th, 2012

Following-on from my earlier post, Mobile Video Editing Hardware: Thoughts, Ideas & Dreams, where I considered an eventual migration from my laptop to a luggable PC, my thoughts veered (possibly having spotted cash-icebergs among them) towards an alternative solution:

  • Use the laptop for lightweight editing & compositing.
  • Use the desktop as a number-crunching RADI-attached server.

The two could be linked by:

  • Remote access / remote sessions (some of which via smartphone)
  • DropBox, e.g. have an active folder where I can drop Adobe Premiere XML and have it processed remotely by Adobe apps installed there.

Some links:

  • http://www.theregister.co.uk/2012/01/11/virtualisation_at_home_part_2/
    • (There’s no equivalent “_part_1” page.  I guess it’s just “Part 2” of that guy’s story).
    • DIY virtual machines: Rigging up at home, by Trevor Pott, 11th January 2012 14:33 GMT
    • Personal Virtual Machine (PVM) (in use) for about seven years with retail boxed version of Windows XP.
    • VM has been moved from virtualization platform to virtualization platform over the years … the most recent incarnation … inside Hyper-V.
    • …nothing beats Windows Server 2008 R2. It comes with a top-notch virtualisation platform (Hyper-V), and added RemoteFX support with Service Pack 1. You can still use the desktop operating system for all your HTPC needs, and a single Server 2008 R2 Standard license allows you to run both a host copy and a single virtual instance of Server 2008 R2.
    • In my case, the host instance does little more than play movies on the projector via VLC. The virtual instance of Server runs my Plex media server, and aggregates my many storage devices into a single share using DFS.
  • Shuttle Inc (Taiwan)

Mobile Video Editing Hardware: Thoughts, Ideas & Dreams

Tuesday, January 10th, 2012

Want a mobile “suitcase” editing system, something more (and more expandable) than a laptop but not too expensive.  Primarily to be used for Adobe CS5.5 for media enhancement / editing / compositing etc.

Nearest I found was NextDimension’s range around $7000 I think (but just guesswork – could be way off – would need to get a quote).   That would (if true) be around £4500 at current rates.  Plus import…  NextDimension call such machines “flextops” (Maybe they coined the term? Google searches on it mostly come up with them.)

Apart from the (mil/broadcast-lite but me-heavy) price, it might possibly be undesirably heavy to lug around much.   If so (just guessing, not assuming), it would make more sense to go for a modular quick-setup system.  So, starting to “think different” in this direction:

  • Standard tower, capable of taking new CUDA etc. graphics cards etc. as they emerge, but no need for more than say a couple of disks, maybe if SSD could even get away with just a single disk? (For system and media – inadvisable for traditional disks of course, what about for SSD’s?  I have much to learn about SSD’s though).
  • “Laptop-Lite” to talk to it.  With robust shuttered-stereoscopic HD monitor.
  • Gigabit network to NAS fast storage (SSD and/or RAID ?).

Maybe in that case it would be far more logical/affordable to use an existing laptop as a client working together with a luggable tower server, sufficiently light and robust for frequent dis/re -connection and travel.  And remote access of course (no heavy data to be exchanged, assume that’s already sync’d).  And some means to easily swap/sync applications and projects (data) between laptop and tower, giving the option to use just the (old) laptop on its own if needed.  All such options are handy for the travelling dude (working on train, social visits etc.) who also occasionally has to do heavy processing.  Then would just need a protective suitcase for the tower, plus another one for a decent monitor for grading etc.

I certainly won’t be spending anything just yet, but it’s good to have at least some kind of “radar”.

(more…)

XDCAM-EX to ProRes: How

Saturday, December 10th, 2011

I have a Sony XDCAM-EX clip at 1280x720p25 to be transcoded to ProRes, so it can be used as source for iMovie (for another user on another machine).

In principle it should be very simple: go on Mac, use Compressor to transcode the XDCAM footage to ProRes.  But as usual, things are pernickety…

Sequence:

  • First tried dragging the XDCAM [.mp4] file into compressor.
    • Not recognised.
    • Likewise the BPAV folder.
  • Next, I transcoded the XDCAM footage to “MXF for NLEs” format, using the Mac version of Sony Clip Browser
    • Not recognised.
  •  Next, opened the XDCAM Transfer app.
    • In this app, open the XDCAM’s BPAV folder.
    • The footage displays OK but how do I export it to a QuickTime [MOV] file?
    • Looks like I can’t.  It only offers to export to an [MP4] file.
    • Instead, I guess I’ll have to open it from FCP.
  • FCP
    • I opened a random existing FCP project.
    • The footage is 720p but the project/sequence settings are arbitrary (unknown to me)
    • FCP: File > Import > Sony XDCAM…
    • It imported to somewhere … but where?
    • FCP Browser: file > RightClick > Reveal in Finder
    • It was at [/Volumes/GRm HFS+/_Media/_Projects/2010-05-30 (Esp) Alison Doggies/020 Source/Sony XDCAM Transfer/SxS_01]
  • File System:
    • In other words, at whatever destination was last used by some app – presumably XDCAM Transfer or possibly FCP
    • The destination path was in fact specified in XDCAM Transfer, under its Menu: [XDCAM Transfer > Preferences > Import]
    • Moved the file instead to [/Volumes/GRm HFS+/_Media/_Projects/2009-11-22 (JRM) Lady of the Silver Wheel]
  • Compressor:
    • Open it in Compressor
      • Drag it to the “job-strip” (my term) in Compresor.
    • Compressor displays data about that clip (e.g. 1280×720, 25 fps)
    • Select jobstrip settings:
      • Select Setting
        • Settings: Apple > Formats > QuickTime > Apple Pro
          • Name: Apple ProRes 422
          • Description: Apple ProRes 422 with audio pass-through. Settings based off the source resolution and frame rate
      • Apply (Drag) Setting to Jobstrip
    • Destination
      • Leave destination unspecified.  Then it will be the same folder as Source.
    • Processing (transcoding) of this footage (1280x720p25) took about 3 minutes (on MacBook Pro 2009).
    • Result was not that much bigger than the original:
      • Originally recorded [.MP4] file: 1.19 GB
      • Rewrapped [.MOV] from XDCAM Transfer: 1.14 GB
      • ProRes [.MOV] from Compressor: 1.97 GB

Laptop-Based Mobile Editing: GRaid Mini (Out-Shines “Passport” Drive)

Wednesday, November 9th, 2011

Video-editing on-the-move (e.g. on a train) using a MacBook Pro (laptop) with Sony Vegas 9 (64-bit) as NLE (under Boot Camp / Windows 7), my practical experience was that a GRaid Mini external drive was far, far better than a 5400 rpm Western Digital “Passport” drive.   Consistent with the dual use of the MacBook, I partitioned the drive for both NFTS (Windows) and HFS+ (Mac OS), 50-50%.  Due to Boot Camp limitations (explained below), up till now I only ever used it “tethered” to its own mains-based power supply.  But now I see it can also be used mobile, powered from the MacBook – something that up till now I could only achieve under Mac OS, not under Windows.

When using Boot Camp / Windows on the MacBook, I initially tried the shirtpocket-sized Passport drive because it was small, light and powered from the laptop’s USB port.  While its data throughput wasn’t too bad, at least for single-channel HD editing (especially when only 1280720), when it came to cuts from one video clip (hence, in my case, video file) to another, there was a frustrating delay every time.

I also have a GRaid Mini drive, but it wasn’t obvious at first how to use it mobile when using Windows (on a MacBook).  That drive consists of two 7200  drives in RAID-0 configuation (striped, giving speed but no redundancy), and appears just like any single drive to the computer (no RAID management etc. needed).  The drive has not only a USB (2) port but also FireWire 800 (FW800) and eSATA ports.  While the latter two options work fine with the MacBook under Mac OS, they don’t work under Boot Camp / Windows.  I have tried many times and trawled many forums, no solution is apparent.  Under Mac OS the eSATA drive would ordinarily plug into an ExpressCard adaptor plugged into the laptop’s ExpressCard slot, but under Boot Camp / Windows, the ExpressCard slot doesn’t work, while for the FW800 port under Boot Camp / Windows, it appears to work at first but eventually crashes as a device when it attempts to communicate data (e.g. when copying files).

When connected only by USB to the MacBook under BootCamp / Windows, the GRaid Mini is not powered from that port, hence up till now I have relied on a mains power supply to that drive.  However, I discovered if, after first connecting by USB, you subsequently connect also by the FW800 lead, then the drive takes power from the FW800 yet communicates data via the USB lead.   Hooray!  I can use it on-the-move then!

The order in which the leads are connected is vital.  If by mistake the FW lead was connected first, then the drive would sense that as the data communications route, and subsequently fail in use.  It is vital that the USB connection is made first.  Likewise, on disconnecting the drive (following “ejection” by the computer’s file-system), disconnect the FW drive first.  The rule is FW lead: connect last disconnect first.

My experience of editing with the GRaid Mini is far more fluid hence more pleasurable and efficient.  Totally worth it.  None of the per-cut delay effects of the 5400 rpm Passport drive.  And now it can be used on-the-move, even with Boot Camp / Windows on a MacBook.  I just wish Apple would fix that Boot Camp isue with FireWire and ExpressCard ports!

Mobile Editing Blues: FW800 Unusable on MacBook via BootCamp

Thursday, October 27th, 2011

This is a problem I encountered some time ago, when I was running Boot Camp v3.1 on my MacBook Pro.  Since then I upgraded to v3.2.  I know there’s a v3.3 around but before upgrading I thought it worthwhile to see whether v3.2 had fixed that problem (especially since I couldn’t rule out the possibility of v3.3 reintroducing it).   Only one thing to do: prevaricate test.

  • Copy file from GRaid Mini (GRm) to Desktop:
    • 2GB fine
    • 12GB appears ok initially but then fails (to zero b/s transfer rate, then the Grm device “no longer exists”, at least until reboot)
  • Reverse: 2GB fails (same way) almost immediately.

OK not good thus far…

Next tried an alternative approach: run W7 as a Virtual Machine on Mac Os via Parallel.  I have Parallels v6.  Forum search revealed that there is no FW support in either v6 or v7, though the developers seem interested in knowing why people want it.

  • 2GB GRm to W7 Desktop: ok
  • The reverse: ok.

Had to stop there due to other work – and a very full W7 disk.

The next workaround to consider is attaching a NAS.  Ethernet bandwidths can be 1Gbps, hence more than FW800’s 0.8 Gbps, though I wonder if there could be any issues of lag / latency in this approach.  I’ll do some research and put up another post about this idea.

Training: Den Lennie’s “Music Video” Experience

Thursday, October 27th, 2011

I attended, working on one of the camera units.  Had a great time, learnt lots, at all sorts of levels.  Even how to make good use of the Movie Slate application on my iPhone!  Link: http://www.fstopacademy.com/

RAID Sharing Over Home Network: Works OK

Thursday, October 6th, 2011

Just for the record, currently this works:

  • On Mac Pro, in Windows (XP):
    • Choose any folder. For example:
      • On the RAID, this could be either of the two partitions:
        • NTFS_RAID (an NTFS partition)
        • RAID_ProAVIO (an HFS+) partition
    • Then [RtClk > Sharing and Security… > Sharing > Network sharing and security:
      • Share this folder on the network: YES
      • Allow network users to change my files: YES
        • Because some tools create “sidecar” files e.g. audio peaks or video indexes.
  • On Mac Book, in Windows 7:
    • From a file-browser, be it Windows Explorer or part of the File>Open feature of an application, go to [ Network > aComputer > aFolder ]
      • Tested by playing file in Windows Media Player and in Sony Vegas (video NLE).  Worked fine.  Latter added a [.sfk] sidecar-file to the RAID-folder.
      • Worked both for source on NTFS partition and for source on HFS+ partition, except the latter gave rise to prolonged “hourglass” delays before the file was accessed/linked, following which the video played smoothly.
  • What doesn’t work:
    • Unable to see Mac Book from Mac Pro.
    • Also, when MacPro is in Mac OS, MacBook W7 Windows Explorer > Network lists the MacPro as [MACPRO-2E4124] yet cannot connect to it, blaming a firewall (presumably the one on MacPro).
      • Network Error: Windows cannot access \\MACPRO-2E4124
      • Tried exiting Kapersky (on the MacPro>MacOS) but no change.
      • Tried stopping MacPro firewall (via Preferences > Security) but no change.

iPhone 4: Audio: Voice Memo: Audio File Extraction – on a Mac

Monday, September 26th, 2011

I audio-recorded some lectures by using the Voice Memo app in-built on my iPhone 4.  How does one get such recordings out of the phone and into an audio editing (or indeed audio/video editing) app on Mac or PC?  Here’s what I found, mainly by experiment:

  • For a Mac:
    • The iTunes app allows you to transfer/sync an iPhone’s Voice Memo audio recordings onto the computer.  In iTunes, under the device representing the phone, there is a Voice Memos folder. Inside this there is a list of recordings (objects) e.g. as follows:
      • 1  ✓ 21/09/2011 09:33
      • 2  ✓ 21/09/2011 10:43
      • 3  ✓ 21/09/2011 11:01
    • These entries correspond to M4A (FourCC=”MP4A”) files.
      • These files can be transferred (moved/copied) as follows:
        • Copy a file by dragging it out of Voice Memos folder in iTunes’ interface.
        • Move a file by dragging it out of iTunes’ user-specific Voice Memos folder in file system.  That folder is described below.
      • Additionally iTunes can export MP3 equivalent copies of the files:
        • iTunes (Voice Memos): Recording >RtClk> Create MP3 Version
    • The files are stored at (in my case) at:
      • /Users/davidesp/Music/iTunes/iTunes Media/Voice Memos
        • 21_09_2011 09_33.m4a
        • 21_09_2011 10_43.m4a
        • 21_09_2011 11_01.m4a
    • The nature of the files:
      • These files are stated by Mac’s Finder to be of type “MPEG-4 audio”, and are about 30 MB per hour.
      • VideoSpec is able to analyse them, it reports:
        • Container: M4A – QuickTime
        • Encoding: MP4A (FourCC), constant bitrate of 64 kbps, 16 bits, 44.1 kHz, stereo
          • However it makes no sense to encode stereo from a one-microphone device and indeed when imported to an audio editor (Audacity 1.3.13 beta) it only produces a single mono track.
    • Audio Editing/Processing (cuts/envelopes/effects e.g. dynamic range compression) was subsequently achievable by any of these:
      • Audacity (1.3.13 beta) can import the M4A file.  My (multi-platform) old-familiar.
      • Garage Band (e.g. as explained at http://macmost.com/editing-audio-files-in-garageband.html).
      • SoundTrack Pro (but can’t simply drag the file in – instead have to use File>Open).  OK but a little clunky (in my “newbie-to-this-app” opinion)

Avid Basics (not only for Final Cut Pro Users)

Saturday, September 24th, 2011
  • Avid Basics re Projects, Bins, Users, User-Settings
    • Notionally “Avid for Final Cut Pro Users” but generally useful introduction, complete with pragmatics.
    • Overview:
      • In this set of tutorials, it is assumed that a Shared project called “Tempest” will be edited by two people, Laura and David, one on day-shift (say) and the other on nights.  They can each select eithers’ settings (editor configs) via the Project Window’s Settings tab, under the User dropdown.  No need to log-out of Avid or indeed to switch User login sessions.  I guess a given user could create several configs e.g. default, david, david_temp
    • http://community.avid.com/forums/p/70313/393166.aspx
      • Module 1, Lesson 1 – Creating a Project
      • Private/Shared etc., User Settings (independent of User logged-in), Film features
    • http://community.avid.com/forums/p/70314/393171.aspx#393171
      • Module 1, Lesson 2 – Intro to Project Window – Avid for Final Cut Pro Users
      • MenuBar: [Tools > Console] brings up a text logger where progress can be recorded by users (as far as I can tell from the tutorial)
    • http://community.avid.com/forums/p/70315/393176.aspx
      • Module 1, Lesson 3 – Where to find Avid files
      • Explains where the Avid files are stored on the computer’s hard drive, and how to take your settings from system to system.
    • http://community.avid.com/forums/p/70317/393180.aspx
      • Module 1, Lesson 4 – Project Differences
      • Explains Avid Projects, Bins and how one can move & create stuff equivalently in Avid or in Windows Explorer (etc.)
      • Also explains some ways it differs from FCP.

Mac & PC: Inter-Filesystem Read-Write

Wednesday, May 11th, 2011

http://forums.creativecow.net/thread/45/868715

  •  Windows’ Paragon is better than Mac’s MacDrive ?

DropBox – Initial Reactions

Wednesday, April 20th, 2011

You register and get 2GB free storage area, visible in Windows Explorer as “dropbox”, alongside the existing “downloads”.  There, you can drag files (from elsewhere) to some place such as the eisting Public folder or you can create a new folder and make it personal or sharable.  If you make it sharable, you get prompted for an email address of the sharer.  Dropbox then auto-sends an email to that address.  The email includes a probably-unique secure (https) link (incorporating a big pseudorandom number) for them to click on.   OK so it’s not just public.  Great!

Rescuing a damaged file from a writeable DVD

Tuesday, March 1st, 2011

A physical DVD could be played but not copied, despite attempts on multiple computers (each with own drive).  The main VOB file was corrupted.  The disk appeared to the eye to be in good condition.  Wanted to copy a repaired version of that file, maybe with gaps or truncations, whatever could be salvaged.  Workaround was to do a ‘dd’.  That’s a unix command to “convert and copy a file”.  Mac OS X is based on a variant of unix, so chose to execute it from there.  The successful command was as follows:

  • dd if=/Volumes/’DVD VR’/VIDEO_TS/VTS_01_1.VOB of=tmp/vts_01_1.vob conv=noerror

Prior to this, naively-and-unsuccessfully tried Windows 7 PowerShell (PS) ‘cp’ command with ‘-Force’ option, but that was “barking up the wrong tree”.  An alternative suggestion, not attempted, was to use a streaming video processor such as VirtualDub. (more…)

Iomega NAS – Initial Experiences

Sunday, December 12th, 2010

Bought an Iomega Network-Attached Storage (NAS) device for a home network.  Here are my experiences:

  • Plugged it in and (naively) looked for it on Windows Explorer (networks section) but it didn’t appear there.
  • Ran its setup CD. Wary of bloatware, I selected Custom setup.  At the very least, that should list what kinds of thing could potentially be installed, then I can find out about them first.
    • Only installed the basic (non optional) software.
    • There is a backup tool, potentially useful but don’t want to dive into that just yet.
  • The result: a number of extra “drives” (drive letters) appeared, each being mapped to a network drive. The IP address was the same in each case: [192.168.1.68].   Although there was only one NAS, each “Network Drive” (letter) was mapped to a different folder on it, as follows:
    • music
    • photos
    • ActiveFolders
      • The NAS config (webpage) lets you define these, e.g. a folder where you can drop a file and it will turn up at your YouTube account.
    • public
      • What’s that for?  How”public” is it?
    • movies
    • backups
  • Reactions:
    • I had not expected that, just expected a single drive letter underneath which I would make my own folders, one for each user plus a shared one.
    • Presumably they are all intended to be shared among all users.
    • Maybe I can still make my own folders per-user etc.?  Maybe also can configure each user’s account to be tied to the appropriate one of these?
    • The NAS is a server, maybe it doesn’t allow access to root, only to folders?
  • User Manual
    • The Setup (disk) offers to Install the User Manual. Slightly concerned over that – only really want a PDF, not (potentially) more unknown bloatware.  Went ahead anyway.
  • NAS Config
    • To access the NAS Config (webpage):
      • In browser, for this particular device, can simply enter its name.
        • The default name was [Iomega-110496/], the numeric suffix being from the NAS’s MAC address.
        • One can change this name in NAS Settings, explained further below (where I indeed did so).
      • Its initially-displayed page is a demo ActiveFolder which presents a slideshow of any photos dragged into it.  So you can e.g. use your laptop as a photo-frame.  This is but one of several types of ActiveFolder.  More on these later…
      • Go to the Home page (via left-most menu)
        • Go to Remote Access tab (via upper tab-selections)
          • Ensure that Remote Access is disabled (for security).
          • Also can see the full MAC address there.
      • Go to the System page (via left-most menu)
        • Go to Date and Time tab (via upper menu)
          • Change it to the appropriate timezone.  By default was the US’s EST.  Changed it to GMT.
        • Go to Settings tab.
          • Change LED brightness down to minimum.  These gimmicks are simply annoying.
          • Change Machine Name from default
            [Iomega-110496] to memorable [magibox].

            • Test it works by entering magibox as the address in a fresh browser page.  Used Opera, which asked to put a “http://” at the front.
          • Change machine description to read “Julie’s Home Network Drive”.
      • Go to the Folders page (via left-most menu)
        • There is only one tab, “Folders”
          • Click the [+] button at lower-left (scroll-down) to add a new folder.
          • Add folders for each user- in line with my original scheme, to see if I can make that happen.
            • When create folder for self it says something like “since this is the first time you are accessing this folder, would you like to assign a password for it?
      • Go to the Active Folders page.
        • Go to Photos and Files tab
          • See the demo (slideshow) folder is there
          • Add a new folder, [Julie’s YouTube]
            • Inspected its Settings and it appeared already to have some YouTube details (username and password).
  • The NAS offers other features like Bittorrent (in some way) but I haven’t investigated those yet.

Checksum-based Backup Methodology – Thoughts

Friday, December 10th, 2010

Looks like I’ve evolved a hybrid system:

  1. Proper backup / synch tool: SuperFlexibleFileSynchronizer (abbrev to “SuperFlex”).
  2. Procedural with lower-level tools: File drag-copy combined with MD5Summer.

The idea is that normally I would use SuperFlex, but for occasions where I already have manually-created (supposed) mirrors, I can retrospectively check consistency at content (not just datetime and size) level.  Have yet to experiment with SuperFlex to see if it can verify existing copies of files (as opposed to copies that it is making).

File Backup / Sync / Verification

Thursday, December 9th, 2010

I was looking for an app to assist in synchronizing of copies of file systems, for example main and back-up copies.  I chose a good-looking application available for both Windows and Mac:

(more…)

md5 Check-Sum Apps for Windows

Thursday, December 9th, 2010

md5 Checksumming in Windows:

  • There is no in-built checksumming tool I know of in Windows, but they are standard in linux etc.
  • The best third-party md5 checksum tool I have discovered for Windows is MD5Summer (V. 1.2.0.11)  [http://www.md5summer.org/download.html].
    • When you run it, it first checks file associations and offers to set it as default for [.md5] files.  In W7 an attempt to do so may be denied (admin status required).
    • You first select a root-folder then files within it (or all, recursively).  To select files, it’s not sufficient to highlight them, you have to double-click them (or click the Add button), so that they appear in the right-hand pane.  Only then will the OK button be enabled.
    • By running it recursively from a root folder, rather than on individual folders, a single [.md5] file is created that does not pollute the subfolders (e.g. BPAV folders as recorded by XDCAM-EX).
    • Had a slight issue once where I manually copied from XP to W7 machines, then generated MD5 on source machine (XP) and verified on target machine (W7).  At first, the verifier immediately returned all mismatches.  So immediately in fact that it seems likely it was not performing any computation.  Later on it worked as expected.  Not known what changed to cause this, maybe simply the act of opening the md5 file in NotePad..

Concepts (to the best of my understanding):

  • A check-sum of some data is a “fingerprint” numeric value that is probably-unique to that data.  A fingerprint is useful when it is much smaller than the data it represents yet (virtually) uniquely identifies the content of that data.  A checksum of a file reflects that file’s data-contents but is not affected by the file name etc.
  • Various checksum schemes/algorithms exist, one of the most popular being md5.   An md5 fingerprint is extremely unlikely to be the same for any other file and will (for that reason) almost certainly change in value if the contents of a file is changed by even the tiniest degree.
  • A typical md5 tool will take one or more specified files and generate their fingerprints as corresponding entries (each entry being a filename and a fingerprint) in a [.md5] file.  Correspondingly it will take a given [.md5] file and report whether or not the stated and (re-) computed fingerprints agree.
  • An [.md5] file is a text file with one or more lines consisting of a checksum value (in hexadecimal) then a space then an asterisk then a file name, possibly preceded by a folder path (with respect to the folder containing the [.md5] file).  It can also have comment-lines, each beginning with a hash (#) character.  Example entries:
    • eb574b236133e60c989c6f472f07827b *fred.exe
    • [eb574b236133e60c989c6f472f07827b *tmp/fred.exe].
  • Some download sites include [ .md5] files alongside or along with their associated  files.  Some sites just display the fingerprint itself on the webpage.  Typically the purpose is to allow the user to check whether a download was complete or was corrupted.
  • The fingerprint computed by a typical checksumming application is not affected by a file’s name or read-only status etc., only by its data contents.  Thus it is not a total basis for consistency-checking of system configuration.

(more…)

MyBook on a multi-OS network (eg Mac & Win)

Sunday, May 16th, 2010

Can use Wester Digital’s MyBook drive on a network featuring multiple OSs, such as Mac OS as well as Windows, provided one does not install MioNet (bundled with the drive).  My instincts were right then (I did not install it).

In my case, the drive is formatted as NTFS, on a Mac it simply appears automatically in Finder then Mac OS is able to read it (Mac OS is able to read NTFS).  In retrospect, maybe would have been better to format it as HFS+ since then Windows could use MacDrive to not only read but write to it.  Meanwhile on Windows I found it necessary to run the “Discover” application bundled with MyBook, which configures the network drive mapping (to a drive letter).

Mac Aliases are like Windows Shortcuts but Better

Saturday, May 15th, 2010

From [http://www.dummies.com/how-to/content/working-with-aliases-in-mac-os-x.html] as of 2010-05-15]:

  • Aliases in Mac OS are like Shortcuts in Windows but better.   Aliases usually don’t break when you move or rename the original file; shortcuts do.
  • Alias feature has been on the Mac since at least OS 7 circa 1992. Shortcuts first appeared with Windows 95 in 1995.

sha1 Checksums

Saturday, May 15th, 2010

Based on [http://www.bresink.com/osx/sha1.html] as of 2010-05-15:

  • Apple recommends the use of the so-called SHA1 standard (Secure Hash Algorithm Version 1). Apple is using this method for their own security updates as well. For more information please also refer to Apple Support Article HT1652.
  • Perform the following steps:
    • Make sure you have the downloaded file displayed somewhere in the Finder.
    • Open the Terminal application.
    • Enter the command: [/usr/bin/openssl sha1 ] with a space at the end (not a Return).
      • Note the last character of ‘openssl’ is a small ‘L’ not a ‘1’
    • Drag the target file from the Finder window into the Terminal window. A path specification will appear in the Terminal window.
    • Now press the Return key (Enter).
    • You will see output similar to the following example:
      • SHA1(path specification)= 2eb722f340d4e57aa79bb5422b94d556888cbf38.

FCP Project Folder Structures: The (Non?) Fragmentation Issue

Saturday, May 15th, 2010

Reading book “Final Cut Pro Workflows” by Osder & Carman, 2008.  On page 284 it relays advice that it is best to put Project Files [.fcp] on a separate drive to the Media Drive (e.g. Media Drive= XSAN), due to:

  • Safety – not all on one drive
  • Avoid fragmenting the media drive (project files, cache and to a lesser extent render files) are written often (transient files?)

I’m not immediately convinced by these arguments:

How to view degree of fragmentation on an HFS volume:

  • [http://osxbook.com/book/bonus/chapter12/hfsdebug/fragmentation.html]
    • Command-line app to report a variety of storage-volume statistics, including fragmentation.
    • After download, can check the sha1 checksum, but this is of the executable, not the download itself ([.dmg] file).  The ‘sha1’ command is inbuilt to Mac OS, as: [/usr/bin/openssl sha1].  Note the last character of ‘openssl’ is a small ‘L’ niot a ‘1’.

Capture to HFS+, Use from Windows 7: Experiences

Monday, May 3rd, 2010

On MacBook Pro, used Sony Clip Browser (ClipBrowser) to import footage from a Sony XDCAM-EX to Mac OS HFS+.  This machine had MacDrive installed, enabling Windows apps to directly access files on the HFS+ file system.  On same machine, under Boot Camp (BootCamp) and Windows 7, ran Sony Vegas NLE.   Successfully imported and used footage by both of the following methods:

  • Sony Vegas’s Device Explorer [View > Device Explorer].
    • This took several minutes to import.
    • Importing resulted in copying the [.mp4] file (and other files) to the NTFS partition.
  • Direct use of [.mp4] on the HFS+ partition.
    • No need to import as such, just constructed waveforms etc.
    • This completed in seconds.
    • Only downside is that it ewas unable to save the waveform files etc., due to my config of MacDrive (read-only), so it would have to do this every time I opened the project.
      • Have yet to try the same thing when MacDrive has config for full read/write access.

FCP: Migrating, Consolidating, Tidying…e.g.bat ving…

Sunday, May 2nd, 2010

Sometimes the media files for a project are stored in several folders on one or more scratch disks. This happens if you selected a different scratch disk each time you captured media files, or when the name of the project file changes (this often happens when you save often to different filenames).  It would be tidier to consolidate all the media files for a project into one folder.Of course it may be better to avoid making a mess in the first place, as per this person’s advice: <<<I set up each project in its own folder so and then set up capture scratch render files and everything else. This way if I need to move anything you can move the folder onto the hard drive and you dont have to worry about what stuff to move.  >>>There are two approaches (I am aware of):

  • Manually copy/move the relevant files (project, source media, renders) then Reconnect the consequently offline media.
    • May only have to reconnect one file, provided the reconnect all files in this relative path option is checked.
  • Use Media Manager.  This is more efficient, because all of your clips are reconnected automatically after the media files are moved.

The Manual Approach:

  • The Project file.
    • This is the most important file.
      • If all the media in your project came from external media or generated media like titles or color mattes, this is the only file you actually need to move to another computer.  Everything else can be re-imported (e.g. batch-captured) or re-created (e.g. re-rendered).
      • But usually you’ll want to bring other files with you, as below.
  • Source Media files.
    • Recordings e.g. captured video and audio clips. These are usually stored in your Capture Scratch folder in a folder with your project name (unless you saved them elsewhere or moved them). If you move these to another computer, you should only have to reconnect one file, if you have the reconnect all files in this relative path option checked.
    • Other media/project files – you might have Motion or LiveType projects, Photoshop graphics, After Effects animations, or any number of other media files for your project- obviously you’ll want to move these over to the new system, and you might want the original project files also.
  • Render-files.
    • There are exceptions, but it is best not to move render files – this can lead to problems.
    • It’s better to rerender on the new system. If you must move them, they are in the “Render Files” folder.

The Media Manager Approach:

  • OS X:
    • Ensure there is a destination folder for the project.
  • FCP:
    • In the Browser, select all items in the project.
    • Make sure the Browser window is active.
    • Menu: [File > Media Manager]
    • Set the required options e.g. handles (margins), select the destination folder (mentioned above) and press [OK].
    • Files created by the Media Manager:
      • [<Destination Folder>]
        • Project File [<projname>.fcp].
        • [Media]
          • [<projname>]
            • <Media Files>
    • This structure is a bit mad but it’s the FCP convention…

Refs:

Windows 7: [Program Files] & [Program Files x86]

Saturday, May 1st, 2010

 In Windows 7, there are two applications folders.   Presumably the same as in Vista (which I have never used), as reported at [http://www.nvnews.net/vbulletin/showthread.php?t=87478]:

  • Program Files = 64bit apps
  • Program Files x86 = 32bit apps
  • If the installer is a 32bit application, it will install it to the x86 directory. If it’s a 64bit application it will install it to the regular one.

FAT32 Size Limits (Partition and File)

Saturday, May 1st, 2010

http://www.windowsitpro.com/article/file-systems/understanding-file-size-limits-on-ntfs-and-fat.aspx

  • FAT32 file-size limit is (2^32)-1 bytes, or one byte less than a full 4GB.
  • FAT32 has a 2TB limit on partition size.
    • Whereas FAT16 has a 4GB limit on partition size.
  • The file system is not the only source of constraint:
    • Windows XP and Windows 2000 limit partition creation to no larger than 32GB on FAT32. This limitation is by design: Microsoft wants you to use NTFS for large drives.
    • Applications could potentially impose their own limits

http://www.g-technology.com/Support/Pdf/G-RAID-mini-Manual.pdf

  • The Microsoft OS imposed limit of 32GB can be side-stepped by formatting instead from the Apple Mac OS (OS X).

Disk Space Usage / Inventory

Wednesday, April 21st, 2010

For Mac OS:

  • Disk Inventory X

For Windows:

  • WinDirStat
  • FolderSize

They are both pretty similar, in each case displaying filespace usage via a tree map looking like a patchwork of multicoloured PVC, each colour representing a different type of file (audio, video, application, document etc.).  Their advantage over traditional browser trees is you can see all the largest files and folders simultaneously (as a plan-view).  Tree maps (treemaps) are explained at http://w3.win.tue.nl/nl/onderzoek/onderzoek_informatica/visualization/sequoiaview/about_treemaps/ – they are formed by subdividing in alternate dimensions (horizontal/vertical), each time in proportion to relative size of item, be it folder or file.  A variation on this, employed by the above tools, is a cushion treemap [http://www.win.tue.nl/~vanwijk/ctm.pdf], where shading reveals the directory structure.  A further variation is the squarified treemap [http://www.win.tue.nl/~vanwijk/stm.pdf], where subdivision and grouping attempt (no guarantee of success) to make the rectangles as square as possible. (more…)