Blog

Confusions:

• I’m slightly befuddled by these terms, and what the corresponding operations are useful for. What do they do and how are they pragmatically useful? - in particular in a digital and even tapeless world. I first came across the term Decompose when investigating an Offline/Online workflow for AVCHD-Cineform-DNxHD(low bitrate)-DNxHD(high bitrate): http://blog.davidesp.com/archives/328.
• For example “recapturing” sounds at first like an antiquated concept in the digital world.  But maybe they have expanded the term to include re-ingestion of digital footage in higher-bitrate format, e.g. online as opposed to proxy in the case of XDCAM-HD or just different DNxHD bitrates (e.g. 145 instead of 36).  For example initially ingesting all digital footage at DNxHD(36) for offline editing then later recapturing only the used subset of that digital footage at DNxHD (145).  Advantage: reduced disk space requirement.  Just my guesswork though, still trying to work it all out…

Snippets providing Glimpses:

• Avid MC 5.5 Help [Understanding Decompose and Expert Decompose]:
  • Decompose creates new master clips in the bin based on the lengths of the clips edited into a sequence. You can then recapture media for the new master clips. Decompose breaks any links to the original source clips, and only the sequence and its new master clips are linked to the newly captured media files. If decomposing results in a change to the edit rate of some clips in the sequence, your Avid editing application adds Motion Adapter effects, or modifies existing Motion Adapter effects, to manage the edit rate change.
  • Decompose gives you more control over the recapturing process than simply recapturing a sequence without using decompose. You can sort or modify the clips that decompose creates before you recapture media. You can also use the Expert Decompose feature to customize how decompose operates.
  • For film projects, clips you create with decompose retain all the information from the original master clips, including Pullin column information, key numbers, ink numbers, or any other information formerly entered in the bin.
  • You do not need to decompose clips that were linked with the AMA (Avid Media Access) method.
• There is also anAvidHelp on [Decomposing Sequences].  But it is just a set of instructions on how to do it, not a helpful (to me) as an explanation of what it does or what it’s for.  To understand it, you need an existing understanding of the concepts & jargon.
• http://cstest.avid.com/forums/t/64647.aspx
  • Larry Rubin:
    • Consolidating will create a new set of pointer files and associated media files, separately discreet from the original set - essentially a clone.
    • Decomposing creates a new and offline clip list relative to a particular sequence, so that you can batch capture or import only media associated with that sequence instead of all the original source material. This is used primarily to conform a low-rez rough cut to high rez finished material.
• http://community.avid.com/forums/p/50488/283293.aspx
  • Q1:
    • Q: I want to save disk space. Is it possible to decompose an MXF Sequence in order to keep only the portions of footage used in the sequence on my hard disk? The footage was captured on P2 cards and Firestore.
    • A: Decompose is not what you want.  Select your sequence and right-click select consolodate.  Set your handles, then select “delete original media” and that should do it.  Remember that if you digitised anything that is not used in your sequence that you consolodate, this media will remain and you would need to manually delete that media as well. Be warned, though…  make certain that you are not trashing anything used in other sequences as well.
  • Q2:
    • Q: Can I consolidate more than one sequence at a time?
    • A: Yes - just select the sequences and go through the same routine
• http://hershleder.com/content/avid-media-composer-version-4
  • Recapturing and Decomposing: The new Expert Decompose feature allows you to decompose only certain tapes or clips used by a sequence, and allows you to fine tune target formats to which you want to recapture.
• Book: Avid Editing: “A Guide for Beginning and Intermediate Users”, By Sam Kauffmann.  I have that book, but a web-link version is still handy.  Paraphrasing:
  • Clip Menu (Page 341)
    • Consolidate / Transcode:
      • Consolidate lets you migrate media files to other drives, e.g. for better organization of files.
      • Transcode lets you change the format of a clip or sequence, e.g. from HD to SD.
    • Decompose:
      • Breaks a sequence into its component clips.  Example: you captured 1000 master clips at low resolution, e.g. DNxHD (36), to save space.  Thefinal sequence only uses 50 clips. After decomposing, you have those 50 clips in a bin and you only recapture those 50 clips at higher resolution e.g. DNxHD (220).
  • Uprezing Your Offline Sequence (Page 308).  That is, moving from offline to online resolution; nothing to do with sub-pixel enhancement etc.  Example: given an initial edit based on DNxHD (36), you want to replace those clips with equivalents “recaptured” (retranscoded from original recording, digital or otherwise) at DNxHD (220).
  • Recapture (Page 311).
  • Decompose (Page 311) breaks a Sequence into its component clips.  This enables you to organize these clips in bins, recapture them etc.  Without this feature, a recapture would take place over all clips at once - when you really only want certain clips recaptured, or maybe put off some until another day.
  • Recapture
• http://forums.creativecow.net/thread/45/866918#866918
  • Scott Simmons:
    • Decomposing gives you a new sequence with offline media that has had all the clips trimmed to only include the media used in the edit + handles. If you’ve edited at low resolution you then recapture only the media you need at the online rez.
• http://c2.avid.com/forums/t/98814.aspx
  • Randall L. Rike: Consolidate isn’t designed to create offline media.  It is used to move/copy media.  Similar to Decompose, it can move/copy only the segments of clips required to support a sequence.
  • Larry Rubin: The consolidation function is also useful for creating duplicate sets of clips and media that are linked to a different project than the one they were originally created in.
  • Job ter Burg:
    • Consolidate a sequence: make a dupe of all media that is being used in the sequence (with handles if you want). Consolidate master clips: make a dupe of the clip on a new destination drive. Consolidate subclips: make a dupe of this part of the media on a new destination drive. Decompose a sequence: generate offline clips for all media that is being used in the sequence (with handles if you want). You can’t decompose master clips.
    • Decompose I would use when finishing a tape based offline and preparing for tape capture to hi res media-right?Absolutely. Performing a Batch Capture would sort of have the same result, except Decomposer gives you more control over the batch process (as you can select which clips, etc).
    • You can’t decompose master clips
    • What I meant was if I consolidate a low res sequence, I am removing the unwanted media that does not reside in the sequence.Consolidating will not necessarily delete any unused media. And if you do select the option to delete originals, you still may have a lot of unused clips on board (that are completely unreferenced in the sequuence. I personally don’t consolidate that much anymore, as drive space is so cheap. If you do, I’d recommend consolidating onto an empty drive/partition, then hide/delete/remove the originals on your other media drives.
  • Pat Horridge:
    • Decompose is useful in that you create a new sequence and new master clips with whatever handles you set.  You can the decide how to re-capture/re-import those clips rather than with just batch capturing/batch importing the sequence which will just prompt for the required tapes/files.
    • Consolidate is normally used to create a new sequence and subset of Avid Media files for media management (so you can blow away the rest) or moving to a different system.
      • Take care that not all media can be consolidated (re AMA and “foreign types”) This skips those clips that link to media that can’t be made into Avid media files and your consolidated sequence will still be linked to the original AMA’d media.
      • So with AMA media and a mixture of sources you may need to consolidate and then transcode to get a complete set of media. (I’ve made a feature request to have this as a single function)
• http://aviddssupport.avid.com/pdf/v10.2/AvidDS_HiRes_Workflow_Color_Guide.pdf
  • ?? (Not yet read) ??

A year or two ago I did a web-survey of project planners, and I liked the sound of this one, it sounded more realistic to use than standard ones.  So now I have time to “think” again, I registered for a 30-day trial.

• http://www.liquidplanner.com

Also as I guessed/hoped, there is an associated App for mobile phones:

•  http://www.liquidplanner.com/mobile

Some SharePoint (2010) training material (free):

• http://forums.whirlpool.net.au/archive/988174
• http://technet.microsoft.com/en-us/library/cc262788.aspx
• https://www.microsoft.com/downloads/details.aspx?FamilyID=7BB3A2A3-6A9F-49F4-84E8-FF3FB71046DF&displaylang=en#filelist

Some non-free material:

• http://www.lynda.com/Office-2007-tutorials/sharepoint-2007-essential-training/503-2.html

I was looking for an app to assist in synchronizing of copies of file systems, for example main and back-up copies.  I chose a good-looking application available for both Windows and Mac:

•  Super Flexible File Synchronizer [http://www.superflexible.com]

(more…)

In Windows XP, wanted a folder right-click menu option to bring up a MSDOS command shell.  This proved very simple to add, and gave the further benefit of the text-based (command-line) user interface (TUI?) retaining customisations (e.g. green text).  The steps were (for Windows XP):

• Windows Explorer: Tools > Folder Options.
• Select the File Types tab.
• Go to NONE / Folder.
• Select the entry labeled Folder
•  Press Advanced button.
• Select New
• In the action block type “Command Prompt” (without the quotes).
• In the app block type “cmd.exe” (without the quotes).
• Save and exit Folder Options.

(more…)

This is how I’m doing it today:

• Local System Disk
  • Render Files
  • Thumbnail Cache Files
  • Waveform Cache Files
• Local RAIDs (One stationary, one portable)
  • _App_Specific
    • Final_Cut
      • _Projects (just for misc [.fcp] files)
      • Audio Render Files
      • Autosave Vault
      • Capture Scratch
      • Render Files
  • _Media
    • _Library
    • _Projects
      • 2010-04-30 (Client) Event
        • 010 Preparation
        • 020 Source
        • 030 Projects
          • FCP
            • EventTitle v001.fcp

Notes:

• I would have put everything on the RAID but for the Final Cut Settings interface, which only allows the first three items above to be on one location.  In contrast, the other items can be specified in a small set of possible locations, each of which can be toggled (enabled/disabled).
• The structure below [Projects] should mainly branch by function then by application.  But it will vary from (real-world) project to (real-world) project.

From [http://support.apple.com/kb/TA27694?viewlocale=en_US] and own experiences, my own advice to myself is as follows.  Note however:

• It is not guaranteed to be correct or complete!  Just my own best-practice, so far.
• It will only delete from scratch areas it actually knows about i.e. as defined in FCP’s System Settings.
•  Even then, it doesn’t seem to delete everything.  After allegedly deleting all render files of all projects, I went through afterwards and found (and deleted) a number of files under [Render Files/Constant Frames].  Maybe a result of my messy initial novice practices - who knows!

My process:

• Open the project in Final Cut Pro
• Delete Project’s Render-Files:
  • Tools > Render Manager.
  • Use checkbox in the Remove column next to the name of the project whose render-files you wish to remove.
    • Warning: Do not check other projects or those projects’ render files will also be removed.
  • Click OK.
• Delete Media Files in the Browser:
  • Activate the Browser window.
  • Select everything you want to delete (e.g. Select All).
    • Warning: Don’t select any clips, images, audio or anything else that is used in another project, or is used by another application (a photo that you are also using in a DVD project, for example), , as you will not want them to be deleted.
  • Modify > Make Offline.
  • Click the Move Them to the Trash button.
  • Click OK.
  • Close your Final Cut Pro project (and don’t bother saving!)
• Delete the Project File
  • Drag it to the Trash after you have quit Final Cut Pro.
    • But how do I know where it is located, e.g. if it was the latest project, auto-loaded into FCP, I might not remember where it is.
      • Save Project As reveals the project file name e.g. [LenWed RecepLine Expt 001 copy.fcp].  Expanding the Save..As Finder reveals it is in a directory called [FCP Projects].  But not obvious where that folder is located.
  • There may also be project files in the AutoSave Vault(s).
    • In my case, this vault is on the System drive, because I save my main project files to real-world-project -specific folders on a separate Media drive (a local RAID).
• Check any scratch areas etc. on other disks, e.g. System disk, in case anything got (accidentally) written there, e.g. if system was accidentally powered-up when RAID was not running.
• Empty the Trash

What’s a good folder structure for FCP?  I read and heard lots of tips from great sources, but some of them (e.g. keep [.fcp] files on local drive not Media drive) sounded questionable, at least from my context, and anyway I always want to know the underlying reason for anything.  So it’s research-time again…

• .

There are several aspects:

• .

Context

• Participants
  • Individual, small team or large team
  • Standards-based, methodical or haphazard
• Application(s)
  • Apple (FCS etc.) -centric?
  • Combination of several apps e.g. several makes of NLE
• Media
  • File size and value.
• Storage system(s)?
  • Local
    • Normal or (relatively) slow drive
    • Fast drive e.g. RAID
  • Remote (probably shared).
    • Exceedingly Slow (e.g. web via standard broadband)
    • Slow (e.g. NAS such as WD MyBook)
    • Fast but with possible latency (e.g. “Fibre Channel” / SAN)
  • Integrated
    • Final Cut Server giving seamless access to all storage including near-line (easily-retrievable archive)?
      • Near-line storage is any medium that is used to copy and store data from the hard drive to a source that is easily retrieved [http://www.wisegeek.com/what-is-near-line-storage.htm].

Requirements

• Tidy organization
  • Easy to find stuff, including serendipitously.
  • Easy to manage stuff, e.g. archiving / shelving and reinstating.
• Performance
  • User-level.
    • Keep
  • System-level
• Security

My Conclusions (so far):

• The typical professional situation involves multiple users on a SAN.   In this case:
  • Each user should configure their apps (e.g. FCP) to save small and transient files to local disk.
    • Local disk has less latency and minimization of small-file traffic on SAN improves its performance to all users.
  • An individual (or item) -specific project file, which counts as a “small and transient file”, should not be saved routinely or automatically to shared media server but only saved there on an occasional basis (e.g. at end of day or project).
    • Restricting this operation to end-of-project might “discourage” users from corrupting each other’s files, though really that’s what Permissions are for (in Mac OS X / unix).
• Much advice relates to the “typical professional situation”, not all of it is appropriate to other situations.
• .

(more…)

Based on [http://www.bresink.com/osx/sha1.html] as of 2010-05-15:

• Apple recommends the use of the so-called SHA1 standard (Secure Hash Algorithm Version 1). Apple is using this method for their own security updates as well. For more information please also refer to Apple Support Article HT1652.
• Perform the following steps:
  • Make sure you have the downloaded file displayed somewhere in the Finder.
  • Open the Terminal application.
  • Enter the command: [/usr/bin/openssl sha1 ] with a space at the end (not a Return).
    • Note the last character of ‘openssl’ is a small ‘L’ not a ‘1′
  • Drag the target file from the Finder window into the Terminal window. A path specification will appear in the Terminal window.
  • Now press the Return key (Enter).
  • You will see output similar to the following example:
    • SHA1(path specification)= 2eb722f340d4e57aa79bb5422b94d556888cbf38.

Reading book “Final Cut Pro Workflows” by Osder & Carman, 2008.  On page 284 it relays advice that it is best to put Project Files [.fcp] on a separate drive to the Media Drive (e.g. Media Drive= XSAN), due to:

• Safety - not all on one drive
• Avoid fragmenting the media drive (project files, cache and to a lesser extent render files) are written often (transient files?)

I’m not immediately convinced by these arguments:

• Media drive should be backed-up in any case.  I use (slower) USB drives and file-synch.
• Fragmentation may be an old-school issue, less of an issue on Mac’s HFS +.
  • Example: article at [http://www.macmousecalls.com/files/1f6efbb25f58d77390499af1f984d2c7-33.html] linking-on to [http://www.kernelthread.com/mac/apme/fragmentation]
  • Example: Larry Jordan, “Final Cut Pro Power Skills” 2010, page 34, Chapter 2, “Defragmenting Your Hard Drive”: Once popular, this technique is no longer recommended.

How to view degree of fragmentation on an HFS volume:

• [http://osxbook.com/book/bonus/chapter12/hfsdebug/fragmentation.html]
  • Command-line app to report a variety of storage-volume statistics, including fragmentation.
  • After download, can check the sha1 checksum, but this is of the executable, not the download itself ([.dmg] file).  The ’sha1′ command is inbuilt to Mac OS, as: [/usr/bin/openssl sha1].  Note the last character of ‘openssl’ is a small ‘L’ niot a ‘1′.