Blog

I installed the (free) LE version of Avid’s DNxHD onto bothWindows and Mac, in order to round-trip between Sony Vegas and FCP.  Having received no responses to my threads on forums for Sony Vegas and for FCP, I tried Avid’s forums.

• The “Avid on Windows” Forum:
  • PC MediaComposer to MAC FCP http://community.avid.com/forums/p/74442/416691.aspx#416691
    • Thread from August 2009.  Might well be out of date now.
    • If you export DVCProHD from your PC Avid you’ll want to check the “Use AvidDV Codec” box.  This will encode the file as AvidDV100 (which is Avid’s DVCProHD codec).  With the Avid codecs installed on your FCP system you’ll be able to read them.
    • Why don’t they mention DNxHD ?
• The “Avid on Mac” Forum:
  • Search on [fcp dnxhd]
    • http://community.avid.com/forums/p/84564/477328.aspx#477328
      • Error while loading DNxHD QT .mov into FCP
      • Someone with same issue as me: they rendered from Sony Vegas to DNxHD to import to FCP, and the media wasn’t recognized.
      • Thus far, that thread is inconclusive, with suspicions focussing on the precise format settings etc. and the fact that the media was rendered from Sony Vegas (as I guess the latter is relatively “unknown territory” to the Avid/DNxHD folks).
      • I copied this info (& thread link) to the Sony Vegas forum http://www.sonycreativesoftware.com/forums/ShowMessage.asp?MessageID=720994&Replies=11.  Useful replies:
        • …a smalll but critical point; the codec DNxHD does all the encoding of the video stream, vegas does not touch the encoding process. At most, vegas may update the file headers and starting meta data as it closes the file. If Sony has an issue it will be there, but I would question FCP file handling when it opening the file.
      • ?

Which is best?   Depends on the camera, scene and shot dynamics I guess.  The same point is queried at the following thread:

• http://www.dvinfo.net/forum/sony-xdcam-ex-cinealta/238593-steady-shot-smoothcam-both-what-2.html#post1551809

Some general advice from a computer-post-savvy author: definitely use the camera’s SteadyShot:

• http://www.gcvcc.com/March%202008%20%20Camera%20Techniques%20No2.pdf

Limitations of post

• Stabilization necessitates motion estimation and image reconstruction, which are extremely CPU-heavy, hence really slow to execute.
• Most stabilization apps (in post) can’t currently cope with motion-blurred edges or parallax effects (though both should be possible in principle, by deconvolution and 3D modelling both informed from multiple frames).
• For rolling-shutter-ed footage (e.g. CMOS sensors as in Sony Exmor as in Sony XDCAM-EX e.g. EX1 & EX3), there exist options to reduce the effect (don’t expect perfection, but may suffice):
  • Sony’s ClipBrowser 2.6 can remove this http://www.dvinfo.net/forum/sony-xdcam-ex-cinealta/467511-clipbrowser-takes-out-rolling-shutter.html.
  • Gunnar Thalin’s Deshaker plugin for VirtualDub can compensate for this.

My experiences:

• Stabilizing Tools:
  • Gunnar Thalin’s Deshaker works really well.  And it is multi-threaded, really speeds up the process.  The author says it is more intended for handheld pans etc. than fast-shaking shots from vehicles etc. (but has nevertheless seen good results in such situations).
    • The author says [http://forum.doom9.org/showthread.php?p=1418923] to try “to stabilize only on the most distant parts in the frames, since the moving inwards-effect is less there”.  And “you should probably increase the value for [discard motion of blocks that move > X pixels in wrong direction]. That’s to allow the blocks to move “freely” a little, since Deshaker can’t handle the “moving inwards”-effect.
    • Possibly equally applicable to other smooth/stabilize/deshake tools ?
  • Boris’s Optical/Motion Stabilizer (in Boris Red 4.3.3 on XP) is only single-threaded and I find it slower, clunkier and less intuitive than Deshaker.  Has a Smooth mode, which is like the others here, as well as a Stabilize mode (try to keep frame static, no good for motion then).  The other tools can be configured to do the same thing.
  • Mercalli in Sony Vegas has no mode for 720p50 but otherwise is pretty good and very intuitive and configurable.
  • FCP’s SmoothCam Effect worked best for a challenging clip for wobbly-hand-held camera tracking close past an object (a Formula-1 car) hence huge degree of moving-inwards effect.  The default settings worked straight away.  The result quality was way above that of the other tools.  On the other hand sometimes it’s not the best (sorry, forgot the exact situation).
• Cameras & Shots:
  • Historically, using a TRV33 DV HandyCam indoors (hence low-light hence long shutter time):
    • Way back in the past, using a (now ancient) TRV33 DV handy-cam (which has huge sensor margin i.e. spare pixels), when I shot big zooms to lecture audience individuals (e.g. question-time) I had the camera’s steady-shot (digital, not mirror) enabled  and used Gunnar Thalin’s Deshaker (VirtualDub plugin) also.  The result was astoundingly steady.
    • The same arrangement worked OK with hand or shoulder mounted cam for walk-throughs past nearby objects (e.g. walls, people, furniture).
    • An attempt to do the same thing without steady-shot enabled on the camera resulted in seriously motion-blurred edges.
  • Now, using a Sony EX3:
    • With camera Steady-Shot set to Medium, hand-held pans and motion past nearby objects seem to acquire a positional instability, as if the camera feedback mechanism needs greater damping. Maybe the camera’s internal mirror “suspension” has to be tighter (than the TRV33 digital equivalent) because it lacks the generous pixels margin of the TRV33?  or maybe something to do with the mirror’s inertia?  Or (real-time-constrained) processing-power?
      • Experimentation is needed with the camera’s other SteadyShot modes (High, Low).
      • In the absence of more generous sensor pixel margins, I wish it could be loosened-up e.g. to allow black borders (to crop in post) so as to permit smoother rides overall.

Problem:

• In Windows I export from Sony Vegas to AVI (CineForm).  In OS X I read the file into FCP and apply the SmoothCam effect, then export to ProRes.  In Windows, Sony Vegas, I replace the original file with the smoothed one.  The levels/gamma are wrong.

Solution (Search):

• Sony Vegas forum http://www.sonycreativesoftware.com/forums/ShowMessage.asp?ForumID=4&MessageID=718371
  • Use DNxHD
    •  Couple of tips re DNxHD:  709 color level assumes 16-235, and RGB assumes 0-255.
  • Force it back again: www.kenstone.net/fcp_homepage/gamma_mac_pc.html
    • But this presumably implies getting re-quantized twice (the roundtrip issue and the forcing), which for 8-bit footage I imagine could reduce the quality (banding).
• Uncertainties
  • Where and how does this gamma get applied?  In FCP I didn’t (knowingly) alter the levels (eg until it looked right), I just applied the SmoothCam filter.  So I guess it would look wrong on the (pre-SnowLeopard) Mac but I wouldn’t care.  Wouldn’t FCP then export back whatever it got but smoothed?  This one is really confusing.    Experiments needed (when I get time…) I guess.

http://digitalfilms.wordpress.com/2010/06/12/will-cs5-and-mc5-toast-fcp

• Great article comparing their pragmatics for various kinds of business and workflow situation.
• Also speculating on their likely future developments.

This is how I’m doing it today:

• Local System Disk
  • Render Files
  • Thumbnail Cache Files
  • Waveform Cache Files
• Local RAIDs (One stationary, one portable)
  • _App_Specific
    • Final_Cut
      • _Projects (just for misc [.fcp] files)
      • Audio Render Files
      • Autosave Vault
      • Capture Scratch
      • Render Files
  • _Media
    • _Library
    • _Projects
      • 2010-04-30 (Client) Event
        • 010 Preparation
        • 020 Source
        • 030 Projects
          • FCP
            • EventTitle v001.fcp

Notes:

• I would have put everything on the RAID but for the Final Cut Settings interface, which only allows the first three items above to be on one location.  In contrast, the other items can be specified in a small set of possible locations, each of which can be toggled (enabled/disabled).
• The structure below [Projects] should mainly branch by function then by application.  But it will vary from (real-world) project to (real-world) project.

From [http://support.apple.com/kb/TA27694?viewlocale=en_US] and own experiences, my own advice to myself is as follows.  Note however:

• It is not guaranteed to be correct or complete!  Just my own best-practice, so far.
• It will only delete from scratch areas it actually knows about i.e. as defined in FCP’s System Settings.
•  Even then, it doesn’t seem to delete everything.  After allegedly deleting all render files of all projects, I went through afterwards and found (and deleted) a number of files under [Render Files/Constant Frames].  Maybe a result of my messy initial novice practices - who knows!

My process:

• Open the project in Final Cut Pro
• Delete Project’s Render-Files:
  • Tools > Render Manager.
  • Use checkbox in the Remove column next to the name of the project whose render-files you wish to remove.
    • Warning: Do not check other projects or those projects’ render files will also be removed.
  • Click OK.
• Delete Media Files in the Browser:
  • Activate the Browser window.
  • Select everything you want to delete (e.g. Select All).
    • Warning: Don’t select any clips, images, audio or anything else that is used in another project, or is used by another application (a photo that you are also using in a DVD project, for example), , as you will not want them to be deleted.
  • Modify > Make Offline.
  • Click the Move Them to the Trash button.
  • Click OK.
  • Close your Final Cut Pro project (and don’t bother saving!)
• Delete the Project File
  • Drag it to the Trash after you have quit Final Cut Pro.
    • But how do I know where it is located, e.g. if it was the latest project, auto-loaded into FCP, I might not remember where it is.
      • Save Project As reveals the project file name e.g. [LenWed RecepLine Expt 001 copy.fcp].  Expanding the Save..As Finder reveals it is in a directory called [FCP Projects].  But not obvious where that folder is located.
  • There may also be project files in the AutoSave Vault(s).
    • In my case, this vault is on the System drive, because I save my main project files to real-world-project -specific folders on a separate Media drive (a local RAID).
• Check any scratch areas etc. on other disks, e.g. System disk, in case anything got (accidentally) written there, e.g. if system was accidentally powered-up when RAID was not running.
• Empty the Trash

What’s a good folder structure for FCP?  I read and heard lots of tips from great sources, but some of them (e.g. keep [.fcp] files on local drive not Media drive) sounded questionable, at least from my context, and anyway I always want to know the underlying reason for anything.  So it’s research-time again…

• .

There are several aspects:

• .

Context

• Participants
  • Individual, small team or large team
  • Standards-based, methodical or haphazard
• Application(s)
  • Apple (FCS etc.) -centric?
  • Combination of several apps e.g. several makes of NLE
• Media
  • File size and value.
• Storage system(s)?
  • Local
    • Normal or (relatively) slow drive
    • Fast drive e.g. RAID
  • Remote (probably shared).
    • Exceedingly Slow (e.g. web via standard broadband)
    • Slow (e.g. NAS such as WD MyBook)
    • Fast but with possible latency (e.g. “Fibre Channel” / SAN)
  • Integrated
    • Final Cut Server giving seamless access to all storage including near-line (easily-retrievable archive)?
      • Near-line storage is any medium that is used to copy and store data from the hard drive to a source that is easily retrieved [http://www.wisegeek.com/what-is-near-line-storage.htm].

Requirements

• Tidy organization
  • Easy to find stuff, including serendipitously.
  • Easy to manage stuff, e.g. archiving / shelving and reinstating.
• Performance
  • User-level.
    • Keep
  • System-level
• Security

My Conclusions (so far):

• The typical professional situation involves multiple users on a SAN.   In this case:
  • Each user should configure their apps (e.g. FCP) to save small and transient files to local disk.
    • Local disk has less latency and minimization of small-file traffic on SAN improves its performance to all users.
  • An individual (or item) -specific project file, which counts as a “small and transient file”, should not be saved routinely or automatically to shared media server but only saved there on an occasional basis (e.g. at end of day or project).
    • Restricting this operation to end-of-project might “discourage” users from corrupting each other’s files, though really that’s what Permissions are for (in Mac OS X / unix).
• Much advice relates to the “typical professional situation”, not all of it is appropriate to other situations.
• .

(more…)

Reading book “Final Cut Pro Workflows” by Osder & Carman, 2008.  On page 284 it relays advice that it is best to put Project Files [.fcp] on a separate drive to the Media Drive (e.g. Media Drive= XSAN), due to:

• Safety - not all on one drive
• Avoid fragmenting the media drive (project files, cache and to a lesser extent render files) are written often (transient files?)

I’m not immediately convinced by these arguments:

• Media drive should be backed-up in any case.  I use (slower) USB drives and file-synch.
• Fragmentation may be an old-school issue, less of an issue on Mac’s HFS +.
  • Example: article at [http://www.macmousecalls.com/files/1f6efbb25f58d77390499af1f984d2c7-33.html] linking-on to [http://www.kernelthread.com/mac/apme/fragmentation]
  • Example: Larry Jordan, “Final Cut Pro Power Skills” 2010, page 34, Chapter 2, “Defragmenting Your Hard Drive”: Once popular, this technique is no longer recommended.

How to view degree of fragmentation on an HFS volume:

• [http://osxbook.com/book/bonus/chapter12/hfsdebug/fragmentation.html]
  • Command-line app to report a variety of storage-volume statistics, including fragmentation.
  • After download, can check the sha1 checksum, but this is of the executable, not the download itself ([.dmg] file).  The ’sha1′ command is inbuilt to Mac OS, as: [/usr/bin/openssl sha1].  Note the last character of ‘openssl’ is a small ‘L’ niot a ‘1′.

Sometimes the media files for a project are stored in several folders on one or more scratch disks. This happens if you selected a different scratch disk each time you captured media files, or when the name of the project file changes (this often happens when you save often to different filenames).  It would be tidier to consolidate all the media files for a project into one folder.Of course it may be better to avoid making a mess in the first place, as per this person’s advice: <<<I set up each project in its own folder so and then set up capture scratch render files and everything else. This way if I need to move anything you can move the folder onto the hard drive and you dont have to worry about what stuff to move.  >>>There are two approaches (I am aware of):

• Manually copy/move the relevant files (project, source media, renders) then Reconnect the consequently offline media.
  • May only have to reconnect one file, provided the reconnect all files in this relative path option is checked.
• Use Media Manager.  This is more efficient, because all of your clips are reconnected automatically after the media files are moved.

The Manual Approach:

• The Project file.
  • This is the most important file.
    • If all the media in your project came from external media or generated media like titles or color mattes, this is the only file you actually need to move to another computer.  Everything else can be re-imported (e.g. batch-captured) or re-created (e.g. re-rendered).
    • But usually you’ll want to bring other files with you, as below.
• Source Media files.
  • Recordings e.g. captured video and audio clips. These are usually stored in your Capture Scratch folder in a folder with your project name (unless you saved them elsewhere or moved them). If you move these to another computer, you should only have to reconnect one file, if you have the reconnect all files in this relative path option checked.
  • Other media/project files - you might have Motion or LiveType projects, Photoshop graphics, After Effects animations, or any number of other media files for your project- obviously you’ll want to move these over to the new system, and you might want the original project files also.
• Render-files.
  • There are exceptions, but it is best not to move render files - this can lead to problems.
  • It’s better to rerender on the new system. If you must move them, they are in the “Render Files” folder.

The Media Manager Approach:

• OS X:
  • Ensure there is a destination folder for the project.
• FCP:
  • In the Browser, select all items in the project.
  • Make sure the Browser window is active.
  • Menu: [File > Media Manager]
  • Set the required options e.g. handles (margins), select the destination folder (mentioned above) and press [OK].
  • Files created by the Media Manager:
    • [<Destination Folder>]
      • Project File [<projname>.fcp].
      • [Media]
        • [<projname>]
          • <Media Files>
  • This structure is a bit mad but it’s the FCP convention…

Refs:

• [http://www.dvcreators.net/how-do-you-move-a-final-cut-project-from-one-computer-to-another]
• [http://forums.creativecow.net/thread/200/880525]
• [http://documentation.apple.com/en/finalcutpro/usermanual/index.html#chapter=94%26section=9]

 By “systems” i chiefly mean file systems, but also computer systems i.e. instances of FCP.  For simplicity, I ignore here the possibility of different versions of FCP.

• Google: [migrating “final cut” disk]
  • http://forums.creativecow.net/thread/8/1020585
    • “You can add multiple scratch disks in FCP, or just change it to relate to your current project, or when you import the new project put the all the media in the relevant location to your current scratch disks…”
      • “When I take my portable disk to the other machine to work, rather than work from the portable disk, I want to copy the entire project folder to the other machine’s drive. It’s the main workstation where the work will ultimately reside, so my portable drive is simply the “rough-in” drive. I work from home on this drive, then take what I’ve done over to the main machine and copy everything over. ”
      • “…it worked. I didn’t need to change that machine’s scratch location. I just copied the folder over, launced the project file and boom. No rendering necessary. “
    • .
• .