Archive for May, 2009

HD (compressed & uncompressed) and Computer (Capture & RAID) Bandwidths

Sunday, May 31st, 2009

I like to have a feel for bandwidths – roughly how big these things are and why.  Especially when I start running out of storage space and having to purchase additional disk drives etc…   Also I like to get a handle on how adequate my RAID will be for capturing uncompressed HD.  Google-search & calculations:

  • In the case of HD-SDI (1080i50), bitrate is around 1.5 Gbps [http://www.filmmakermagazine.com/fall2006/line_items/hdmi.php]
  • In our case we have 720p50, but that apparently has about the same uncompressed bandwidth as 1080i50, namely 1.5Gbps [http://www.dtvforum.info/index.php?showtopic=9100]
  • My system has a RocketRaid+ProAVIO RAID5 arrangement.  I have not yet tested its uncompressed capture ability (I havea BlackMagic card), but it (if I’ve calculated it right), it looks like it should handle it easily.
    • RocketRaid themselves report tests [http://www.highpoint-tech.com/PDF/RR2522/RR2522_Performance_Report.pdf] involving an AJA capture card and a RAID5 array (as is mine) giving read and write speeds at about 400MBps (=3.2Gbps, because B i.e. Bytes = 8*b = Bits). 
      • If I understand this right, my system has a capture bandwidth twice that an uncompressed HD signal.  Plenty of headroom then.
  • ProRes’s 123 Mbps is not far off the “100Mbps” figure often mentioned in relation to the Convergent Design flash memory recorder, at which level the recording, compressed (as is ProRes) in a variant of MPEG2, is regarded as almost indistinguishable from uncompressed.

Sequence Renders: What & Where?

Sunday, May 31st, 2009

Suppose you added an effect to the timeline and as a result it shows a red “Unrendered” (or “Needs Rendering”) line above it.  You press Command-R (say) and rendering happens and now you can play it in real-time.  But what exactly is this rendering?  What format is it in and where does it go, i.e. what file(s) in what folder(s) ?

  •  The rendered file appears in a project-specific subfolder of the Rendering folder of FCP’s Scratch area.  In my case I defined the latter to be on the RAID.
    • Location was: [RAID_ProAVIO/App-Specific/Final_Cut/ FCP_Scratch/Render Files/Voda 2009 Estab 001/]
    • Example file name is: “Sequence 1-WP1-00000001
  • The coding format is ProRes 422 – Standard, not High Quality (which would be overkill).  This is good news!
    • VideoSpec displays its FourCC code to be “apcn”
    • The rendered file’s bitrate, according to VideoSpec, is an average of 123Mbps, around four times that of the HDCAM EX high-quality mode (35Mbps).  Nevertheless a great reduction on the uncompressed bitrate (around 1.5 Gbps) and broadly comparable to Convergent Design’s 100Mbps, judged by some to be virtually indistinguishable from uncompressed quality.  So it bodes well.

    Final Cut Effects for Video Denoising

    Sunday, May 31st, 2009

    Denoising: We’re talking video here (not audio), with twinkling speckles, e.g. due to low exposure or high gain hence a poor signal to noise ratio.   There aren’t any video denoisers built in to FCP, but plugins exist, both free and commercial.  There is some discussion of these at DvInfo forum [www.dvinfo.net/conf/non-linear-editing-mac/137219-de-noise-plug-fcp.html#post1151393].  The best one I have found so far, at the cost of money and rendering intensity, is Neat Video,  available for most of the major NLE’s, including FCP.

    •  The “Too Much Too Soon” (TMTS) filter package [http://www.toolfarm.com/plugins/index.php/Too_Much_Too_Soon_Free_FXScrips] includes a denoiser.  And it is free.
      • Didn’t remove much noise for me, despite setting it to max denoising.
    • Joe’s Filters (filter package), commercial, includes a Smart Denoiser.  
      • This worked reasonably well.  It removed most othe dynamic (twinkling) noise but left static pixel variations, presumably associated with pixel variation in the camera’s sensor which as a result were more obvious.  The result was like looking at a clean moving image through a dirt-patchy glass.
      • It was found to be best placed after the levels&colors adjustments, i.e. at the bottom of the filter chain.
    • Neat Video [www.neatvideo.com], the best denoiser I have ever been able to find on Windows, is now available for FCP on Mac.
      • Tried the FCP version.  Totally outclasses the ones so far seen.  At the cost of money and (intensive) rendering time.  Well worth it though if you need the highest quality.
      • There is a free demo download but it is crippled to only affects a central rectangle of the image.  So don’t judge it by what happens at the margins region, rather use it as a basis for comparing the denoised central bit.
    • CHV – the Repair filter collection, including a denoiser, at [http://www.chv-plugins.com/cms/Fx-Script/Repair-collection/NoiseReduction.php]
      • (I have not yet tried this)

    Final Cut – Find/Try the Basic Enhancement Effects

    Sunday, May 31st, 2009

    Basic effects from my point of view are color curves, color correction and denoising.  Indeed the test footage I had was in dire need of all these enhancements.  Here, I began seeking how to do a simple levels adjustment, hoping to move on afterwards to S-Curves as in Sony Vegas.  However even for simple levels adjustment the process in FCP turned out not to be as simple as I hoped…  Probably the best solution was FCP’s 3-Way Color Corrector.  Had high hopes for using Apple Color (the application, not a fruit-specific effect) but for some reason FCP’s “Send to Color” option was greyed-out.

    •  .

     To begin with, here are some handy tips: 

    • FCP: FCP > User Preferences: Undo Levels -> 99, Recent Clips -> 20 (max poss)
    • Playing the Unrendered: Some kinds of effects cause a red line to appear above the affected part of the timeline, meaning essentially that the effect is so heavy on CPU that it can’t be played in real-time and must first be rendered (e.g. hit Command-R), then the red line disappears and it plays smoothly.   For me, such renders typically take a minute or two – boring and frustrating when all you want to do is experiment with settings etc.  There is however an alternative: Alt-P for “force rendered playback”.  Not so smooth but far less boring.  Reference: [http://forums.dvdoctor.net/showthread.php?t=42108]

    Now the actual Effects experiments:

    • Levels Adjustment – Attempt 1 (works but awkward): The in-built Levels effect.  This effect is not what I expected – it is a little weird and non-intuitive.  Others agree.  
    • Levels Adjustment – Attempt 2 (easier): Joes Filters [www.joesfilters.com] makes a (commercial) filter with 5-way controls (min & max input & output plus gamma) as in Vegas.  The sliders are tiny and fiddly – but presumably that’s an Apple thing (?)
    • Levels Adjustment – Attempt 3 (failed so far): Apple Color.  That’s reportedly far more sophisticated in capability than FCP’s filters and allegedly [http://www.dvinfo.net/conf/non-linear-editing-mac/141670-color-curves-fcp.html] includes S-curve levels adjustments equivalent to Color Curves as in Sony Vegas. It is advised by some [www.dvinfo.net/conf/non-linear-editing-mac/141670-color-curves-fcp-2.html] to use that instead.   Allegedly [www.dvinfo.net/conf/non-linear-editing-mac/141670-color-curves-fcp.html] a good instructional DVD on it is Creative COW Master Series : Stop Staring and Start Grading with Apple Color by Walter Biscardi.
      • So I tried to use Color, as follows: [FCP: (Sequence) > (RtClk) > Send To > Color].  However Color was greyed out (unavailable).  Why?  Color is installed OK and can be started up as an app in its own right.  Tried doing that then “..Send To” it again, but no difference – still greyed-out.
    • Some other people [http://www.dvinfo.net/conf/non-linear-editing-mac/144594-apple-color-send-fcp-doesnt-work.html] had some other difficulties in FCP<>Color exchange.  Though not exactly as mine, it illustrates kind of the fiddly fussiness that can become an issue.
    • Yet others [www.dvinfo.net/conf/non-linear-editing-mac/141670-color-curves-fcp.html] suggest instead using the Low/Mid/High levels sliders of the 3-Way Color Corrector to achieve the same effect as an S-Curve.  
      • There is a free tutorial for this at Creative Cow, explaining not only levels adjustment but also my standard “face cropping & vectorscope” method of flesh color correction.  There are plenty more FCP tutorials there of similar calibre.  
      • I tried the 3-Way and yes it is by far the easiest method, very simple and intuitive, though once again the (Apple-standard?) sliders are a bit too small and fiddly.
      • A further advantage of using the 3-Way is that it also in Apple Color and can an even (in some sense) be migrated from FCP to Color, if that’s where you later prefer to work on the grading.
    • Still wish I had a proper interactive S (or Bezier) levels/colors curve effect though…

    Import Sony EX XDCAM 720p50 into Final Cut: On a Surer Footing

    Sunday, May 31st, 2009

    Following advice passed on from Sony (by Obi Lidobe/Ejukene), I installed latest Mac versions of Sony XDCAM support software [from www.sonybiz.net/EX under Tools/Downloads], namely:

    • Final Cut Pro XDCAM Transfer v2.9.0
    • Clipbrowser v2.5
    • SxS Device Driver
    • XDCAM EX Log&Transfer Utility v1.0
    • This set of items presumably constitutes the “separate plug‑in from Sony … required to enable (the XDCAM EX) features”.

      Tried using FCP to see if the new XDCAM EX features are now available:

    • FCP: FCP >Easy Setup: The nearest obvious template was “XDCAM HD”.  But does that cover “XDCAM EX” or only their older optical disk based XDCAM ?
    • FCP: File > Import > Sony XDCAM Transfer
    • Transfer: required setup of cache folders etc.  Default was on personal area of Mac HD.  Chose instead to put it on my RAID: [/App-Specific/Sony XDCAM Transfer], with subfolders /Cache, /Import, /Export Scratch.  Can change these later on under [Sony XDCAM Transfer > Settings]
    • Transfer: File > Add Sources.  Can select multiple files.  Can access NTFS-captured files in original Sony (BPAV) folder structure.  Automatically queues job to build thumbnails (only does that, doesn’t convert the files to anything yet).  
    • Transfer: Thumbnails appear.  Can multi-select them.  
    • Do [Transfer: (Files) > (RtClk) > Import].  This generates equivalent QuickTime (“.mov”) files to the Import folder you specified earlier under Settings.  QUESTION: Does one have to define such settings individually for each project?  How best to organize their location?  The size of this equivalent file is almost identical to that of the original “.mp4” file (in the EX’s BPAV folder).  Presumably it is the same codec (data), just re-wrapped.  QUESTION: Would it be better to import them to ProRes (since this – unlike the XDCAM EX format – is a non-GoP format)?
    • Incidentally, the [Transfer: (File) > (RtClk) > Export Clips to Folder] option generates equivalent “.mxf” files, again broadly the same size, prompting for the destination folder.  QUESTION: Is this intended for foreign NLE’s such as Avid or Vegas rather than FCP?  Is it “export” in the sense of “from FCP to outside world”?
    • As a result of the [Transfer: …Import] operation, “.mov” files exist in the Import folder (as defined in Settings) and also they are listed in FCP’s Browser (top-left pane). 
    • FCP: Drag one of these files to Timeline.  Prompts: “Not the same format – change?”.  Say YES.   So I guess my doubts about the appropriateness of the HDCAM HD template were justified.  QUESTION: What format is it now then?
    • FCP: (Sequence) > (RtClk) > Properties: 50fps, 1280×720, Compresor = (XDCAM EX 720p50 (35 Mb/s VBR) ).  QUESTION: Does that imply that when the Sequence is rendered (as in getting rid of the “Needs Rendering” red line), it is rendered to this same format?  QUESTION: To reduce generational losses (in this highly compressed format), would I be better off setting this to be ProRes, and if so then what format?  Presumably if the original clips had been imported to ProRes, I would have been automatically prompted for that Sequence setting when I dragged those imports onto the Timeline (from FCP Browser).  
    • Also presumably the ProRes approach would benefit external (to the FCS system) workflows e.g. enhancement in VirtualDub (via the Windows read-only version of the ProRes codec, just a “Dec”oder).  In that scenario, the external Windows app would have to write to some other broadly equivalent format such as Cineform.  Is there a Cineform decoder for Mac?  If I had it, would the [FCP: File > Import] or some other way be able to import (convert to FCP-friendly format) that format?   Not just a case of re-wrapping but re-compression.  Would I have to use Compressor in principle – and is it capable of it in practice?

    Import Sony EX XDCAM 720p50 into Final Cut: Websearch

    Sunday, May 31st, 2009
    • Googled for any well-known solution to the lack of 720p50 support.  An article from Nov 2007 at the Aulich & ADamski website [http://www.aulich-adamski.de/en/perm/720p50-capturing-editing-in-final-cut-pro] said that the previous (to then) lack of 50p support in FCP had been addressed in updates (at that time) to Final Cut Pro, Motion, DVD Studio Pro, Color, Cinema Tools, Soundtrack Pro and QuickTime Pro.
    • At [www.apple.com/support/releasenotes/en/Final_Cut_Pro_6.0_rn] the Release Notes of Final Cut indicated that XDCAM EX support had been added in FCP version 6.0.2.  This support included for XDCAM EX 720p50 VBR as per my footage.  
    • Notes also said: “Important: A separate plug‑in from Sony is required to enable these features”.  But no actual link etc. was stated…
    • Notes also said: “Once you ingest your XDCAM EX footage to QuickTime media files on your scratch disk, you can simply choose the XDCAM EX Easy Setup that corresponds to your footage and edit as you would with any other native format in Final Cut Pro”.  So at least now I know my footage should be ingested to QuickTime not MXF.
    • Notes also said, re “50p Support”: “Along with support for a number of recent 50p video formats, Final Cut Pro 6.0.2 includes support for 50 fps timecode in all timecode fields and project properties. A new 50 @ 25 timecode format has been added for deck support and EDL compatibility with 50 fps formats.  Note: Motion, Color, and Soundtrack Pro now support 50p footage as well.
    • Notes also said, re “Updating Motion and Motion Templates”:Final Cut Pro 6.0.2 master templates require Motion version 3.0.2 or later. By upgrading to Motion version 3.0.2 or later, you take advantage of important fixes and improvements made in the Motion application and templates.”
    • I checked the version of my installed Motion and it was indeed 3.0.2.
    • Setting up an FCP project (though not specifically XDCAM EX): article at [http://www.kenstone.net/fcp_homepage/project_setup_fcp6_balis.html]
    • Organizing FCP project disks/folders/files in a tidy fashion: article at http://www.larryjordan.biz/articles/lj_archive.html
    • How to render for a DVD [http://discussions.apple.com/thread.jspa?threadID=1991874&tstart=0]: “Make sure you sequence is rendered and then export a Quicktime using current settings with compression markers and do not make it self-contained (assuming you compressing on the same machine). In Compressor pick a DVD setting that works for you delivery.”

      Import Sony EX XDCAM 720p50 into Final Cut: Initial Stumbles

      Sunday, May 31st, 2009

      Initial Stumbles 

      • Shot footage on an EX3 suitable for DVD and web.  Following advice of “gurus” such as Alister Chapman, shot it in 720p50 mode.  Having done so, wanted to get it into Final Cut for editing etc.
      • Initially, used ClipBrowser (v2.0) to ingest the footage.  Didn’t know if I should do it the same way I did on Windows for Sony Vegas, that is by generating a “.mxf” file or by generating a “.mov” file.  Tried both.  These are containers, not codecs.  The “.mxf” file is Material Exchange Format while “.mov” is QuickTime.
      • Wanted to know more about the contents e.g. the codecs used and their settings.  To get this, used VideoSpec – a video analyzer broadly like GSpot on Windows)
      • MXF contents: FourCC “mpg2” (MPEG-2), Bitrate 35000 kbps,  fps 50, 1280×720, PAR 1:1, DAR: 16:9, Chroma subsampling format YUV420p.
      • MOV contents: FourCC “xdva” (XDCAM), Bitrate 34900 kbps, fps 50, 1280×720, PAR 1:1, DAR 16:9, Chroma YUV420p.
      • In FCP, tried to find a standard setting suitable for this, but nothing matched. In particular there were 60p formats but not 50p formats – frustrating.   Instead made a “best guess” at the most closely matching format and customized it.  I think I ended up with format “HDV 720p50” but was concerned that HDV may have different standards (e.g. number and aspect ratio of pixels) to that of my EX XDCAM footage.