David Esp Video Technical Blog

When I start-up any application, I like to understand at least the main side-effects it’s having on my system.  In the case of Adobe’s primary video-editing apps, Premiere and After Effects, my experience (on Windows 7) is that they save intermediate preview-renders to the system volume.  This causes me the following concerns:

• System Volume may serve poorly as a media drive.
  • Larry Jordan, at least in the recent past, advises against using the system drive for media read/write.  On the upside, such drives may have high-bandwidth to the system, but on the downside, the system can interrupt their use with highest priority, whuch may (I guess) pose a risk to smooth playback (though I am aware that buffering may possibly reduce this risk, I haven’t done or seen any such calculations).  Cache files are indeed media files that are written and read.
  • On the other hand, an informed representative of a well-known UK supplier of video editing laptops advised me that in his experience, most users of laptops with only a single internal drive (as system drive) do use that drive in this way (for portability).
• System drive can become “clogged up”
  • System drive can become clogged-up by many or large video files of which that the user is only partially aware, their creation having happened implicitly during their use of the NLE etc.  Like temporary files only worse!
  • Ultimately the system drive can even become full, making the operating system itself sluggish or even less stable (and video playback less smooth.
  • Backup of a system drive that includes media files will typically require significantly greater archive space and will take significantly greater time (than a clean system).
• Migrate-ability is reduced
  • I like the idea of a video project being a free-floating data-object.  That is, it should not be tied to any particular instance of a data storage volume, let alone a particular computer (system).  It should be possible for all files relevant to a project to be stored on any volume, migrated to any other volume, plugged into any computer having appropriate installed applications, and everything to work the same way as when the project was on its original volume being edited on the original system.  That includes not only the source media files etc. but also the intermediate rendered files.

So what do the Adobe editing applications provide to enable my preferred working arrangement?

• Premiere:
  • [Edit > Preferences > Media]
    •  This defines the location of the folder [Media Cache Files], which contains pseudorandomly-named files.  Example Files:
      • [929_4372_01-125eeda9-ba0d-a8ea-4418-3480000001f0.ims]
      • [Rendered – 68721ea9-25e9-4f56-8430-4ca10101ace7-04602910-cd54-1a45-a5d7-557b000001f2.ims]
    • Default location: [c:\Users\…\AppData\roaming\Adobe\Common]
    • [Yes] Save Media Cache files next to originals when possible
      • e.g. for my XDCAM-EX files, inside a CLIPR folder (which contains the EX’s “essence” (.mp4) files, in this case [929_4491_01.MP4]), appeared the following files:
        • 929_4491_01.MP4 48000.cfa (9.2 MB)
          • Unknown, but the “48000” and the “a” on the end of “cfa” are suggestive of audio.
        • 929_4491_01.MP4 48000.pek (37KB)
          • Simply the peaks (waveform graphics data) file for the audio component of the essence-file.
      • Experience:
        • I clicked the [Browse] button and selected an area on my external media drive (a GRaid Mini) as: [H:\_App_Specific\Adobe].
        • Consequently, at my specified location the following folder appeared: [Media Cache Files]
  • Media Cache Database
    • Note:
      • When Premiere Pro imports video and audio in some formats, it processes and caches versions of these items that it can readily access when generating previews. Imported audio files are each conformed to a new .cfa file, and MPEG files are indexed to a new .mpgindex file. The media cache greatly improves performance for previews, because the video and audio items do not need to be reprocessed for each preview.
      • When you first import a file, you may experience a delay while the media is being processed and cached.
        A database retains links to each of the cached media files. This media cache database is shared with Adobe Media Encoder, After Effects, Premiere Pro, Encore, and Soundbooth, so each of these applications can each read from and write to the same set of cached media files.
    • Location: [c:\Users\…\AppData\roaming\Adobe\Common]
      • [Browse]
        • If you change the location of the database from within any of these applications, the location is updated for the other applications, too.
        • Each application can use its own cache folder, but the same database keeps track of them.
        • Example Experience:
          • I clicked the [Browse] button and selected an area on my external media drive (a GRaid Mini) as: [H:\_App_Specific\Adobe].
          • In response, a prompt came up saying “Move the existing media cache database to the new location, or delete it (Buttons: [Move] [Delete] [Cancel] ).
          • I clicked [Move]
          • Consequently, at my specified location the following folder appeared: [Media Cache]
    • Purging
      • [Clean]
        • This removes “orphan” cache files.
        • To remove conformed and indexed files from the cache and to remove their entries from the database, click [Clean]. This command only removes files associated with footage items for which the source file is no longer available.
          • Important: Before clicking the [Clean] button, make sure that any storage devices that contain your currently used source media are connected to your computer.
    • [Yes] Write XMP ID To Files On Import
      • Check this box to write ID information into XMP metadata fields.
      • e.g. hence [929_4491_01M01.XMP] sidecar-file (containing XMP metadata) got written into the CLIPR folder containing its associated media  file, here an XDCAM-EX essence file, [929_4491_01.MP4].
    • [Yes] Enable Clip And XMP Metadata Linking Check this box to link clip metadata to XMP metadata, so that changing one changes the other.
  • Intermediate-Preview Render Files
    • None of the above measures affect where intermediate/preview files get rendered to…
      • I proved this (in Windows 7) by deliberately causing a render then searching on “Huge” (>16MB) files created “Today”
      • Rendered files location was:[C:\Users\…\Documents\Adobe\Premiere Pro\5.5\Adobe Premiere Pro Preview Files\Untitled3.PRV]
    • A brief Google revealed several articles where the (sadly now obsolete) solution was a setting under [Edit > Preferences > Scratch Disks]
    • Eventually discovered that in CS5.5 these settings now resided in: [Project > Project Settings > Scratch Disks]
      • Here, all settings (affecting Capture and  Preview) were set to [Same as Project]
      • As it happened, my project location was [C:\Users\David\Documents\Adobe\Premiere Pro\5.5].
    • Experiment with a different project location:
      • Save As: [H:\_Media\_Projects\MyProject\030 Projects\Adobe\Experiments]
      • • The file [Expt A v001.prproj] appeared in it, but that was all.
      • Save a Copy: [H:\_Media\_Projects\MyProject\030 Projects\Adobe\Experiments]
        • The file [Expt A v001 Copy.prproj] appeared there, but that was all.
      • (The file open within Premiere remained as the original, not the copy)
      • After a while, the project file was joined by: [Adobe Premiere Pro Auto-Save]
    • Experiment to migrate the Preview-Render files:
      • In Windows Explorer, I created a folder named [Adobe Premiere Pro Preview Files]
      • Into that folder, from the similarly-named folder on the system volume, I dragged the existing folder [Untitled3.PRV]
    • Experiment: Premiere “knows” when render-files have gone, and prompts for their possible new location.
      • I deleted the project-specific render-files folder on the (external) project-drive.
      • I re-rendered, resulting in a fresh such folder, re-populated.
      • In Premiere, I Closed the project
      • Then in Windows I renamed the project-specific render-files folder, then back in Premiere I re-opened the project.
        • Premiere prompted with Browser titled: “Where is the File ‘Rendered – 5ac…..228.mpeg’?”
        • I selected the stated file in its newly-renamed folder, Premiere then found all the others there.
      • Result: the timeline render-region went green (i.e. “probably playable”).
    • Experiment: Once render-files are regarded as “gone”, they cannot be restored.
      • I closed the project, ensuring I did not save changes (such as the new location of the render files), then re-opened it.
        • As in the previous experiment (since I did not save changes), the render files folder could not be found.
        • Premiere thus prompted with Browser titled: “Where is the File ‘Rendered – 5ac…..228.mpeg’?”
        • This time however I simply used the [Skip All] option.
      • Closed the project.
      • Renamed the rendered-files folder back to its original name.
      • In Premiere, re-opened the project.
        • The timeline region remained red, indication no render-files were associated.
    • Experiment: Tidy migration of a project to a new location.
      • Warnig: in the case of doing a Copy (which is Windows’ default drag operation between different volumes), take care to ensure the Project (file) is not simply referencing the original preview files at the old location…
      • Drag both Project and its folders (including render-file folder) to a new location (e.g. on a new disk).
      • If name and relative location of folder are unchanged (as they ought to be, in good practice) then the files will be automatically detected and used, not even a user-prompt.
        • Just be sure though that the project isn’t simply referencing the render-files in their original location, if they are still present there.  Premiere is “lazy” in this respect.
    • Experiment: The relative location of the Rendered Files folder does matter (relative to the project file).
      • Tried putting the render files in a non-standard location.
        • The “Locate/Browse” prompt appeared
        • I located the file
        • All at first appeared well, and the corresponding section of the timeline went green
        • However, the “Composer” window simply displayed “Media Pending”.  That never went away.
    • Experiment:
      • When migrating, also need to move (or copy):
        • The Media Cache directories
          • Actually I’m not so sure about this. I tried exiting Premiere, renaming these directories and opening Premiere.  It created and repopulated the same directories in their original location, which in my case was an external drive.
        • The Source Media files
  • http://forums.adobe.com/thread/784220?tstart=0
    • I suggest marking each external with the drive letter that the user assigns to it, say Z:\. Then, whenever Z:\ is plugged in, it will always be seen as Z:\. This way, the NLE can keep up with where the Assets are located, starting with the drive letter.
    • If one is migrating Projects between computers, they will repeat this exact process in the OS of each computer.
    • Note: when doing the migration, ALL Assets, Scratch Disks, and the Project file, MUST be included on that external

Work Procedure for Migrate-ability:

• By associating cache and XMP files with the media (or its essence), Adobe projects are migratable.  However adding such files into the BPAV/CLIPR folder structure is considered by some applications to be an adulteration of that structure, requiring their deletion.   However, such deletion on an as-needed basis is not too onerous – given it is easy to do and in any case this situaion should rarely arise in practice.
• When using different disks, remember to re-define (in Preferences) the location of cache files etc.
  • One work-around would be to -re-set the cache location before opening any individual project.
    • Might be hard to remember to do when opening a project from within the NLE, easier to remember when double-clicking a project file in Windows Explorer.still
  • I’m not 100% sure what to do about these…
• As noted earlier:
  • When doing the migration, ALL Assets (Sources), Scratch Disks (Renders), and the Project file, MUST be included on that external.
    • I note that this says nothing about Cache Files etc. …