Improved Profiles (400+ Export Profiles, New Profile UI, and more) (#5116)

* Fixing invalid legacy profiles (invalid sample ratios, and invalid name)

* Adding initial profile definitions, which will be used to generate over 400 unique profile resolution + frame rates. These include the ATSC standards, Sony D1, CIF, Blu-ray, DVD, DVB, HDV, VCD/SVCD, Facebook/Instagram, NTSC, PAL, and VGA.

* Fixing legacy profile for square quarter wide, to use more accurate 16x9 size

* Updating JSON profile definitions with sample aspect ratios, for anamorphic sizes, and initial checkin of manage.py script to help parse, test, and update these formats for the future.

* Updating ATSC and VGA profiles to fix some invalid aspect ratios

* Moving old legacy export profiles into their own legacy directory

* Adding initial generated export profiles (364)

* Improve export preset manage.py script, to find legacy profiles, and generate valid export presets from our JSON definitions

* Migrate default-profile on launch of OpenShot - search legacy profiles and find equivalent new preset

* Updating Profile and Preference dialogs, to display the Profile.LongNameWithDesc() function - standardizing profile names

* Migrate new profile manage.py script to use libopenshot Profile class for storing and processing

* Updating profile naming scheme to explude "anomorphic", and to use the Key() from libopenshot::Profile class. Re-generating all profiles.

* Fixing issue with Profile interlace detection and updating profile files

* Fixing issue with Profile interlace detection in manage.py script. Also, adding initial Presets update logic into manage.py script.

* Simplifying profile definitions with the purpose of more uniform generated titles

* Big update to profile manage.py script, to allow for identification of Anamorphic formats (and adding Note automatically), and better generation of profiles with a more uniform description.

* Updating actual profile files with modified descriptions / more uniform

* Moving merged profiles into legacy folder

* Adding a couple new FPS for vertical FHD profiles

* Fixed manage.py script to correctly compare legacy profiles to new generated profiles

* Fix CIF definitions to no longer combine NTSC and PAL formats on a single definition, and regenerate affected profiles

* Adding 18 new generated profiles

* Rename Square to SQ in profiles. Update profile naming logic in manage.py script, to move NTSC/PAL/SD/HD to the front of the description, and Anamorphic to the end of the description.

* Update profiles with updated description formatting

* Update profile manage.py script to correctly replace preset.xml profile names, and correctly order FHD, UHD, and other labels to the front of the description

* Fix bug in manage.py script for profiles, to correctly output the DAR ratio, and fixed vertical resolutiosn to use the original width (i.e. 720p Vertical instead of 1280p Vertical), and regenerating all profiles

* Updating profile descriptions to use 2.5K and 3K labels for certain resolutions, and regenerating affected profiles

* Updating presets to use new profile description names

* Initial commit of new Profile treeview and UI, which is filterable/searchable. Now requires user to click "Ok" button to prevent accidental changes to Profile.

* - Remove caching settings from New/Open projects - and move it to the TimelineSync object (where the JSON is applied or loaded in libopenshot)
- Add playback caching setting when Profile dialog changes profile in accept() method

* - integrating new Profile UI with Export UI
- separate Profile UI from default project id, and allow it to be more generic
- Allow profile to initially select any profile by description or Key()
- Correctly apply the profile of a loaded project, and fallback to a default if not found

* Add double click handler to selecting a new Profile

* Use same profile format for Preferences and Export dialogs.

* - Allow export presets to include empty audio and video codecs, as well as 0 sample rate and 0 channels (indicating missing video or audio tracks)
- Allow up to 384000 sample rate (and 0)
- Update GIF preset to remove audio info
- Update MP3 preset to remove video info
- Improve manage.py profile script, to auto-generate our *.rst profile documenation (list of presets and profiles)
- Update profiles documentation (presets list and profiles list)

* Fixing some Codacy nitpicks:
- Removing unused variables
- Removing unreachable code
- Wrapping some long code lines
- Removing unneeded elif

* Fixing some Codacy nitpicks:
- Replaced minidomxml with defusedxml (safer)
- Removed unneeded else's
- Removed unneeded overloaded method
- Removed unused key in dict loop
- Fixed some long lines that needed to wrap

* Fixing some Codacy nitpicks:
- Avoid using built-in variable
- Removing unneeded elif
- Reduce complexity of Exception string

* Adding the adjusted width after applying SAR/PAR to our documentation and profiles.rst file - to help demonstrate how SAR is applied to the final display resolution

* Updating profiles.rst documentation and screenshots which have changed - related to our new Profiles UI. Also tweaking the column widths of the generated profile list syntax.

* Experiment with file renaming for Windows gitlab runner issues - removing these profile files

* Re-generating all profiles, with updated Key() / filenames - to exclude the ":" character, since Windows has issues with this character in file paths.

* Updating DVD presets to use ac3 audio, since aac seems invalid on newer FFmpeg versions

* - Wrapping profile changing into undo/redo transaction
- Fixing bug when rescaling FPS to a new FPS, to prevent undo/redo history from being erased

Author: jonoomph

doc/images/export-profiles.jpg

@@ -285,39 +285,82 @@ def new(self):
 
         # Get default profile
         s = get_app().get_settings()
-        default_profile = s.get("default-profile")
+        default_profile_desc = s.get("default-profile")
+
+        # Get profile (if any)
+        profile = self.get_profile(profile_desc=default_profile_desc)
+        if not profile:
+            # Fallback, in case of missing/invalid default profile
+            profile = self.get_profile(profile_desc="HD 720p 30 fps")
+
+        # Update default profile (if it changed)
+        if profile and default_profile_desc != profile.info.description:
+            log.info(f"Updating default-profile from legacy `{default_profile_desc}` to `{profile.info.description}`.")
+            s.set("default-profile", profile.info.description)
+
+        # Apply default audio playback settings to this data structure
+        self.apply_default_audio_settings()
+
+        # Set default project ID
+        self._data["id"] = self.generate_id()
+
+    def get_profile(self, profile_desc=None, profile_key=None):
+        """Attempt to find a specific profile"""
+        profile = None
+
+        # Loop through legacy profiles
+        LEGACY_PROFILE_PATH = os.path.join(info.PROFILES_PATH, "legacy")
+        legacy_profile = None
+        for legacy_filename in os.listdir(LEGACY_PROFILE_PATH):
+            legacy_profile_path = os.path.join(LEGACY_PROFILE_PATH, legacy_filename)
+            try:
+                # Load Profile and append description
+                temp_profile = openshot.Profile(legacy_profile_path)
+                if profile_desc == temp_profile.info.description:
+                    legacy_profile = temp_profile
+                    break
+            except RuntimeError:
+                # Ignore legacy parsing errors
+                pass
 
         # Loop through profiles
         profile_dirs = [info.USER_PROFILES_PATH, info.PROFILES_PATH]
         available_dirs = [f for f in profile_dirs if os.path.exists(f)]
         for profile_folder in available_dirs:
-            for file in os.listdir(profile_folder):
+            for file in reversed(sorted(os.listdir(profile_folder))):
                 profile_path = os.path.join(profile_folder, file)
+                if os.path.isdir(profile_path):
+                    continue
                 try:
                     # Load Profile and append description
-                    profile = openshot.Profile(profile_path)
-
-                    if default_profile == profile.info.description:
-                        log.info("Setting default profile to %s" % profile.info.description)
-
-                        # Update default profile
-                        self._data["profile"] = profile.info.description
-                        self._data["width"] = profile.info.width
-                        self._data["height"] = profile.info.height
-                        self._data["fps"] = {"num": profile.info.fps.num, "den": profile.info.fps.den}
-                        self._data["display_ratio"] = {"num": profile.info.display_ratio.num, "den": profile.info.display_ratio.den}
-                        self._data["pixel_ratio"] = {"num": profile.info.pixel_ratio.num, "den": profile.info.pixel_ratio.den}
+                    temp_profile = openshot.Profile(profile_path)
+
+                    if profile_desc == temp_profile.info.description:
+                        profile = self.apply_profile(temp_profile)
+                        break
+                    if legacy_profile and legacy_profile.Key() == temp_profile.Key():
+                        # Switch from legacy profile to new profile
+                        profile = self.apply_profile(temp_profile)
                         break
 
                 except RuntimeError as e:
                     # This exception occurs when there's a problem parsing the Profile file - display a message and continue
                     log.error("Failed to parse file '%s' as a profile: %s" % (profile_path, e))
 
-        # Apply default audio playback settings to this data structure
-        self.apply_default_audio_settings()
+        return profile
 
-        # Set default project ID
-        self._data["id"] = self.generate_id()
+    def apply_profile(self, profile):
+        """Apply a specific profile to the current project data"""
+        log.info("Setting profile to %s" % profile.info.description)
+
+        # Update default profile
+        self._data["profile"] = profile.info.description
+        self._data["width"] = profile.info.width
+        self._data["height"] = profile.info.height
+        self._data["fps"] = {"num": profile.info.fps.num, "den": profile.info.fps.den}
+        self._data["display_ratio"] = {"num": profile.info.display_ratio.num, "den": profile.info.display_ratio.den}
+        self._data["pixel_ratio"] = {"num": profile.info.pixel_ratio.num, "den": profile.info.pixel_ratio.den}
+        return profile
 
     def load(self, file_path, clear_thumbnails=True):
         """ Load project from file """
@@ -386,6 +429,13 @@ def load(self, file_path, clear_thumbnails=True):
             # Upgrade any data structures
             self.upgrade_project_data_structures()
 
+            # Get profile (if any)
+            project_profile_desc = self._data.get("profile", "HD 720p 30 fps")
+            profile = self.get_profile(profile_desc=project_profile_desc)
+            if not profile:
+                # Fallback, in case of missing/invalid default profile
+                profile = self.get_profile(profile_desc="HD 720p 30 fps")
+
             # Apply default audio playback settings to this data structure
             self.apply_default_audio_settings()
 

doc/images/export-simple.jpg

@@ -78,8 +78,14 @@ def changed(self, action):
                                                               "layers", "scale", "profile"]:
             return
 
+        # Disable video caching temporarily
+        openshot.Settings.Instance().ENABLE_PLAYBACK_CACHING = False
+
         try:
             if action.type == "load":
+                # Clear any selections in UI (since we are clearing the timeline)
+                self.window.clearSelections()
+
                 # Clear any existing clips & effects (free memory)
                 self.timeline.Close()
                 self.timeline.Clear()
@@ -105,6 +111,9 @@ def changed(self, action):
             log.info("Error applying JSON to timeline object in libopenshot: %s. %s" %
                      (e, action.json(is_array=True)))
 
+        # Enable video caching
+        openshot.Settings.Instance().ENABLE_PLAYBACK_CACHING = True
+
     def MaxSizeChangedCB(self, new_size):
         """Callback for max sized change (i.e. max size of video widget)"""
         while not self.window.initialized:

doc/images/profiles-dialog.jpg

@@ -354,13 +354,14 @@ def dispatch_action(self, action):
         self.update_watchers()
 
     # Perform load action (loading all project data), clearing history for taking a new path
-    def load(self, values):
+    def load(self, values, reset_history=True):
         """ Load all project data via an UpdateAction into the UpdateManager
         (this action will then be distributed to all listeners) """
 
         self.last_action = UpdateAction('load', '', values)
-        self.redoHistory.clear()
-        self.actionHistory.clear()
+        if reset_history:
+            self.redoHistory.clear()
+            self.actionHistory.clear()
         self.pending_action = None
         self.dispatch_action(self.last_action)
 

doc/images/profiles.jpg

@@ -17,7 +17,7 @@
 	 med=""
 	 high="256 kb/s"></audiobitrate>
 	<samplerate>48000</samplerate>
-	<projectprofile>HD 1080i 25 fps</projectprofile>
-	<projectprofile>HD 1080i 30 fps</projectprofile>
-	<projectprofile>HD 1080p 25 fps</projectprofile>
+	<projectprofile>FHD PAL 1080i 25 fps</projectprofile>
+	<projectprofile>FHD 1080i 30 fps</projectprofile>
+	<projectprofile>FHD PAL 1080p 25 fps</projectprofile>
 </export-option>

doc/profiles.rst

@@ -5,7 +5,7 @@
 	<title translatable="True">DVD-NTSC</title>
 	<videoformat>dvd</videoformat>
 	<videocodec>mpeg2video</videocodec>
-	<audiocodec>aac</audiocodec>
+	<audiocodec>ac3</audiocodec>
 	<audiochannels>2</audiochannels>
 	<audiochannellayout>3</audiochannellayout>
 	<videobitrate
@@ -17,6 +17,6 @@
 	 med="192 kb/s"
 	 high="256 kb/s"></audiobitrate>
 	<samplerate>48000</samplerate>
-	<projectprofile>DV/DVD NTSC</projectprofile>
-	<projectprofile>DV/DVD Widescreen NTSC</projectprofile>
+	<projectprofile>NTSC SD Anamorphic 480i 29.97 fps</projectprofile>
+	<projectprofile>NTSC SD Widescreen Anamorphic 480i 29.97 fps</projectprofile>
 </export-option>

src/classes/project_data.py

@@ -5,7 +5,7 @@
 	<title translatable="True">DVD-PAL</title>
 	<videoformat>dvd</videoformat>
 	<videocodec>mpeg2video</videocodec>
-	<audiocodec>aac</audiocodec>
+	<audiocodec>ac3</audiocodec>
 	<audiochannels>2</audiochannels>
 	<audiochannellayout>3</audiochannellayout>
 	<videobitrate
@@ -17,6 +17,6 @@
 	 med="192 kb/s"
 	 high="256 kb/s"></audiobitrate>
 	<samplerate>48000</samplerate>
-	<projectprofile>DV/DVD PAL</projectprofile>
-	<projectprofile>DV/DVD Widescreen PAL</projectprofile>
+	<projectprofile>PAL SD Anamorphic 576i 25 fps</projectprofile>
+	<projectprofile>PAL SD Widescreen Anamorphic 576i 25 fps</projectprofile>
 </export-option>

src/classes/timeline.py

@@ -19,6 +19,6 @@
 	<samplerate>48000</samplerate>
 	<projectprofile>HD 720p 25 fps</projectprofile>
 	<projectprofile>HD 720p 29.97 fps</projectprofile>
-	<projectprofile>HD 1080p 25 fps</projectprofile>
-	<projectprofile>HD 1080p 29.97 fps</projectprofile>
+	<projectprofile>FHD PAL 1080p 25 fps</projectprofile>
+	<projectprofile>FHD 1080p 29.97 fps</projectprofile>
 </export-option>

src/classes/updates.py

@@ -6,16 +6,16 @@
     <export-to>Video Only</export-to>
 	<videoformat>gif</videoformat>
 	<videocodec>gif</videocodec>
-	<audiocodec>aac</audiocodec>
-	<audiochannels>2</audiochannels>
-	<audiochannellayout>3</audiochannellayout>
+	<audiocodec></audiocodec>
+	<audiochannels>0</audiochannels>
+	<audiochannellayout>4</audiochannellayout>
 	<videobitrate
 	 low="384 kb/s"
 	 med="5 Mb/s"
 	 high="15.00 Mb/s"></videobitrate>
 	<audiobitrate
-	 low="96 kb/s"
-	 med="128 kb/s"
-	 high="192 kb/s"></audiobitrate>
-	<samplerate>48000</samplerate>
+	 low="0 kb/s"
+	 med="0 kb/s"
+	 high="0 kb/s"></audiobitrate>
+	<samplerate>0</samplerate>
 </export-option>

src/presets/avchd.xml

@@ -5,14 +5,14 @@
 	<title translatable="True">MP3 (audio only)</title>
     <export-to>Audio Only</export-to>
 	<videoformat>mp3</videoformat>
-	<videocodec>mpeg2video</videocodec>
+	<videocodec></videocodec>
 	<audiocodec>libmp3lame</audiocodec>
 	<audiochannels>2</audiochannels>
 	<audiochannellayout>3</audiochannellayout>
 	<videobitrate
-	 low="384 kb/s"
-	 med="5 Mb/s"
-	 high="15.00 Mb/s"></videobitrate>
+	 low="0 kb/s"
+	 med="0 Mb/s"
+	 high="0 Mb/s"></videobitrate>
 	<audiobitrate
 	 low="96 kb/s"
 	 med="128 kb/s"

src/presets/dvd_ntsc.xml

@@ -19,10 +19,10 @@
 	<samplerate>48000</samplerate>
 	<projectprofile>HD 720p 25 fps</projectprofile>
 	<projectprofile>HD 720p 30 fps</projectprofile>
-	<projectprofile>HD 1080p 25 fps</projectprofile>
-	<projectprofile>HD 1080p 30 fps</projectprofile>
+	<projectprofile>FHD PAL 1080p 25 fps</projectprofile>
+	<projectprofile>FHD 1080p 30 fps</projectprofile>
 	<projectprofile>HD Vertical 720p 25 fps</projectprofile>
 	<projectprofile>HD Vertical 720p 30 fps</projectprofile>
-	<projectprofile>HD Vertical 1080p 25 fps</projectprofile>
-	<projectprofile>HD Vertical 1080p 30 fps</projectprofile>
+	<projectprofile>FHD Vertical 1080p 25 fps</projectprofile>
+	<projectprofile>FHD Vertical 1080p 30 fps</projectprofile>
 </export-option>

src/presets/dvd_pal.xml

@@ -17,5 +17,5 @@
 	 med="256 kb/s"
 	 high="320 kb/s"></audiobitrate>
 	<samplerate>44100</samplerate>
-	<projectprofile>VGA NTSC</projectprofile>
+	<projectprofile>NTSC SD SQ VGA 480p 29.97 fps</projectprofile>
 </export-option>

src/presets/flickr_HD.xml

@@ -17,5 +17,5 @@
 	 med="256 kb/s"
 	 high="320 kb/s"></audiobitrate>
 	<samplerate>48000</samplerate>
-	<projectprofile>Mobile 360p</projectprofile>
+	<projectprofile>NTSC SD 1/4 QVGA 240p 29.97 fps</projectprofile>
 </export-option>

src/presets/format_gif.xml

@@ -17,5 +17,5 @@
 	 med="256 kb/s"
 	 high="320 kb/s"></audiobitrate>
 	<samplerate>44100</samplerate>
-	<projectprofile>VGA NTSC</projectprofile>
+	<projectprofile>NTSC SD SQ VGA 480p 29.97 fps</projectprofile>
 </export-option>

src/presets/format_mp3.xml

@@ -19,10 +19,10 @@
 	<samplerate>48000</samplerate>
 	<projectprofile>HD 720p 25 fps</projectprofile>
 	<projectprofile>HD 720p 30 fps</projectprofile>
-	<projectprofile>HD 1080p 25 fps</projectprofile>
-	<projectprofile>HD 1080p 30 fps</projectprofile>
+	<projectprofile>FHD PAL 1080p 25 fps</projectprofile>
+	<projectprofile>FHD 1080p 30 fps</projectprofile>
 	<projectprofile>HD Vertical 720p 25 fps</projectprofile>
 	<projectprofile>HD Vertical 720p 30 fps</projectprofile>
-	<projectprofile>HD Vertical 1080p 25 fps</projectprofile>
-	<projectprofile>HD Vertical 1080p 30 fps</projectprofile>
+	<projectprofile>FHD Vertical 1080p 25 fps</projectprofile>
+	<projectprofile>FHD Vertical 1080p 30 fps</projectprofile>
 </export-option>

src/presets/instagram.xml

@@ -17,6 +17,6 @@
 	 med="256 kb/s"
 	 high="320 kb/s"></audiobitrate>
 	<samplerate>48000</samplerate>
-	<projectprofile>VGA NTSC</projectprofile>
-	<projectprofile>VGA Widescreen NTSC</projectprofile>
+	<projectprofile>NTSC SD SQ VGA 480p 29.97 fps</projectprofile>
+	<projectprofile>NTSC SD Wide FWVGA 480p 29.97 fps</projectprofile>
 </export-option>

src/presets/metacafe.xml

@@ -22,9 +22,9 @@
 	<projectprofile>HD 720p 25 fps</projectprofile>
 	<projectprofile>HD 720p 29.97 fps</projectprofile>
 	<projectprofile>HD 720p 30 fps</projectprofile>
-	<projectprofile>HD 1080p 23.98 fps</projectprofile>
-	<projectprofile>HD 1080p 24 fps</projectprofile>
-	<projectprofile>HD 1080p 25 fps</projectprofile>
-	<projectprofile>HD 1080p 29.97 fps</projectprofile>
-	<projectprofile>HD 1080p 30 fps</projectprofile>
+	<projectprofile>FHD 1080p 23.98 fps</projectprofile>
+	<projectprofile>FHD 1080p 24 fps</projectprofile>
+	<projectprofile>FHD PAL 1080p 25 fps</projectprofile>
+	<projectprofile>FHD 1080p 29.97 fps</projectprofile>
+	<projectprofile>FHD 1080p 30 fps</projectprofile>
 </export-option>

src/presets/nokia_nHD.xml

@@ -17,5 +17,5 @@
 	 med="128 kb/s"
 	 high="192 kb/s"></audiobitrate>
 	<samplerate>48000</samplerate>
-	<projectprofile>QVGA 29.97 fps</projectprofile>
+	<projectprofile>NTSC SD 1/4 QVGA 240p 29.97 fps</projectprofile>
 </export-option>

src/presets/picasa.xml

@@ -17,7 +17,7 @@
 	 med="256 kb/s"
 	 high="320 kb/s"></audiobitrate>
 	<samplerate>48000</samplerate>
-	<projectprofile>DV/DVD Widescreen NTSC</projectprofile>
+	<projectprofile>NTSC SD Widescreen Anamorphic 480i 29.97 fps</projectprofile>
 	<projectprofile>HD 720p 29.97 fps</projectprofile>
-	<projectprofile>HD 1080p 29.97 fps</projectprofile>
+	<projectprofile>FHD 1080p 29.97 fps</projectprofile>
 </export-option>

src/presets/twitter.xml

@@ -17,14 +17,14 @@
 	 med="256 kb/s"
 	 high="320 kb/s"></audiobitrate>
 	<samplerate>48000</samplerate>
-	<projectprofile>VGA NTSC</projectprofile>
-	<projectprofile>VGA Widescreen NTSC</projectprofile>
+	<projectprofile>NTSC SD SQ VGA 480p 29.97 fps</projectprofile>
+	<projectprofile>NTSC SD Wide FWVGA 480p 29.97 fps</projectprofile>
 	<projectprofile>HD 720p 23.98 fps</projectprofile>
 	<projectprofile>HD 720p 24 fps</projectprofile>
 	<projectprofile>HD 720p 25 fps</projectprofile>
 	<projectprofile>HD 720p 29.97 fps</projectprofile>
 	<projectprofile>HD 720p 30 fps</projectprofile>
-	<projectprofile>HD 720p 50 fps</projectprofile>
+	<projectprofile>PAL HD 720p 50 fps</projectprofile>
 	<projectprofile>HD 720p 59.94 fps</projectprofile>
 	<projectprofile>HD 720p 60 fps</projectprofile>
 	<projectprofile>HD Vertical 720p 23.98 fps</projectprofile>

src/presets/vimeo.xml

@@ -17,12 +17,12 @@
 	 med="256 kb/s"
 	 high="320 kb/s"></audiobitrate>
 	<samplerate>48000</samplerate>
-	<projectprofile>2.5K QHD 1440p 23.98 fps</projectprofile>
-	<projectprofile>2.5K QHD 1440p 24 fps</projectprofile>
-	<projectprofile>2.5K QHD 1440p 25 fps</projectprofile>
-	<projectprofile>2.5K QHD 1440p 29.97 fps</projectprofile>
-	<projectprofile>2.5K QHD 1440p 30 fps</projectprofile>
-	<projectprofile>2.5K QHD 1440p 50 fps</projectprofile>
-	<projectprofile>2.5K QHD 1440p 59.94 fps</projectprofile>
-	<projectprofile>2.5K QHD 1440p 60 fps</projectprofile>
+	<projectprofile>2.5K WQHD 1440p 23.98 fps</projectprofile>
+	<projectprofile>2.5K WQHD 1440p 24 fps</projectprofile>
+	<projectprofile>2.5K WQHD 1440p 25 fps</projectprofile>
+	<projectprofile>2.5K WQHD 1440p 29.97 fps</projectprofile>
+	<projectprofile>2.5K WQHD 1440p 30 fps</projectprofile>
+	<projectprofile>2.5K WQHD 1440p 50 fps</projectprofile>
+	<projectprofile>2.5K WQHD 1440p 59.94 fps</projectprofile>
+	<projectprofile>2.5K WQHD 1440p 60 fps</projectprofile>
 </export-option>