feat: Rebrand to Proxima

README.md

@@ -1,16 +1,24 @@
-# Resolve Proxy Encoder
+# Proxima :sparkles:
+#### (Previously *Resolve Proxy Encoder*)
 
-![GitHub](https://img.shields.io/github/license/in03/Resolve-Proxy-Encoder) 
+![GitHub](https://img.shields.io/github/license/in03/proxima) 
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
-![GitHub branch checks state](https://img.shields.io/github/checks-status/in03/Resolve-Proxy-Encoder/main)
-[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/in03/Resolve-Proxy-Encoder/main.svg)](https://results.pre-commit.ci/latest/github/in03/Resolve-Proxy-Encoder/main)
+![GitHub branch checks state](https://img.shields.io/github/checks-status/in03/proxima/main)
+[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/in03/proxima/main.svg)](https://results.pre-commit.ci/latest/github/in03/proxima/main)
 
-![GitHub last commit](https://img.shields.io/github/last-commit/in03/Resolve-Proxy-Encoder)
-![GitHub Repo stars](https://img.shields.io/github/stars/in03/Resolve-Proxy-Encoder?style=social)
+![GitHub last commit](https://img.shields.io/github/last-commit/in03/proxima)
+![GitHub Repo stars](https://img.shields.io/github/stars/in03/proxima?style=social)
 
-##### Resolve Proxy Encoder makes queuing proxies from DaVinci Resolve a breeze. Launch the worker on as many computers as you have free and it'll pool all the free CPU threads together to encode multiple proxies at once. Only have the one computer? Encoding runs entirely on the CPU, leaving GPU-heavy Resolve with plenty of resources to continue editing while you pump out proxies. Once they're finished, they're automatically linked.
 
-![](https://github.com/in03/Resolve-Proxy-Encoder/blob/main/docs/images/rprox_worker-min.gif)
+##### Proxima makes queuing proxies from DaVinci Resolve a breeze. Launch the worker on as many computers as you have free and it'll pool all the free CPU threads together to encode multiple proxies at once. Only have the one computer? Encoding runs entirely on the CPU, leaving GPU-heavy Resolve with plenty of resources to continue editing while you pump out proxies. Once they're finished, they're automatically linked.
+
+> **Warning**
+>
+> **Broker support has changed!**
+> Since introducing queuer-side progress indicators, only Redis is supported as a broker. Any other brokers officially supported by Celery will no longer work. 
+> Passing progress data back to the queuer through Celery proved to be a little obtuse. Choosing and working with a single broker was the path forward.
+
+![](https://github.com/in03/proxima/blob/main/docs/images/rprox_worker-min.gif)
 
 ## Why make proxies? ##
 DaVinci Resolve's greatly benefits from having all-intra media, like ProRes or DNxHD. If you shoot in h.264 or h.265 like many do, you're likely to see great performance improvements using proxies. This application makes queuing, encoding and linking them quick and easy.
@@ -25,14 +33,21 @@ DaVinci Resolve's greatly benefits from having all-intra media, like ProRes or D
 - [x] Easy YAML based user configuration
 - [x] Advanced configuration validation
 - [x] Automatically checks for updates
-
-## Soon to come! ##
-- [ ] Resolve 18 support
-- [ ] Mac M1 support
+- [x] Control Celery with Celery commands from CLI, even when installed with pipx
+- [x] Multi progress-bar [#190](https://github.com/in03/proxima/issues/190)
+
+## Roadmap ##
+- [ ] Better Resolve 18 integration
+- [ ] Better Apple silicon support
+- [ ] Faster config parsing and validation with TOML [#207](https://github.com/in03/proxima/issues/207)
+- [ ] Interactive timeline selection [#161](https://github.com/in03/proxima/issues/161)
 - [ ] Better resource-utilization with encode-chunking
-- [ ] Better queuer-side monitoring - multi progress-bar
+- [ ] Multiple, customizable proxy-presets with rule-based matching [#201](https://github.com/in03/proxima/issues/201)
 - [ ] Easier cross-platform paths via path-mapping
+- [ ] Manage configuration from CLI
+- [ ] Pre-flight checks - encoding settings validation 
 - [ ] UI improvements
+- [ ] Monitor Web App
 
 ## What about Blackmagic's Proxy Generator? ##
 I started this for the company I work for, well before BPG was on the scene. If BPG works better for you, go for it! As it stands BPG won't do any all-intra codecs on Windows, which is a dealbreaker for us. It also works on a watch-folder basis with no filename whitelisting or path-filtering rules. That means EVERY video file becomes a proxy, whether you need it or not. Our workflow often sees the shooter doing a rough assembly of chosen takes as an exported timeline. We simply import this timeline and queue proxies from it. If you work with chronic-overshooters, you'll save a heap of disk space and encoding time queuing proxies from a roughly-organised timeline. 
@@ -52,38 +67,41 @@ I started this for the company I work for, well before BPG was on the scene. If
 > 
 > This Python version is end-of-life. No bug-fixes or security-patches are being released anymore. 
 > As such, many popular Python packages we depend on are dropping support for Python 3.6. 
-> Once Resolve 18 is out of public beta, there will be a final release for Resolve 17 and future development will be in a higher python version.
-> Until then, development must continue in 3.6.
+> Resolve 18 is now out of public beta. Once Proxima's integration and testing are complete for compatability with 18,
+> there will be a final release for Resolve 17 and future development will continue in a later version of Python 3.
+> Until then the full, working feature-set is available on the main branch.
 > To mitigate dependency conflicts you can try:
 >
-> - Calling Resolve Proxy Encoder from a Python 3.6 virtual environment.
-> - Install Python 3.6 for Resolve Proxy Encoder and install a newer Python alongside it for your other needs.
+> - Calling Proxima from a Python 3.6 virtual environment.
+> - Install Python 3.6 for Proxima and install a newer Python alongside it for your other needs.
 > - Install a tool like *pipx* that isolates Python CLI tools with their own virtual environments but keeps them on path (recommended)
 
 ### Installation
-Resolve Proxy Encoder is composed of three major parts:
+Proxima is composed of three major parts:
 
 - the 'queuer' responsible for interfacing with DaVinci Resolve and sending tasks to the broker
-- the 'broker' (Redis or RabbitMQ) that distributes jobs to the workers
+- the 'broker' (Redis) that distributes jobs to the workers
 - the 'worker' one of potentially many workers responsible for doing the actual encoding
 
 #### CLI
 The 'queuer' and 'worker' are bundled together in the CLI app. They are both installed from the same source, called from the same command and share the same configuration file. As such, any computer that has the CLI app installed can both queue proxies (so long as Resolve is set-up) and run workers. Install it with pipx:
 
 ``` 
-pipx install git+https://github.com/in03/Resolve-Proxy-Encoder
+pipx install git+https://github.com/in03/proxima
 ```
 
 #### Broker
 The broker is best installed on an always-on computer or server. If it's not running neither queuers nor workers can communicate.
-Very little configuration is required. Just make sure it's accessible over LAN. Redis is best, but RabbitMQ is also supported.
-Install it with docker:
+Very little configuration is required. Just make sure it's accessible over LAN. Since [#190](https://github.com/in03/proxima/pull/190), brokers other than Redis are no longer supported.
+
+Install Redis with docker:
 ```
 docker run -d --name some-redis -p 6379:6379 redis-server --append-only yes
 ```
 
 #### Monitor
-If you want to monitor your jobs, it's a good idea to install Flower.
+An in-house web-app monitor for Proxima jobs is coming and will include some really helpful, specific features.
+Until it's ready, if you want to monitor your jobs, it's a good idea to install Flower - an official monitor for Celery tasks and workers.
 Install it with docker alongside your broker:
 ```
 docker run --name flower -e $CELERY_BROKER_URL=redis://192.168.1.171:6379/0 -e FLOWER_PURGE_OFFLINE_WORKERS=300 -d flower
@@ -95,7 +113,7 @@ Consider setting `FLOWER_PURGE_OFFLINE_WORKERS` if you don't have a well-defined
 ## How do I use it?
 
 ```
-Usage: rprox [OPTIONS] COMMAND [ARGS]...
+Usage: proxima [OPTIONS] COMMAND [ARGS]...
 
 Options:
   --install-completion  Install completion for the current shell.
@@ -110,12 +128,13 @@ Commands:
   purge  Purge all proxy jobs from all queues
   queue  Queue proxies from the currently open DaVinci Resolve timeline
   work   Prompt to start Celery workers on local machine
+  celery Inject Celery commands to control tasks and workers
   ```
 
 ## Configuration
 On first run, you'll be prompted to alter your settings. The app will copy the default settings to the OS user configuration folder. 
-- **Linux/Mac:** `$XDG_HOME_CONFIG/resolve_proxy_encoder/user_settings.yml` (may not open settings automatically)
-- **Windows:** `%homepath%/resolve_proxy_encoder/user_settings.yml`
+- **Linux/Mac:** `$XDG_HOME_CONFIG/proxima/user_settings.yml` (may not open settings automatically)
+- **Windows:** `%homepath%/proxima/user_settings.yml`
 
 
 ### Some Key Settings
@@ -132,35 +151,31 @@ Celery runs all queuer/worker/broker communications.
 Make sure you set all of the below addresses as per your environment!
 
 ```
-celery:
-  host_address: 192.168.1.171
-  broker_url: redis://192.168.1.171:6379/0
-  flower_url: http://192.168.1.171:5555
-  result_backend: redis://192.168.1.171:6379/0
-  result_expires: 60 # 10 mins
+broker:
+  url:  redis://192.168.1.19:6379/0
+  job_expires: 3600 # 1 hour (cleared if not received by worker)
+  result_expires: 86400 # 1 day (Needed for webapp monitor)
 ```
 
 > **Warning**
 > 
-> Make sure you set `result_expires!` to a reasonable value otherwise your broker may run out of memory!
-> If you need persistent results, consider configuring your broker for persistent storage. 
-> Both Redis and RabbitMQ have options for persistence, though they come with some trade-offs. Consider your needs carefully..
+> Make sure you set `result_expires!` to a reasonable value otherwise Redis may run out of memory!
+> If you need persistent results, consider configuring Redis for persistence.
 
-#### `concurrency`
-Windows doesn't support preforking for concurrency. Actually, Celery doesn't officially support Windows anymore at all. Running workers on Mac, Linux or containerised gets around this limitation. By default the configuration encourages starting multiple workers processes as 'solo' to work with Windows. Change this as necessary to reduce overhead:
+#### `worker`
 ```
 worker:
-  concurrency: 1
+  loglevel: INFO
+  terminal_args: [] # use alternate shell? Recommend windows terminal ("wt") on Windows.
+  celery_args: [-l, INFO, -P, solo, --without-mingle, --without-gossip]
 ```
+Queuer and worker have separate loglevels to make debugging a little easier if you've got just one worker playing up.
+
+### A Note on Concurrency
+Some pretty dangerous concurrency settings were moved out of the configuration settings to prevent accidents. Windows doesn't support pre-forking for concurrency. Actually, Celery doesn't officially support Windows anymore at all. Mac and Linux can pre-fork, but the worker launcher invoked with the `proxima work` command gets around this by launching separate processes with workers named `worker1@host`, `worker2@host`, etc. We're going for the lowest common denominator here. This works fine with Mac and Linux too. It makes monitoring easier and cross-platform behaviour more consistent. For those concerned about the extra overhead who would like to squeeze out every last bit of performance on Mac or Linux, consider injecting the `celery multi` command with `proxima celery`. See [Celery daemonization](https://docs.celeryq.dev/en/master/userguide/daemonizing.html?#init-script-celeryd) 
 
-#### `prefetch_multiplier`
-Encoding is a long-running task. Tasks won't get divided nicely between workers if they fetch more than one task at a time:
-```
-worker:
-  prefetch_multiplier: 1 
-```
 
 
 ## How can I contribute?
-Hey! Thanks! Any help is appreciated. Please check the [Contribution Guide](https://github.com/in03/Resolve-Proxy-Encoder/wiki/Contribution-Guide).
+Hey! Thanks! Any help is appreciated. Please check the [Contribution Guide](https://github.com/in03/proxima/wiki/Contribution-Guide).
 

mkdocs.yml

@@ -1,3 +1,3 @@
-site_name: Resolve Proxy Encoder
+site_name: Proxima
 theme:
   name: material

resolve_proxy_encoder/app/broker.py → proxima/app/broker.py

@@ -15,7 +15,7 @@
 from rich.live import Live
 from cryptohash import sha1
 
-from resolve_proxy_encoder.settings.manager import SettingsManager
+from proxima.settings.manager import SettingsManager
 
 
 class RedisConnection:

resolve_proxy_encoder/app/checks.py → proxima/app/checks.py

@@ -82,7 +82,7 @@ def check_for_updates(github_url: str, package_name: str) -> Union[dict, None]:
         logger.warning(
             "[yellow]Update available.\n"
             + "Fully uninstall and reinstall when possible:[/]\n"
-            + '"pip uninstall resolve-proxy-encoder"\n'
+            + '"pip uninstall resolve-proxima"\n'
             + f'"pip install git+{github_url}"\n'
         )
 

resolve_proxy_encoder/app/cli.py → proxima/app/cli.py

@@ -33,15 +33,20 @@
 
 
 @cli_app.callback(invoke_without_command=True)
+def run_without_args():
+    draw_banner()
+    print("Run [bold]proxima --help[/] for a list of commands")
+
+
 def draw_banner():
 
     # Print CLI title
-    fig = Figlet()
-    text = fig.renderText("Resolve Proxy Encoder")
-    print(f"[green]{text}\n")
+    fig = Figlet(font="rectangles")
+    text = fig.renderText("proxima")
+    print(text + "\n")
 
     # Get build info
-    build_info = pkg_info.get_build_info("Resolve-Proxy-Encoder")
+    build_info = pkg_info.get_build_info("proxima")
 
     # Print banner data
     if build_info["build"] == "release":
@@ -75,7 +80,7 @@ def run_checks():
     # Check for any updates and inject version info into user settings.
     version_info = checks.check_for_updates(
         github_url=settings["app"]["update_check_url"],
-        package_name="resolve_proxy_encoder",
+        package_name="proxima",
     )
 
     settings.update({"version_info": version_info})
@@ -200,23 +205,34 @@ def purge():
     console.rule(f"[red bold]Purge all tasks! :fire:", align="left")
     print("\n")
 
-    subprocess.run(
-        ["celery", "-A", "resolve_proxy_encoder.worker", "purge", "-Q", VC_KEY]
-    )
+    subprocess.run(["celery", "-A", "proxima.worker", "purge", "-Q", VC_KEY])
 
 
-# TODO: Would be great if we can pass options unparsed by Typer
-# This command could serve as a gateway to all Celery commands,
-# but typer parses 'f' in 'broker purge -f' as an undefined option
-@cli_app.command()
-def celery(command: List[str]):
-    """Pass celery commands to Celery buried within venv"""
+@cli_app.command(
+    context_settings={"allow_extra_args": True, "ignore_unknown_options": True}
+)
+def celery(
+    ctx: typer.Context,
+    celery_command: List[str] = typer.Argument(..., help="A command to pass to Celery"),
+):
+    """
+    Pass commands to Celery buried in venv.
+
+    Runs `celery -A proxima.worker [celery_command]`
+    at the absolute location of the package's Celery executable.
+    Useful when the celery project is buried in a virtual environment and you want
+    to do something a little more custom like purge jobs from a custom queue name.
+
+    See https://docs.celeryq.dev/en/latest/reference/cli.html for proper usage.
+    """
+
+    print(ctx.params["celery_command"])
 
     print("\n")
     console.rule(f"[cyan bold]Celery command :memo:", align="left")
     print("\n")
 
-    subprocess.run(["celery", "-A", "resolve_proxy_encoder.worker", command])
+    subprocess.run(["celery", "-A", "proxima.worker", *celery_command])
 
 
 @cli_app.command()

resolve_proxy_encoder/app/utils/core.py → proxima/app/utils/core.py

@@ -72,7 +72,7 @@ def app_exit(level: int = 0, timeout: int = 0, cleanup_funcs: list = []):
     sys.exit(level)
 
 
-def notify(message: str, title: str = "Resolve Proxy Encoder"):
+def notify(message: str, title: str = "Proxima"):
     """Cross platform system notification
 
     Args:

resolve_proxy_encoder/app/utils/pkg_info.py → proxima/app/utils/pkg_info.py

@@ -67,7 +67,7 @@ def get_build_info(package_name: str) -> dict:
     except subprocess.CalledProcessError as e:
         pass
 
-    release_version = pkg_resources.get_distribution("Resolve-Proxy-Encoder").version
+    release_version = pkg_resources.get_distribution("proxima").version
     return {
         "build": "release",
         "installed": True,

resolve_proxy_encoder/settings/default_settings.yml → proxima/settings/default_settings.yml

@@ -1,4 +1,4 @@
----  # Resolve Proxy Encoder User configuration file
+---  # Proxima User configuration file
 
 # NOTE: 
 # Proxy paths, codecs, resolution, filters, etc are all inherited from the queuer's user configuration.
@@ -7,7 +7,7 @@
 app:
   loglevel: WARNING
   check_for_updates: true
-  update_check_url: "https://github.com/in03/resolve-proxy-encoder" # If you fork the repo, change this to your fork
+  update_check_url: "https://github.com/in03/proxima" # If you fork the repo, change this to your fork
   disable_version_constrain: false # DANGEROUS! Allows any version of worker to take jobs. Must be set on queuer and worker. 
 
 paths:

resolve_proxy_encoder/settings/manager.py → proxima/settings/manager.py

@@ -31,7 +31,7 @@
 USER_SETTINGS_FILE = os.path.join(
     Path.home(),
     ".config",
-    "resolve_proxy_encoder",
+    "proxima",
     "user_settings.yml",
 )
 

resolve_proxy_encoder/worker/celery.py → proxima/worker/celery.py

@@ -20,7 +20,7 @@
 
 app.autodiscover_tasks(
     [
-        "resolve_proxy_encoder.worker.tasks.encode.tasks.encode_proxy",
+        "proxima.worker.tasks.encode.tasks.encode_proxy",
     ]
 )
 

resolve_proxy_encoder/worker/launch_workers.py → proxima/worker/launch_workers.py

@@ -154,7 +154,7 @@ def get_new_console():
         get_new_console(),
         *settings["worker"]["terminal_args"],
         f'"{get_celery_binary_path()}"',
-        "-A resolve_proxy_encoder.worker",
+        "-A proxima.worker",
         "worker",
         get_worker_name(id),
         get_worker_queue(),

pyproject.toml

@@ -1,5 +1,5 @@
 [tool.poetry]
-name = "resolve-proxy-encoder"
+name = "proxima"
 version = "0.1.0"
 description = "Transcode source media in DaVinci Resolve using multiple machines. Great for quickly creating proxies without interrupting work."
 authors = ["Caleb Trevatt <[email protected]>"]
@@ -37,7 +37,7 @@ commitizen = "^2.27.1"
 
 [tool.semantic_release]
 version_variable = [
-    "resolve_proxy_encoder/__init__.py:__version__",
+    "proxima/__init__.py:__version__",
     "pyproject.toml:version"
 ]
 branch = "main"
@@ -51,4 +51,4 @@ requires = ["poetry-core>=1.0.0"]
 build-backend = "poetry.core.masonry.api"
 
 [tool.poetry.scripts]
-rprox = "resolve_proxy_encoder.app.cli:main"
+proxima = "proxima.app.cli:main"

version_constraint_key

@@ -1 +1 @@
-slightly-sharp-shark
+constantly-cool-corgi