Skip to content

Commit 9f5aa99

Browse files
in03in03
authored
docs: update README
 - Added badges - Added basic UI demo GIF - Added warning callouts - Added contribution guide link - Addressed Blackmagic Proxy Generator - Added sections detailing existing features and features to come - Restyled some sections - Updated old configuration references
1 parent 16a1d35 commit 9f5aa99

File tree

1 file changed

+123
-113
lines changed

1 file changed

+123
-113
lines changed

‎README.md

+123-113
Original file line numberDiff line numberDiff line change
@@ -1,156 +1,166 @@
11
# Resolve Proxy Encoder
2-
[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/in03/Resolve-Proxy-Encoder/main.svg)](https://results.pre-commit.ci/latest/github/in03/Resolve-Proxy-Encoder/main) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
32

4-
## Work in progress! ⚠️
3+
![GitHub](https://img.shields.io/github/license/in03/Resolve-Proxy-Encoder)
4+
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
5+
![GitHub branch checks state](https://img.shields.io/github/checks-status/in03/Resolve-Proxy-Encoder/main)
6+
[![pre-commit.ci status](https://results.pre-commit.ci/badge/github/in03/Resolve-Proxy-Encoder/main.svg)](https://results.pre-commit.ci/latest/github/in03/Resolve-Proxy-Encoder/main)
57

6-
Please excuse anything that doesn't make sense in this repository. I code when I can find the time (which is scarce!) and often I'll leave quick-fixes and band-aid- solutions sitting pretty. I'm doing my best to separate configuration from code and keep things cross platform, but at times I bend the rules to keep this working everyday in our specific office environment. If you have the time to contribute to this repo, please do! Forks, PRs, issues and suggestions welcome.
8+
![GitHub last commit](https://img.shields.io/github/last-commit/in03/Resolve-Proxy-Encoder)
9+
![GitHub Repo stars](https://img.shields.io/github/stars/in03/Resolve-Proxy-Encoder?style=social)
710

8-
---
9-
10-
## What's it for? ##
11-
DaVinci Resolve Studio has a fantastic remote-rendering system built-in that allows queuing renders on other networked Resolve computers.
12-
Unfortunately Resolve doesn't have a remote-rendering, or even background-rendering solution for proxies.
13-
*Resolve Proxy Encoder* is a Python application that can be used in a similar way to Resolve's remote-rendering system, but for proxies.
14-
15-
## How does it work? ##
16-
**Resolve Proxy Encoder works using three major parts:**
17-
- the Celery app that manages and runs the workers
18-
- the `resolve_queue_proxies` module responsible for interfacing with DaVinci Resolve's python API
19-
- the broker (Redis or RabbitMQ server) that's responsible for sending data between Celery instances
20-
21-
## What does it need?
22-
**This app has a few non-negotiable prerequisites:**
23-
- Python 3.6 **ONLY** (DaVinci Resolve's Python API requires it)
24-
- DaVinci Resolve Studio, with scripting set up (read Resolve's scripting README)
25-
- Redis or RabbitMQ broker (preferably running on a NAS or server)
26-
- Worker computers (decent resources, connected to LAN, can access source media and proxies output folder)
27-
- *Resolve Proxy Encoder* installed on every 'queuer' and 'worker'. Not necessary to run broker.
28-
29-
**It also makes a few assumptions about the way you have your environment set up:**
30-
- Running Windows (Mac OS, Linux and BSD untested - though should be easy to make work)
31-
- All source media is accessible from a single networked volume / mapped-drive
32-
- All proxies are to be encoded to and accessible from a single networked volume / mapped-drive
33-
34-
35-
## How do I install it?
36-
37-
### A Warning about Python 3.6
38-
39-
Because DaVinci Resolve requires Python 3.6 to communicate with its API, no versions over Python 3.6 will work with *Resolve Proxy Encoder*.
40-
Unfortunately this means that *Resolve Proxy Encoder* may get stuck using older versions of certain packages as they begin to drop support for 3.6.
41-
It also means that security patches for some dependencies won't make it into *Resolve Proxy Encoder*
42-
This kind of setup almost guarantees dependency conflicts if you have multiple Python CLI tools you keep installed.
43-
To mitigate this you can:
44-
45-
- Use Python 3.6 for *Resolve Proxy Encoder* **ONLY** and install a newer Python alongside for your other needs.
46-
47-
- Install a tool like *pipx* that isolates Python CLI tools with their own virtual environments but keeps them on path.
11+
##### Resolve Proxy Encoder makes queuing proxies from DaVinci Resolve a breeze. Launch the worker on as many computers as you have free and it'll pool all the free CPU threads together to encode multiple proxies at once. Only have the one computer? Encoding runs entirely on the CPU, leaving GPU-heavy Resolve with plenty of resources to continue editing while you pump out proxies. Once they're finished, they're automatically linked.
4812

13+
![](https://github.com/in03/Resolve-Proxy-Encoder/blob/main/docs/images/rprox_worker-min.gif)
14+
15+
## Why make proxies? ##
16+
DaVinci Resolve's greatly benefits from having all-intra media, like ProRes or DNxHD. If you shoot in h.264 or h.265 like many do, you're likely to see great performance improvements using proxies. This application makes queuing, encoding and linking them quick and easy.
17+
18+
## Features ##
19+
- [x] Distributed encoding on multiple machines
20+
- [x] Utilizes all CPU threads for encoding
21+
- [x] Queue only used proxies straight from a DaVinci Resolve timeline
22+
- [x] Automatically link proxies after encoding
23+
- [x] Flexible queuer/worker arrangement allows background encoding on same machine
24+
- [x] Version constrain prevents incompatabilities between different machines
25+
- [x] Easy YAML based user configuration
26+
- [x] Advanced configuration validation
27+
- [x] Automatically checks for updates
28+
29+
## Soon to come! ##
30+
- [ ] Resolve 18 support
31+
- [ ] Mac M1 support
32+
- [ ] Better resource-utilization with encode-chunking
33+
- [ ] Better queuer-side monitoring - multi progress-bar
34+
- [ ] Easier cross-platform paths via path-mapping
35+
- [ ] UI improvements
36+
37+
## What about Blackmagic's Proxy Generator? ##
38+
I started this for the company I work for, well before BPG was on the scene. If BPG works better for you, go for it! As it stands BPG won't do any all-intra codecs on Windows, which is a dealbreaker for us. It also works on a watch-folder basis with no filename whitelisting or path-filtering rules. That means EVERY video file becomes a proxy, whether you need it or not. Our workflow often sees the shooter doing a rough assembly of chosen takes as an exported timeline. We simply import this timeline and queue proxies from it. If you work with chronic-overshooters, you'll save a heap of disk space and encoding time queuing proxies from a roughly-organised timeline.
39+
40+
41+
## Ooh, I want it!
42+
43+
### Prerequisites
44+
- One or more editing computers with DaVinci Resolve Studio installed and scripting enabled
45+
- An always on computer to run the broker (e.g. server, NAS, primary-desktop, Raspberry Pi)
46+
- Worker computers (decent resources, can overlap with editing computers)
47+
- all above machines on LAN and able to access same files via same filepath.
48+
49+
> **Warning**
50+
>
51+
> **DaVinci Resolve 17 requires Python 3.6.**
52+
>
53+
> This Python version is end-of-life. No bug-fixes or security-patches are being released anymore.
54+
> As such, many popular Python packages we depend on are dropping support for Python 3.6.
55+
> Once Resolve 18 is out of public beta, there will be a final release for Resolve 17 and future development will be in a higher python version.
56+
> Until then, development must continue in 3.6.
57+
> To mitigate dependency conflicts you can try:
58+
>
59+
> - Calling Resolve Proxy Encoder from a Python 3.6 virtual environment.
60+
> - Install Python 3.6 for Resolve Proxy Encoder and install a newer Python alongside it for your other needs.
61+
> - Install a tool like *pipx* that isolates Python CLI tools with their own virtual environments but keeps them on path (recommended)
62+
63+
### Installation
64+
Resolve Proxy Encoder is composed of three major parts:
65+
66+
- the 'queuer' responsible for interfacing with DaVinci Resolve and sending tasks to the broker
67+
- the 'broker' (Redis or RabbitMQ) that distributes jobs to the workers
68+
- the 'worker' one of potentially many workers responsible for doing the actual encoding
69+
70+
#### CLI
71+
The 'queuer' and 'worker' are bundled together in the CLI app. They are both installed from the same source, called from the same command and share the same configuration file. As such, any computer that has the CLI app installed can both queue proxies (so long as Resolve is set-up) and run workers. Install it with pipx:
4972

50-
### CLI / Worker
51-
The CLI app is bundled with everything necessary to queue from Resolve and start workers that run the encoding.
52-
```
53-
py -3.6 -m pip install git+https://github.com/in03/Resolve-Proxy-Encoder
54-
```
55-
Or with *pipx*
5673
```
5774
pipx install git+https://github.com/in03/Resolve-Proxy-Encoder
5875
```
59-
### Broker
60-
The broker needs to be accessible by each computer over LAN.
61-
Install Redis on an always-on computer or server (nice and easy to run in a docker container):
76+
77+
#### Broker
78+
The broker is best installed on an always-on computer or server. If it's not running neither queuers nor workers can communicate.
79+
Very little configuration is required. Just make sure it's accessible over LAN. Redis is best, but RabbitMQ is also supported.
80+
Install it with docker:
6281
```
6382
docker run -d --name some-redis -p 6379:6379 redis-server --append-only yes
6483
```
65-
OR RabbitMQ:
66-
```
67-
docker run -d --hostname my-rabbit --name some-rabbit -p 5672:5672 -p 8080:15672 rabbitmq:3-management
68-
```
6984

70-
### Monitor / Notifications
85+
#### Monitor
7186
If you want to monitor your jobs, it's a good idea to install Flower.
87+
Install it with docker alongside your broker:
88+
```
89+
docker run --name flower -e $CELERY_BROKER_URL=redis://192.168.1.171:6379/0 -e FLOWER_PURGE_OFFLINE_WORKERS=300 -d flower
7290
```
73-
docker run --name flower -e $POSTGRES_PASSWORD -e $POSTGRES_USER -d flower
91+
Set the IP address and port to the same as your broker container. Don't forget the Redis protocol `redis://`.
92+
Consider setting `FLOWER_PURGE_OFFLINE_WORKERS` if you don't have a well-defined set of computers to run the workers, or there's potential to change worker hostnames. `300` for example, would clear the list of offline workers every 5 minutes. This just keeps things tidy.
93+
94+
95+
## How do I use it?
96+
7497
```
75-
Make sure you set the environment variables for `CELERY_BROKER_URL` or Flower won't be able to connect to the broker. Also consider setting `FLOWER_PURGE_OFFLINE_WORKERS` if you don't have a well-defined set of computers to run the workers, or there's potential to change hostnames. `300` for example, would clear the list of offline workers every 5 minutes. This just keeps things tidy.
98+
Usage: rprox [OPTIONS] COMMAND [ARGS]...
7699
77-
If you opt not to run Flower, keep in mind some non-essential CLI commands may not work.
100+
Options:
101+
--install-completion Install completion for the current shell.
102+
--show-completion Show completion for the current shell, to copy it or
103+
customize the installation.
104+
105+
--help Show this message and exit.
106+
107+
Commands:
108+
link Manually link proxies from directory to source media in open...
109+
mon Launch Flower Celery monitor in default browser new window
110+
purge Purge all proxy jobs from all queues
111+
queue Queue proxies from the currently open DaVinci Resolve timeline
112+
work Prompt to start Celery workers on local machine
113+
```
78114

79115
## Configuration
80116
On first run, you'll be prompted to alter your settings. The app will copy the default settings to the OS user configuration folder.
81-
- **Linux/Mac:** `$XDG_HOME_CONFIG/resolve_proxy_encoder/user_settings.yml`
117+
- **Linux/Mac:** `$XDG_HOME_CONFIG/resolve_proxy_encoder/user_settings.yml` (may not open settings automatically)
82118
- **Windows:** `%homepath%/resolve_proxy_encoder/user_settings.yml`
83119

84120

85-
86121
### Some Key Settings
87122

88-
The default global log-level:
89-
```
90-
loglevel: WARNING
91-
```
92-
123+
#### `proxy_path_root`
93124
All proxies will be encoded to this directory. They'll retain the source media's directory structure:
94125
```
95126
paths:
96127
proxy_path_root: R:/ProxyMedia
97128
```
98-
These are celery specific settings. Make sure you set all of the below addresses as per your environment!
99-
```
100-
celery_settings:
101129

102-
host_address: 192.168.1.19
103-
broker_url: redis://192.168.1.19:6379/0
104-
flower_url: http://192.168.1.19:5555
105-
result_backend: redis://192.168.1.19:6379/0
106-
```
130+
#### `celery`
131+
Celery runs all queuer/worker/broker communications.
132+
Make sure you set all of the below addresses as per your environment!
107133

108-
Set this to a reasonable value in minutes so that the broker doesn't fill with results and run out of memory!
109-
If you need persistent results, consider configuring your broker for persistent storage. Both Redis and RabbitMQ have options for persistence, though they come with some trade-offs. Consider your needs carefully.
110134
```
111-
celery_settings:
112-
result_expires: 60
135+
celery:
136+
host_address: 192.168.1.171
137+
broker_url: redis://192.168.1.171:6379/0
138+
flower_url: http://192.168.1.171:5555
139+
result_backend: redis://192.168.1.171:6379/0
140+
result_expires: 60 # 10 mins
113141
```
142+
143+
> **Warning**
144+
>
145+
> Make sure you set `result_expires!` to a reasonable value otherwise your broker may run out of memory!
146+
> If you need persistent results, consider configuring your broker for persistent storage.
147+
> Both Redis and RabbitMQ have options for persistence, though they come with some trade-offs. Consider your needs carefully..
148+
149+
#### `concurrency`
114150
Windows doesn't support preforking for concurrency. Actually, Celery doesn't officially support Windows anymore at all. Running workers on Mac, Linux or containerised gets around this limitation. By default the configuration encourages starting multiple workers processes as 'solo' to work with Windows. Change this as necessary to reduce overhead:
115151
```
116-
celery_settings:
117-
worker_concurrency: 1
118-
```
119-
120-
Encoding is a long-running task. Tasks won't get divided nicely between workers if they fetch more than one task at a time:
121-
```
122-
celery_settings:
123-
worker_prefetch_multiplier: 1
152+
worker:
153+
concurrency: 1
124154
```
125155

126-
## How can I contribute?
127-
Clone the repo, install dependencies, call from poetry shell:
156+
#### `prefetch_multiplier`
157+
Encoding is a long-running task. Tasks won't get divided nicely between workers if they fetch more than one task at a time:
128158
```
129-
git clone https://github.com/in03/Resolve-Proxy-Encoder
130-
cd Resolve-Proxy-Encoder
131-
py -3.6 -m pip install poetry
132-
py -3.6 -m poetry shell
133-
poetry install
134-
rprox
159+
worker:
160+
prefetch_multiplier: 1
135161
```
136-
If you're unfamiliar with using Poetry for dependency management and packaging, [give it a look](https://python-poetry.org/docs/basic-usage).
137162

138-
## How do I use it?
139163

140-
```
141-
Usage: rprox [OPTIONS] COMMAND [ARGS]...
142-
143-
Options:
144-
--install-completion Install completion for the current shell.
145-
--show-completion Show completion for the current shell, to copy it or
146-
customize the installation.
147-
148-
--help Show this message and exit.
164+
## How can I contribute?
165+
Hey! Thanks! Any help is appreciated. Please check the [Contribution Guide](https://github.com/in03/Resolve-Proxy-Encoder/wiki/Contribution-Guide).
149166

150-
Commands:
151-
link Manually link proxies from directory to source media in open...
152-
mon Launch Flower Celery monitor in default browser new window
153-
purge Purge all proxy jobs from all queues
154-
queue Queue proxies from the currently open DaVinci Resolve timeline
155-
work Prompt to start Celery workers on local machine
156-
```

0 commit comments

Comments
 (0)