You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/concepts/managed-llms/managed-language-models.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -21,7 +21,7 @@ Each cloud provider offers their own managed Large Language Model services. AWS
21
21
22
22
In order to leverage cloud-native managed language models from your Defang services, all you need to do is add the `x-defang-llm` extension to the service config and Defang will configure the approprate roles and permissions for you.
23
23
24
-
:::tip
24
+
:::info
25
25
Ensure you have the necessary permissions to access the model you intend to use. To do this, you can check your [AWS Bedrock model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access-modify.html) or [GCP Vertex AI model access](https://cloud.google.com/vertex-ai/generative-ai/docs/control-model-access).
Copy file name to clipboardExpand all lines: docs/concepts/managed-llms/openai-access-gateway.md
+5-1
Original file line number
Diff line number
Diff line change
@@ -37,7 +37,11 @@ The `x-defang-llm` extension is used to configure the appropriate roles and perm
37
37
38
38
## Model Mapping
39
39
40
-
Defang supports model mapping through the [openai-access-gateway](https://github.com/DefangLabs/openai-access-gateway) on AWS and GCP. This takes a model with a Docker naming convention (e.g. `ai/llama3.3`) and maps it to the closest matching model name on the target platform. If no such match can be found it can fallback onto a known existing model (e.g. `ai/mistral`). These environment variables are `USE_MODEL_MAPPING` (default to true) and `FALLBACK_MODEL` (no default), respectively.
40
+
Defang supports model mapping through the [openai-access-gateway](https://github.com/DefangLabs/openai-access-gateway) on AWS and GCP. This takes a model with a Docker naming convention (e.g. `ai/llama3.3`) and maps it to the closest matching model name on the target platform. If no such match can be found it can fallback onto a known existing model (e.g. `ai/mistral`).
41
+
42
+
This can be configured through the following environment variables:
43
+
* `USE_MODEL_MAPPING` (default to true) - configures whether or not model mapping should be enabled.
44
+
* `FALLBACK_MODEL` (no default) - configure a model which will be used if model mapping fails to find a target model.
Copy file name to clipboardExpand all lines: docs/tutorials/deploy-openai-apps/aws-bedrock.mdx
+6-5
Original file line number
Diff line number
Diff line change
@@ -11,8 +11,11 @@ import {useColorMode} from '@docusaurus/theme-common';
11
11
Let's assume you have an app that uses an OpenAI client library and you want to deploy it to the cloud on **AWS Bedrock**.
12
12
13
13
This tutorial shows you how **Defang** makes it easy.
14
+
:::info
15
+
You must [configure AWS Bedrock model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access-modify.html) for each model you intend to use in your AWS account.
16
+
:::
14
17
15
-
Suppose you start with a Compose file like this:
18
+
Suppose you start with a `compose.yaml` file with one `app` service, like this:
16
19
17
20
```yaml
18
21
services:
@@ -31,9 +34,7 @@ services:
31
34
32
35
## Add an LLM Service to Your Compose File
33
36
34
-
You need to add a new service that acts as a proxy between your app and the backend LLM provider (Bedrock).
You can use AWS Bedrock without changing your `app` code by introducing a new [`defangio/openai-access-gateway`](https://github.com/DefangLabs/openai-access-gateway) service. We'll call the new service `llm`. This new service will act as a proxy between your application and AWS Bedrock, and will transparently handle converting your OpenAI requests into AWS Bedrock requests and Bedrock responses into OpenAI responses. This allows you to use AWS Bedrock with your existing OpenAI client SDK.
37
38
38
39
```diff
39
40
+ llm:
@@ -161,4 +162,4 @@ You now have a single app that can:
161
162
162
163
- Talk to **AWS Bedrock**
163
164
- Use the same OpenAI-compatible client code
164
-
- Easily switch cloud providers by changing a few environment variables
165
+
- Easily switch between models or cloud providers by changing a few environment variables
Copy file name to clipboardExpand all lines: docs/tutorials/deploy-openai-apps/gcp-vertex.mdx
+9-8
Original file line number
Diff line number
Diff line change
@@ -11,8 +11,11 @@ import {useColorMode} from '@docusaurus/theme-common';
11
11
Let's assume you have an application that uses an OpenAI client library and you want to deploy it to the cloud using **GCP Vertex AI**.
12
12
13
13
This tutorial shows you how **Defang** makes it easy.
14
+
:::info
15
+
You must [configure GCP Vertex AI model access](https://cloud.google.com/vertex-ai/generative-ai/docs/control-model-access) for each model you intend to use in your GCP account.
16
+
:::
14
17
15
-
Suppose you start with a Compose file like this:
18
+
Suppose you start with a `compose.yaml` file with one `app` service, like this:
16
19
17
20
```yaml
18
21
services:
@@ -31,9 +34,7 @@ services:
31
34
32
35
## Add an LLM Service to Your Compose File
33
36
34
-
You need to add a new service that acts as a proxy between your app and the backend LLM provider (Vertex AI).
You can use Vertex AI without changing your `app` code by introducing a new [`defangio/openai-access-gateway`](https://github.com/DefangLabs/openai-access-gateway) service. We'll call the new service `llm`. This new service will act as a proxy between your application and Vertex AI, and will transparently handle converting your OpenAI requests into Vertex AI requests and Vertex AI responses into OpenAI responses. This allows you to use Vertex AI with your existing OpenAI client SDK.
0 commit comments