Skip to content

Commit fc84200

Browse files
Apply suggestions from code review
Co-authored-by: Jordan Stephens <[email protected]>
1 parent e920df0 commit fc84200

File tree

4 files changed

+21
-15
lines changed

4 files changed

+21
-15
lines changed

docs/concepts/managed-llms/managed-language-models.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ Each cloud provider offers their own managed Large Language Model services. AWS
2121

2222
In order to leverage cloud-native managed language models from your Defang services, all you need to do is add the `x-defang-llm` extension to the service config and Defang will configure the approprate roles and permissions for you.
2323

24-
:::tip
24+
:::info
2525
Ensure you have the necessary permissions to access the model you intend to use. To do this, you can check your [AWS Bedrock model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access-modify.html) or [GCP Vertex AI model access](https://cloud.google.com/vertex-ai/generative-ai/docs/control-model-access).
2626
:::
2727

docs/concepts/managed-llms/openai-access-gateway.md

+5-1
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,11 @@ The `x-defang-llm` extension is used to configure the appropriate roles and perm
3737

3838
## Model Mapping
3939

40-
Defang supports model mapping through the [openai-access-gateway](https://github.com/DefangLabs/openai-access-gateway) on AWS and GCP. This takes a model with a Docker naming convention (e.g. `ai/llama3.3`) and maps it to the closest matching model name on the target platform. If no such match can be found it can fallback onto a known existing model (e.g. `ai/mistral`). These environment variables are `USE_MODEL_MAPPING` (default to true) and `FALLBACK_MODEL` (no default), respectively.
40+
Defang supports model mapping through the [openai-access-gateway](https://github.com/DefangLabs/openai-access-gateway) on AWS and GCP. This takes a model with a Docker naming convention (e.g. `ai/llama3.3`) and maps it to the closest matching model name on the target platform. If no such match can be found it can fallback onto a known existing model (e.g. `ai/mistral`).
41+
42+
This can be configured through the following environment variables:
43+
* `USE_MODEL_MAPPING` (default to true) - configures whether or not model mapping should be enabled.
44+
* `FALLBACK_MODEL` (no default) - configure a model which will be used if model mapping fails to find a target model.
4145

4246
## Current Support
4347

docs/tutorials/deploy-openai-apps/aws-bedrock.mdx

+6-5
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,11 @@ import {useColorMode} from '@docusaurus/theme-common';
1111
Let's assume you have an app that uses an OpenAI client library and you want to deploy it to the cloud on **AWS Bedrock**.
1212

1313
This tutorial shows you how **Defang** makes it easy.
14+
:::info
15+
You must [configure AWS Bedrock model access](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access-modify.html) for each model you intend to use in your AWS account.
16+
:::
1417

15-
Suppose you start with a Compose file like this:
18+
Suppose you start with a `compose.yaml` file with one `app` service, like this:
1619

1720
```yaml
1821
services:
@@ -31,9 +34,7 @@ services:
3134
3235
## Add an LLM Service to Your Compose File
3336
34-
You need to add a new service that acts as a proxy between your app and the backend LLM provider (Bedrock).
35-
36-
Add **Defang's [openai-access-gateway](https://github.com/DefangLabs/openai-access-gateway)** service:
37+
You can use AWS Bedrock without changing your `app` code by introducing a new [`defangio/openai-access-gateway`](https://github.com/DefangLabs/openai-access-gateway) service. We'll call the new service `llm`. This new service will act as a proxy between your application and AWS Bedrock, and will transparently handle converting your OpenAI requests into AWS Bedrock requests and Bedrock responses into OpenAI responses. This allows you to use AWS Bedrock with your existing OpenAI client SDK.
3738

3839
```diff
3940
+ llm:
@@ -161,4 +162,4 @@ You now have a single app that can:
161162

162163
- Talk to **AWS Bedrock**
163164
- Use the same OpenAI-compatible client code
164-
- Easily switch cloud providers by changing a few environment variables
165+
- Easily switch between models or cloud providers by changing a few environment variables

docs/tutorials/deploy-openai-apps/gcp-vertex.mdx

+9-8
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,11 @@ import {useColorMode} from '@docusaurus/theme-common';
1111
Let's assume you have an application that uses an OpenAI client library and you want to deploy it to the cloud using **GCP Vertex AI**.
1212

1313
This tutorial shows you how **Defang** makes it easy.
14+
:::info
15+
You must [configure GCP Vertex AI model access](https://cloud.google.com/vertex-ai/generative-ai/docs/control-model-access) for each model you intend to use in your GCP account.
16+
:::
1417

15-
Suppose you start with a Compose file like this:
18+
Suppose you start with a `compose.yaml` file with one `app` service, like this:
1619

1720
```yaml
1821
services:
@@ -31,9 +34,7 @@ services:
3134
3235
## Add an LLM Service to Your Compose File
3336
34-
You need to add a new service that acts as a proxy between your app and the backend LLM provider (Vertex AI).
35-
36-
Add **Defang's [openai-access-gateway](https://github.com/DefangLabs/openai-access-gateway)** service:
37+
You can use Vertex AI without changing your `app` code by introducing a new [`defangio/openai-access-gateway`](https://github.com/DefangLabs/openai-access-gateway) service. We'll call the new service `llm`. This new service will act as a proxy between your application and Vertex AI, and will transparently handle converting your OpenAI requests into Vertex AI requests and Vertex AI responses into OpenAI responses. This allows you to use Vertex AI with your existing OpenAI client SDK.
3738

3839
```diff
3940
+ llm:
@@ -54,8 +55,8 @@ Add **Defang's [openai-access-gateway](https://github.com/DefangLabs/openai-acce
5455
- The container image is based on [aws-samples/bedrock-access-gateway](https://github.com/aws-samples/bedrock-access-gateway), with enhancements.
5556
- `x-defang-llm: true` signals to **Defang** that this service should be configured to use target platform AI services.
5657
- New environment variables:
57-
- `REGION` is the zone where the services runs (e.g. us-central1)
58-
- `GCP_PROJECT_ID` is your project to deploy to (e.g. my-project-456789)
58+
- `REGION` is the zone where the services runs (e.g. `us-central1`)
59+
- `GCP_PROJECT_ID` is your project to deploy to (e.g. `my-project-456789`)
5960

6061
:::tip
6162
**OpenAI Key**
@@ -148,7 +149,7 @@ services:
148149
mode: host
149150
environment:
150151
- OPENAI_API_KEY
151-
- GCP_PROJECT_ID # required if using GCP Vertex AI
152+
- GCP_PROJECT_ID
152153
- REGION
153154
```
154155

@@ -168,4 +169,4 @@ You now have a single app that can:
168169

169170
- Talk to **GCP Vertex AI**
170171
- Use the same OpenAI-compatible client code
171-
- Easily switch cloud providers by changing a few environment variables
172+
- Easily switch between models or cloud providers by changing a few environment variables

0 commit comments

Comments
 (0)