Skip to content

Commit 63cccba

Browse files
ishaan-jaffishaan-jaff
committed
docs - audit logs
1 parent 3dc05bf commit 63cccba

File tree

2 files changed

+194
-142
lines changed

2 files changed

+194
-142
lines changed

‎docs/my-website/docs/proxy/enterprise.md

+193-141
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ import Image from '@theme/IdealImage';
22
import Tabs from '@theme/Tabs';
33
import TabItem from '@theme/TabItem';
44

5-
# ✨ Enterprise Features - Content Mod, SSO, Custom Swagger
5+
# ✨ Enterprise Features - SSO, Audit Logs, Guardrails
66

77
Features here are behind a commercial license in our `/enterprise` folder. [**See Code**](https://github.com/BerriAI/litellm/tree/main/enterprise)
88

@@ -14,18 +14,203 @@ Features here are behind a commercial license in our `/enterprise` folder. [**Se
1414

1515
Features:
1616
- ✅ [SSO for Admin UI](./ui.md#✨-enterprise-features)
17+
- ✅ [Audit Logs](./ui.md#✨-enterprise-features)
18+
- ✅ [Tracking Spend for Custom Tags](#tracking-spend-for-custom-tags)
1719
- ✅ Content Moderation with LLM Guard, LlamaGuard, Google Text Moderations
1820
- ✅ [Prompt Injection Detection (with LakeraAI API)](#prompt-injection-detection-lakeraai)
1921
- ✅ Reject calls from Blocked User list
2022
- ✅ Reject calls (incoming / outgoing) with Banned Keywords (e.g. competitors)
21-
- ✅ Don't log/store specific requests to Langfuse, Sentry, etc. (eg confidential LLM requests)
22-
- ✅ Tracking Spend for Custom Tags
2323
- ✅ Custom Branding + Routes on Swagger Docs
24-
- ✅ Audit Logs for `Created At, Created By` when Models Added
24+
25+
## Audit Logs
26+
27+
Store Audit logs for **Create, Update Delete Operations** done on `Teams` and `Virtual Keys`
28+
29+
**Step 1** Switch on audit Logs
30+
```shell
31+
litellm_settings:
32+
store_audit_logs: true
33+
```
34+
35+
Start the litellm proxy with this config
36+
37+
**Step 2** Test it - Create a Team
38+
39+
```shell
40+
curl --location 'http://0.0.0.0:4000/team/new' \
41+
--header 'Authorization: Bearer sk-1234' \
42+
--header 'Content-Type: application/json' \
43+
--data '{
44+
"max_budget": 2
45+
}'
46+
```
47+
48+
**Step 3** Expected Log
49+
50+
```json
51+
{
52+
"id": "e1760e10-4264-4499-82cd-c08c86c8d05b",
53+
"updated_at": "2024-06-06T02:10:40.836420+00:00",
54+
"changed_by": "109010464461339474872",
55+
"action": "created",
56+
"table_name": "LiteLLM_TeamTable",
57+
"object_id": "82e725b5-053f-459d-9a52-867191635446",
58+
"before_value": null,
59+
"updated_values": {
60+
"team_id": "82e725b5-053f-459d-9a52-867191635446",
61+
"admins": [],
62+
"members": [],
63+
"members_with_roles": [
64+
{
65+
"role": "admin",
66+
"user_id": "109010464461339474872"
67+
}
68+
],
69+
"max_budget": 2.0,
70+
"models": [],
71+
"blocked": false
72+
}
73+
}
74+
```
75+
76+
77+
## Tracking Spend for Custom Tags
78+
79+
Requirements:
80+
81+
- Virtual Keys & a database should be set up, see [virtual keys](https://docs.litellm.ai/docs/proxy/virtual_keys)
82+
83+
### Usage - /chat/completions requests with request tags
84+
85+
86+
<Tabs>
87+
88+
89+
<TabItem value="openai" label="OpenAI Python v1.0.0+">
90+
91+
Set `extra_body={"metadata": { }}` to `metadata` you want to pass
92+
93+
```python
94+
import openai
95+
client = openai.OpenAI(
96+
api_key="anything",
97+
base_url="http://0.0.0.0:4000"
98+
)
99+
100+
# request sent to model set on litellm proxy, `litellm --model`
101+
response = client.chat.completions.create(
102+
model="gpt-3.5-turbo",
103+
messages = [
104+
{
105+
"role": "user",
106+
"content": "this is a test request, write a short poem"
107+
}
108+
],
109+
extra_body={
110+
"metadata": {
111+
"tags": ["model-anthropic-claude-v2.1", "app-ishaan-prod"]
112+
}
113+
}
114+
)
115+
116+
print(response)
117+
```
118+
</TabItem>
119+
120+
<TabItem value="Curl" label="Curl Request">
121+
122+
Pass `metadata` as part of the request body
123+
124+
```shell
125+
curl --location 'http://0.0.0.0:4000/chat/completions' \
126+
--header 'Content-Type: application/json' \
127+
--data '{
128+
"model": "gpt-3.5-turbo",
129+
"messages": [
130+
{
131+
"role": "user",
132+
"content": "what llm are you"
133+
}
134+
],
135+
"metadata": {"tags": ["model-anthropic-claude-v2.1", "app-ishaan-prod"]}
136+
}'
137+
```
138+
</TabItem>
139+
<TabItem value="langchain" label="Langchain">
140+
141+
```python
142+
from langchain.chat_models import ChatOpenAI
143+
from langchain.prompts.chat import (
144+
ChatPromptTemplate,
145+
HumanMessagePromptTemplate,
146+
SystemMessagePromptTemplate,
147+
)
148+
from langchain.schema import HumanMessage, SystemMessage
149+
150+
chat = ChatOpenAI(
151+
openai_api_base="http://0.0.0.0:4000",
152+
model = "gpt-3.5-turbo",
153+
temperature=0.1,
154+
extra_body={
155+
"metadata": {
156+
"tags": ["model-anthropic-claude-v2.1", "app-ishaan-prod"]
157+
}
158+
}
159+
)
160+
161+
messages = [
162+
SystemMessage(
163+
content="You are a helpful assistant that im using to make a test request to."
164+
),
165+
HumanMessage(
166+
content="test from litellm. tell me why it's amazing in 1 sentence"
167+
),
168+
]
169+
response = chat(messages)
170+
171+
print(response)
172+
```
173+
174+
</TabItem>
175+
</Tabs>
176+
177+
178+
### Viewing Spend per tag
179+
180+
#### `/spend/tags` Request Format
181+
```shell
182+
curl -X GET "http://0.0.0.0:4000/spend/tags" \
183+
-H "Authorization: Bearer sk-1234"
184+
```
185+
186+
#### `/spend/tags`Response Format
187+
```shell
188+
[
189+
{
190+
"individual_request_tag": "model-anthropic-claude-v2.1",
191+
"log_count": 6,
192+
"total_spend": 0.000672
193+
},
194+
{
195+
"individual_request_tag": "app-ishaan-local",
196+
"log_count": 4,
197+
"total_spend": 0.000448
198+
},
199+
{
200+
"individual_request_tag": "app-ishaan-prod",
201+
"log_count": 2,
202+
"total_spend": 0.000224
203+
}
204+
]
205+
206+
```
207+
208+
209+
<!-- ## Tracking Spend per Key
25210
26211
27212
## Content Moderation
28-
### Content Moderation with LLM Guard
213+
#### Content Moderation with LLM Guard
29214
30215
Set the LLM Guard API Base in your environment
31216
@@ -160,7 +345,7 @@ curl --location 'http://0.0.0.0:4000/v1/chat/completions' \
160345
</TabItem>
161346
</Tabs>
162347
163-
### Content Moderation with LlamaGuard
348+
#### Content Moderation with LlamaGuard
164349
165350
Currently works with Sagemaker's LlamaGuard endpoint.
166351
@@ -194,7 +379,7 @@ callbacks: ["llamaguard_moderations"]
194379
195380
196381
197-
### Content Moderation with Google Text Moderation
382+
#### Content Moderation with Google Text Moderation
198383
199384
Requires your GOOGLE_APPLICATION_CREDENTIALS to be set in your .env (same as VertexAI).
200385
@@ -250,7 +435,7 @@ Here are the category specific values:
250435
251436
252437
253-
### Content Moderation with OpenAI Moderations
438+
#### Content Moderation with OpenAI Moderations
254439
255440
Use this if you want to reject /chat, /completions, /embeddings calls that fail OpenAI Moderations checks
256441
@@ -417,139 +602,6 @@ curl --location 'http://0.0.0.0:4000/chat/completions' \
417602
}
418603
'
419604
```
420-
## Tracking Spend for Custom Tags
421-
422-
Requirements:
423-
424-
- Virtual Keys & a database should be set up, see [virtual keys](https://docs.litellm.ai/docs/proxy/virtual_keys)
425-
426-
### Usage - /chat/completions requests with request tags
427-
428-
429-
<Tabs>
430-
431-
432-
<TabItem value="openai" label="OpenAI Python v1.0.0+">
433-
434-
Set `extra_body={"metadata": { }}` to `metadata` you want to pass
435-
436-
```python
437-
import openai
438-
client = openai.OpenAI(
439-
api_key="anything",
440-
base_url="http://0.0.0.0:4000"
441-
)
442-
443-
# request sent to model set on litellm proxy, `litellm --model`
444-
response = client.chat.completions.create(
445-
model="gpt-3.5-turbo",
446-
messages = [
447-
{
448-
"role": "user",
449-
"content": "this is a test request, write a short poem"
450-
}
451-
],
452-
extra_body={
453-
"metadata": {
454-
"tags": ["model-anthropic-claude-v2.1", "app-ishaan-prod"]
455-
}
456-
}
457-
)
458-
459-
print(response)
460-
```
461-
</TabItem>
462-
463-
<TabItem value="Curl" label="Curl Request">
464-
465-
Pass `metadata` as part of the request body
466-
467-
```shell
468-
curl --location 'http://0.0.0.0:4000/chat/completions' \
469-
--header 'Content-Type: application/json' \
470-
--data '{
471-
"model": "gpt-3.5-turbo",
472-
"messages": [
473-
{
474-
"role": "user",
475-
"content": "what llm are you"
476-
}
477-
],
478-
"metadata": {"tags": ["model-anthropic-claude-v2.1", "app-ishaan-prod"]}
479-
}'
480-
```
481-
</TabItem>
482-
<TabItem value="langchain" label="Langchain">
483-
484-
```python
485-
from langchain.chat_models import ChatOpenAI
486-
from langchain.prompts.chat import (
487-
ChatPromptTemplate,
488-
HumanMessagePromptTemplate,
489-
SystemMessagePromptTemplate,
490-
)
491-
from langchain.schema import HumanMessage, SystemMessage
492-
493-
chat = ChatOpenAI(
494-
openai_api_base="http://0.0.0.0:4000",
495-
model = "gpt-3.5-turbo",
496-
temperature=0.1,
497-
extra_body={
498-
"metadata": {
499-
"tags": ["model-anthropic-claude-v2.1", "app-ishaan-prod"]
500-
}
501-
}
502-
)
503-
504-
messages = [
505-
SystemMessage(
506-
content="You are a helpful assistant that im using to make a test request to."
507-
),
508-
HumanMessage(
509-
content="test from litellm. tell me why it's amazing in 1 sentence"
510-
),
511-
]
512-
response = chat(messages)
513-
514-
print(response)
515-
```
516-
517-
</TabItem>
518-
</Tabs>
519-
520-
521-
### Viewing Spend per tag
522-
523-
#### `/spend/tags` Request Format
524-
```shell
525-
curl -X GET "http://0.0.0.0:4000/spend/tags" \
526-
-H "Authorization: Bearer sk-1234"
527-
```
528-
529-
#### `/spend/tags`Response Format
530-
```shell
531-
[
532-
{
533-
"individual_request_tag": "model-anthropic-claude-v2.1",
534-
"log_count": 6,
535-
"total_spend": 0.000672
536-
},
537-
{
538-
"individual_request_tag": "app-ishaan-local",
539-
"log_count": 4,
540-
"total_spend": 0.000448
541-
},
542-
{
543-
"individual_request_tag": "app-ishaan-prod",
544-
"log_count": 2,
545-
"total_spend": 0.000224
546-
}
547-
]
548-
549-
```
550-
551-
552-
<!-- ## Tracking Spend per Key
553605
554606
## Tracking Spend per User -->
555607

‎docs/my-website/sidebars.js

+1-1
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ const sidebars = {
3636
label: "📖 All Endpoints (Swagger)",
3737
href: "https://litellm-api.up.railway.app/",
3838
},
39+
"proxy/enterprise",
3940
"proxy/demo",
4041
"proxy/configs",
4142
"proxy/reliability",
@@ -44,7 +45,6 @@ const sidebars = {
4445
"proxy/customers",
4546
"proxy/billing",
4647
"proxy/user_keys",
47-
"proxy/enterprise",
4848
"proxy/virtual_keys",
4949
"proxy/alerting",
5050
{

0 commit comments

Comments
 (0)