Skip to content

Commit d239d39

Browse files
asd981256asd981256
committed
add ensemble example
Signed-off-by: asd981256 <[email protected]>
1 parent acbc8ed commit d239d39

File tree

6 files changed

+287
-0
lines changed

6 files changed

+287
-0
lines changed

‎docs/modelserving/inference_graph/ensemble_vote/AvgVote.py

+42
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
import argparse
2+
from typing import Dict, Union
3+
import numpy as np
4+
from kserve import (
5+
Model,
6+
ModelServer,
7+
model_server,
8+
InferRequest,
9+
InferOutput,
10+
InferResponse,
11+
logging,
12+
)
13+
from kserve.utils.utils import get_predict_response
14+
15+
class AvgVote(Model):
16+
def __init__(self, name: str):
17+
super().__init__(name)
18+
self.model = None
19+
self.ready = False
20+
self.load()
21+
22+
def load(self):
23+
self.ready = True
24+
25+
def predict(self, payload: Union[Dict, InferRequest], headers: Dict[str, str] = None) -> Union[Dict, InferResponse]:
26+
tmp = []
27+
for isvcName, output in payload.items():
28+
prediction = output['predictions']
29+
tmp.append(prediction)
30+
31+
result = [sum(x)/len(tmp) for x in zip(*tmp)] # assume same number of label
32+
return get_predict_response(payload, result, self.name)
33+
34+
parser = argparse.ArgumentParser(parents=[model_server.parser])
35+
args, _ = parser.parse_known_args()
36+
37+
if __name__ == "__main__":
38+
if args.configure_logging:
39+
logging.configure_logging(args.log_config_file)
40+
41+
model = AvgVote(args.model_name)
42+
ModelServer().start([model])

‎docs/modelserving/inference_graph/ensemble_vote/DummyClassifier1.py

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
import argparse
2+
from typing import Dict, Union
3+
import numpy as np
4+
from kserve import (
5+
Model,
6+
ModelServer,
7+
model_server,
8+
InferRequest,
9+
InferOutput,
10+
InferResponse,
11+
logging,
12+
)
13+
from kserve.utils.utils import get_predict_response
14+
15+
16+
class DummyClassifier1(Model):
17+
def __init__(self, name: str):
18+
super().__init__(name)
19+
self.model = None
20+
self.ready = False
21+
self.load()
22+
23+
def load(self):
24+
self.ready = True
25+
26+
def predict(self, payload: Union[Dict, InferRequest], headers: Dict[str, str] = None) -> Union[Dict, InferResponse]:
27+
return {"predictions": [0.8, 0.2]}
28+
29+
parser = argparse.ArgumentParser(parents=[model_server.parser])
30+
args, _ = parser.parse_known_args()
31+
32+
if __name__ == "__main__":
33+
if args.configure_logging:
34+
logging.configure_logging(args.log_config_file)
35+
36+
model = DummyClassifier1(args.model_name)
37+
ModelServer().start([model])

‎docs/modelserving/inference_graph/ensemble_vote/DummyClassifier2.py

+37
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
import argparse
2+
from typing import Dict, Union
3+
import numpy as np
4+
from kserve import (
5+
Model,
6+
ModelServer,
7+
model_server,
8+
InferRequest,
9+
InferOutput,
10+
InferResponse,
11+
logging,
12+
)
13+
from kserve.utils.utils import get_predict_response
14+
15+
16+
class DummyClassifier2(Model):
17+
def __init__(self, name: str):
18+
super().__init__(name)
19+
self.model = None
20+
self.ready = False
21+
self.load()
22+
23+
def load(self):
24+
self.ready = True
25+
26+
def predict(self, payload: Union[Dict, InferRequest], headers: Dict[str, str] = None) -> Union[Dict, InferResponse]:
27+
return {"predictions": [0.6, 0.4]}
28+
29+
parser = argparse.ArgumentParser(parents=[model_server.parser])
30+
args, _ = parser.parse_known_args()
31+
32+
if __name__ == "__main__":
33+
if args.configure_logging:
34+
logging.configure_logging(args.log_config_file)
35+
36+
model = DummyClassifier2(args.model_name)
37+
ModelServer().start([model])

‎docs/modelserving/inference_graph/ensemble_vote/README.md

+169
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
# Deploy Ensemble Learning with InferenceGraph
2+
The tutorial demonstrate how to deploy Ensemble Learning model using `InferenceGraph`. The case should be that the classifiers are heavy or something else and you can't make them in one custom_model.
3+
4+
## Deploy the individual InferenceServices
5+
6+
### Build InferenceServices
7+
We focus on how ensemble node gather classifiers outputs and give an example of how to extract them with python code. Therefore we skip classifier part and just use [dummy classifier 1, and 2](DummyClassifier1.py) to return fixed result for demonstartion.
8+
9+
#### Ensemble Node outputs
10+
If `name` in `steps` is set, ensemble node will use it as key for its correspond `InferenceService` output. Otherwise, it use index of `InferenceService` in `steps` instead.
11+
12+
For example, Ensemble node deployed as following
13+
```yaml
14+
routerType: Ensemble
15+
steps:
16+
- serviceName: classifier-1
17+
name: classifier-1
18+
- serviceName: classifier-2
19+
```
20+
will result in similar result like this.
21+
```jsonld
22+
{"1":{"predictions":[0.6,0.4]},"classifier-1":{"predictions":[0.8,0.2]}}
23+
```
24+
#### Vote
25+
In this tutorial, we use following [python code](AvgVote.py) to build image for average vote.
26+
```python
27+
import argparse
28+
from typing import Dict, Union
29+
import numpy as np
30+
from kserve import (
31+
Model,
32+
ModelServer,
33+
model_server,
34+
InferRequest,
35+
InferOutput,
36+
InferResponse,
37+
logging,
38+
)
39+
from kserve.utils.utils import get_predict_response
40+
41+
class AvgVote(Model):
42+
def __init__(self, name: str):
43+
super().__init__(name)
44+
self.model = None
45+
self.ready = False
46+
self.load()
47+
48+
def load(self):
49+
self.ready = True
50+
51+
def predict(self, payload: Union[Dict, InferRequest], headers: Dict[str, str] = None) -> Union[Dict, InferResponse]:
52+
tmp = []
53+
for isvcName, output in payload.items():
54+
prediction = output['predictions']
55+
tmp.append(prediction)
56+
57+
result = [sum(x)/len(tmp) for x in zip(*tmp)] # assume same number of label
58+
return get_predict_response(payload, result, self.name)
59+
60+
parser = argparse.ArgumentParser(parents=[model_server.parser])
61+
args, _ = parser.parse_known_args()
62+
63+
if __name__ == "__main__":
64+
if args.configure_logging:
65+
logging.configure_logging(args.log_config_file)
66+
67+
model = AvgVote(args.model_name)
68+
ModelServer().start([model])
69+
```
70+
71+
#### Build Image
72+
We are skipping this part for now. Take a look at [custom_model buildpacks](../../v1beta1/custom/custom_model/#build-custom-serving-image-with-buildpacks), or use else tools that help you build image.
73+
74+
### Deploy InferenceServices
75+
```bash
76+
kubectl apply -f - <<EOF
77+
apiVersion: serving.kserve.io/v1beta1
78+
kind: InferenceService
79+
metadata:
80+
name: avg-vote
81+
spec:
82+
predictor:
83+
containers:
84+
- name: avg-vote
85+
image: {avg-vote-image}
86+
args:
87+
- --model_name=avg-vote
88+
---
89+
apiVersion: serving.kserve.io/v1beta1
90+
kind: InferenceService
91+
metadata:
92+
name: classifier-1
93+
spec:
94+
predictor:
95+
containers:
96+
- name: classifier-1
97+
image: {classifier-1-image}
98+
args:
99+
- --model_name=classifier-1
100+
---
101+
apiVersion: serving.kserve.io/v1beta1
102+
kind: InferenceService
103+
metadata:
104+
name: classifier-2
105+
spec:
106+
predictor:
107+
containers:
108+
- name: classifier-2
109+
image: {classifier-2-image}
110+
args:
111+
- --model_name=classifier-2
112+
EOF
113+
```
114+
All InferenceSerivces should be ready.
115+
```bash
116+
kubectl get isvc
117+
NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE
118+
avg-vote http://avg-vote.default.example.com True 100 avg-vote-predictor-00001
119+
classifier-1 http://classifier-1.default.example.com True 100 classifier-1-predictor-00001
120+
classifier-2 http://classifier-2.default.example.com True 100 classifier-2-predictor-00001
121+
```
122+
123+
124+
## Deploy InferenceGraph
125+
126+
```bash
127+
kubectl apply -f - <<EOF
128+
apiVersion: "serving.kserve.io/v1alpha1"
129+
kind: "InferenceGraph"
130+
metadata:
131+
name: "ensemble-2-avg-vote"
132+
spec:
133+
nodes:
134+
root:
135+
routerType: Sequence
136+
steps:
137+
- nodeName: ensemble-2
138+
name: ensemble-2
139+
- serviceName: avg-vote
140+
name: avg-vote
141+
data: $response
142+
ensemble-2:
143+
routerType: Ensemble
144+
steps:
145+
- serviceName: classifier-1
146+
name: classifier-1
147+
- serviceName: classifier-2
148+
name: classifier-2
149+
EOF
150+
```
151+
152+
## Test the InferenceGraph
153+
First, check the `InferenceGraph` ready state
154+
```bash
155+
kubectl get ig ensemble-2-avg-vote
156+
NAME URL READY AGE
157+
ensemble-2-avg-vote http://ensemble-2-avg-vote.default.example.com True
158+
```
159+
Second, [determine the ingress IP and ports](../../../get_started/first_isvc.md#4-determine-the-ingress-ip-and-ports) and set `INGRESS_HOST` and `INGRESS_PORT`. Now, can test by sending [data](input.json).
160+
161+
```bash
162+
SERVICE_HOSTNAME=$(kubectl get ig ensemble-2-avg-vote -o jsonpath='{.status.url}' | cut -d "/" -f 3)
163+
curl -v -H "Host: ${SERVICE_HOSTNAME}" -H "Content-Type: application/json" http://${INGRESS_HOST}:${INGRESS_PORT} -d @./input.json
164+
```
165+
!!! success "Expected Output"
166+
```{ .json .no-copy }
167+
{"predictions":[0.7,0.30000000000000004]
168+
```
169+

‎docs/modelserving/inference_graph/ensemble_vote/input.json

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"inputs": "sample single input string"}

‎mkdocs.yml

+1
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,7 @@ nav:
6363
- Inference Graph:
6464
- Concept: modelserving/inference_graph/README.md
6565
- Image classification inference graph: modelserving/inference_graph/image_pipeline/README.md
66+
- Ensemble Learning inference graph: modelserving/inference_graph/ensemble_vote/README.md
6667
- Model Storage:
6768
- Storage Containers: modelserving/storage/storagecontainers.md
6869
- Azure: modelserving/storage/azure/azure.md

0 commit comments

Comments
 (0)