Skip to content

Commit 6b5cdc7

Browse files
alexmtalexmt
authored
docs: document server side pagination proposal (#17222)
Signed-off-by: Alexander Matyushentsev <[email protected]>
1 parent d85a1de commit 6b5cdc7

File tree

1 file changed

+180
-0
lines changed

1 file changed

+180
-0
lines changed

‎docs/proposals/server-side-pagination.md

+180
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
---
2+
title: Server Side Pagination for Applications List and Watch APIs
3+
authors:
4+
- "@alexmt"
5+
sponsors:
6+
- "@jessesuen"
7+
reviewers:
8+
- TBD
9+
approvers:
10+
- TBD
11+
12+
creation-date: 2024-02-14
13+
last-updated: 2024-02-14
14+
---
15+
16+
# Introduce Server Side Pagination for Applications List and Watch APIs
17+
18+
Improve Argo CD performance by introducing server side pagination for Applications List and Watch APIs.
19+
20+
## Open Questions [optional]
21+
22+
This is where to call out areas of the design that require closure before deciding to implement the
23+
design.
24+
25+
26+
## Summary
27+
28+
The Argo CD API server currently returns all applications in a single response. This can be a performance
29+
bottleneck when there are a large number of applications. This proposal is to introduce server side pagination
30+
for the Applications List and Watch APIs.
31+
32+
## Motivation
33+
34+
The main motivation for this proposal it to improve the Argo CD UI responsiveness when there are a large number
35+
of applications. The API server memory usage increases with the number of applications however this is not critical
36+
and can be mitigated by increasing memory limits for the API server deployment. The UI however becames unresponsive
37+
even on a powerful machine when the number of applications increases 2000. The server side pagination will allow
38+
to reduce amount of data returned by the API server and improve the UI responsiveness.
39+
40+
### Goals
41+
42+
* Support server side pagination for Applications List and Watch APIs
43+
44+
* Leverage pagination in the Argo CD UI to improve responsiveness
45+
46+
* Leverage pagination in the Argo CD CLI to improve performance and reduce load on the API server
47+
48+
### Non-Goals
49+
50+
* The API Server is known to use a lot of CPU while applying very large RBAC policies to a large number of applications.
51+
Even with pagination API still need to apply RBAC policies to return "last page" response. So the issueis not addressed by this proposal.
52+
53+
## Proposal
54+
55+
**Pagination Cursor**
56+
57+
It is proposed to add `offset` and `limit` fields for pagination support in Application List API.
58+
The The Watch API is a bit more complex. Both Argo CD user interface and CLI are relying on the Watch API to display real time updates of Argo CD applications.
59+
The Watch API currently supports filtering by a project and an application name. In order to effectively
60+
implement server side pagination for the Watch API we cannot rely on the order of the applications returned by the API server. Instead of
61+
relying on the order it is proposed to rely on the application name and use it as a cursor for pagination. Both the Applications List and Watch
62+
APIs will be extended with the optional `minName` and `maxName` fields. The `minName` field will be used to specify the application name to start from
63+
and the `maxName` field will be used to specify the application name to end at. The fields should be added to the `ApplicationQuery` message
64+
which is used as a request payload for the Applications List and Watch APIs.
65+
66+
```proto
67+
message ApplicationQuery {
68+
// ... existing fields
69+
// New proto fields for server side pagination
70+
// the application name to start from (app with min name is included in response)
71+
optional string minName = 9;
72+
// the application name to end at (app with max name is included in response)
73+
optional string maxName = 10;
74+
// offset
75+
optional int64 offset = 18;
76+
// limit
77+
optional int64 limit = 19;
78+
}
79+
```
80+
81+
**Server Side Filtering**
82+
83+
In order to support server side pagination the filtering has to be moved to the server side as well. `ApplicationQuery` message needs to be extended with the following fields:
84+
85+
```proto
86+
message ApplicationQuery {
87+
// ... existing fields
88+
// New proto fields for server side filtering
89+
// the repos filter
90+
repeated string repos = 11;
91+
// the clusters filter
92+
repeated string clusters = 12;
93+
// the namespaces filter
94+
repeated string namespaces = 13;
95+
// the auth sync filter
96+
optional bool autoSyncEnabled = 14;
97+
// the sync status filter
98+
repeated string syncStatuses = 15;
99+
// the health status filter
100+
repeated string healthStatuses = 16;
101+
// search
102+
optional string search = 17;
103+
}
104+
```
105+
106+
The Argo CD UI should be updated to populate fields in the List and Watch API requests instead of performing filtering on the client side.
107+
108+
**Applications Stats**
109+
110+
The Argo CD UI displays the breakdown of the applications by the sync status, health status etc. Stats numbers are calculated on the client side
111+
and rely on the full list of applications returned by the API server. The server side pagination will break the stats calculation. The proposal is to
112+
intoduce a new `stats` field to the Applications List API response. The field will contain the breakdown of the applications by various statuses.
113+
114+
```golang
115+
type ApplicationLabelStats struct {
116+
Key string `json:"key" protobuf:"bytes,1,opt,name=key"`
117+
Values []string `json:"values" protobuf:"bytes,2,opt,name=values"`
118+
}
119+
120+
// ApplicationListStats holds additional information about the list of applications
121+
type ApplicationListStats struct {
122+
Total int64 `json:"total" protobuf:"bytes,1,opt,name=total"`
123+
TotalBySyncStatus map[SyncStatusCode]int64 `json:"totalBySyncStatus,omitempty" protobuf:"bytes,2,opt,name=totalBySyncStatus"`
124+
TotalByHealthStatus map[health.HealthStatusCode]int64 `json:"totalByHealthStatus,omitempty" protobuf:"bytes,3,opt,name=totalByHealthStatus"`
125+
AutoSyncEnabledCount int64 `json:"autoSyncEnabledCount" protobuf:"bytes,4,opt,name=autoSyncEnabledCount"`
126+
Destinations []ApplicationDestination `json:"destinations" protobuf:"bytes,5,opt,name=destinations"`
127+
Namespaces []string `json:"namespaces" protobuf:"bytes,6,opt,name=namespaces"`
128+
Labels []ApplicationLabelStats `json:"labels,omitempty" protobuf:"bytes,7,opt,name=labels"`
129+
}
130+
```
131+
132+
The `stats` filter should be populated with information about all applications returned by the API server even when single page is loaded.
133+
The Argo CD UI should be updated to use the stats returned by the API server instead of calculating the stats on the client side.
134+
135+
**Argo CD CLI**
136+
137+
The Argo CD CLI should be updated to support server side pagination. The `argocd app list` command should be updated to support `--offset` and `--limit` flags.
138+
If the `--offset` and `--limit` flags are not specified the CLI should use pagination to load all applications in batches of 500 applications.
139+
140+
### Use cases
141+
142+
Add a list of detailed use cases this enhancement intends to take care of.
143+
144+
#### User Server Side Pagination in Argo CD User Interface to improve responsiveness:
145+
As a user, I would like to be able to navigate through the list of applications using the pagination controls.
146+
147+
#### User Server Side Pagination in Argo CD CLI to reduce load on the API server:
148+
As a user, I would like to use Argo CD CLI to list applications while leveraging the pagination without overloading the API server.
149+
150+
### Implementation Details/Notes/Constraints [optional]
151+
152+
**Application Stats**
153+
154+
**CLI Backward Compatibility**
155+
156+
Typically we bump minimal supported API version when we introduce a new feature in CLI. In this case I
157+
suggest to support gracefully missing pagination support in CLI. If the API server returns more applications than
158+
specified in `limit` the CLI should assume pagination is not supported and response has full list of applications.
159+
This way the user can downgrade API server without downgrading CLI.
160+
161+
### Security Considerations
162+
163+
The proposal does not introduce any new security risks.
164+
165+
### Risks and Mitigations
166+
167+
We might need to bump minimal supported API version in CLI to support pagination. The `Implementation Details` section
168+
contains the proposal to avoid doing it.
169+
170+
### Upgrade / Downgrade Strategy
171+
172+
The proposal does not introduce any breaking changes. The API server should gracefully handle requests without pagination fields.
173+
174+
## Drawbacks
175+
176+
None.
177+
178+
## Alternatives
179+
180+
****

0 commit comments

Comments
 (0)