Skip to content

Commit 95048dc

Browse files
miledxzmiledxz
committed
Rewrite Client IP Enhancement Proposal (nginx#2329)
Problem: As a user of NGF, I want to rewrite the client's IP address to the original client's IP when fronting NGF with a LoadBalancer, so that the real client's IP address is forwarded to my application, or so that I can log the client's IP address. Solution: Add an enhancement proposal for rewriting the client's IP address
1 parent a496d61 commit 95048dc

File tree

1 file changed

+297
-0
lines changed

1 file changed

+297
-0
lines changed

‎docs/proposals/rewrite-client-ip.md

+297
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,297 @@
1+
# Enhancement Proposal-2335: Rewrite Client IP
2+
3+
- Issue: https://github.com/nginxinc/nginx-gateway-fabric/issues/2325
4+
- Status: Implementable
5+
6+
## Summary
7+
8+
This Enhancement Proposal extends the [NginxProxy API](gateway-settings.md), to allow users to configure a method to rewrite the client's IP address to the original client's IP when NGF is fronted by another load balancer or proxy.
9+
10+
## Goals
11+
12+
- Define the API for rewriting the client's IP address.
13+
14+
## Non-Goals
15+
16+
- Provide implementation details for implementing the new API.
17+
18+
## Introduction
19+
20+
When requests travel through one or more proxies or load balancers before reaching NGINX Gateway Fabric, the client IP address is set to the IP address of the server that last handled the request.
21+
22+
For example, consider this request flow:
23+
24+
```mermaid
25+
flowchart LR
26+
C(Client 1.1.1.1) --> P1(Proxy1 2.2.2.2) --> P2(Proxy2 3.3.3.3) --> NGF(NGINX Gateway Fabric)
27+
```
28+
29+
When the request reaches NGINX Gateway Fabric, the client's IP address, stored in the NGINX variable `$remote_addr`, is set to `3.3.3.3`. A user may want to preserve the original client's IP address, in this case `1.1.1.1`, and pass that to their backend applications.
30+
31+
### Methods for preserving client IP addresses
32+
33+
- X-Forwarded-For: A multi-value HTTP header that is appended to by each proxy. Each proxy appends the IP address of the host from which it received the request. Resulting header should look like `X-Forwarded-For: client, proxy1, proxy2`. Other headers for passing port, host, and proto information: X-Forwarded-Host, X-Forwarded-Port, X-Forwarded-Proto.
34+
- [Forwarded](https://datatracker.ietf.org/doc/html/rfc7239): A multi-value HTTP header of key-value pairs separated by semicolons. `Forwarded: for=client;port=80;proto=https, for=proxy;port=80;proto=https`. Similar to X-Forwarded-For, each proxy appends to this header.
35+
- X-Real-IP: a single value header that contains just the client IP address.
36+
- [PROXY protocol](http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt): allows TCP proxies to inject data about original source and dest addresses to their upstream servers without knowledge of underlying protocol. Can operate at L4, instead of L7.
37+
- Custom header: determine the client IP address for a request based on a trusted custom HTTP header.
38+
39+
The most popular methods are PROXY protocol and X-Forwarded-For. Choosing a method will depend on how the Load Balancer fronting NGF preserves the client IP address, and what the user wants to do with the client IP. If passing the client IP to the backend, then it's important to consider how the backend expects to receive this information.
40+
41+
Initially, this design will expose these two methods only, but it can be extended in the future to support the additional methods.
42+
43+
### Required NGINX directives and their behavior
44+
45+
#### `proxy_protocol` listen param
46+
47+
The `proxy_protocol` listen parameter configures NGINX server to accept the PROXY protocol. NGINX will also set the `$proxy_protocol_addr` and `$proxy_protocol_port` variables to the original client address and port.
48+
49+
#### real ip module
50+
51+
The [real-ip module](https://nginx.org/en/docs/http/ngx_http_realip_module.html) rewrites the values in the `$remote_addr` and `$remote_port` variables to the client IP address and port. Without this module, the `$remote_addr` and `$remote_port` variables are set to the IP address and port of the load balancer.
52+
53+
How the real-ip modules determines the client IP address and port depends on how you configure it.
54+
55+
#### `set_real_ip_from`
56+
57+
The [`set_real_ip_from`](https://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from) directive tells NGINX to only trust replacement IPs from these addresses.
58+
59+
If not provided, the `$remote_addr` and `$remote_port` variables will never be replaced.
60+
61+
To trust all addresses, set to `0.0.0.0/0`.
62+
63+
This directive is also used by the `real_ip_recursive` directive.
64+
65+
#### `real_ip_header`
66+
67+
The [`real_ip_header`](https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header) directive sets the header whose value will be used to replace the client address. By default, NGINX will use the value of the X-Real-IP header. This directive can be set to X-Forwarded-To, proxy_protocol, or any other header name. If set to proxy_protocol, proxy_protocol must be enabled on the server.
68+
69+
#### `real_ip_recursive`
70+
71+
The [`real_ip_recursive`](https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_recursive) directive configures whether recursive search is used when selecting the client's address from a multi-value header. Only makes sense when the header specified in `real_ip_header` is a multi-value header (e.g. contains a list of addresses). Commonly used with X-Forwarded-For. For example:
72+
73+
Say you have the following setup:
74+
75+
```mermaid
76+
flowchart LR
77+
C(Client 1.1.1.1) --> P1(Proxy1 2.2.2.2) --> P2(Proxy2 3.3.3.3) --> NGF(NGINX Gateway Fabric)
78+
```
79+
80+
and the following NGINX config:
81+
82+
```nginx configuration
83+
set_real_ip_from 3.3.3.3;
84+
real_ip_header X-Forwarded-For;
85+
real_ip_recursive on;
86+
```
87+
88+
Once the request hits NGF, the `X-Forwarded-For` header contains three IP addresses: `X-Forwarded-For: [1.1.1.1, 2.2.2.2, 3.3.3.3]`
89+
90+
Because `real_ip_recursive` is on, NGINX will set `$remote_addr` to 2.2.2.2. This is because it recurses on the values in X-Forwarded-Header from end of array to start of array and selects the first untrusted ip. If you wanted to set `$remote_addr` to the user's IP address instead, and you trust the Proxy, you could achieve that by also specifying the Proxy's IP using `set_real_ip_from`:
91+
92+
```nginx configuration
93+
set_real_ip_from 3.3.3.3;
94+
set_real_ip_from 2.2.2.2;
95+
real_ip_header X-Forwarded-For;
96+
real_ip_recursive on;
97+
```
98+
99+
If `real_ip_recursive` is off, NGINX will set `$remote_addr` to 3.3.3.3 because it will select the rightmost address.
100+
101+
## API, Customer Driven Interfaces, and User Experience
102+
103+
This API will be added to the `NginxProxy` CRD that is a part of the `gateway.nginx.org` Group. It will be referenced in the `parametersRef` field of a GatewayClass. It will live at the cluster scope.
104+
105+
This is a dynamic configuration that can be changed by a user at any time, and NGF will propagate those changes to NGINX.
106+
107+
For example, an `NginxProxy` named `proxy-settings` would be referenced as follows:
108+
109+
```yaml
110+
kind: GatewayClass
111+
metadata:
112+
name: nginx
113+
spec:
114+
controllerName: gateway.nginx.org/nginx-gateway-controller
115+
parametersRef:
116+
group: gateway.nginx.org/v1alpha1
117+
kind: NginxProxy
118+
name: proxy-settings
119+
```
120+
121+
Below is the Golang API for the `RewriteClientIP` field on `NginxProxy`. Note, all other `NginxProxy` fields have been omitted to keep the focus on `RewriteClientIP`.
122+
123+
### Go
124+
125+
```go
126+
package v1alpha1
127+
128+
import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
129+
130+
type NginxProxy struct {
131+
metav1.TypeMeta `json:",inline"`
132+
metav1.ObjectMeta `json:"metadata,omitempty"`
133+
134+
// Spec defines the desired state of the NginxProxy.
135+
Spec NginxProxySpec `json:"spec"`
136+
}
137+
138+
type NginxProxySpec struct {
139+
// RewriteClientIP contains configuration for rewriting the client IP to the original client's IP.
140+
// +optional
141+
RewriteClientIP *RewriteClientIP `json:"rewriteClientIP,omitempty"`
142+
}
143+
144+
// RewriteClientIP specifies the configuration for rewriting the client's IP address.
145+
// The client's IP will be stored in the $remote_addr NGINX variable and passed to the backends in the X-Real-IP and X-Forwarded-For* headers.
146+
type RewriteClientIP struct {
147+
// Mode defines how NGINX will rewrite the client's IP address.
148+
// There are two possible modes:
149+
// - ProxyProtocol: NGINX will rewrite the client's IP using the PROXY protocol header.
150+
// - XForwardedFor: NGINX will rewrite the client's IP using the X-Forwarded-For header.
151+
// +optional
152+
Mode *RewriteClientIPModeType `json:"mode,omitempty"`
153+
154+
// TrustedAddresses specifies the addresses that are trusted to send correct client IP information.
155+
// If a request comes from a trusted address, NGINX will rewrite the client IP information, and forward it to the backend in the X-Forwarded-For* and X-Real-IP headers.
156+
// If the request does not come from a trusted address, NGINX will not rewrite the client IP information.
157+
// Addresses must be provided as CIDR blocks: 10.0.0.0/32, 192.33.21/0.
158+
// To trust all addresses (not recommended), set to 0.0.0.0/0.
159+
// If no addresses are provided, NGINX will not rewrite the client IP information.
160+
// Sets the set_real_ip_from directive in NGINX: https://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from.
161+
// This field is required if mode is set.
162+
TrustedAddresses []string `json:"trustedAddresses,omitempty"`
163+
164+
// SetIPRecursively configures whether recursive search is used when selecting the client's address from the X-Forwarded-For header.
165+
// It is used in conjunction with TrustedAddresses.
166+
// If enabled, NGINX will recurse on the values in X-Forwarded-Header from the end of array to start of array and select the first untrusted IP.
167+
// For example, if X-Forwarded-For is [11.11.11.11, 22.22.22.22, 55.55.55.1], and TrustedAddresses is set to 55.55.55.1/0, NGINX will rewrite the client IP to 22.22.22.22.
168+
// If disabled, NGINX will select the IP at the end of the array. In the previous example, 55.55.55.1 would be selected.
169+
//
170+
// +optional
171+
SetIPRecursively *bool `json:"setIPRecursively,omitempty"`
172+
}
173+
174+
// RewriteClientIPModeType defines how NGINX Gateway Fabric will determine the client's original IP address.
175+
// +kubebuilder:validation:Enum=ProxyProtocol;XForwardedFor
176+
type RewriteClientIPModeType string
177+
178+
const (
179+
// RewriteClientIPModeProxyProtocol configures NGINX to accept PROXY protocol and set the client's IP address to the IP address in the PROXY protocol header.
180+
// Sets the proxy_protocol parameter to the listen directive on all servers, and sets real_ip_header to proxy_protocol: https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header.
181+
RewriteClientIPModeProxyProtocol RewriteClientIPModeType = "ProxyProtocol"
182+
183+
// RewriteClientIPModeXForwardedFor configures NGINX to set the client's IP address to the IP address in the X-Forwarded-For HTTP header.
184+
// Sets real_ip_header to XForwardedFor: https://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header.
185+
RewriteClientIPModeXForwardedFor RewriteClientIPModeType = "XForwardedFor"
186+
)
187+
188+
```
189+
190+
The benefits of this API are:
191+
192+
1. Allow users to easily configure the two most common methods for rewriting the client IP address.
193+
2. Express configuration in terms of use cases.
194+
3. Secure by default. By requiring that TrustedAddresses is set, users will have to explicitly enable addresses to trust.
195+
4. Require minimal knowledge of NGINX configuration while still correlating fields to NGINX directives or behavior.
196+
5. Allow for extension to support other methods of rewriting IP addresses in the future.
197+
198+
### Validation
199+
200+
The Go API above does not contain validation annotation. Annotations should be added to enforce the following rules.
201+
202+
- If `mode` is set, then `trustedAddresses` is required.
203+
- `trustedAddresses` can have up to 16 CIDR blocks.
204+
- `trustedAddresses` must be in CIDR block notation.
205+
206+
### Status
207+
208+
Status is set on the GatewayClass, not the `NginxProxy` resource. If the `NginxProxy` is invalid, set the `Accepted` condition reason on the GatewayClass to `InvalidParameters` but still mark `Accepted` as `True`. See [gateway settings proposal](gateway-settings.md#status) for more details on status.
209+
210+
### Future Work
211+
212+
- If requested by a user, add more `RewriteClientIPModes`, such as custom header or Forwarded.
213+
- Allow users to rate limit or apply security policies using the value of `$remote_addr`.
214+
- The `set_real_ip_from` directive accepts IP addresses, CIDR blocks, hostnames and the special value `unix:;` which trusts all UNIX-domain sockets. For simplicity, we will begin by only allowed CIDR blocks since this will cover most use cases. However, if a user requests it, we can extend the TrustedAddresses field to accept other types of addresses.
215+
216+
## Use Cases
217+
218+
- As a Cluster Operator, I want to configure NGINX to rewrite the client's IP address using PROXY protocol for all applications associated with the GatewayClass.
219+
- As a Cluster Operator, I want to configure NGINX to rewrite the client's IP address using the X-Forwarded-For header for all applications associated with the GatewayClass.
220+
221+
## Testing
222+
223+
- Unit tests
224+
- Functional tests that verify the attachment of the CRD to the GatewayClass, and that NGINX behaves properly based on the configuration. This includes verifying client IP is propagated to the backends.
225+
226+
## Security Considerations
227+
228+
Validating all fields in the `NginxProxy` is critical to ensuring that the NGINX config generated by NGINX Gateway Fabric is correct and secure.
229+
230+
All fields in the `NginxProxy` will be validated with Open API Schema. If the Open API Schema validation rules are not sufficient, we will use [CEL](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#validation-rules).
231+
232+
RBAC via the Kubernetes API server will ensure that only authorized users can update the CRD.
233+
234+
## Alternatives
235+
236+
### API that maps to NGINX directives
237+
238+
Rather than create an expressive API with use-case driven language, we can simply expose the NGINX directives:
239+
240+
```go
241+
type NginxProxySpec struct {
242+
// RewriteClientIP contains configuration for rewriting the client IP to the original client's IP.
243+
// +optional
244+
RewriteClientIP *RewriteClientIP `json:"rewriteClientIP,omitempty"`
245+
}
246+
247+
// RewriteClientIP specifies
248+
type RewriteClientIP struct {
249+
EnableProxyProtocol *bool `json:"rewriteClientIP,omitempty"`
250+
SetRealIPFrom []string `json:"setRealIPFrom,omitempty"`
251+
RealIPHeader string `json:"realIPHeader,omitempty"`
252+
RealIPRecursive *bool `json:"realIPRecursive,omitempty"`
253+
}
254+
```
255+
256+
then, we could add user documentation describing how to implement the two most common use cases:
257+
258+
1. PROXY protocol:
259+
260+
```yaml
261+
apiVersion: gateway.nginx.org/v1alpha1
262+
kind: NginxProxy
263+
metadata:
264+
name: proxy-protocol
265+
spec:
266+
rewriteClientIP:
267+
enableProxyProtocol: true
268+
setRealIPFrom:
269+
- 0.0.0.0/0
270+
realIPHeader: proxy_protocol
271+
```
272+
273+
2. X-Forwarded-For
274+
275+
```yaml
276+
apiVersion: gateway.nginx.org/v1alpha1
277+
kind: NginxProxy
278+
metadata:
279+
name: x-forwarded-for
280+
spec:
281+
rewriteClientIP:
282+
setRealIPFrom:
283+
- 0.0.0.0/0
284+
realIPHeader: x-forwarded-for
285+
realIPRecursive: true
286+
```
287+
288+
The benefit to this approach is that it gives the users the full power of the NGINX real IP module and immediately allows them to configure all methods to rewrite a client's IP address.
289+
This is ideal for users that are familiar with NGINX configuration and know what these NGINX directives are and how they work. However, for non-NGINX users or NGINX newcomers, this API would be more challenging to use.
290+
Without reading the documentation, it would be difficult to figure out how to configure PROXY protocol. Additionally, there's a higher chance of misconfiguration since each use case requires three fields to be in sync with each other.
291+
292+
## References
293+
294+
- [NGINX Extensions Enhancement Proposal](nginx-extensions.md)
295+
- [Attaching Policy to GatewayClass](https://gateway-api.sigs.k8s.io/geps/gep-713/#attaching-policy-to-gatewayclass)
296+
- [Kubernetes API Conventions](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md)
297+
- [Gateway Settings (NginxProxy CRD)](gateway-settings.md)

0 commit comments

Comments
 (0)