[Bug]: Missing vary caching header

[MrKaplan-lw]

Requirements

• This is a bug report, and if not, please post to https://lemmy.ml/c/lemmy_support instead.
• Please check to see if this issue already exists.
• It's a single bug. Do not report multiple bugs in one issue.
• It's a frontend issue, not a backend issue; Otherwise please create an issue on the backend repo instead.

Summary

Lemmy has various endpoints used for AP requests and also by browsers.
Some of these endpoints have overlapping URLs and are also considered cacheable, including headers like cache-control: public, max-age=60.
Lemmy does not currently include the accept header in the list of headers returned in the Vary header, which would inform caches to treat requests by browsers different form requests by ActivityPub clients.
This can lead to cache confusion, where a cache server may serve HTML to ActivityPub clients or activities to web browsers.

Steps to Reproduce

1. Set up Lemmy with a cache in front of it
2. Issue request with ActivityPub accept header to prime cache
3. Issue request without ActivityPub accept header
4. See JSON returned

Technical Details

https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Vary
https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/Caching#vary

Some caches, most prominently Cloudflare, do not support the Vary header, which means that these overlapping URLs are not cacheable by those caches.

curl -v -o /dev/null https://lemmy.ml
> GET / HTTP/2
> Host: lemmy.ml
> accept: application/activity+json, application/ld+json
...
< HTTP/2 200 
< content-type: text/html; charset=utf-8
< cache-control: public, max-age=60

No vary header is returned.

Lemmy-ansible currently works around this by explicitly including the accept header in the cache key for nginx.

Related:

• [Bug]: Incorrect vary caching header lemmy#5632
• Add setting to mark paths varying by accept header as uncacheable #3100
• Add setting to mark paths varying by accept header as uncacheable lemmy#5633

Lemmy Instance Version

0.19.11

Lemmy Instance URL

No response

[dessalines]

Here's the only place we insert headers in the back end: https://github.com/LemmyNet/lemmy/blob/main/crates/routes/src/middleware/session.rs#L83

[MrKaplan-lw]

in the backend, it seems that this may be coming from the Compress middleware: https://docs.rs/actix-web/latest/actix_web/middleware/struct.Compress.html