gRPC Storage Service

[VineethReddy02]

What this PR does:
This PR contains gRPC service client i.e

1. Index Client
2. Storage Client
3. Table Client

This is still in work in progress PR this needs testing with gRPC storage server implementation

Which issue(s) this PR fixes:
Fixes #1681

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[gouthamve]

Looks good for the first attempt! Left some cursory feedback, lets iterate on this!

[gouthamve]

One final comment and then we're good to go!

[VineethReddy02]

Thanks @pracucci for taking the time & reviewing. I made all the suggested changes.
Looking forward in getting this merged. :)

[pracucci]

LGTM! I just left a couple of minor nits and then we should good to go.

[pstibrany]

Since this API is supposedly to be used for writing 3rd party storage implementations, I think it lacks documentation (at least inside grpc.proto file, but more generic documentation about how Cortex uses the storage and what should the user of this API reasonably expect).

Is there any implementation using this API? How do we know it satisfies the needs of external storage client?

(I am also skeptical about including this in Cortex in the first place. Many failure modes related to gRPC (grpc server not running, big responses, bad config, ...) would be automatically removed if client was included in the Cortex)

[gouthamve]

I think it lacks documentation

Absolutely. We still have follow up work to document and give a reference in-memory implementation. And to use the in-mem implementation to write an integration test.

would be automatically removed if client was included in the Cortex

What do you mean?

[pstibrany]

Absolutely. We still have follow up work to document and give a reference in-memory implementation. And to use the in-mem implementation to write an integration test.

I think we need real implementation or two, not just in-mem one.

would be automatically removed if client was included in the Cortex

What do you mean?

I was talking about failure modes related to gRPC -- which wouldn't be a problem if client was in Cortex directly, and not using gRPC (well, obviously). Just saying that using gRPC adds more complexity.

[gouthamve]

I kinda disagree :) I see gRPC having the same amount of failure modes as a remote Bigtable/Cassandra. They are both endpoints.

grpc server not running, big responses, bad config, ...

We faced these with Bigtable tbh, specially the big responses. We had to up our gRPC message limit with a previous schema, hence came up with the v9/v10 schema. I would say gRPC makes the failure modes more likely, but doesn't introduce new ones.

And while I agree that having the client directly in Cortex would be the best, I am not super enthused about a ton of storage clients in the tree.

[gouthamve]

I think we need real implementation or two, not just in-mem one.

The first one was going to be Postgres/MySQL so Loki could use it as the index. Not sure what the next one would be tbh.

[pstibrany]

And while I agree that having the client directly in Cortex would be the best, I am not super enthused about a ton of storage clients in the tree.

I personally think that biggest issue in people contributing more clients is lack of good documentation, and pointers about how to start.

Anyway, please feel free to merge when PR is ready, my comments are not merge-blockers.

[pracucci]

Let's gooo 🚀 ! I will merge as soon as the CI passes