[RFC] Add A Resource API in Query Enhancements

[kathyguo18]

Background

The query enhancements plugin provides a query API for executing queries using PPL, SQL, and other languages. Currently, there is no standardized method to retrieve and manage resources, such as metadata, resource labels, rules, associated with these queries. This RFC proposes an approach to implement a generic Resource API in the OpenSearch Dashboards query enhancements plugin to retrieve and manage resources across different backend systems.

Problem Statement

Currently, there is no standardized method to retrieve resources. We need to build a generic way that can retrieve and manage resources supported by backend, from any component/plugin. This should allow filtering and pagination. The API response should not be significantly larger than backend response, and should preserve all information from the backend response. It should be extensible and backward compatible when we add more resource types, datasource types, UI components.

Proposed Approach

Expose a resources API in Nodejs server

We can expose APIs in Nodejs server that acts as a proxy to call the underlying SQL resources APIs. Below is a diagram using prometheus as an example

1. Create APIs in Nodejs server that acts as a proxy to call backend resources APIs

Sample POST API

POST /api/enhancements/resources
{
  "connection": {
    "id": "string",
    "type": "string"
  },
  "resource": {
    "name": "string", // optional
    "type": "string"
  },
  "content": {} // arbitrary JSON
}

Sample GET API

const route = {
  path: "/api/enhancements/resources/{connectionType}/{connectionId}/{resourceType}/{resourceName?}",
  validate: {
    params: schema.object({
      connectionType: schema.string(),
      connectionId: schema.string(),
      resourceType: schema.string(),
      resourceName: schema.maybe(schema.string()),
    }),
    query: schema.object({}, { unknowns: "allow" }),
  },
};

2. Allow register BaseConnectionManager implementations through a ResourceManagerService in query enhancements plugin.

export abstract class BaseConnectionManager {
  constructor() {}

  abstract handlePostRequest(
    context: RequestHandlerContext,
    request: OpenSearchDashboardsRequest
  ): Promise<HttpResponsePayload>;

  // ... methods for GET, PUT, DELETE
}

This creates shared resources APIs that plugins will use to retrieve resources like metadata, labels, schema, call mutating actions, etc. The API will be responsible to call the underlying client. It does minimal transformation in the client response to provide flexibility for the frontend component.

Plugins to use this API will register a connectionManager extending the abstract class BaseConnectionManager to handle one type of resource retrieval. This approach enables plugins to define their own logic to retrieve resources from external datasources, and expose them to other plugins in OSD through a single API.

Pros

• simpler architecture
• better flexibility in supporting different plugins: they interact with resources using a REST API and schema defined by underlying datasource, which is an existing/common contract

Cons

• less code reusability: if multiple plugins use the same logic (to convert schema, handle errors), they need to implement the logic individually

Alternative approach

Create a resource interceptor and resource strategy abstraction layer

We can follow similar design of the existing search interceptor and search strategy to provide resources retrieval from the underlying datasource.

The search interceptor provides a common interface for managing search requests (search, abort, pending, error), and the search strategy provides a common interface for making the search request to the underlying datasource. We can create a resource interceptor that handles the common operations of resource requests.

Pros

• more uniformed approach to handle resource requests (can have shared transformation logic, error handling logic, etc)

Cons

• added complexity
• small abstraction benefit due to the diverse nature of different resource requests

Recommendation

The proposed option is preferred because

Query APIs have a known output structure, where the output always contains the field schema and field values. But for resources, due to the diverse resource types and plugin integrations we may add in the future, it is not as useful to create the abstraction level to account for the different use cases. The proposed option keeps the use case flexible.

The current implementation of the Resource API POST method is available for review in OpenSearch Dashboards Pull Request #9770. We will develop GET, PUT, and DELETE methods to expand API functionality later.

Note on API methods

Ideally we should follow REST API conventions and use GET to get resources, but due to certain get/search requests can be very large, we'll expose all methods but let the underlying datasource manager decide what to do. This has been discussed in #9731 (comment).

For example, datasources can strictly follow REST API conventions, or they can allow both GET and POST methods to retrieve resources if necessary. The latter is aligned with Prometheus and OpenSearch API behavior. See #9731 and #9770 for relevant discussions.

[joshuali925]

Thanks @kathyguo18 for adding this, I edited the issue to add diagrams, alternate approach and API methods we discussed about