[RFC] New Discover experience for Observability Workspaces

[angle943]

New Discover Experience for Observability Workspaces

The OpenSearch Dashboard(OSD) team is working on vastly improving the Discover experience in 2025 for Observability Workspaces. We want OSD to be the tool users reach for when they want to dive deep into logs. To achieve this goal, we are making several changes:

Make PPL the default language

While Dashboard Query Language (DQL) has been a great experience for many users in discovering their OpenSearch data, it is not compatible with exploring data outside of OpenSearch indexes. Therefore, one of the decisions we are taking for observability workspaces is that we are going to make Piped Processing Language (PPL) the default query language for Discover. We are also expanding our PPL language support to include commands such as JOIN, LOOKUP, and SUBQUERY operations. This decision also opens up other means for users to explore their data, such as...

Aggregated Queries will generate Visualizations

One powerful feature PPL gives us is to calculate an aggregation from a search result, such as using the stats command. But instead of viewing these results in a table format, OSD will auto-detect an aggregated query and generate sensible visualizations such as a bar chart or line chart, depending on the query. These visualizations will be customizable by the user, and even be able to switch the visualization type if needed. And these visualizations will be able to be saved to the dashboard.

The way we are going to achieve is is by having separate tabs within the Discover page. We will have a "logs" tab that will work almost identically as our current Discover experience. We will create a "visualization" tab that will show you the associated visualization based on your query. If users want to see the underlying logs that makes up the visualization, they can switch to the logs tab.

Curated experience for log patterns

Another feature that PPL provides are patterns. Users when digging into logs will often times want to find patterns in their results, and to enable a seamless experience for this we are building a third tab called "Log Patterns". When a user switches from the Logs tab or Visualize tab and goes to the log pattern tab, OSD will automatically fire a pattern query and display the results in a table format that we are building exclusively for log patterns.

Query-Driven Experience

To achieve a seamless and consistent experience to the users, we are going with what we are calling a "query driven experience." The query will now be the source of truth for the results that are displayed, including things such as filters, sort, and displayed columns. For example, adding a filter for a specific value by selecting it from our left filter panel will now update the query string.

UX tailored for data discovery and consumption

We will also be making several changes to the UI/UX, including big changes to the query editor as well as the results table. Our goal will be to make the UI intuitive to all users as well as optimize the space for users to be able to dive into the data with minimal UI disturbances that deter from that experience.

How to enable the new experience

Run yarn start:explore and open an observability workspace to see our current development!

Questions and Answers

When will this experience launch?

We are currently in active development, but we are planning on releasing a working build of this new experience in mid July 2025. After this launch, we will continue on developing features for this experience.

Why only observability workspace?

We are currently only focusing on observability workspace, but in the future we do have plans to provide a tailored experience to other workspace type, such as Security Analytics.

What if I want to continue using the old Discover experience?

We will have a mean for users to switch between the new experience and the "classic" experience.