Skip to content

[receiver/sqlserver] add scraper to record the currently executing query in mssql #38632

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 16 commits into from
Mar 28, 2025

Conversation

cuichenli
Copy link
Contributor

Co-authored-by: @sincejune (CLA signed)

This PR is associated with #37958 Since the other PR is still under review, we created this one separately to accelerate the review process. Reviewers who have completed the review of the previous PR can prioritize this one.

Description

We introduced Query Sample collection in this PR. The scraper will record all the currently executing queries once (in most case) and report related metrics. This enables users to monitor executed queries and correlate them with Top Query data for deeper insights into troubleshooting and performance optimization.

Configuration

We introduced one configuration for the feature(see receiver's README for details):

  1. max_sample_query_count: this one is also added in the other PR. In this change, it is mainly used to configure the cache size for the reported samples. It would report the sample only if the query (identified by queryHash and queryPlanHash) is not in the cache

New Log Attributes

  • sqlserver.db_name
  • sqlserver.client_address
  • sqlserver.client_port
  • sqlserver.query_start
  • sqlserver.session_id
  • sqlserver.session_status
  • sqlserver.request_status
  • sqlserver.host_name
  • sqlserver.command
  • db.query.text
  • sqlserver.blocking_session_id
  • sqlserver.wait_type
  • sqlserver.wait_time
  • sqlserver.wait_resource
  • sqlserver.open_transaction_count
  • sqlserver.transaction_id
  • sqlserver.percent_complete
  • sqlserver.estimated_completion_time
  • sqlserver.cpu_time
  • sqlserver.total_elapsed_time
  • sqlserver.reads
  • sqlserver.writes
  • sqlserver.logical_reads
  • sqlserver.transaction_isolation_level
  • sqlserver.lock_timeout
  • sqlserver.deadlock_priority
  • sqlserver.row_count
  • sqlserver.query_hash
  • sqlserver.query_plan_hash
  • sqlserver.context_info
  • sqlserver.username
  • sqlserver.wait_code
  • sqlserver.wait_category

Additional dependency

Example Output

resourceLogs:
  - resource:
      attributes:
        - key: db.system.type
          value:
            stringValue: microsoft.sql_server
    scopeLogs:
      - logRecords:
          - attributes:
              - key: sqlserver.db_name
                value:
                  stringValue: master
              - key: sqlserver.client_address
                value:
                  stringValue: "172.19.0.1"
              - key: sqlserver.client_port
                value:
                  intValue: 59286
              - key: sqlserver.query_start
                value:
                  stringValue: 2025-02-12T16:37:54.843+08:00
              - key: sqlserver.session_id
                value:
                  intValue: 60
              - key: sqlserver.session_status
                value:
                  stringValue: running
              - key: sqlserver.request_status
                value:
                  stringValue: running
              - key: sqlserver.host_name
                value:
                  stringValue: DESKTOP-GHAEGRD
              - key: sqlserver.command
                value:
                  stringValue: SELECT
              - key: db.query.text
                value:
                  stringValue: "SELECT DB_NAME ( r.database_id ), ISNULL ( c.client_net_address, ? ), ISNULL ( c.client_tcp_port, ? ), CONVERT ( NVARCHAR, TODATETIMEOFFSET ( r.start_time, DATEPART ( TZOFFSET, SYSDATETIMEOFFSET ( ) ) ), ? ), s.session_id, s.STATUS, r.STATUS, ISNULL ( s.host_name, ? ), r.command, SUBSTRING ( o.TEXT, ( r.statement_start_offset / ? ) + ? ( ( CASE r.statement_end_offset WHEN - ? THEN DATALENGTH ( o.TEXT ) ELSE r.statement_end_offset END - r.statement_start_offset ) / ? ) + ? ), r.blocking_session_id, ISNULL ( r.wait_type, ? ), r.wait_time, r.wait_resource, r.open_transaction_count, r.transaction_id, r.percent_complete, r.estimated_completion_time, r.cpu_time, r.total_elapsed_time, r.reads, r.writes, r.logical_reads, r.transaction_isolation_level, r.LOCK_TIMEOUT, r.DEADLOCK_PRIORITY, r.row_count, r.query_hash, r.query_plan_hash, ISNULL ( r.context_info, CONVERT ( VARBINARY, ? ) ), s.login_name FROM sys.dm_exec_requests r INNER JOIN sys.dm_exec_sessions s ON r.session_id = s.session_id INNER JOIN sys.dm_exec_connections c ON s.session_id = c.session_id CROSS APPLY sys.dm_exec_sql_text ( r.plan_handle )"
              - key: sqlserver.blocking_session_id
                value:
                  intValue: 0
              - key: sqlserver.wait_type
                value:
                  stringValue: ""
              - key: sqlserver.wait_time
                value:
                  intValue: 0
              - key: sqlserver.wait_resource
                value:
                  stringValue: ""
              - key: sqlserver.open_transaction_count
                value:
                  intValue: 0
              - key: sqlserver.transaction_id
                value:
                  intValue: 11089
              - key: sqlserver.percent_complete
                value:
                  doubleValue: 0
              - key: sqlserver.estimated_completion_time
                value:
                  doubleValue: 0
              - key: sqlserver.cpu_time
                value:
                  intValue: 6
              - key: sqlserver.total_elapsed_time
                value:
                  intValue: 6
              - key: sqlserver.reads
                value:
                  intValue: 0
              - key: sqlserver.writes
                value:
                  intValue: 0
              - key: sqlserver.logical_reads
                value:
                  intValue: 38
              - key: sqlserver.transaction_isolation_level
                value:
                  intValue: 2
              - key: sqlserver.lock_timeout
                value:
                  intValue: -1
              - key: sqlserver.deadlock_priority
                value:
                  intValue: 0
              - key: sqlserver.row_count
                value:
                  intValue: 1
              - key: sqlserver.query_hash
                value:
                  stringValue: "307837304133423133304231303438443444"
              - key: sqlserver.query_plan_hash
                value:
                  stringValue: "307831343032313046363442373838434239"
              - key: sqlserver.context_info
                value:
                  stringValue: ""
              - key: sqlserver.username
                value:
                  stringValue: sa
              - key: sqlserver.wait_code
                value:
                  intValue: 0
              - key: sqlserver.wait_category
                value:
                  stringValue: Unknown
            body:
              stringValue: sample
        scope:
          name: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/sqlserverreceiver
          version: v0.0.1

Link to tracking issue

Part of #36462

Testing

Added

Documentation

Updated

@atoulme
Copy link
Contributor

atoulme commented Mar 18, 2025

Please resolve conflict and build issues.

@atoulme atoulme marked this pull request as draft March 18, 2025 19:03
@cuichenli cuichenli marked this pull request as ready for review March 19, 2025 01:50
@cuichenli
Copy link
Contributor Author

the other pr has been merged, so i updated it accordingly and did a force push

@cuichenli cuichenli requested a review from crobert-1 as a code owner March 25, 2025 02:47
@github-actions github-actions bot added the Run Windows Enable running windows test on a PR label Mar 25, 2025
@cuichenli
Copy link
Contributor Author

@crobert-1 i have updated based on the comments. please take a look. thanks!

Copy link
Member

@crobert-1 crobert-1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One minor request, looks good otherwise!

Copy link
Member

@crobert-1 crobert-1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, thanks for all your work here @cuichenli!

@mx-psi mx-psi merged commit 6223d55 into open-telemetry:main Mar 28, 2025
204 checks passed
@github-actions github-actions bot added this to the next release milestone Mar 28, 2025
dmathieu pushed a commit to dmathieu/opentelemetry-collector-contrib that referenced this pull request Apr 8, 2025
…ery in mssql (open-telemetry#38632)

Co-authored-by:  @sincejune (CLA signed)

This PR is associated with
open-telemetry#37958
Since the other PR is still under review, we created this one separately
to accelerate the review process. Reviewers who have completed the
review of the previous PR can prioritize this one.

#### Description

We introduced Query Sample collection in this PR. The scraper will
record all the currently executing queries once (in most case) and
report related metrics. This enables users to monitor executed queries
and correlate them with Top Query data for deeper insights into
troubleshooting and performance optimization.

#### Configuration
We introduced one configuration for the feature(see receiver's README
for details):
1. max_sample_query_count: this one is also added in the other PR. In
this change, it is mainly used to configure the cache size for the
reported samples. It would report the sample only if the query
(identified by queryHash and queryPlanHash) is not in the cache

#### New Log Attributes

- `sqlserver.db_name`
- `sqlserver.client_address`
- `sqlserver.client_port`
- `sqlserver.query_start`
- `sqlserver.session_id`
- `sqlserver.session_status`
- `sqlserver.request_status`
- `sqlserver.host_name`
- `sqlserver.command`
- `db.query.text`
- `sqlserver.blocking_session_id`
- `sqlserver.wait_type`
- `sqlserver.wait_time`
- `sqlserver.wait_resource`
- `sqlserver.open_transaction_count`
- `sqlserver.transaction_id`
- `sqlserver.percent_complete`
- `sqlserver.estimated_completion_time`
- `sqlserver.cpu_time`
- `sqlserver.total_elapsed_time`
- `sqlserver.reads`
- `sqlserver.writes`
- `sqlserver.logical_reads`
- `sqlserver.transaction_isolation_level`
- `sqlserver.lock_timeout`
- `sqlserver.deadlock_priority`
- `sqlserver.row_count`
- `sqlserver.query_hash`
- `sqlserver.query_plan_hash`
- `sqlserver.context_info`
- `sqlserver.username`
- `sqlserver.wait_code`
- `sqlserver.wait_category`

#### Additional dependency
- hashicorp/golang-lru/v2
	- License: MPL-2.0
- Link:
[pkg.go.dev/github.com/hashicorp/golang-lru/v2](https://pkg.go.dev/github.com/hashicorp/golang-lru/v2)
	- Already been used in the repo
- DataDog/datadog-agent/pkg/obfuscate
	- License: Apache 2.0
- Link:
[pkg.go.dev/github.com/DataDog/datadog-agent/pkg/obfuscate](https://pkg.go.dev/github.com/DataDog/datadog-agent/pkg/obfuscate)
	- Already been used in the repo

#### Example Output
```yaml
resourceLogs:
  - resource:
      attributes:
        - key: db.system.type
          value:
            stringValue: microsoft.sql_server
    scopeLogs:
      - logRecords:
          - attributes:
              - key: sqlserver.db_name
                value:
                  stringValue: master
              - key: sqlserver.client_address
                value:
                  stringValue: "172.19.0.1"
              - key: sqlserver.client_port
                value:
                  intValue: 59286
              - key: sqlserver.query_start
                value:
                  stringValue: 2025-02-12T16:37:54.843+08:00
              - key: sqlserver.session_id
                value:
                  intValue: 60
              - key: sqlserver.session_status
                value:
                  stringValue: running
              - key: sqlserver.request_status
                value:
                  stringValue: running
              - key: sqlserver.host_name
                value:
                  stringValue: DESKTOP-GHAEGRD
              - key: sqlserver.command
                value:
                  stringValue: SELECT
              - key: db.query.text
                value:
                  stringValue: "SELECT DB_NAME ( r.database_id ), ISNULL ( c.client_net_address, ? ), ISNULL ( c.client_tcp_port, ? ), CONVERT ( NVARCHAR, TODATETIMEOFFSET ( r.start_time, DATEPART ( TZOFFSET, SYSDATETIMEOFFSET ( ) ) ), ? ), s.session_id, s.STATUS, r.STATUS, ISNULL ( s.host_name, ? ), r.command, SUBSTRING ( o.TEXT, ( r.statement_start_offset / ? ) + ? ( ( CASE r.statement_end_offset WHEN - ? THEN DATALENGTH ( o.TEXT ) ELSE r.statement_end_offset END - r.statement_start_offset ) / ? ) + ? ), r.blocking_session_id, ISNULL ( r.wait_type, ? ), r.wait_time, r.wait_resource, r.open_transaction_count, r.transaction_id, r.percent_complete, r.estimated_completion_time, r.cpu_time, r.total_elapsed_time, r.reads, r.writes, r.logical_reads, r.transaction_isolation_level, r.LOCK_TIMEOUT, r.DEADLOCK_PRIORITY, r.row_count, r.query_hash, r.query_plan_hash, ISNULL ( r.context_info, CONVERT ( VARBINARY, ? ) ), s.login_name FROM sys.dm_exec_requests r INNER JOIN sys.dm_exec_sessions s ON r.session_id = s.session_id INNER JOIN sys.dm_exec_connections c ON s.session_id = c.session_id CROSS APPLY sys.dm_exec_sql_text ( r.plan_handle )"
              - key: sqlserver.blocking_session_id
                value:
                  intValue: 0
              - key: sqlserver.wait_type
                value:
                  stringValue: ""
              - key: sqlserver.wait_time
                value:
                  intValue: 0
              - key: sqlserver.wait_resource
                value:
                  stringValue: ""
              - key: sqlserver.open_transaction_count
                value:
                  intValue: 0
              - key: sqlserver.transaction_id
                value:
                  intValue: 11089
              - key: sqlserver.percent_complete
                value:
                  doubleValue: 0
              - key: sqlserver.estimated_completion_time
                value:
                  doubleValue: 0
              - key: sqlserver.cpu_time
                value:
                  intValue: 6
              - key: sqlserver.total_elapsed_time
                value:
                  intValue: 6
              - key: sqlserver.reads
                value:
                  intValue: 0
              - key: sqlserver.writes
                value:
                  intValue: 0
              - key: sqlserver.logical_reads
                value:
                  intValue: 38
              - key: sqlserver.transaction_isolation_level
                value:
                  intValue: 2
              - key: sqlserver.lock_timeout
                value:
                  intValue: -1
              - key: sqlserver.deadlock_priority
                value:
                  intValue: 0
              - key: sqlserver.row_count
                value:
                  intValue: 1
              - key: sqlserver.query_hash
                value:
                  stringValue: "307837304133423133304231303438443444"
              - key: sqlserver.query_plan_hash
                value:
                  stringValue: "307831343032313046363442373838434239"
              - key: sqlserver.context_info
                value:
                  stringValue: ""
              - key: sqlserver.username
                value:
                  stringValue: sa
              - key: sqlserver.wait_code
                value:
                  intValue: 0
              - key: sqlserver.wait_category
                value:
                  stringValue: Unknown
            body:
              stringValue: sample
        scope:
          name: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/sqlserverreceiver
          version: v0.0.1

```

#### Link to tracking issue
Part of
open-telemetry#36462

#### Testing
Added

#### Documentation
Updated

---------

Co-authored-by: Chao Weng <[email protected]>
Fiery-Fenix pushed a commit to Fiery-Fenix/opentelemetry-collector-contrib that referenced this pull request Apr 24, 2025
…ery in mssql (open-telemetry#38632)

Co-authored-by:  @sincejune (CLA signed)

This PR is associated with
open-telemetry#37958
Since the other PR is still under review, we created this one separately
to accelerate the review process. Reviewers who have completed the
review of the previous PR can prioritize this one.

#### Description

We introduced Query Sample collection in this PR. The scraper will
record all the currently executing queries once (in most case) and
report related metrics. This enables users to monitor executed queries
and correlate them with Top Query data for deeper insights into
troubleshooting and performance optimization.

#### Configuration
We introduced one configuration for the feature(see receiver's README
for details):
1. max_sample_query_count: this one is also added in the other PR. In
this change, it is mainly used to configure the cache size for the
reported samples. It would report the sample only if the query
(identified by queryHash and queryPlanHash) is not in the cache

#### New Log Attributes

- `sqlserver.db_name`
- `sqlserver.client_address`
- `sqlserver.client_port`
- `sqlserver.query_start`
- `sqlserver.session_id`
- `sqlserver.session_status`
- `sqlserver.request_status`
- `sqlserver.host_name`
- `sqlserver.command`
- `db.query.text`
- `sqlserver.blocking_session_id`
- `sqlserver.wait_type`
- `sqlserver.wait_time`
- `sqlserver.wait_resource`
- `sqlserver.open_transaction_count`
- `sqlserver.transaction_id`
- `sqlserver.percent_complete`
- `sqlserver.estimated_completion_time`
- `sqlserver.cpu_time`
- `sqlserver.total_elapsed_time`
- `sqlserver.reads`
- `sqlserver.writes`
- `sqlserver.logical_reads`
- `sqlserver.transaction_isolation_level`
- `sqlserver.lock_timeout`
- `sqlserver.deadlock_priority`
- `sqlserver.row_count`
- `sqlserver.query_hash`
- `sqlserver.query_plan_hash`
- `sqlserver.context_info`
- `sqlserver.username`
- `sqlserver.wait_code`
- `sqlserver.wait_category`

#### Additional dependency
- hashicorp/golang-lru/v2
	- License: MPL-2.0
- Link:
[pkg.go.dev/github.com/hashicorp/golang-lru/v2](https://pkg.go.dev/github.com/hashicorp/golang-lru/v2)
	- Already been used in the repo
- DataDog/datadog-agent/pkg/obfuscate
	- License: Apache 2.0
- Link:
[pkg.go.dev/github.com/DataDog/datadog-agent/pkg/obfuscate](https://pkg.go.dev/github.com/DataDog/datadog-agent/pkg/obfuscate)
	- Already been used in the repo

#### Example Output
```yaml
resourceLogs:
  - resource:
      attributes:
        - key: db.system.type
          value:
            stringValue: microsoft.sql_server
    scopeLogs:
      - logRecords:
          - attributes:
              - key: sqlserver.db_name
                value:
                  stringValue: master
              - key: sqlserver.client_address
                value:
                  stringValue: "172.19.0.1"
              - key: sqlserver.client_port
                value:
                  intValue: 59286
              - key: sqlserver.query_start
                value:
                  stringValue: 2025-02-12T16:37:54.843+08:00
              - key: sqlserver.session_id
                value:
                  intValue: 60
              - key: sqlserver.session_status
                value:
                  stringValue: running
              - key: sqlserver.request_status
                value:
                  stringValue: running
              - key: sqlserver.host_name
                value:
                  stringValue: DESKTOP-GHAEGRD
              - key: sqlserver.command
                value:
                  stringValue: SELECT
              - key: db.query.text
                value:
                  stringValue: "SELECT DB_NAME ( r.database_id ), ISNULL ( c.client_net_address, ? ), ISNULL ( c.client_tcp_port, ? ), CONVERT ( NVARCHAR, TODATETIMEOFFSET ( r.start_time, DATEPART ( TZOFFSET, SYSDATETIMEOFFSET ( ) ) ), ? ), s.session_id, s.STATUS, r.STATUS, ISNULL ( s.host_name, ? ), r.command, SUBSTRING ( o.TEXT, ( r.statement_start_offset / ? ) + ? ( ( CASE r.statement_end_offset WHEN - ? THEN DATALENGTH ( o.TEXT ) ELSE r.statement_end_offset END - r.statement_start_offset ) / ? ) + ? ), r.blocking_session_id, ISNULL ( r.wait_type, ? ), r.wait_time, r.wait_resource, r.open_transaction_count, r.transaction_id, r.percent_complete, r.estimated_completion_time, r.cpu_time, r.total_elapsed_time, r.reads, r.writes, r.logical_reads, r.transaction_isolation_level, r.LOCK_TIMEOUT, r.DEADLOCK_PRIORITY, r.row_count, r.query_hash, r.query_plan_hash, ISNULL ( r.context_info, CONVERT ( VARBINARY, ? ) ), s.login_name FROM sys.dm_exec_requests r INNER JOIN sys.dm_exec_sessions s ON r.session_id = s.session_id INNER JOIN sys.dm_exec_connections c ON s.session_id = c.session_id CROSS APPLY sys.dm_exec_sql_text ( r.plan_handle )"
              - key: sqlserver.blocking_session_id
                value:
                  intValue: 0
              - key: sqlserver.wait_type
                value:
                  stringValue: ""
              - key: sqlserver.wait_time
                value:
                  intValue: 0
              - key: sqlserver.wait_resource
                value:
                  stringValue: ""
              - key: sqlserver.open_transaction_count
                value:
                  intValue: 0
              - key: sqlserver.transaction_id
                value:
                  intValue: 11089
              - key: sqlserver.percent_complete
                value:
                  doubleValue: 0
              - key: sqlserver.estimated_completion_time
                value:
                  doubleValue: 0
              - key: sqlserver.cpu_time
                value:
                  intValue: 6
              - key: sqlserver.total_elapsed_time
                value:
                  intValue: 6
              - key: sqlserver.reads
                value:
                  intValue: 0
              - key: sqlserver.writes
                value:
                  intValue: 0
              - key: sqlserver.logical_reads
                value:
                  intValue: 38
              - key: sqlserver.transaction_isolation_level
                value:
                  intValue: 2
              - key: sqlserver.lock_timeout
                value:
                  intValue: -1
              - key: sqlserver.deadlock_priority
                value:
                  intValue: 0
              - key: sqlserver.row_count
                value:
                  intValue: 1
              - key: sqlserver.query_hash
                value:
                  stringValue: "307837304133423133304231303438443444"
              - key: sqlserver.query_plan_hash
                value:
                  stringValue: "307831343032313046363442373838434239"
              - key: sqlserver.context_info
                value:
                  stringValue: ""
              - key: sqlserver.username
                value:
                  stringValue: sa
              - key: sqlserver.wait_code
                value:
                  intValue: 0
              - key: sqlserver.wait_category
                value:
                  stringValue: Unknown
            body:
              stringValue: sample
        scope:
          name: github.com/open-telemetry/opentelemetry-collector-contrib/receiver/sqlserverreceiver
          version: v0.0.1

```

#### Link to tracking issue
Part of
open-telemetry#36462

#### Testing
Added

#### Documentation
Updated

---------

Co-authored-by: Chao Weng <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants