7.64.3

Agent

Prelude

Release on: 2025-04-10

Enhancement Notes

• Agents are now built with Go 1.23.8.

Bug Fixes

• Add a configuration option, enable_nvml_detection, disabled by default, to stop the Agent from trying to find the NVML library on startup.
• Fix remote tagger initialization log message to be logged only once.

Datadog Cluster Agent

Prelude

Released on: 2025-04-10 Pinned to datadog-agent v7.64.3: CHANGELOG.

7.64.2

Agent

Prelude

Release on: 2025-04-02

• Please refer to the 7.64.2 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• Process Agent files are added to the flare archive instead of displaying "no session token provided".

Datadog Cluster Agent

Prelude

Released on: 2025-04-02 Pinned to datadog-agent v7.64.2: CHANGELOG.

7.64.1

Agent

Prelude

Release on: 2025-03-20

• Please refer to the 7.64.1 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• Fixed Python package dependency issue preventing integrations from loading on Windows

Datadog Cluster Agent

Prelude

Released on: 2025-03-20 Pinned to datadog-agent v7.64.1: CHANGELOG

7.64.0

Agent

Known Issues

This version contains a Python package dependency (cryptography) issue that can prevent Python integrations from loading on Windows. A workaround is to set the environment variable CRYPTOGRAPHY_OPENSSL_NO_LEGACY=1 at the machine level and restart the Agent. If the issue persists, the recommendation at this time is to downgrade to Agent v7.63.3 or upgrade to v7.64.1 when it becomes available.

Prelude

Release on: 2025-03-19

• Please refer to the 7.64.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• Add the new following docker images, containing every components of the Datadog Agent, including the new Otel-Agent:
  • 7.X.Y-full
  • 7-full
  • latest-full
• The datadog-fips-agent now builds with the 'requirefips' buildtag in the 7.64 Linux release and no longer needs the GOFIPS=1 environment variable set to enable FIPS mode. When upgrading from 7.63, this can be removed.
• Enables IMDSv2 by default on all EC2 instance hosts by updating theec2_imdsv2_transition_payload_enabled flag from false to true. If IMDSv2 hasn’t been explicitly enabled and the hostname isn’t set to the instance ID, the display name may change to the instance ID without affecting Agent behavior. For more information, see the IMDSv2 Enablement by Default documentation.
• Bump the Python version to 3.12.9
• ECS task collection is now enabled by default (see ecs_task_collection_enabled in the datadog.yaml configuration).

New Features

• Added support for obfuscation for valkey command. This feature is enabled by default. To disable it, set DD_APM_OBFUSCATION_VALKEY_ENABLED=false. To replace all valkey command arguments with a single ?, set DD_APM_OBFUSCATION_VALKEY_REMOVE_ALL_ARGS=true (default: false).
• APM: Add support for multi-region failover. This feature is controlled via Remote Config and allows the trace-agent to switch to a failover data center when enabled.
• Add new card: common field to DogStatsD Datagram specification to allow customer to specify the cardinality of the metric. This field is optional.
• Log Agent now officially supports http2 transport to proxy.

Enhancement Notes

• Add per-process GPU tagging when running process checks on the Core Agent.
• Enable pymem.inuse metric when Agent telemetry is enabled
• In rare cases, when the Agent's Network Performance Monitor is enabled and the Agent is identified as contributing to a "Blue Screen of Death" (BSOD) event, Agent telemetry is used to generate a payload that includes Agent's driver code offset for further analysis.
• Adds a flag for setting the inactivity timeout and adds support for "check name" for the analyze logs subcommand
• Agents are now built with Go 1.23.6.
• Added a new feature flag disable_receive_resource_spans_v2 in DD_APM_FEATURES that replaces enable_receive_resource_spans_v2 - the refactored implementation of ReceiveResourceSpans for OTLP is now opt-out instead of opt-in.
• Enable IMDSv2 by default for all EC2 instance hosts if IMDSv2 usage was not explicitly enabled.
• Added capability to generate profiling data in a flare requested via Remote Config
• Image layer digests will no longer report as "<missing>" from Docker runtimes.
• Upgraded github.com/DataDog/go-sqllexer to v0.1.1, resulting in reduced CPU usage and improved memory allocation efficiency.
• Upgraded github.com/DataDog/go-sqllexer to v0.1.3 to fix a bug that caused the lexer to panic when trimming identifier quotes.
• Added a new language detector that uses the apm-inject propagated language, if available.
• The KSM core check is now capable of retrying its coordination with the Kubernetes API Server. Failed queries in the check configuration will no longer block the check from being scheduled.
• • Add logging for log compression configuration.
  • The Agent now validates the log compression setting and falls back to the default compression if an invalid option is provided.
• Report resource requirements as native sidecars when restartPolicy=Always is used.
• Improved the behavior of the SQL obfuscator cache to allow caching of both obfuscated and normalized queries
• APM: Fix a formatting bug where the trace-agent's PID from "agent status" could be displayed in scientific notation for large PIDs.

Deprecation Notes

• logs.processed and logs.sent metrics are no longer emitted by the Agent
• Deprecation warnings added for the fips-proxy configuration keys (e.g. fips.*) as this is planned to be unsupported in 7.65+ releases of the datadog-agent. Please use the datadog-fips-agent for FIPS compliance instead.
• APM: The existing /config/set endpoint on the trace-agent is now deprecated in favor of the /config/set endpoint on the debug port on the trace-agent (default: 5012). The old endpoint will be removed in a future version.

Bug Fixes

• Fixed parsing of group resource strings to support groups with periods.
• Fixed a bug in the DogStatsD Unix socket server that caused metrics to miss container tags and the Agent to report matched PID for the process is 0 warnings.
• Fix issue with FIPS image where some build tags were missing for the process and trace agents during packaging.
• Makes CWS report the correct container ID on EKS Fargate.
• Fix an issue where ingestion_reason:probabilistic is set even when an OTLP span was sampled by the Error Sampler. To enable the Error Sampler for OTLP spans, you need to set DD_OTLP_CONFIG_TRACES_PROBABILISTIC_SAMPLER_SAMPLING_PERCENTAGE to 99 or lower, or enable DD_APM_PROBABILISTIC_SAMPLER_ENABLED and set DD_APM_PROBABILISTIC_SAMPLER_PERCENTAGE to 99 or lower.
• version-manifest.json and version-manifest.txt files now correctly reflect the packages content.
• Prevent journald and windows event logs from being errantly marked as truncated in specific circumstances.
• Obfuscation Cache Size Calculation: Resolved an issue where the cache item size was underestimated by not accounting for the Go struct overhead (including struct fields and headers for strings and slices). This fix ensures a more accurate calculation of cache item memory usage, leading to better memory efficiency and preventing over-allocation of NumCounters in cache configurations.
• Fix potential panic in journald and Windows event tailers during system shutdown
• Remove leading expressions in parentheses during SQL normalization.
• APM: Fix a rare panic that can occur when using client side stats in the tracers.
• APM: Fix an issue where the environment tag was normalized incorrectly. This resulted in some valid envs, like 123foo, having the leading digits removed. This fix allows these envs to pass through unedited.

Other Notes

• Add multi line log aggregation telemetry.

Datadog Cluster Agent

Prelude

Released on: 2025-03-19 Pinned to datadog-agent v7.64.0: CHANGELOG.

Upgrade Notes

• Datadog Autoscaling is upgraded to use DatadogPodAutoscaler CRD v1alpha2 instead of v1alpha1. Remote (created in Datadog) autoscalers are automatically migrated. In-cluster (Local) autoscalers need to be migrated manually.

New Features

• Enable collection of Pod Disruption Budgets by default in the orchestrator check.

• Target-based workload selection is now available for Single Step Instrumentation. This feature enables you to instrument specific workloads using pod and namespace label selectors. By applying user-defined labels, you can select workloads for instrumentation without modifying applications. For example, the following configuration injects the Python tracer with a default version for pods labeled with language=python:

instrumentation:
  enabled: true
  targets:
    - name: "Python Services"
      podSelector:
        matchLabels:
          language: "python"
      ddTraceVersions:
        python: "default"

• Targets can also be chained together, with the first matching rule taking precedence. For example, the following configuration installs the Python tracer for pods labeled language=python and the Java tracer for pods in a namespace labeled language=java. If a pod matches both rules, the first match takes precedence:

instrumentation:
  enabled: true
  targets:
    - name: "Python Services"
      podSelector:
        matchLabels:
          language: "python"
      ddTraceVersions:
        python: "default"
    - name: "Java Namespaces"
      namespaceSelector:
        matchLabels:
          language: "java"
      ddTraceVersions:
        python: "default"

• Targets support tracer configuration options in the form of environment variables. All options must have the DD_ prefix. The following example installs the Python tracer with profiling and data jobs enabled:

instrumentation:
  enabled: true
  targets:
    - name: "Python Apps"
      podSelector:
        matchLabels:
          l...

7.63.3

Agent

Prelude

Release on: 2025-03-04

• Please refer to the 7.63.3 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• Fully disable the X25519Kyber768Draft00 key exchange mechanism to avoid issues with firewalls not supporting multi-packet key exchanges, in particular AWS Network Firewall and Suricata.

Datadog Cluster Agent

Prelude

Released on: 2025-03-04 Pinned to datadog-agent v7.63.3: CHANGELOG.

7.63.2

Agent

Known issues

This version contains a TLS change that can in some circumstances prevent the Agent from communicating with our backend through AWS Network Firewalls due to an upstream issue. If you are using this combination of systems, the recommendation at this time is to downgrade to Agent v7.61 or upgrade to v7.63.3 when it becomes available.

Prelude

Release on: 2025-02-28

• Please refer to the 7.63.2 tag on integrations-core for the list of changes on the Core Checks

Datadog Cluster Agent

Prelude

Released on: 2025-02-28 Pinned to datadog-agent v7.63.2: CHANGELOG.

7.63.1

Agent

Known issues

This version contains a TLS change that can in some circumstances prevent the Agent from communicating with our backend through AWS Network Firewalls due to an upstream issue. If you are using this combination of systems, the recommendation at this time is to downgrade to Agent v7.61 or upgrade to v7.63.3 when it becomes available.

Prelude

Release on: 2025-02-26

• Please refer to the 7.63.1 tag on integrations-core for the list of changes on the Core Checks

Bug Fixes

• Publish image tags of the datadog-fips-agent build.

Datadog Cluster Agent

Prelude

Released on: 2025-02-26 Pinned to datadog-agent v7.63.1: CHANGELOG.

7.63.0

Agent

Known issues

This version contains a TLS change that can in some circumstances prevent the Agent from communicating with our backend through AWS Network Firewalls due to an upstream issue. If you are using this combination of systems, the recommendation at this time is to downgrade to Agent v7.61 or upgrade to v7.63.3 when it becomes available.

Prelude

Release on: 2025-02-19

• Please refer to the 7.63.0 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• Bump the Python version to 3.12.8

New Features

• Add support of CIS AlmaLinux 9 Benchmark in CSPM.

Enhancement Notes

• Adds a kube_cronjob tag to kubernetes_state.job.duration metric.
• Increase the Agent's default ecs_metadata_timeout from 500ms to 1000ms to avoid timeouts.
• Enhanced the Containerd Check to use a cache for container image sizes, reducing redundant API calls and improving performance.
• Add apm_config.obfuscation.cache.max_size to set the maximum size of the cache in bytes.
• Add TCP diagnosis check for logs_config.force_use_tcp.
• Added the Linux kernel's dmesg logs into the Agent flare. This information will appear in system-probe/dmesg.log.
• Begin collecting metrics from all internal Prometheus registries. Previously, the default registry was ignored, resulting in the omission of the point.sent and point.dropped metrics. This change ensures that all metrics are collected.
• Agents are now built with Go 1.23.5.
• Include Datadog Process Monitor (ddprocmon) service status in flare on Windows
• Language detection adds support for detecting PHP.
• When apm.features.enable_receive_resource_spans_v2 is set, trace agent OTLPReceiver now maps HTTP attributes from OTLP conventions to DD conventions. See the full list of attributes here: https://docs.datadoghq.com/opentelemetry/schema_semantics/semantic_mapping/?tab=datadogexporter#http
• Adds initial Windows support for UDP probes in Network Path.
• Updated Oracle check to lazily initialize the obfuscator. This should improve performance each time the Oracle check runs and collects SQL statements.
• The Windows Agent MSI now shows the user an error message if the provided password contains a semicolon.
• APM: Introduce sql_obfuscation_mode parameter. The value obfuscate_and_normalize is recommended for DBM customers to enhance APM/DBM correlation.
• APM: Adds span events as a top level payload field. Span events received this way will be altered according to rules defined by DD_APM_REPLACE_TAGS. Credit card obfuscation will also be applied to span event attributes.
• APM: If apm_config.obfuscation.remove_stack_traces is enabled the trace agent will now also remove the value at span tag exception.stacktrace replacing it with a "?".

Security Notes

• Update OpenSSL from 3.3.2 to 3.3.3 addressing CVE-2024-12797.
• On Windows, the named pipe \pipedd_system_probe from system probe is now restricted to Local System, Administrators, and the ddagentuser. Any other custom users are not supported.

Bug Fixes

• Fixes some existing metric transformer unit tests by correcting their assertions.

• Datadog span.Type and span.Resource attributes are set correctly for OTel spans processed via OTel Agent and Datadog Exporter when client span type is a database span.Type.

  span.Type logic update is limited to ReceiveResourceSpansV2 logic, set using "enable_receive_resource_spans_v2" in DD_APM_FEATURES

  span.Resource logic update is limited to OperationAndResourceNameV2 logic, set using "enable_operation_and_resource_name_logic_v2" in DD_APM_FEATURES

  Users should set a span.type attribute on their telemetry if they wish to override the default span type.

• Agent flare service status search for datadog services is now case insensitive on Windows

• Fixed an issue where the "source" and "service" tags were incorrectly set to "kubernetes" in logs when the Agent runs on ECS EC2.

• Bypass sending blank logs configs to the integrations launcher to prevent the launcher from sending JSON parse error logs.

• Respect proxy config in symdb endpoint.

• Fix IsUserAnAdmin call on Windows to use correct API.

• Fixed a bug that occurs when reinstalling marketplace/extra integrations for a RPM package after an Agent upgrade.

• Windows installer will not abort if the LanmanServer (Server) service is not running (regression introduced in 7.47.0).

• Fix the removal of non-core integrations during Agent upgrades on Windows platforms. To enable persisting non-core integration during install, set INSTALL_PYTHON_THIRD_PARTY_DEPS="1" property during the installation of the MSI.

Datadog Cluster Agent

Prelude

Released on: 2025-02-19 Pinned to datadog-agent v7.63.0: CHANGELOG.

Enhancement Notes

• Added support for kubernetesResourcesLabelsAsTags and kubernetesResourcesAnnotationsAsTags in the orchestrator check. Kubernetes resources processed by the orchestrator check can now include labels and annotations as tags, improving consistency with existing tagging configurations.
• The Cluster Agent is now able to delete ValidatingAdmissionWebhook and MutatingAdmissionWebhook depending on the admission_controller.validation.enabled and admission_controller.mutation.enabled settings. Note that admission_controller.enabled must be set to true to allow the Cluster Agent to interact with the Kubernetes Admission Controller.

Bug Fixes

• Fixes an issue with the datadog.cluster_agent.cluster_checks.configs_dispatched metric emitted by the Cluster Agent telemetry. The metric values could become inaccurate after the Cluster Agent loses and then regains leader status.

7.62.3

Agent

Known issues

This version contains a TLS change that can in some circumstances prevent the Agent from communicating with our backend through AWS Network Firewalls due to an upstream issue. If you are using this combination of systems, the recommendation at this time is to downgrade to Agent v7.61 or upgrade to v7.63.3 when it becomes available.

Prelude

Release on: 2025-02-14

• Please refer to the 7.62.3 tag on integrations-core for the list of changes on the Core Checks

Datadog Cluster Agent

Prelude

Released on: 2025-02-14 Pinned to datadog-agent v7.62.3: CHANGELOG.

7.62.2

Agent

Known issues

This version contains a TLS change that can in some circumstances prevent the Agent from communicating with our backend through AWS Network Firewalls due to an upstream issue. If you are using this combination of systems, the recommendation at this time is to downgrade to Agent v7.61 or upgrade to v7.63.3 when it becomes available.

Prelude

Release on: 2025-02-10

• Please refer to the 7.62.2 tag on integrations-core for the list of changes on the Core Checks

Upgrade Notes

• Removes Datadog user's full control of the Datadog data directory on Windows. If you are using custom configured values for log files, confd_path, run_path, or additional_checksd that are within the Datadog ProgramData folder, then you will have to explicitly give the Datadog user write permissions to the folders and files configured.

Security Notes

• Removes Datadog user's full control of the Datadog data directory on Windows.

Bug Fixes

• On Windows, ensures the ipc_perm.pem file's permissions are updated during installation.
• Disables fentry by default in event stream.

Datadog Cluster Agent

Prelude

Released on: 2025-02-10 Pinned to datadog-agent v7.62.2: CHANGELOG.

Bug Fixes

• Fix issue where annotations as tags were not showing up properly when certain resource collectors were enabled.