feat(azure_logs_ingestion): Initial `azure_logs_ingestion` sink #22912

jlaundry · 2025-04-20T23:12:14Z

Summary

The current azure_monitor_logs sink uses the Data Collector API, which has been deprecated and will be removed in September 2026.

This sink uses the replacement Logs Ingestion API.

While I did consider making this a drop-in replacement for the existing sink, users need to make numerous breaking infrastructure changes, including:

Creating new Data Collection Endpoint and Data Collection Rule resources
Moving from a workspace-based secret key to an OAuth credential (App Registration, Managed Identity, etc.)
(optionally) Re-configuring logs to use the built-in tables, instead of _CL custom tables.

Change Type

Bug fix
New feature
Non-functional (chore, refactoring, docs)
Performance

Is this a breaking change?

Yes
No

How did you test this PR?

Following the Tutorial steps, create a Log Analytics workspace, App Registration, Data Collection Endpoint, and Data Collection Rule
Set the AZURE_TENANT_ID, AZURE_CLIENT_ID, and AZURE_CLIENT_SECRET environment variables from the App Registration
Use the following vector.yaml:

sources:
  stdin:
    type: stdin

sinks:
  azure:
    type: azure_logs_ingestion
    inputs:
      - stdin
    endpoint: https://dce-e42z.westus2-1.ingest.monitor.azure.com
    dcr_immutable_id: dcr-00000000000000000000000000000000
    stream_name: Custom-vector_CL

Does this PR include user facing changes?

Yes. Please add a changelog fragment based on our guidelines.
No. A maintainer will apply the "no-changelog" label to this PR.

References

Closes: Azure DCR-based custom logs -> Logs ingestion API #20978
Mentioned in: Enhance Azure Identity Integration in Vector: Ability to Utilize Passwordless Connections #20625 - while this PR doesn't resolve this issue for azure_blob, by using the Azure Identity crate, this sink supports passwordless credentials.

Signed-off-by: Jed Laundry <[email protected]>

bits-bot · 2025-04-20T23:12:18Z

All committers have signed the CLA.

Signed-off-by: Jed Laundry <[email protected]>

check-spelling run (pull_request_target) for feature-azure_logs_ingestion Signed-off-by: check-spelling-bot <[email protected]> on-behalf-of: @check-spelling <[email protected]>

pront

Hi @jlaundry, thanks for this contribution!

It looks good. Two things:

We will need some documentation files. See an example here (all files under website). Note that base/ is generated by make generate-component-docs.
Is the intention to complete replace the azure_monitor_logs sink? If that's the case maybe we can mark the existing one as deprecated in favor of this new sink.

rtrieu

Thank you for your contribution! Approving with one very minor suggestion.

rtrieu · 2025-04-21T18:54:54Z

website/cue/reference/components/sinks/base/azure_logs_ingestion.cue

+					The maximum size of a batch that is processed by a sink.
+
+					This is based on the uncompressed size of the batched events, before they are
+					serialized/compressed.


Suggested change

serialized/compressed.

serialized or compressed.

This text is imported from https://github.com/vectordotdev/vector/blob/master/src/sinks/util/batch.rs#L104, and appears in most of the sinks/base/*.cue files - should we raise a separate PR to change this?

Sounds reasonable.

jlaundry · 2025-04-22T20:03:27Z

We will need some documentation files. See an example here (all files under website). Note that base/ is generated by make generate-component-docs.

Apologies, I thought this page was auto-generated as well - added now.

Is the intention to complete replace the azure_monitor_logs sink? If that's the case maybe we can mark the existing one as deprecated in favor of this new sink.

Good point, added 🙂

pront · 2025-04-25T19:27:07Z

src/sinks/azure_logs_ingestion/service.rs

+
+static CONTENT_TYPE_VALUE: LazyLock<HeaderValue> =
+    LazyLock::new(|| HeaderValue::from_static(CONTENT_TYPE));
+// static X_MS_CLIENT_REQUEST_ID_HEADER: LazyLock<HeaderName> =


Delete if not needed

pront · 2025-04-25T19:27:46Z

src/sinks/azure_logs_ingestion/mod.rs

+// #[cfg(test)]
+// mod tests;


Did you mean to add a tests.rs? If not you can delete this.

Actually it would be great to have a simple integration test here. But I don't want to block the PR because of this. If you look at other integrations tests, it should be easy to add one (assuming we have a azure logs docker image)

I've spent some time this weekend looking for a nice way to do testing... and unfortunately it's not as easy as it seems.

Because the API expects an OAuth token, we need to mock the OAuth endpoint, which would be simple, except:

to mock the token endpoint, we need the azure_identity crate to be updated to 0.23.0 to use a ClientSecretCredential with specific authority endpoint, and

that update is blocked, because it and azure_core have substantial refactors, and azure_storage hasn't yet been updated (the azure_blob sink uses azure_storage)

The only other option is to use a real Azure App Registration as part of the test suite, and ignore the OAuth token in the mock API - which would be simple to setup for the GitHub Actions tests, but would then require individual devs to have Azure CLI / their own AZURE_CLIENT_ID, which doesn't seem nice.

I've stashed a WIP: jlaundry/vector@feature-azure_logs_ingestion...jlaundry:vector:feature-azure_logs_ingestion-tests and will tinker with this for a bit, but we might be best waiting until the azure_storage crate update is unblocked.

pront · 2025-04-25T19:28:10Z

src/sinks/azure_logs_ingestion/service.rs

+    ) -> crate::Result<Self> {
+        let mut parts = endpoint.into_parts();
+        parts.path_and_query = Some(
+            // https://my-dce-5kyl.eastus-1.ingest.monitor.azure.com/dataCollectionRules/dcr-000a00a000a00000a000000aa000a0aa/streams/Custom-MyTable?api-version=2023-01-01


Delete debug comment

jlaundry added 17 commits November 2, 2024 03:49

update azure crates to 0.21

efa78f4

Signed-off-by: Jed Laundry <[email protected]>

initial azure_logs_ingestion implementation

b0f8438

Signed-off-by: Jed Laundry <[email protected]>

tidy up error messages

2c76ddf

make token_scope configurable

43e2a77

Signed-off-by: Jed Laundry <[email protected]>

parameterise the dcr_immutable_id and stream_name

9d022e5

Signed-off-by: Jed Laundry <[email protected]>

spelling mistake

a5a4565

add reminder from upstream changes

b42fd4d

initial azure_logs_ingestion implementation

92bf424

Signed-off-by: Jed Laundry <[email protected]>

tidy up error messages

430294f

make token_scope configurable

f038731

Signed-off-by: Jed Laundry <[email protected]>

parameterise the dcr_immutable_id and stream_name

ed5bcfb

Signed-off-by: Jed Laundry <[email protected]>

spelling mistake

40450b4

add reminder from upstream changes

4f3fc79

rebase from upstream

be60a11

Signed-off-by: Jed Laundry <[email protected]>

update for 1.83 toolchain

3a3164a

Signed-off-by: Jed Laundry <[email protected]>

Add configurable timestamp field

2279f92

Signed-off-by: Jed Laundry <[email protected]>

add docs

dd18960

Signed-off-by: Jed Laundry <[email protected]>

jlaundry requested review from a team as code owners April 20, 2025 23:12

github-actions bot added domain: sinks Anything related to the Vector's sinks domain: external docs Anything related to Vector's external, public documentation labels Apr 20, 2025

jlaundry added 2 commits April 20, 2025 23:17

add changelog fragment

55297e7

Signed-off-by: Jed Laundry <[email protected]>

Update metadata

967771a

check-spelling run (pull_request_target) for feature-azure_logs_ingestion Signed-off-by: check-spelling-bot <[email protected]> on-behalf-of: @check-spelling <[email protected]>

pront self-assigned this Apr 21, 2025

pront reviewed Apr 21, 2025

View reviewed changes

rtrieu approved these changes Apr 21, 2025

View reviewed changes

jlaundry added 3 commits April 22, 2025 19:48

check-fmt lint

4d72879

initial website docs

d39e51d

add deprecation notice to azure_monitor_logs

b5c85d4

add deprecation notice to azure_monitor_logs

0061348

pront reviewed Apr 25, 2025

View reviewed changes

remove debug comments

f8aad84

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(azure_logs_ingestion): Initial `azure_logs_ingestion` sink #22912

feat(azure_logs_ingestion): Initial `azure_logs_ingestion` sink #22912

jlaundry commented Apr 20, 2025

bits-bot commented Apr 20, 2025 •

edited

Loading

pront left a comment

rtrieu left a comment

rtrieu Apr 21, 2025

jlaundry Apr 22, 2025

pront Apr 22, 2025

jlaundry commented Apr 22, 2025

pront Apr 25, 2025

jlaundry May 4, 2025

pront Apr 25, 2025

pront Apr 25, 2025

jlaundry May 4, 2025

pront Apr 25, 2025

jlaundry May 4, 2025

feat(azure_logs_ingestion): Initial azure_logs_ingestion sink #22912

Are you sure you want to change the base?

feat(azure_logs_ingestion): Initial azure_logs_ingestion sink #22912

Conversation

jlaundry commented Apr 20, 2025

Summary

Change Type

Is this a breaking change?

How did you test this PR?

Does this PR include user facing changes?

References

bits-bot commented Apr 20, 2025 • edited Loading

pront left a comment

Choose a reason for hiding this comment

rtrieu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jlaundry commented Apr 22, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

feat(azure_logs_ingestion): Initial `azure_logs_ingestion` sink #22912

feat(azure_logs_ingestion): Initial `azure_logs_ingestion` sink #22912

bits-bot commented Apr 20, 2025 •

edited

Loading