feat(services/azdls): list start from and sas token support

[alexwilcoxson-rel]

Which issue does this PR close?

Closes #.

Rationale for this change

This changes proposes implementation of the start_after workaround used by Hadoop to improve listing in ADLS from a particular point.

This is important for table formats like Delta Lake that make use of list_with_offset in object store to get the latest version of a table.

In my testing listing directories with > 10k records similar to what Delta Lake would do improves times from seconds to milliseconds (~3seconds to 60ms).

Also this adds SAS token support which could be split into its own PR but needed it for testing against my company's ADLS accounts.

What changes are included in this PR?

Changes to azdls module to include list with start_after functionality. This includes a module crc64 which computes crc for use in the continuation token generated for start after

Are there any user-facing changes?

Yes, start_after will function for azdls, and SAS token support.

---

@Xuanwo I'm opening this as a draft to get initial feedback. This is not directly supported by Microsoft, or I should say documented. In pursuit of them adding this capability to the azblob endpoints, they were the ones that directed me to Hadoop example, so I have confidence in it but could see if its something they would officially support/document.

One caveat as well is I believe (per the hadoop code) the change is will only work if hierarchical namespace (xns) is enabled which isn't something I believe we can tell when the list request is being made. It could be the user has to opt in and inform if xns is enabled. Hadoop code does another request to figure that out dynamically which I'd personally want to avoid.

[Xuanwo]

Hi, thank you @alexwilcoxson-rel for working on this. I've been considering porting logic from Hadoop to OpenDAL for a while but haven't taken action yet.

This feature is undocumented, so it's best not to enable it by default. How about adding a new flag called enable_list_with_start_after, where we set the capability list_with_start_after to true?

[Xuanwo]

Reqsign has sas_token support; we can use it here: https://docs.rs/reqsign/0.16.0/reqsign/struct.AzureStorageConfig.html#structfield.sas_token

Maybe it's worth a separate PR?

[alexwilcoxson-rel]

Are you suggesting we use reqsign::AzureStorageConfig on AzdlsBuilder or AzdlsConfig or something else?

[Xuanwo]

Yes, we can construct like s3 does:

opendal/core/src/services/s3/backend.rs

Lines 745 to 783 in 3fe8caa

	// This is our current config.
	let mut cfg = AwsConfig::default();
	if !self.config.disable_config_load {
	#[cfg(not(target_arch = "wasm32"))]
	{
	cfg = cfg.from_profile();
	cfg = cfg.from_env();
	}
	}
	if let Some(v) = self.config.region.clone() {
	cfg.region = Some(v);
	}
	if cfg.region.is_none() {
	return Err(Error::new(
	ErrorKind::ConfigInvalid,
	"region is missing. Please find it by S3::detect_region() or set them in env.",
	)
	.with_operation("Builder::build")
	.with_context("service", Scheme::S3));
	}
	let region = cfg.region.to_owned().unwrap();
	debug!("backend use region: {region}");
	// Building endpoint.
	let endpoint = self.build_endpoint(&region);
	debug!("backend use endpoint: {endpoint}");
	// Setting all value from user input if available.
	if let Some(v) = self.config.access_key_id {
	cfg.access_key_id = Some(v)
	}
	if let Some(v) = self.config.secret_access_key {
	cfg.secret_access_key = Some(v)
	}
	if let Some(v) = self.config.session_token {
	cfg.session_token = Some(v)
	}

[alexwilcoxson-rel]

I will open a follow up issue for this if this gets merged

• follow up issue

[alexwilcoxson-rel]

hey @Xuanwo as I was refactoring the code I wrote a test against one of our actual storage accounts. I then executed the list call with recursive = true as the object_store integration does. However when I do that list result from azure contains everything.

I'm trying to figure out how recursive could be impacting the azdls list but I don't see it used by azdls service.

EDIT: ah I see the CompleteLayer, looking into this.

[alexwilcoxson-rel]

Ok so I found the FlatLister does not retain the OpList args.

Adding modifying FlatLister to take in the original args and pass them to the inner lister seems to work for this case, but I'm not sure how recursive listing and start_after should interact for all edge cases.

The underlying Azure REST API does have a recursive parameter, but when actually using that:

1. the recursive listing performance in general is not great on azdls
2. start_after continuation token causes 500 when recursive = true
   so I am hesitant to enable it

[Xuanwo]

Thank you @alexwilcoxson-rel for catching this.

Adding modifying FlatLister to take in the original args and pass them to the inner lister seems to work for this case, but I'm not sure how recursive listing and start_after should interact for all edge cases.

It should be fine as long as we handle the capability correctly.

The underlying Azure REST API does have a recursive parameter, but when actually using that:

1. the recursive listing performance in general is not great on azdls
2. start_after continuation token causes 500 when recursive = true
   so I am hesitant to enable it

That seems to be a problem. Would you like to raise a separate issue for this? We can track it and switch to azdls's own recursive when available.

[Xuanwo]

Yes, we can construct like s3 does:

opendal/core/src/services/s3/backend.rs

Lines 745 to 783 in 3fe8caa

	// This is our current config.
	let mut cfg = AwsConfig::default();
	if !self.config.disable_config_load {
	#[cfg(not(target_arch = "wasm32"))]
	{
	cfg = cfg.from_profile();
	cfg = cfg.from_env();
	}
	}
	if let Some(v) = self.config.region.clone() {
	cfg.region = Some(v);
	}
	if cfg.region.is_none() {
	return Err(Error::new(
	ErrorKind::ConfigInvalid,
	"region is missing. Please find it by S3::detect_region() or set them in env.",
	)
	.with_operation("Builder::build")
	.with_context("service", Scheme::S3));
	}
	let region = cfg.region.to_owned().unwrap();
	debug!("backend use region: {region}");
	// Building endpoint.
	let endpoint = self.build_endpoint(&region);
	debug!("backend use endpoint: {endpoint}");
	// Setting all value from user input if available.
	if let Some(v) = self.config.access_key_id {
	cfg.access_key_id = Some(v)
	}
	if let Some(v) = self.config.secret_access_key {
	cfg.secret_access_key = Some(v)
	}
	if let Some(v) = self.config.session_token {
	cfg.session_token = Some(v)
	}

[alexwilcoxson-rel]

The underlying Azure REST API does have a recursive parameter, but when actually using that:

1. the recursive listing performance in general is not great on azdls
2. start_after continuation token causes 500 when recursive = true
   so I am hesitant to enable it

That seems to be a problem. Would you like to raise a separate issue for this? We can track it and switch to azdls's own recursive when available.

• Yes I will create a follow up issue to switch to use recursive parameter available on azdls list. Mentioning though the seeming incompatibility with this start_after workaround

[alexwilcoxson-rel]

This feature is undocumented, so it's best not to enable it by default. How about adding a new flag called enable_list_with_start_after, where we set the capability list_with_start_after to true?

• gate behind feature

[alexwilcoxson-rel]

Hey @Xuanwo just letting you know I probably won't get back to this until later in the week. I will share though that I am using this in our fork of version ~0.48 I think. We're using it along size the base object_store azure store (since those have all the put support for delta), and it has gone well!

Looking forward to getting this in along with the put if-not-match changes that are in progress as well!

[Xuanwo]

Hi, @alexwilcoxson-rel, do we have a status change here? I'm open to get this feature merged under a new config called enable_list_with_start_after.

[alexwilcoxson-rel]

Hey @Xuanwo sorry for the delay on this, I have some cycles to try to get it across the line

[alexwilcoxson-rel]

This feature is undocumented, so it's best not to enable it by default. How about adding a new flag called enable_list_with_start_after, where we set the capability list_with_start_after to true?

done

[alexwilcoxson-rel]

In testing this with deltalake table loads, i see the object_store integration is doing a stat per list entry

is there a way around this if the list response contains the needed metadata? i notice the metadata key stuff was removed (for the better 😄 )

[Xuanwo]

sas_token support has been added in #6205