Skip to content

Commit 68d465f

Browse files
tasherif-msftjhendrixMSFTealsurrichardpark-msftazure-sdk
authored
[AzDatalake] Cleanup + Improvements (#21222)
* Enable gocritic during linting (#20715) Enabled gocritic's evalOrder to catch dependencies on undefined behavior on return statements. Updated to latest version of golangci-lint. Fixed issue in azblob flagged by latest linter. * Cosmos DB: Enable merge support (#20716) * Adding header and value * Wiring and tests * format * Fixing value * change log * [azservicebus, azeventhubs] Stress test and logging improvement (#20710) Logging improvements: * Updating the logging to print more tracing information (per-link) in prep for the bigger release coming up. * Trimming out some of the verbose logging, seeing if I can get it a bit more reasonable. Stress tests: * Add a timestamp to the log name we generate and also default to append, not overwrite. * Use 0.5 cores, 0.5GB as our baseline. Some pods use more and I'll tune them more later. * update proxy version (#20712) Co-authored-by: Scott Beddall <[email protected]> * Return an error when you try to send a message that's too large. (#20721) This now works just like the message batch - you'll get an ErrMessageTooLarge if you attempt to send a message that's too large for the link's configured size. NOTE: there's a patch to `internal/go-amqp/Sender.go` to match what's in go-amqp's main so it returns a programmatically useful error when the message is too large. Fixes #20647 * Changes in test that is failing in pipeline (#20693) * [azservicebus, azeventhubs] Treat 'entity full' as a fatal error (#20722) When the remote entity is full we get a resource-limit-exceeded condition. This isn't something we should keep retrying on and it's best to just abort and let the user know immediately, rather than hoping it might eventually clear out. This affected both Event Hubs and Service Bus. Fixes #20647 * [azservicebus/azeventhubs] Redirect stderr and stdout to tee (#20726) * Update changelog with latest features (#20730) * Update changelog with latest features Prepare for upcoming release. * bump minor version * pass along the artifact name so we can override it later (#20732) Co-authored-by: scbedd <[email protected]> * [azeventhubs] Fixing checkpoint store race condition (#20727) The checkpoint store wasn't guarding against multiple owners claiming for the first time - fixing this by using IfNoneMatch Fixes #20717 * Fix azidentity troubleshooting guide link (#20736) * [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 (#20437) * [Release] sdk/resourcemanager/paloaltonetworksngfw/armpanngfw/0.1.0 generation from spec commit: 85fb4ac6f8bfefd179e6c2632976a154b5c9ff04 * client factory * fix * fix * update * add sdk/resourcemanager/postgresql/armpostgresql live test (#20685) * add sdk/resourcemanager/postgresql/armpostgresql live test * update assets.json * set subscriptionId default value * format * add sdk/resourcemanager/eventhub/armeventhub live test (#20686) * add sdk/resourcemanager/eventhub/armeventhub live test * update assets * add sdk/resourcemanager/compute/armcompute live test (#20048) * add sdk/resourcemanager/compute/armcompute live test * skus filter * fix subscriptionId default value * fix * gofmt * update recording * sdk/resourcemanager/network/armnetwork live test (#20331) * sdk/resourcemanager/network/armnetwork live test * update subscriptionId default value * update recording * add sdk/resourcemanager/cosmos/armcosmos live test (#20705) * add sdk/resourcemanager/cosmos/armcosmos live test * update assets.json * update assets.json * update assets.json * update assets.json * Increment package version after release of azcore (#20740) * [azeventhubs] Improperly resetting etag in the checkpoint store (#20737) We shouldn't be resetting the etag to nil - it's what we use to enforce a "single winner" when doing ownership claims. The bug here was two-fold: I had bad logic in my previous claim ownership, which I fixed in a previous PR, but we need to reflect that same constraint properly in our in-memory checkpoint store for these tests. * Eng workflows sync and branch cleanup additions (#20743) Co-authored-by: James Suplizio <[email protected]> * [azeventhubs] Latest start position can also be inclusive (ie, get the latest message) (#20744) * Update GitHubEventProcessor version and remove pull_request_review procesing (#20751) Co-authored-by: James Suplizio <[email protected]> * Rename DisableAuthorityValidationAndInstanceDiscovery (#20746) * fix (#20707) * AzFile (#20739) * azfile: Fixing connection string parsing logic (#20798) * Fixing connection string parse logic * Update README * [azadmin] fix flaky test (#20758) * fix flaky test * charles suggestion * Prepare azidentity v1.3.0 for release (#20756) * Fix broken podman link (#20801) Co-authored-by: Wes Haggard <[email protected]> * [azquery] update doc comments (#20755) * update doc comments * update statistics and visualization generation * prep-for-release * Fixed contribution section (#20752) Co-authored-by: Bob Tabor <[email protected]> * [azeventhubs,azservicebus] Some API cleanup, renames (#20754) * Adding options to UpdateCheckpoint(), just for future potential expansion * Make Offset an int64, not a *int64 (it's not optional, it'll always come back with ReceivedEvents) * Adding more logging into the checkpoint store. * Point all imports at the production go-amqp * Add supporting features to enable distributed tracing (#20301) (#20708) * Add supporting features to enable distributed tracing This includes new internal pipeline policies and other supporting types. See the changelog for a full description. Added some missing doc comments. * fix linter issue * add net.peer.name trace attribute sequence custom HTTP header policy before logging policy. sequence logging policy after HTTP trace policy. keep body download policy at the end. * add span for iterating over pages * Restore ARM CAE support for azcore beta (#20657) This reverts commit 9020972. * Upgrade to stable azcore (#20808) * Increment package version after release of data/azcosmos (#20807) * Updating changelog (#20810) * Add fake package to azcore (#20711) * Add fake package to azcore This is the supporting infrastructure for the generated SDK fakes. * fix doc comment * Updating CHANGELOG.md (#20809) * changelog (#20811) * Increment package version after release of storage/azfile (#20813) * Update changelog (azblob) (#20815) * Updating CHANGELOG.md * Update the changelog with correct version * [azquery] migration guide (#20742) * migration guide * Charles feedback * Richard feedback --------- Co-authored-by: Charles Lowell <[email protected]> * Increment package version after release of monitor/azquery (#20820) * [keyvault] prep for release (#20819) * prep for release * perf tests * update date * fixed datalake errors + moved to internal path * delegation key + constants * removed test * further cleanup * renamed support and client fixes * added tests * handle error for rename * fixed response formatting * cleanup --------- Co-authored-by: Joel Hendrix <[email protected]> Co-authored-by: Matias Quaranta <[email protected]> Co-authored-by: Richard Park <[email protected]> Co-authored-by: Azure SDK Bot <[email protected]> Co-authored-by: Scott Beddall <[email protected]> Co-authored-by: siminsavani-msft <[email protected]> Co-authored-by: scbedd <[email protected]> Co-authored-by: Charles Lowell <[email protected]> Co-authored-by: Peng Jiahui <[email protected]> Co-authored-by: James Suplizio <[email protected]> Co-authored-by: Sourav Gupta <[email protected]> Co-authored-by: gracewilcox <[email protected]> Co-authored-by: Wes Haggard <[email protected]> Co-authored-by: Bob Tabor <[email protected]> Co-authored-by: Bob Tabor <[email protected]>
1 parent 9a351b9 commit 68d465f

32 files changed

+1538
-1183
lines changed

sdk/storage/azdatalake/assets.json

+1-1
Original file line numberDiff line numberDiff line change
@@ -2,5 +2,5 @@
22
"AssetsRepo": "Azure/azure-sdk-assets",
33
"AssetsRepoPrefixPath": "go",
44
"TagPrefix": "go/storage/azdatalake",
5-
"Tag": "go/storage/azdatalake_820b86faa9"
5+
"Tag": "go/storage/azdatalake_db1de4a48b"
66
}

sdk/storage/azdatalake/common.go

+45
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
package azdatalake
88

99
import (
10+
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/lease"
1011
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/exported"
1112
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/sas"
1213
)
@@ -34,3 +35,47 @@ func ParseURL(u string) (URLParts, error) {
3435
// ending at offset+count. A zero-value HTTPRange indicates the entire resource. An HTTPRange
3536
// which has an offset but no zero value count indicates from the offset to the resource's end.
3637
type HTTPRange = exported.HTTPRange
38+
39+
// ===================================== LEASE CONSTANTS ============================================================
40+
41+
// StatusType defines values for StatusType
42+
type StatusType = lease.StatusType
43+
44+
const (
45+
StatusTypeLocked StatusType = lease.StatusTypeLocked
46+
StatusTypeUnlocked StatusType = lease.StatusTypeUnlocked
47+
)
48+
49+
// PossibleStatusTypeValues returns the possible values for the StatusType const type.
50+
func PossibleStatusTypeValues() []StatusType {
51+
return lease.PossibleStatusTypeValues()
52+
}
53+
54+
// DurationType defines values for DurationType
55+
type DurationType = lease.DurationType
56+
57+
const (
58+
DurationTypeInfinite DurationType = lease.DurationTypeInfinite
59+
DurationTypeFixed DurationType = lease.DurationTypeFixed
60+
)
61+
62+
// PossibleDurationTypeValues returns the possible values for the DurationType const type.
63+
func PossibleDurationTypeValues() []DurationType {
64+
return lease.PossibleDurationTypeValues()
65+
}
66+
67+
// StateType defines values for StateType
68+
type StateType = lease.StateType
69+
70+
const (
71+
StateTypeAvailable StateType = lease.StateTypeAvailable
72+
StateTypeLeased StateType = lease.StateTypeLeased
73+
StateTypeExpired StateType = lease.StateTypeExpired
74+
StateTypeBreaking StateType = lease.StateTypeBreaking
75+
StateTypeBroken StateType = lease.StateTypeBroken
76+
)
77+
78+
// PossibleStateTypeValues returns the possible values for the StateType const type.
79+
func PossibleStateTypeValues() []StateType {
80+
return lease.PossibleStateTypeValues()
81+
}

sdk/storage/azdatalake/datalakeerror/error_codes.go

+117-123
Large diffs are not rendered by default.

sdk/storage/azdatalake/directory/client.go

+9-8
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ func NewClient(directoryURL string, cred azcore.TokenCredential, options *Client
5050
ClientOptions: options.ClientOptions,
5151
}
5252
blobClient, _ := blockblob.NewClient(blobURL, cred, &blobClientOpts)
53-
dirClient := base.NewPathClient(directoryURL, blobURL, blobClient, azClient, nil, (*base.ClientOptions)(conOptions))
53+
dirClient := base.NewPathClient(directoryURL, blobURL, blobClient, azClient, nil, &cred, (*base.ClientOptions)(conOptions))
5454

5555
return (*Client)(dirClient), nil
5656
}
@@ -78,7 +78,7 @@ func NewClientWithNoCredential(directoryURL string, options *ClientOptions) (*Cl
7878
ClientOptions: options.ClientOptions,
7979
}
8080
blobClient, _ := blockblob.NewClientWithNoCredential(blobURL, &blobClientOpts)
81-
dirClient := base.NewPathClient(directoryURL, blobURL, blobClient, azClient, nil, (*base.ClientOptions)(conOptions))
81+
dirClient := base.NewPathClient(directoryURL, blobURL, blobClient, azClient, nil, nil, (*base.ClientOptions)(conOptions))
8282

8383
return (*Client)(dirClient), nil
8484
}
@@ -113,7 +113,7 @@ func NewClientWithSharedKeyCredential(directoryURL string, cred *SharedKeyCreden
113113
return nil, err
114114
}
115115
blobClient, _ := blockblob.NewClientWithSharedKeyCredential(blobURL, blobSharedKey, &blobClientOpts)
116-
dirClient := base.NewPathClient(directoryURL, blobURL, blobClient, azClient, cred, (*base.ClientOptions)(conOptions))
116+
dirClient := base.NewPathClient(directoryURL, blobURL, blobClient, azClient, cred, nil, (*base.ClientOptions)(conOptions))
117117

118118
return (*Client)(dirClient), nil
119119
}
@@ -158,6 +158,10 @@ func (d *Client) sharedKey() *exported.SharedKeyCredential {
158158
return base.SharedKeyComposite((*base.CompositeClient[generated.PathClient, generated.PathClient, blockblob.Client])(d))
159159
}
160160

161+
func (d *Client) identityCredential() *azcore.TokenCredential {
162+
return base.IdentityCredentialComposite((*base.CompositeClient[generated.PathClient, generated.PathClient, blockblob.Client])(d))
163+
}
164+
161165
// DFSURL returns the URL endpoint used by the Client object.
162166
func (d *Client) DFSURL() string {
163167
return d.generatedDirClientWithDFS().Endpoint()
@@ -168,6 +172,8 @@ func (d *Client) BlobURL() string {
168172
return d.generatedDirClientWithBlob().Endpoint()
169173
}
170174

175+
//TODO: create method to get file client - this will require block blob to have a method to get another block blob
176+
171177
// Create creates a new directory (dfs1).
172178
func (d *Client) Create(ctx context.Context, options *CreateOptions) (CreateResponse, error) {
173179
return CreateResponse{}, nil
@@ -230,8 +236,3 @@ func (d *Client) SetHTTPHeaders(ctx context.Context, httpHeaders HTTPHeaders, op
230236
// TODO: call into blob
231237
return SetHTTPHeadersResponse{}, nil
232238
}
233-
234-
// UndeletePath restores the specified path that was previously deleted. (dfs op/blob2).
235-
func (d *Client) UndeletePath(ctx context.Context, path string, options *UndeletePathOptions) (UndeletePathResponse, error) {
236-
return UndeletePathResponse{}, nil
237-
}

sdk/storage/azdatalake/directory/constants.go

+4-29
Original file line numberDiff line numberDiff line change
@@ -7,37 +7,12 @@
77
package directory
88

99
import (
10-
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
11-
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/generated"
10+
"github.com/Azure/azure-sdk-for-go/sdk/storage/azdatalake/internal/path"
1211
)
1312

14-
type ResourceType = generated.PathResourceType
13+
type EncryptionAlgorithmType = path.EncryptionAlgorithmType
1514

16-
// TODO: consider the possibility of not exposing this and just pass it under the hood
1715
const (
18-
ResourceTypeFile ResourceType = generated.PathResourceTypeFile
19-
ResourceTypeDirectory ResourceType = generated.PathResourceTypeDirectory
20-
)
21-
22-
type RenameMode = generated.PathRenameMode
23-
24-
// TODO: consider the possibility of not exposing this and just pass it under the hood
25-
const (
26-
RenameModeLegacy RenameMode = generated.PathRenameModeLegacy
27-
RenameModePosix RenameMode = generated.PathRenameModePosix
28-
)
29-
30-
type SetAccessControlRecursiveMode = generated.PathSetAccessControlRecursiveMode
31-
32-
const (
33-
SetAccessControlRecursiveModeSet SetAccessControlRecursiveMode = generated.PathSetAccessControlRecursiveModeSet
34-
SetAccessControlRecursiveModeModify SetAccessControlRecursiveMode = generated.PathSetAccessControlRecursiveModeModify
35-
SetAccessControlRecursiveModeRemove SetAccessControlRecursiveMode = generated.PathSetAccessControlRecursiveModeRemove
36-
)
37-
38-
type EncryptionAlgorithmType = blob.EncryptionAlgorithmType
39-
40-
const (
41-
EncryptionAlgorithmTypeNone EncryptionAlgorithmType = blob.EncryptionAlgorithmTypeNone
42-
EncryptionAlgorithmTypeAES256 EncryptionAlgorithmType = blob.EncryptionAlgorithmTypeAES256
16+
EncryptionAlgorithmTypeNone EncryptionAlgorithmType = path.EncryptionAlgorithmTypeNone
17+
EncryptionAlgorithmTypeAES256 EncryptionAlgorithmType = path.EncryptionAlgorithmTypeAES256
4318
)

0 commit comments

Comments
 (0)