Skip to content

fix(services/s3): Remove not needed check for batch delete #6206

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
May 21, 2025

Conversation

Xuanwo
Copy link
Member

@Xuanwo Xuanwo commented May 21, 2025

Which issue does this PR close?

None

Rationale for this change

One of opendal users report a panic in opendal:

2025-05-16T06:55:27.909244+00:00[Etc/UTC]  WARN storage::service: service.rs:103 opendal will retry after 2.563914299s because: RateLimited (temporary) at delete => S3Error { code: "SlowDown", message: "Please reduce your request rate.", resource: "", request_id: "x" }

Context:
   uri: http://xxx.s3.cn-northwest-1.amazonaws.com.cn/?delete
   response: Parts { status: 503, version: HTTP/1.1, headers: {"x-amz-request-id": "x", "x-amz-id-2": "x", "content-type": "application/xml", "transfer-encoding": "chunked", "date": "Fri, 16 May 2025 06:55:27 GMT", "connection": "close", "server": "AmazonS3"} }
   service: s3
   deleted: 0
 node_id=x nodegroup=x
2025-05-16T06:55:27.921869+00:00[Etc/UTC] ERROR runtime::global: global.rs:105 panic occurred: panicked at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opendal-0.53.1/src/services/s3/delete.rs:73:36:
removal index (is 0) should be < len (is 0)
backtrace:
   0: std::backtrace_rs::backtrace::libunwind::trace
             at /rustc/6bc57c6bf7d0024ad9ea5a2c112f3fc9c383c8a4/library/std/src/../../backtrace/src/backtrace/libunwind.rs:117:9
   1: std::backtrace_rs::backtrace::trace_unsynchronized
             at /rustc/6bc57c6bf7d0024ad9ea5a2c112f3fc9c383c8a4/library/std/src/../../backtrace/src/backtrace/mod.rs:66:14
   2: std::backtrace::Backtrace::create
             at /rustc/6bc57c6bf7d0024ad9ea5a2c112f3fc9c383c8a4/library/std/src/backtrace.rs:331:13
   3: runtime::global::set_panic_hook::{{closure}}
             at /build/crates/runtime/src/global.rs:104:25
   4: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
             at /rustc/6bc57c6bf7d0024ad9ea5a2c112f3fc9c383c8a4/library/alloc/src/boxed.rs:1980:9
   5: std::panicking::rust_panic_with_hook
             at /rustc/6bc57c6bf7d0024ad9ea5a2c112f3fc9c383c8a4/library/std/src/panicking.rs:841:13
   6: std::panicking::begin_panic_handler::{{closure}}
             at /rustc/6bc57c6bf7d0024ad9ea5a2c112f3fc9c383c8a4/library/std/src/panicking.rs:706:13
   7: std::sys::backtrace::__rust_end_short_backtrace
             at /rustc/6bc57c6bf7d0024ad9ea5a2c112f3fc9c383c8a4/library/std/src/sys/backtrace.rs:168:18
   8: __rustc::rust_begin_unwind
             at /rustc/6bc57c6bf7d0024ad9ea5a2c112f3fc9c383c8a4/library/std/src/panicking.rs:697:5
   9: core::panicking::panic_fmt
             at /rustc/6bc57c6bf7d0024ad9ea5a2c112f3fc9c383c8a4/library/core/src/panicking.rs:75:14
  10: core::fmt::Arguments::new_v1
             at /rustc/6bc57c6bf7d0024ad9ea5a2c112f3fc9c383c8a4/library/core/src/fmt/mod.rs:643:9
  11: alloc::vec::Vec<T,A>::remove::assert_failed
             at /rustc/6bc57c6bf7d0024ad9ea5a2c112f3fc9c383c8a4/library/alloc/src/vec/mod.rs:2071:13
  12: alloc::vec::Vec<T,A>::remove
             at /rustc/6bc57c6bf7d0024ad9ea5a2c112f3fc9c383c8a4/library/alloc/src/vec/mod.rs:2076:13
  13: <opendal::services::s3::delete::S3Deleter as opendal::raw::oio::delete::batch_delete::BatchDelete>::delete_batch::{{closure}}
             at /usr/local/cargo/registry/src/index.crates.io-1949cf8c6b5b557f/opendal-0.53.1/src/services/s3/delete.rs:73:23

It appears that AWS S3 can return an Ok response even if no files were actually removed successfully. The best thing here we can do is just remove this check.

What changes are included in this PR?

Remove the early check to return error.

Are there any user-facing changes?

No.

@dosubot dosubot bot added size:XS This PR changes 0-9 lines, ignoring generated files. releases-note/fix The PR fixes a bug or has a title that begins with "fix" labels May 21, 2025
@Xuanwo Xuanwo requested a review from tisonkun May 21, 2025 07:48
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label May 21, 2025
@Xuanwo Xuanwo merged commit 074fa7a into main May 21, 2025
103 checks passed
@Xuanwo Xuanwo deleted the fix-s3-delete-objects branch May 21, 2025 07:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lgtm This PR has been approved by a maintainer releases-note/fix The PR fixes a bug or has a title that begins with "fix" size:XS This PR changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants