feat(io): Add delete_stream to Storage trait#2216
Conversation
| } | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
I didn't add this for gcs because fake-gcs-server doesn't support batch delete
There was a problem hiding this comment.
Sorry, I don't get your point.
There was a problem hiding this comment.
This works for S3 but the same test would fail for gcs, because we use fake-gcs-server for testing, and fake-gcs-server doesn't support batch delete.
crates/storage/opendal/src/lib.rs
Outdated
|
|
||
| // Use relativize_path for remaining paths to avoid rebuilding the operator each time. | ||
| while let Some(path) = paths.next().await { | ||
| let relative_path = self.relativize_path(&path)?; |
There was a problem hiding this comment.
I see some problems with this approach, what if we are deleting things as following:
s3://bucket1/a.txt
s3://bucketb/b.txt
There was a problem hiding this comment.
I thought about this as well, but I think it would be tricky to solve this. Because OpenDal's operator is tied to a bucket, and if user passes in locations from multiple buckets then we will have to create operator for each of the incoming locations --- it's gonna be slow
Also I'd say this is an acceptable limitation because most users will not have data spanned across several buckets
There was a problem hiding this comment.
We don't need to create operator for each location, we just need to create operator for each bucket.
There was a problem hiding this comment.
I've updated the implementation to include a HashMap to map from bucket name to deleter
| } | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
Sorry, I don't get your point.
| } | ||
|
|
||
| #[tokio::test] | ||
| async fn test_file_io_s3_delete_stream() { |
2 participants
Which issue does this PR close?
What changes are included in this PR?
delete_streamtoStoragetrait to support batch deletedelete_streaminFileIOas wellAre these changes tested?
Added uts
Addded integtests for opendal