Send to HN

I’m looking to build a tool to migrate files between cloud providers (ie blob storage, s3, etc).

I’m looking to learn best practices for doing so. The volume is a few petabytes but can be done incrementally.

I was thinking some queue that handles N transfers concurrently running in a k8s cluster. Queue I was thinking Redis or Kafka. However, I’d love to build something nice and resilient