data pipeline

[Jignesh Patel]

If there is a need to transfer the data between 10000 tenants via kafka to s3, is it advisable to create 10000 topics - one for each tenant and then similarly creating one bucket for each tenant?
is there better architecture solution than this?

[Tim Cooke]

That's a lot of buckets.

Since you don't present much information about the problem you are trying to solve it's hard to make any meaningful suggestions.

[Tim Holloway]

I held off on responding because I've not done much with buckets and always considered them best for big BLOBs such as VM images.

I would be more inclined to slap a filesystem over the bucket or forgo buckets entirely. I'm fairly sure that the object store on my own servers would make more efficient use of the available storage if it was subdivided (many objects in one bucket) than many buckets.