Experimental: External Payload Storage

[jmaeagle99]

What was changed

Add external storage APIs to allow customers to specify how large payloads or payloads that match some specific customer-defined quality can be stored via customer-supplied external storage systems.

• Customers can choose their external storage endpoints by providing storage drivers. They can either use prebuilt ones or write their own via the StorageDriver type. This change does not contain any prebuilt drivers, just provides the primitives to make them.
• Customers can register several storage drivers. This allows workflows and clients to be able to retrieve payloads that have been stored by different driver types.
• Customers can dynamically choose which driver to use for storing each payload by specifying a driver selector. The selector can be a simple callable or could be a StorageDriverSelector to participate in serialization context transforms.
• External storage is situated outside of the workflow sandbox and occurs between payload conversion and codec application. This allows customers to have the best visibility for inspecting payload content to make storage decisions and to apply separate codec behavior, if desired.
• External storage is off by default and must be opted into via specifying DataConverter.external_storage. There are no breaking changes regardless of opted into or not.

Why?

• Allow customers to mitigate situations where payload size may be too large to store into temporal history.
• Allow customers to externally store payloads based on other payload and contextual qualities such a serialization context and payload metadata.

Checklist

1. Closes SDK External Payload Storage #1342
2. How was this tested: Add additional unit and workflow tests.
3. Any docs updates needed? Yes, new docs for external storage should be created.

[drewhoskins-temporal]

Very excited! One important comment (which we already covered in our afternoon sync) and a bunch of polish suggestions.

[drewhoskins-temporal]

nit: can we be more specific than data ? It's super vague, hard to grep for, and could be confused with the payload itself. Maybe reference_metadata or claim_metadata or claim or something?

Also, is there nothing standardized here? Like the name of the storage driver? I feel like users building custom drivers will want a bit more guidance, and the community could likely benefit from some commonality.