Minimizing RAM usage (feature)

[desiprisg]

While trying some other providers (coming from vertex), I noticed the usual RAM usage spike because arbitrary URLs aren't supported in their API and ai sdk will download and pass them as base64 strings. Haven't done a deep dive yet, but I believe that there's not anything "smart" going one behind the scenes.

Most codebases will be using URLs with the lib, and not conditionally swap to base64 to try to minimize RAM usage (and I think that's the point of this library too, to abstract these things away). I find it quite frequent (at least in my codebase) that the same files are being used in more than one generation, at the same time. This seems to mean that RAM usage is multiplied.

I thought that adding some shared state here for attachments would be a good idea, where the lib could reuse the already occupied memory for any other streamText/Object that comes in. I know this essentially means that you "cache" the URL, but that would only be the case for as long as an active generation is using that URL, so it doesn't seem like that big of a deal.

Essentially,

• generateText A with file part example.com/foo.pdf starts
• generateText B with file part example.com/foo.pdf starts
• Both use the same downloaded base64 string from this shared state (an object with key being the URL would do)

[github-actions[bot]]

This discussion was automatically locked because it has not been updated in over 30 days. If you still have questions about this topic, please ask us at community.vercel.com/ai-sdk