Add `force_download` support for Anthropic and OAIR models and clarify proper `BinaryContent` base64 handling #3694

dsfaccini · 2025-12-10T12:50:25Z

Closes #3569 - Review multi-modal input handling.

Changes

BinaryContent improvements:

Add base64 property that returns raw base64-encoded string
Update data field docstring to clarify it should contain raw bytes, not base64-encoded strings

Anthropic provider:

Add force_download support for ImageUrl - downloads and sends as base64 when force_download=True
Add text/plain support for BinaryContent - decodes UTF-8 and sends as plain text document
Extract _map_binary_content() helper to reduce complexity of _map_user_prompt()

Test fixes:

Fix test_mcp_sampling.py: was storing base64.b64encode(b'data') instead of raw b'data'
Fix test_mistral.py: was storing base64 ASCII characters as bytes
Update test_anthropic.py error message match for audio content

OpenAI Responses API:

Add force_download support for AudioUrl and DocumentUrl - uses file_url to send URLs directly when force_download=False (default), downloads and sends as file_data when force_download=True
Consolidate AudioUrl and DocumentUrl handling into single branch

New tests:

test_binary_content_base64 - verifies new property
test_image_url_input_force_download - verifies Anthropic force_download
test_text_document_as_binary_content_input - verifies Anthropic text/plain support
test_document_url_input_response_api - verifies URL is sent directly (default behavior)
test_document_url_input_force_download_response_api - verifies download when force_download=True

github-actions · 2025-12-10T13:06:56Z

Docs Preview

commit:	`45d35e5`
Preview URL:	https://9f96efab-pydantic-ai-previews.pydantic.workers.dev

dsfaccini · 2025-12-10T14:44:27Z

work's still ongoing to address the other points mentioned in the issue

pydantic_ai_slim/pydantic_ai/models/anthropic.py

DouweM · 2025-12-10T20:46:37Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

+                    yield AnthropicModel._map_binary_content(item)
+                elif isinstance(item, ImageUrl):
+                    if item.force_download:
+                        downloaded = await download_item(item, data_format='bytes')


We should also respect force_download for DocumentUrl + item.media_type == 'application/pdf' further down, right?

Would it make sense to use _map_binary_content there, if we make it more generic so it can take the result of download_item? (Or take a FileUrl | BinaryContent and if it gets a FileUrl, do the download first)

DouweM · 2025-12-10T20:47:38Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

+                type='image',
+            )
+        elif item.media_type == 'application/pdf':
+            return BetaBase64PDFBlockParam(


This is now an alias for BetaRequestDocumentBlockParam, so let's use the newer name

DouweM · 2025-12-10T20:49:49Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

+            )
+        elif item.media_type == 'application/pdf':
+            return BetaBase64PDFBlockParam(
+                source=BetaBase64PDFSourceParam(


I'm looking at what other sources this supports, and there's also BetaFileDocumentSourceParam, which takes a file_id for the file upload API.

We're adding support for uploaded files in #2611, but that PR has been stale for a bit so may be interesting for you to pick up.

DouweM · 2025-12-10T20:53:46Z

tests/models/test_anthropic.py

+            ),
+        ]
+    )
+    assert result.output == snapshot(


Can we test that the forced download is actually happening?

DouweM · 2025-12-10T20:56:36Z

tests/models/test_openai.py

+
+    document_url = DocumentUrl(
+        url='https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf',
+        force_download=True,


Same as up -- we're not testing that we're respecting this, although I suppose that fact that those lines have test coverage sort of does? But it'd be good if we could assert that explicitly

pydantic_ai_slim/pydantic_ai/messages.py

This commit addresses Douwe's review comments on PR #3694: 1. **Anthropic: Add force_download for DocumentUrl PDF** - DocumentUrl with PDF now respects force_download parameter - When force_download=True: downloads and sends as base64 - When force_download=False: sends URL directly 2. **Anthropic: Rename BetaBase64PDFBlockParam → BetaRequestDocumentBlockParam** - Updated to use newer type name (BetaBase64PDFBlockParam is now an alias) - Updated imports and all usages 3. **Anthropic: Refactor _map_binary_content to handle FileUrl** - Created new _map_binary_to_block helper method - Eliminates code duplication between BinaryContent, ImageUrl force_download, and DocumentUrl force_download - Supports image/*, application/pdf, and text/plain media types 4. **Messages: Use base64 property in otel_message_parts** - Replaced manual base64.b64encode calls with item.base64 property - 3 replacements in UserPromptPart.otel_message_parts and ModelResponse.otel_* methods 5. **Tests: Add force_download verification tests** - 4 new tests for Anthropic (ImageUrl + DocumentUrl) - 5 new tests for OpenAI (ImageUrl + DocumentUrl + AudioUrl) - Tests verify download_item is called when force_download=True All tests pass (306 tests in test_anthropic.py, test_openai.py, test_messages.py). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <[email protected]>

DouweM · 2025-12-12T20:42:52Z

pydantic_ai_slim/pydantic_ai/messages.py


    data: bytes
-    """The binary data."""
+    """Arbitrary binary data.


"Arbitrary" may be a little too strong, as it is specifically a file matching the type in media_type 😄

I think if we make this The binary file data, we can remove the specific mention of base64 that's added in the next lines

I'd like to keep one as it reduces the likelihood of users using it incorrectly

DouweM · 2025-12-12T20:44:37Z

pydantic_ai_slim/pydantic_ai/models/anthropic.py

+            raise RuntimeError(f'Unsupported binary content media type for Anthropic: {media_type}')
+
+    @staticmethod
+    def _map_binary_content(item: BinaryContent) -> BetaContentBlockParam:


We don't need this method anymore as it's just one line that can be inlined

I wonder if download_item could be refactored to return a BinaryContent

would've been cool if it had been like BinaryContent.download or .from_url but I don't think the replacement is worth the hassle now

pydantic_ai_slim/pydantic_ai/models/anthropic.py

dsfaccini · 2025-12-13T04:26:27Z

changed the docs as follows

force_download snippet -> better than describing it (personally, but also for llms)

dsfaccini · 2025-12-14T16:43:38Z

Addendum

force_download=False passthrough mode (addresses #3724):

Changed force_download from bool to bool | None to support explicit URL passthrough:

Value	Behavior
None (default)	Provider decides - downloads if needed, sends URL if supported
True	Always download content locally and send as bytes
False	Never download - pass URL directly to API (useful for proxies)

This enables use cases where users have middleware/proxies that intercept requests and handle custom URL schemes (e.g., file://).

Provider changes:

All providers: Updated conditionals from if item.force_download: to if item.force_download is True:
Google/Gemini: force_download=False bypasses google-gla download requirement
Anthropic: force_download=False on text/plain DocumentUrl passes URL through (API will reject, it though unless the user knows what they're doing)
Bedrock: Raises RuntimeError when force_download=False since Bedrock doesn't support URLs at all

New tests:

test_document_url_passthrough_non_http_url - verifies non-http URLs pass through without validation
test_url_force_download_false_raises_error - verifies Bedrock raises clear error
test_text_document_url_input_no_force_download - verifies Anthropic API error for unsupported passthrough

Docs:

Updated docs/input.md with force_download behavior table and examples

This is how the doc looks like after this change:

DouweM · 2025-12-15T18:24:59Z

docs/input.md

 ## User-side download vs. direct file URL

-When you provide a URL using any of `ImageUrl`, `AudioUrl`, `VideoUrl` or `DocumentUrl`, Pydantic AI will typically send the URL directly to the model API so that the download happens on their side.
+When using one of `ImageUrl`, `AudioUrl`, `VideoUrl` or `DocumentUrl`, Pydantic AI will default to sending the URL to the model, so the file is downloaded on their side.


Suggested change

When using one of `ImageUrl`, `AudioUrl`, `VideoUrl` or `DocumentUrl`, Pydantic AI will default to sending the URL to the model, so the file is downloaded on their side.

When using one of `ImageUrl`, `AudioUrl`, `VideoUrl` or `DocumentUrl`, Pydantic AI will default to sending the URL to the model provider, so the file is downloaded on their side.

DouweM · 2025-12-15T18:25:18Z

docs/input.md

+When using one of `ImageUrl`, `AudioUrl`, `VideoUrl` or `DocumentUrl`, Pydantic AI will default to sending the URL to the model, so the file is downloaded on their side.

-Some model APIs do not support file URLs at all or for specific file types. In the following cases, Pydantic AI will download the file content and send it as part of the API request instead:
+Support for file URLs varies depending on type and provider. Pydantic AI handles this as follows:


Suggested change

Support for file URLs varies depending on type and provider. Pydantic AI handles this as follows:

Support for file URLs varies depending on type and provider:

DouweM · 2025-12-15T18:27:48Z

docs/input.md

-When you provide a URL using any of `ImageUrl`, `AudioUrl`, `VideoUrl` or `DocumentUrl`, Pydantic AI will typically send the URL directly to the model API so that the download happens on their side.
+When using one of `ImageUrl`, `AudioUrl`, `VideoUrl` or `DocumentUrl`, Pydantic AI will default to sending the URL to the model, so the file is downloaded on their side.

-Some model APIs do not support file URLs at all or for specific file types. In the following cases, Pydantic AI will download the file content and send it as part of the API request instead:


We still need a sentence like this to explain what will happen if the table below has "Sends URL directly" as false, especially because the "false" is not explicitly stated but implied because of something like "ImageUrl, AudioUrl, DocumentUrl | ImageUrl only". It's not currently clear to the user what happens for AudioUrl and DocumentUrl

Maybe it should be columns: "Send URL directly | Download URL and send bytes | Unsupported", listing each type explicitly for each provider

DouweM · 2025-12-15T18:35:47Z

docs/input.md

+DocumentUrl(url='https://example.com/doc.pdf', force_download=True)
+```
+
+Use `force_download=False` to pass URLs directly without any download attempt. This is useful when you have a proxy or middleware that intercepts requests:


So this is only relevant for models that support URLs of a given type, but for some URLs of that type prefer a download anyway, and now we're giving the user the option to in those cases send the URL anyway?

I think that's way too niche a feature to support, as it just affects people putting a file-URL-interpreting proxy in front of Google GLA, and the specific Anthropic+text/plain combination, which seems incredibly uncommon.

This doesn't solve #3724 problem anyway, as they were using OpenAIChatModel which doesn't support document URLs anyway, only image URLs.

So please revert these changes, reopen that issue, and maybe we can find out what he would expect to happen because OpenAI Chat doesn't have a way of passing document URLs though anyway.

Edit: Also note that this is the risk of adding scope to an existing PR; I may have been able to merge this now and then we could have done this new feature in a followup PR, but now this entire PR is blocked waiting for it to be removed

DouweM · 2025-12-15T18:36:15Z

experiments/test_anthropic_changes.py

No experiments please

review multi modal input handling

c642ac7

dsfaccini marked this pull request as draft December 10, 2025 14:44

enable url media for openai responses

754c0f0

dsfaccini changed the title ~~Add force_download support for anthropic model and clarify proper BinaryContent base64 handling~~ Add force_download support for Anthropic and OAIR models and clarify proper BinaryContent base64 handling Dec 10, 2025

Update documentation for BinaryContent.base64 usage

2f5bcf6

dsfaccini marked this pull request as ready for review December 10, 2025 18:02

dsfaccini changed the title ~~Add force_download support for Anthropic and OAIR models and clarify proper BinaryContent base64 handling~~ Add force_download support for Anthropic and OAIR models and clarify proper BinaryContent base64 handling Dec 10, 2025

dsfaccini marked this pull request as draft December 10, 2025 18:06

dsfaccini commented Dec 10, 2025

View reviewed changes

pydantic_ai_slim/pydantic_ai/models/anthropic.py Show resolved Hide resolved

dsfaccini marked this pull request as ready for review December 10, 2025 18:21

dsfaccini requested a review from DouweM December 10, 2025 18:22

dsfaccini added 2 commits December 10, 2025 13:22

Merge branch 'main' into review-multimodal

5d99ed2

Merge branch 'main' into review-multimodal

ff5d79a

DouweM requested changes Dec 10, 2025

View reviewed changes

DouweM self-assigned this Dec 10, 2025

DouweM added the awaiting author revision label Dec 10, 2025

dsfaccini and others added 5 commits December 10, 2025 17:48

Merge branch 'main' into review-multimodal

887a79f

upstream cerebras merge stuff

7389467

AsyncClient

8ad810d

base64 replacements

da9ec78

dsfaccini mentioned this pull request Dec 11, 2025

Cohere: Implement multimodal support for Command A Vision #3703

Open

add force download support for mistral

9a55ce2

DouweM requested changes Dec 12, 2025

View reviewed changes

dsfaccini added 2 commits December 12, 2025 19:58

address review comments

703b772

update docs

35d8745

allow explicit download disallowing

d9e8a01

dsfaccini added 2 commits December 14, 2025 09:25

Merge remote-tracking branch 'origin/main' into review-multimodal

0bcd2e8

fix linting issue and update doc

8a98f77

dsfaccini mentioned this pull request Dec 14, 2025

Is it possible to disable downloads + URL check for DocumentUrl when using openai provider? #3724

Closed

fix tests

45d35e5

DouweM requested changes Dec 15, 2025

View reviewed changes

	When using one of `ImageUrl`, `AudioUrl`, `VideoUrl` or `DocumentUrl`, Pydantic AI will default to sending the URL to the model, so the file is downloaded on their side.
	When using one of `ImageUrl`, `AudioUrl`, `VideoUrl` or `DocumentUrl`, Pydantic AI will default to sending the URL to the model provider, so the file is downloaded on their side.

	Support for file URLs varies depending on type and provider. Pydantic AI handles this as follows:
	Support for file URLs varies depending on type and provider:

Add force_download support for Anthropic and OAIR models and clarify proper BinaryContent base64 handling #3694

Are you sure you want to change the base?

Add force_download support for Anthropic and OAIR models and clarify proper BinaryContent base64 handling #3694

Uh oh!

Conversation

dsfaccini commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Uh oh!

github-actions bot commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Docs Preview

Uh oh!

dsfaccini commented Dec 10, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dsfaccini commented Dec 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dsfaccini commented Dec 14, 2025

Addendum

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

DouweM Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add `force_download` support for Anthropic and OAIR models and clarify proper `BinaryContent` base64 handling #3694

Add `force_download` support for Anthropic and OAIR models and clarify proper `BinaryContent` base64 handling #3694

dsfaccini commented Dec 10, 2025 •

edited

Loading

github-actions bot commented Dec 10, 2025 •

edited

Loading

dsfaccini commented Dec 13, 2025 •

edited

Loading

DouweM Dec 15, 2025 •

edited

Loading