Consistently raise `ContentFilterError` when model response is empty because of content filter #3634

AlanPonnachan · 2025-12-03T17:52:16Z

Unified Content Filtering Exception Handling

This PR closes #1035

Standardizing how content filtering events are handled across different model providers.

Previously, triggering a content filter resulted in inconsistent behaviors: generic ModelHTTPError (Azure), UnexpectedModelBehavior (Google), or silent failures depending on the provider. This PR introduces a dedicated exception hierarchy to allow users to catch and handle prompt refusals and response interruptions programmatically and consistently.

Key Changes:

New Exceptions: Added ContentFilterError (base), PromptContentFilterError (for input rejections, e.g., Azure 400), and ResponseContentFilterError (for output refusals).
OpenAI & Azure: Updated logic to raise PromptContentFilterError for Azure's specific 400 error body and ResponseContentFilterError when finish_reason='content_filter'.
Google Gemini: Updated _process_response to raise ResponseContentFilterError instead of UnexpectedModelBehavior when safety thresholds are triggered.
Anthropic: Added mapping for refusal stop reasons to raise ResponseContentFilterError.
Tests: Added comprehensive tests covering synchronous and streaming scenarios for OpenAI, Google, and Anthropic in tests/models/.

Example Usage:

from pydantic_ai import Agent
from pydantic_ai.exceptions import ContentFilterError

agent = Agent('openai:gpt-4o')

try:
    # If the prompt or generation triggers a safety filter
    await agent.run("Generate unsafe content...")
except ContentFilterError as e:
    # Catches both PromptContentFilterError and ResponseContentFilterError
    print(f"Request halted by safety filter: {e}")

rahim-figs · 2025-12-04T12:54:34Z

Is it possible to handle AWS Bedrock as well? Thanks.

dsfaccini

Hey @AlanPonnachan thank you for the PR! I've requested a couple small changes, let me know if you have any questions

pydantic_ai_slim/pydantic_ai/exceptions.py

pydantic_ai_slim/pydantic_ai/models/anthropic.py

pydantic_ai_slim/pydantic_ai/models/openai.py

tests/models/test_anthropic.py

tests/models/test_google.py

tests/models/test_openai.py

dsfaccini · 2025-12-04T19:25:57Z

one more thing I missed, please include the name of the new exception in the fallback model docs

ModelAPIError, which includes ModelHTTPError and ...

AlanPonnachan · 2025-12-05T11:06:46Z

@dsfaccini Thank you for the review. I’ve made the requested changes.

pydantic_ai_slim/pydantic_ai/models/openai.py

tests/models/test_openai.py

pydantic_ai_slim/pydantic_ai/exceptions.py

pydantic_ai_slim/pydantic_ai/models/openai.py

dsfaccini · 2025-12-05T16:31:33Z

hey @AlanPonnachan thanks a lot for your work! It looks very good now, I requested a couple more changes but once that's done and coverage passes I think the PR will be ready for merge.

AlanPonnachan · 2025-12-06T06:33:59Z

@dsfaccini Thanks again for the review! I’ve applied the requested changes. Test coverage is now at 100%.

DouweM

@AlanPonnachan Thanks for working on this! My main concern is that this doesn't actually make it consistent for all models that respond with finish_reason=='content_filter', just for Anthropic/Google/OpenAI.

pydantic_ai_slim/pydantic_ai/exceptions.py

pydantic_ai_slim/pydantic_ai/models/anthropic.py

pydantic_ai_slim/pydantic_ai/models/google.py

pydantic_ai_slim/pydantic_ai/models/openai.py

docs/models/overview.md

pydantic_ai_slim/pydantic_ai/_agent_graph.py

DouweM · 2025-12-12T00:15:39Z

pydantic_ai_slim/pydantic_ai/models/openai.py

+                        chat.chat_completion.Choice(
+                            finish_reason='content_filter',
+                            index=0,
+                            message=chat.ChatCompletionMessage(content='', role='assistant'),


Do we need this? I'd want it to result in a ModelResponse with 0 parts, and this may result in a TextPart(content='')

It may be cleaner to just build that ModelResponse directly...

a ModelResponse directly would be cleaner but _completions_create is strictly typed to return ChatCompletion (or AsyncStream).
I updated to constructing a ChatCompletion with content=None (not '') which will successfully results in a ModelResponse with 0 parts via _process_response.

AlanPonnachan · 2025-12-13T17:52:38Z

@DouweM , I’ve made a few changes. Let me know your thoughts.

DouweM · 2025-12-15T17:55:13Z

pydantic_ai_slim/pydantic_ai/exceptions.py



+class ContentFilterError(UnexpectedModelBehavior):
+    """Raised when content filtering is triggered by the model provider."""


Suggested change

"""Raised when content filtering is triggered by the model provider."""

"""Raised when content filtering is triggered by the model provider resulting in an empty response."""

DouweM · 2025-12-15T17:57:45Z

pydantic_ai_slim/pydantic_ai/models/openai.py



+def _check_azure_content_filter(e: APIStatusError) -> bool:
+    """Check if the error is an Azure content filter error."""


Can we do this only if self.system == 'azure'?

DouweM · 2025-12-15T18:01:16Z

pydantic_ai_slim/pydantic_ai/_agent_graph.py

+                    # Check for content filter on empty response
+                    if self.model_response.finish_reason == 'content_filter':
+                        raise exceptions.ContentFilterError(
+                            f'Content filter triggered for model {self.model_response.model_name}'


Instead of naming the model (which we don't do anywhere else), I'd prefer to include model_response.provider_details['finish_reason'] if it exists, like we did in the Google exception

Related to what I wrote in google.py about storing more of the actual error response context on provider_details, what do you think about including (JSONified) model_response on this exception, so that it can be accessed by the user (or in an observability platform)?

DouweM · 2025-12-15T18:01:52Z

tests/models/test_google.py

    )
+
+
+async def test_google_stream_empty_chunk(


Do we need this?

DouweM · 2025-12-15T18:03:57Z

pydantic_ai_slim/pydantic_ai/models/google.py

        raw_finish_reason = candidate.finish_reason
        if raw_finish_reason:  # pragma: no branch
            vendor_details = {'finish_reason': raw_finish_reason.value}
            finish_reason = _FINISH_REASON_MAP.get(raw_finish_reason)


Are you able to test against Vertex AI?

It may be worth exposing these extra data points on ModelResponse.provider_details https://docs.cloud.google.com/vertex-ai/generative-ai/docs/multimodal/configure-safety-filters#gemini-api-in-vertex-ai

Similar for the content_filter_result returned by Azure:

{ 'message': "The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766", 'type': None, 'param': 'prompt', 'code': 'content_filter', 'status': 400, 'innererror': { 'code': 'ResponsibleAIPolicyViolation', 'content_filter_result': { 'hate': { 'filtered': True, 'severity': 'high' }, 'jailbreak': { 'filtered': False, 'detected': False }, 'self_harm': { 'filtered': False, 'severity': 'safe' }, 'sexual': { 'filtered': False, 'severity': 'safe' }, 'violence': { 'filtered': False, 'severity': 'medium' } } } }

AlanPonnachan and others added 2 commits December 3, 2025 17:35

add content filtering exceptions

4ac81d4

Merge branch 'main' into feature/content-filtering-exceptions

658f407

dsfaccini requested changes Dec 4, 2025

View reviewed changes

AlanPonnachan and others added 5 commits December 5, 2025 09:31

resolve issues

404a833

add docs

3771c79

Merge branch 'main' into feature/content-filtering-exceptions

28e20c8

test coverage

70bcb74

improve tests

e861a50

dsfaccini requested changes Dec 5, 2025

View reviewed changes

AlanPonnachan added 3 commits December 6, 2025 05:13

resolve issues

51506b2

test coverage

b67d216

fix ci fails

4ac1608

DouweM requested changes Dec 9, 2025

View reviewed changes

DouweM self-assigned this Dec 9, 2025

DouweM added the awaiting author revision label Dec 9, 2025

AlanPonnachan and others added 2 commits December 11, 2025 17:30

make consistent for all models

264f9e6

Merge branch 'main' into feature/content-filtering-exceptions

ed242de

DouweM requested changes Dec 12, 2025

View reviewed changes

AlanPonnachan and others added 4 commits December 13, 2025 15:02

improvements

a0a85cb

Merge branch 'main' into feature/content-filtering-exceptions

0327196

test coverage

bba451d

test coverage

d969325

DouweM changed the title ~~Feat: Content Filtering Exception Handling~~ Consistently raise ContentFilterError when model response is empty because of content filter Dec 15, 2025

DouweM requested changes Dec 15, 2025

View reviewed changes



		class ContentFilterError(UnexpectedModelBehavior):
		"""Raised when content filtering is triggered by the model provider."""



		def _check_azure_content_filter(e: APIStatusError) -> bool:
		"""Check if the error is an Azure content filter error."""

Consistently raise ContentFilterError when model response is empty because of content filter #3634

Are you sure you want to change the base?

Consistently raise ContentFilterError when model response is empty because of content filter #3634

Conversation

AlanPonnachan commented Dec 3, 2025

Unified Content Filtering Exception Handling

Uh oh!

rahim-figs commented Dec 4, 2025

Uh oh!

dsfaccini left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dsfaccini commented Dec 4, 2025

Uh oh!

AlanPonnachan commented Dec 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dsfaccini commented Dec 5, 2025

Uh oh!

AlanPonnachan commented Dec 6, 2025

Uh oh!

DouweM left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlanPonnachan commented Dec 13, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Consistently raise `ContentFilterError` when model response is empty because of content filter #3634

Consistently raise `ContentFilterError` when model response is empty because of content filter #3634