Fix BlobNotFound error in entity history retrieval#1269
Closed
torosent wants to merge 2 commits intoAzure:mainfrom
Closed
Fix BlobNotFound error in entity history retrieval#1269torosent wants to merge 2 commits intoAzure:mainfrom
torosent wants to merge 2 commits intoAzure:mainfrom
Conversation
Fixes Azure/azure-functions-durable-extension#3264 When retrieving history events for entities with large properties stored in blob storage, a BlobNotFound error can occur if the blob was deleted due to a race condition (e.g., ContinueAsNew deleted blobs from a previous execution while a late message is still being processed). This change extends the fix from PR #1189 to also handle the entity-specific code path in DecompressLargeEntityProperties: 1. Changed DecompressLargeEntityProperties to return bool indicating success 2. Added try-catch for RequestFailedException with status 404 (BlobNotFound) 3. Also handles the wrapped DurableTaskStorageException case 4. When blob is not found, logs a warning and returns false 5. Caller skips the history entry instead of failing the orchestration This allows the orchestration to continue processing even when stale history entries reference blobs that no longer exist.
…erties These tests verify: 1. Method signature returns Task<bool> to indicate success/failure 2. RequestFailedException with status 404 can be caught by the when clause 3. DurableTaskStorageException wrapping 404 can also be caught 4. Other status codes (e.g., 500) are NOT caught by the 404 filter See: Azure/azure-functions-durable-extension#3264
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes Azure/azure-functions-durable-extension#3264
This PR extends the fix from #1189 to also handle the entity-specific code path in
DecompressLargeEntityProperties.Problem
When retrieving history events for entities (instance IDs starting with
@) with large properties stored in blob storage, aBlobNotFounderror can occur under heavy load. This happens due to a race condition similar to what PR #1189 addressed for orchestrations:ContinueAsNewdeleted the blobsThe stack trace shows the failure occurring in
DecompressLargeEntityProperties:While PR #1189 added execution ID validation at the history retrieval level, there's still a window where:
DecompressLargeEntityPropertiescallSolution
DecompressLargeEntityPropertiesto returnboolindicating successRequestFailedExceptionwith status 404 (BlobNotFound)DurableTaskStorageExceptioncasefalseThis allows the orchestration to continue processing even when stale history entries reference blobs that no longer exist.
Testing
Related Issues