Skip to content

perf: Optimize metadata records processing in SqlStorageClient #1551

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Mantisus wants to merge 23 commits into
base: master
Choose a base branch
from
+611 −238
Open

perf: Optimize metadata records processing in SqlStorageClient #1551

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
5d0802b
add buffer tables
Mantisus Nov 10, 2025
4b9386f
some optimization
Mantisus Nov 11, 2025
7848a21
index optimization
Mantisus Nov 11, 2025
3665a4c
fix
Mantisus Nov 11, 2025
4236b20
Merge branch 'master' into sql-metadata-buffer
Mantisus Nov 11, 2025
4e74b70
recalculate only for is_empty
Mantisus Nov 11, 2025
251253c
add lenght for all String fields
Mantisus Nov 14, 2025
d7171db
Merge branch 'master' into sql-metadata-buffer
Mantisus Nov 14, 2025
c8054b9
consistent use `_update_metadata`
Mantisus Nov 16, 2025
83209ce
up block time interval
Mantisus Nov 16, 2025
1ee46da
fix
Mantisus Nov 16, 2025
d4cd262
up lengrh for data
Mantisus Nov 16, 2025
eae231d
up docs
Mantisus Nov 17, 2025
c3fd6e8
Merge branch 'master' into sql-metadata-buffer
Mantisus Jan 26, 2026
e0d07b4
remove foreign key constraint for buffer tables
Mantisus Jan 26, 2026
4ed54a9
Update src/crawlee/storage_clients/_sql/_request_queue_client.py
Mantisus Jan 26, 2026
aea99e7
fixes
Mantisus Jan 26, 2026
7bf9e1e
merge
Mantisus Jan 26, 2026
c48f179
Update src/crawlee/storage_clients/_sql/_db_models.py
Mantisus Jan 26, 2026
24fe92d
Update src/crawlee/storage_clients/_sql/_client_mixin.py
Mantisus Jan 26, 2026
83225de
Update src/crawlee/storage_clients/_sql/_request_queue_client.py
Mantisus Jan 26, 2026
bedcd8d
update types
Mantisus Jan 26, 2026
9d143a7
change level message
Mantisus Jan 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 36 additions & 0 deletions docs/guides/storage_clients.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -199,6 +199,7 @@ class datasets {
+ created_at
+ modified_at
+ item_count
+ buffer_locked_until
}

class dataset_records {
Expand All @@ -208,6 +209,15 @@ class dataset_records {
+ data
}

class dataset_metadata_buffer {
<<table>>
+ id (PK)
+ accessed_at
+ modified_at
+ dataset_id (FK)
+ delta_item_count
}

%% ========================
%% Key-Value Store Tables
%% ========================
Expand All @@ -220,6 +230,7 @@ class key_value_stores {
+ accessed_at
+ created_at
+ modified_at
+ buffer_locked_until
}

class key_value_store_records {
Expand All @@ -231,15 +242,25 @@ class key_value_store_records {
+ size
}

class key_value_store_metadata_buffer {
<<table>>
+ id (PK)
+ accessed_at
+ modified_at
+ key_value_store_id (FK)
}

%% ========================
%% Client to Table arrows
%% ========================

SqlDatasetClient --> datasets
SqlDatasetClient --> dataset_records
SqlDatasetClient --> dataset_metadata_buffer

SqlKeyValueStoreClient --> key_value_stores
SqlKeyValueStoreClient --> key_value_store_records
SqlKeyValueStoreClient --> key_value_store_metadata_buffer
```
```mermaid
---
Expand Down Expand Up @@ -274,6 +295,7 @@ class request_queues {
+ handled_request_count
+ pending_request_count
+ total_request_count
+ buffer_locked_until
}

class request_queue_records {
Expand All @@ -294,13 +316,27 @@ class request_queue_state {
+ forefront_sequence_counter
}

class request_queue_metadata_buffer {
<<table>>
+ id (PK)
+ accessed_at
+ modified_at
+ request_queue_id (FK)
+ client_id
+ delta_handled_count
+ delta_pending_count
+ delta_total_count
+ need_recalc
}

%% ========================
%% Client to Table arrows
%% ========================

SqlRequestQueueClient --> request_queues
SqlRequestQueueClient --> request_queue_records
SqlRequestQueueClient --> request_queue_state
SqlRequestQueueClient --> request_queue_metadata_buffer
```

Configuration options for the <ApiLink to="class/SqlStorageClient">`SqlStorageClient`</ApiLink> can be set through environment variables or the <ApiLink to="class/Configuration">`Configuration`</ApiLink> class:
Expand Down
Loading
Loading