feat(async): Set priorities to async tasks#14195
Open
kiblik wants to merge 10000 commits intoDefectDojo:devfrom
Open
feat(async): Set priorities to async tasks#14195kiblik wants to merge 10000 commits intoDefectDojo:devfrom
kiblik wants to merge 10000 commits intoDefectDojo:devfrom
Conversation
…tialization command (DefectDojo#14002)
* dedupe reopen: add test cases that prove the bug * remove obsolete method * dedupe reopen: proceed with next candidate if candidate is mitigated * rename methods
…Dojo#14017) * added code to remove unwanted vulnerability ids * Update dojo/finding/helper.py --------- Co-authored-by: Cody Maffucci <46459665+Maffooch@users.noreply.github.com>
…efectDojo#14018) Bumps [drf-spectacular-sidecar](https://github.com/tfranzel/drf-spectacular-sidecar) from 2025.12.1 to 2026.1.1. - [Commits](tfranzel/drf-spectacular-sidecar@2025.12.1...2026.1.1) --- updated-dependencies: - dependency-name: drf-spectacular-sidecar dependency-version: 2026.1.1 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…14019) Bumps [django-polymorphic](https://github.com/jazzband/django-polymorphic) from 4.5.1 to 4.5.2. - [Release notes](https://github.com/jazzband/django-polymorphic/releases) - [Changelog](https://github.com/jazzband/django-polymorphic/blob/master/docs/changelog.rst) - [Commits](jazzband/django-polymorphic@v4.5.1...v4.5.2) --- updated-dependencies: - dependency-name: django-polymorphic dependency-version: 4.5.2 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [pillow](https://github.com/python-pillow/Pillow) from 12.0.0 to 12.1.0. - [Release notes](https://github.com/python-pillow/Pillow/releases) - [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst) - [Commits](python-pillow/Pillow@12.0.0...12.1.0) --- updated-dependencies: - dependency-name: pillow dependency-version: 12.1.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…thub/workflows/renovate.yaml) (DefectDojo#14025) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…efectDojo#14026) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…Dojo#14021) Bumps [pdfmake](https://github.com/bpampuch/pdfmake) from 0.2.21 to 0.3.0. - [Release notes](https://github.com/bpampuch/pdfmake/releases) - [Changelog](https://github.com/bpampuch/pdfmake/blob/master/CHANGELOG.md) - [Commits](bpampuch/pdfmake@0.2.21...0.3.0) --- updated-dependencies: - dependency-name: pdfmake dependency-version: 0.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [gitpython](https://github.com/gitpython-developers/GitPython) from 3.1.45 to 3.1.46. - [Release notes](https://github.com/gitpython-developers/GitPython/releases) - [Changelog](https://github.com/gitpython-developers/GitPython/blob/main/CHANGES) - [Commits](gitpython-developers/GitPython@3.1.45...3.1.46) --- updated-dependencies: - dependency-name: gitpython dependency-version: 3.1.46 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…json) (DefectDojo#14023) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Bumps [celery](https://github.com/celery/celery) from 5.6.0 to 5.6.1. - [Release notes](https://github.com/celery/celery/releases) - [Changelog](https://github.com/celery/celery/blob/main/Changelog.rst) - [Commits](celery/celery@v5.6.0...v5.6.1) --- updated-dependencies: - dependency-name: celery dependency-version: 5.6.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ckerfile.integration-tests-debian) (DefectDojo#14008) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Add permission classes and refine queryset in BurpRawRequestResponseViewSet
docs: add new page on custom trust
[docs] Pro changelog: Dec 30
[docs] create sitemap at root on Hugo deploy
Release 2.54.0: Merge Bugfix into Dev
Release: Merge release into master from: release/2.54.0
…x/2.54.0-2.55.0-dev Release: Merge back 2.54.0 into bugfix from: master-into-bugfix/2.54.0-2.55.0-dev
….54.0-2.55.0-dev Release: Merge back 2.54.0 into dev from: master-into-dev/2.54.0-2.55.0-dev
…14030) Bumps [django-polymorphic](https://github.com/jazzband/django-polymorphic) from 4.5.2 to 4.6.0. - [Release notes](https://github.com/jazzband/django-polymorphic/releases) - [Changelog](https://github.com/jazzband/django-polymorphic/blob/master/docs/changelog.rst) - [Commits](jazzband/django-polymorphic@v4.5.2...v4.6.0) --- updated-dependencies: - dependency-name: django-polymorphic dependency-version: 4.6.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…efectDojo#14266) The `has_jira_issue` filter only checks if the finding itself has a JIRA issue. For findings in a group where the group has a JIRA issue, this filter doesn't match. A `FindingHasJIRAFilter` that checks both already existed but was never wired up to be visible in the UI or API. Rename it to `has_any_jira_issue`, add help text, and include it in `get_finding_filterset_fields()` so it appears in the finding filters when both JIRA and finding groups are enabled. Closes DefectDojo#12670
…ctDojo#14267) * Add finding group support to jira_status_reconciliation command The jira_status_reconciliation management command only processed individual findings with direct JIRA issues. Finding groups that were pushed to JIRA as groups were completely skipped because their JIRA issue is attached to the Finding_Group model, not to individual findings. This adds a second processing loop for Finding_Group objects with JIRA issues, supporting all three modes (reconcile, push_status_to_jira, import_status_from_jira). The group's aggregate status is derived from its member findings. To avoid pushing the same JIRA issue twice, we use push_status_to_jira directly on the group object (not push_finding_group_to_jira which would also push individual finding JIRA issues already handled by the existing loop). Also adds --include-findings/--no-include-findings and --include-finding-groups/--no-include-finding-groups flags so users can control which types are processed. Closes DefectDojo#14031 * add upgrade notes
…Dojo#14262) * Enhance JIRA synchronization logic in importers and serializers - Updated push_to_jira conditions to include sync behavior based on JIRA instance settings. - Refactored JIRA push logic to check for sync status in FindingSerializer and DefaultImporter. - Improved handling of JIRA instance retrieval and sync checks in DefaultReImporter and BaseImporter. - Added support for prefetched JIRA instance in is_keep_in_sync_with_jira function. * Refactor JIRA sync flag to use 'finding_jira_sync' for consistency in importers and reimporters * Refactor is_keep_in_sync_with_jira function to use a generic object parameter for improved flexibility * Refactor is_keep_in_sync_with_jira function to improve JIRA issue detection and sync logic * Add tests * bulk edit: push groups to JIRA when sync is enabled (DefectDojo#14265) --------- Co-authored-by: valentijnscholten <valentijnscholten@gmail.com>
Release: Merge release into master from: release/2.55.2
….55.2-2.56.0-dev Release: Merge back 2.55.2 into dev from: master-into-dev/2.55.2-2.56.0-dev
Contributor
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
Bumps [ruff](https://github.com/astral-sh/ruff) from 0.14.14 to 0.15.0. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md) - [Commits](astral-sh/ruff@0.14.14...0.15.0) --- updated-dependencies: - dependency-name: ruff dependency-version: 0.15.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
….12 to v (dockerfile.nginx-alpine) (DefectDojo#14263) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
…3.12 to v (dockerfile.integration-tests-debian) (DefectDojo#14264) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Bumps [setuptools](https://github.com/pypa/setuptools) from 80.10.2 to 82.0.0. - [Release notes](https://github.com/pypa/setuptools/releases) - [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst) - [Commits](pypa/setuptools@v80.10.2...v82.0.0) --- updated-dependencies: - dependency-name: setuptools dependency-version: 82.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…b/workflows/renovate.yaml) (DefectDojo#14268) Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Resolved conflicts by: - Removing @dojo_async_task decorators that were removed in upstream - Keeping priority parameters from the PR - Adapting to refactored notification system (standalone task functions) - Using correct priority values (3 for notifications/jira, 1 for webhooks, 4 for cleanup tasks, 0 for status checks)
Contributor
|
Conflicts have been resolved. A maintainer will review the pull request shortly. |
valentijnscholten
removed
the
affects_pro
Member
|
@kiblik I resolved conflicts and published the PR so we can merge it and test it with Pro. The PR does not affect Pro in the sense that Pro will keep working and the Pro celery tasks will get priority 3 which is fine. But we can/should do a Pro PR together or after this one to move some of the Pro celery tasks to other priority levels. |
valentijnscholten
approved these changes
Feb 12, 2026
Maffooch
approved these changes
Feb 13, 2026
…tasks - post_process_finding_save: priority=3 (user-triggered, regular task) - post_process_findings_batch: priority=4 (background batch processing) - calculate_grade: priority=4 (background processing) - All JIRA tasks (push_finding_to_jira, push_finding_group_to_jira, push_engagement_to_jira, close_epic, update_epic, add_epic): priority=4 (integrations tier, should not compete with imports for worker slots)
Member
|
@kiblik At first I thought this was a nice way to implement separate queues without having to actually create separate queues in Valkey. However with this approach there's a risk of starvation on the prio 4/5 tasks if an instance is extremely busy (dupe_delete for example). It also may make the task ordering unexpected/unpredictable making it harder to analyse performance problems. What do you think? Since we've never had any reports on the need for priorities or separate queues I have become hesitant to merge this. A compromise could be to have only two priorities 0 (critical) stuff that is needed to keep the instance alive/healthy (1) everything else. |
Contributor
|
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
14 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is adding support for priority queues. More important tasks will be processed sooner and vice versa.
The status of each queue is listed in the system settings overview:
Queue with id
0is automatically marked without a suffix (all other are using naming conventioncelery:1,celery:2...). Thanks to this fact, we are keeping backward compatibility (if there are any tasks in the queue during migration to the new version, we will still process those tasks after the upgrade).It is a bit harder to write tests for this. But I tested manually (stop worker, play around, start worker, observe logs) and it was working as expected.
The role of each queue is described in
settings.dist.py(and reflected in the overview in system settings). If task do not have a priority assigned, priority is set to the default value:3I'm open to feedback if priority of any task should be changed.