Skip to content

Migrate Gentoo importer to advisory V2#2090

Open
ziadhany wants to merge 14 commits intoaboutcode-org:mainfrom
ziadhany:gentoo-migration
Open

Migrate Gentoo importer to advisory V2#2090
ziadhany wants to merge 14 commits intoaboutcode-org:mainfrom
ziadhany:gentoo-migration

Conversation

@ziadhany
Copy link
Collaborator

@ziadhany ziadhany commented Dec 30, 2025

@ziadhany ziadhany changed the title Add initial migration to Gentoo importer v2 Migrate Gentoo importer to advisory V2 Dec 30, 2025
@ziadhany ziadhany marked this pull request as ready for review January 1, 2026 13:58
@ziadhany
Copy link
Collaborator Author

ziadhany commented Jan 1, 2026

gentoo importer V2 logs:

/home/ziad-hany/PycharmProjects/vulnerablecode/venv/bin/python /home/ziad-hany/PycharmProjects/vulnerablecode/manage.py import localhost:8000 --all 
Importing data using gentoo_importer_v2
INFO 2026-02-19 00:43:39.965910 UTC Pipeline [GentooImporterPipeline] starting
INFO 2026-02-19 00:43:39.966115 UTC Step [clone] starting
INFO 2026-02-19 00:43:39.966230 UTC Cloning `git+https://anongit.gentoo.org/git/data/glsa.git`
INFO 2026-02-19 00:43:46.580024 UTC Step [clone] completed in 7 seconds
INFO 2026-02-19 00:43:46.580225 UTC Step [collect_and_store_advisories] starting
INFO 2026-02-19 00:43:46.611900 UTC Collecting 3,814 advisories
INFO 2026-02-19 00:43:57.193975 UTC Progress: 10% (382/3814) ETA: 95 seconds (1.6 minutes)
INFO 2026-02-19 00:44:05.434729 UTC InvalidVersion constraints version: 1.3* error:'1.3*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-19 00:44:07.580986 UTC Progress: 20% (763/3814) ETA: 84 seconds (1.4 minutes)
INFO 2026-02-19 00:44:19.843280 UTC Progress: 30% (1145/3814) ETA: 78 seconds (1.3 minutes)
INFO 2026-02-19 00:44:25.308116 UTC InvalidVersion constraints version: 3.24.48:3 error:'3.24.48:3' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-19 00:44:25.308312 UTC InvalidVersion constraints version: 3.24.48:3 error:'3.24.48:3' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-19 00:44:29.940508 UTC Progress: 40% (1526/3814) ETA: 65 seconds (1.1 minutes)
INFO 2026-02-19 00:44:40.136244 UTC Progress: 50% (1907/3814) ETA: 54 seconds
INFO 2026-02-19 00:44:44.267492 UTC InvalidVersion constraints version: 7.3* error:'7.3*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-19 00:44:44.267669 UTC InvalidVersion constraints version: 7.4* error:'7.4*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-19 00:44:50.277876 UTC Progress: 60% (2289/3814) ETA: 42 seconds
INFO 2026-02-19 00:44:51.104486 UTC InvalidVersion constraints version: 7.4* error:'7.4*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-19 00:44:57.657790 UTC InvalidVersion constraints version: 6.9.3:6 error:'6.9.3:6' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-19 00:44:57.657974 UTC InvalidVersion constraints version: 6.9.3:6 error:'6.9.3:6' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-19 00:44:59.602151 UTC InvalidVersion constraints version: 7.3* error:'7.3*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-19 00:44:59.602339 UTC InvalidVersion constraints version: 7.4* error:'7.4*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-19 00:45:00.146105 UTC Progress: 70% (2670/3814) ETA: 32 seconds
INFO 2026-02-19 00:45:04.708364 UTC InvalidVersion constraints version: 7.3* error:'7.3*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-19 00:45:04.708537 UTC InvalidVersion constraints version: 7.4* error:'7.4*' is not a valid <class 'univers.versions.GentooVersion'>
INFO 2026-02-19 00:45:10.335491 UTC Progress: 80% (3052/3814) ETA: 21 seconds
INFO 2026-02-19 00:45:20.078882 UTC Progress: 90% (3433/3814) ETA: 10 seconds
INFO 2026-02-19 00:45:29.943624 UTC Progress: 100% (3814/3814)
INFO 2026-02-19 00:45:29.970652 UTC Successfully collected 3,814 advisories
INFO 2026-02-19 00:45:29.970781 UTC Step [collect_and_store_advisories] completed in 103 seconds (1.7 minutes)
INFO 2026-02-19 00:45:29.970850 UTC Step [clean_downloads] starting
INFO 2026-02-19 00:45:29.970908 UTC Removing cloned repository
INFO 2026-02-19 00:45:30.062081 UTC Step [clean_downloads] completed in 0 seconds
INFO 2026-02-19 00:45:30.062238 UTC Pipeline completed in 110 seconds (1.8 minutes)

Process finished with exit code 0
from vulnerabilities.models import AdvisoryV2
from django.db.models import Count
duplicates = (
    AdvisoryV2.objects
    .values('avid')
    .annotate(count=Count('id'))
    .filter(count__gt=1)
)
len(duplicates)
Out[2]: 0
AdvisoryV2.objects.count()
Out[3]: 3814

gentoo importer V1 logs:

Importing data using vulnerabilities.importers.gentoo.GentooImporter
Invalid safe_version 3.24.48:3 - error: '3.24.48:3' is not a valid <class 'univers.versions.GentooVersion'>
Invalid safe_version 6.9.3:6 - error: '6.9.3:6' is not a valid <class 'univers.versions.GentooVersion'>
Successfully imported data using vulnerabilities.importers.gentoo.GentooImporter

@ziadhany
Copy link
Collaborator Author

ziadhany commented Jan 1, 2026

Copy link
Member

@keshav-space keshav-space left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ziadhany Thanks, see some feedback below.

Comment on lines 144 to 145
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ziadhany why do we invert fixed range, we should report fixed range as is.

Copy link
Collaborator Author

@ziadhany ziadhany Feb 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@keshav-space we have two type of package version

  • unaffected
  • vulnerable
GLSA-201405-01 , CVE-2014-0004
  <affected>
    <package name="sys-fs/udisks" auto="yes" arch="*">
      <unaffected range="rge">1.0.5</unaffected>
      <unaffected range="ge">2.1.3</unaffected>
      <vulnerable range="lt">2.1.3</vulnerable>
    </package>
  </affected>

we invert the unaffected (safe_version) to get the affected version if it isn’t specified.
see line: 159, 162

the main question is that does the unaffected mean fixed range, if yes I should update this

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the main question is that does the unaffected mean fixed range, if yes I should update this

@ziadhany Yes, unaffected is fixed range see resolution section here https://security.gentoo.org/glsa/201405-01 so if we get unaffected range it should be treated as fixed range.

Also rge means revision greater than equals and rgt means revision greater than lets mention this in comments.

<unaffected range="ge">2.1.3</unaffected> is pretty straight forward it translates to fixed range vers:ebuild/>=2.1.3. But revision range are bit tricky here <unaffected range="rge">1.0.5</unaffected> translates to vers:ebuild/>=1.0.5|<1.1 see the interpretation here https://security.gentoo.org/glsa/201405-01

similarly for this advisory https://security.gentoo.org/glsa/202004-13 we have these unaffected range

<unaffected range="rge">2.23.3</unaffected>
<unaffected range="rge">2.24.3</unaffected>
<unaffected range="rge">2.25.4</unaffected>
<unaffected range="rge">2.26.2</unaffected>

and these would be interpreted as fixed range

vers:ebuild/>=2.23.3|<2.24
vers:ebuild/>=2.24.3|<2.25
vers:ebuild/>=2.25.4|<2.26
vers:ebuild/>=2.26.2|<2.27

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ziadhany I went through your pr aboutcode-org/univers#181 and I also reviewed the gentoo version documentation https://projects.gentoo.org/pms/8/pms.html#x1-250003.2. It turns out that gentoo packages do not follow any particular versioning standard, so it becomes difficult to create a range from a single version such that the range includes only revisions of that version.
The only reliable way to generate a bounded range for a revisione version is to retrieve all versions of a particular gentoo package and sort them as described here https://projects.gentoo.org/pms/8/pms.html#x1-260003.3 and then select the last revision of that version as the terminating bound. We cannot do that in this pipeline and it should be handled separately in an improver pipeline.

For now we can only capture what we are presented with, in the above example.

<unaffected range="rge">2.23.3</unaffected>
<unaffected range="rge">2.24.3</unaffected>
<unaffected range="rge">2.25.4</unaffected>
<unaffected range="rge">2.26.2</unaffected>

we should create impacted package with these fixed version range.

vers:ebuild/>=2.23.3
vers:ebuild/>=2.24.3
vers:ebuild/>=2.25.4
vers:ebuild/>=2.26.2

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. I was also hesitant about a the PR implementation I will close this PR

I have created an issue so we can track this properly and improve it in the future using an improver pipeline, as you suggested.

@ziadhany
Copy link
Collaborator Author

@keshav-space Is this correct implementation?

https://security.gentoo.org/glsa/201709-09

<package name="dev-vcs/subversion" auto="yes" arch="*">
    <unaffected range="ge">1.9.7</unaffected>      <---- fixed_version_range  >=1.9.7
    <unaffected range="rgt">1.8.18</unaffected>   <---- fixed_version_range >1.8.18|<1.9.0
    <vulnerable range="lt">1.9.7</vulnerable>
    <vulnerable range="eq">0.1.1</vulnerable>
</package>
"affected_version_range": "vers:ebuild/0.1.1|<1.9.7",
"fixed_version_range": "vers:ebuild/>1.8.18|<1.9|>=1.9.7",

@ziadhany ziadhany force-pushed the gentoo-migration branch 2 times, most recently from cc9e228 to d73249f Compare February 16, 2026 14:38
@ziadhany
Copy link
Collaborator Author

@keshav-space I am also getting an InvalidConstraintsError even after using the new implementation.
aboutcode-org/univers#181

see:
gentoo.zip

@ziadhany
Copy link
Collaborator Author

As we discussed in the weekly VulnerableCode meeting, we decided to have an affected_package for every index in package (unaffected, vulnerable). to avoid conflict on version range constraints.

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Update the Gentoo get_safe_and_affected_versions function in advisory v2

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
… and fixed_version_range for non-revision versions

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Update the pipeline to use the new AdvisoryDataV2

Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Signed-off-by: ziad hany <ziadhany2016@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments