feat: add support for parallel kind processing with threads#840
feat: add support for parallel kind processing with threads#840bhearsum merged 4 commits intotaskcluster:mainfrom
Conversation
bhearsum
force-pushed
the
push-yvzvqryxtvsq
branch
2 times, most recently
from
2a48d4e to
05e295a
Compare
|
Msgspec does support freethreading in python: https://github.com/jcrist/msgspec/pull/877/files Initially thought we are planning to have both support for msgspec and voluptuous |
Oh, that's wonderful to hear about msgspec! Supporting both should be fine as far as I'm concerned for this work; it just means I'll need to rebase this after the schema work lands. |
|
msgspec support for 3.14 has been released https://github.com/jcrist/msgspec/releases/tag/0.20.0 |
bhearsum
force-pushed
the
push-yvzvqryxtvsq
branch
6 times, most recently
from
2afc838 to
f4b9a78
Compare
bhearsum
force-pushed
the
push-yvzvqryxtvsq
branch
from
f4b9a78 to
019cf90
Compare
This will make a subsequent commit that adds support for a ThreadPoolExecutor cleaner.
Voluptuous contains thread-unsafe code; work around this by locking before calling into it.
Even with 3.14 free-threaded python, this is still a bit slower than multiprocessing on Linux, but it will allow us to start experimenting with it more, and may allow users on macOS and Windows to immediately see a speed-up.
bhearsum
force-pushed
the
push-yvzvqryxtvsq
branch
from
019cf90 to
71fdab5
Compare
hneiva
left a comment
hneiva
left a comment
There was a problem hiding this comment.
One non-blocking question
LGTM!
| # | ||
| # On all platforms, multiple threads can be enabled by setting | ||
| # TASKGRAPH_USE_THREADS in the environment. Taskgraph must be running | ||
| # from a free-threaded Python build to see any performance benefits. |
There was a problem hiding this comment.
Should we also add this info to taskgraph --help ?
There was a problem hiding this comment.
I think, because it's experimental, I'd rather avoid advertising it for the moment (aside from a mention in the changelog during the next release).
4 participants
This adds experimental, opt-in support for generating kinds in parallel across multiple threads. When used in conjunction with a free-threaded python it can be a significant speed-up in configuration with many expensive to generate kinds (eg: Gecko) where multiprocessing is not available.
In the short and medium term this should stay opt-in for a few reasons:
zstandard) are not available, which makes taskgraph more painful to install with this python.In the long term, I hope to get rid of multiprocessing support entirely once the above are no longer issues, because it imposes many limitations on how taskgraph generation can happen, how cached data can be shared, etc.
Ideally this PR would come with some multithreaded testing on Windows and macOS. Once #869 is done we ought to be able to add mac tests. Windows is trickier -- we don't have a compiler available on the workers, and zstandard (and probably other packages) don't have wheels available for free threaded python, which means we can't even install taskgraph in a free-threaded python virtualenv.
The code coverage failure is a bit misleading -- there's little in the way of new code here, and the new paths for the Linux side is covered with the multithread test job, which coverage isn't aware of.