-
-
Notifications
You must be signed in to change notification settings - Fork 228
feat(darwin): add nix-darwin host and linux-builder for NixOS tests #1999
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,189 @@ | ||
| ## Prerequisites | ||
|
|
||
| Running NixOS tests on macOS requires a Linux builder VM because NixOS tests need a Linux environment. | ||
| This project includes a nix-darwin configuration that sets up a linux-builder VM automatically. | ||
|
|
||
| You need: | ||
| - macOS with Apple Silicon (aarch64-darwin) | ||
| - Nix installed (see [Getting Started](start-here.md)) preferably a recent version (2.30+) | ||
|
|
||
| ## Setup | ||
|
|
||
| Run the setup script to configure nix-darwin with the linux-builder: | ||
|
|
||
| ```bash | ||
| nix run .#setup-darwin-linux-builder | ||
| ``` | ||
|
|
||
| Note that you don't have to checkout the repository to run the setup, you can run it directly from GitHub: | ||
|
|
||
| ```bash | ||
| nix run github:supabase/postgres#setup-darwin-linux-builder | ||
| ``` | ||
|
|
||
| This command will: | ||
| - Back up existing system files (`/etc/nix/nix.conf`, `/etc/bashrc`, `/etc/zshrc`) | ||
| - Configure nix-darwin with the linux-builder VM | ||
| - Install helper scripts for managing the builder | ||
|
|
||
| The linux-builder VM is configured with: | ||
| - 6 CPU cores | ||
| - 8GB RAM | ||
| - 40GB disk | ||
| - Support for both x86_64-linux and aarch64-linux builds | ||
| - The `nixos-test` feature required for running NixOS tests | ||
|
|
||
| After setup completes, restart your shell to access the helper commands. | ||
|
|
||
| ## Verify the setup | ||
|
|
||
| The setup script runs verification automatically after configuration. | ||
| You can also run verification manually at any time: | ||
|
|
||
| ```bash | ||
| nix run .#verify-darwin-linux-builder | ||
| ``` | ||
|
|
||
| Or after setup, use the installed command: | ||
|
|
||
| ```bash | ||
| verify-darwin-linux-builder | ||
| ``` | ||
|
|
||
| The verification script checks: | ||
|
|
||
| 1. Launchd service status (running vs loaded-but-stopped) | ||
| 2. Nix configuration via `nix config show` (substituters, trusted keys, experimental features) | ||
| 3. Builder features (`/etc/nix/machines` includes `nixos-test`) | ||
| 4. Builder responsiveness (test build of `nixpkgs#hello` for aarch64-linux) | ||
|
|
||
| Each check reports pass/fail with actionable guidance on failures. | ||
|
|
||
| You can also manually test that the linux-builder is working by building a simple package for Linux: | ||
|
|
||
| ```bash | ||
| nix build --system x86_64-linux nixpkgs#hello | ||
| nix build --system aarch64-linux nixpkgs#hello | ||
| ``` | ||
|
|
||
| If both commands succeed, the linux-builder is ready for NixOS tests. | ||
|
|
||
| ## Running NixOS tests | ||
|
|
||
| NixOS tests are defined in `nix/ext/tests/` and exposed as flake checks. | ||
| To run a test on macOS, use the `aarch64-darwin` system attribute: | ||
|
|
||
| ```bash | ||
| nix build .#checks.aarch64-darwin.ext-pgjwt -L | ||
| ``` | ||
|
|
||
| The `-L` flag shows logs during the build, which is helpful for seeing test progress and debugging failures. | ||
|
|
||
| If the nix build exit immediately with success, it means that the result was fetched from cache and the test passed previously. | ||
| To force a re-run of the test, use the `--rebuild` flag: | ||
|
|
||
| ```bash | ||
| nix build .#checks.aarch64-darwin.ext-pgjwt -L --rebuild | ||
| ``` | ||
|
|
||
| ### Available tests | ||
|
|
||
| List all available checks with: | ||
|
|
||
| ```bash | ||
| nix flake show --json 2>/dev/null | jq -r '.checks["aarch64-darwin"] | keys[]' | sort | ||
| ``` | ||
|
|
||
| Extension tests follow the naming pattern `ext-<extension_name>`: | ||
|
|
||
| ```bash | ||
| nix build .#checks.aarch64-darwin.ext-pgjwt -L | ||
| nix build .#checks.aarch64-darwin.ext-postgis -L | ||
| nix build .#checks.aarch64-darwin.ext-vector -L | ||
| nix build .#checks.aarch64-darwin.ext-pg_graphql -L | ||
| ``` | ||
|
|
||
| ## Managing the linux-builder VM | ||
|
|
||
| The setup installs two helper commands for controlling the VM: | ||
|
|
||
| ```bash | ||
| stop-linux-builder # Stop the VM (pauses resource usage) | ||
| start-linux-builder # Start the VM again | ||
| ``` | ||
|
|
||
| As the VM can consume significant resources, you may want to stop it when not running tests using `stop-linux-builder`. | ||
| When stopped with `stop-linux-builder`, the service is unloaded to prevent automatic restart. | ||
| Use `start-linux-builder` to re-enable and start the service. | ||
|
|
||
| ### Checking VM status | ||
|
|
||
| ```bash | ||
| sudo launchctl list | grep linux-builder | ||
| ``` | ||
|
|
||
| If the VM is running, you'll see a line containing `org.nixos.linux-builder`. | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| ### Tests fail with "builder not available" | ||
|
|
||
| Ensure the linux-builder is running: | ||
|
|
||
| ```bash | ||
| start-linux-builder | ||
| ``` | ||
|
|
||
| Then verify with a simple build: | ||
|
|
||
| ```bash | ||
| nix build --system aarch64-linux nixpkgs#hello | ||
| ``` | ||
|
|
||
| ### VM won't start after reboot | ||
|
|
||
| If the VM doesn't start automatically, run: | ||
|
|
||
| ```bash | ||
| start-linux-builder | ||
| ``` | ||
|
|
||
| The VM is configured as ephemeral, meaning it's recreated fresh on each start. | ||
| This ensures a clean environment but requires re-downloading cached build artifacts. | ||
|
|
||
| ### Slow first build | ||
|
|
||
| The first NixOS test run may download significant data. | ||
| Subsequent runs benefit from the Nix store cache and the project's binary cache at `nix-postgres-artifacts.s3.amazonaws.com`. | ||
|
|
||
| ## How it works | ||
|
|
||
| The linux-builder is a QEMU virtual machine managed by nix-darwin. | ||
| When you run a build targeting Linux (like NixOS tests), Nix automatically delegates the build to this VM. | ||
|
|
||
| Key configuration from `nix/hosts/darwin-nixostest/darwin-configuration.nix`: | ||
|
|
||
| ```nix | ||
| nix.linux-builder = { | ||
| enable = true; | ||
| ephemeral = true; | ||
| maxJobs = 4; | ||
| supportedFeatures = [ | ||
| "kvm" | ||
| "benchmark" | ||
| "big-parallel" | ||
| "nixos-test" # Required for NixOS integration tests | ||
| ]; | ||
| config = { | ||
| virtualisation = { | ||
| darwin-builder = { | ||
| diskSize = 40 * 1024; # 40GB | ||
| memorySize = 8 * 1024; # 8GB | ||
| }; | ||
| cores = 6; | ||
| }; | ||
| }; | ||
| }; | ||
| ``` | ||
|
|
||
| The `nixos-test` supported feature is what enables running NixOS VM tests from macOS. | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| { inputs, self, ... }: | ||
| { | ||
| flake = { | ||
| darwinConfigurations = { | ||
| darwin-nixostest = inputs.nix-darwin.lib.darwinSystem { | ||
| specialArgs = { inherit self; }; | ||
| modules = [ ./hosts/darwin-nixostest/darwin-configuration.nix ]; | ||
| }; | ||
| }; | ||
| }; | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,135 @@ | ||
| { | ||
| lib, | ||
| pkgs, | ||
| self, | ||
| ... | ||
| }: | ||
| let | ||
| start-linux-builder = pkgs.writeShellApplication { | ||
| name = "start-linux-builder"; | ||
| text = '' | ||
| echo "Starting linux-builder..." | ||
|
|
||
| if sudo launchctl list | grep -q org.nixos.linux-builder; then | ||
| echo "linux-builder is already running" | ||
| exit 0 | ||
| fi | ||
|
|
||
| # Use load instead of start to re-enable the service | ||
| if sudo launchctl load -w /Library/LaunchDaemons/org.nixos.linux-builder.plist 2>/dev/null; then | ||
| echo "linux-builder started successfully" | ||
| else | ||
| echo "Error: Could not start linux-builder" | ||
| echo "Make sure nix-darwin is configured with linux-builder enabled" | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Check if it's running | ||
| sleep 2 | ||
| if sudo launchctl list | grep -q org.nixos.linux-builder; then | ||
| echo "linux-builder is now running" | ||
| else | ||
| echo "Warning: linux-builder may not have started properly" | ||
| fi | ||
| ''; | ||
|
Comment on lines
+8
to
+34
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🌐 Web query:
💡 Result:
Detect whether a specific job is running (using
|
||
| }; | ||
| stop-linux-builder = pkgs.writeShellApplication { | ||
| name = "stop-linux-builder"; | ||
| text = '' | ||
| echo "Stopping linux-builder..." | ||
|
|
||
| # Use unload instead of stop because KeepAlive=true will restart it | ||
| if sudo launchctl unload -w /Library/LaunchDaemons/org.nixos.linux-builder.plist 2>/dev/null; then | ||
| echo "linux-builder stopped successfully" | ||
| else | ||
| echo "Warning: Could not stop linux-builder (it may not be running)" | ||
| fi | ||
|
|
||
| # Check if it's still running | ||
| sleep 1 | ||
| if sudo launchctl list | grep -q org.nixos.linux-builder; then | ||
| echo "Warning: linux-builder is still running" | ||
| STATUS=$(sudo launchctl list | grep org.nixos.linux-builder || true) | ||
| echo "Current status: $STATUS" | ||
|
Comment on lines
+50
to
+53
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same The stop script has the same problem: a loaded-but-not-running job will trigger the "still running" warning incorrectly. Consider using the same PID-aware check suggested for the start script. 🐛 Proposed fix- if sudo launchctl list | grep -q org.nixos.linux-builder; then
+ if sudo launchctl list | awk '$3=="org.nixos.linux-builder" && $1 != "-" {found=1} END{exit !found}'; then
echo "Warning: linux-builder is still running"🤖 Prompt for AI Agents |
||
| else | ||
| echo "linux-builder is not running" | ||
| fi | ||
| ''; | ||
| }; | ||
| verify-darwin-linux-builder = self.packages.aarch64-darwin.verify-darwin-linux-builder; | ||
| in | ||
| { | ||
| nixpkgs.hostPlatform = "aarch64-darwin"; | ||
|
|
||
| # Install builder control scripts | ||
| environment.systemPackages = [ | ||
| start-linux-builder | ||
| stop-linux-builder | ||
| verify-darwin-linux-builder | ||
| ]; | ||
jfroche marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| nix.settings = { | ||
| experimental-features = [ | ||
| "nix-command" | ||
| "flakes" | ||
| ]; | ||
| always-allow-substitutes = true; | ||
| max-jobs = "auto"; | ||
| trusted-users = [ "@admin" ]; | ||
| extra-substituters = [ "https://nix-postgres-artifacts.s3.amazonaws.com" ]; | ||
| extra-trusted-substituters = [ "https://nix-postgres-artifacts.s3.amazonaws.com" ]; | ||
| extra-trusted-public-keys = [ | ||
| "nix-postgres-artifacts:dGZlQOvKcNEjvT7QEAJbcV6b6uk7VF/hWMjhYleiaLI=" | ||
| ]; | ||
| }; | ||
|
|
||
| nix.extraOptions = '' | ||
| !include nix.custom.conf | ||
| ''; | ||
|
|
||
| # accept existing nix.custom.conf | ||
| system.activationScripts.checks.text = lib.mkForce ""; | ||
| system.activationScripts.nix-daemon.text = lib.mkForce '' | ||
| if ! diff /etc/nix/nix.conf /run/current-system/etc/nix/nix.conf &> /dev/null || ! diff /etc/nix/machines /run/current-system/etc/nix/machines &> /dev/null; then | ||
| echo "reloading nix-daemon..." >&2 | ||
| launchctl kill HUP system/org.nixos.nix-daemon | ||
| fi | ||
| max_wait=30 | ||
| waited=0 | ||
| while ! nix-store --store daemon -q --hash ${pkgs.stdenv.shell} &>/dev/null; do | ||
| if [ $waited -ge $max_wait ]; then | ||
| echo "ERROR: nix-daemon failed to start after $max_wait seconds" >&2 | ||
| exit 1 | ||
| fi | ||
| echo "waiting for nix-daemon" >&2 | ||
| launchctl kickstart system/org.nixos.nix-daemon | ||
| sleep 1 | ||
| waited=$((waited + 1)) | ||
| done | ||
| ''; | ||
|
|
||
| nix.linux-builder = { | ||
| enable = true; | ||
| ephemeral = true; | ||
| maxJobs = 4; | ||
| supportedFeatures = [ | ||
| "kvm" | ||
| "benchmark" | ||
| "big-parallel" | ||
| "nixos-test" | ||
| ]; | ||
| config = { | ||
| virtualisation = { | ||
| darwin-builder = { | ||
| diskSize = 40 * 1024; | ||
| memorySize = 8 * 1024; | ||
| }; | ||
| cores = 6; | ||
| }; | ||
| }; | ||
| }; | ||
jfroche marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| nix.distributedBuilds = true; | ||
|
|
||
| system.stateVersion = 6; | ||
| } | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fix grammar and improve clarity.
The sentence has a grammatical error ("exit" should be "exits") and could be clearer.
📝 Suggested fix
📝 Committable suggestion
🧰 Tools
🪛 LanguageTool
[style] ~60-~60: ‘with success’ might be wordy. Consider a shorter alternative.
Context: ...res. If the nix build exit immediately with success, it means that the result was fetched f...
(EN_WORDINESS_PREMIUM_WITH_SUCCESS)
🤖 Prompt for AI Agents