feat: add portable CLI binaries for multi-platform Supabase CLI distribution#2033
Merged
feat: add portable CLI binaries for multi-platform Supabase CLI distribution#2033
Conversation
LGUG2Z
force-pushed
the
feature/sb-cli-binaries
branch
30 times, most recently
from
cebd363 to
57718b4
Compare
LGUG2Z
force-pushed
the
feature/sb-cli-binaries
branch
from
3b7417d to
d9cb921
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In `@nix/checks.nix`:
- Around line 215-256: The cliSkipTests array in nix/checks.nix includes "plv8"
but there is no corresponding test file; remove the "plv8" entry from the
cliSkipTests list (or alternatively add a matching test file under
nix/tests/sql/ if the test is required) so filteredTestList and the test
basename mapping no longer reference a non-existent test; update the
cliSkipTests array (symbol: cliSkipTests) and re-run the filter logic (symbol:
filteredTestList) to ensure no missing basenames remain.
In `@nix/packages/cli-config/pg_hba.conf.template`:
- Around line 5-9: The pg_hba.conf.template currently requires scram-sha-256 for
loopback TCP ("host all all 127.0.0.1/32 scram-sha-256" and "host all all
::1/128 scram-sha-256"), which will reject initdb-created passwordless superuser
connections; update the CLI-friendly template to use "trust" for those loopback
host rules or alternatively make the init wrapper set a password before enabling
SCRAM—modify the entries in pg_hba.conf.template for 127.0.0.1/32 and ::1/128 to
"trust" (or implement password-setting in the init script that runs before
switching these entries to scram-sha-256).
Comment on lines
+215
to
+256
| # Tests to skip for CLI variants (require extensions not in CLI) | ||
| cliSkipTests = [ | ||
| # Extension-specific tests | ||
| "evtrigs" | ||
| "http" | ||
| "hypopg" | ||
| "index_advisor" | ||
| "pg_hashids" | ||
| "pg_jsonschema" | ||
| "pg_partman" | ||
| "pg_repack" | ||
| "pg_tle" | ||
| "pgtap" | ||
| "pgmq" | ||
| "pgroonga" | ||
| "pgrouting" | ||
| "plpgsql-check" | ||
| "plv8" | ||
| "postgis" | ||
| "postgres_fdw" | ||
| # Tests that depend on extensions not in CLI | ||
| "security" # depends on various extensions | ||
| "extensions_schema" # tests extension loading | ||
| "roles" # includes roles/schemas from extensions not in CLI (pgtle, pgmq, repack, topology) | ||
| # Version-specific extension tests | ||
| "z_17_ext_interface" | ||
| "z_17_pg_stat_monitor" | ||
| "z_17_pgvector" | ||
| "z_17_rum" | ||
| "z_17_roles" # version-specific roles test, includes pgtle_admin | ||
| ]; | ||
|
|
||
| # Convert filtered tests to a sorted list of basenames (without extension) | ||
| testList = pkgs.lib.mapAttrsToList ( | ||
| name: _: builtins.substring 0 (pkgs.lib.stringLength name - 4) name | ||
| ) filteredSqlTests; | ||
| sortedTestList = builtins.sort (a: b: a < b) testList; | ||
|
|
||
| # Filter out CLI-incompatible tests if this is a CLI variant | ||
| filteredTestList = | ||
| if isCliVariant then | ||
| builtins.filter (test: !(builtins.elem test cliSkipTests)) testList | ||
| else |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Compare cliSkipTests entries against nix/tests/sql/*.sql basenames.
python - <<'PY'
import re, pathlib
text = pathlib.Path("nix/checks.nix").read_text()
m = re.search(r"cliSkipTests\s*=\s*\[(.*?)\];", text, re.S)
skip = re.findall(r'"([^"]+)"', m.group(1)) if m else []
tests = [p.stem for p in pathlib.Path("nix/tests/sql").glob("*.sql")]
missing = [s for s in skip if s not in tests]
extra = [t for t in tests if t in skip]
print("Skip entries without matching test file:", missing)
print("Skip entries that match:", extra)
PYRepository: supabase/postgres
Length of output: 467
Remove plv8 from cliSkipTests—it has no corresponding test file.
Verification shows plv8 is listed in the skip list but lacks a matching test file in nix/tests/sql/. Most other entries (including plpgsql-check) correctly match their test basenames. Remove plv8 or create the missing test file.
🤖 Prompt for AI Agents
In `@nix/checks.nix` around lines 215 - 256, The cliSkipTests array in
nix/checks.nix includes "plv8" but there is no corresponding test file; remove
the "plv8" entry from the cliSkipTests list (or alternatively add a matching
test file under nix/tests/sql/ if the test is required) so filteredTestList and
the test basename mapping no longer reference a non-existent test; update the
cliSkipTests array (symbol: cliSkipTests) and re-run the filter logic (symbol:
filteredTestList) to ensure no missing basenames remain.
00bdb0f to
d3de8d8
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Fix all issues with AI agents
In `@nix/packages/postgres-portable.nix`:
- Around line 50-61: Uncomment and set the getkey_script paths in the
postgresql.conf.template so the portable bundle points to the installed script
path (installed by configBundle) — update the pgsodium.getkey_script and
vault.getkey_script entries in postgresql.conf.template to reference the bundle
location (share/supabase-cli/config/pgsodium_getkey.sh) or implement a startup
hook in the CLI to rewrite those config keys at launch; ensure references align
with the install path created by configBundle (the directory created in
installPhase) and document the behavior so pgsodium and vault can locate the
getkey script when the portable bundle is used.
Comment on lines
50
to
65
| configBundle = stdenv.mkDerivation { | ||
| name = "cli-config-bundle"; | ||
| src = configDir; | ||
| dontPatchShebangs = true; | ||
| installPhase = '' | ||
| mkdir -p $out/share/supabase-cli/config | ||
| cp postgresql.conf.template $out/share/supabase-cli/config/ | ||
| cp pg_hba.conf.template $out/share/supabase-cli/config/ | ||
| cp pg_ident.conf.template $out/share/supabase-cli/config/ | ||
| cp pgsodium_getkey.sh $out/share/supabase-cli/config/ | ||
| chmod +x $out/share/supabase-cli/config/pgsodium_getkey.sh | ||
| ''; |
Contributor
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
fd -t f "\.(template|sh)$" nix/packages/Repository: supabase/postgres
Length of output: 244
🏁 Script executed:
echo "=== pg_hba.conf.template ===" && cat nix/packages/cli-config/pg_hba.conf.template && \
echo -e "\n=== pgsodium_getkey.sh ===" && cat nix/packages/cli-config/pgsodium_getkey.sh && \
echo -e "\n=== postgresql.conf.template ===" && head -50 nix/packages/cli-config/postgresql.conf.templateRepository: supabase/postgres
Length of output: 2208
🏁 Script executed:
cat -n nix/packages/postgres-portable.nix | head -80Repository: supabase/postgres
Length of output: 3062
🏁 Script executed:
rg -i "getkey_script|pgsodium.*path|postgresql\.conf" --type sh --type rs --type ts --type go -A 3 -B 1 | head -100Repository: supabase/postgres
Length of output: 87
🏁 Script executed:
rg -i "getkey_script|pgsodium.*path" -A 2 -B 1 | head -80Repository: supabase/postgres
Length of output: 5047
🏁 Script executed:
grep -r "run-server\|launcher\|wrapper" nix/packages/postgres-portable.nix || echo "Not found in postgres-portable.nix"Repository: supabase/postgres
Length of output: 345
🏁 Script executed:
rg "configBundle|postgresql\.conf\.template" nix/packages/ -A 5 -B 5 | head -60Repository: supabase/postgres
Length of output: 2061
🏁 Script executed:
cd nix/packages && grep -A 30 "buildPhase\|installPhase" postgres-portable.nix | grep -E "run-server|launcher|bin/" | head -20Repository: supabase/postgres
Length of output: 261
🏁 Script executed:
# Check what's actually included in the final output
grep -A 50 "stdenv.mkDerivation {" nix/packages/postgres-portable.nix | grep -E "mkdir|cp|bin/|share/" | head -30Repository: supabase/postgres
Length of output: 1025
🏁 Script executed:
# Look for documentation or CLI integration
find . -name "README*" -o -name "*.md" | head -10Repository: supabase/postgres
Length of output: 362
🏁 Script executed:
# Search for references to share/supabase-cli/config
rg "share/supabase-cli/config|cli-config" --type md --type txt -A 2 -B 1Repository: supabase/postgres
Length of output: 43
🏁 Script executed:
cat README.md | grep -A 20 -i "portable\|cli\|config" | head -50Repository: supabase/postgres
Length of output: 3259
🏁 Script executed:
# Search for where CLI might process these templates
rg "share/supabase-cli" -lRepository: supabase/postgres
Length of output: 144
🏁 Script executed:
# Look for any comments or issues about template configuration in the codebase
rg "placeholder|path.*template|getkey.*path" --type nix -B 2 -A 2 | head -80Repository: supabase/postgres
Length of output: 379
🏁 Script executed:
# Check if there's a mechanism in the nix package to handle path substitution
tail -100 nix/packages/postgres-portable.nixRepository: supabase/postgres
Length of output: 4291
Uncomment and configure getkey_script paths in postgresql.conf.template for portable bundle usage.
The template has pgsodium.getkey_script and vault.getkey_script commented out with a placeholder path. Since the portable bundle installs the script to share/supabase-cli/config/pgsodium_getkey.sh, these lines should be uncommented and configured to reference that location, or a documented mechanism must exist for the CLI to set these paths at startup. Without this configuration, the pgsodium and vault extensions won't function. The pg_hba.conf configuration (trust for local, scram-sha-256 for network) is appropriate for passwordless init.
🤖 Prompt for AI Agents
In `@nix/packages/postgres-portable.nix` around lines 50 - 61, Uncomment and set
the getkey_script paths in the postgresql.conf.template so the portable bundle
points to the installed script path (installed by configBundle) — update the
pgsodium.getkey_script and vault.getkey_script entries in
postgresql.conf.template to reference the bundle location
(share/supabase-cli/config/pgsodium_getkey.sh) or implement a startup hook in
the CLI to rewrite those config keys at launch; ensure references align with the
install path created by configBundle (the directory created in installPhase) and
document the behavior so pgsodium and vault can locate the getkey script when
the portable bundle is used.
7b41b7c to
fc53a9e
Compare
Contributor
Author
|
I've updated the PR to
I have not made any changes to the template - I think it's normal to update a config template at runtime Adhoc codesigning isn't working and is hard to debug in CI, I think it's ok to continue with client-side codesigning for macOS for now since that is what homebrew kegs do when installing Let's discuss the init with others on Slack and then we can update this PR with whatever is decided |
LGUG2Z
force-pushed
the
feature/sb-cli-binaries
branch
from
fc53a9e to
a35290e
Compare
Contributor
Author
|
@CodeRabbit pause |
Contributor
✅ Actions performedReviews paused. |
LGUG2Z
force-pushed
the
feature/sb-cli-binaries
branch
11 times, most recently
from
26c381c to
09c97c8
Compare
…ibution
This commit implements portable, self-contained PostgreSQL binaries for
the Supabase CLI across macOS (ARM), Linux (x64), and Linux (ARM64),
along with automated CI/CD workflows for building and releasing these
artifacts.
The Supabase CLI needs to ship PostgreSQL binaries that work on user
machines without requiring Nix or other system dependencies. This means
extracting the actual binaries from Nix's wrapper scripts, bundling all
necessary shared libraries, and patching them to use relative paths
instead of hardcoded Nix store paths.
A `variant` parameter was added to the postgres build pipeline to
distinguish between "full" (all extensions) and "cli" (minimal
extensions for Supabase CLI). The `cliExtensions` list contains 6
extensions required for running Supabase migrations: supautils,
pg_graphql, pgsodium, supabase_vault, pg_net, and pg_cron. Built-in
extensions (uuid-ossp, pgcrypto, pg_stat_statements) are included
automatically with PostgreSQL. `makeOurPostgresPkgs`/`makePostgresBin`
were modified to accept this parameter. A new `psql_17_cli` package is
created using `variant = "cli"`, while the full extension set is
preserved for base packages (`psql_15`, `psql_17`, `psql_orioledb-17`).
The portable CLI variant (`psql_17_cli_portable`) includes 6 extensions
for migration support while maintaining a significantly smaller size
than the full build. The implementation in
`nix/packages/postgres-portable.nix` extracts binaries from
`psql_17_cli` using a `resolve_binary()` function that follows wrapper
layers to find the actual ELF/Mach-O binaries behind Nix's environment
setup scripts.
All Nix-provided libraries (ICU, readline, zlib, etc.) are bundled while
excluding system libraries (`libc`, `libpthread`, `libm`, `glibc`,
`libdl`) that must come from the host. This distinction is critical:
Linux bundles must exclude glibc due to kernel ABI dependencies, while
macOS can include more libs due to its different linking model.
Dependency resolution runs multiple passes to catch transitive deps
(e.g., ICU → charset → etc.).
Platform-specific patching is applied: Linux binaries use the system
interpreter (`/lib64/ld-linux-*.so.2`) and `$ORIGIN`-based RPATHs, while
macOS binaries use `@rpath` with `@executable_path`. Wrapper scripts set
`LD_LIBRARY_PATH` (Linux) or `DYLD_LIBRARY_PATH` (macOS) to find bundled
libraries. The bundle includes PostgreSQL config templates
(`postgresql.conf`, `pg_hba.conf`, `pg_ident.conf`) tailored for CLI
usage with minimal local dev settings, plus the complete Supabase
migration script (`migrate.sh`) with all init-scripts and migration SQL
files (55 files, 236KB).
A Docker-style initialization script (`supabase-postgres-init.sh`)
provides one-command database setup via environment variables
(`POSTGRES_USER`, `POSTGRES_PASSWORD`, `POSTGRES_DB`, `PGDATA`).
To achieve true portability, a `portable` parameter was added to the
PostgreSQL build in `nix/postgresql/generic.nix`. When `portable =
true`, three critical hardcoded paths are excluded from the build:
1. `--with-system-tzdata` is removed from configure flags, allowing
PostgreSQL to use bundled timezone data from the `share/` directory
instead of a hardcoded `/nix/store/.../tzdata` path;
2. the `locale-binary-path.patch` is excluded, so PostgreSQL calls
`locale -a` from system PATH rather than using an absolute path to
glibc's locale command;
3. the `postFixup` initdb wrapper is disabled to avoid hardcoding glibc
paths. The `portable` parameter defaults to `false` in
`nix/postgresql/default.nix` but is overridden to `true` for the CLI
variant in `nix/packages/postgres.nix` using `.override { portable =
true; }`.
This ensures standard PostgreSQL builds remain unchanged while the CLI
variant produces truly portable binaries without any `/nix/store`
references.
A GitHub Actions workflow builds portable binaries across all three
platforms using a matrix strategy. Each build runs automated portability
checks that verify no `/nix/store` references remain, validate RPATH
configuration, confirm transitive dependencies are bundled, ensure
system libraries are NOT bundled, and check wrapper scripts contain
proper library path setup. Post-build testing validates binaries work
without Nix (`postgres --version`, `psql --version`). On tagged releases
(`v*-cli`), the workflow creates GitHub releases with tarball artifacts
and checksums.
The test infrastructure needed significant changes to support variants
with different extension sets. An `isCliVariant` parameter was added to
`makeCheckHarness`, and the hardcoded `shared_preload_libraries` list in
`postgresql.conf.in` was replaced with a `@PRELOAD_LIBRARIES@`
placeholder. A `generatePreloadLibs` script now parses `receipt.json` at
test time and dynamically builds the preload list based on available
extensions, removing the previous timescaledb removal hack for OrioleDB.
LGUG2Z
force-pushed
the
feature/sb-cli-binaries
branch
from
09c97c8 to
4bad4ae
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What kind of change does this PR introduce?
Adds a new, portable
psql_17_cli_portabletarget.What is the current behavior?
All current builds are not portable (i.e. they have hard dependencies on the
/nix/store), and they all include the full list of extensions.What is the new behavior?
The
psql_17_cli_portabletarget can run without a/nix/storepresent, and only includes thesupautilsextension.Additional context
Summary by CodeRabbit