Skip to content

Non null arrays to allow index usage #819

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
abnegate wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Non null arrays to allow index usage #819

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
f76a0de
(fix): Make array columns NOT NULL with default empty array for index…
abnegate Feb 19, 2026
4f02d63
(test): Add array index schema and query tests, update expectations f…
abnegate Feb 19, 2026
31d3db1
Add gin index support check
abnegate Feb 19, 2026
1ada153
Assert pg check
abnegate Feb 19, 2026
6cdbe98
Fix upsert + test
abnegate Feb 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions src/Database/Adapter.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -979,6 +979,13 @@ abstract public function getSupportForIndexArray(): bool;
*/
abstract public function getSupportForCastIndexArray(): bool;

/**
* Does the adapter use GIN indexes for array columns?
*
* @return bool
*/
abstract public function getSupportForGinIndex(): bool;

/**
* Is unique index supported?
*
Expand Down
6 changes: 3 additions & 3 deletions src/Database/Adapter/MariaDB.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1684,7 +1684,7 @@ protected function getSQLType(string $type, int $size, bool $signed = true, bool
return $this->getSpatialSQLType($type, $required);
}
if ($array === true) {
return 'JSON';
return 'JSON NOT NULL DEFAULT (JSON_ARRAY())';
}
Comment on lines 1686 to 1688
Copy link
Contributor

@coderabbitai coderabbitai bot Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# First, let's examine the updateAttribute method in MariaDB.php
head -n 450 src/Database/Adapter/MariaDB.php | tail -n 80

Repository: utopia-php/database

Length of output: 2845

🏁 Script executed:

# Check the full context of getSQLType for array handling
rg -n "getSQLType" src/Database/Adapter/MariaDB.php -A 3 -B 3

Repository: utopia-php/database

Length of output: 1486

🏁 Script executed:

# Look for any UPDATE or backfill logic related to array columns
rg -n "JSON_ARRAY\(\)" src/Database/Adapter/MariaDB.php -B 5 -A 5

Repository: utopia-php/database

Length of output: 3009

🏁 Script executed:

# Check if there are any migration or backfill mechanisms in Database.php or SQL.php
rg -n "updateAttribute" src/Database/ -A 5 -B 2 --type=php

Repository: utopia-php/database

Length of output: 16758

🏁 Script executed:

# Check for NULL backfill patterns in the codebase
rg -n "IS NULL.*JSON_ARRAY\|JSON_ARRAY.*IS NULL" src/Database/ --type=php

Repository: utopia-php/database

Length of output: 45

🏁 Script executed:

# Get the full updateAttribute method in Database.php around line 2616-2920
sed -n '2616,2935p' src/Database/Database.php | cat -n

Repository: utopia-php/database

Length of output: 16616

🏁 Script executed:

# Specifically look for any UPDATE statements in the updateAttribute flow
rg -n "UPDATE" src/Database/Database.php | grep -A 5 -B 5 "2616\|2700\|2800\|2900"

Repository: utopia-php/database

Length of output: 45

🏁 Script executed:

# Check for any JSON_ARRAY backfill patterns in the entire Database.php
rg -n "JSON_ARRAY" src/Database/Database.php -B 5 -A 5

Repository: utopia-php/database

Length of output: 45

🏁 Script executed:

# Look for migration files or migration strategy
find . -type f -name "*migration*" -o -name "*migrate*" | head -20

Repository: utopia-php/database

Length of output: 45

🏁 Script executed:

# Check if there are tests for updateAttribute with arrays
rg -n "updateAttribute.*array|array.*updateAttribute" tests/ --type=php -B 3 -A 3

Repository: utopia-php/database

Length of output: 707

🏁 Script executed:

# Look for any test that validates updating array attributes on existing tables
rg -n "testUpdateAttribute" tests/ --type=php -l

Repository: utopia-php/database

Length of output: 279

🏁 Script executed:

# Check what the test file mentioned in learnings says about array attributes
sed -n '6418,6439p' tests/e2e/Adapter/Scopes/DocumentTests.php

Repository: utopia-php/database

Length of output: 1068

🏁 Script executed:

# Examine the AttributeTests.php for array attribute tests
rg -n "updateAttribute" tests/e2e/Adapter/Scopes/AttributeTests.php -B 5 -A 10 | head -100

Repository: utopia-php/database

Length of output: 4513

🏁 Script executed:

# Look for tests that create data with NULL or then update attribute to NOT NULL
rg -n "createDocument\|updateAttribute" tests/e2e/Adapter/Scopes/AttributeTests.php -B 2 -A 2 | head -80

Repository: utopia-php/database

Length of output: 45

🏁 Script executed:

# Check if getSQLType array behavior is consistent across adapters
rg -n "array === true" src/Database/Adapter/ --type=php -B 2 -A 2

Repository: utopia-php/database

Length of output: 860

🏁 Script executed:

# Look for any schemaless adapter handling
rg -n "getSupportForAttributes\|schemaless" src/Database/Adapter/ --type=php -B 3 -A 3 | head -80

Repository: utopia-php/database

Length of output: 45

🏁 Script executed:

# Check if there's a specific test for updating array attributes on existing collections
rg -n "array.*true|createAttribute.*array" tests/e2e/Adapter/Scopes/AttributeTests.php -B 3 -A 8

Repository: utopia-php/database

Length of output: 3855

🏁 Script executed:

# Check what createAttribute does for array types
rg -n "createAttribute" src/Database/Adapter/MariaDB.php -A 30 | head -60

Repository: utopia-php/database

Length of output: 45

🏁 Script executed:

# Look at parent SQL adapter for array handling
rg -n "array === true|JSON NOT NULL" src/Database/Adapter/SQL.php -B 5 -A 5

Repository: utopia-php/database

Length of output: 45

🏁 Script executed:

# Check if there's a difference between initial creation and update for NOT NULL handling
git log --oneline -20 -- src/Database/Adapter/MariaDB.php

Repository: utopia-php/database

Length of output: 159

🏁 Script executed:

# Look at the recent commit that mentions NOT NULL arrays
git show 4f02d63f --stat | head -40

Repository: utopia-php/database

Length of output: 2426

🏁 Script executed:

# Check the parent class SQL.php to see if createAttribute creates with NOT NULL
rg -n "createAttribute\|getSQLType" src/Database/Adapter/SQL.php -A 15 | head -100

Repository: utopia-php/database

Length of output: 45

🏁 Script executed:

# Look for the createAttribute implementation in MariaDB or parent
rg -n "public function createAttribute" src/Database/Adapter/ -A 20 | head -80

Repository: utopia-php/database

Length of output: 5879

🏁 Script executed:

# Check if there's any test that actually calls updateAttribute on array fields
rg -n "updateAttribute.*array.*true|array.*true.*updateAttribute" tests/ -B 5 -A 10

Repository: utopia-php/database

Length of output: 1608

🏁 Script executed:

# Verify when createAttribute is used vs updateAttribute for array fields
rg -n "createAttribute|updateAttribute" tests/e2e/Adapter/Scopes/AttributeTests.php | grep -i array

Repository: utopia-php/database

Length of output: 364

🏁 Script executed:

# Check if Postgres adapter has the same issue
cat src/Database/Adapter/Postgres.php | grep -A 5 "array === true"

Repository: utopia-php/database

Length of output: 221

🏁 Script executed:

# Check if there's a test that specifically tests updating an existing array column 
# that might have NULL values (e.g., converting non-array to array)
rg -n "createDocument\|createAttribute" tests/e2e/Adapter/Scopes/AttributeTests.php -B 2 -A 5 | grep -A 20 "non-array\|convert"

Repository: utopia-php/database

Length of output: 45

🏁 Script executed:

# Look for any migration or update scenario in tests where data exists before updateAttribute
rg -n "createDocument.*\$|createDocument" tests/e2e/Adapter/Scopes/AttributeTests.php | head -20

Repository: utopia-php/database

Length of output: 1508

🏁 Script executed:

# Specifically check if there's a test scenario where documents are created, 
# then an attribute is changed to array=true
rg -n "array.*true" tests/e2e/Adapter/Scopes/AttributeTests.php -B 20 | grep -A 20 "createDocument"

Repository: utopia-php/database

Length of output: 45

🏁 Script executed:

# Check if there's validation preventing changing array from false to true
rg -n "originalArray\|array.*change\|array.*update" src/Database/Database.php -B 3 -A 3

Repository: utopia-php/database

Length of output: 45

🏁 Script executed:

# Look for what happens when originalArray != array (is this even allowed?)
sed -n '2616,2650p' src/Database/Database.php

Repository: utopia-php/database

Length of output: 1706

🏁 Script executed:

# Check if there's a scenario in tests where array is changed
rg -n "originalArray" src/Database/ --type=php -B 5 -A 5

Repository: utopia-php/database

Length of output: 1632

🏁 Script executed:

# Check TYPE_ARRAY_APPEND and TYPE_ARRAY_PREPEND implementations
rg -n "TYPE_ARRAY_APPEND|TYPE_ARRAY_PREPEND" src/Database/Adapter/MariaDB.php -B 2 -A 15

Repository: utopia-php/database

Length of output: 1066

🏁 Script executed:

# Verify the issue: check if array attribute can be changed from false to true on existing table
rg -n "array.*false\|array.*true" tests/e2e/Adapter/Scopes/AttributeTests.php -B 5 -A 5 | head -50

Repository: utopia-php/database

Length of output: 45

🏁 Script executed:

# Check if there's any constraint preventing changing array=false to array=true
rg -n "array.*originalArray\|changing.*array\|convert.*array" src/Database/ --type=php -B 3 -A 3

Repository: utopia-php/database

Length of output: 45

🏁 Script executed:

# Final check: look for validation that prevents this specific change
sed -n '2642,2750p' src/Database/Database.php | cat -n

Repository: utopia-php/database

Length of output: 5648

🏁 Script executed:

# Confirm: check if there are other operators that still have IFNULL
rg -n "TYPE_ARRAY_UNIQUE|TYPE_ARRAY_REMOVE|TYPE_ARRAY_INTERSECT|TYPE_ARRAY_DIFF|TYPE_ARRAY_FILTER" src/Database/Adapter/MariaDB.php -A 8 | grep -E "IFNULL|TYPE_ARRAY"

Repository: utopia-php/database

Length of output: 598

🏁 Script executed:

# Final verification: confirm IFNULL was removed from APPEND/PREPEND but kept elsewhere
git show HEAD:src/Database/Adapter/MariaDB.php | grep -A 5 "TYPE_ARRAY_APPEND\|TYPE_ARRAY_PREPEND" | head -20

Repository: utopia-php/database

Length of output: 539

MODIFY/CHANGE COLUMN … NOT NULL will fail for existing tables with any NULL array values.

updateAttribute() (Database.php, line 2915) passes getSQLType() output directly to the adapter's updateAttribute() method, which executes ALTER TABLE … MODIFY/CHANGE COLUMN with no preceding UPDATE to backfill NULLs. Any production collection with existing NULL values in an array column will cause the ALTER TABLE to fail with a data-integrity error when this code path is triggered.

The PHP-side null-to-[] coercion only governs future writes and does not retroactively fix existing DB NULLs.

A safe approach is to issue an UPDATE to coerce NULLs to JSON_ARRAY() before the ALTER TABLE:

🛡️ Proposed fix: backfill NULLs before MODIFY 
 public function updateAttribute(string $collection, string $id, string $type, int $size, bool $signed = true, bool $array = false, ?string $newKey = null, bool $required = false): bool
 {
     $name = $this->filter($collection);
     $id = $this->filter($id);
     $newKey = empty($newKey) ? null : $this->filter($newKey);
     $type = $this->getSQLType($type, $size, $signed, $array, $required);
+
+    // Backfill NULLs before adding NOT NULL constraint on array columns
+    if ($array) {
+        $backfill = "UPDATE {$this->getSQLTable($name)} SET `{$id}` = JSON_ARRAY() WHERE `{$id}` IS NULL";
+        $this->getPDO()->prepare($backfill)->execute();
+    }
+
     if (!empty($newKey)) {
         $sql = "ALTER TABLE {$this->getSQLTable($name)} CHANGE COLUMN `{$id}` `{$newKey}` {$type};";
     } else {
         $sql = "ALTER TABLE {$this->getSQLTable($name)} MODIFY `{$id}` {$type};";
     }

Additionally, TYPE_ARRAY_APPEND and TYPE_ARRAY_PREPEND (lines 2085, 2090) dropped their IFNULL guards, while TYPE_ARRAY_REMOVE, TYPE_ARRAY_UNIQUE, TYPE_ARRAY_INTERSECT, TYPE_ARRAY_DIFF, and TYPE_ARRAY_FILTER (lines 2103–2161) retain them. This asymmetry is harmless with the NOT NULL constraint but creates inconsistency that will mislead future maintainers. Consider applying IFNULL consistently across all array operators for clarity.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@src/Database/Adapter/MariaDB.php` around lines 1686 - 1688, The ALTER path
that emits "JSON NOT NULL DEFAULT (JSON_ARRAY())" will fail when existing rows
have NULLs — modify the MariaDB adapter's updateAttribute() flow to run an
UPDATE table SET column = JSON_ARRAY() WHERE column IS NULL (or equivalent)
before executing ALTER TABLE MODIFY/CHANGE using getSQLType() so NULLs are
backfilled; also restore consistent IFNULL() guarding for array operators by
wrapping TYPE_ARRAY_APPEND and TYPE_ARRAY_PREPEND expressions with
IFNULL(column, JSON_ARRAY()) to match the other operators (e.g.,
TYPE_ARRAY_REMOVE/UNIQUE/INTERSECT/DIFF/FILTER) so behavior is uniform and safe.


switch ($type) {
Expand Down Expand Up @@ -2082,12 +2082,12 @@ protected function getOperatorSQL(string $column, Operator $operator, int &$bind
case Operator::TYPE_ARRAY_APPEND:
$bindKey = "op_{$bindIndex}";
$bindIndex++;
return "{$quotedColumn} = JSON_MERGE_PRESERVE(IFNULL({$quotedColumn}, JSON_ARRAY()), :$bindKey)";
return "{$quotedColumn} = JSON_MERGE_PRESERVE({$quotedColumn}, :$bindKey)";

case Operator::TYPE_ARRAY_PREPEND:
$bindKey = "op_{$bindIndex}";
$bindIndex++;
return "{$quotedColumn} = JSON_MERGE_PRESERVE(:$bindKey, IFNULL({$quotedColumn}, JSON_ARRAY()))";
return "{$quotedColumn} = JSON_MERGE_PRESERVE(:$bindKey, {$quotedColumn})";

case Operator::TYPE_ARRAY_INSERT:
$indexKey = "op_{$bindIndex}";
Expand Down
5 changes: 5 additions & 0 deletions src/Database/Adapter/Mongo.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3221,6 +3221,11 @@ public function getSupportForCastIndexArray(): bool
return false;
}

public function getSupportForGinIndex(): bool
{
return false;
}

public function getSupportForUpserts(): bool
{
return true;
Expand Down
5 changes: 5 additions & 0 deletions src/Database/Adapter/Pool.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -388,6 +388,11 @@ public function getSupportForCastIndexArray(): bool
return $this->delegate(__FUNCTION__, \func_get_args());
}

public function getSupportForGinIndex(): bool
{
return $this->delegate(__FUNCTION__, \func_get_args());
}

public function getSupportForUniqueIndex(): bool
{
return $this->delegate(__FUNCTION__, \func_get_args());
Expand Down
50 changes: 40 additions & 10 deletions src/Database/Adapter/Postgres.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -892,6 +892,26 @@ public function createIndex(string $collection, string $id, string $type, array
$collection = $this->filter($collection);
$id = $this->filter($id);

// JSONB array columns need GIN indexes for containment queries (@>).
$isArrayIndex = false;
if ($type === Database::INDEX_KEY) {
$metadataCollection = new Document(['$id' => Database::METADATA]);
$collectionDoc = $this->getDocument($metadataCollection, $collection);
if (!$collectionDoc->isEmpty()) {
$collectionAttributes = \json_decode($collectionDoc->getAttribute('attributes', '[]'), true);
$arrayCount = 0;
foreach ($attributes as $attr) {
foreach ($collectionAttributes as $collectionAttribute) {
if (\strtolower($collectionAttribute['$id']) === \strtolower($attr) && !empty($collectionAttribute['array'])) {
$arrayCount++;
break;
}
}
}
$isArrayIndex = $arrayCount > 0 && $arrayCount === \count($attributes);
}
}

foreach ($attributes as $i => $attr) {
$order = empty($orders[$i]) || Database::INDEX_FULLTEXT === $type ? '' : $orders[$i];
$isNestedPath = isset($indexAttributeTypes[$attr]) && \str_contains($attr, '.') && $indexAttributeTypes[$attr] === Database::VAR_OBJECT;
Expand Down Expand Up @@ -933,13 +953,13 @@ public function createIndex(string $collection, string $id, string $type, array
$sql = "CREATE {$sqlType} \"{$keyName}\" ON {$this->getSQLTable($collection)}";

// Add USING clause for special index types
$sql .= match ($type) {
Database::INDEX_SPATIAL => " USING GIST ({$attributes})",
Database::INDEX_HNSW_EUCLIDEAN => " USING HNSW ({$attributes} vector_l2_ops)",
Database::INDEX_HNSW_COSINE => " USING HNSW ({$attributes} vector_cosine_ops)",
Database::INDEX_HNSW_DOT => " USING HNSW ({$attributes} vector_ip_ops)",
Database::INDEX_OBJECT => " USING GIN ({$attributes})",
Database::INDEX_TRIGRAM =>
$sql .= match (true) {
$type === Database::INDEX_SPATIAL => " USING GIST ({$attributes})",
$type === Database::INDEX_HNSW_EUCLIDEAN => " USING HNSW ({$attributes} vector_l2_ops)",
$type === Database::INDEX_HNSW_COSINE => " USING HNSW ({$attributes} vector_cosine_ops)",
$type === Database::INDEX_HNSW_DOT => " USING HNSW ({$attributes} vector_ip_ops)",
$type === Database::INDEX_OBJECT, $isArrayIndex => " USING GIN ({$attributes})",
$type === Database::INDEX_TRIGRAM =>
" USING GIN (" . implode(', ', array_map(
fn ($attr) => "$attr gin_trgm_ops",
array_map(fn ($attr) => trim($attr), explode(',', $attributes))
Expand Down Expand Up @@ -1937,7 +1957,7 @@ protected function getFulltextValue(string $value): string
protected function getSQLType(string $type, int $size, bool $signed = true, bool $array = false, bool $required = false): string
{
if ($array === true) {
return 'JSONB';
return "JSONB NOT NULL DEFAULT '[]'::jsonb";
}
Comment on lines 1957 to 1961
Copy link
Contributor

@coderabbitai coderabbitai bot Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# Search for the getSQLType method implementation to confirm the behavior with arrays
rg -A 30 "protected function getSQLType" src/Database/Adapter/Postgres.php

Repository: utopia-php/database

Length of output: 1197

🏁 Script executed:

# Search for updateAttribute method to see how it uses getSQLType
rg -A 50 "protected function updateAttribute" src/Database/Adapter/Postgres.php | head -80

Repository: utopia-php/database

Length of output: 45

🌐 Web query:

PostgreSQL ALTER TABLE ALTER COLUMN TYPE syntax - does it support NOT NULL DEFAULT modifiers

💡 Result:

ALTER TABLE ... ALTER COLUMN ... TYPE (aka SET DATA TYPE) does not accept inline column-definition modifiers like NOT NULL or DEFAULT. It only changes the data type (optionally with USING/COLLATE). [1][2]

To change those properties, use separate subcommands:

ALTER TABLE t
  ALTER COLUMN c TYPE new_type USING ...,
  ALTER COLUMN c SET DEFAULT new_default,
  ALTER COLUMN c SET NOT NULL;

PostgreSQL will also try to cast the existing column default (if any) and constraints that reference the column when you change the type, which is why doing SET DEFAULT explicitly after a type change is sometimes clearer/safer. [1]

Sources:
[1] PostgreSQL docs (current): “Modifying Tables” (ALTER COLUMN TYPE / SET DEFAULT / SET NOT NULL) (postgresql.org)
[2] PostgreSQL ALTER TABLE reference (Postgres Pro mirror): SET DATA TYPE, SET/DROP DEFAULT, SET/DROP NOT NULL (postgrespro.com)

Citations:

🏁 Script executed:

# Search for updateAttribute method with more context
rg -A 100 "protected function updateAttribute" src/Database/Adapter/Postgres.php | head -120

Repository: utopia-php/database

Length of output: 45

🏁 Script executed:

# Also search for where getSQLType is called within updateAttribute context
rg -B 5 -A 15 "getSQLType.*\$array" src/Database/Adapter/Postgres.php

Repository: utopia-php/database

Length of output: 45

🏁 Script executed:

# Search for ALTER COLUMN in Postgres.php to find where updateAttribute uses the type
rg -B 10 "ALTER COLUMN" src/Database/Adapter/Postgres.php | head -100

Repository: utopia-php/database

Length of output: 299

🏁 Script executed:

# Try to find updateAttribute by searching for the method name more broadly
rg "updateAttribute" src/Database/Adapter/Postgres.php

Repository: utopia-php/database

Length of output: 251

🏁 Script executed:

# Get the full updateAttribute implementation
rg -A 150 "public function updateAttribute" src/Database/Adapter/Postgres.php | head -200

Repository: utopia-php/database

Length of output: 5027

updateAttribute will generate invalid SQL for array attributes.

getSQLType now returns "JSONB NOT NULL DEFAULT '[]'::jsonb" when $array === true. This string is fed directly into the ALTER COLUMN TYPE clause:

ALTER TABLE schema.table ALTER COLUMN "col" TYPE JSONB NOT NULL DEFAULT '[]'::jsonb

PostgreSQL's ALTER COLUMN TYPE grammar accepts only data_type [COLLATE …] [USING …]. The NOT NULL and DEFAULT clauses must be issued as separate ALTER COLUMN sub-commands; their presence in the TYPE clause causes a syntax error.

This is a regression: the previous return value (JSONB alone) was valid for the TYPE clause.

Proposed fix in updateAttribute 
         $type = $this->getSQLType(
             $type,
             $size,
             $signed,
             $array,
             $required,
         );

+        // For ALTER COLUMN TYPE the NOT NULL / DEFAULT modifiers must be issued
+        // as separate ALTER COLUMN sub-commands; strip them from the type string.
+        $alterType = preg_replace('/\s+NOT NULL.*$/i', '', $type);
+
         if ($type == 'TIMESTAMP(3)') {
-            $type = "TIMESTAMP(3) without time zone USING TO_TIMESTAMP(\"$id\", 'YYYY-MM-DD HH24:MI:SS.MS')";
+            $alterType = "TIMESTAMP(3) without time zone USING TO_TIMESTAMP(\"$id\", 'YYYY-MM-DD HH24:MI:SS.MS')";
         }

         // ... (use $alterType in the ALTER COLUMN sql, keep $type for other uses)
         $sql = "
             ALTER TABLE {$this->getSQLTable($name)}
-            ALTER COLUMN \"{$id}\" TYPE {$type}
+            ALTER COLUMN \"{$id}\" TYPE {$alterType}
         ";
🧰 Tools
🪛 PHPMD (2.15.0)

[warning] 1957-1957: Avoid unused parameters such as '$signed'. (undefined)

(UnusedFormalParameter)

[warning] 1957-1957: Avoid unused parameters such as '$required'. (undefined)

(UnusedFormalParameter)

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@src/Database/Adapter/Postgres.php` around lines 1957 - 1961, getSQLType
currently returns "JSONB NOT NULL DEFAULT '[]'::jsonb" for array columns which
is invalid inside an ALTER COLUMN ... TYPE clause; change getSQLType(array=true)
to return only "JSONB" and update updateAttribute to emit separate ALTER COLUMN
subcommands for NOT NULL and DEFAULT (e.g., ALTER TABLE ... ALTER COLUMN "col"
TYPE JSONB, ALTER TABLE ... ALTER COLUMN "col" SET DEFAULT '[]'::jsonb, and
ALTER TABLE ... ALTER COLUMN "col" SET NOT NULL / DROP NOT NULL as needed).
Adjust logic in updateAttribute to detect array columns (or call to getSQLType)
and append separate ALTER clauses for defaults and nullability rather than
embedding them in the TYPE string.


switch ($type) {
Expand Down Expand Up @@ -2117,6 +2137,16 @@ public function getSupportForTimeouts(): bool
return true;
}

public function getSupportForCastIndexArray(): bool
{
return true;
}

public function getSupportForGinIndex(): bool
{
return true;
}

/**
* Does the adapter handle Query Array Overlaps?
*
Expand Down Expand Up @@ -2688,12 +2718,12 @@ protected function getOperatorSQL(string $column, Operator $operator, int &$bind
case Operator::TYPE_ARRAY_APPEND:
$bindKey = "op_{$bindIndex}";
$bindIndex++;
return "{$quotedColumn} = COALESCE({$columnRef}, '[]'::jsonb) || :$bindKey::jsonb";
return "{$quotedColumn} = {$columnRef} || :$bindKey::jsonb";

case Operator::TYPE_ARRAY_PREPEND:
$bindKey = "op_{$bindIndex}";
$bindIndex++;
return "{$quotedColumn} = :$bindKey::jsonb || COALESCE({$columnRef}, '[]'::jsonb)";
return "{$quotedColumn} = :$bindKey::jsonb || {$columnRef}";

case Operator::TYPE_ARRAY_UNIQUE:
return "{$quotedColumn} = COALESCE((
Expand Down
19 changes: 19 additions & 0 deletions src/Database/Adapter/SQL.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1545,6 +1545,11 @@ public function getSupportForCastIndexArray(): bool
return false;
}

public function getSupportForGinIndex(): bool
{
return false;
}

public function getSupportForRelationships(): bool
{
return true;
Expand Down Expand Up @@ -2599,8 +2604,12 @@ public function upsertDocuments(
$spatialAttributes = $this->getSpatialAttributes($collection);

$attributeDefaults = [];
$arrayAttributes = [];
foreach ($collection->getAttribute('attributes', []) as $attr) {
$attributeDefaults[$attr['$id']] = $attr['default'] ?? null;
if ($attr['array'] ?? false) {
$arrayAttributes[$attr['$id']] = true;
}
}

$collection = $collection->getId();
Expand Down Expand Up @@ -2671,6 +2680,11 @@ public function upsertDocuments(
foreach ($allColumnNames as $attributeKey) {
$attrValue = $currentRegularAttributes[$attributeKey] ?? null;

// Array columns are NOT NULL DEFAULT '[]', so coerce null to empty array
if ($attrValue === null && isset($arrayAttributes[$attributeKey])) {
$attrValue = '[]';
}
Comment on lines +2683 to +2686
Copy link
Contributor

@coderabbitai coderabbitai bot Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# First, check the file structure and locate the relevant methods
fd "SQL.php" --type f | head -5

Repository: utopia-php/database

Length of output: 122

🏁 Script executed:

# Get line count for SQL.php to understand the file size
wc -l src/Database/Adapter/SQL.php

Repository: utopia-php/database

Length of output: 96

🏁 Script executed:

# Check the createDocuments method around line 2451-2586
sed -n '2451,2500p' src/Database/Adapter/SQL.php | cat -n

Repository: utopia-php/database

Length of output: 2189

🏁 Script executed:

# Check the $arrayAttributes collection and coercion in upsertDocuments (lines 2607-2686)
sed -n '2607,2700p' src/Database/Adapter/SQL.php | cat -n

Repository: utopia-php/database

Length of output: 4999

🏁 Script executed:

# Check the second coercion block mentioned (lines 2815-2818)
sed -n '2800,2830p' src/Database/Adapter/SQL.php | cat -n

Repository: utopia-php/database

Length of output: 1895

🏁 Script executed:

# Search for encode method in Database.php to check for upstream coercion
rg -n "public function encode|private function encode" src/Database/Database.php -A 15 | head -80

Repository: utopia-php/database

Length of output: 899

🏁 Script executed:

# See the rest of createDocuments to find the value binding logic
sed -n '2500,2586p' src/Database/Adapter/SQL.php | cat -n

Repository: utopia-php/database

Length of output: 4231

🏁 Script executed:

# See the full Database::encode() method to check for array coercion
sed -n '8177,8250p' src/Database/Database.php | cat -n

Repository: utopia-php/database

Length of output: 3322

🏁 Script executed:

# Verify that createDocuments is called in context where encode() may not be applied first
rg -n "createDocuments" src/Database/Database.php -B 3 -A 3 | head -50

Repository: utopia-php/database

Length of output: 662

🏁 Script executed:

# Check if Database::createDocuments calls encode() before adapter->createDocuments()
sed -n '5118,5200p' src/Database/Database.php | cat -n

Repository: utopia-php/database

Length of output: 4233

createDocuments is missing the same null-to-'[]' guard added here.

upsertDocuments correctly coerces null → '[]' for array-typed attributes (lines 2683–2686 and 2815–2818). However, createDocuments builds $attributeKeys as the union of all documents in the batch, then falls back to $attributes[$key] ?? null for each key. When document A has an optional array attribute and document B does not, document B binds NULL for a NOT NULL DEFAULT '[]' column → constraint violation.

🛡️ Proposed fix in createDocuments (SQL.php ~line 2456)

Collect the $arrayAttributes map the same way as upsertDocuments does, just before the loop that iterates over documents:

     $spatialAttributes = $this->getSpatialAttributes($collection);
     $collection = $collection->getId();
+
+    $arrayAttributes = [];
+    foreach ($collection->getAttribute('attributes', []) as $attr) {
+        if ($attr['array'] ?? false) {
+            $arrayAttributes[$attr['$id']] = true;
+        }
+    }

Then apply the coercion when binding values:

     foreach ($attributeKeys as $key) {
         $value = $attributes[$key] ?? null;
+        // Array columns are NOT NULL DEFAULT '[]', so coerce null to empty array
+        if ($value === null && isset($arrayAttributes[$key])) {
+            $value = '[]';
+        }
         if (\is_array($value)) {
             $value = \json_encode($value);
         }
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@src/Database/Adapter/SQL.php` around lines 2683 - 2686, createDocuments is
missing the null-to-'[]' coercion for array-typed attributes which causes NULL
to be bound for NOT NULL DEFAULT '[]' columns; mirror the fix in upsertDocuments
by building the $arrayAttributes map just before the loop that iterates over
documents (same approach used in upsertDocuments) and, in the value-binding loop
inside createDocuments where you currently use $attributes[$key] ?? null, coerce
$attrValue === null && isset($arrayAttributes[$key]) to '[]' before binding;
reference createDocuments, upsertDocuments, $attributeKeys, $attributes,
$arrayAttributes and the binding loop when applying the change.


if (\is_array($attrValue)) {
$attrValue = \json_encode($attrValue);
}
Expand Down Expand Up @@ -2798,6 +2812,11 @@ public function upsertDocuments(
foreach ($allColumnNames as $attributeKey) {
$attrValue = $currentRegularAttributes[$attributeKey] ?? null;

// Array columns are NOT NULL DEFAULT '[]', so coerce null to empty array
if ($attrValue === null && isset($arrayAttributes[$attributeKey])) {
$attrValue = '[]';
}

if (\is_array($attrValue)) {
$attrValue = \json_encode($attrValue);
}
Expand Down
7 changes: 7 additions & 0 deletions src/Database/Database.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8200,6 +8200,13 @@ public function encode(Document $collection, Document $document, bool $applyDefa

// Continue on optional param with no default
if (is_null($value) && is_null($default)) {
// Non-required array columns are NOT NULL with DEFAULT '[]', so coerce null to empty array.
// Required arrays must remain null so the Structure validator catches the missing value.
// Only coerce when applying defaults (inserts) to avoid clearing existing values during partial updates.
$required = $attribute['required'] ?? false;
if ($applyDefaults && $array && !$required) {
$document->setAttribute($key, []);
}
continue;
}

Expand Down
5 changes: 5 additions & 0 deletions tests/e2e/Adapter/Base.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,11 @@ abstract class Base extends TestCase
*/
abstract protected function getDatabase(): Database;

/**
* @return \Utopia\Database\PDO|\PDO|null
*/
abstract protected function getPDO(): mixed;

/**
* @param string $collection
* @param string $column
Expand Down
5 changes: 5 additions & 0 deletions tests/e2e/Adapter/MariaDBTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,11 @@ public function getDatabase(bool $fresh = false): Database
return self::$database = $database;
}

protected function getPDO(): mixed
{
return self::$pdo;
}

protected function deleteColumn(string $collection, string $column): bool
{
$sqlTable = "`" . $this->getDatabase()->getDatabase() . "`.`" . $this->getDatabase()->getNamespace() . "_" . $collection . "`";
Expand Down
5 changes: 5 additions & 0 deletions tests/e2e/Adapter/MirrorTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -313,6 +313,11 @@ public function testDeleteMirroredDocument(): void
$this->assertTrue($database->getDestination()->getDocument('testDeleteMirroredDocument', $document->getId())->isEmpty());
}

protected function getPDO(): mixed
{
return self::$sourcePdo;
}

protected function deleteColumn(string $collection, string $column): bool
{
$sqlTable = "`" . self::$source->getDatabase() . "`.`" . self::$source->getNamespace() . "_" . $collection . "`";
Expand Down
5 changes: 5 additions & 0 deletions tests/e2e/Adapter/MongoDBTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,11 @@ public function testKeywords(): void
$this->assertTrue(true);
}

protected function getPDO(): mixed
{
return null;
}

protected function deleteColumn(string $collection, string $column): bool
{
return true;
Expand Down
5 changes: 5 additions & 0 deletions tests/e2e/Adapter/MySQLTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,11 @@ public function getDatabase(): Database
return self::$database = $database;
}

protected function getPDO(): mixed
{
return self::$pdo;
}

protected function deleteColumn(string $collection, string $column): bool
{
$sqlTable = "`" . $this->getDatabase()->getDatabase() . "`.`" . $this->getDatabase()->getNamespace() . "_" . $collection . "`";
Expand Down
12 changes: 12 additions & 0 deletions tests/e2e/Adapter/PoolTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,18 @@ public function getDatabase(): Database
return self::$database = $database;
}

protected function getPDO(): mixed
{
$pdo = null;
self::$pool->use(function (Adapter $adapter) use (&$pdo) {
$class = new ReflectionClass($adapter);
$property = $class->getProperty('pdo');
$property->setAccessible(true);
$pdo = $property->getValue($adapter);
});
return $pdo;
}

protected function deleteColumn(string $collection, string $column): bool
{
$sqlTable = "`" . $this->getDatabase()->getDatabase() . "`.`" . $this->getDatabase()->getNamespace() . "_" . $collection . "`";
Expand Down
5 changes: 5 additions & 0 deletions tests/e2e/Adapter/PostgresTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,11 @@ public function getDatabase(): Database
return self::$database = $database;
}

protected function getPDO(): mixed
{
return self::$pdo;
}

protected function deleteColumn(string $collection, string $column): bool
{
$sqlTable = '"' . $this->getDatabase()->getDatabase(). '"."' . $this->getDatabase()->getNamespace() . '_' . $collection . '"';
Expand Down
5 changes: 5 additions & 0 deletions tests/e2e/Adapter/SQLiteTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,11 @@ public function getDatabase(): Database
return self::$database = $database;
}

protected function getPDO(): mixed
{
return self::$pdo;
}

protected function deleteColumn(string $collection, string $column): bool
{
$sqlTable = "`" . $this->getDatabase()->getNamespace() . "_" . $collection . "`";
Expand Down
3 changes: 2 additions & 1 deletion tests/e2e/Adapter/Scopes/AttributeTests.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1620,10 +1620,11 @@ public function testArrayAttribute(): void
$this->assertEquals('Invalid query: Cannot query contains on attribute "age" because it is not an array, string, or object.', $e->getMessage());
}

// Array columns are NOT NULL with DEFAULT empty array, so isNull returns nothing
$documents = $database->find($collection, [
Query::isNull('long_size')
]);
$this->assertCount(1, $documents);
$this->assertCount(0, $documents);

$documents = $database->find($collection, [
Query::contains('tv_show', ['love'])
Expand Down
6 changes: 3 additions & 3 deletions tests/e2e/Adapter/Scopes/CollectionTests.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -569,10 +569,10 @@ public function testSchemaAttributes(): void

$attribute = $attributes['string_list'];
$this->assertEquals('string_list', $attribute['$id']);
$this->assertTrue(in_array($attribute['dataType'], ['json', 'longtext'])); // mysql vs maria
$this->assertTrue(in_array($attribute['columnType'], ['json', 'longtext']));
$this->assertTrue(in_array($attribute['dataType'], ['json', 'jsonb', 'longtext']));
$this->assertTrue(in_array($attribute['columnType'], ['json', 'jsonb', 'longtext']));
$this->assertTrue(in_array($attribute['characterMaximumLength'], [null, '4294967295']));
$this->assertEquals('YES', $attribute['isNullable']);
$this->assertEquals('NO', $attribute['isNullable']);

$attribute = $attributes['dob'];
$this->assertEquals('dob', $attribute['$id']);
Expand Down
Loading
Loading