Skip to content

Implement v0.9 spec and "Bring Your Own LLM" API#694

Closed
gspencergoog wants to merge 67 commits intoflutter:mainfrom
gspencergoog:feature/v0.9-migration-byo-llm-2
Closed

Implement v0.9 spec and "Bring Your Own LLM" API#694
gspencergoog wants to merge 67 commits intoflutter:mainfrom
gspencergoog:feature/v0.9-migration-byo-llm-2

Conversation

@gspencergoog
Copy link
Collaborator

Description

This implements the A2UI v0.9 spec and reorganizes the API around a "Bring your own LLM" strategy which eliminates ContentGenerators and their associated packages.

Refactor message keys and tool usage to match the A2UI v0.9 specification:
- Rename `SurfaceUpdate` related handling to `UpdateComponents`.
- Rename `BeginRendering` related handling to `CreateSurface`.
- Update `A2uiAgentConnector` to recognize `updateComponents`, `updateDataModel`, `createSurface`.
- Update AI content generators (`dartantic`, `firebase_ai`, `google_generative_ai`) to use `UpdateComponentsTool` and `CreateSurfaceTool`.
- Fix tests and example app to reflect these changes.
- Implement  for two-way binding.
- Implement  for schema validation.
- Add  schema.
- Add tests for new functionality.
Fix schema generation for empty catalogs, clean up formatting, and update custom_backend test data to match v0.9 Component schema.
- Update Catalog definition to include `functions` schema property.
- Add standard client function definitions (required, regex, etc.) to A2uiSchemas.
- Add unit tests for A2uiMessage serialization/deserialization.
- Ensure all schema helper methods in A2uiSchemas are static.

Detailed changes:
- `packages/genui/lib/src/model/catalog.dart`: Added `functions` to `Catalog.definition`.
- `packages/genui/lib/src/model/a2ui_schemas.dart`: Implemented schemas for `required`, `regex`, `length`, `numeric`, `email`, `formatString`, `formatNumber`, `formatCurrency`, `formatDate`. Corrected missing static modifiers.
- `packages/genui/test/model/a2ui_message_test.dart`: New test file.
- Update all core widgets to use flattened properties and V0.9 schema.
- Implement  to replace .
- Implement  and standard client-side functions.
- Update  to support  resolution.
- Add  for ${...} expression evaluation.
- Update  with function definitions.
- Fix examples and tests to align with v0.9 specification.
- Remove legacy  widget.
- Update `Action` schema to v0.9 (event/functionCall).
- Implement `checks` support in `TextField` for client-side validation.
- Add `closeModal` client-side function support in `Button` and `TextField`.
- Fix `formatString` and implementation of `openUrl` in `FunctionRegistry`.
- Resolve checks and updates tests to new schema.
- Improve public API documentation across the package.
- Remove "stream of consciousness" comments.
- Remove references to specific spec versions (e.g. v0.9).
- Update README.md and DESIGN.md to reflect current architecture and naming.
- Update README.md for genui_a2ui, genui_firebase_ai, genui_dartantic, and genui_google_generative_ai.
- Improve public API documentation in all integration packages.
- Remove legacy terms and outdated comments.
- Fix legacy class references (e.g. GenUiManager -> A2uiMessageProcessor).
Updates READMEs, dartdocs, and removes legacy references (e.g. GenUiManager, v0.8) across genui_a2ui, genui_firebase_ai, genui_dartantic, and genui_google_generative_ai.
This commit addresses the remaining identified shortfalls for Phase 2 of the
GenUI v0.9 migration.

Key changes:
- Tabs: Added 'activeTab' support to the Tabs component, allowing bidirectional
  binding of the active tab index to the data model.
- TextField: Refined the 'checks' schema to use 'func' instead of 'call' and
  implemented robust validation triggering on initialization and external updates.
- ExpressionParser: Standardized on using 'func' as the key for function calls
  instead of 'call', ensuring consistency with the A2UI schema.
- A2uiSchemas: Added 'logicExpression' and 'validationCheck' schema definitions.
- WidgetUtilities: Added 'subscribeToNumber' extension method for cleaner
  numeric data binding.

Verified with new tests in 'tabs_test.dart' and 'text_field_test.dart'.
…t data model)

This commit completes Phase 3 of the GenUI v0.9 migration.

Key changes:
- `ContentGenerator`: Added `eventStream`, `addInterceptor`, `removeInterceptor`.
- `ContentGeneratorMixin`: New mixin for shared event/interceptor logic.
- `GenUiEvent`: Added `ToolStartEvent`, `ToolEndEvent`, etc.
- `GenUiConversation`: Exposes `processingEvents` and passes `clientDataModel` to generators.
- Generators: Updated `GoogleGenerativeAiContentGenerator`, `FirebaseAiContentGenerator`, `DartanticContentGenerator`, and `A2uiContentGenerator` to support the new API, emit events, and handle interceptors (including Mock and Cancel actions).
- `A2uiAgentConnector`: Updated to include `clientDataModel` in request metadata.

BREAKING CHANGE: `ContentGenerator.sendRequest` now requires a `clientDataModel` parameter (nullable) and `dispose` must call `disposeMixin` if using the mixin.
Updates Google, Firebase, and Dartantic providers to inject the A2UI v0.9 `standard_catalog.json` and rules directly into the system prompt.
This ensures models have the necessary schema and rules to generate valid v0.9 A2UI content.

Changes:
- Adds `StandardCatalogEmbed` to `genui` package containing the standard catalog and rules.
- Updates `GoogleGenerativeAiContentGenerator` to append catalog/rules to system instructions.
- Updates `FirebaseAiContentGenerator` to append catalog/rules to system instructions.
- Updates `DartanticContentGenerator` to inject catalog/rules via extra instructions.
- Fixes `A2uiMessageProcessor` tests to align with `SurfaceAdded` -> `ComponentsUpdated` lifecycle events.
- Fixes `ui_tools_test.dart` to verify `ComponentsUpdated` events.

BREAKING-CHANGE: System prompts for all providers now include the A2UI Standard Catalog by default. This may affect token usage and model behavior.
- Add input validation `checks` support to `TextField` and `DateTimeInput`.
- Implement concurrency guards in `GenUiConversation`.
- Add surface ID validation to `A2uiMessageProcessor`.
- Fix duplicate tool registration in `GoogleGenerativeAiContentGenerator`.
- Enhance `GenUiSurfaceManager` and add `GenUiFallback` widget.
- Clean up lints and documentation comments.
The `Action` schema in A2UI v0.9 requires the action to be wrapped in an `event`
object (or `functionCall`). The examples in `Trailhead`, `TravelCarousel`, and
`InputGroup` were using a legacy flat structure. This change wraps them
correctly to pass validation tests.

Also includes a dev note in `GEMINI.md` about running tests efficiently.
… schema

- Remove hardcoded standardCatalogJson from StandardCatalogEmbed to support dynamic schemas.
- Update Google, Firebase, and Dartantic content generators to inject catalog.definition.toJson() into system prompts.
- Add JsonBlockParser to extract JSON from LLM text responses (prompt-first approach).
- Refactor content generators to parse JSON from text responses instead of tool calls.
- Fix DartanticContentGenerator to use Agent.send and parse text output.
…tors

- Conditionally set toolConfig in GoogleGenerativeAiContentGenerator and FirebaseAiContentGenerator only when tools are present.
- Prevents 'Function calling config is set without function_declarations' error from the API.
- Replace catalog.definition with A2uiMessage.a2uiMessageSchema(catalog) in content generators.
- Ensures LLM receives schema for top-level messages (createSurface, etc.) not just components.
- Fixes issues where LLM output was missing required fields like 'catalogId'.
- Add explicit JSON examples to StandardCatalogEmbed to guide LLM for createSurface/updateComponents.
- Add debug logging to A2uiMessage.fromJson to print JSON content when parsing fails.
- This helps diagnose and prevent JSON hallucination issues.
- Add JsonBlockParser.stripJsonBlock to strip JSON from text responses.
- Update content generators in genui_google_generative_ai, genui_firebase_ai, and genui_dartantic to use this.
- Update StandardCatalogEmbed examples to enforce 'id': 'root' for the top-level component, preventing 'no root component' errors.
- Verified that schema descriptions are correctly included in the JSON schema.
- Update JsonBlockParser to handle multiple JSON blocks in a single text response.
- Standardize extracting all JSON blocks and stripping them from the text in all content generators.
- Enable full prompt logging in GoogleGenerativeAiContentGenerator to aid debugging.
- This fixes issues where separate 'CreateSurface' and 'UpdateComponents' blocks were not both being processed or stripped.
…d with literal values

Updated standard catalog input components (TextField, CheckBox, ChoicePicker, DateTimeInput, Slider, Tabs) and travel_app components (OptionsFilterChipInput, TextInputChip, DateInputChip, CheckboxFilterChipsInput) to support implicit data binding.

When these components are initialized with a literal value (e.g., "value": "some string"), they now default to binding to '${context.id}.value' (or similar) in the DataModel.

This ensures that user interactions correctly update the DataModel and preserve state, which was previously failing for literal initializations.
Replaced the unreliable Future.delayed(Duration.zero) call with await subscription.cancel() to ensure stream completion before proceeding, avoiding race conditions.

refactor(travel_app): implement AiClient interface for better extensibility

Introduces `AiClient` interface and updates `GoogleGenerativeAiClient` and `FakeAiClient` to implement it.
Refactors `TravelPlannerPage` to use `AiClient` instead of `Object` and removes type checks in wiring methods.
This simplifies `_wireClient` and `_sendRequest` and makes the client pluggable.
refactor(travel_app): use JsonBlockParser for JSON extraction

Replaces local `_parseJsonBlocks` and `_stripJsonBlock` helpers with `JsonBlockParser` from `package:genui` to remove duplication and use the centralized utility.
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request primarily focuses on migrating the GenUI framework and its examples to A2UI protocol version 0.9, which involves significant renaming and restructuring of core components and message types. Key changes include replacing BeginRendering with CreateSurface, SurfaceUpdate with UpdateComponents, and DataModelUpdate with UpdateDataModel. The component structure in JSON payloads has been flattened, moving from nested objects like {"Text": {...}} to a more direct {"component": "Text", ...} format. The GenUiSurface widget's host parameter has been renamed to genUiContext across the codebase. The ContentGenerator abstraction has been replaced by a new AiClient interface and GenUiController for managing LLM interactions and UI state, leading to the removal of Firebase-specific AI integrations and the introduction of dartantic_ai in the simple_chat example. Platform-specific plugin registrations were updated to remove Firebase and add url_launcher dependencies. Review comments highlight a bug in the _generate method of GoogleGenerativeAiClient causing duplicate error emissions, a concern about the custom JSON schema validator in UiDefinition being prone to errors and incomplete, and an ambiguous handling of null values in list updates within DataModel._updateValue. Additionally, the _handleChatMessage method in ChatSession was noted for only logging events instead of triggering further LLM requests, making the example incomplete.

Comment on lines 553 to 555
genUiLogger.severe('Error from service.generateContent', e, st);
_errorController.add(ContentGeneratorError(e, st));
_errorController.add(e);
rethrow;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The _generate method catches exceptions, adds them to the _errorController, and then re-throws them. The calling method, sendRequest, also has a try-catch block that will catch this re-thrown exception and add it to the same _errorController. This will result in duplicate error events being emitted for a single failure.

The migration plan correctly identifies this as BUG-004.

Suggested change
genUiLogger.severe('Error from service.generateContent', e, st);
_errorController.add(ContentGeneratorError(e, st));
_errorController.add(e);
rethrow;
_errorController.add(e);
return;

Comment on lines +156 to +318
void validate(Schema schema) {
final String jsonOutput = schema.toJson();
final schemaMap = jsonDecode(jsonOutput) as Map<String, dynamic>;

List<Map<String, dynamic>> allowedSchemas = [];
if (schemaMap.containsKey('oneOf')) {
allowedSchemas = (schemaMap['oneOf'] as List)
.cast<Map<String, dynamic>>();
} else if (schemaMap.containsKey('properties') &&
(schemaMap['properties'] as Map).containsKey('components')) {
final componentsProp =
(schemaMap['properties'] as Map)['components']
as Map<String, dynamic>;
if (componentsProp.containsKey('items')) {
final items = componentsProp['items'] as Map<String, dynamic>;
if (items.containsKey('oneOf')) {
allowedSchemas = (items['oneOf'] as List)
.cast<Map<String, dynamic>>();
} else {
allowedSchemas = [items];
}
}
}

if (allowedSchemas.isEmpty) {
return;
}

for (final Component component in components.values) {
var matched = false;
List<String> errors = [];
final JsonMap instanceJson = component.toJson();

for (final s in allowedSchemas) {
if (_schemaMatchesType(s, component.type)) {
try {
_validateInstance(instanceJson, s, '/components/${component.id}');
matched = true;
break;
} catch (e) {
errors.add(e.toString());
}
}
}

if (!matched) {
if (errors.isNotEmpty) {
throw GenUiValidationException(
surfaceId: surfaceId,
message:
'Validation failed for component ${component.id} '
'(${component.type}): ${errors.join("; ")}',
path: '/components/${component.id}',
);
}
throw GenUiValidationException(
surfaceId: surfaceId,
message: 'Unknown component type: ${component.type}',
path: '/components/${component.id}',
);
}
}
}

bool _schemaMatchesType(Map<String, dynamic> schema, String type) {
if (schema.containsKey('properties')) {
final props = schema['properties'] as Map;
if (props.containsKey('component')) {
final compProp = props['component'] as Map<String, dynamic>;
if (compProp.containsKey('const') && compProp['const'] == type) {
return true;
}
if (compProp.containsKey('enum') &&
(compProp['enum'] as List).contains(type)) {
return true;
}
}
}
return false;
}

void _validateInstance(
Object? instance,
Map<String, dynamic> schema,
String path,
) {
if (instance == null) {
return;
}

if (schema.containsKey('const')) {
final Object? constVal = schema['const'];
if (instance != constVal) {
throw GenUiValidationException(
surfaceId: surfaceId,
message: 'Value mismatch. Expected $constVal, got $instance',
path: path,
);
}
}

if (schema.containsKey('enum')) {
final enums = schema['enum'] as List;
if (!enums.contains(instance)) {
throw GenUiValidationException(
surfaceId: surfaceId,
message: 'Value not in enum: $instance',
path: path,
);
}
}

if (schema.containsKey('required') && instance is Map) {
final List<String> required = (schema['required'] as List).cast<String>();
for (final key in required) {
if (!instance.containsKey(key)) {
throw GenUiValidationException(
surfaceId: surfaceId,
message: 'Missing required property: $key',
path: path,
);
}
}
}

if (schema.containsKey('properties') && instance is Map) {
final props = schema['properties'] as Map<String, dynamic>;
for (final MapEntry<String, dynamic> entry in props.entries) {
final String key = entry.key;
final propSchema = entry.value as Map<String, dynamic>;
if (instance.containsKey(key)) {
_validateInstance(instance[key], propSchema, '$path/$key');
}
}
}

if (schema.containsKey('items') && instance is List) {
final itemsSchema = schema['items'] as Map<String, dynamic>;
for (var i = 0; i < instance.length; i++) {
_validateInstance(instance[i], itemsSchema, '$path/$i');
}
}

if (schema.containsKey('oneOf')) {
final List<Map<String, dynamic>> oneOfs = (schema['oneOf'] as List)
.cast<Map<String, dynamic>>();
var oneMatched = false;
for (final s in oneOfs) {
try {
_validateInstance(instance, s, path);
oneMatched = true;
break;
} catch (_) {}
}
if (!oneMatched) {
throw GenUiValidationException(
surfaceId: surfaceId,
message: 'Value did not match any oneOf schema',
path: path,
);
}
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The new validate method implements a custom JSON schema validator from scratch. Re-implementing a validator is complex and prone to errors. This custom implementation appears to be missing support for several JSON schema features like numeric/string constraints (minimum, maxLength), allOf, etc., which could lead to incomplete validation and allow invalid data to pass.

Consider using the validation capabilities of an established library like json_schema_builder (which is already a dependency) to ensure comprehensive and correct validation. If a custom implementation is necessary for performance or other reasons, it would be beneficial to document its limitations.

Comment on lines 346 to 354
if (index < current.length) {
current[index] = value;
if (value == null) {
current[index] = value;
} else {
current[index] = value;
}
} else if (index == current.length) {
current.add(value);
} else {
throw ArgumentError(
'Index out of bounds for list update: index ($index) is greater '
'than list length (${current.length}).',
);
if (value != null) current.add(value);
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The logic for updating lists within _updateValue handles null values ambiguously. When value is null, current[index] = value sets an element to null rather than removing it, which is inconsistent with how null values are handled for maps (where the key is removed). Additionally, if index == current.length, a null value is added to the list. This behavior could lead to unexpected null entries in lists and deviate from the intended semantics of null meaning 'delete'.

Suggested change
if (index < current.length) {
current[index] = value;
if (value == null) {
current[index] = value;
} else {
current[index] = value;
}
} else if (index == current.length) {
current.add(value);
} else {
throw ArgumentError(
'Index out of bounds for list update: index ($index) is greater '
'than list length (${current.length}).',
);
if (value != null) current.add(value);
}
if (index < current.length) {
if (value == null) {
current.removeAt(index);
} else {
current[index] = value;
}
} else if (index == current.length && value != null) {
current.add(value);
}

Comment on lines 107 to 109
void _handleChatMessage(ChatMessage event) {
genUiLogger.info('Received chat message: ${event.toJson()}');
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The _handleChatMessage method, which listens for UI interaction events, currently only logs the event. In a functional chat application, this should trigger a new request to the LLM to continue the conversation. As it stands, the example is incomplete and could be misleading for developers using it as a reference.

  void _handleChatMessage(ChatMessage event) {
    genUiLogger.info('Received chat message: ${event.toJson()}');
    // For this example, we'll extract the text from the interaction and send it back.
    // A real app might have more complex logic to handle different actions.
    final interactionPart = event.parts.whereType<UiInteractionPart>().firstOrNull;
    if (interactionPart != null) {
      // The interaction part contains a JSON string. For simplicity, we send it as text.
      sendMessage(interactionPart.interaction);
    }
  }

@gspencergoog gspencergoog changed the title Add Migration Plan Implement v0.9 spec and "Bring Your Own LLM" API Feb 3, 2026
- Rename global `GenUiContext` to `GenUiHost`.
- Introduce new `GenUiContext` scoped to individual surfaces.
- Update `GenUiSurface` and `GenUiSurfaceManager` to use the new interfaces.
- Remove deprecated members from `GenUiHost` (`catalogs`, `dataModels`, etc.).
- Update documentation and diagrams to reflect these architectural changes.

fix(genui): relax A2UI message version validation
- Allow both "v0.9" and "0.9" as valid version strings in `A2uiMessage` parsing to handle LLM outputs that miss the 'v'.

feat(simple_chat): implement interaction handling
- Update `ChatSession` to properly serialize `UiInteractionPart` events and send them back to the LLM as text messages.

docs: update API design and usage guides
- Update `DESIGN.md`, `guide.md`, and class diagrams.
- Add `api_design.md`.

BREAKING CHANGE: `GenUiContext` is now `GenUiHost`. Custom implementations of `GenUiContext` must be updated to implement `GenUiHost` or the new scoped `GenUiContext` depending on usage. `GenUiSurface` constructor now requires a scoped `GenUiContext`.
///
/// It wraps the [A2uiParserTransformer] to provide an imperative, push-based
/// interface that is easier to integrate into imperative loops.
class GenUiController {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that this does not own an A2UIMessageProcessor etc, it seems like it's adding very little on top of A2uiParserTransformer. Could we remove it and just have A2uiParserTransformer used directly?

@gspencergoog
Copy link
Collaborator Author

Please check the feature/v0.9-migration branch instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments