Implement ExternalService Framework #17009

Wei-hao-Li · 2026-01-12T01:10:27Z

No description provided.

Signed-off-by: Weihao Li <[email protected]>

JackieTien97

partial review, finish CN and antlr parts.

...lational-grammar/src/main/antlr4/org/apache/iotdb/db/relational/grammar/sql/RelationalSql.g4

iotdb-core/confignode/src/test/resources/confignode1conf/iotdb-system.properties

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java

...e/src/main/java/org/apache/iotdb/confignode/manager/externalservice/ExternalServiceInfo.java

Signed-off-by: Weihao Li <[email protected]>

JackieTien97

Add all ITs you list in the documents and the recovery process is missing

.../node-commons/src/main/java/org/apache/iotdb/commons/schema/column/ColumnHeaderConstant.java

iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/service/ServiceType.java

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/service/DataNode.java

...ode/src/main/java/org/apache/iotdb/db/queryengine/plan/relational/sql/parser/AstBuilder.java

.../main/java/org/apache/iotdb/db/queryengine/plan/execution/config/TableConfigTaskVisitor.java

...rg/apache/iotdb/db/queryengine/plan/execution/config/executor/ClusterConfigTaskExecutor.java

...tanode/src/main/java/org/apache/iotdb/db/service/exernalservice/BuiltinExternalServices.java

Signed-off-by: Weihao Li <[email protected]>

Copilot

Pull request overview

This pull request implements a comprehensive ExternalService Framework that enables management of both built-in and user-defined services in IoTDB. The framework provides complete lifecycle management (create, start, stop, drop, show) for services through SQL statements, with persistence on ConfigNode and execution on DataNode.

Changes:

Adds Thrift protocol definitions for external service communication between nodes
Implements SQL grammar for service management operations (CREATE/START/STOP/DROP/SHOW SERVICE)
Creates service management infrastructure on both DataNode and ConfigNode with consensus-based persistence

Reviewed changes

Copilot reviewed 78 out of 78 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
datanode.thrift, confignode.thrift, common.thrift	Defines Thrift structures and RPC methods for service communication
RelationalSql.g4, IoTDBSqlParser.g4	Adds SQL grammar rules for service management statements
ServiceInfo.java	Core data structure representing service state and metadata
ExternalServiceManagementService.java	DataNode service managing local service instances
ExternalServiceManager.java, ExternalServiceInfo.java	ConfigNode components for global service coordination
IExternalService.java	API interface for implementing external services
ShowExternalServiceTask.java, etc.	Query execution tasks for service operations
IoTDBServicesIT.java	Integration tests validating service functionality

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-14T01:22:21Z

iotdb-protocol/thrift-commons/src/main/thrift/common.thrift

+  1: required string serviceName
+  2: required string className
+  3: required byte state
+  4: required i32 dataNodId


Corrected spelling of 'dataNodId' to 'dataNodeId'.

...c/main/java/org/apache/iotdb/db/service/exernalservice/ExternalServiceManagementService.java

.../db/queryengine/plan/execution/config/metadata/externalservice/StartExternalServiceTask.java

Copilot · 2026-01-14T01:22:22Z

...b/db/queryengine/plan/execution/config/metadata/externalservice/ShowExternalServiceTask.java

+      TExternalServiceEntry externalServiceEntry, ColumnBuilder[] columnBuilders) {
+    columnBuilders[0].writeBinary(
+        new Binary(externalServiceEntry.getServiceName(), TSFileConfig.STRING_CHARSET));
+    columnBuilders[1].writeInt(externalServiceEntry.getDataNodId());


Corrected method name from 'getDataNodId' to 'getDataNodeId'.

...e/src/main/java/org/apache/iotdb/confignode/manager/externalservice/ExternalServiceInfo.java

...c/main/java/org/apache/iotdb/db/service/exernalservice/ExternalServiceManagementService.java

...g/apache/iotdb/confignode/consensus/request/read/exernalservice/ShowExternalServicePlan.java

...test/java/org/apache/iotdb/relational/it/query/recent/informationschema/IoTDBServicesIT.java

Copilot · 2026-01-14T01:22:24Z

...e/src/main/java/org/apache/iotdb/confignode/manager/externalservice/ExternalServiceInfo.java

+      throws IOException {
+    int length = ReadWriteIOUtils.readInt(inputStream);
+    byte[] bytes = new byte[length];
+    inputStream.read(bytes);


Method deserializeServiceInfoConsiderCRC ignores exceptional return value of InputStream.read.

Suggested change

inputStream.read(bytes);

int offset = 0;

while (offset < length) {

int read = inputStream.read(bytes, offset, length - offset);

if (read == -1) {

throw new IOException("Unexpected end of stream while reading service info.");

}

offset += read;

}

iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/parser/ASTVisitor.java

Signed-off-by: Weihao Li <[email protected]>

don't persist on CN when start or stop built-in service Signed-off-by: Weihao Li <[email protected]>

Wei-hao-Li added 8 commits December 23, 2025 17:27

add api

4d61221

Signed-off-by: Weihao Li <[email protected]>

Merge branch 'master' of github.com:apache/iotdb into exteranlservice

86e3dfa

add process in CN: ExternalServiceInfo

8f93984

Signed-off-by: Weihao Li <[email protected]>

draft

720c7c5

Signed-off-by: Weihao Li <[email protected]>

done

3e4cf34

Signed-off-by: Weihao Li <[email protected]>

Merge branch 'master' of github.com:apache/iotdb into exteranlservice

88568d0

fix Some

cd49eee

Signed-off-by: Weihao Li <[email protected]>

add snapshot test

3ff5dd2

Signed-off-by: Weihao Li <[email protected]>

JackieTien97 requested changes Jan 12, 2026

View reviewed changes

fix according review

f7aff40

Signed-off-by: Weihao Li <[email protected]>

JackieTien97 requested changes Jan 12, 2026

View reviewed changes

Wei-hao-Li added 6 commits January 13, 2026 01:05

add restore logic when DN start

b1f1ed6

Signed-off-by: Weihao Li <[email protected]>

fix according DN review

4f4c5bf

Signed-off-by: Weihao Li <[email protected]>

Merge branch 'master' of github.com:apache/iotdb into exteranlservice

9b906d2

fix tests

e3dda43

Signed-off-by: Weihao Li <[email protected]>

disable service management SQL

db6c1f3

Signed-off-by: Weihao Li <[email protected]>

modify snapshot logic

3b94e8c

Signed-off-by: Weihao Li <[email protected]>

JackieTien97 requested a review from Copilot January 14, 2026 01:13

Copilot started reviewing on behalf of JackieTien97 January 14, 2026 01:13 View session

Copilot AI reviewed Jan 14, 2026

View reviewed changes

Wei-hao-Li added 2 commits January 14, 2026 10:47

modify according

23e44b8

Signed-off-by: Weihao Li <[email protected]>

disable some columns when show services &

8ce8ec2

don't persist on CN when start or stop built-in service Signed-off-by: Weihao Li <[email protected]>

-    inputStream.read(bytes);
+    int offset = 0;
+    while (offset < length) {
+      int read = inputStream.read(bytes, offset, length - offset);
+      if (read == -1) {
+        throw new IOException("Unexpected end of stream while reading service info.");
+      }
+      offset += read;
+    }

Implement ExternalService Framework #17009

Are you sure you want to change the base?

Implement ExternalService Framework #17009

Uh oh!

Conversation

Wei-hao-Li commented Jan 12, 2026

Uh oh!

JackieTien97 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JackieTien97 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Jan 14, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants