[FLINK-14925][table] Support precision-aware TO_TIMESTAMP with format-based inference by raminqaf · Pull Request #27793 · apache/flink

raminqaf · 2026-03-20T08:53:34Z

What is the purpose of the change

This pull request makes the TO_TIMESTAMP function precision-aware when a format pattern is provided. Previously,
TO_TIMESTAMP always returned TIMESTAMP(3) regardless of the format pattern's fractional second precision, which forced users to lose sub-millisecond data. This is the TO_TIMESTAMP counterpart to the TO_TIMESTAMP_LTZ precision support added in FLINK-39244.

The output type for the 1-arg variant remains TIMESTAMP(3) for backward compatibility. For the 2-arg variant, precision is
inferred from the format pattern's trailing S count (e.g., SSSSSS → TIMESTAMP(6)), with a minimum of 3.

As part of this change, TO_TIMESTAMP is migrated from the legacy Calcite-native function pattern (FlinkSqlOperatorTable + StringCallGen codegen) to the modern bridging function pattern (BuiltInFunctionDefinition + runtimeClass), matching how
TO_TIMESTAMP_LTZ is implemented. This was made possible by fixing the function name from camelCase "toTimestamp" to "TO_TIMESTAMP", which allows CoreModule to resolve it correctly for SQL queries without needing a separate FlinkSqlOperatorTable entry.

Brief change log

Type strategy (ToTimestampTypeStrategy): New output type strategy that returns TIMESTAMP(3) for the 1-arg variant
and TIMESTAMP(max(sCount, 3)) for the 2-arg variant, where sCount is inferred from the format pattern's trailing S
characters.
Runtime function (ToTimestampFunction): New runtime class with eval(StringData) and eval(StringData, StringData)
methods. The 2-arg variant passes precisionFromFormat(format) to parseTimestampData for precision-aware parsing.
Function definition (BuiltInFunctionDefinitions): Changed name from "toTimestamp" to "TO_TIMESTAMP" (removing the need for explicit sqlName), added runtimeClass, and switched output type strategy to SpecificTypeStrategies.TO_TIMESTAMP.
Removed legacy plumbing: Removed FlinkSqlOperatorTable.TO_TIMESTAMP, DirectConvertRule mapping, StringCallGen
cases, and BuiltInMethods.STRING_TO_TIMESTAMP / STRING_TO_TIMESTAMP_WITH_FORMAT — all superseded by the bridging function mechanism.
Documentation: Updated sql_functions.yml, sql_functions_zh.yml, and Python expressions.py / expression.py
docstrings with precision-dependent output types and examples.

Verifying this change

This change added tests and can be verified as follows:

Added type strategy unit tests in ToTimestampTypeStrategyTest covering 1-arg default precision, 2-arg format-based
precision (SSS/SSSSSS/SSSSSSSSS/no-S), invalid argument types, and argument count validation.
Added integration tests in TimeFunctionsITCase for 1-arg truncation to precision 3, 2-arg precision 6/9 from format, SSS format staying at precision 3, fewer input digits than format precision, unparsable string, and null input.
Removed redundant legacy tests from TemporalTypesTest.scala that are now covered by the new TimeFunctionsITCase tests.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): no
The serializers: no
The runtime per-record code paths (performance sensitive): no — the ToTimestampFunction.eval() methods call the same DateTimeUtils.parseTimestampData methods as before.
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? docs / JavaDocs / PyDocs

…-based inference

flinkbot · 2026-03-20T08:56:51Z

CI report:

edd47b2 Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

snuyanzin · 2026-03-20T09:28:40Z

docs/data/sql_functions.yml

+      Converts a datetime string to a TIMESTAMP without time zone.
+
+      - string1: the datetime string to parse
+      - string2: the format pattern (default 'yyyy-MM-dd HH:mm:ss'). The pattern follows Java's DateTimeFormatter syntax, where 'S' represents fractional seconds (e.g., 'SSS' for milliseconds, 'SSSSSS' for microseconds, 'SSSSSSSSS' for nanoseconds).


looks like we are going to heavily introduce java classes here....

Would be great if we have it consistent at least.

I'm asking since here we are talking about DateTimeFormatter, in case of DATE_FORMAT there is SimpleDateFormatter...

I think we should have at least a link to their doc explaining the formats, it will simplify the search for non java people (sql, python)

davidradl · 2026-03-20T10:15:32Z

docs/data/sql_functions.yml

+
+      The output precision depends on the variant used:
+      - 1-arg variant: always returns TIMESTAMP(3).
+      - 2-arg variant: precision is inferred from the number of trailing 'S' characters in the format pattern, with a minimum of 3. E.g., format 'yyyy-MM-dd HH:mm:ss.SS' returns TIMESTAMP(3), format 'yyyy-MM-dd HH:mm:ss.SSSSSS' returns TIMESTAMP(6).


what happens when the number of S's is not 3, 6 or 9.

snuyanzin · 2026-03-20T10:33:01Z

...able-planner/src/test/java/org/apache/flink/table/planner/functions/TimeFunctionsITCase.java

+                                toTimestamp("2023-01-01 12:30:00"),
+                                "TO_TIMESTAMP('2023-01-01 12:30:00')",
+                                LocalDateTime.of(2023, 1, 1, 12, 30, 0),
+                                TIMESTAMP(3).nullable())


can we have a couple of tests for non 0, 3, 6, 9 precision?

snuyanzin · 2026-03-20T10:33:41Z

...ntime/src/main/java/org/apache/flink/table/runtime/functions/scalar/ToTimestampFunction.java

+        super(BuiltInFunctionDefinitions.TO_TIMESTAMP, context);
+    }
+
+    public @Nullable TimestampData eval(StringData timestamp) {


Suggested change

public @Nullable TimestampData eval(StringData timestamp) {

public @Nullable TimestampData eval(@Nullable StringData timestamp) {

snuyanzin · 2026-03-20T10:34:04Z

...ntime/src/main/java/org/apache/flink/table/runtime/functions/scalar/ToTimestampFunction.java

+        return parseTimestampData(timestamp.toString());
+    }
+
+    public @Nullable TimestampData eval(StringData timestamp, StringData format) {


Suggested change

public @Nullable TimestampData eval(StringData timestamp, StringData format) {

public @Nullable TimestampData eval(@Nullable StringData timestamp, @Nullable StringData format) {

snuyanzin · 2026-03-20T10:36:31Z

docs/data/sql_functions_zh.yml

+
+      The output precision depends on the variant used:
+      - 1-arg variant: always returns TIMESTAMP(3).
+      - 2-arg variant: precision is inferred from the number of trailing 'S' characters in the format pattern, with a minimum of 3. E.g., format 'yyyy-MM-dd HH:mm:ss.SS' returns TIMESTAMP(3), format 'yyyy-MM-dd HH:mm:ss.SSSSSS' returns TIMESTAMP(6).


E.g., format 'yyyy-MM-dd HH:mm:ss.SS' returns TIMESTAMP(3)

why do we have such behavior?

what is the behavior for other vendors?

This is kept for backward compatibility. Same behavior we have for TO_TIMESTAMP_LTZ https://issues.apache.org/jira/browse/FLINK-39244

isn't release notes telling it a bit different?

TO_TIMESTAMP_LTZ() function now supports up to precision 9 for both numeric and string conversions. While we kept backwards compatibility for numeric precision 0-3 which always returns TIMESTAMP_LTZ(3), precisions p=4-9 now return TIMESTAMP_LTZ(p). Function calls taking string formats such as ".SSSS" now return precision 4 instead of silently loosing sub second information.

[FLINK-14925][table] Support precision-aware TO_TIMESTAMP with format…

edd47b2

…-based inference

snuyanzin marked this pull request as ready for review March 20, 2026 08:55

raminqaf marked this pull request as draft March 20, 2026 08:58

raminqaf marked this pull request as ready for review March 20, 2026 08:58

snuyanzin reviewed Mar 20, 2026

View reviewed changes

davidradl reviewed Mar 20, 2026

View reviewed changes

snuyanzin reviewed Mar 20, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-14925][table] Support precision-aware TO_TIMESTAMP with format-based inference#27793

[FLINK-14925][table] Support precision-aware TO_TIMESTAMP with format-based inference#27793
raminqaf wants to merge 1 commit intoapache:masterfrom
raminqaf:FLINK-14925

raminqaf commented Mar 20, 2026

Uh oh!

flinkbot commented Mar 20, 2026 •

edited

Loading

Uh oh!

snuyanzin Mar 20, 2026

Uh oh!

davidradl Mar 20, 2026

Uh oh!

snuyanzin Mar 20, 2026

Uh oh!

snuyanzin Mar 20, 2026

Uh oh!

snuyanzin Mar 20, 2026

Uh oh!

snuyanzin Mar 20, 2026

Uh oh!

raminqaf Mar 20, 2026

Uh oh!

snuyanzin Mar 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	public @Nullable TimestampData eval(StringData timestamp) {
	public @Nullable TimestampData eval(@Nullable StringData timestamp) {

Conversation

raminqaf commented Mar 20, 2026

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

Uh oh!

flinkbot commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CI report:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

flinkbot commented Mar 20, 2026 •

edited

Loading