[FLINK-37175][table] Support JSON built-in function for JSON_OBJECT #26022

gustavodemorais · 2025-01-20T11:54:13Z

What is the purpose of the change

It is currently not possible to declare a SQL string that contains existing JSON as valid JSON for JSON_OBJECT. Something like JSON_OBJECT(KEY 'K' VALUE '{"value": 42}') returns {"K", "{"value": 42}"}, where the value is a string a not a json object.

This PR adds support for the JSON() function. It's the initial support for it, until this function returns the JSON datatype (what we still don't have in flink).

Example:

jsonObject(JsonOnNull.NULL, "nested", json('{"value": 42}'))
JSON_OBJECT(KEY 'K' VALUE JSON('{"value": 42}'))
// {"nested":{"value":42}}

Brief change log

Add JSON BuiltInFunction definition
Update JsonObjectCallGen to support JSON function logic
Update expressions
Add tests for json function

Verifying this change

This change added tests for multiple uses cases of the function. Also added tests to make sure the fucnction is only called within JSON_OBJECT.

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): yes
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? docs, JavaDocs

flinkbot · 2025-01-20T13:54:49Z

CI report:

42db42a Azure: FAILURE

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

docs/data/sql_functions.yml

twalthr · 2025-01-20T12:02:04Z

flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Expressions.java

@@ -861,9 +862,13 @@ public static ApiExpression withoutColumns(Object head, Object... tail) {
     * jsonObject(JsonOnNull.ABSENT, "K1", nullOf(DataTypes.STRING())) // "{}"
     *
     * // {"K1":{"K2":"V"}}
+     * jsonObject(JsonOnNull.NULL, "K1", json('{"K2":"V"}'))


Suggested change

* jsonObject(JsonOnNull.NULL, "K1", json('{"K2":"V"}'))

* jsonObject(JsonOnNull.NULL, "K1", json("{'K2':'V'}"))

or escaped? not sure how strict the user-defined JSON needs to be? maybe we should check and document this.

The string doesn't have to be escaped, I've updated the javadoc for the java expression and added one example using string literals that show that there's no need to escape it

* // {"K":{"K2":{"K3":42}}} * jsonObject( * JsonOnNull.NULL, * "K", * json(""" * { * "K2": { * "K3": 42 * } * } * """))

We escape in the java code since java also uses " to express strings. If using single quotes in python or string literals in java, there's no need to escape. If they're escaped, they are also processed properly as well and result in the same json object.

thanks for updating the JavaDocs of json(). Seems this line still needs an update.

flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Expressions.java

...e/flink-table-common/src/main/java/org/apache/flink/table/types/logical/LogicalTypeRoot.java

...-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/JsonGenerateUtils.scala

...flink-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/CodeGenUtils.scala

gustavodemorais · 2025-01-22T11:24:56Z

Made the changes as we discussed @twalthr. One additional thing I've changed is that we not only parse the json, but convert the json back to string before storing it. I think that makes sense, so we optimize the storage space by getting rid of unnecessary whitespaces/line breaks and so on before returning the value. It's a similar behavior to other frameworks and what JSON type probably will eventually do.

twalthr

Looks very good! We are almost there :-)

twalthr · 2025-01-24T12:05:42Z

flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/Expressions.java

@@ -861,9 +862,13 @@ public static ApiExpression withoutColumns(Object head, Object... tail) {
     * jsonObject(JsonOnNull.ABSENT, "K1", nullOf(DataTypes.STRING())) // "{}"
     *
     * // {"K1":{"K2":"V"}}
+     * jsonObject(JsonOnNull.NULL, "K1", json('{"K2":"V"}'))


thanks for updating the JavaDocs of json(). Seems this line still needs an update.

twalthr · 2025-01-24T12:06:06Z

...-table-common/src/main/java/org/apache/flink/table/functions/BuiltInFunctionDefinitions.java

+                    .kind(SCALAR)
+                    .inputTypeStrategy(sequence(logical(LogicalTypeFamily.CHARACTER_STRING)))
+                    .outputTypeStrategy(nullableIfArgs(explicit(DataTypes.STRING())))
+                    .runtimeClass("org.apache.flink.table.runtime.functions.scalar.JsonFunction")


replace with runtimeDeferred()?

My first take at it gave me the impression that we would have even more boilerplate code if we go with runtimeDeferred - I can't just swap them. Would it be a bad practice using the runtime class here?

twalthr · 2025-01-24T12:08:08Z

...-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/ExprCodeGenerator.scala

+    // throw exception if json function is called outside JSON_OBJECT
+    if (isJsonFunctionOperand(call)) {
+      throw new ValidationException(
+        "The JSON function is currently only supported inside a JSON_OBJECT." +


Suggested change

"The JSON function is currently only supported inside a JSON_OBJECT." +

"The JSON() function is currently only supported inside a JSON_OBJECT() function." +

twalthr · 2025-01-24T12:12:48Z

...-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/ExprCodeGenerator.scala

@@ -476,6 +485,11 @@ class ExprCodeGenerator(ctx: CodeGeneratorContext, nullableInput: Boolean)
            call.getOperator.getReturnTypeInference == ReturnTypes.ARG0 =>
        generateNullLiteral(resultType)

+      // We only support JSON function operands within JSON_OBJECT
+      case (operand: RexNode, _)


Suggested change

case (operand: RexNode, _)

case (operand: RexNode, i)

also check that the index is even. otherwise JSON() function could be used for the key?

twalthr · 2025-01-24T12:14:58Z

...-table-planner/src/main/scala/org/apache/flink/table/planner/codegen/ExprCodeGenerator.scala

@@ -476,6 +485,11 @@ class ExprCodeGenerator(ctx: CodeGeneratorContext, nullableInput: Boolean)
            call.getOperator.getReturnTypeInference == ReturnTypes.ARG0 =>
        generateNullLiteral(resultType)

+      // We only support JSON function operands within JSON_OBJECT
+      case (operand: RexNode, _)
+          if isJsonObjectOrArrayOperand(call) && isJsonFunctionOperand(operand) =>


Since we use OrArrayOperand? Do we support JSON_ARRAY?

twalthr · 2025-01-24T12:19:00Z

...able-runtime/src/main/java/org/apache/flink/table/runtime/functions/scalar/JsonFunction.java

+
+/** Implementation of {@link BuiltInFunctionDefinitions#JSON}. */
+@Internal
+public class JsonFunction extends BuiltInScalarFunction {


davidradl

I look at Oracle and see that they use json_object for the nested json_object.
also I see there is an example of a nested object using json_object in the flink docs:

JSON_OBJECT(
KEY 'K1'
VALUE JSON_OBJECT(
KEY 'K2'
VALUE 'V'
)
)
why do we need a new function when we can already do this with json_object? Or am I missing something?

davidradl · 2025-01-24T16:46:20Z

Reviewed by Chi on 23/01/2025 Go back to the submitter with review comments.

flinkbot added the component=TableSQL/API label Jan 20, 2025

gustavodemorais force-pushed the FLINK-37175 branch from 6cc5305 to dc6eed5 Compare January 20, 2025 13:38

davidradl reviewed Jan 20, 2025

View reviewed changes

docs/data/sql_functions.yml Show resolved Hide resolved

twalthr reviewed Jan 20, 2025

View reviewed changes

gustavodemorais force-pushed the FLINK-37175 branch from b6516b6 to 6cc950a Compare January 21, 2025 10:30

gustavodemorais closed this Jan 22, 2025

gustavodemorais force-pushed the FLINK-37175 branch from 6cc950a to dccb782 Compare January 22, 2025 09:26

gustavodemorais reopened this Jan 22, 2025

[FLINK-37175][table] Support JSON built-in function

42db42a

gustavodemorais force-pushed the FLINK-37175 branch from 1dea414 to 42db42a Compare January 22, 2025 11:14

gustavodemorais requested a review from twalthr January 22, 2025 11:25

twalthr reviewed Jan 24, 2025

View reviewed changes

davidradl suggested changes Jan 24, 2025

View reviewed changes

gustavodemorais mentioned this pull request Jan 24, 2025

[FLINK-37177][Doc] Update chinese documentation of json function #26055

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FLINK-37175][table] Support JSON built-in function for JSON_OBJECT #26022

[FLINK-37175][table] Support JSON built-in function for JSON_OBJECT #26022

gustavodemorais commented Jan 20, 2025

flinkbot commented Jan 20, 2025 •

edited

Loading

twalthr Jan 20, 2025

twalthr Jan 20, 2025

gustavodemorais Jan 22, 2025

twalthr Jan 24, 2025

gustavodemorais Jan 24, 2025

gustavodemorais commented Jan 22, 2025 •

edited

Loading

twalthr left a comment

twalthr Jan 24, 2025

twalthr Jan 24, 2025

gustavodemorais Jan 24, 2025

twalthr Jan 24, 2025

twalthr Jan 24, 2025

twalthr Jan 24, 2025

twalthr Jan 24, 2025

davidradl left a comment

davidradl commented Jan 24, 2025

	* jsonObject(JsonOnNull.NULL, "K1", json('{"K2":"V"}'))
	* jsonObject(JsonOnNull.NULL, "K1", json("{'K2':'V'}"))

	"The JSON function is currently only supported inside a JSON_OBJECT." +
	"The JSON() function is currently only supported inside a JSON_OBJECT() function." +

[FLINK-37175][table] Support JSON built-in function for JSON_OBJECT #26022

Are you sure you want to change the base?

[FLINK-37175][table] Support JSON built-in function for JSON_OBJECT #26022

Conversation

gustavodemorais commented Jan 20, 2025

What is the purpose of the change

Brief change log

Verifying this change

Does this pull request potentially affect one of the following parts:

Documentation

flinkbot commented Jan 20, 2025 • edited Loading

CI report:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gustavodemorais commented Jan 22, 2025 • edited Loading

twalthr left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidradl left a comment

Choose a reason for hiding this comment

davidradl commented Jan 24, 2025

flinkbot commented Jan 20, 2025 •

edited

Loading

gustavodemorais commented Jan 22, 2025 •

edited

Loading