[FLINK-37187][doc] Add doc for Submitting Refresh Job of Materialized…

… Table to Yarn/K8s
apache · Jan 24, 2025 · 4158587 · 4158587
1 parent 5749a07
commit 4158587
Show file tree

Hide file tree

Showing 4 changed files with 118 additions and 8 deletions.
diff --git a/docs/content.zh/docs/dev/table/materialized-table/overview.md b/docs/content.zh/docs/dev/table/materialized-table/overview.md
@@ -28,10 +28,6 @@ under the License.
 
 物化表是 Flink SQL 引入的一种新的表类型，旨在简化批处理和流处理数据管道，提供一致的开发体验。在创建物化表时，通过指定数据新鲜度和查询，Flink 引擎会自动推导出物化表的 Schema ，并创建相应的数据刷新管道，以达到指定的新鲜度。
 
-{{< hint warning >}}
-**注意**：该功能目前是一个 MVP（最小可行产品）功能，仅在 [SQL Gateway]({{< ref "docs/dev/table/sql-gateway/overview" >}})中可用，并且只支持部署作业到 Flink [Standalone]({{< ref "docs/deployment/resource-providers/standalone/overview" >}})集群。
-{{< /hint >}}
-
 # 核心概念
 
 物化表包含以下核心概念：数据新鲜度、刷新模式、查询定义和 `Schema` 。

diff --git a/docs/content.zh/docs/dev/table/sqlClient.md b/docs/content.zh/docs/dev/table/sqlClient.md
@@ -681,6 +681,65 @@ After execution, SQL Client will print the cluster id on the terminal. The scrip
 <span class="label label-danger">Attention</span> When deploying a script to the cluster, SQL Client only supports running with `--jars` startup option, other options, e.g. `--init`
 are not supported.
 
+### Execute Materialized Table Refresh Jobs in Kubernetes/YARN
+
+#### Execute Refresh Jobs in Session Mode
+
+For Kubernetes session mode:
+```sql
+Flink SQL> SET 'execution.mode' = 'kubernetes-session';
+[INFO] Execute statement succeeded.
+
+Flink SQL> SET 'kubernetes.cluster-id' = 'flink-cluster-mt-session-1';
+[INFO] Execute statement succeeded.
+
+FLINK SQL> CREATE MATERIALIZED TABLE continuous_users_shops
+> ...
+[INFO] Execute statement succeeded.
+```
+Set `execution.mode` to `kubernetes-session` and specify a valid `kubernetes.cluster-id` corresponding to an existing Kubernetes session cluster.
+
+For YARN session mode:
+```sql
+Flink SQL> SET 'execution.mode' = 'yarn-session';
+[INFO] Execute statement succeeded.
+
+Flink SQL> SET 'yarn.application.id' = 'application-xxxx';
+[INFO] Execute statement succeeded.
+
+FLINK SQL> CREATE MATERIALIZED TABLE continuous_users_shops
+> ...
+[INFO] Execute statement succeeded.
+```
+Set `execution.mode` to `yarn-session` and specify a valid `yarn.application.id` corresponding to an existing YARN session cluster.
+
+#### Execute Refresh Jobs in Application Mode
+
+For Kubernetes application mode:
+```sql
+Flink SQL> SET 'execution.mode' = 'kubernetes-application';
+[INFO] Execute statement succeeded.
+
+Flink SQL> SET 'kubernetes.cluster-id' = 'flink-cluster-mt-application-1';
+[INFO] Execute statement succeeded.
+
+FLINK SQL> CREATE MATERIALIZED TABLE continuous_users_shops
+> ...
+[INFO] Execute statement succeeded.
+```
+Set `execution.mode` to `kubernetes-application`. The `kubernetes.cluster-id` is optional; if not set, it will be automatically generated.
+
+For YARN application mode:
+```sql
+Flink SQL> SET 'execution.mode' = 'yarn-application';
+[INFO] Execute statement succeeded.
+
+FLINK SQL> CREATE MATERIALIZED TABLE continuous_users_shops
+> ...
+[INFO] Execute statement succeeded.
+```
+Only set `execution.mode` to `yarn-application`. The `yarn.application.id` doesn't need to be set; it will be automatically generated during submission.
+
 ### Execute a set of SQL statements
 
 SQL Client execute each INSERT INTO statement as a single Flink job. However, this is sometimes not

diff --git a/docs/content/docs/dev/table/materialized-table/overview.md b/docs/content/docs/dev/table/materialized-table/overview.md
@@ -28,10 +28,6 @@ under the License.
 
 Materialized Table is a new table type introduced in Flink SQL, aimed at simplifying both batch and stream data pipelines, providing a consistent development experience. By specifying data freshness and query when creating Materialized Table, the engine automatically derives the schema for the materialized table and creates corresponding data refresh pipeline to achieve the specified freshness.
 
-{{< hint warning >}}
-**Note**: This feature is currently an MVP (“minimum viable product”) feature and only available within [SQL Gateway]({{< ref "docs/dev/table/sql-gateway/overview" >}}) which connected to a [Standalone]({{< ref "docs/deployment/resource-providers/standalone/overview" >}}) deployed Flink cluster.
-{{< /hint >}}
-
 # Core Concepts
 
 Materialized Table encompass the following core concepts: Data Freshness, Refresh Mode, Query Definition and Schema.

diff --git a/docs/content/docs/dev/table/sqlClient.md b/docs/content/docs/dev/table/sqlClient.md
@@ -619,6 +619,65 @@ After execution, SQL Client will print the cluster id on the terminal. The scrip
 <span class="label label-danger">Attention</span> When deploying a script to the cluster, SQL Client only supports running with `--jars` startup option, other options, e.g. `--init` 
 are not supported.
 
+### Execute Materialized Table Refresh Jobs in Kubernetes/YARN
+
+#### Execute Refresh Jobs in Session Mode
+
+For Kubernetes session mode:
+```sql
+Flink SQL> SET 'execution.mode' = 'kubernetes-session';
+[INFO] Execute statement succeeded.
+
+Flink SQL> SET 'kubernetes.cluster-id' = 'flink-cluster-mt-session-1';
+[INFO] Execute statement succeeded.
+
+FLINK SQL> CREATE MATERIALIZED TABLE continuous_users_shops
+> ...
+[INFO] Execute statement succeeded.
+```
+Set `execution.mode` to `kubernetes-session` and specify a valid `kubernetes.cluster-id` corresponding to an existing Kubernetes session cluster.
+
+For YARN session mode:
+```sql
+Flink SQL> SET 'execution.mode' = 'yarn-session';
+[INFO] Execute statement succeeded.
+
+Flink SQL> SET 'yarn.application.id' = 'application-xxxx';
+[INFO] Execute statement succeeded.
+
+FLINK SQL> CREATE MATERIALIZED TABLE continuous_users_shops
+> ...
+[INFO] Execute statement succeeded.
+```
+Set `execution.mode` to `yarn-session` and specify a valid `yarn.application.id` corresponding to an existing YARN session cluster.
+
+#### Execute Refresh Jobs in Application Mode
+
+For Kubernetes application mode:
+```sql
+Flink SQL> SET 'execution.mode' = 'kubernetes-application';
+[INFO] Execute statement succeeded.
+
+Flink SQL> SET 'kubernetes.cluster-id' = 'flink-cluster-mt-application-1';
+[INFO] Execute statement succeeded.
+
+FLINK SQL> CREATE MATERIALIZED TABLE continuous_users_shops
+> ...
+[INFO] Execute statement succeeded.
+```
+Set `execution.mode` to `kubernetes-application`. The `kubernetes.cluster-id` is optional; if not set, it will be automatically generated.
+
+For YARN application mode:
+```sql
+Flink SQL> SET 'execution.mode' = 'yarn-application';
+[INFO] Execute statement succeeded.
+
+FLINK SQL> CREATE MATERIALIZED TABLE continuous_users_shops
+> ...
+[INFO] Execute statement succeeded.
+```
+Only set `execution.mode` to `yarn-application`. The `yarn.application.id` doesn't need to be set; it will be automatically generated during submission.
+
 ### Execute a set of SQL statements
 
 SQL Client execute each INSERT INTO statement as a single Flink job. However, this is sometimes not