Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue 479: Support Docusaurus format #594

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 8 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions documentation/src/docs/batch.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Batch
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,8 +18,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Batch Connector

The Flink Connector library for Pravega makes it possible to use a Pravega Stream as a data source and data sink in a batch program. See the below sections for details.

## Table of Contents
Expand Down
6 changes: 4 additions & 2 deletions documentation/src/docs/configurations.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Configurations
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,8 +18,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Configurations

The Flink connector library for Pravega supports the **Flink Streaming API**, **Table API** and **Batch API**, using a common configuration class.

## Table of Contents
Expand Down
4 changes: 4 additions & 0 deletions documentation/src/docs/dev-guide.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Developer's Guide
---

<!--
Copyright Pravega Authors.

Expand Down
8 changes: 7 additions & 1 deletion documentation/src/docs/getting-started.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Getting Started
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,12 +18,14 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Pravega Flink Connectors
import TOCInline from '@theme/TOCInline';

This repository implements connectors to read and write [Pravega](http://pravega.io/) Streams with [Apache Flink](http://flink.apache.org/) stream processing framework.

The connectors can be used to build end-to-end stream processing pipelines (see [Samples](https://github.com/pravega/pravega-samples)) that use Pravega as the stream storage and message bus, and Apache Flink for computation over the streams.

<TOCInline toc={toc} />

## Features & Highlights

- **Exactly-once processing guarantees** for both Reader and Writer, supporting **end-to-end exactly-once processing pipelines**
Expand Down
6 changes: 4 additions & 2 deletions documentation/src/docs/metrics.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Metrics
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,8 +18,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Metrics

Pravega metrics are collected and exposed via Flink metrics framework when using [`FlinkPravegaReader`](streaming.md#flinkpravegareader) or [`FlinkPravegaWriter`](streaming.md#flinkpravegawriter).

## Reader Metrics
Expand Down
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Overview
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,8 +18,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Apache Flink Connectors for Pravega

This documentation describes the connectors API and it's usage to read and write [Pravega](http://pravega.io/) streams with [Apache Flink](http://flink.apache.org/) stream processing framework.

Build end-to-end stream processing pipelines that use Pravega as the stream storage and message bus, and Apache Flink for computation over the streams. See the [Pravega Concepts](http://pravega.io/docs/pravega-concepts/) page for more information.
Expand Down
6 changes: 4 additions & 2 deletions documentation/src/docs/quickstart.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Quick Start
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,8 +18,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Getting Started

## Creating a Flink Stream Processing Project

**Note**: _You can skip this step if you have a streaming project set up already._
Expand Down
6 changes: 4 additions & 2 deletions documentation/src/docs/serialization.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Serialization
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,8 +18,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Serialization

**Serialization** refers to converting a data element in your Flink program to/from a message in a Pravega stream.

Flink defines a standard interface for data serialization to/from byte messages delivered by various connectors. The core interfaces are:
Expand Down
18 changes: 18 additions & 0 deletions documentation/src/docs/sidebars.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
module.exports = {
mainSidebar: [
'overview',
'getting-started',
'quickstart',
'dev-guide',
{
'Features': [
'streaming',
'batch',
'table-api',
],
},
'metrics',
'configurations',
'serialization',
]
};
6 changes: 4 additions & 2 deletions documentation/src/docs/streaming.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Streaming
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,8 +18,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Streaming Connector

The Flink Connector library for Pravega provides a data source and data sink
for use with the Flink Streaming API. See the below sections for details.

Expand Down
14 changes: 8 additions & 6 deletions documentation/src/docs/table-api.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
---
title: Table API
---

<!--
Copyright Pravega Authors.

Expand All @@ -14,8 +18,6 @@ See the License for the specific language governing permissions and
limitations under the License.
-->

# Table Connector

The Flink connector library for Pravega provides a table source and table sink for use with the Flink Table API.
The Table API provides a unified table source API for both the Flink streaming and batch environment, and also sink for the Flink streaming environment.

Expand Down Expand Up @@ -89,9 +91,9 @@ WITH (
| security.trust-store | optional | (none) | String | Trust Store for Pravega client |
| scan.execution.type | optional | streaming | String | Execution type for scan source. Valid values are 'streaming', 'batch'. |
| scan.reader-group.name | optional | (none) | String | Pravega reader group name |
| scan.streams | required for source | (none) | List<String> | Semicolon-separated list of stream names from which the table is read. |
| scan.start-streamcuts | optional | (none) | List<String> | Semicolon-separated list of base64 encoded strings for start streamcuts, begin of the stream if not specified |
| scan.end-streamcuts | optional | (none) | List<String> | Semicolon-separated list of base64 encoded strings for end streamcuts, unbounded end if not specified |
| scan.streams | required for source | (none) | List&lt;String&gt; | Semicolon-separated list of stream names from which the table is read. |
| scan.start-streamcuts | optional | (none) | List&lt;String&gt; | Semicolon-separated list of base64 encoded strings for start streamcuts, begin of the stream if not specified |
| scan.end-streamcuts | optional | (none) | List&lt;String&gt; | Semicolon-separated list of base64 encoded strings for end streamcuts, unbounded end if not specified |
| scan.reader-group.max-outstanding-checkpoint-request | optional | 3 | Integer | Maximum outstanding checkpoint requests to Pravega |
| scan.reader-group.refresh.interval | optional | 3 s | Duration | Refresh interval for reader group |
| scan.event-read.timeout.interval | optional | 1 s | Duration | Timeout for the call to read events from Pravega |
Expand All @@ -109,7 +111,7 @@ WITH (
`scan.execution.type` can be specified as user's choice to perform batch read or streaming read.
In the streaming environment, the table source uses a [`FlinkPravegaReader`](streaming.md#flinkpravegareader) connector.
In the batch environment, the table source uses a [`FlinkPravegaInputFormat`](batch.md#flinkpravegainputformat) connector.
Please see the documentation of [Streaming Connector](streaming.md) and [Batch Connector](#batch.md) to have a better understanding on the below mentioned parameter list.
Please see the documentation of [Streaming Connector](streaming.md) and [Batch Connector](batch.md) to have a better understanding on the below mentioned parameter list.

### Specify start and end streamcut

Expand Down