[stable] multi-cluster support

[stevenctl]

beta issue: #13

This document covers the stability of multi-primary and primary-remote configurations.

This does not include:

• external control plane
• multi-network

Stable Feature Requirements

This page lists the requirements for promoting multicluster to stable. Please check off and document the steps as they are completed.

Requirements

Features/Fixes:

• Allow opt-out of multicluster SDS istio#32364
• headless services (Multi-Cluster: Can only reach local Headless service istio#27342, Istio MultiPrimary Deployment cannot resolve headless service in remote cluster  istio#29969)
• Exposing headless service with statefulset (in multi primary setup) is not resolvable across clusters. istio#31787
• Fix and test multicluster with .global stub domain istio#29335
• MultiCluster communications fail randomly after 30 minutes istio#31464
• Refactor Multicluster Analyzer istio#28906
• istioctl x create-remote-secret createsinvalid certificate chain istio#30625
• [Multicluster]the istioctl debug tool in Primary-Remote model behavior improperly istio#29900
• Remote pilot address is bind to port 15012 (for xDs). It would be great if we can override this port, making it able to support NodePort for remode istiod. istio#29482
• Make istiod-less Remote Cluster the default for multicluster istio#27420
  • Re-enable tests (temporarily in a separate prow job(s) using test framework settting
  • [] A workaround for our test infra that avoids patching ISTIOD_CUSTOM_HOST (to closer match what we document for users).
  • istiodless remote cluster tests failing istio#33619
  • Document installation steps. (multi-cluster: remote clusters should not install local istiod istio.io#9618)
• bug-report support bug-report: support multicluster istio#32321

Tests:

• multicluster networking ingress coverage (cover multi-cluster in networking ingress test  istio#32282)
• Draining traffic to a cluster and removing it from a mesh
• security test flakes (Test Flake: Multicluster security tests istio#32172)
• security test gaps (remove SkippedForMulticluster in security reachability tests istio#32287)
• test failures: corsPolity breaks using VM + Multi-Cluster (debug corsPolicy not working when calling VM that's registered in another cluster istio#31532)
• gRPC in security tests (fix cross-cluster gRPC in security tests istio#32280)
• enhance mirroring test checks (make traffic mirroring test check cross cluster load balancing istio#32284)

Docs:

Each task here also includes providing an automated test for the document on istio.io.

• headless/statefulsets - Document multi-cluster support for headless services and stateful sets istio.io#8832
• Add best practices guide for multicluster istio.io#8605
• Rework upgrade docs to cover multicluster istio.io#8603
• Create Task: Draining a Cluster istio.io#8469
• multi-cluster: document expansion from a single-cluster mesh to be multi-cluster istio.io#9559
• How to configure a service to be cluster-local  istio.io#9558
• instructions for revisioned upgrade of east-west gateway (Provide instructions on revisioned upgrade of east-west gateway istio#30526)

Other:

• Revisit the supportability review for beta

Approvals:

• The appropriate work group(s) have reviewed and approved promotion of the feature.
• The supportability review panel has reviewed promotion of the feature.
• The TOC has reviewed and approved promotion of the feature as part of the
  road map for a release.

[hzxuzhonghu]

We also need cluster-aware load balancing when clusters are in different networks

[stevenctl]

Multi-network is a separate concern from multi-cluster

[brian-avery]

@stevenctl can you please create a PR for this following https://github.com/istio/enhancements/blob/master/FEATURES.md? Thanks

[stevenctl]

@brian-avery I started moving it over... but I'll do that as the last step. That format is pretty hard to use as a living checklist while the work is in progress.