Support Nouveau in CHT Deploy script and upgrades and docs

[mrjones-plip]

Describe the issue
Current, the CHT k8s Deploy script and k8s upgrade service is not aware of the upcoming CHT Core Nouveau feature.

Describe the improvement you'd like
We should update the deploy script such that a happy path new deployment works when we ship CHT 4.x with Nouveau. Additionally, upgrade paths from prior versions of k8s based deployments should account for this change to ensure services don't break. This may be just documentation improvements or it maybe code additions or helm chart additions.

[mrjones-plip]

@jkuester or @sugat009 - do we know how a deployment looks when it has both a clustered couch deployment AND Nouveau? That is - if there's 3 couch containers running, is there 1 Nouveau container or 3 Nouveau containers?

cc @dianabarsan

[jkuester]

😓 Mokhtar might know the answer to that question, but unfortunately I do not have any hard answers yet. Definitely something we need to confirm.

[mrjones-plip]

Gotcha! I'll do a naive POC to see what happens on the latest CI branch but with multi-node couch compose file to see if I can find out anything.

[mrjones-plip]

Ah - I don't think CI has ever passed, so the only way to POC is in dev mode. I'll see what I can do! I'll also see if CouchDB docs have any info.

[mrjones-plip]

@jkuester or @sugat009 - are there steps on how to run Nouveau against a developer setup?

Otherwise, I can't find an answer through a POC (b/c I can't bootstrap the system in the first place) and don't see Nouveau setup instructions involving a cluster anywhere - I'm hopping in there Slack server to see what I can find!

Hey all! I'm continuing Mokhtar's work at Medic with Nouveau . We currently offer a performance boost by deploying our app against a clustered couch install with 3 couch nodes behind HAProxy (these are a naive "raid 0" type cluster with q=12 and n=1 - so just for speed, not for redundancy)

My question is - with our clustered setup - how many instances of Nouveau would we need: 1:1 (so 3 in our case) or can multiple couch nodes speak to a single Nouveau since the data is all the same?

[mrjones-plip]

Good news! Jan Lehnardt (CouchDB dev and PMC chair) reports:

Nouveau can run in 1:N configurations, but I don’t know how far the fan out is useful in practice. But Bob will comment when he sees this.

So we can very likely proceed with just the one instance. I've asked a follow up per the "Fan out" comment if we'd deploy a cluster of Nouveau nodes or something 🤔

---

Ok - confirmed! we can safely use 1:N Nouveau instances \o/ The only possible limitation is performance - which will be directly based on how Lucene performs. I think it's safe to keep it as is (default to "just another container") and we can scale it vertically or horizontally if needed in the future.

Robert Newson @ Apache replied in our Slack thread:

It's a good question. in your setup you could have 1, 2 or 3 nouveau servers but not more (though it's an interesting idea...)

whether 1 or 2 servers would suffice you'll have to measure for yourself, I can only confirm that it's possible (where it is not possible for clouseau)

the config is simple. each couchdb node has a config item which is the url to the nouveau server. two couchdb nodes can point safely to the same nouveau server, as long as they have distinct erlang node names (which they have to in a cluster anyway)

(all the requests to nouveau include the requesting erlang nodes name in the request path)

[mrjones-plip]

cc @dianabarsan 👆

[dianabarsan]

Thanks for the update @mrjones-plip ! Great news!