Integrate service discovery into core?

[dominikriemer]

Hi,
currently, StreamPipes has a dependency on Consul for two reasons:

• Extensions services register at Consul and the core makes use of Consul's service discovery to find endpoints that support pipeline elements and adapters
• The core uses Consul's key/value store to store configs such as email, authentication settings, broker settings and more. In addition, extensions can define configurations which are also stored in Consul.

I think it should be pretty easy to integrate the service discovery logic directly into the core and store the configurations in CouchDB, which would make our Consul dependency obsolete, reducing system complexity.

Approach

The approach would be:

• Extensions services register at the core upon startup, using a new endpoint which stores the service configuration in CouchDB. The core's hostname would be backend by default in the docker-compose installation, link to the backend service in k8s and could be overridden with an environment variable similar to SP_CONSUL_LOCATION for development purposes.
• The core then uses its own service discovery mechanism based on stored service configurations when pipelines and adapters are started.
• All configurations of the core which are currently stored in Consul would move to a new CouchDB database core-configuration.
• Extensions services could store their configs in another CouchDB database which can be accessed over the client from the core.

Migration

• We don't need a migration for the service discovery change, assuming that extensions services and the core are running the same StreamPipes version
• For the key/value store configurations, we need a migration. We could add a migration in release 0.93.0 which transfers all Consul configs to the CouchDB database, and remove Consul in the release thereafter.

What do ou think of this idea?

[tenthe]

Hi @dominikriemer,
I think this idea is great and will reduce the size of StreamPipes installation.
+1 from my side.

I didn't quite understand why the migration is split into two versions. Wouldn't that mean we have to maintain the data in two places in release 0.93.0?

Cheers,
Philipp

[dominikriemer]

Thanks for asking @tenthe!
In v0.93.0, we would add an migration to copy all existing configs from the key value store to CouchDB. All variables will then already be read from and written to CouchDB. In the next release, we can then remove Consul.
We need Consul in another release to make sure that customized configs are properly copied to CouchDB.

[tenthe]

Ah ok, thanks for the explanation.
That makes perfect sense. So if a user wants to get rid of Consul manually, it could be removed after the migration script was executed, right?

[dominikriemer]

@tenthe right now I don't think this can be done since we probably can remove the BackendConfig.INSTANCE.isConfigured() call which is needed in StreamPipesCoreApplication only after the migration has taken place to ensure that the system can properly identify an unconfigured system and start the initial installation procedure.

[bossenti]

Thanks for bringing this up for discussion and for the well-written suggestion @dominikriemer
In general, I like the idea of moving towards a slimmer setup of StreamPipes.
However, I think when making such a decision, we should always weigh the benefit of getting rid of an external dependency against the complexity we add to our codebase by implementing the functionality ourselves.
Can you estimate how much and how complex custom code we would add to StreamPipes by replacing Consul?

We don't need a migration for the service discovery change, assuming that extensions services and the core are running the same StreamPipes version

Can we really hold on to this assumption? Isn't it to restrictive?

Other than that, your migration plan sounds got to me 👍🏼

[dominikriemer]

@bossenti it's not overly complex since we can reuse most existing APIs, so that the change actually deletes more code than it adds (I guess). So I think it reduces complexity by a lot.

Concerning the version, I think we've never supported or tested compatibility between core and extensions services that do not have the same version, so I don't think it's an overly restrictive assumption.

[bossenti]

it's not overly complex since we can reuse most existing APIs, so that the change actually deletes more code than it adds (I guess). So I think it reduces complexity by a lot.

I will mark your words here 😉

Joke aside, then I'm also +1 for this idea

Concerning the version, I think we've never supported or tested compatibility between core and extensions services that do not have the same version, so I don't think it's an overly restrictive assumption.

Indeed