Ways to implement Host-Plugin interaction

[namit-chandwani]

There are many ways by which a host can interact with plugins and these can be divided mainly into 2 categories:

1. Multi-process (or external process) architecture: Plugins run as separate processes and the host process communicates with the plugins using various communication techniques (for eg. RPC).

2. Single process (or in-process) architecture: Plugins and the host run on the same process.

 
These approaches are described in detail below:

1. Multi-process architecture

1.1) Using RPC via stdin/stdout

• Description:

  • Here the host process starts a separate plugin process
  • The host process uses RPC (Remote Procedure Call) via stdin/stdout connection to communicate with plugin processes

• Example: natefinch/pie

---

1.2) Using RPC via network

• Description:

  • The way RPC calls are implemented here differs slightly from the prior approach
  • RPC calls in this approach are made over the (local) network rather than the stdin/stdout connection

• Example: hashicorp/go-plugin

 
 
Pros and cons of both the above approaches that involve RPC-based communication:

• Pros:

  • Plugins are technology-independent, ie. they don't have to be written in the same language as the host CLI (but they must share a similar RPC protocol).
  • Plugins can be versioned to help support backward-compatible changes
  • The plugins and host CLI are running in separate processes so if any plugin crashes, the impact on the host CLI is limited.
  • Plugins can be hot-reloaded

• Cons:

  • Lower performance than dynamic library function calls (native Golang Plugins)
  • Additional runtime processes, binaries to be deployed, and dependencies between them, all add to the complexity
  • Relatively higher latency due to out of the process communication

---

1.3) Via message queue

• Description:

  Message queue systems, particularly brokerless ones, provide a great foundation for developing plugin systems. They can be conveniently converted to a plugin architecture.

---

 

2. Single process architecture

2.1) Using the Golang Plugin package

• Description:

  • Golang Plugins is a way of extending Golang applications natively

  • This approach uses the ability of Go packages that enable them to be compiled as shared object (.so) libraries and to be loaded dynamically at run time

  • These work by compiling plugins in the following way:

    go build -buildmode=plugin

  • This build mode can output a .so file with symbols that the host CLI can load dynamically.

• Pros:

  • Part of the Golang stdlib
  • Simple and sparse
  • High performance as it just calls functions in a library using native interfaces

• Cons:

  • The environment for building a Go plugin such as OS, Go language version, and dependency versions must match exactly with the host CLI
  • Unloading plugins are not allowed without restarting the program
  • We cannot replace a plugin with a newer version during runtime (ie. cannot be hot-reloaded)
  • As of now, we can only build plugins on Linux, FreeBSD, and macOS using this approach

• Examples: Krakend, Gotify

---

2.2) Embedded scripting

• Description:

  • Scripts are executed in-process using an embedded scripting engine.
  • As the scripts are interpreted rather than compiled, their implementation can be extended and changed without modifying the existing application

• Pros:

  • Dynamic invocation at runtime
  • Relatively simple as there are no additional runtime processes involved
  • Language bindings allow in-process communication and data passing between existing application and scripts

• Cons:

  • Usually require knowledge of another language e.g. Lua, JavaScript
  • When compared to compiled code, interpreted code may have a reduced performance

• Examples:

  • Agora (a scripting language with Go-like syntax)
  • Otto (a JavaScript interpreter)
  • GopherLua (an interpreter for the Lua scripting language)

---

2.3) Including plugins at compile time

• Description:

  • This approach is used by Golang internally to support extensions e.g. image formats and database drivers
  • In this approach, the host CLI developers will need to compile the existing application themselves along with the source for plugins (developed by third parties) without needing to modify any plugin source code

• Pros:

  • Speed, reliability and ease of use
  • Relatively simple interface with no additional runtime processes, extra deployment artifacts or dependencies
  • Simple deployment as there's only a single binary
  • All code is written in Golang
  • Follows the design patterns of stdlib packages

• Cons:

  • Adding, removing, or updating a plugin requires re-compiling of the entire main application
  • The host CLI developer needs to compile the main program and the plugins together as a single binary
  • The host CLI developer cannot use a pre-packaged binary and connect to it

• Example:

  HTTP middleware is one of the most frequent forms of compile-time plugins. The net/http package of Golang makes it easier to add new handlers to an HTTP server by following these steps:

  • Creating a package that contains one or more functions implementing the Handler interface, or functions that have the signature: func(w http.ResponseWriter, r *http.Request)
  • Importing the net/http package into the application
  • Registering a handler by calling: http.Handle(<pattern>, <pkg_name.handler_name>) or http.HandleFunc(<pattern>, <pkg_name.handlefunc_name>)

---

2.4) Using Goroutines (Not recommended in most cases)

---

2.5) Using Config files (Not recommended in most cases)