Conveyor: pipelines approach opposite to channels

Releases
1

To build a kafka/rabbitmq consuming pipeline with saturated stages in Go you normally need to spin a goroutine per stage and connect them with channels (whose buffer can’t be change in runtime). The logic of processing one message (or batch of messages) gets split between multiple functions not allowing you to use local variables and making it complicated to sync different branches of execution (in case of fan-outs).

Conveyor’s approach is orthogonal to that:

  • define a single function that processes an item (message or a batch),
  • add stage.MoveTo() gates between the stages in the function and
  • let the lib manage goroutines and ordering.
  • Supported pipeline topologies are deadlock-free: execution always moves forwards and earlier items have priority over later.
  • Keep the state of item processing in local variables in the function and use them at any stage you need.
  • Graceful queue size and concurrency limit changes, graceful shutdown and observability out of the box

Ready to answer questions. Don’t miss the great WebAssembly interactive demo

2

Interesting approach, here is some opinionated critique:

API-Ergonomics. I don’t like the “stage.moveTo()” syntax. It is not clear at all that something is locked and without an explicit “release” it is very unclear when it is released. Go provides a nice option with aquire() defer release(), which feels very Go-Native and provides stability with early returns or panics. I would prefer a readStage.aquire() and defer readStage.release()

I’m missing an easy option to parallelize the processing of individual stages - as it stands I would have to code this myself when using the library - whereas channels can easily have multiple concurrent readers, which execute multiple packets in parallel. In your implementation, if a single item is very slow in a single stage, that stage will be locked until this item is completely processed, implicitly blocking all prior stages from advancing. So a single slow network call would slow down the whole pipeline.

No automatic scaling/back pressure. A good pipeline system would provide some feedback via back pressure from later stages, so earlier stages can adjust their throughput dynamically to provide an even load.