Jetway.Producer (Jetway v0.1.0)

Copy Markdown View Source

A Broadway producer that pulls messages from a NATS JetStream durable pull consumer.

Every producer of the pipeline subscribes to its own inbox and long-polls the same durable consumer with pull requests of up to :max_batch_size messages, bounded by the demand of the pipeline. JetStream spreads the messages of the consumer across all producers, so the pipeline can run any number of producers (concurrency), and several nodes can run the same pipeline against the same consumer.

The producer talks to NATS through a Gnat connection that you start and supervise yourself, usually with Gnat.ConnectionSupervisor, before the pipeline:

children = [
  {Gnat.ConnectionSupervisor,
   %{name: :gnat, connection_settings: [%{host: "localhost", port: 4222}]}},
  MyBroadway
]

When the connection goes down, the producer waits for Gnat.ConnectionSupervisor to bring it back and resumes pulling. Messages that were in flight are redelivered by JetStream after the consumer's ack_wait.

Options

  • :connection_name - Required. The registered name of the Gnat connection, usually the :name given to Gnat.ConnectionSupervisor. A pid is accepted too, but it becomes stale as soon as the connection is restarted, so prefer a name.

  • :stream_name (String.t/0) - The name of the stream to consume from. Required unless :consumer is given.

  • :consumer_name (String.t/0) - The name of the durable pull consumer to fetch messages from. Required unless :consumer is given.

  • :consumer (struct of type Gnat.Jetstream.API.Consumer) - A Gnat.Jetstream.API.Consumer definition. When given, every producer creates the durable consumer on connect if it does not exist yet (an existing consumer is used as is). The stream name, consumer name and domain are taken from the struct, so this option cannot be combined with :stream_name, :consumer_name or :domain.

  • :domain (String.t/0) - The JetStream domain of the stream.

  • :inbox_prefix (String.t/0) - The prefix of the subject each producer receives its messages on. The default value is "_INBOX.".

  • :connection_retries (non_neg_integer/0) - How many times the producer retries to find the Gnat connection or to reach the consumer before it stops. The default value is 10.

  • :connection_retry_timeout (non_neg_integer/0) - The time in milliseconds between those retries. The default value is 1000.

  • :request_expires (pos_integer/0) - The time in milliseconds a pull request stays open on the server while waiting for messages. When it expires, the producer immediately issues a new one. The default value is 5000.

  • :receive_interval (non_neg_integer/0) - The time in milliseconds to wait before pulling again after the server rejected a pull request. The default value is 5000.

  • :max_batch_size (pos_integer/0) - The maximum number of messages requested by a single pull request. The producer requests min(demand, max_batch_size) messages at a time. The default value is 100.

  • :on_success - The acknowledgement action for successful messages. See the "Acknowledgements" section below. The default value is :ack.

  • :on_failure - The acknowledgement action for failed messages. See the "Acknowledgements" section below. The default value is :nack.

Consumer requirements

The consumer must be a durable pull consumer (no deliver_subject) with ack_policy: :explicit. All producers of the pipeline share it. Either create it upfront, for example with the nats CLI:

nats consumer add ORDERS order-worker --pull --ack explicit --max-deliver 5

or pass a Gnat.Jetstream.API.Consumer struct as the :consumer option and let the producers create it.

JetStream stops delivering once a consumer has max_ack_pending unacknowledged messages, so set it to at least the number of messages the pipeline holds in flight (all processors and batchers). If the consumer sets max_batch, keep :max_batch_size less than or equal to it, otherwise the server rejects the pull requests with 409 Exceeded MaxRequestBatch.

Redelivery

A message must be acknowledged within the consumer's ack_wait (30 seconds by default), counted from its delivery. The producer does not extend that window, so a message still sitting in a processor or batcher when ack_wait elapses is delivered again, possibly to another producer, while the first copy is still being handled. Set ack_wait above your longest handling time plus batch_timeout, and make handlers idempotent, for example keyed on message.metadata.jetstream.stream_seq or on a Nats-Msg-Id header set by the publisher.

max_deliver is unlimited (-1) unless the consumer sets it, both in NATS and in Gnat.Jetstream.API.Consumer. With the default on_failure: :nack, a message that always fails is then redelivered immediately and forever. Always set max_deliver, and use {:nack, delay_ms} to space out retries: a plain :nack ignores the consumer's backoff, which only applies to messages left unacknowledged past ack_wait (:noop).

JetStream has no dead-letter queue. When a message reaches max_deliver, the server publishes an advisory on $JS.EVENT.ADVISORY.CONSUMER.MAX_DELIVERIES.<stream>.<consumer> whose stream_seq identifies the message. Subscribe to it and fetch the message with Gnat.Jetstream.API.Stream.get_message/4 to handle dead letters.

Acknowledgements

You can use the :on_success and :on_failure options to control how messages are acknowledged. You can set these options when starting the producer or change them for each message through Broadway.Message.configure_ack/2. By default, successful messages are acked (:ack) and failed messages are negatively acknowledged (:nack).

The possible values for :on_success and :on_failure are:

  • :ack - acknowledges the message. JetStream will not redeliver it.

  • :nack - negatively acknowledges the message. JetStream redelivers it right away, up to the consumer's max_deliver (unlimited by default, see "Redelivery").

  • {:nack, delay_ms} - negatively acknowledges the message. JetStream redelivers it after delay_ms milliseconds, up to the consumer's max_deliver.

  • :term - terminates the message. JetStream will not redeliver it, but it is not considered successfully processed either (it is reported as an advisory).

  • :noop - does nothing. JetStream redelivers the message once the consumer's ack_wait elapses.

For example, to give a failed message a minute before it is retried:

def handle_message(_, message, _) do
  case process(message.data) do
    :ok ->
      message

    {:error, :rate_limited} ->
      message
      |> Broadway.Message.configure_ack(on_failure: {:nack, 60_000})
      |> Broadway.Message.failed(:rate_limited)
  end
end

Message metadata

Each Broadway.Message has the following metadata:

  • :subject - the subject the message was published to.
  • :headers - the message headers as a list of {name, value} tuples ([] when there are none).
  • :reply_to - the $JS.ACK.… subject used to acknowledge the message.
  • :jetstream - a Gnat.Jetstream.API.Message.Metadata struct with the stream and consumer sequences, the number of deliveries, the number of pending messages and the timestamp of the message.

Example

defmodule MyBroadway do
  use Broadway

  def start_link(_opts) do
    Broadway.start_link(__MODULE__,
      name: __MODULE__,
      producer: [
        module:
          {Jetway.Producer,
           connection_name: :gnat,
           stream_name: "ORDERS",
           consumer_name: "order-worker"},
        concurrency: 2
      ],
      processors: [
        default: [concurrency: 10]
      ]
    )
  end

  @impl true
  def handle_message(_processor, message, _context) do
    IO.inspect(message.data, label: "Got message")
    message
  end
end