topics

[remkoboschker]

We (myself and @albertjan) use bull (and oxen; a bull implementation in f#) and would like to add support for "topics". With "topic" I mean an identifier for a grouping of queues I can add a job to. Each queue would have the same semantics as it currently does.

We were thinking along the following lines:

• two Queue constructors
  1. as it is now: Queue(queuename, port, host, opts)
  2. Queue(topicname, queuename, port, host, opts)
• otherwise the api stays the same
• queues in redis can either have bull:queuename:wait etc. identifiers or bull:topicname:queuename:wait etc. identifiers
• jobs can have either bull:queuename:id or bull:topicname:queuename:id identifiers; this means copying the jobs themselves instead of referencing the same job in all queues in the topic, but otherwise we could get into trouble updating the job's progess.
• a queue.add() where the queue was constructed with ii. will add the job to all queue's that have the same topicname; however if the Queue instance only adds jobs an unused wait set would be created. This could be prevented if you construct a queue with i. and add a topic: true option to the opts in Queue.add; the queuename parameter would then be used as the topicname.
• if a topic exists, but the queue does not, then the queuename will be added to a bull:topicname:queues set of queue ids. Normal queue semantics apply, no existing jobs are copied.

Does anyone have similar use-cases? What would be requirements? Are we missing anything?

We intend to write an implementation and submit a pull request to bull en to oxen.

@manast would you consider accepting such a pull request?

[manast]

Hello,

Could you please explain a high level use-case were topics are useful and how they are supposed to work from the users perspective?

thanks.

[remkoboschker]

Hi,

We are using bull to decouple the production of a series of reports based on a user survey. When the survey finishes a message is sent to several queues where workers do part of the processing. When they are finished the intermediary result is used by several other workers that receive a message that the intermediary result is available and so on. This means that parts need to know about other parts and increases coupling.

We would like reverse the direction where a part can determine what messages it is interested in. Because there could be many workers listening to the same message and we want to ensure that each type of worker only processes a job once and ensure that it gets processed at least one. We need something that is more reliable than pub-sub, but with the one-to-many semantics, and that ensures that a job is processed no more than once whilst allowing for scaling using multiple workers. Topics would be a good solution. We got the terminology from Azure Service Bus

[manast]

thanks for the explanation, I would need some time to digest this information.

[manast]

so one question that arises is, must this functionality be implemented in the core or can it be implemented on top of bull in a higher level module?

[albertjan]

I guess it doesn't have to be in the core. We could monkey patch a new constructor onto bull. Or add a Topic type. But we thought it would be something more people would run into using queue's and therefore a nice addition to bull. Do you think it would make more sense as a separate module?

[manast]

Yes I think that if it is feasible it would be better to have it in a separate module (maybe bull-topic), so that we can keep the core as small and stable as possible. The new module can be referenced from the main README so that all users of bull can know about its existence.

[kentnek]

Hi, any update on this?

[remkoboschker]

I do not expect to find the time for this anytime soon. Our original requirement is still there, but less urgent. We can handle the current load easily, so we do not need to scale horizontally. The other use was for easier configuration ( a receiving service can subscribe to a topic and the sender does not need to know anything about subscribers) is manageable. Good luck

On 18 Mar 2016, at 18:27, Kent notifications@github.com wrote:

Hi, any update on this?

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub

[manast]

I like the concept of topics and @remkoboschker proposal implementation seems like a good start.
Some things I would like to solve better. Adding a job from a queue that does not process (I would like to separate queues between clients and workers, so this will not be a problem), and the problem of adding jobs to a topic when there are no workers yet listening to that topic.

[darbio]

Did this go anywhere? I have a requirement for this at the moment which I am currently fulfilling using NRP 'fanning out' to bull job queues.

The NRP 'topic' queue is listened to (or not) by (a) process(es) which knows what that when it receives a certain type of event, it creates 'x' jobs on pre-determined bull queues.

The problem is that NRP uses Redis PubSub, which doesn't persist events - meaning that if the process which ingests NRP events dies, the downstream workers don't get the event and jobs fail to be processed.

This can be overcome by caching events in a database, but it increases the complexity of the system which I am reluctant to do.

[darbio]

I put up a simple repo with an example of what I am doing at the moment (sans DB persistence for the events published to the topic). Hope it helps someone somewhere:

https://github.com/darbio/redis-topic-queues

[manast]

So one thing that strikes me is why can't you use an extra queue for sending and receiving the events? The worker for such a queue would basically do the same as you are doing on the domains event listener: https://github.com/darbio/redis-topic-queues/blob/master/eventSubscriber.js#L22
But the difference will be that no events will be lost if any of the services goes down...

[darbio]

Because the event subscriber uses a pattern match to catch all the events ('domain.events.*'). Can bull do this too?

[manast]

No it can't. But can't the publisher just add to a queue all the events that start with domain.events instead of emitting an event?

[darbio]

I'll have a play today. The way I can see it happening is how you described:

Domain events go to a queue called 'domain.events' which listens for all events. The attached worker process then works in the same way.

Should work - I'll update the example.

[darbio]

Updated the example repo. It no longer uses NRP and only uses Bull to provide the topic. Thanks for the peer review @manast 👍