Extend the ActivityQueueService to optionally use Celery

added 45m of time spent at 2021-05-05

Proposal

DjangoLDP Core

#380 - estimate: TODO

Dependency injection

ActivityPubService can take as an init parameter an Activity broker, where at the moment it makes calls to ActivityQueueService.send_activity

ActivityQueueService.start call moved from DjangoLDP app config to ActivityQueueService.__init__
ActivityPubService.init will take an (optional) queue_service parameter which can be overridden (default ActivityQueueService). It will be initialised by the DjangoLDP app startup

Estimate: 1h

Documentation

Documentation on writing a custom Queue

Estimation: 2h

New DjangoLDP-ActivityQueue-Celery Package

repository setup etc

Estimate: 2h

Django code

Celery file
task for send_activity
injecting celery queue service which triggers the send_activity celery task
retrying failed tasks with celery

Estimate: 4h

unit tests
Note that one of the points in #332 involves looking into how we might conduct federation (multi-server) unit tests, which would be useful here if successful
tests on staging

Estimate: 1d

Deployment

only support Redis broker for now
we'll need to deploy a Redis broker for Celery (auto-deployment)
the Redis broker will need some settings configured

we'll need to deploy a Redis broker for Celery (auto-deployment)

@plup will need you for estimating this part

@calummackervoy I was about to write: sorry pal, but our hoster doesn't allow us to install our own services but you're in luck, just checking there blog and the functionnality appeared few days ago.

Because our Cloud platform is particularly loved by Pythonistas, you’ll be pleased to learn that running a Celery process (associated to a RabbitMQ, already available, or a Redis in its own instance) became as simple as declaring a Site !

https://blog.alwaysdata.com/2021/04/08/services-kill-the-daemons/#observability-for-your-apps

I never tried so I might need some time to make it happen.

What would you need ? Just a redis backend ? (If we go on with RabbitMQ I can make you a POC in 2 hours).

A priori Redis is lighter and more widely used but I'm happy if we use RabbitMQ, I skimmed a couple of online comparisons

Redis is a database that can be used as a message-broker. On the other hand, RabbitMQ has been designed as a dedicated message-broker. RabbitMQ outperforms Redis as a message-broker in most scenarios

I had some budget (from FNK?) to estimate this but I don't know how far that extends. I left another comment below about how we use ActivityPub in general, I think we probably want to resolve that before doing a POC

@balessan @jbpasquier I've provided most of an estimation above (#382 (comment 62561))

I think that we should discuss the overall direction of the ActivityPubService first, maybe it will be easier in a call

At the moment we detect backlinks on a changes to any Model, totally automatically. Much of the complex functionality in ActivityQueueService, and the delay, are both related to catching unnecessary activities which can't be caught in the post_save. Even so there is a bug that some superfluous activities are going through. The original ActivityQueueService takes care to use only one worker, and things will undoubtedly get more difficult to visualise when Celery has multiple workers running simultaneously

A concurrent issue is with the need to attach the user as the actor of an activity and no clear-cut solutions to that one have presented themselves yet

added 1h 30m of time spent at 2021-06-07

mentioned in issue management/core-team#2

mentioned in issue #380

mentioned in issue energie-partagee/djangoldp_energiepartagee#8