⚓ T1051 Consume ghtorrent events as oneshot tasks with priority normal

		Status	Assigned	Task
		Migrated	gitlab-migration	T1031 Speed up updates - meta task
		Migrated	gitlab-migration	T1051 Consume ghtorrent events as oneshot tasks with priority normal

ardumont changed the task status from Open to Work in Progress.May 9 2018, 10:33 AM

ardumont triaged this task as Normal priority.

ardumont created this task.

ardumont updated the task description. (Show Details)May 9 2018, 10:38 AM

Heads up on this, i'm currently stuck with the ghtorrent infra. Following the documentation, I'm not able to stream yet (as of monday).
I have notified in the ghtorrent repo (as a comment to an existing and identical issue [1]).

In the mean time, I'm working around this in using ad-hoc fake event generator (ghtorrent's mechanism-like).
I'll start from there, abstract from it and adapt later when unstuck.

[1] https://github.com/ghtorrent/ghtorrent.org/issues/397#issuecomment-387052462

Following a ping email to @gousiosg, he restarted the failed service on their end.

Quoting him [1]:

It turned out to be an issue with the service that gets the events from MongoDB and publishes them to RabbitMQ. It is now started with supervise so hopefully it won't die again :-)

Now, everything seems to be ok, i have data streaming on my end.

[1] https://github.com/ghtorrent/ghtorrent.org/issues/397#issuecomment-388761915

ardumont mentioned this in rDSCH5de5eb9e3b41: scheduler.updater: Add publish/subscribe fake ghtorrent class.May 25 2018, 5:27 PM

ardumont mentioned this in rDSCHccdc1346032b: swh.scheduler.updater: Add SWHEvent class and tests around it.

ardumont mentioned this in rDSCH464d75960e61: Reference tryout code work to assert we can work with ghtorrent.

ardumont mentioned this in rDSCH9dd3cd344779: swh.scheduler.updater.backend: Bootstrap backend api.

ardumont mentioned this in rDSCH8d5bb5b7496d: updater: Actually consume ghtorrent event.

ardumont mentioned this in rDSCHd925b052a9f8: swh.scheduler.backend: Drop duplicate tasks at creation time.

ardumont mentioned this in rSPPROF5e67f10302d2: deploy/scheduler: Run the scheduler updater consumer service.May 30 2018, 5:16 PM

ardumont mentioned this in rSPPROFa6796a4da914: deploy/scheduler: Run the scheduler updater consumer service.May 30 2018, 5:22 PM

ardumont renamed this task from Use ghtorrent to speed up updates to Consume ghtorrent events as oneshot tasks with priority normal.May 31 2018, 12:02 PM

ardumont updated the task description. (Show Details)

ardumont closed this task as Resolved.May 31 2018, 12:43 PM

ardumont claimed this task.

ardumont mentioned this in rSPSITE5e67f10302d2: deploy/scheduler: Run the scheduler updater consumer service.Jun 15 2018, 2:30 PM

ardumont mentioned this in rSPSITEa6796a4da914: deploy/scheduler: Run the scheduler updater consumer service.