Page MenuHomeSoftware Heritage
Differential D6772

Clean up old scheduler archival task related services
ClosedPublic
Actions

Authored by ardumont on Dec 7 2021, 2:37 PM.

Details

Reviewers
olasd
Group Reviewers
Reviewers
Maniphest Tasks
T3782: Drop tasks archival service
Commits
rDSCHa8edbdbb0071: Clean up disabled scheduler archival task related services
Summary

This is dead code now as this has long been stopped and disabled in production.

Note:
Also, it was probably not that well tested (and the coverage peek we have with
this diff seems to approve)

Related to T3782

Test Plan

tox (should be happy)

Diff Detail

Repository
rDSCH Scheduling utilities
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

ardumont created this revision.Dec 7 2021, 2:37 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptDec 7 2021, 2:37 PM
swh-public-ci added a comment.Dec 7 2021, 2:38 PM

Build has FAILED

Patch application report for D6772 (id=24570)

Rebasing onto 0a6aac583a...

Current branch diff-target is up to date.
Changes applied before test
commit 4ca80a31e20c33146cd653535a37d9f28ddb8a5c
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Dec 7 14:31:34 2021 +0100

    Clean up old ghtorrent scheduler related services
    
    This is dead code now as hhis has long been stopped and disabled in production.
    
    Related to T3777

Link to build: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/498/
See console output for more information: https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/498/console

Harbormaster returned this revision to the author for changes because remote builds failed.Dec 7 2021, 2:38 PM
Harbormaster failed remote builds in B25442: Diff 24570!
ardumont updated this revision to Diff 24571.Dec 7 2021, 2:45 PM

Drop some more code

swh-public-ci added a comment.Dec 7 2021, 2:48 PM

Build is green

Patch application report for D6772 (id=24571)

Rebasing onto 0a6aac583a...

Current branch diff-target is up to date.
Changes applied before test
commit de2ed04ec5e3293e6cc55cb31254ff1abfa2b692
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Dec 7 14:31:34 2021 +0100

    Clean up old ghtorrent scheduler related services
    
    This is dead code now as hhis has long been stopped and disabled in production.
    
    Related to T3777

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/499/ for more details.

Harbormaster completed remote builds in B25443: Diff 24571.Dec 7 2021, 2:48 PM
ardumont requested review of this revision.Dec 7 2021, 2:48 PM
ardumont edited the summary of this revision. (Show Details)Dec 7 2021, 2:49 PM
ardumont updated this revision to Diff 24606.Dec 8 2021, 10:28 AM

Rebase

swh-public-ci added a comment.Dec 8 2021, 10:30 AM

Build is green

Patch application report for D6772 (id=24606)

Rebasing onto 5de8ba42c2...

Current branch diff-target is up to date.
Changes applied before test
commit f0369228b933f76c5f70e6aef5cf59594f8bf9bb
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Dec 7 14:31:34 2021 +0100

    Clean up old ghtorrent scheduler related services
    
    This is dead code now as hhis has long been stopped and disabled in production.
    
    Related to T3777

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/501/ for more details.

Harbormaster completed remote builds in B25466: Diff 24606.Dec 8 2021, 10:30 AM
ardumont retitled this revision from Clean up old ghtorrent scheduler related services to Clean up old scheduler archival task related services.EditedDec 8 2021, 10:40 AM
ardumont added a subscriber: olasd.

Thank @olasd to make me realize it's not ghtorrent stuff (it's already been removed a while back), it's about the old task archival which got disabled as it was no longer running with decent time.

olasd added a comment.Dec 8 2021, 10:50 AM

So, I think archiving old task entries is still something that we should want to do, but then again as this scheduler is slowly being phased out it may not be as critical as it used to be. Meh.

Aren't there some (test?) requirements that can be cleaned up now?

ardumont added a comment.Dec 8 2021, 10:57 AM

So, I think archiving old task entries is still something that we should want to do, but then again as this scheduler is slowly being phased out it may not be as critical as it used to be. Meh.

I agree.

Like i said in irc, given (in any order)

  • the somewhat complexity of the code
  • the poor coverage (of the time) and my lazyness to actually parse through it and improve the actual coverage
  • the fact that's not running.
  • what you said about this scheduler being phased out
  • you are neutral on the matter

I'd pretty much vote to drop that code.
That's less code to maintain or discover (with possibly questions which i probably no longer have answers for) for newcomers.

Aren't there some (test?) requirements that can be cleaned up now?

I'll double check.

vlorentz added a subscriber: vlorentz.Dec 8 2021, 11:13 AM

You should be able to remove elasticsearch from requirements.txt

ardumont updated this revision to Diff 24613.Dec 8 2021, 11:14 AM
  • Amend commit
  • Drop no longer necessary requirement
ardumont added a comment.Dec 8 2021, 11:14 AM

You should be able to remove elasticsearch from requirements.txt

yes, done ;)

swh-public-ci added a comment.Dec 8 2021, 11:17 AM

Build is green

Patch application report for D6772 (id=24613)

Rebasing onto 5de8ba42c2...

Current branch diff-target is up to date.
Changes applied before test
commit a8edbdbb00717f62e44ce87bfe6c1f698abfd728
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Dec 7 14:31:34 2021 +0100

    Clean up disabled scheduler archival task related services
    
    This is dead code now as this has long been stopped and disabled in production.
    
    Related to T3777

See https://jenkins.softwareheritage.org/job/DSCH/job/tests-on-diff/502/ for more details.

Harbormaster completed remote builds in B25472: Diff 24613.Dec 8 2021, 11:17 AM
ardumont edited the summary of this revision. (Show Details)Dec 8 2021, 11:53 AM
ardumont removed a task: T3777: Cleanup old ghtorrent scheduler services.
olasd accepted this revision.Dec 9 2021, 10:33 AM
This revision is now accepted and ready to land.Dec 9 2021, 10:33 AM
ardumont edited the summary of this revision. (Show Details)Dec 9 2021, 10:52 AM
ardumont added a task: T3782: Drop tasks archival service.
Closed by commit rDSCHa8edbdbb0071: Clean up disabled scheduler archival task related services (authored by ardumont). · Explain WhyDec 9 2021, 11:06 AM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDSCHa8edbdbb0071: Clean up disabled scheduler archival task related services.

Revision Contents
Changeset List

Diff 24674

View Options

requirements.txt

Loading...
View Options

swh/scheduler/backend_es.py

Loading...
View Options

swh/scheduler/cli/task.py

Loading...
View Options

swh/scheduler/elasticsearch_memory.py

Loading...
View Options

swh/scheduler/tests/es/__init__.py

Loading...
View Options

swh/scheduler/tests/es/conftest.py

Loading...
View Options

swh/scheduler/tests/es/test_backend_es.py

Loading...
View Options

swh/scheduler/tests/es/test_cli_task.py

Loading...
View Options

swh/scheduler/tests/es/test_elasticsearch_memory.py

Loading...
About · Developer information · Legal