Page MenuHomeSoftware Heritage
Feed Advanced Search

Yesterday

tenma updated the diff for D4843: Reimplement Bitbucket lister using new Lister API.

Have test coverage for credentials setting

Mon, Jan 18, 8:32 PM · Lister, Bitbucket lister
tenma updated the diff for D4843: Reimplement Bitbucket lister using new Lister API.
  • apply @vlorentz suggestions
  • use tenacity retrying decorator through lister.utils thanks to D4869
  • remove setters no more useful after D4872
  • adapt tests accordingly
Mon, Jan 18, 7:43 PM · Lister, Bitbucket lister
tenma updated the diff for D4843: Reimplement Bitbucket lister using new Lister API.
  • fix assert
  • fix typing
  • add a warning on credentials
Mon, Jan 18, 9:59 AM · Lister, Bitbucket lister

Sun, Jan 17

tenma updated the diff for D4867: Reimplement PyPI lister using new Lister API.
  • remove useless code as per feeback
  • improve tests
Sun, Jan 17, 11:19 PM · Lister

Fri, Jan 15

tenma added a comment to D4867: Reimplement PyPI lister using new Lister API.

In short, let's KISS!

Fri, Jan 15, 3:52 PM · Lister
tenma accepted D4869: lister: Add utility module to ease HTTP requests rate limit handling.

It would be great to parametrize the request and error type, but unfortunately it does not seem easy to do...

Fri, Jan 15, 2:42 PM
tenma added inline comments to D4866: Populate origin_visit_stats table out of the origin_visit_status topic.
Fri, Jan 15, 12:29 PM
tenma added a comment to D4866: Populate origin_visit_stats table out of the origin_visit_status topic.

incremental review

Fri, Jan 15, 12:08 PM
tenma requested changes to D4858: Add persistence of the field OriginVisitStatus.type.

did not see the requested change on clustering keys by vlorentz

Fri, Jan 15, 11:46 AM
tenma accepted D4858: Add persistence of the field OriginVisitStatus.type.

TGTM, but needs a look by someone more knowledgeable about the DBs

Fri, Jan 15, 11:43 AM
tenma added inline comments to D4864: phabricator: Add test for new lister implementation.
Fri, Jan 15, 10:51 AM
tenma moved T2956: Port PyPI lister to the new Lister API from todo to code review on the Sprint 2021 01 board.
Fri, Jan 15, 10:16 AM · Sprint 2021 01, Lister
tenma claimed T2956: Port PyPI lister to the new Lister API.
Fri, Jan 15, 10:16 AM · Sprint 2021 01, Lister
tenma retitled D4867: Reimplement PyPI lister using new Lister API from [WIP] Reimplement PyPI lister using new Lister API to Reimplement PyPI lister using new Lister API.
Fri, Jan 15, 10:12 AM · Lister
tenma updated the diff for D4867: Reimplement PyPI lister using new Lister API.

Use requests.Response.raise_for_status to handle errors generically

Fri, Jan 15, 10:06 AM · Lister
tenma updated the test plan for D4867: Reimplement PyPI lister using new Lister API.
Fri, Jan 15, 9:52 AM · Lister

Thu, Jan 14

tenma updated the diff for D4867: Reimplement PyPI lister using new Lister API.

Rebase

Thu, Jan 14, 7:37 PM · Lister
tenma requested review of D4867: Reimplement PyPI lister using new Lister API.
Thu, Jan 14, 7:36 PM · Lister
tenma added a revision to T2956: Port PyPI lister to the new Lister API: D4867: Reimplement PyPI lister using new Lister API.
Thu, Jan 14, 6:56 PM · Sprint 2021 01, Lister
tenma added a comment to D4843: Reimplement Bitbucket lister using new Lister API.

Thanks for the remarks, will go through them.

Thu, Jan 14, 3:02 PM · Lister, Bitbucket lister
tenma triaged T2971: Port Gitea lister to the new Lister API as Normal priority.
Thu, Jan 14, 12:59 PM · Lister
tenma updated the summary of D4843: Reimplement Bitbucket lister using new Lister API.
Thu, Jan 14, 12:40 PM · Lister, Bitbucket lister
tenma added a revision to T2955: Port Bitbucket lister to the new Lister API: D4843: Reimplement Bitbucket lister using new Lister API.
Thu, Jan 14, 12:40 PM · Sprint 2021 01, Bitbucket lister, Lister
tenma updated the test plan for D4843: Reimplement Bitbucket lister using new Lister API.
Thu, Jan 14, 10:46 AM · Lister, Bitbucket lister
tenma updated the test plan for D4843: Reimplement Bitbucket lister using new Lister API.
Thu, Jan 14, 10:45 AM · Lister, Bitbucket lister
tenma moved T2956: Port PyPI lister to the new Lister API from Backlog to todo on the Sprint 2021 01 board.
Thu, Jan 14, 10:33 AM · Sprint 2021 01, Lister
tenma moved T2955: Port Bitbucket lister to the new Lister API from in-progress to code review on the Sprint 2021 01 board.
Thu, Jan 14, 10:33 AM · Sprint 2021 01, Bitbucket lister, Lister
tenma added a revision to T2442: Provide a unified API for listers to interact with the scheduler: D4843: Reimplement Bitbucket lister using new Lister API.
Thu, Jan 14, 10:33 AM · Sprint 2021 01, Scheduling utilities
tenma updated the summary of D4843: Reimplement Bitbucket lister using new Lister API.
Thu, Jan 14, 10:33 AM · Lister, Bitbucket lister
tenma updated the summary of D4843: Reimplement Bitbucket lister using new Lister API.
Thu, Jan 14, 10:27 AM · Lister, Bitbucket lister
tenma retitled D4843: Reimplement Bitbucket lister using new Lister API from [WIP] Reimplement Bitbucket lister using new Lister API to Reimplement Bitbucket lister using new Lister API.
Thu, Jan 14, 10:26 AM · Lister, Bitbucket lister

Wed, Jan 13

tenma updated the diff for D4843: Reimplement Bitbucket lister using new Lister API.

Fix copyrights and rebase

Wed, Jan 13, 7:24 PM · Lister, Bitbucket lister
tenma updated the diff for D4843: Reimplement Bitbucket lister using new Lister API.

Improved tests

Wed, Jan 13, 6:28 PM · Lister, Bitbucket lister
tenma updated the diff for D4843: Reimplement Bitbucket lister using new Lister API.

refactor auth and backoff

Wed, Jan 13, 12:12 PM · Lister, Bitbucket lister
tenma requested review of D4843: Reimplement Bitbucket lister using new Lister API.
Wed, Jan 13, 11:49 AM · Lister, Bitbucket lister

Tue, Jan 12

tenma created P920 pytest lister celery timeout.
Tue, Jan 12, 8:11 PM · Lister
tenma created P919 tox packages celery timeout pkg_resources.ContextualVersionConflict.
Tue, Jan 12, 8:03 PM · Lister

Mon, Jan 11

tenma updated the task description for T2956: Port PyPI lister to the new Lister API.
Mon, Jan 11, 8:09 PM · Sprint 2021 01, Lister
tenma updated the task description for T2956: Port PyPI lister to the new Lister API.
Mon, Jan 11, 8:09 PM · Sprint 2021 01, Lister
tenma updated the task description for T2956: Port PyPI lister to the new Lister API.
Mon, Jan 11, 8:08 PM · Sprint 2021 01, Lister
tenma triaged T2956: Port PyPI lister to the new Lister API as Normal priority.
Mon, Jan 11, 8:08 PM · Sprint 2021 01, Lister
tenma updated the task description for T2955: Port Bitbucket lister to the new Lister API.
Mon, Jan 11, 7:20 PM · Sprint 2021 01, Bitbucket lister, Lister
tenma updated the task description for T2955: Port Bitbucket lister to the new Lister API.
Mon, Jan 11, 7:17 PM · Sprint 2021 01, Bitbucket lister, Lister
tenma updated the task description for T2955: Port Bitbucket lister to the new Lister API.
Mon, Jan 11, 7:15 PM · Sprint 2021 01, Bitbucket lister, Lister
tenma triaged T2955: Port Bitbucket lister to the new Lister API as Normal priority.
Mon, Jan 11, 7:04 PM · Sprint 2021 01, Bitbucket lister, Lister
tenma edited P918 2021 New lister template.
Mon, Jan 11, 5:50 PM · Lister
tenma created P918 2021 New lister template.
Mon, Jan 11, 5:39 PM · Lister

Fri, Jan 8

tenma updated the diff for D4731: WIP Configuration system.
  • Take into account feedback from ardumont and vlorentz
  • Take into account feedback from douardda
  • still TODO: rename "singletons"
Fri, Jan 8, 5:24 PM · feedback-required, Core & foundations

Thu, Jan 7

tenma added a comment to D4731: WIP Configuration system.

@douardda: taking into account your suggestions.

Thu, Jan 7, 10:47 AM · feedback-required, Core & foundations

Wed, Jan 6

tenma updated the diff for D4766: WIP git bare revision cooker.
  • use directory_ls rather than dir_iterator to lower overhead and control iteration
  • use a set to test early for known objects
  • only update once master head with the given revision id
  • do not chunk files ids because in the end the storage requests are not chunked
  • partition new and old files to adapt processing
Wed, Jan 6, 8:35 PM

Mon, Jan 4

tenma lowered the priority of T2743: Investigate why logging is inconsistent from Normal to Low.
Mon, Jan 4, 5:00 PM · Vault, Object storage, Core & foundations

Wed, Dec 23

tenma added projects to D4731: WIP Configuration system: Core & foundations, feedback-required.
Wed, Dec 23, 10:50 AM · feedback-required, Core & foundations

Tue, Dec 22

tenma updated the diff for D4731: WIP Configuration system.

Be more consistent in naming and abbreviating

Tue, Dec 22, 4:09 PM · feedback-required, Core & foundations
tenma updated the diff for D4731: WIP Configuration system.

Take into account feedback from ardumont and vlorentz

Tue, Dec 22, 1:21 PM · feedback-required, Core & foundations
tenma committed rDDOC2908cd7d47e4: Add note for pip package resolution issues (authored by tenma).
Add note for pip package resolution issues
Tue, Dec 22, 10:38 AM
tenma closed D4751: Add hint for pip package resolution issues.
Tue, Dec 22, 10:38 AM
tenma updated the test plan for D4751: Add hint for pip package resolution issues.
Tue, Dec 22, 10:37 AM
tenma updated the diff for D4751: Add hint for pip package resolution issues.

Used a "note" directive

Tue, Dec 22, 10:30 AM

Mon, Dec 21

tenma added a comment to D4731: WIP Configuration system.

But for components, we wanted to instantiate every instance only one time

What is the benefit of doing that (vs. the cost of the extra complexity)?

Mon, Dec 21, 8:41 PM · feedback-required, Core & foundations
tenma added a comment to D4731: WIP Configuration system.

First, thanks for the feedback, every help is welcome because I am still pretty the newbie in the project as a whole, and this is a more complex topic that initially anticipated.

Until now, configuration handling was implemented independently for each service.

What do you mean by that? It has always been handled by SWHConfig in swh.core.

Mon, Dec 21, 5:07 PM · feedback-required, Core & foundations
tenma added a comment to D4771: SWHID parsing: simplify and deduplicate validation logic.
@vlorentz wrote:

btw, you don't have to expose SWHID_RE_RAW directly, it's available as SWHID_RE.pattern

Mon, Dec 21, 3:03 PM
tenma added a revision to T843: Vault: Add a "git bare" tarball cooker: D4766: WIP git bare revision cooker.
Mon, Dec 21, 10:27 AM · Vault
tenma updated the summary of D4766: WIP git bare revision cooker.
Mon, Dec 21, 10:27 AM

Dec 18 2020

tenma updated the diff for D4766: WIP git bare revision cooker.

Little cleanup

Dec 18 2020, 12:03 PM
tenma created D4766: WIP git bare revision cooker.
Dec 18 2020, 10:59 AM

Dec 16 2020

tenma updated the task description for T1410: Kill implicit configuration: new configuration scheme.
Dec 16 2020, 10:48 AM · Core & foundations
tenma created D4751: Add hint for pip package resolution issues.
Dec 16 2020, 10:45 AM

Dec 15 2020

tenma updated the task description for T2885: investigate the persistent seemingly benign errors on cooking progress.
Dec 15 2020, 8:33 PM · Vault
tenma updated the task description for T2885: investigate the persistent seemingly benign errors on cooking progress.
Dec 15 2020, 8:32 PM · Vault
tenma committed rDENVdd20e4f4c158: Data persistence for a devel setting (authored by tenma).
Data persistence for a devel setting
Dec 15 2020, 6:27 PM
tenma closed D4673: Data persistence for a devel setting.
Dec 15 2020, 6:27 PM
tenma updated the diff for D4673: Data persistence for a devel setting.

Rebase

Dec 15 2020, 6:27 PM
tenma updated the diff for D4673: Data persistence for a devel setting.

Finally fixed this PREsistent typo!

Dec 15 2020, 6:24 PM
tenma updated the diff for D4731: WIP Configuration system.

Reflow the whole source to 88 chars lines

Dec 15 2020, 5:48 PM · feedback-required, Core & foundations
tenma updated the diff for D4731: WIP Configuration system.

Added a TODO about a section on guidelines

Dec 15 2020, 5:17 PM · feedback-required, Core & foundations
tenma updated the diff for D4731: WIP Configuration system.

Cleaned and wrote up several sections

Dec 15 2020, 4:31 PM · feedback-required, Core & foundations

Dec 14 2020

tenma updated the task description for T2885: investigate the persistent seemingly benign errors on cooking progress.
Dec 14 2020, 3:01 PM · Vault
tenma triaged T2885: investigate the persistent seemingly benign errors on cooking progress as Normal priority.
Dec 14 2020, 3:00 PM · Vault
tenma closed T1191: Vault: reschedule failed tasks after a manual/non-manual resolution as Resolved.

In addition to failed cooking retry, failed RPC calls are now also retried.

Dec 14 2020, 12:10 PM · Vault
tenma updated the summary of D4720: developer-setup: update system dependencies.
Dec 14 2020, 10:56 AM

Dec 11 2020

tenma added a revision to T1410: Kill implicit configuration: new configuration scheme: D4731: WIP Configuration system.
Dec 11 2020, 7:22 PM · Core & foundations
tenma created D4731: WIP Configuration system.
Dec 11 2020, 7:22 PM · feedback-required, Core & foundations
tenma updated the diff for D4673: Data persistence for a devel setting.

Removed usage of external host volumes, as asked by @douardda, for brievety.

Dec 11 2020, 4:05 PM
tenma committed rDDOCb8d68f27ece3: developer-setup: update system dependencies (authored by tenma).
developer-setup: update system dependencies
Dec 11 2020, 2:37 PM
tenma closed D4720: developer-setup: update system dependencies.
Dec 11 2020, 2:37 PM
tenma updated the diff for D4720: developer-setup: update system dependencies.

Keep postgresql without major version pin, as we follow debian versions.

Dec 11 2020, 2:23 PM
tenma created D4720: developer-setup: update system dependencies.
Dec 11 2020, 12:12 PM

Dec 10 2020

tenma added inline comments to D4673: Data persistence for a devel setting.
Dec 10 2020, 7:32 PM
tenma added inline comments to D4673: Data persistence for a devel setting.
Dec 10 2020, 7:28 PM
tenma updated the diff for D4673: Data persistence for a devel setting.

Usa a docker-compose.override definition.
Rephrased where appropriate.

Dec 10 2020, 6:40 PM

Dec 9 2020

tenma added inline comments to D4673: Data persistence for a devel setting.
Dec 9 2020, 7:05 PM
tenma abandoned D4630: WIP backend: add retrying on IO error with remote services.

Another strategy preferred: using retrying proxies for storage (existing and activated) and scheduler (to be done).

Dec 9 2020, 4:44 PM

Dec 8 2020

tenma triaged T2867: Webapp search UI: set focus on search input by default as Normal priority.
Dec 8 2020, 6:00 PM · Web app, Archive search
tenma accepted D4677: vault: Add retry ability to the storage.

Tested in docker env, the config is valid after D4685 applied.

Dec 8 2020, 5:01 PM
tenma committed rDVAU1b6a10fded60: vault: Remove deprecated services default config (authored by tenma).
vault: Remove deprecated services default config
Dec 8 2020, 3:42 PM
tenma closed D4685: server: Remove deprecated services default config.
Dec 8 2020, 3:42 PM
tenma committed rDVAU91f64b9e7a6f: cli: Remove deprecated logging configuration (authored by tenma).
cli: Remove deprecated logging configuration
Dec 8 2020, 3:42 PM
tenma closed D4680: cli: Remove deprecated logging configuration.
Dec 8 2020, 3:42 PM
tenma retitled D4685: server: Remove deprecated services default config from vault: Remove deprecated services default config to server: Remove deprecated services default config.
Dec 8 2020, 3:01 PM
tenma updated the summary of D4685: server: Remove deprecated services default config.
Dec 8 2020, 2:59 PM