Page MenuHomeSoftware Heritage
Feed Advanced Search

Jun 11 2021

ardumont moved T3367: scheduler: Separate scheduling save code now tasks from the recurrent tasks from Backlog to deployed/landed/monitoring on the System administration board.
Jun 11 2021, 11:58 AM · System administration, Scheduling utilities, Save Code Now
ardumont added a project to T3367: scheduler: Separate scheduling save code now tasks from the recurrent tasks: System administration.
Jun 11 2021, 11:58 AM · System administration, Scheduling utilities, Save Code Now
ardumont added a comment to T3367: scheduler: Separate scheduling save code now tasks from the recurrent tasks.

Deployed first yesterday on staging (manually) to check the behavior.
Everything was fine.

Jun 11 2021, 11:58 AM · System administration, Scheduling utilities, Save Code Now
rdicosmo added a comment to T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete.

@ardumont, @rdicosmo, I just figured out that data we are missing (Content-Length, Last-Modified) from tarballs archived by the Internet archive are in fact available in x-archive-orig-* HTTP response headers,

Jun 11 2021, 11:39 AM · Save Code Now
ardumont added a comment to T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete.

That's great news!

Jun 11 2021, 11:32 AM · Save Code Now
anlambert added a comment to T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete.

@ardumont, @rdicosmo, I just figured out that data we are missing (Content-Length, Last-Modified) from tarballs archived by the Internet archive are in fact available in x-archive-orig-* HTTP response headers, see below:

Jun 11 2021, 11:29 AM · Save Code Now
ardumont added a revision to T3367: scheduler: Separate scheduling save code now tasks from the recurrent tasks: D5854: Install the scheduler-runner-priority in scheduler nodes.
Jun 11 2021, 11:21 AM · System administration, Scheduling utilities, Save Code Now
ardumont claimed T3367: scheduler: Separate scheduling save code now tasks from the recurrent tasks.
Jun 11 2021, 10:11 AM · System administration, Scheduling utilities, Save Code Now

Jun 10 2021

ardumont closed D5851: docker: Explicit the scheduler runner for save-code-now.
Jun 10 2021, 4:01 PM · Scheduling utilities, Save Code Now
ardumont closed D5826: runner: Separate scheduling tasks with and without priority concerns.
Jun 10 2021, 4:01 PM · Scheduling utilities, Save Code Now
vsellier accepted D5851: docker: Explicit the scheduler runner for save-code-now.

LGTM

Jun 10 2021, 4:00 PM · Scheduling utilities, Save Code Now
ardumont added a comment to D5826: runner: Separate scheduling tasks with and without priority concerns.

still not a big fan of the usage of random in the tests ;), but otherwise, it matches what you explain to me this morning

Jun 10 2021, 3:58 PM · Scheduling utilities, Save Code Now
ardumont added inline comments to D5851: docker: Explicit the scheduler runner for save-code-now.
Jun 10 2021, 3:57 PM · Scheduling utilities, Save Code Now
vsellier accepted D5826: runner: Separate scheduling tasks with and without priority concerns.

LGTM, still not a big fan of the usage of random in the tests ;), but otherwise, it matches what you explain to me this morning

Jun 10 2021, 3:56 PM · Scheduling utilities, Save Code Now
ardumont added projects to D5826: runner: Separate scheduling tasks with and without priority concerns: Save Code Now, Scheduling utilities.
Jun 10 2021, 3:26 PM · Scheduling utilities, Save Code Now
ardumont updated the test plan for D5851: docker: Explicit the scheduler runner for save-code-now.
Jun 10 2021, 3:26 PM · Scheduling utilities, Save Code Now
ardumont added a revision to T3367: scheduler: Separate scheduling save code now tasks from the recurrent tasks: D5851: docker: Explicit the scheduler runner for save-code-now.
Jun 10 2021, 3:22 PM · System administration, Scheduling utilities, Save Code Now
ardumont added a revision to T3367: scheduler: Separate scheduling save code now tasks from the recurrent tasks: D5846: Refactor and extract a get_available_slots utility.
Jun 10 2021, 12:17 PM · System administration, Scheduling utilities, Save Code Now

Jun 8 2021

ardumont added a revision to T3367: scheduler: Separate scheduling save code now tasks from the recurrent tasks: D5826: runner: Separate scheduling tasks with and without priority concerns.
Jun 8 2021, 5:39 PM · System administration, Scheduling utilities, Save Code Now
ardumont edited projects for T3252: Better handling of erroneous origins submitted to save code now, added: System administration; removed System administrators.
Jun 8 2021, 4:53 PM · System administration, Save Code Now, Web app
ardumont edited projects for T3082: Improve Save Code Now handling, added: System administration; removed System administrators.
Jun 8 2021, 4:53 PM · System administration, Save Code Now, meta-task, Roadmap 2021, Web app
ardumont added a comment to T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete.

I guess we could avoid walking on subtrees in that case and just take the most recent
date on the first level of an archive content.

Jun 8 2021, 4:46 PM · Save Code Now
anlambert added a comment to T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete.

Then, if not present, we could fallback as walking the arborescence tree to determine the most
recent date (but i have no idea how long that could take for big archives)

Jun 8 2021, 1:54 PM · Save Code Now
ardumont added a comment to T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete.

Yes, that sounds like something achievable from afar.

Jun 8 2021, 12:43 PM · Save Code Now
anlambert added a comment to T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete.

@ardumont, maybe we could use the timestamps of the files extracted from an archive to compute the author/committer date when not available from the HEAD response ?

Jun 8 2021, 11:48 AM · Save Code Now
ardumont added a comment to T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete.

What could be a decent heuristic to handle this kind of situation?

Jun 8 2021, 10:56 AM · Save Code Now

Jun 7 2021

rdicosmo added a comment to T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete.

Thanks @ardumont for investigating this. The fact that the IA does not provide the LastModified information may make sense for their specific case (it is possible that they do not have kept the LastModified info from the original location).

Jun 7 2021, 11:08 PM · Save Code Now
ardumont renamed T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete from save code now: Failure to ingest new 'archives' type to save code now: Failure to ingest new 'archives' type when head response is incomplete.
Jun 7 2021, 5:06 PM · Save Code Now
ardumont updated the task description for T3367: scheduler: Separate scheduling save code now tasks from the recurrent tasks.
Jun 7 2021, 4:55 PM · System administration, Scheduling utilities, Save Code Now
ardumont updated the task description for T3367: scheduler: Separate scheduling save code now tasks from the recurrent tasks.
Jun 7 2021, 4:54 PM · System administration, Scheduling utilities, Save Code Now
ardumont added a comment to T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete.

I did not realize immediately that the archive urls used for the ingestion (submitted by
the users) are the ones from the internet archive! Neat trick!

Jun 7 2021, 2:52 PM · Save Code Now
ardumont updated the task description for T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete.
Jun 7 2021, 2:40 PM · Save Code Now
ardumont added a comment to T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete.

ah. It's an edge case from the start, nice!

Jun 7 2021, 12:39 PM · Save Code Now
ardumont added a comment to T3213: Enable save code now of software source code archives for specific users.

It's deployed now.

Jun 7 2021, 12:31 PM · Save Code Now, Web app
ardumont triaged T3365: save code now: Failure to ingest new 'archives' type when head response is incomplete as High priority.
Jun 7 2021, 12:29 PM · Save Code Now
ardumont added a comment to T3213: Enable save code now of software source code archives for specific users.

@rdicosmo That's the issue ^ (unrelated to T3361 in the end), the workers were not suscribed to consume from that queue yet.

Jun 7 2021, 12:15 PM · Save Code Now, Web app
ardumont added a comment to T3361: "Save code now" seems to be stuck.

fwiw, the urls mentioned are now browsable.

Jun 7 2021, 12:06 PM · Save Code Now
mwagner added a comment to T3361: "Save code now" seems to be stuck.

Thanks for the explanation. The strange thing here is, that only the repository I requested to be archived are not yet browsable. Other requests (even much newer) are browsable for me

Jun 7 2021, 10:36 AM · Save Code Now
douardda added a project to T1226: Save code now email notification for submitter: Save Code Now.
Jun 7 2021, 10:35 AM · Save Code Now, Web app
douardda added a project to T3257: Template the save code now rejection reason to ease operator workflow: Save Code Now.
Jun 7 2021, 10:34 AM · Save Code Now, Easy hack, Web app
douardda added a project to T3256: Propose reason for rejecting a save code now: Save Code Now.
Jun 7 2021, 10:34 AM · Save Code Now, Easy hack, Web app
douardda added a project to T3252: Better handling of erroneous origins submitted to save code now: Save Code Now.
Jun 7 2021, 10:34 AM · System administration, Save Code Now, Web app
douardda added a project to T3213: Enable save code now of software source code archives for specific users: Save Code Now.
Jun 7 2021, 10:33 AM · Save Code Now, Web app
douardda added a project to T1524: save code now: also add new origins for unknown repos: Save Code Now.
Jun 7 2021, 10:33 AM · Save Code Now, Web app
ardumont added a comment to T3361: "Save code now" seems to be stuck.

If you give it a bit of time, that should eventually be browsable.
Internally, the webapp uses a replicated database.

Jun 7 2021, 10:27 AM · Save Code Now
mwagner added a comment to T3361: "Save code now" seems to be stuck.

Ok. Thanks a lot. In the dashboard most entries now are marked as "succeeded". However, I'm not able to access the recently archived repositories

Jun 7 2021, 10:13 AM · Save Code Now
ardumont added a comment to T3361: "Save code now" seems to be stuck.

Thanks for the bump! It was stuck and not it got unstuck.

Jun 7 2021, 10:05 AM · Save Code Now
zack renamed Save Code Now from SaveCodeNow to Save Code Now.
Jun 7 2021, 9:45 AM
zack triaged T3361: "Save code now" seems to be stuck as High priority.
Jun 7 2021, 9:44 AM · Save Code Now

Jun 2 2021

vsellier closed T3355: Running save code now request are never detected as completed by the webapp as Resolved.
  • The fix was deployed on webapp1 and moma
  • The refresh script was manually launched:
root@webapp1:~# /usr/local/bin/refresh-savecodenow-statuses
Successfully updated 140 save request(s).

The previous requests were correctly refreshed and are now displaying the right status.

Jun 2 2021, 3:14 PM · Save Code Now, Web app
vsellier added a comment to T3355: Running save code now request are never detected as completed by the webapp .

Will be deployed with version v0.0.310 of the webapp (build in progress)

Jun 2 2021, 2:22 PM · Save Code Now, Web app
vsellier added a revision to T3355: Running save code now request are never detected as completed by the webapp : D5810: Update running save origin request status.
Jun 2 2021, 12:07 PM · Save Code Now, Web app
vsellier renamed T3355: Running save code now request are never detected as completed by the webapp from Running save code now request are never finalized to Running save code now request are never detected as completed by the webapp .
Jun 2 2021, 11:58 AM · Save Code Now, Web app
vsellier changed the status of T3355: Running save code now request are never detected as completed by the webapp from Open to Work in Progress.
Jun 2 2021, 11:57 AM · Save Code Now, Web app

May 28 2021

ardumont added a comment to T1481: add metric to monitor "save code now" efficiency.

Now what's missing here (not sure how hard it is) is the mean and max ingestion time
of save code now requests (time between they being accepted and the loader task is
over)

May 28 2021, 11:54 AM · Save Code Now, System administration, Metrics/monitoring

May 27 2021

ardumont closed T3346: save code now: Some entries are missing information after update as Resolved.
May 27 2021, 4:03 PM · Save Code Now
ardumont added a comment to T3346: save code now: Some entries are missing information after update.

Improve the current way of fetching save code now requests to update. Status of those
requests even if updated with their latest status (which is derived out of the
scheduler) will get fetched again for update until the information retrieved out of the
main archive is also configured ("replication lag subsides").

May 27 2021, 4:02 PM · Save Code Now
anlambert closed T3347: Archive loader should have a snapshot append mode as Resolved by committing rDLDBASE584777f3a5d2: package/archive: Add snapshot_append parameter to ArchiveLoader.
May 27 2021, 2:04 PM · Save Code Now, Core Loader
anlambert added a revision to T3347: Archive loader should have a snapshot append mode: D5789: package/archive: Add snapshot_append parameter to ArchiveLoader.
May 27 2021, 11:52 AM · Save Code Now, Core Loader
ardumont added a revision to T3346: save code now: Some entries are missing information after update: D5788: common/origin_save: Update missing information when available.
May 27 2021, 11:35 AM · Save Code Now
ardumont added a comment to T3347: Archive loader should have a snapshot append mode.

Your diff sounds quite enough.

May 27 2021, 9:34 AM · Save Code Now, Core Loader

May 26 2021

anlambert added a comment to T3347: Archive loader should have a snapshot append mode.

Or even directly the archive loader's default behavior (append previously seen branch
from early snapshot/visit). As discussed, I'm wondering whether an archive loader (gnu
or cran [1]) would not benefit from always displaying previously seen branches (whether
they are still present in the current main api we list/visit or not [2]).

May 26 2021, 5:59 PM · Save Code Now, Core Loader
ardumont added a comment to T3347: Archive loader should have a snapshot append mode.

This could be implemented by adding a new option to the loader.

May 26 2021, 5:03 PM · Save Code Now, Core Loader
anlambert triaged T3347: Archive loader should have a snapshot append mode as Normal priority.
May 26 2021, 4:46 PM · Save Code Now, Core Loader
ardumont updated the task description for T3346: save code now: Some entries are missing information after update.
May 26 2021, 4:25 PM · Save Code Now
ardumont updated the task description for T3346: save code now: Some entries are missing information after update.
May 26 2021, 4:23 PM · Save Code Now

May 10 2021

anlambert closed T3272: Authenticated users should be able to browse their save code now requests, a subtask of T3082: Improve Save Code Now handling, as Resolved.
May 10 2021, 11:19 AM · System administration, Save Code Now, meta-task, Roadmap 2021, Web app

May 8 2021

rdicosmo moved T3082: Improve Save Code Now handling from Backlog to Work in progress on the Roadmap 2021 board.
May 8 2021, 11:12 AM · System administration, Save Code Now, meta-task, Roadmap 2021, Web app

May 3 2021

anlambert changed the status of T3272: Authenticated users should be able to browse their save code now requests, a subtask of T3082: Improve Save Code Now handling, from Open to Work in Progress.
May 3 2021, 2:09 PM · System administration, Save Code Now, meta-task, Roadmap 2021, Web app

Apr 29 2021

ardumont closed T3296: save code now detailed task window can display erratic creation date time as Resolved.

Deployment is done.

Apr 29 2021, 3:00 PM · Save Code Now
ardumont closed T3280: Separate save code now request updates from the save code now ui as Resolved.
Apr 29 2021, 11:16 AM · Monitoring, Save Code Now
ardumont added a comment to T3296: save code now detailed task window can display erratic creation date time.

Deployment in progress.

Apr 29 2021, 11:16 AM · Save Code Now
ardumont added a revision to T3280: Separate save code now request updates from the save code now ui: D5641: docker-dev: Refresh save code now statuses periodically.
Apr 29 2021, 10:52 AM · Monitoring, Save Code Now

Apr 28 2021

ardumont changed the status of T3296: save code now detailed task window can display erratic creation date time from Open to Work in Progress.

Needs deployment now.

Apr 28 2021, 4:13 PM · Save Code Now
ardumont claimed T3296: save code now detailed task window can display erratic creation date time.
Apr 28 2021, 4:13 PM · Save Code Now

Apr 27 2021

ardumont added a comment to T3280: Separate save code now request updates from the save code now ui.

I'll keep it open until the docker env is ok as well (see the diff D5615).

Apr 27 2021, 5:38 PM · Monitoring, Save Code Now
ardumont added a revision to T3296: save code now detailed task window can display erratic creation date time: D5624: Fix malformatted date value when null.
Apr 27 2021, 5:29 PM · Save Code Now
ardumont added a comment to T3296: save code now detailed task window can display erratic creation date time.

Well, trying out the following in your browser (console or devtools):

Apr 27 2021, 5:23 PM · Save Code Now
ardumont updated the task description for T3296: save code now detailed task window can display erratic creation date time.
Apr 27 2021, 5:23 PM · Save Code Now
ardumont added a comment to T3280: Separate save code now request updates from the save code now ui.

Deployed (production/staging)

Apr 27 2021, 3:22 PM · Monitoring, Save Code Now
ardumont added a project to T3296: save code now detailed task window can display erratic creation date time: Save Code Now.
Apr 27 2021, 1:59 PM · Save Code Now

Apr 26 2021

ardumont added a revision to T3280: Separate save code now request updates from the save code now ui: D5615: docker: Install save code now refresh status cron.
Apr 26 2021, 6:18 PM · Monitoring, Save Code Now

Apr 23 2021

vlorentz assigned T3082: Improve Save Code Now handling to ardumont.
Apr 23 2021, 4:48 PM · System administration, Save Code Now, meta-task, Roadmap 2021, Web app
ardumont added a comment to T3278: Check older pending save code now requests apparently stuck and reschedule those.

Closing this now as the gist of this is done and the remaining fix is to be dealt with in another task [1].

Apr 23 2021, 1:52 PM · System administration, Save Code Now
ardumont triaged T3285: Dynamize webapp save code now status listing page update as Normal priority.
Apr 23 2021, 1:43 PM · Easy hack, Save Code Now
ardumont added a revision to T3280: Separate save code now request updates from the save code now ui: D5583: Separate save code now status refresh routine from the listing ui.
Apr 23 2021, 10:28 AM · Monitoring, Save Code Now

Apr 22 2021

ardumont updated the task description for T3280: Separate save code now request updates from the save code now ui.
Apr 22 2021, 5:10 PM · Monitoring, Save Code Now
ardumont claimed T3280: Separate save code now request updates from the save code now ui.
Apr 22 2021, 5:07 PM · Monitoring, Save Code Now
ardumont moved T3266: Improve save code now failed/uneventful status reporting from Pending validation to Done on the Roadmap 2021 board.
Apr 22 2021, 4:21 PM · Save Code Now, Roadmap 2021, System administrators, Web app
ardumont closed T3266: Improve save code now failed/uneventful status reporting, a subtask of T3082: Improve Save Code Now handling, as Resolved.
Apr 22 2021, 4:21 PM · System administration, Save Code Now, meta-task, Roadmap 2021, Web app
ardumont closed T3266: Improve save code now failed/uneventful status reporting as Resolved.

Deployed.

Apr 22 2021, 4:20 PM · Save Code Now, Roadmap 2021, System administrators, Web app
anlambert added a revision to T3266: Improve save code now failed/uneventful status reporting: D5575: tests/api/origin_save: Explicitely populate SaveAuthorizedOrigin table.
Apr 22 2021, 3:32 PM · Save Code Now, Roadmap 2021, System administrators, Web app
ardumont added a comment to T3266: Improve save code now failed/uneventful status reporting.

Trying to bisect the issue and failed (multiple dimensions got the best of me, master ok, debian build in stable ko... i need to improve my tooling there).

Apr 22 2021, 12:37 PM · Save Code Now, Roadmap 2021, System administrators, Web app
ardumont added a comment to T3266: Improve save code now failed/uneventful status reporting.

So, the debian/unstable broke which is fixed now [1] (it wasd missing one new deps to
test the migration part).

Apr 22 2021, 10:19 AM · Save Code Now, Roadmap 2021, System administrators, Web app
ardumont moved T3266: Improve save code now failed/uneventful status reporting from Backlog to Pending validation on the Roadmap 2021 board.
Apr 22 2021, 9:20 AM · Save Code Now, Roadmap 2021, System administrators, Web app

Apr 21 2021

ardumont changed the status of T3266: Improve save code now failed/uneventful status reporting, a subtask of T3082: Improve Save Code Now handling, from Open to Work in Progress.
Apr 21 2021, 7:08 PM · System administration, Save Code Now, meta-task, Roadmap 2021, Web app
ardumont changed the status of T3266: Improve save code now failed/uneventful status reporting from Open to Work in Progress.
Apr 21 2021, 7:08 PM · Save Code Now, Roadmap 2021, System administrators, Web app
ardumont added a comment to T3266: Improve save code now failed/uneventful status reporting.

Landed. Remains to deploy. I'll attend to that tomorrow.

Apr 21 2021, 7:08 PM · Save Code Now, Roadmap 2021, System administrators, Web app
ardumont moved T3278: Check older pending save code now requests apparently stuck and reschedule those from deployed/landed/monitoring to done on the System administration board.
Apr 21 2021, 6:57 PM · System administration, Save Code Now
ardumont closed T2117: Save Code Now: End to End monitoring, a subtask of T3082: Improve Save Code Now handling, as Resolved.
Apr 21 2021, 6:57 PM · System administration, Save Code Now, meta-task, Roadmap 2021, Web app