Lazy loading for content objects (stored as files on the filesystem) gives us a choice where to put the cursor between the two ends of the eternal tradeoff between:

Jun 25 2020, 4:15 PM · Core Loader

olasd added a project to T2466: Re design the lazy loading feature for Content objects: Core Loader.

Jun 25 2020, 4:07 PM · Core Loader

May 29 2020

ardumont closed T2428: Unstuck failing loaders in staging as Resolved.

May 29 2020, 5:38 PM · Staging environment, Core Loader

ardumont added a comment to T2428: Unstuck failing loaders in staging.

Deployed.

May 29 2020, 5:38 PM · Staging environment, Core Loader

ardumont closed D3196: package.loader: Check for snapshot id prior to reifying model object.

May 29 2020, 3:59 PM · Core Loader

swh-public-ci added a comment to D3196: package.loader: Check for snapshot id prior to reifying model object.

Build is green

May 29 2020, 3:58 PM · Core Loader

ardumont updated the diff for D3196: package.loader: Check for snapshot id prior to reifying model object.

Rebase on latest master

May 29 2020, 3:57 PM · Core Loader

swh-public-ci added a comment to D3196: package.loader: Check for snapshot id prior to reifying model object.

Build is green

May 29 2020, 1:24 PM · Core Loader

ardumont retitled D3196: package.loader: Check for snapshot id prior to reifying model object from package.loader: Check for snapshot id prior to reify the model object to package.loader: Check for snapshot id prior to reifying model object.

May 29 2020, 1:23 PM · Core Loader

swh-public-ci added a comment to D3196: package.loader: Check for snapshot id prior to reifying model object.

Build is green

May 29 2020, 1:23 PM · Core Loader

ardumont updated the diff for D3196: package.loader: Check for snapshot id prior to reifying model object.

Rework commit message

May 29 2020, 1:23 PM · Core Loader

ardumont updated the summary of D3196: package.loader: Check for snapshot id prior to reifying model object.

May 29 2020, 1:22 PM · Core Loader

ardumont updated the diff for D3196: package.loader: Check for snapshot id prior to reifying model object.

Adapt according to review
Align git commit with diff description

May 29 2020, 1:21 PM · Core Loader

olasd accepted D3196: package.loader: Check for snapshot id prior to reifying model object.

Please rename the snap variable to something meaningful :)

May 29 2020, 12:52 PM · Core Loader

ardumont updated the summary of D3196: package.loader: Check for snapshot id prior to reifying model object.

May 29 2020, 12:44 PM · Core Loader

ardumont changed the status of T2428: Unstuck failing loaders in staging from Open to Work in Progress.

May 29 2020, 12:40 PM · Staging environment, Core Loader

ardumont updated the summary of D3196: package.loader: Check for snapshot id prior to reifying model object.

May 29 2020, 12:31 PM · Core Loader

ardumont added projects to T2428: Unstuck failing loaders in staging: Core Loader, Staging environment.

May 29 2020, 12:28 PM · Staging environment, Core Loader

ardumont removed a project from D3196: package.loader: Check for snapshot id prior to reifying model object: Staging environment.

May 29 2020, 12:28 PM · Core Loader

ardumont added projects to D3196: package.loader: Check for snapshot id prior to reifying model object: Core Loader, Staging environment.

May 29 2020, 12:28 PM · Core Loader

Apr 22 2020

ardumont changed the status of T2371: nixguix: fails to use previous visit snapshot from Open to Work in Progress.

Apr 22 2020, 12:17 PM · Package Loader

ardumont added a comment to T2371: nixguix: fails to use previous visit snapshot.

tl; dr: That issue must be invalid. I'm ensuring it is. I've triggered the
necessary checks and waiting for the conclusion.

Apr 22 2020, 12:16 PM · Package Loader

ardumont added a comment to T2371: nixguix: fails to use previous visit snapshot.

Note: the snapshot is a brand new one, no shared state with a prior one.

Apr 22 2020, 11:58 AM · Package Loader

ardumont added a comment to T2371: nixguix: fails to use previous visit snapshot.

Something is off though.

Apr 22 2020, 11:28 AM · Package Loader

ardumont updated the task description for T2371: nixguix: fails to use previous visit snapshot.

Apr 22 2020, 11:27 AM · Package Loader

ardumont removed a project from T1991: Implement a Guix/Nix loader: Lister.

Apr 22 2020, 10:50 AM · Package Loader

ardumont triaged T2371: nixguix: fails to use previous visit snapshot as Normal priority.

Apr 22 2020, 10:49 AM · Package Loader

Apr 20 2020

ardumont added a subtask for T1991: Implement a Guix/Nix loader: T2357: nixguix: fails to finish as downloading artifacts step hangs.

Apr 20 2020, 9:44 AM · Package Loader

ardumont added a subtask for T1991: Implement a Guix/Nix loader: T2352: nixguix: Fails to finish as it's stuck in a loop up to memory error.

Apr 20 2020, 9:43 AM · Package Loader

Apr 14 2020

ardumont updated the task description for T2357: nixguix: fails to finish as downloading artifacts step hangs.

Apr 14 2020, 2:29 PM · Package Loader

ardumont added a revision to T2352: nixguix: Fails to finish as it's stuck in a loop up to memory error: D3012: retry: Make content_add endpoints maximize content writes to storage.

Apr 14 2020, 10:28 AM · Package Loader, Storage manager

ardumont added a revision to T2352: nixguix: Fails to finish as it's stuck in a loop up to memory error: D2966: storage.buffer: Add a new clear_buffers operation for the buffer proxy.

Apr 14 2020, 10:25 AM · Package Loader, Storage manager

ardumont added a revision to T2352: nixguix: Fails to finish as it's stuck in a loop up to memory error: D2973: package.loader: Insert consistently data to storage, clear buffer proxy state in case of error when skipping artifact.

Apr 14 2020, 10:25 AM · Package Loader, Storage manager

ardumont added a revision to T2352: nixguix: Fails to finish as it's stuck in a loop up to memory error: D3014: package.loader: Flush and clear regularly internal proxy storage states.

Apr 14 2020, 10:21 AM · Package Loader, Storage manager

Apr 11 2020

ardumont renamed T2357: nixguix: fails to finish as downloading artifacts step hangs from nixguix: fails to finish as downloading artifacts step to nixguix: fails to finish as downloading artifacts step hangs.

Apr 11 2020, 4:50 PM · Package Loader

ardumont renamed T2357: nixguix: fails to finish as downloading artifacts step hangs from nixguix: fails to finish as download hanging to nixguix: fails to finish as downloading artifacts step.

Apr 11 2020, 12:58 PM · Package Loader

ardumont updated the task description for T2357: nixguix: fails to finish as downloading artifacts step hangs.

Apr 11 2020, 12:00 PM · Package Loader

ardumont triaged T2357: nixguix: fails to finish as downloading artifacts step hangs as Normal priority.

Apr 11 2020, 11:58 AM · Package Loader

ardumont created T2357: nixguix: fails to finish as downloading artifacts step hangs.

Apr 11 2020, 11:58 AM · Package Loader

ardumont renamed T2352: nixguix: Fails to finish as it's stuck in a loop up to memory error from nixguix: Fails to finish to nixguix: Fails to finish as it's stuck in a loop up to memory error.

Apr 11 2020, 11:51 AM · Package Loader, Storage manager

Apr 9 2020

ardumont added a comment to T2352: nixguix: Fails to finish as it's stuck in a loop up to memory error.

In the mean time (pending reviews), a new load was triggered without the proxy storage and all went fine.

Apr 9 2020, 9:39 AM · Package Loader, Storage manager

Apr 7 2020

ardumont added a comment to T2352: nixguix: Fails to finish as it's stuck in a loop up to memory error.

Right now, heading for 2. for now as the solution for 3. is still a pending question [2]

Apr 7 2020, 3:05 PM · Package Loader, Storage manager

ardumont updated the task description for T2352: nixguix: Fails to finish as it's stuck in a loop up to memory error.

Apr 7 2020, 2:55 PM · Package Loader, Storage manager

ardumont added projects to T2352: nixguix: Fails to finish as it's stuck in a loop up to memory error: Core Loader, Storage manager.

Apr 7 2020, 11:56 AM · Package Loader, Storage manager

Apr 2 2020

ardumont updated the task description for T1991: Implement a Guix/Nix loader.

Apr 2 2020, 4:34 PM · Package Loader

Mar 25 2020

zack triaged T2334: Manage failing SSL requests as Normal priority.

Mar 25 2020, 11:03 AM · Core Loader

Mar 24 2020

lewo created T2334: Manage failing SSL requests.

Mar 24 2020, 3:52 PM · Core Loader

Mar 20 2020

ardumont created P620 staging: functional loader crash .

Mar 20 2020, 11:16 AM · Core Loader

Mar 19 2020

lewo updated the task description for T1991: Implement a Guix/Nix loader.

Mar 19 2020, 11:09 AM · Package Loader

Mar 18 2020

ardumont added a comment to T1991: Implement a Guix/Nix loader.

First runs triggered on staging, errors will be reported in sentry [1]

Mar 18 2020, 4:50 PM · Package Loader

ardumont added a comment to P618 functional loader through cli.

Related to T1991

Mar 18 2020, 4:49 PM · Core Loader

ardumont added a comment to P618 functional loader through cli.

well, still no message in sentry.
I'll stop that script and use the proper scheduler stanza on staging.

Mar 18 2020, 4:24 PM · Core Loader

ardumont added a comment to P618 functional loader through cli.

well the export instructions were wrong. Fixed!

Mar 18 2020, 2:49 PM · Core Loader

ardumont edited P618 functional loader through cli.

Mar 18 2020, 2:47 PM · Core Loader

ardumont added a comment to P618 functional loader through cli.

some error not reported in sentry:

DEBUG:swh.loader.package.utils:extrinsic_metadata                                                                                                                                                                                     [84/8086]
ERROR:swh.loader.package.loader:Fail to uncompress https://github.com/lihaoyi/Ammonite/releases/download/1.6.9/2.12-1.6.9
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/swh/core/tarball.py", line 72, in uncompress
    shutil.unpack_archive(tarpath, extract_dir=dest)
  File "/usr/lib/python3.7/shutil.py", line 995, in unpack_archive
    raise ReadError("Unknown archive format '{0}'".format(filename))
shutil.ReadError: Unknown archive format '/tmp/tmpmnd54wj6/2.12-1.6.9'

Mar 18 2020, 2:46 PM · Core Loader

ardumont added a comment to P618 functional loader through cli.

so sentry see the issues...

Mar 18 2020, 2:34 PM · Core Loader

ardumont created P618 functional loader through cli.

Mar 18 2020, 2:31 PM · Core Loader

Mar 10 2020

lewo updated the task description for T1991: Implement a Guix/Nix loader.

Mar 10 2020, 6:49 PM · Package Loader

lewo updated the task description for T1991: Implement a Guix/Nix loader.

Mar 10 2020, 6:42 PM · Package Loader

Mar 5 2020

ardumont assigned T1991: Implement a Guix/Nix loader to lewo.

Mar 5 2020, 7:30 PM · Package Loader

ardumont renamed T1991: Implement a Guix/Nix loader from Implement a Guix/GuixSD/Nix/NixOS loader to Implement a Guix/Nix loader.

Mar 5 2020, 7:30 PM · Package Loader

ardumont added a comment to T1991: Implement a Guix/Nix loader.

Implementation of a lister started at D2025.

Mar 5 2020, 7:30 PM · Package Loader

ardumont renamed T1991: Implement a Guix/Nix loader from Implement a Guix/GuixSD/Nix/NixOS lister to Implement a Guix/GuixSD/Nix/NixOS loader.

Mar 5 2020, 7:28 PM · Package Loader

Feb 6 2020

ardumont closed T1626: loader-core: Unify counter with the storage's results as Resolved.

It's been done and removed... so closed.

Feb 6 2020, 7:12 PM · Core Loader

Jan 22 2020

ardumont renamed T2241: lister/loader: Ingest archived artifacts from cran mirror from lister/loader: Ingest all known artifacts from cran mirror to lister/loader: Ingest archived artifacts from cran mirror.

Jan 22 2020, 10:46 AM · Core Loader, Lister, Origin-CRAN

Jan 21 2020

ardumont added a comment to T2241: lister/loader: Ingest archived artifacts from cran mirror.

This should take care of [1]

Jan 21 2020, 12:18 PM · Core Loader, Lister, Origin-CRAN

ardumont updated the task description for T2241: lister/loader: Ingest archived artifacts from cran mirror.

Jan 21 2020, 11:52 AM · Core Loader, Lister, Origin-CRAN

ardumont triaged T2241: lister/loader: Ingest archived artifacts from cran mirror as Normal priority.

Jan 21 2020, 11:52 AM · Core Loader, Lister, Origin-CRAN

Jan 6 2020

ardumont closed D2495: package.loader: Fail loading early when failing to create an origin/visit.

Jan 6 2020, 4:37 PM · Core Loader

swh-public-ci added a comment to D2495: package.loader: Fail loading early when failing to create an origin/visit.

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/305/ for more details.

Jan 6 2020, 3:39 PM · Core Loader

ardumont added inline comments to D2495: package.loader: Fail loading early when failing to create an origin/visit.

Jan 6 2020, 3:38 PM · Core Loader

ardumont updated the diff for D2495: package.loader: Fail loading early when failing to create an origin/visit.

Fix error message
plug to master branch

Jan 6 2020, 3:37 PM · Core Loader

douardda accepted D2495: package.loader: Fail loading early when failing to create an origin/visit.

Jan 6 2020, 3:34 PM · Core Loader

swh-public-ci added a comment to D2495: package.loader: Fail loading early when failing to create an origin/visit.

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/304/ for more details.

Jan 6 2020, 3:13 PM · Core Loader

ardumont updated the diff for D2495: package.loader: Fail loading early when failing to create an origin/visit.

Rebase on latest master

Jan 6 2020, 3:11 PM · Core Loader

ardumont updated the summary of D2495: package.loader: Fail loading early when failing to create an origin/visit.

Jan 6 2020, 3:06 PM · Core Loader

Dec 20 2019

ardumont closed D2486: package.utils: Drop unneeded hashes from download computation.

Dec 20 2019, 2:21 PM · Debian loader, Core Loader

swh-public-ci added a comment to D2486: package.utils: Drop unneeded hashes from download computation.

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/297/ for more details.

Dec 20 2019, 2:21 PM · Debian loader, Core Loader

ardumont updated the diff for D2486: package.utils: Drop unneeded hashes from download computation.

Fix unneeded dict comprehension

Dec 20 2019, 2:19 PM · Debian loader, Core Loader

olasd accepted D2486: package.utils: Drop unneeded hashes from download computation.

Just one tiny inline change, but looks good, thanks!

Dec 20 2019, 2:17 PM · Debian loader, Core Loader

swh-public-ci added a comment to D2486: package.utils: Drop unneeded hashes from download computation.

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/296/ for more details.

Dec 20 2019, 2:17 PM · Debian loader, Core Loader

ardumont updated the diff for D2486: package.utils: Drop unneeded hashes from download computation.

Drop unneeded instructions about response.headers

Dec 20 2019, 2:15 PM · Debian loader, Core Loader

ardumont updated the summary of D2486: package.utils: Drop unneeded hashes from download computation.

Dec 20 2019, 2:05 PM · Debian loader, Core Loader

swh-public-ci added a comment to D2486: package.utils: Drop unneeded hashes from download computation.

Build is green
See https://jenkins.softwareheritage.org/job/DLDBASE/job/tox/295/ for more details.

Dec 20 2019, 2:04 PM · Debian loader, Core Loader

ardumont retitled D2486: package.utils: Drop unneeded hashes from download computation from debian.loader: Fix wrong request content-length received to package.utils: Drop unneeded hashes from download computation.

Dec 20 2019, 2:03 PM · Debian loader, Core Loader