loader: Rename loaders to more meaningful names
ClosedPublic
Actions

Authored by ardumont on Dec 17 2018, 2:10 PM.

Details

Reviewers

zack
olasd
vlorentz

Group Reviewers

Reviewers

Commits

rDLDG53f484f6236f: Update configuration files with more meaningful names
rDLDG7f673b52697f: test_loader: Add missing copyright header
rDLDG9e4f2ce64da4: git.loader: Rename BulkUpdater to GitLoader
rDLDGac28890a3fae: loader.git: Rename loader to from_disk
rDLDGee6e14fae1d3: git.updater: Rename to git.loader
rDLDG2219999a697e: loader.git.from_disk: Rename GitLoader to GitLoaderFromDisk
rDLDG2d239694b774: loader.git.updater: Drop redundant SWH prefix name

Summary

git.updater: Rename to git.loader
loader.git.from_disk: Rename GitLoader to GitLoaderFromDisk
test_loader: Add missing copyright header
git.loader: Rename BulkUpdater to GitLoader
Update configuration files to be named appropriately
Update doc
Adaptations of the expected configuration files for the least surprise effect on the users.

This does not deal yet with other impacted modules (main
documentation, swh-site, etc...).

Test Plan

tox

And/or from swh-docker-dev to start the relevant services:

docker-compose up

then, from remote test:

python3 -m swh.loader.git.loader --origin-url https://github.com/SoftwareHeritage/swh-storage.git

2018-12-17 14:01:19,948 12740 Creating git origin for https://github.com/SoftwareHeritage/swh-storage.git
2018-12-17 14:01:19,951 12740 Starting new HTTP connection (1): localhost:5002
2018-12-17 14:01:19,958 12740 http://localhost:5002 "POST /origin/add HTTP/1.1" 200 1
2018-12-17 14:01:19,959 12740 Done creating git origin for https://github.com/SoftwareHeritage/swh-storage.git
2018-12-17 14:01:19,959 12740 Creating origin_visit for origin 1 at time 2018-12-17 13:01:19.948234+00:00
2018-12-17 14:01:19,961 12740 Resetting dropped connection: localhost
2018-12-17 14:01:19,965 12740 http://localhost:5002 "POST /origin/visit/add HTTP/1.1" 200 16
2018-12-17 14:01:19,965 12740 Done Creating origin_visit for origin 1 at time 2018-12-17 13:01:19.948234+00:00
2018-12-17 14:01:19,967 12740 Resetting dropped connection: localhost
2018-12-17 14:01:19,971 12740 http://localhost:5002 "POST /fetch_history/start HTTP/1.1" 200 1
2018-12-17 14:01:19,977 12740 Resetting dropped connection: localhost
2018-12-17 14:01:19,982 12740 http://localhost:5002 "POST /snapshot/latest HTTP/1.1" 200 1
2018-12-17 14:01:19,982 12740 Starting new HTTPS connection (1): github.com:443
2018-12-17 14:01:20,352 12740 https://github.com:443 "GET /SoftwareHeritage/swh-storage.git/info/refs?service=git-upload-pack HTTP/1.1" 200 None
2018-12-17 14:01:20,460 12740 Resetting dropped connection: localhost
2018-12-17 14:01:20,466 12740 http://localhost:5002 "POST /object/find_by_sha1_git HTTP/1.1" 200 5776
2018-12-17 14:01:20,775 12740 https://github.com:443 "POST /SoftwareHeritage/swh-storage.git/git-upload-pack HTTP/1.1" 200 None
Enumerating objects: 241, done.
Counting objects: 100% (241/241), done.
Compressing objects: 100% (145/145), done.
Total 7480 (delta 130), reused 181 (delta 84), pack-reused 7239
2018-12-17 14:01:21,464 12740 Resetting dropped connection: localhost
2018-12-17 14:01:21,469 12740 http://localhost:5002 "POST /object/find_by_sha1_git HTTP/1.1" 200 5776
2018-12-17 14:01:21,470 12740 Listed 137 refs for repo https://github.com/SoftwareHeritage/swh-storage.git
2018-12-17 14:01:22,276 12740 Resetting dropped connection: localhost
2018-12-17 14:01:22,314 12740 http://localhost:5002 "POST /content/missing HTTP/1.1" 200 53375
2018-12-17 14:01:22,830 12740 Sending 2426 contents
2018-12-17 14:01:22,852 12740 Resetting dropped connection: localhost
2018-12-17 14:01:25,424 12740 http://localhost:5002 "POST /content/add HTTP/1.1" 200 1
2018-12-17 14:01:25,425 12740 Done sending 2426 contents
2018-12-17 14:01:25,430 12740 Resetting dropped connection: localhost
2018-12-17 14:01:25,456 12740 http://localhost:5002 "POST /directory/missing HTTP/1.1" 200 84637
2018-12-17 14:01:25,754 12740 Sending 3847 directories
2018-12-17 14:01:25,767 12740 Resetting dropped connection: localhost
2018-12-17 14:01:26,275 12740 http://localhost:5002 "POST /directory/add HTTP/1.1" 200 1
2018-12-17 14:01:26,276 12740 Done sending 3847 directories
2018-12-17 14:01:26,284 12740 Resetting dropped connection: localhost
2018-12-17 14:01:26,292 12740 http://localhost:5002 "POST /revision/missing HTTP/1.1" 200 23961
2018-12-17 14:01:26,523 12740 Sending 1089 revisions
2018-12-17 14:01:26,526 12740 Resetting dropped connection: localhost
2018-12-17 14:01:27,597 12740 http://localhost:5002 "POST /revision/add HTTP/1.1" 200 1
2018-12-17 14:01:27,597 12740 Done sending 1089 revisions
2018-12-17 14:01:27,601 12740 Resetting dropped connection: localhost
2018-12-17 14:01:27,605 12740 http://localhost:5002 "POST /release/missing HTTP/1.1" 200 2599
2018-12-17 14:01:27,827 12740 Sending 118 releases
2018-12-17 14:01:27,829 12740 Resetting dropped connection: localhost
2018-12-17 14:01:27,847 12740 http://localhost:5002 "POST /release/add HTTP/1.1" 200 1
2018-12-17 14:01:27,847 12740 Done sending 118 releases
2018-12-17 14:01:27,849 12740 Resetting dropped connection: localhost
2018-12-17 14:01:27,860 12740 http://localhost:5002 "POST /snapshot/add HTTP/1.1" 200 1
2018-12-17 14:01:27,862 12740 Resetting dropped connection: localhost
2018-12-17 14:01:27,865 12740 http://localhost:5002 "POST /fetch_history/end HTTP/1.1" 200 1
2018-12-17 14:01:27,866 12740 Updating origin_visit for origin 1 with status full
2018-12-17 14:01:27,868 12740 Resetting dropped connection: localhost
2018-12-17 14:01:27,870 12740 http://localhost:5002 "POST /origin/visit/update HTTP/1.1" 200 1
2018-12-17 14:01:27,871 12740 Done updating origin_visit for origin 1 with status full

from disk test:

python3 -m swh.loader.git.from_disk --origin-url swh-loader-git --git-directory $(pwd)
2018-12-17 14:02:03,006 13195 Creating git origin for swh-loader-git
2018-12-17 14:02:03,008 13195 Done creating git origin for swh-loader-git
2018-12-17 14:02:03,008 13195 Creating origin_visit for origin 6 at time 2018-12-17 13:02:03.003626+00:00
2018-12-17 14:02:03,009 13195 Done Creating origin_visit for origin 6 at time 2018-12-17 13:02:03.003626+00:00
2018-12-17 14:02:04,044 13195 Sending 1000 contents
2018-12-17 14:02:04,345 13195 Done sending 1000 contents
2018-12-17 14:02:04,435 13195 Sending 1000 contents
2018-12-17 14:02:04,690 13195 Done sending 1000 contents
2018-12-17 14:02:04,717 13195 Sending 292 contents
2018-12-17 14:02:04,800 13195 Done sending 292 contents
2018-12-17 14:02:04,975 13195 Sending 2500 directories
2018-12-17 14:02:05,222 13195 Done sending 2500 directories
2018-12-17 14:02:05,304 13195 Sending 1375 directories
2018-12-17 14:02:05,429 13195 Done sending 1375 directories
2018-12-17 14:02:05,532 13195 Sending 1000 revisions
2018-12-17 14:02:05,896 13195 Done sending 1000 revisions
2018-12-17 14:02:05,932 13195 Sending 465 revisions
2018-12-17 14:02:05,981 13195 Done sending 465 revisions
2018-12-17 14:02:05,990 13195 Sending 89 releases
2018-12-17 14:02:06,001 13195 Done sending 89 releases
2018-12-17 14:02:06,029 13195 Updating origin_visit for origin 6 with status full
2018-12-17 14:02:06,029 13195 Done updating origin_visit for origin 6 with status full

Diff Detail

Repository

rDLDG Git loader

Branch

master

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 3082
Build 3940: tox-on-jenkins	Jenkins
Build 3939: arc lint + arc unit

Event Timeline

ardumont created this revision.Dec 17 2018, 2:10 PM

Build is green
See https://jenkins.softwareheritage.org/job/DLDG/job/tox/67/ for more details.

Harbormaster completed remote builds in B3082: Diff 2653.Dec 17 2018, 2:12 PM

ardumont edited the test plan for this revision. (Show Details)Dec 17 2018, 2:15 PM

ardumont edited the test plan for this revision. (Show Details)

zack retitled this revision from loader: Rename loaders to more expected names to loader: Rename loaders to more meaningful names.Dec 17 2018, 2:20 PM

vlorentz accepted this revision.Dec 17 2018, 2:24 PM

This revision is now accepted and ready to land.Dec 17 2018, 2:24 PM