Page MenuHomeSoftware Heritage

Recreate DOBJS commit history
ClosedPublic

Authored by qcampos on Jun 27 2016, 1:37 PM.

Details

Reviewers
zack
ardumont
Group Reviewers
Reviewers
Maniphest Tasks
T460: Extract the objstorage component from storage module to an independent module
Commits
rDOBJS3662fc28456c: Revert "Client to backup contents to an objstorage server instance"
rDOBJS84bc789758a6: Revert "Client to backup contents to an objstorage server instance"
rDOBJS3226739f3e9e: exc: Create a new exception type specific to the object storage. Also, add…
rDOBJS3ed1894784b1: Move all files to the correct repository swh/objstorage
rDOBJS8e79ecc58f5f: Import scripts from swh.storage
rDOBJS24993f350253: Move tests to the correct location
rDOBJS89bf2a98d033: imports: correction due to the module extraction
rDOBJS601d984e0568: Move the swh.storage.checker module into swh.objstorage.checker as this…
rDOBJS3effd252a53d: api: Move the http api-related code from storage to objstorage to keep the…
rDOBJS3b808772841b: objstorage.multiplexer: add some use case example to the…
rDOBJS78e4f662baf5: Update debian packaging config
rDOBJS38eac8bb167c: objstorage.multiplexer: add tests for the multiplexer object storage
rDOBJSa386b9d90ab8: objstorage: move the constant ID_HASH_ALGO to ObjStorage
rDOBJS8501d47df55a: Open (and fix) swh.storage.storage.revision_log_by
rDOBJS5a145eeccda0: objstorage.multiplexer.filter: Implement the multiplexer filter mecanism
rDOBJSf99f8197a018: objstorage.multiplexer: Implement the object storage multiplexer
rDOBJSaf20300a3870: objstorage.multiplexer.filter: add tests for the filters implementations
rDOBJSfce4177991af: objstorage.multiplexer.filter: add a way to easily create filter chaining
rDOBJS662fef2e7175: test_archiver: remove debug prints
rDOBJS946073fc88e2: Create a base API that define the objstorage behavior
rDOBJSa157a6d47970: Update the content integrity checker in order to follow objstorage changes
rDOBJS3a05041faa2f: Add an implementation of the object storage api and a test class
rDOBJSafc53da694eb: archiver: change objstorage initialization into the archiver
rDOBJSf14c963076b2: Fix an object id encoding into a test of objstorage api
rDOBJS9deabcbdd67b: Create a base API that define the objstorage behavior
rDOBJSc3f12b0de6c3: Improve content integrity checker's test coverage
rDOBJSb86631ce7af0: Add an implementation of the object storage api and a test class
rDOBJS4f8327f43ccf: objstorage: PathslicingObjStorage now allow arbitrary slicing
rDOBJS38f3b7eec82f: objstorage.api: Change instantiation of the object storage
rDOBJS8aa04719df7c: objstorage: PathslicingObjStorage now allow arbitrary slicing
rDOBJS43c5cdb25afe: Change the objstorage http server in order to follow the changes of the local…
rDOBJS639096443f98: checker: change the objstorage initialization of the checker
rDOBJS802832c36dba: converters: update to reflect decoding logic moved to swh.core.utils
rDOBJS89a115c74098: The checker now have the possibility to have multiple backup servers
rDOBJS5f9bc2686a0a: Change the objstorage http server in order to follow the changes of the local…
rDOBJS46f6641ed963: objstorage.api: Change instantiation of the object storage
rDOBJSe15367114f58: Add run_from_webserver function for objstorage api server
rDOBJS79ca04e1e05b: Also, add a get_random_contents access to the remote API
rDOBJS900e5d166ae3: Add some methods to the object storage in order to allow a content integrity…
rDOBJS4156533a6193: Add some methods to the object storage in order to allow a content integrity…
rDOBJS1ec84256ea10: Add some tests for the content integrity checker
rDOBJS689dcf66197a: Command line launch improvements
rDOBJSab064e3f4c80: Add unique identifier message on default api server route endpoints
rDOBJS547c219872c1: Also, add a get_random_contents access to the remote API
rDOBJS6376a2aeffdc: Change test attribute in order to skip some tests
rDOBJS553df58f2514: Add synchronous first implementation of the archiver
rDOBJSc11a89f44c23: Make the worker asynchronous optional
rDOBJSe0667961d26e: Improve test coverage of the archiver
rDOBJS5605f85df596: storage: add entity_get_one to retrieve a single entity
rDOBJS21b7717ec6fa: sql/swh-func: drop temporary entity-related tables on commit
rDOBJSd744acd7cb7e: Change the objstorage file to a module
rDOBJScc21d001c60d: Add tests for the remote objstorage API and share some code with storage.api…
rDOBJS43e8918c74fa: db: allow passing UUIDs back and forth with the database
rDOBJS355e30dabdbd: sql/swh-schema: add a fullname field to person
rDOBJSf4f7709977aa: Add an http API for objstorage and refactor the shared code with storage.api
rDOBJS3bb3c4d9097c: sql/swh-schema: rename entity{_history}.doap to metadata
rDOBJSd502912ed606: swh-schema: fold the entity.lister column in entity.lister_metadata
rDOBJS59ca69809b00: revision_log from multiple root revisions
rDOBJS349d282882a9: swh.storage: Allow storage of extra git headers in revisions
rDOBJS35a49a0b32a0: Open entity_get api
rDOBJS807d436a679f: Open content_missing_per_sha1
rDOBJS7e1232514210: Open occurrence_get(origin_id) to retrieve latest occurrences per origin
rDOBJS24a4163afadd: Deal with occurrence_history.branch, release.name as bytes
rDOBJSf708c2504046: Rename directory_get to directory_ls
rDOBJS728ddf49f266: Let the person's id flow
rDOBJSb818e0d6a837: Open directory_get to retrieve information on directory by id
rDOBJSc1f45f71f020: sql/swh-schema: Handle negative UTC offsets in revision and release dates
rDOBJS23a48526e4d2: storage: add new method object_find_by_sha1_git
rDOBJS4fd2682a1e46: revision_get_by: branch name filtering is optional
rDOBJSf10c61f16897: Wrap miscellaneous psycopg2 failures for client in StorageBackendError
rDOBJS42753cfc6c9b: storage.py: add a shortlog method only returning revision ids and parents
rDOBJS7867dab133fd: Open release_get_by to retrieve a release by origin.
rDOBJS8dfe8a87ad56: sql/upgrades/039: add 038→039 upgrade script
rDOBJSce73d5f157ba: sql: make occurrences and revisions use type agnostic targets
rDOBJS273cb6f0594f: sql: Use a new schema for occurrences
rDOBJS6a333ea4d4be: test_storage: move backend-agnostic tests to the base class
rDOBJS73043bc5e3fa: Open directory_entry_get_by_path api
rDOBJS945abad6cb68: List direct parents and children sha1s from a given revision
rDOBJS35f2b0a65e08: Open revision_get_by to retrieve a revision by occurrence criterion filtering
rDOBJSa6e11fed9ed4: swh.storage: new release schema
rDOBJSbea6691b008d: revision_log is now 'parents' aware
rDOBJS272df815b052: Unify content_find api to return dict instead of tuples
rDOBJSeb8cc20b9268: Open revision_log api
rDOBJSb232df4ae41d: Update validity timestamp filtering
rDOBJS221b1db661e9: swh.storage: use new schema for revisions
rDOBJS7d7ad16ce76f: Revert "List direct parents and children sha1s from a given revision"
rDOBJS6d7a50f91123: sql/swh-func: get_revision_by should only return one revision
rDOBJSc4f1f6e1e382: Simplify test
rDOBJS3755a19432c4: Remove redundant content_exist and use content_find which returns None if not…
rDOBJS835c05f494df: revision_log: Implement a limit on the number of revisions listed
rDOBJS646eb1eae229: Retrieve origin information
rDOBJS6c11f3247ea9: Fix [null...] parents list
rDOBJSbde53fa5665e: Fix db.origin_add returns tuple not dict
rDOBJSad8cedbfcb10: Refactor - Simplify and add test
rDOBJS126e1f3bfdcd: Open person_get api to retrieve person information
rDOBJS53feebafa3b6: fix
rDOBJS57ff8af10994: Open release_get : [sha1] -> [Release]
rDOBJS1a65f77137d5: Add endpoints to list entities from the lister metadata
rDOBJSb9b39bf57392: Unify release_get's implementation with revision_get
rDOBJS9f8531e487eb: Add origin lookup per id
rDOBJS5f679c4a5545: Add endpoints to create entities
rDOBJS617884c8cfa8: sql: Add contents to directory_walk_one
rDOBJS38197324d80a: objstorage: create files as mode 0o644
rDOBJS8a9dec7a43ae: new SQL function: swh_stat_counters()
rDOBJS550a19740b8c: storage: add revision_get
rDOBJSfa2193f32c88: tests: fix flake8
rDOBJS4d1598d27528: Add metadata field to revision table
rDOBJS4d82f9cbf56d: test_storage: use proper types for revisions
rDOBJS61bbbe1209ea: objstorage: add test for directory modes
rDOBJS80925cc93960: Write/read `synthetic` column in revision/release
rDOBJSa1795a9b8e9e: storage: Add entry points for fetch_history
rDOBJSefe2dc312611: sql: update schema of organizations and projects
rDOBJSa9c2454dd63d: test_storage: add collision detection test
rDOBJS35dd9fbabba0: swh.storage.tests: Actually check the error message
rDOBJS40ab1c42e208: sql/swh-func: add table aliases to "where not exists" queries
rDOBJS8dfb906e4926: db_testing.py: allow to customize DB restore
rDOBJSaf4005522d58: storage.content_find: better testing
rDOBJSda274930196a: swh-schema: person (name, email) -> bytea
rDOBJSd250617427ad: swh.storage.tests: clean the database on teardown
rDOBJS0847618cb425: swh.storage.storage: make occurrence_add support overlapping intervals
rDOBJSff2661c8e940: Remove atime, mtime, ctime from directory entries
rDOBJS730766edb587: storage.content_find: refactor argument checking
rDOBJSff0c444c2e16: test_api_client: use absolute import to allow rm __init__.py
rDOBJS54134618a387: Improve output response for find occurrence api
rDOBJS41becead4548: test_storage: don't break if the storage has no objstorage
rDOBJS08bbfa766ae3: Implement content_find_occurrence up to the db
rDOBJS487e4a29ba38: swh.storage.storage: add origin_get and origin_add_one
rDOBJSef7b096357b2: sql/swh-schema: unix_path to bytea
rDOBJS59cd05e6be73: Refactor - Migrate content_present to content_find
rDOBJS3079f675482f: tests: add documentation for Test{,Remote}Storage
rDOBJSb0e2a6678554: swh.storage.api: Properly escape arbitrary byte sequences in arguments
rDOBJSea0ac366f30d: swh.storage.storage: Allow storing "skipped" contents
rDOBJS6d89128b9b8e: tests: make both storage tests inherit from an abstract class
rDOBJS30fcc29356a5: Add a remote API server and a client for remote storage
rDOBJSd4a37d7c6b96: Refactor - Permit reuse of the content_find api
rDOBJSd964f372a69a: swh.storage.api: Serialize/deserialize exceptions
rDOBJSfbdf7bf2ea65: Move remote_storage and api to api.client and api.server
rDOBJS2f969e885e40: move db_test.py from swh-storage to swh-core
rDOBJS8b012b2cc280: Simplify the api content_present
rDOBJSa2a2a65fdde3: Open /content/present api
rDOBJS6e5364e3d73b: test_remote_storage: Add code to run the server automatically
rDOBJSa9102e766c0f: Implement the content_present api
rDOBJS9d5da47dfdcd: Add offsets to the dates that need them in the schema
rDOBJS79b28a205057: Commit first implementation of directory_add
rDOBJS4aa636a5aa54: DB testing: update doc and tests to use swh-storage-testdata
rDOBJSb48911d097a9: swh.storage.storage: implement occurrence_add
rDOBJSc2a5a2d88206: swh.storage.tests: refactor storage test
rDOBJS1b5f80a80532: Implement revision_add for swh.storage.Storage
rDOBJSf02b1a0bc081: swh.storage.storage: Add content_missing method to Storage
rDOBJS4d4633a1cbd0: swh.storage.db/storage: Allow listing one directory's contents
rDOBJS1811f5916880: add back needed __init__.py to non-namespace packages
rDOBJS8c5aec2d794e: get rid of __init__.py
rDOBJS8cb72d223c39: test_objstorage.py: use assert(Not)In where possible (style)
rDOBJScf387f8b2d40: db_testing: bug fix: call specific super() methods
rDOBJSad760ca40473: db_testing.py: fixture to ease testing using the DB
rDOBJSa334ddd93523: db_testing: do not try to rollback closed connections
rDOBJS4d7ca904aeb3: test_storage.py: test add_content()
rDOBJS942a0fd6750d: stored proc: generalize mktemp function
rDOBJS375e2fda05b5: Rename Storage.add_content to content_add
rDOBJS75a597cd45bb: Update ObjStorage API for the new hashes
rDOBJS000d2098afd0: test_db.py: (trivial) tests for swh.storage.db
rDOBJS48f11ca9c99d: refactor db.Db to be a proxy with storage procedure wrappers
rDOBJSa05b79a3b210: db.Db: refactor copy_to() function to COPY to arbitrary tables
rDOBJS8e416232bcf6: make swh.storage.tests a proper python module
rDOBJS9f120b6b29cb: Update swh.storage.db for the new hash API
rDOBJS09e1d5717352: objstorage: simplify exc hierarchy to have a top-level Error exc
rDOBJS995efafc3885: objstorage: return object id upon add()
rDOBJSc6e50451862a: objstorage: idempotent addition, ignore dups
rDOBJS7c183da88c37: ensure we can easily switch to sha1 algo
rDOBJS8cd1c7c88e10: tests: preseed obj_id on add wherever possible
rDOBJSddd94f2301d0: objstorage: use __contains__ for membership test
rDOBJS810bde68afe5: tests: better organization for setUp values
rDOBJS9012042f9a68: objstorage: add support for len() and __iter__()
rDOBJSd3c33d6dbef7: add check() method
rDOBJS99f01a2f9a7f: add get() method(s)
rDOBJS75750b3b1907: objstorage: switch to sha1 object ids (from sha1_git)
rDOBJSe8733fa77194: add has() method and clobbering support
rDOBJSf17eaf8ad521: add gzip compression
rDOBJSae1f67019537: push tests down, inside swh.storage
Summary

The previous push didn't work correctly and the commit history were erased.

Diff Detail

Repository
rDOBJS Object storage
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

qcampos updated this revision to Diff 225.Jun 27 2016, 1:37 PM
qcampos retitled this revision from to Recreate DOBJS commit history.
qcampos updated this object.
qcampos added a reviewer: ardumont.
zack accepted this revision.Jun 27 2016, 1:38 PM
zack added a reviewer: zack.
This revision is now accepted and ready to land.Jun 27 2016, 1:38 PM
qcampos updated this revision to Diff 226.Jun 27 2016, 1:42 PM
qcampos edited edge metadata.

Update differential

This revision now requires review to proceed.Jun 27 2016, 1:42 PM
zack accepted this revision.Jun 27 2016, 1:42 PM
zack edited edge metadata.
This revision is now accepted and ready to land.Jun 27 2016, 1:42 PM
This revision was automatically updated to reflect the committed changes.