This first version does create a compressed folder of an archive
directory but is not linked to any API or notification system.
This diff is submitted for architecture and code review and will evolve.
Details
Details
- Reviewers
ardumont - Group Reviewers
Reviewers - Maniphest Tasks
- T531: Vault cookers
- Commits
- rDSTOCb11cfe34a03e: First version of the directory cooker & cache
R65:b11cfe34a03e: First version of the directory cooker & cache
rDVAUb84c5152a4ea: First version of the directory cooker & cache
rDSTOb11cfe34a03e: First version of the directory cooker & cache
Diff Detail
Diff Detail
- Repository
- rDSTO Storage manager
- Branch
- vault
- Lint
No Linters Available - Unit
No Unit Test Coverage - Build Status
Buildable 394 Build 582: Software Heritage Python tests Build 581: arc lint + arc unit
Event Timeline
| swh/storage/vault/cache.py | ||
|---|---|---|
| 18 | What does the TempStorage is for? It looks like an iterative step which could be removed now. | |
| swh/storage/vault/cooker.py | ||
| 22 | describes | |
| 38 | must implement | |
| 69 | You could use generator: Something like this could work: import itertools datas = self.storage.directory_ls(dir_id, recursive=True) file_data, dir_data = itertools.tee(datas, 2) file_data = (entry for entry in datas if entry['type'] == 'file') dir_data = (entry for entry in datas if entry['type'] == 'dir') If this work, you could even go further as to deal with absent, hidden and normal files here by creating 5 generators and do the filtering here. file_hidden_data = (entry for entry in datas if entry['type'] == 'file' and entry['status'] == 'hidden') file_absent_data = (entry for entry in datas if entry['type'] == 'file' and entry['status'] == 'absent') ... Note: data is invariable, so it should really be data (and not datas) ^^ | |
| 70 | s/If this work,/If this works/ | |
| 84 | You could use generator here using ( and ) and even possibly remove altogether the function (cf. previous comment). | |
| 106 | method | |
| 124 | beware the print ^^ | |
| swh/storage/vault/cooker.py | ||
|---|---|---|
| 74 | You could probably remove the map here and only retrieves the list of names line 90. Also, i stand by my previous remark, you could inline the _split_data method directly here. | |
| 90 | If entry is not useful, you could directly return entry['name']. | |
| 128 | This could probably be improved later. | |
| swh/storage/vault/cooker.py | ||
|---|---|---|
| 128 | Remember the performance issue we got on a remote storage? The current implementation is the reason. Working on a local storage made the execution time acceptable, but changing it is scheduled! | |