User Details
- User Since
- May 11 2019, 6:37 PM (281 w, 6 d)
Sep 22 2019
The problem is with the python setup something to do with typing.py giving errors if I remember correctly. I will try it again and add a error log.
I am sorry for the delay. Due to some dependency issue, I am unable to run the tests locally which is why I have been making so many pushes and am unable to solve it effectively. I will reinstall all the whole environment and try to fix it, if the environment works I will complete this. If I am unable to fix my environment then sadly I will have to drop this one.
Aug 20 2019
- Remove unneeded checks
- Remove unneeded checks
- Remove unneeded checks
- Remove unneeded checks
Aug 14 2019
- Fix return value for license indexer
Aug 13 2019
Overiding just the test method doesn't work for TestFossologyLicenseRangeIndexer. Tests test__index_contents and test__index_contents_with_indexed_data call the function self.indexer._index_contents which is implemented in ContentRangeIndexer.
- Fix return value for license indexer
- Fix return value for license indexer
Aug 12 2019
- Override test methods
- Override test methods
It appears that two different tests TestFossologyLicenseRangeIndexer and TestMimetypeRangeIndexer are inheriting from the same class namely, CommonContentIndexerRangeTest as a result both use the same assert_results_ok method. Both test classes have the same structure for expected results, hence the test works for both, as I experimented in the previous commit, changing the test for the new format of TestFossologyLicenseRangeIndexer will fail for TestMimetypeRangeIndexer. How do I resolve this? Should I override the method in both classes?
Aug 11 2019
- Change tests and expected results
Aug 8 2019
Aug 6 2019
I haven't been able to give enough time to it. I'll complete the diff by this weekend.
Jul 22 2019
Squash and change commit message
- Refactoring
- Change expected result
Jul 21 2019
Fixed skipped_content counter bug
Jul 19 2019
Rebase on master
Rebase and update
- Change index storage mechanism
Best solution is tox -- -k test_name
Jul 16 2019
- Add break to prevent multiple yields
- Use all hashes in a content
Jul 13 2019
- Remove dependency
- Modify in_memory content_add to add skipped_content
I did not find any mechanism in db.py that is actually storing skipped_content. db.py line 51, is passed without implementation.
In db.py line 128, the query does not compare blake2s256 despite content_hash_keys = ['sha1', 'sha1_git', 'sha256', 'blake2s256']. Does this mean that skipped content will never be hashed with blake2s256?
I have a concern here, storage.py line 120. The function self.content_missing can throw an exception in case of a hash collision. Shouldn't line 120 be in a try except block to catch that error and ignore that particular content?
Jul 11 2019
Use db_transaction annotation
Jul 10 2019
small edit and rebase
Jul 9 2019
I went through all the tests in test_storage.py. It appears that only content_fossology_license_get needs to be refactored. All other storage methods return a dictionary or a list of dictionaries, where each dictionary has multiple keys.
Jul 7 2019
I am familiar with the web APIs and I went through the discussion in T782. When you say output a single dictionary, I believe you mean something like this
{ sha1: [ {tool: TOOL, licenses: [licences]}, {tool: TOOL, licenses: [licences]} ],
Jul 6 2019
This logic is similar to one being used in db.skipped_content_missing. However the current implementation in_memory._content_add does not populate _skipped_contents and _skipped_content_indexes.
Jul 2 2019
After re-reading the documentation, I realized that the configuration files given in swh-indexer is actually used by swh-scheduler and swh-storage, indicating that these two modules are at the root of the dependency tree.
I looked at how the docs look after being built, for e.g. take swh-indexer at https://docs.softwareheritage.org/devel/swh-indexer/dev-info.html. It seems like the configuration information along with instructions to run and test it are best suited to this page. Have you considered adding comments about configuration parameters in this page itself, rather than making a top level file, because because only someone hacking on swh-indexer would be interested in the configuration.
I can begin working on it, once I understand what is required. Is my interpretation of the task correct?
Jul 1 2019
Jun 29 2019
This is related to T1388.
Jun 28 2019
Rebase from master
To expand on this further, I think the pre-push hook scripts you have configured only work when using git in a terminal. I am using vs code and curiously, I used the gui to push changes. I believe the scripts are not able to check such a situation and I can commit changes without review. This is probably a security flaw, that should be considered seriously. @zack
It seems like my revision is already in origin master. There are no changes to push. Should I close this revision.
Jun 27 2019
Aren't all doc strings rendered at some place with documentation for packages, sub modules the functions they contain?
While formatting the documents I realized that whoever was writing was try to keep line lengths short but they followed an arbitrary line length which was longer than 80 character. Since # noqa was applied these formatting errors did not pop up. I think a lot of this can be resolved if there can be a guideline to add a vertical ruler to the editor at 80 chars.
Fix more docstrings
Reformat docstring wherever possible
Reformat docstring for max line length
I'll close this revision.
I get it now, that's an ingenious way of keeping documentation up to date. However if this is evaluated why is adding whitespace changing the output, the dictionary and list should still have valid items.
Then swh-indexer does not require any changes I will close this diff. Wow, I never new docstrings could be evaluated. Why is this required though?
I will close this diff since no changes required for swh-deposit. With complex cases like this there is no way this process can be automated.
I followed swh-docker-dev documentation to host the setup locally. But it only hosts the web portal, I couldn't access locally hosted documentation. How can I see the effect my changes are making?
It is odd that py3 test cases are failing although I have only changed to formatting of the docstrings.
Jun 26 2019
what is the expected formatting for snippets like these
the 80 character mark is -----------------------------------------------------------------| @browse_route(r'origin/(?P<origin_type>[a-z]+)/url/(?P<origin_url>.+)/visit/(?P<timestamp>.+)/directory/', # noqa r'origin/(?P<origin_type>[a-z]+)/url/(?P<origin_url>.+)/visit/(?P<timestamp>.+)/directory/(?P<path>.+)/', # noqa r'origin/(?P<origin_type>[a-z]+)/url/(?P<origin_url>.+)/directory/', # noqa r'origin/(?P<origin_type>[a-z]+)/url/(?P<origin_url>.+)/directory/(?P<path>.+)/', # noqa r'origin/(?P<origin_url>.+)/visit/(?P<timestamp>.+)/directory/', # noqa r'origin/(?P<origin_url>.+)/visit/(?P<timestamp>.+)/directory/(?P<path>.+)/', # noqa r'origin/(?P<origin_url>.+)/directory/', # noqa r'origin/(?P<origin_url>.+)/directory/(?P<path>.+)/', # noqa view_name='browse-origin-directory') def origin_directory_browse(request, origin_url, origin_type=None, timestamp=None, path=None): """Django view for browsing the content of a directory associated to an origin for a given visit.