Page MenuHomeSoftware Heritage

DanSeraf (Daniele Serafini)
User

Projects

User Details

User Since
Sep 23 2019, 3:10 PM (28 w, 1 d)

Recent Activity

Yesterday

DanSeraf added a comment to D2960: swh-model: exclude path support for the function from_disk in Directory.

I can define a dir_filter function but how could i get the values i need, like the full path generated inside from_disk, without touching the implementation?

I'm not sure i understand the question.
from_disk has a save_path parameter. If set to True, the paths are stored alongside the output result.
Isn't that enough?

Mon, Apr 6, 8:27 PM
DanSeraf added a comment to D2960: swh-model: exclude path support for the function from_disk in Directory.

I think in the end, this should be implemented with the dir_filter parameter instead.

Please, check that you can do this. If you can, this will, there is no need to change the current implementation.
And instead, either:

  • not touch swh.model at all, then define your ignore_path function in swh-scanner and use it when calling from_disk.
  • if you think, this can be a shared behavior (i guess it can), define a function callable here in swh-model (like accept_all_directories and the other ones next to it).

In both cases, this won't touch Directory.from_disk though.

Mon, Apr 6, 6:10 PM
DanSeraf updated the summary of D2960: swh-model: exclude path support for the function from_disk in Directory.
Mon, Apr 6, 4:10 PM
DanSeraf created D2960: swh-model: exclude path support for the function from_disk in Directory.
Mon, Apr 6, 4:10 PM

Thu, Apr 2

DanSeraf committed rDTSCNa618bf17c049: new visualization format: sunburst chart (authored by DanSeraf).
new visualization format: sunburst chart
Thu, Apr 2, 2:08 PM
DanSeraf committed rDTSCNe4bb51b7d894: changed structure of temp_folder fixture (authored by DanSeraf).
changed structure of temp_folder fixture
Thu, Apr 2, 2:08 PM
DanSeraf committed rDTSCNa49c4e5ec053: model: get information about directories (authored by DanSeraf).
model: get information about directories
Thu, Apr 2, 2:08 PM
DanSeraf closed D2863: scanner: show result with a sunburst chart.
Thu, Apr 2, 2:08 PM
DanSeraf updated the diff for D2863: scanner: show result with a sunburst chart.

rebase

Thu, Apr 2, 2:07 PM

Wed, Apr 1

DanSeraf added a comment to D2863: scanner: show result with a sunburst chart.

Could you add tests that check this bug doesn't happen again?

Wed, Apr 1, 5:09 PM
DanSeraf updated the diff for D2863: scanner: show result with a sunburst chart.

ids of sunburst must be unique

Wed, Apr 1, 1:42 PM

Fri, Mar 27

DanSeraf updated the diff for D2863: scanner: show result with a sunburst chart.

requested changes
resolved bug due to wrong percentage calculation

Fri, Mar 27, 1:20 PM
DanSeraf added a comment to D2863: scanner: show result with a sunburst chart.

Sorry I didn't do it earlier, but I finally got around to actually running this code, and I found a bug: if two directories have the same name (or if there's one named "total"), then the swh-scanner process exits early, and nothing is shown in the browser.

Fri, Mar 27, 10:58 AM

Thu, Mar 26

DanSeraf updated the diff for D2863: scanner: show result with a sunburst chart.

requested changes

Thu, Mar 26, 7:35 PM
DanSeraf created P632 (An Untitled Masterwork).
Thu, Mar 26, 2:32 PM

Wed, Mar 25

DanSeraf updated the diff for D2863: scanner: show result with a sunburst chart.
  • test dataframe generation
  • requested changes
Wed, Mar 25, 7:14 PM
DanSeraf added a comment to T2336: scanner: add support for an exclusion list.

The problem here is that actually swh-model il calculating the persistent identifier of the input path, so it also has to exclude the specified paths.

Wed, Mar 25, 3:28 PM · Code scanner

Mon, Mar 23

DanSeraf added inline comments to D2863: scanner: show result with a sunburst chart.
Mon, Mar 23, 9:25 PM

Fri, Mar 20

DanSeraf updated the summary of D2863: scanner: show result with a sunburst chart.
Fri, Mar 20, 7:08 PM
DanSeraf created D2863: scanner: show result with a sunburst chart.
Fri, Mar 20, 7:02 PM
DanSeraf committed rDTSCN5c62848288b0: dump json instead of dict (authored by DanSeraf).
dump json instead of dict
Fri, Mar 20, 12:17 PM
DanSeraf closed D2855: scanner: dump json instead of dict.
Fri, Mar 20, 12:17 PM
DanSeraf updated the diff for D2855: scanner: dump json instead of dict.

rebase

Fri, Mar 20, 12:17 PM
DanSeraf updated the diff for D2855: scanner: dump json instead of dict.

better json output

Fri, Mar 20, 11:54 AM

Thu, Mar 19

DanSeraf updated the diff for D2855: scanner: dump json instead of dict.

tests

Thu, Mar 19, 7:44 PM
DanSeraf added a comment to D2855: scanner: dump json instead of dict.

/with jedi mind trick voice

You want to add types
You want to update the tests

;)

Thu, Mar 19, 7:33 PM
DanSeraf reclaimed D2855: scanner: dump json instead of dict.
Thu, Mar 19, 4:18 PM
DanSeraf abandoned D2855: scanner: dump json instead of dict.
Thu, Mar 19, 2:12 PM
DanSeraf created D2855: scanner: dump json instead of dict.
Thu, Mar 19, 12:42 PM

Thu, Mar 12

DanSeraf committed rDTSCN3d462c8364ed: added test for the model (authored by DanSeraf).
added test for the model
Thu, Mar 12, 6:42 PM
DanSeraf committed rDTSCN8448ae0a840e: get_subpaths test adapted to the new temp_folder structure (authored by DanSeraf).
get_subpaths test adapted to the new temp_folder structure
Thu, Mar 12, 6:42 PM
DanSeraf committed rDTSCN3de533865fef: changed folder structure in temp_folder fixture (authored by DanSeraf).
changed folder structure in temp_folder fixture
Thu, Mar 12, 6:42 PM
DanSeraf closed D2811: scanner: added test for the model.
Thu, Mar 12, 6:42 PM
DanSeraf updated the diff for D2811: scanner: added test for the model.

changes

Thu, Mar 12, 6:02 PM

Wed, Mar 11

DanSeraf created D2811: scanner: added test for the model.
Wed, Mar 11, 2:57 PM
DanSeraf closed T2300: scanner: print a nicer error message when rate limit is hit as Resolved by committing rDTSCNcb5faf6e866a: exceptions: APIError response.
Wed, Mar 11, 9:51 AM · Easy hack, Code scanner
DanSeraf committed rDTSCNcb5faf6e866a: exceptions: APIError response (authored by DanSeraf).
exceptions: APIError response
Wed, Mar 11, 9:51 AM
DanSeraf committed rDTSCN1857499820d0: let live_server start directly (authored by DanSeraf).
let live_server start directly
Wed, Mar 11, 9:51 AM
DanSeraf closed D2794: exceptions: custom error response.
Wed, Mar 11, 9:51 AM
DanSeraf updated the diff for D2794: exceptions: custom error response.

removed --no-start-live-server option in pytest.ini

Wed, Mar 11, 9:34 AM
DanSeraf added inline comments to D2794: exceptions: custom error response.
Wed, Mar 11, 9:29 AM

Tue, Mar 10

DanSeraf updated the diff for D2794: exceptions: custom error response.

rebase

Tue, Mar 10, 6:02 PM
DanSeraf updated the diff for D2794: exceptions: custom error response.
flake8
Tue, Mar 10, 3:14 PM
DanSeraf updated the diff for D2794: exceptions: custom error response.
assert APIError raises when the payload is too large
live_server override, only one per session
Tue, Mar 10, 3:10 PM
DanSeraf added a comment to D2794: exceptions: custom error response.

Now that swh-scanner has tests, it would be nice if every new feature and bugfix added (or changed) one or more tests.

Tue, Mar 10, 1:40 PM
DanSeraf created D2794: exceptions: custom error response.
Tue, Mar 10, 12:11 PM
DanSeraf closed T2299: scanner: add integration tests as Resolved by committing rDTSCNa5555dd12464: basic test for the scanner.
Tue, Mar 10, 10:46 AM · Code scanner
DanSeraf committed rDTSCNa5555dd12464: basic test for the scanner (authored by DanSeraf).
basic test for the scanner
Tue, Mar 10, 10:46 AM
DanSeraf closed D2791: swh-scanner: integration tests.
Tue, Mar 10, 10:46 AM

Mon, Mar 9

DanSeraf updated the diff for D2791: swh-scanner: integration tests.

make tox pass

Mon, Mar 9, 5:50 PM
DanSeraf updated the diff for D2791: swh-scanner: integration tests.

rebase

Mon, Mar 9, 3:19 PM
DanSeraf closed D2785: scanner: check that pid exists before yield the sub path.
Mon, Mar 9, 12:36 PM
DanSeraf committed rDTSCNb984695a79a8: scanner: behavior when a symlink is encountered (authored by DanSeraf).
scanner: behavior when a symlink is encountered
Mon, Mar 9, 12:36 PM
DanSeraf updated the diff for D2785: scanner: check that pid exists before yield the sub path.

if a symbolic link is encountered, the persistent identifier is parsed by pid_of_file (as in swh.model)

Mon, Mar 9, 12:29 PM

Mar 6 2020

DanSeraf updated the diff for D2791: swh-scanner: integration tests.

rebase

Mar 6 2020, 5:08 PM
DanSeraf updated the diff for D2785: scanner: check that pid exists before yield the sub path.

comments

Mar 6 2020, 5:03 PM
DanSeraf added a comment to D2785: scanner: check that pid exists before yield the sub path.

When does this happen? Could you add a test for this behavior?

When, for example, there is a broken symlink. It can't read if it's a file or a directory.

Could you add this as a comment?

Mar 6 2020, 4:56 PM
DanSeraf added a comment to D2791: swh-scanner: integration tests.

I'm restarting the tests

Mar 6 2020, 4:48 PM
DanSeraf updated the diff for D2791: swh-scanner: integration tests.

clean flask_api.py

Mar 6 2020, 4:47 PM
DanSeraf updated the diff for D2791: swh-scanner: integration tests.

make flake8 pass

Mar 6 2020, 4:44 PM
DanSeraf created D2791: swh-scanner: integration tests.
Mar 6 2020, 4:40 PM
DanSeraf committed rDTSCNe21189377049: model: getJsonTree, wrong name on recursive call (authored by DanSeraf).
model: getJsonTree, wrong name on recursive call
Mar 6 2020, 4:04 PM
DanSeraf closed D2784: model: getJsonTree, wrong name on recursive call.
Mar 6 2020, 4:04 PM
DanSeraf added a comment to D2785: scanner: check that pid exists before yield the sub path.

When does this happen? Could you add a test for this behavior?

Mar 6 2020, 4:02 PM
DanSeraf created D2785: scanner: check that pid exists before yield the sub path.
Mar 6 2020, 2:37 PM
DanSeraf created D2784: model: getJsonTree, wrong name on recursive call.
Mar 6 2020, 2:28 PM
DanSeraf added a watcher for Code scanner: DanSeraf.
Mar 6 2020, 10:46 AM

Mar 3 2020

DanSeraf committed rDTSCN9c3954efa07a: model: using f-strings in printNode function (authored by DanSeraf).
model: using f-strings in printNode function
Mar 3 2020, 4:44 PM
DanSeraf closed D2758: model: using f-strings in printNode function.
Mar 3 2020, 4:43 PM
DanSeraf created D2758: model: using f-strings in printNode function.
Mar 3 2020, 3:00 PM
DanSeraf committed rDTSCNcedf6fc9b068: model: output result to json file (authored by DanSeraf).
model: output result to json file
Mar 3 2020, 2:46 PM
DanSeraf closed T2298: scanner: support alternative output formats as Resolved by committing rDTSCNcedf6fc9b068: model: output result to json file.
Mar 3 2020, 2:46 PM · Code scanner
DanSeraf closed D2748: model: output result to json file.
Mar 3 2020, 2:46 PM
DanSeraf updated the diff for D2748: model: output result to json file.

cli: --format option, text and json
model:

  • changed function name from getJsonChild to getJsonTree
  • show function: output based on format
Mar 3 2020, 1:36 PM

Mar 2 2020

DanSeraf updated the diff for D2748: model: output result to json file.

cli: flake8

Mar 2 2020, 7:22 PM
DanSeraf created D2748: model: output result to json file.
Mar 2 2020, 7:17 PM

Feb 28 2020

DanSeraf committed rDTSCN08059a547774: wrong url parsing (authored by DanSeraf).
wrong url parsing
Feb 28 2020, 4:57 PM
DanSeraf triaged T2298: scanner: support alternative output formats as Low priority.
Feb 28 2020, 3:36 PM · Code scanner
DanSeraf committed rDTSCNdd9004026c04: color only when stdout is connected to terminal, removed logger (authored by DanSeraf).
color only when stdout is connected to terminal, removed logger
Feb 28 2020, 3:12 PM
DanSeraf committed rDTSCN1541e9f44a9e: default url (authored by DanSeraf).
default url
Feb 28 2020, 3:12 PM
DanSeraf committed rDTSCNc0dbe0b1907b: cli: scanner group with scan option, updated description (authored by DanSeraf).
cli: scanner group with scan option, updated description
Feb 28 2020, 3:12 PM
DanSeraf committed rDTSCNed5a359e2ef4: added: model, logger and exceptions (authored by DanSeraf).
added: model, logger and exceptions
Feb 28 2020, 3:12 PM
DanSeraf committed rDTSCN40442b4d6bf8: scanner: asynchronous operation, type annotation (authored by DanSeraf).
scanner: asynchronous operation, type annotation
Feb 28 2020, 3:12 PM
DanSeraf committed rDTSCNf39b3d2c3374: test requirements (authored by DanSeraf).
test requirements
Feb 28 2020, 3:12 PM
DanSeraf committed rDTSCNb09cf5b29168: code scanner prototype (authored by DanSeraf).
code scanner prototype
Feb 28 2020, 3:12 PM
DanSeraf closed D2657: code scanner prototype.
Feb 28 2020, 3:12 PM
DanSeraf updated the summary of D2657: code scanner prototype.
Feb 28 2020, 2:53 PM
DanSeraf updated the diff for D2657: code scanner prototype.
default url
Feb 28 2020, 1:29 PM
DanSeraf updated the diff for D2657: code scanner prototype.
  • cli: scanner group with scan option, updated description
  • color only when stdout is connected to terminal
Feb 28 2020, 1:24 PM

Feb 27 2020

DanSeraf added a comment to D2657: code scanner prototype.

It's still missing a description of what it does (ie. check that files exist in swh), in the CLI help and/or README.

and I still think you should use a subcommand, in case you want to add new features in the future without breaking scripts that depend on this one.

Feb 27 2020, 4:12 PM
DanSeraf added a comment to D2657: code scanner prototype.
Feb 27 2020, 3:56 PM

Feb 26 2020

DanSeraf updated the diff for D2657: code scanner prototype.
The cli supports only one operation: swh scanner [options] path, where path is the root path we want to scan.
The output now is a tree like structure with a green/blue text if the path is found, red otherwise.
Feb 26 2020, 4:38 PM

Feb 17 2020

DanSeraf committed rDWAPPS792f7263ffc6: input size limit for /known/ endpoint (authored by DanSeraf).
input size limit for /known/ endpoint
Feb 17 2020, 11:54 AM
DanSeraf closed D2661: Web API: /known/ input size limit.
Feb 17 2020, 11:54 AM
DanSeraf updated the diff for D2661: Web API: /known/ input size limit.

rebase

Feb 17 2020, 11:54 AM

Feb 14 2020

DanSeraf updated the diff for D2661: Web API: /known/ input size limit.

limit is set inside the endpoint

Feb 14 2020, 11:12 AM

Feb 12 2020

DanSeraf updated the diff for D2661: Web API: /known/ input size limit.

rebase

Feb 12 2020, 4:36 PM
DanSeraf added inline comments to D2661: Web API: /known/ input size limit.
Feb 12 2020, 4:09 PM
DanSeraf added inline comments to D2661: Web API: /known/ input size limit.
Feb 12 2020, 3:38 PM
DanSeraf updated the summary of D2661: Web API: /known/ input size limit.
Feb 12 2020, 3:18 PM