Page MenuHomeSoftware Heritage
Differential D4735

Move parse_swh_reference to swh.deposit.utils namespace
ClosedPublic
Actions

Authored by ardumont on Dec 14 2020, 5:35 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Maniphest Tasks
T2886: support metadata-only deposit in deposit client cli
Commits
rDDEP36f0b632966b: Move parse_swh_reference to swh.deposit.utils namespace
Summary

So it can be reused in other non django part of the deposit (e.g
deposit client cli)

Related to T2886

Test Plan

tox

Diff Detail

Repository
rDDEP Push deposit
Branch
master
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 17934
Build 27707: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 27706: arc lint + arc unit

Event Timeline

ardumont created this revision.Dec 14 2020, 5:35 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptDec 14 2020, 5:35 PM
swh-public-ci added a comment.Dec 14 2020, 5:38 PM

Build has FAILED

Patch application report for D4735 (id=16772)

Rebasing onto 73ee587df9...

Current branch diff-target is up to date.
Changes applied before test
commit 36527db3c35089f11b56858df8de6fc3a221f6e3
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Dec 14 17:34:22 2020 +0100

    Move parse_swh_reference in swh.deposit.utils namespace
    
    So it can be reused in other non django part of the deposit.
    
    Related to T2886

Link to build: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/479/
See console output for more information: https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/479/console

Harbormaster failed remote builds in B17932: Diff 16772!Dec 14 2020, 5:38 PM
ardumont added inline comments.Dec 14 2020, 5:43 PM
swh/deposit/tests/cli/test_client.py
642 ↗(On Diff #16772)

oops, not meant here ¯\_(ツ)_/¯, this is what fails the build...

ardumont updated this revision to Diff 16774.Dec 14 2020, 5:45 PM

Drop test code not meant for that diff

zack retitled this revision from Move parse_swh_reference in swh.deposit.utils namespace to Move parse_swh_reference to swh.deposit.utils namespace.Dec 14 2020, 5:46 PM
swh-public-ci added a comment.Dec 14 2020, 5:47 PM

Build is green

Patch application report for D4735 (id=16774)

Rebasing onto 73ee587df9...

Current branch diff-target is up to date.
Changes applied before test
commit 2e0c77b59f4ef11fc6385465caf4f0dd48485309
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Dec 14 17:43:36 2020 +0100

    Move parse_swh_reference in swh.deposit.utils namespace
    
    So it can be reused in other non django part of the deposit.
    
    Related to T2886

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/480/ for more details.

Harbormaster completed remote builds in B17934: Diff 16774.Dec 14 2020, 5:47 PM
ardumont updated this revision to Diff 16775.Dec 14 2020, 5:50 PM

Rework commit message

swh-public-ci added a comment.Dec 14 2020, 5:52 PM

Build is green

Patch application report for D4735 (id=16775)

Rebasing onto 73ee587df9...

Current branch diff-target is up to date.
Changes applied before test
commit 36f0b632966bf94ff2d4b9880e1ca89c6b14a68e
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Mon Dec 14 17:43:36 2020 +0100

    Move parse_swh_reference to swh.deposit.utils namespace
    
    So it can be reused in other non django part of the deposit.
    
    Related to T2886

See https://jenkins.softwareheritage.org/job/DDEP/job/tests-on-diff/481/ for more details.

Harbormaster completed remote builds in B17935: Diff 16775.Dec 14 2020, 5:52 PM
vlorentz accepted this revision.Dec 15 2020, 10:02 AM
This revision is now accepted and ready to land.Dec 15 2020, 10:02 AM
Closed by commit rDDEP36f0b632966b: Move parse_swh_reference to swh.deposit.utils namespace (authored by ardumont). · Explain WhyDec 15 2020, 11:45 AM
This revision was automatically updated to reflect the committed changes.
ardumont added a commit: rDDEP36f0b632966b: Move parse_swh_reference to swh.deposit.utils namespace.

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
deposit/
api/
3 lines95%
Loading...
100 lines100%
Loading...
tests/
api/
115 lines100%
Loading...
111 lines100%
Loading...
106 lines98%
Loading...

Diff 16774

View Options

swh/deposit/api/common.py

  • This file was added.
# Copyright (C) 2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
"""
The purpose of this module is to display and to interact with the result of the
scanner contained in the model.
vlorentzUnsubmitted
Not Done
Inline Actions

this group of imports should be after the other ones. See the second item here: https://www.python.org/dev/peps/pep-0008/#imports

vlorentz: this group of imports should be after the other ones. See the second item here: https://www.
The `sunburst` function generates a navigable sunburst chart from the
directories information retrieved from the model. The chart displays for
each directory the total number of files and the percentage of file known.
The size of the directory is defined by the total number of contents whereas
the color gradient is generated relying on the percentage of contents known.
"""
from typing import List, Dict, Tuple
from pathlib import PosixPath
from plotly.offline import offline # type: ignore
import plotly.graph_objects as go # type: ignore
import pandas as pd # type: ignore
import numpy as np # type: ignore
vlorentzUnsubmitted
Not Done
Inline Actions

module docstring must be before imports

vlorentz: module docstring must be before imports
def build_hierarchical_df(
dirs_dataframe: pd.DataFrame, levels: List[str],
metrics_columns: List[str], root_name: str) -> pd.DataFrame:
"""
Build a hierarchy of levels for Sunburst or Treemap charts.
vlorentzUnsubmitted
Not Done
Inline Actions

Could you add a description of each argument? (and possibly change the name df to express what it contains)

vlorentz: Could you add a description of each argument? (and possibly change the name `df` to express…
vlorentzUnsubmitted
Not Done
Inline Actions

renaming df to dataframe isn't more explicit about what it contains :/

vlorentz: renaming `df` to `dataframe` isn't more explicit about what it contains :/
For each directory the new dataframe will have the following
vlorentzUnsubmitted
Not Done
Inline Actions

"For each directory level stored, the new dataframe will have"?

vlorentz: "For each directory level store**d,** the new dataframe will have"?
information:
id: the directory name
parent: the parent directory of id
contents: the total number of contents of the directory id and
vlorentzUnsubmitted
Not Done
Inline Actions

Could you add an example of inputs, and what the outputs are? (like you did for generate_df)

vlorentz: Could you add an example of inputs, and what the outputs are? (like you did for `generate_df`)
the relative subdirectories
known: the percentage of contents known relative to computed
'contents'
Example:
Given the following dataframe:
.. code-block:: none
vlorentzUnsubmitted
Not Done
Inline Actions

forgot this print

vlorentz: forgot this print
lev0 lev1 contents known
'' '' 20 2 //root
kernel kernel/subdirker 5 0
telnet telnet/subdirtel 10 4
The output hierarchical dataframe will be like the following:
.. code-block:: none
vlorentzUnsubmitted
Not Done
Inline Actions

I think I understand what is going on here, but could you add a comment with an example of how the data is reshaped from df to df_tree step-by-step, for future readers?

vlorentz: I think I understand what is going on here, but could you add a comment with an example of how…
id parent contents known
20 10.00
vlorentzUnsubmitted
Not Done
Inline Actions

The data from the first df_trees will be copied into a new dataframe for each run of the loop. This is inefficient (quadratic complexity). You might want to build a list of dataframes and concatenate everything at the end.

vlorentz: The data from the first `df_tree`s will be copied into a new dataframe for each run of the loop.
kernel/subdirker kernel 5 0.00
telnet/subdirtel telnet 10 40.00
total 20 10.00
kernel total 5 0.00
telnet total 10 40.00
total 35 17.14
To create the hierarchical dataframe we need to iterate through
the dataframe given in input relying on the number of levels.
Based on the previous example we have to do two iterations:
vlorentzUnsubmitted
Not Done
Inline Actions

dirs_path: List[PosixPath]

vlorentz: `dirs_path: List[PosixPath]`
iteration 1
vlorentzUnsubmitted
Not Done
Inline Actions

also, would be nice to write in this docstring what data is in the dataframe

vlorentz: also, would be nice to write in this docstring what data is in the dataframe
The generated dataframe 'df_tree' will be:
.. code-block:: none
id parent contents known
20 10.0
kernel/subdirker kernel 5 0.0
telnet/subdirtel telnet 10 40.0
iteration 2
The generated dataframe 'df_tree' will be:
.. code-block:: none
id parent contents known
total 20 10.0
kernel total 5 0.0
vlorentzUnsubmitted
Not Done
Inline Actions

You could rename it generate_df_from_dirs.

And you can add a type annotation: dirs: Dict[PosixPath, Tuple[int, int]].

vlorentz: You could rename it `generate_df_from_dirs`. And you can add a type annotation: `dirs: Dict…
telnet total 10 40.0
Note that since we have reached the last level, the parent given
to the directory id is the directory root.
The 'total' row il computed by adding the number of contents of the
dataframe given in input and the average of the contents known on
vlorentzUnsubmitted
Not Done
Inline Actions

Great!

vlorentz: Great!
the total number of contents.
"""
def compute_known_percentage(contents: pd.Series, known: pd.Series
) -> pd.Series:
"""This function compute the percentage of known contents and generate
the new known column with the percentage values.
It also assures that if there is no contents inside a directory
the percentage is zero
"""
known_values = []
for idx, content_val in enumerate(contents):
if content_val == 0:
known_values.append(0)
else:
percentage = known[idx] / contents[idx] * 100
known_values.append(percentage)
return pd.Series(np.array(known_values))
complete_df = pd.DataFrame(columns=['id', 'parent', 'contents', 'known'])
# revert the level order to start from the deepest
levels = [level for level in reversed(levels)]
contents_col = metrics_columns[0]
vlorentzUnsubmitted
Not Done
Inline Actions

parts_list = list(dir_path.parts)

vlorentz: `parts_list = list(dir_path.parts)`
known_col = metrics_columns[1]
df_tree_list = []
for i, level in enumerate(levels):
df_tree = pd.DataFrame(columns=['id', 'parent', 'contents', 'known'])
dfg = dirs_dataframe.groupby(levels[i:]).sum()
dfg = dfg.reset_index()
df_tree['id'] = dfg[level].copy()
if i < len(levels) - 1:
# copy the parent directories (one level above)
df_tree['parent'] = dfg[levels[i+1]].copy()
else:
# last level reached
df_tree['parent'] = root_name
# copy the contents column
df_tree['contents'] = dfg[contents_col]
# compute the percentage relative to the contents
df_tree['known'] = compute_known_percentage(
dfg[contents_col], dfg[known_col])
df_tree_list.append(df_tree)
complete_df = complete_df.append(df_tree_list, ignore_index=True)
# create the main parent
total_contents = dirs_dataframe[contents_col].sum()
total_known = dirs_dataframe[known_col].sum()
total_avg = total_known / total_contents * 100
total = pd.Series(dict(id=root_name, parent='',
contents=total_contents,
known=total_avg))
complete_df = complete_df.append(total, ignore_index=True)
return complete_df
def compute_max_depth(dirs_path: List[PosixPath], root: PosixPath) -> int:
"""Compute the maximum depth level of the given directory paths.
Example: for `var/log/kernel/` the depth level is 3
"""
max_depth = 0
for dir_path in dirs_path:
if dir_path == root:
continue
dir_depth = len(dir_path.parts)
if dir_depth > max_depth:
max_depth = dir_depth
return max_depth
def generate_df_from_dirs(dirs: Dict[PosixPath, Tuple[int, int]],
columns: List[str], root: PosixPath, max_depth: int
) -> pd.DataFrame:
"""Generate a dataframe from the directories given in input.
Example:
given the following directories as input
.. code-block:: python
dirs = {
'/var/log/': (23, 2),
'/var/log/kernel': (5, 0),
'/var/log/telnet': (10, 3)
}
The generated dataframe will be:
.. code-block:: none
lev0 lev1 lev2 contents known
'var' 'var/log' '' 23 2
'var' 'var/log' 'var/log/kernel' 5 0
'var' 'var/log' 'var/log/telnet' 10 3
"""
def get_parents(path: PosixPath):
parts = path.parts[1:] if path.parts[0] == '/' else path.parts
for i in range(1, len(parts)+1):
yield '/'.join(parts[0:i])
def get_dirs_array():
for dir_path, contents_info in dirs.items():
empty_lvl = max_depth - len(dir_path.parts)
if dir_path == root:
# ignore the root but store contents information
yield ['']*(max_depth) + list(contents_info)
else:
yield list(get_parents(dir_path)) + \
['']*empty_lvl + \
list(contents_info)
df = pd.DataFrame(np.array(
[dir_array for dir_array in get_dirs_array()]), columns=columns)
df['contents'] = pd.to_numeric(df['contents'])
df['known'] = pd.to_numeric(df['known'])
return df
def sunburst(directories: Dict[PosixPath, Tuple[int, int]],
root: PosixPath) -> None:
"""Show the sunburst chart from the directories given in input.
"""
max_depth = compute_max_depth(list(directories.keys()), root)
metrics_columns = ['contents', 'known']
levels_columns = ['lev'+str(i) for i in range(max_depth)]
df_columns = levels_columns + metrics_columns
dirs_df = generate_df_from_dirs(directories, df_columns, root, max_depth)
hierarchical_df = build_hierarchical_df(
dirs_df, levels_columns, metrics_columns, str(root))
known_avg = dirs_df['known'].sum() / dirs_df['contents'].sum()
fig = go.Figure()
fig.add_trace(go.Sunburst(
labels=hierarchical_df['id'],
parents=hierarchical_df['parent'],
values=hierarchical_df['contents'],
branchvalues='total',
marker=dict(
colors=hierarchical_df['known'],
colorscale='RdBu',
cmid=known_avg),
hovertemplate='''<b>%{label}</b>
<br>Files: %{value}
<br>Known: <b>%{color:.2f}%</b>''',
name=''
))
offline.plot(fig, filename='sunburst.html')
View Options

swh/deposit/parsers.py

  • This file was added.
# Copyright (C) 2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
"""
The purpose of this module is to display and to interact with the result of the
scanner contained in the model.
vlorentzUnsubmitted
Not Done
Inline Actions

this group of imports should be after the other ones. See the second item here: https://www.python.org/dev/peps/pep-0008/#imports

vlorentz: this group of imports should be after the other ones. See the second item here: https://www.
The `sunburst` function generates a navigable sunburst chart from the
directories information retrieved from the model. The chart displays for
each directory the total number of files and the percentage of file known.
The size of the directory is defined by the total number of contents whereas
the color gradient is generated relying on the percentage of contents known.
"""
from typing import List, Dict, Tuple
from pathlib import PosixPath
from plotly.offline import offline # type: ignore
import plotly.graph_objects as go # type: ignore
import pandas as pd # type: ignore
import numpy as np # type: ignore
vlorentzUnsubmitted
Not Done
Inline Actions

module docstring must be before imports

vlorentz: module docstring must be before imports
def build_hierarchical_df(
dirs_dataframe: pd.DataFrame, levels: List[str],
metrics_columns: List[str], root_name: str) -> pd.DataFrame:
"""
Build a hierarchy of levels for Sunburst or Treemap charts.
vlorentzUnsubmitted
Not Done
Inline Actions

Could you add a description of each argument? (and possibly change the name df to express what it contains)

vlorentz: Could you add a description of each argument? (and possibly change the name `df` to express…
vlorentzUnsubmitted
Not Done
Inline Actions

renaming df to dataframe isn't more explicit about what it contains :/

vlorentz: renaming `df` to `dataframe` isn't more explicit about what it contains :/
For each directory the new dataframe will have the following
vlorentzUnsubmitted
Not Done
Inline Actions

"For each directory level stored, the new dataframe will have"?

vlorentz: "For each directory level store**d,** the new dataframe will have"?
information:
id: the directory name
parent: the parent directory of id
contents: the total number of contents of the directory id and
vlorentzUnsubmitted
Not Done
Inline Actions

Could you add an example of inputs, and what the outputs are? (like you did for generate_df)

vlorentz: Could you add an example of inputs, and what the outputs are? (like you did for `generate_df`)
the relative subdirectories
known: the percentage of contents known relative to computed
'contents'
Example:
Given the following dataframe:
.. code-block:: none
vlorentzUnsubmitted
Not Done
Inline Actions

forgot this print

vlorentz: forgot this print
lev0 lev1 contents known
'' '' 20 2 //root
kernel kernel/subdirker 5 0
telnet telnet/subdirtel 10 4
The output hierarchical dataframe will be like the following:
.. code-block:: none
vlorentzUnsubmitted
Not Done
Inline Actions

I think I understand what is going on here, but could you add a comment with an example of how the data is reshaped from df to df_tree step-by-step, for future readers?

vlorentz: I think I understand what is going on here, but could you add a comment with an example of how…
id parent contents known
20 10.00
vlorentzUnsubmitted
Not Done
Inline Actions

The data from the first df_trees will be copied into a new dataframe for each run of the loop. This is inefficient (quadratic complexity). You might want to build a list of dataframes and concatenate everything at the end.

vlorentz: The data from the first `df_tree`s will be copied into a new dataframe for each run of the loop.
kernel/subdirker kernel 5 0.00
telnet/subdirtel telnet 10 40.00
total 20 10.00
kernel total 5 0.00
telnet total 10 40.00
total 35 17.14
To create the hierarchical dataframe we need to iterate through
the dataframe given in input relying on the number of levels.
Based on the previous example we have to do two iterations:
vlorentzUnsubmitted
Not Done
Inline Actions

dirs_path: List[PosixPath]

vlorentz: `dirs_path: List[PosixPath]`
iteration 1
vlorentzUnsubmitted
Not Done
Inline Actions

also, would be nice to write in this docstring what data is in the dataframe

vlorentz: also, would be nice to write in this docstring what data is in the dataframe
The generated dataframe 'df_tree' will be:
.. code-block:: none
id parent contents known
20 10.0
kernel/subdirker kernel 5 0.0
telnet/subdirtel telnet 10 40.0
iteration 2
The generated dataframe 'df_tree' will be:
.. code-block:: none
id parent contents known
total 20 10.0
kernel total 5 0.0
vlorentzUnsubmitted
Not Done
Inline Actions

You could rename it generate_df_from_dirs.

And you can add a type annotation: dirs: Dict[PosixPath, Tuple[int, int]].

vlorentz: You could rename it `generate_df_from_dirs`. And you can add a type annotation: `dirs: Dict…
telnet total 10 40.0
Note that since we have reached the last level, the parent given
to the directory id is the directory root.
The 'total' row il computed by adding the number of contents of the
dataframe given in input and the average of the contents known on
vlorentzUnsubmitted
Not Done
Inline Actions

Great!

vlorentz: Great!
the total number of contents.
"""
def compute_known_percentage(contents: pd.Series, known: pd.Series
) -> pd.Series:
"""This function compute the percentage of known contents and generate
the new known column with the percentage values.
It also assures that if there is no contents inside a directory
the percentage is zero
"""
known_values = []
for idx, content_val in enumerate(contents):
if content_val == 0:
known_values.append(0)
else:
percentage = known[idx] / contents[idx] * 100
known_values.append(percentage)
return pd.Series(np.array(known_values))
complete_df = pd.DataFrame(columns=['id', 'parent', 'contents', 'known'])
# revert the level order to start from the deepest
levels = [level for level in reversed(levels)]
contents_col = metrics_columns[0]
vlorentzUnsubmitted
Not Done
Inline Actions

parts_list = list(dir_path.parts)

vlorentz: `parts_list = list(dir_path.parts)`
known_col = metrics_columns[1]
df_tree_list = []
for i, level in enumerate(levels):
df_tree = pd.DataFrame(columns=['id', 'parent', 'contents', 'known'])
dfg = dirs_dataframe.groupby(levels[i:]).sum()
dfg = dfg.reset_index()
df_tree['id'] = dfg[level].copy()
if i < len(levels) - 1:
# copy the parent directories (one level above)
df_tree['parent'] = dfg[levels[i+1]].copy()
else:
# last level reached
df_tree['parent'] = root_name
# copy the contents column
df_tree['contents'] = dfg[contents_col]
# compute the percentage relative to the contents
df_tree['known'] = compute_known_percentage(
dfg[contents_col], dfg[known_col])
df_tree_list.append(df_tree)
complete_df = complete_df.append(df_tree_list, ignore_index=True)
# create the main parent
total_contents = dirs_dataframe[contents_col].sum()
total_known = dirs_dataframe[known_col].sum()
total_avg = total_known / total_contents * 100
total = pd.Series(dict(id=root_name, parent='',
contents=total_contents,
known=total_avg))
complete_df = complete_df.append(total, ignore_index=True)
return complete_df
def compute_max_depth(dirs_path: List[PosixPath], root: PosixPath) -> int:
"""Compute the maximum depth level of the given directory paths.
Example: for `var/log/kernel/` the depth level is 3
"""
max_depth = 0
for dir_path in dirs_path:
if dir_path == root:
continue
dir_depth = len(dir_path.parts)
if dir_depth > max_depth:
max_depth = dir_depth
return max_depth
def generate_df_from_dirs(dirs: Dict[PosixPath, Tuple[int, int]],
columns: List[str], root: PosixPath, max_depth: int
) -> pd.DataFrame:
"""Generate a dataframe from the directories given in input.
Example:
given the following directories as input
.. code-block:: python
dirs = {
'/var/log/': (23, 2),
'/var/log/kernel': (5, 0),
'/var/log/telnet': (10, 3)
}
The generated dataframe will be:
.. code-block:: none
lev0 lev1 lev2 contents known
'var' 'var/log' '' 23 2
'var' 'var/log' 'var/log/kernel' 5 0
'var' 'var/log' 'var/log/telnet' 10 3
"""
def get_parents(path: PosixPath):
parts = path.parts[1:] if path.parts[0] == '/' else path.parts
for i in range(1, len(parts)+1):
yield '/'.join(parts[0:i])
def get_dirs_array():
for dir_path, contents_info in dirs.items():
empty_lvl = max_depth - len(dir_path.parts)
if dir_path == root:
# ignore the root but store contents information
yield ['']*(max_depth) + list(contents_info)
else:
yield list(get_parents(dir_path)) + \
['']*empty_lvl + \
list(contents_info)
df = pd.DataFrame(np.array(
[dir_array for dir_array in get_dirs_array()]), columns=columns)
df['contents'] = pd.to_numeric(df['contents'])
df['known'] = pd.to_numeric(df['known'])
return df
def sunburst(directories: Dict[PosixPath, Tuple[int, int]],
root: PosixPath) -> None:
"""Show the sunburst chart from the directories given in input.
"""
max_depth = compute_max_depth(list(directories.keys()), root)
metrics_columns = ['contents', 'known']
levels_columns = ['lev'+str(i) for i in range(max_depth)]
df_columns = levels_columns + metrics_columns
dirs_df = generate_df_from_dirs(directories, df_columns, root, max_depth)
hierarchical_df = build_hierarchical_df(
dirs_df, levels_columns, metrics_columns, str(root))
known_avg = dirs_df['known'].sum() / dirs_df['contents'].sum()
fig = go.Figure()
fig.add_trace(go.Sunburst(
labels=hierarchical_df['id'],
parents=hierarchical_df['parent'],
values=hierarchical_df['contents'],
branchvalues='total',
marker=dict(
colors=hierarchical_df['known'],
colorscale='RdBu',
cmid=known_avg),
hovertemplate='''<b>%{label}</b>
<br>Files: %{value}
<br>Known: <b>%{color:.2f}%</b>''',
name=''
))
offline.plot(fig, filename='sunburst.html')
View Options

swh/deposit/tests/api/test_parsers.py

  • This file was added.
# Copyright (C) 2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
"""
The purpose of this module is to display and to interact with the result of the
scanner contained in the model.
vlorentzUnsubmitted
Not Done
Inline Actions

this group of imports should be after the other ones. See the second item here: https://www.python.org/dev/peps/pep-0008/#imports

vlorentz: this group of imports should be after the other ones. See the second item here: https://www.
The `sunburst` function generates a navigable sunburst chart from the
directories information retrieved from the model. The chart displays for
each directory the total number of files and the percentage of file known.
The size of the directory is defined by the total number of contents whereas
the color gradient is generated relying on the percentage of contents known.
"""
from typing import List, Dict, Tuple
from pathlib import PosixPath
from plotly.offline import offline # type: ignore
import plotly.graph_objects as go # type: ignore
import pandas as pd # type: ignore
import numpy as np # type: ignore
vlorentzUnsubmitted
Not Done
Inline Actions

module docstring must be before imports

vlorentz: module docstring must be before imports
def build_hierarchical_df(
dirs_dataframe: pd.DataFrame, levels: List[str],
metrics_columns: List[str], root_name: str) -> pd.DataFrame:
"""
Build a hierarchy of levels for Sunburst or Treemap charts.
vlorentzUnsubmitted
Not Done
Inline Actions

Could you add a description of each argument? (and possibly change the name df to express what it contains)

vlorentz: Could you add a description of each argument? (and possibly change the name `df` to express…
vlorentzUnsubmitted
Not Done
Inline Actions

renaming df to dataframe isn't more explicit about what it contains :/

vlorentz: renaming `df` to `dataframe` isn't more explicit about what it contains :/
For each directory the new dataframe will have the following
vlorentzUnsubmitted
Not Done
Inline Actions

"For each directory level stored, the new dataframe will have"?

vlorentz: "For each directory level store**d,** the new dataframe will have"?
information:
id: the directory name
parent: the parent directory of id
contents: the total number of contents of the directory id and
vlorentzUnsubmitted
Not Done
Inline Actions

Could you add an example of inputs, and what the outputs are? (like you did for generate_df)

vlorentz: Could you add an example of inputs, and what the outputs are? (like you did for `generate_df`)
the relative subdirectories
known: the percentage of contents known relative to computed
'contents'
Example:
Given the following dataframe:
.. code-block:: none
vlorentzUnsubmitted
Not Done
Inline Actions

forgot this print

vlorentz: forgot this print
lev0 lev1 contents known
'' '' 20 2 //root
kernel kernel/subdirker 5 0
telnet telnet/subdirtel 10 4
The output hierarchical dataframe will be like the following:
.. code-block:: none
vlorentzUnsubmitted
Not Done
Inline Actions

I think I understand what is going on here, but could you add a comment with an example of how the data is reshaped from df to df_tree step-by-step, for future readers?

vlorentz: I think I understand what is going on here, but could you add a comment with an example of how…
id parent contents known
20 10.00
vlorentzUnsubmitted
Not Done
Inline Actions

The data from the first df_trees will be copied into a new dataframe for each run of the loop. This is inefficient (quadratic complexity). You might want to build a list of dataframes and concatenate everything at the end.

vlorentz: The data from the first `df_tree`s will be copied into a new dataframe for each run of the loop.
kernel/subdirker kernel 5 0.00
telnet/subdirtel telnet 10 40.00
total 20 10.00
kernel total 5 0.00
telnet total 10 40.00
total 35 17.14
To create the hierarchical dataframe we need to iterate through
the dataframe given in input relying on the number of levels.
Based on the previous example we have to do two iterations:
vlorentzUnsubmitted
Not Done
Inline Actions

dirs_path: List[PosixPath]

vlorentz: `dirs_path: List[PosixPath]`
iteration 1
vlorentzUnsubmitted
Not Done
Inline Actions

also, would be nice to write in this docstring what data is in the dataframe

vlorentz: also, would be nice to write in this docstring what data is in the dataframe
The generated dataframe 'df_tree' will be:
.. code-block:: none
id parent contents known
20 10.0
kernel/subdirker kernel 5 0.0
telnet/subdirtel telnet 10 40.0
iteration 2
The generated dataframe 'df_tree' will be:
.. code-block:: none
id parent contents known
total 20 10.0
kernel total 5 0.0
vlorentzUnsubmitted
Not Done
Inline Actions

You could rename it generate_df_from_dirs.

And you can add a type annotation: dirs: Dict[PosixPath, Tuple[int, int]].

vlorentz: You could rename it `generate_df_from_dirs`. And you can add a type annotation: `dirs: Dict…
telnet total 10 40.0
Note that since we have reached the last level, the parent given
to the directory id is the directory root.
The 'total' row il computed by adding the number of contents of the
dataframe given in input and the average of the contents known on
vlorentzUnsubmitted
Not Done
Inline Actions

Great!

vlorentz: Great!
the total number of contents.
"""
def compute_known_percentage(contents: pd.Series, known: pd.Series
) -> pd.Series:
"""This function compute the percentage of known contents and generate
the new known column with the percentage values.
It also assures that if there is no contents inside a directory
the percentage is zero
"""
known_values = []
for idx, content_val in enumerate(contents):
if content_val == 0:
known_values.append(0)
else:
percentage = known[idx] / contents[idx] * 100
known_values.append(percentage)
return pd.Series(np.array(known_values))
complete_df = pd.DataFrame(columns=['id', 'parent', 'contents', 'known'])
# revert the level order to start from the deepest
levels = [level for level in reversed(levels)]
contents_col = metrics_columns[0]
vlorentzUnsubmitted
Not Done
Inline Actions

parts_list = list(dir_path.parts)

vlorentz: `parts_list = list(dir_path.parts)`
known_col = metrics_columns[1]
df_tree_list = []
for i, level in enumerate(levels):
df_tree = pd.DataFrame(columns=['id', 'parent', 'contents', 'known'])
dfg = dirs_dataframe.groupby(levels[i:]).sum()
dfg = dfg.reset_index()
df_tree['id'] = dfg[level].copy()
if i < len(levels) - 1:
# copy the parent directories (one level above)
df_tree['parent'] = dfg[levels[i+1]].copy()
else:
# last level reached
df_tree['parent'] = root_name
# copy the contents column
df_tree['contents'] = dfg[contents_col]
# compute the percentage relative to the contents
df_tree['known'] = compute_known_percentage(
dfg[contents_col], dfg[known_col])
df_tree_list.append(df_tree)
complete_df = complete_df.append(df_tree_list, ignore_index=True)
# create the main parent
total_contents = dirs_dataframe[contents_col].sum()
total_known = dirs_dataframe[known_col].sum()
total_avg = total_known / total_contents * 100
total = pd.Series(dict(id=root_name, parent='',
contents=total_contents,
known=total_avg))
complete_df = complete_df.append(total, ignore_index=True)
return complete_df
def compute_max_depth(dirs_path: List[PosixPath], root: PosixPath) -> int:
"""Compute the maximum depth level of the given directory paths.
Example: for `var/log/kernel/` the depth level is 3
"""
max_depth = 0
for dir_path in dirs_path:
if dir_path == root:
continue
dir_depth = len(dir_path.parts)
if dir_depth > max_depth:
max_depth = dir_depth
return max_depth
def generate_df_from_dirs(dirs: Dict[PosixPath, Tuple[int, int]],
columns: List[str], root: PosixPath, max_depth: int
) -> pd.DataFrame:
"""Generate a dataframe from the directories given in input.
Example:
given the following directories as input
.. code-block:: python
dirs = {
'/var/log/': (23, 2),
'/var/log/kernel': (5, 0),
'/var/log/telnet': (10, 3)
}
The generated dataframe will be:
.. code-block:: none
lev0 lev1 lev2 contents known
'var' 'var/log' '' 23 2
'var' 'var/log' 'var/log/kernel' 5 0
'var' 'var/log' 'var/log/telnet' 10 3
"""
def get_parents(path: PosixPath):
parts = path.parts[1:] if path.parts[0] == '/' else path.parts
for i in range(1, len(parts)+1):
yield '/'.join(parts[0:i])
def get_dirs_array():
for dir_path, contents_info in dirs.items():
empty_lvl = max_depth - len(dir_path.parts)
if dir_path == root:
# ignore the root but store contents information
yield ['']*(max_depth) + list(contents_info)
else:
yield list(get_parents(dir_path)) + \
['']*empty_lvl + \
list(contents_info)
df = pd.DataFrame(np.array(
[dir_array for dir_array in get_dirs_array()]), columns=columns)
df['contents'] = pd.to_numeric(df['contents'])
df['known'] = pd.to_numeric(df['known'])
return df
def sunburst(directories: Dict[PosixPath, Tuple[int, int]],
root: PosixPath) -> None:
"""Show the sunburst chart from the directories given in input.
"""
max_depth = compute_max_depth(list(directories.keys()), root)
metrics_columns = ['contents', 'known']
levels_columns = ['lev'+str(i) for i in range(max_depth)]
df_columns = levels_columns + metrics_columns
dirs_df = generate_df_from_dirs(directories, df_columns, root, max_depth)
hierarchical_df = build_hierarchical_df(
dirs_df, levels_columns, metrics_columns, str(root))
known_avg = dirs_df['known'].sum() / dirs_df['contents'].sum()
fig = go.Figure()
fig.add_trace(go.Sunburst(
labels=hierarchical_df['id'],
parents=hierarchical_df['parent'],
values=hierarchical_df['contents'],
branchvalues='total',
marker=dict(
colors=hierarchical_df['known'],
colorscale='RdBu',
cmid=known_avg),
hovertemplate='''<b>%{label}</b>
<br>Files: %{value}
<br>Known: <b>%{color:.2f}%</b>''',
name=''
))
offline.plot(fig, filename='sunburst.html')
View Options

swh/deposit/tests/test_utils.py

  • This file was added.
# Copyright (C) 2020 The Software Heritage developers
# See the AUTHORS file at the top-level directory of this distribution
# License: GNU General Public License version 3, or any later version
# See top-level LICENSE file for more information
"""
The purpose of this module is to display and to interact with the result of the
scanner contained in the model.
vlorentzUnsubmitted
Not Done
Inline Actions

this group of imports should be after the other ones. See the second item here: https://www.python.org/dev/peps/pep-0008/#imports

vlorentz: this group of imports should be after the other ones. See the second item here: https://www.
The `sunburst` function generates a navigable sunburst chart from the
directories information retrieved from the model. The chart displays for
each directory the total number of files and the percentage of file known.
The size of the directory is defined by the total number of contents whereas
the color gradient is generated relying on the percentage of contents known.
"""
from typing import List, Dict, Tuple
from pathlib import PosixPath
from plotly.offline import offline # type: ignore
import plotly.graph_objects as go # type: ignore
import pandas as pd # type: ignore
import numpy as np # type: ignore
vlorentzUnsubmitted
Not Done
Inline Actions

module docstring must be before imports

vlorentz: module docstring must be before imports
def build_hierarchical_df(
dirs_dataframe: pd.DataFrame, levels: List[str],
metrics_columns: List[str], root_name: str) -> pd.DataFrame:
"""
Build a hierarchy of levels for Sunburst or Treemap charts.
vlorentzUnsubmitted
Not Done
Inline Actions

Could you add a description of each argument? (and possibly change the name df to express what it contains)

vlorentz: Could you add a description of each argument? (and possibly change the name `df` to express…
vlorentzUnsubmitted
Not Done
Inline Actions

renaming df to dataframe isn't more explicit about what it contains :/

vlorentz: renaming `df` to `dataframe` isn't more explicit about what it contains :/
For each directory the new dataframe will have the following
vlorentzUnsubmitted
Not Done
Inline Actions

"For each directory level stored, the new dataframe will have"?

vlorentz: "For each directory level store**d,** the new dataframe will have"?
information:
id: the directory name
parent: the parent directory of id
contents: the total number of contents of the directory id and
vlorentzUnsubmitted
Not Done
Inline Actions

Could you add an example of inputs, and what the outputs are? (like you did for generate_df)

vlorentz: Could you add an example of inputs, and what the outputs are? (like you did for `generate_df`)
the relative subdirectories
known: the percentage of contents known relative to computed
'contents'
Example:
Given the following dataframe:
.. code-block:: none
vlorentzUnsubmitted
Not Done
Inline Actions

forgot this print

vlorentz: forgot this print
lev0 lev1 contents known
'' '' 20 2 //root
kernel kernel/subdirker 5 0
telnet telnet/subdirtel 10 4
The output hierarchical dataframe will be like the following:
.. code-block:: none
vlorentzUnsubmitted
Not Done
Inline Actions

I think I understand what is going on here, but could you add a comment with an example of how the data is reshaped from df to df_tree step-by-step, for future readers?

vlorentz: I think I understand what is going on here, but could you add a comment with an example of how…
id parent contents known
20 10.00
vlorentzUnsubmitted
Not Done
Inline Actions

The data from the first df_trees will be copied into a new dataframe for each run of the loop. This is inefficient (quadratic complexity). You might want to build a list of dataframes and concatenate everything at the end.

vlorentz: The data from the first `df_tree`s will be copied into a new dataframe for each run of the loop.
kernel/subdirker kernel 5 0.00
telnet/subdirtel telnet 10 40.00
total 20 10.00
kernel total 5 0.00
telnet total 10 40.00
total 35 17.14
To create the hierarchical dataframe we need to iterate through
the dataframe given in input relying on the number of levels.
Based on the previous example we have to do two iterations:
vlorentzUnsubmitted
Not Done
Inline Actions

dirs_path: List[PosixPath]

vlorentz: `dirs_path: List[PosixPath]`
iteration 1
vlorentzUnsubmitted
Not Done
Inline Actions

also, would be nice to write in this docstring what data is in the dataframe

vlorentz: also, would be nice to write in this docstring what data is in the dataframe
The generated dataframe 'df_tree' will be:
.. code-block:: none
id parent contents known
20 10.0
kernel/subdirker kernel 5 0.0
telnet/subdirtel telnet 10 40.0
iteration 2
The generated dataframe 'df_tree' will be:
.. code-block:: none
id parent contents known
total 20 10.0
kernel total 5 0.0
vlorentzUnsubmitted
Not Done
Inline Actions

You could rename it generate_df_from_dirs.

And you can add a type annotation: dirs: Dict[PosixPath, Tuple[int, int]].

vlorentz: You could rename it `generate_df_from_dirs`. And you can add a type annotation: `dirs: Dict…
telnet total 10 40.0
Note that since we have reached the last level, the parent given
to the directory id is the directory root.
The 'total' row il computed by adding the number of contents of the
dataframe given in input and the average of the contents known on
vlorentzUnsubmitted
Not Done
Inline Actions

Great!

vlorentz: Great!
the total number of contents.
"""
def compute_known_percentage(contents: pd.Series, known: pd.Series
) -> pd.Series:
"""This function compute the percentage of known contents and generate
the new known column with the percentage values.
It also assures that if there is no contents inside a directory
the percentage is zero
"""
known_values = []
for idx, content_val in enumerate(contents):
if content_val == 0:
known_values.append(0)
else:
percentage = known[idx] / contents[idx] * 100
known_values.append(percentage)
return pd.Series(np.array(known_values))
complete_df = pd.DataFrame(columns=['id', 'parent', 'contents', 'known'])
# revert the level order to start from the deepest
levels = [level for level in reversed(levels)]
contents_col = metrics_columns[0]
vlorentzUnsubmitted
Not Done
Inline Actions

parts_list = list(dir_path.parts)

vlorentz: `parts_list = list(dir_path.parts)`
known_col = metrics_columns[1]
df_tree_list = []
for i, level in enumerate(levels):
df_tree = pd.DataFrame(columns=['id', 'parent', 'contents', 'known'])
dfg = dirs_dataframe.groupby(levels[i:]).sum()
dfg = dfg.reset_index()
df_tree['id'] = dfg[level].copy()
if i < len(levels) - 1:
# copy the parent directories (one level above)
df_tree['parent'] = dfg[levels[i+1]].copy()
else:
# last level reached
df_tree['parent'] = root_name
# copy the contents column
df_tree['contents'] = dfg[contents_col]
# compute the percentage relative to the contents
df_tree['known'] = compute_known_percentage(
dfg[contents_col], dfg[known_col])
df_tree_list.append(df_tree)
complete_df = complete_df.append(df_tree_list, ignore_index=True)
# create the main parent
total_contents = dirs_dataframe[contents_col].sum()
total_known = dirs_dataframe[known_col].sum()
total_avg = total_known / total_contents * 100
total = pd.Series(dict(id=root_name, parent='',
contents=total_contents,
known=total_avg))
complete_df = complete_df.append(total, ignore_index=True)
return complete_df
def compute_max_depth(dirs_path: List[PosixPath], root: PosixPath) -> int:
"""Compute the maximum depth level of the given directory paths.
Example: for `var/log/kernel/` the depth level is 3
"""
max_depth = 0
for dir_path in dirs_path:
if dir_path == root:
continue
dir_depth = len(dir_path.parts)
if dir_depth > max_depth:
max_depth = dir_depth
return max_depth
def generate_df_from_dirs(dirs: Dict[PosixPath, Tuple[int, int]],
columns: List[str], root: PosixPath, max_depth: int
) -> pd.DataFrame:
"""Generate a dataframe from the directories given in input.
Example:
given the following directories as input
.. code-block:: python
dirs = {
'/var/log/': (23, 2),
'/var/log/kernel': (5, 0),
'/var/log/telnet': (10, 3)
}
The generated dataframe will be:
.. code-block:: none
lev0 lev1 lev2 contents known
'var' 'var/log' '' 23 2
'var' 'var/log' 'var/log/kernel' 5 0
'var' 'var/log' 'var/log/telnet' 10 3
"""
def get_parents(path: PosixPath):
parts = path.parts[1:] if path.parts[0] == '/' else path.parts
for i in range(1, len(parts)+1):
yield '/'.join(parts[0:i])
def get_dirs_array():
for dir_path, contents_info in dirs.items():
empty_lvl = max_depth - len(dir_path.parts)
if dir_path == root:
# ignore the root but store contents information
yield ['']*(max_depth) + list(contents_info)
else:
yield list(get_parents(dir_path)) + \
['']*empty_lvl + \
list(contents_info)
df = pd.DataFrame(np.array(
[dir_array for dir_array in get_dirs_array()]), columns=columns)
df['contents'] = pd.to_numeric(df['contents'])
df['known'] = pd.to_numeric(df['known'])
return df
def sunburst(directories: Dict[PosixPath, Tuple[int, int]],
root: PosixPath) -> None:
"""Show the sunburst chart from the directories given in input.
"""
max_depth = compute_max_depth(list(directories.keys()), root)
metrics_columns = ['contents', 'known']
levels_columns = ['lev'+str(i) for i in range(max_depth)]
df_columns = levels_columns + metrics_columns
dirs_df = generate_df_from_dirs(directories, df_columns, root, max_depth)
hierarchical_df = build_hierarchical_df(
dirs_df, levels_columns, metrics_columns, str(root))
known_avg = dirs_df['known'].sum() / dirs_df['contents'].sum()
fig = go.Figure()
fig.add_trace(go.Sunburst(
labels=hierarchical_df['id'],
parents=hierarchical_df['parent'],
values=hierarchical_df['contents'],
branchvalues='total',
marker=dict(
colors=hierarchical_df['known'],
colorscale='RdBu',
cmid=known_avg),
hovertemplate='''<b>%{label}</b>
<br>Files: %{value}
<br>Known: <b>%{color:.2f}%</b>''',
name=''
))
offline.plot(fig, filename='sunburst.html')
View Options

swh/deposit/utils.py

Loading...
About · Developer information · Legal