Running setup.py bdist_wheel for confluent-kafka: started
  Running setup.py bdist_wheel for confluent-kafka: finished with status 'error'
  Complete output from command /home/jenkins/swh-web-env/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-iv7br_vy/confluent-kafka/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/pip-wheel-1fkykhft --python-tag cp37:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.7
  creating build/lib.linux-x86_64-3.7/confluent_kafka
  copying confluent_kafka/__init__.py -> build/lib.linux-x86_64-3.7/confluent_kafka
  creating build/lib.linux-x86_64-3.7/confluent_kafka/admin
  copying confluent_kafka/admin/__init__.py -> build/lib.linux-x86_64-3.7/confluent_kafka/admin
  creating build/lib.linux-x86_64-3.7/confluent_kafka/kafkatest
  copying confluent_kafka/kafkatest/verifiable_client.py -> build/lib.linux-x86_64-3.7/confluent_kafka/kafkatest
  copying confluent_kafka/kafkatest/__init__.py -> build/lib.linux-x86_64-3.7/confluent_kafka/kafkatest
  copying confluent_kafka/kafkatest/verifiable_consumer.py -> build/lib.linux-x86_64-3.7/confluent_kafka/kafkatest
  copying confluent_kafka/kafkatest/verifiable_producer.py -> build/lib.linux-x86_64-3.7/confluent_kafka/kafkatest
  creating build/lib.linux-x86_64-3.7/confluent_kafka/avro
  copying confluent_kafka/avro/load.py -> build/lib.linux-x86_64-3.7/confluent_kafka/avro
  copying confluent_kafka/avro/cached_schema_registry_client.py -> build/lib.linux-x86_64-3.7/confluent_kafka/avro
  copying confluent_kafka/avro/__init__.py -> build/lib.linux-x86_64-3.7/confluent_kafka/avro
  copying confluent_kafka/avro/error.py -> build/lib.linux-x86_64-3.7/confluent_kafka/avro
  creating build/lib.linux-x86_64-3.7/confluent_kafka/avro/serializer
  copying confluent_kafka/avro/serializer/__init__.py -> build/lib.linux-x86_64-3.7/confluent_kafka/avro/serializer
  copying confluent_kafka/avro/serializer/message_serializer.py -> build/lib.linux-x86_64-3.7/confluent_kafka/avro/serializer
  running build_ext
  building 'confluent_kafka.cimpl' extension
  creating build/temp.linux-x86_64-3.7
  creating build/temp.linux-x86_64-3.7/confluent_kafka
  creating build/temp.linux-x86_64-3.7/confluent_kafka/src
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/jenkins/swh-web-env/include -I/usr/include/python3.7m -c confluent_kafka/src/confluent_kafka.c -o build/temp.linux-x86_64-3.7/confluent_kafka/src/confluent_kafka.o
  In file included from confluent_kafka/src/confluent_kafka.c:17:
  confluent_kafka/src/confluent_kafka.h:22:10: fatal error: librdkafka/rdkafka.h: No such file or directory
   #include <librdkafka/rdkafka.h>
            ^~~~~~~~~~~~~~~~~~~~~~
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for confluent-kafka
  Running setup.py clean for confluent-kafka
  Running setup.py bdist_wheel for psutil: started
  Running setup.py bdist_wheel for psutil: finished with status 'done'
  Stored in directory: /home/jenkins/.cache/pip/wheels/52/41/b0/bf50409fe2b1d3b79afa3eed71b54b3e30fe5b695db2c7ba2e

Nevertheless, inside my virtualenv, the linux binary wheels are correctly downloaded:

13:40 $ pip install --upgrade confluent_kafka
Collecting confluent_kafka
  Downloading https://files.pythonhosted.org/packages/1e/87/2d4d49f2aa93eed080e0a7fe3f37e9e1e228199c6381357928fe45876884/confluent_kafka-1.3.0-cp37-cp37m-manylinux2010_x86_64.whl (9.1MB)
     |████████████████████████████████| 9.1MB 13.7MB/s 
Installing collected packages: confluent-kafka
  Found existing installation: confluent-kafka 1.2.0
    Uninstalling confluent-kafka-1.2.0:
      Successfully uninstalled confluent-kafka-1.2.0
Successfully installed confluent-kafka-1.3.0

The manylinux2010 platform tag is quite recent so upgrading pip to its latest version in the CI job should fix the issue.

Update: Solution confirmed by hot patching the CI job configuration, D2450 created to update the concerned Jenkins jobs.

anlambert mentioned this in D2450: jobs/templates/swh-cypress: Ensure to use latest pip version.Dec 13 2019, 1:59 PM

Build is green
See https://jenkins.softwareheritage.org/job/DWAPPS/job/cypress-diff/451/ for more details.

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/cypress-diff/453/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/cypress-diff/453/console

Rebase

the api_request_factory fixture doesn't seem to be used in all the tests you added it for. Intentional?

This revision is now accepted and ready to land.Dec 13 2019, 2:26 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/cypress-diff/454/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/cypress-diff/454/console

Harbormaster failed remote builds in B9754: Diff 8665!Dec 13 2019, 2:27 PM

Build is green
See https://jenkins.softwareheritage.org/job/DWAPPS/job/tox/854/ for more details.

In D2449#57879, @vlorentz wrote:

the api_request_factory fixture doesn't seem to be used in all the tests you added it for. Intentional?

It is used in every test it is declared, you got mistaken by the Phabricator diff that truncates the files.

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/cypress-diff/455/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/cypress-diff/455/console

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/cypress-diff/456/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/cypress-diff/456/console

In D2449#57885, @anlambert wrote:

In D2449#57879, @vlorentz wrote:

the api_request_factory fixture doesn't seem to be used in all the tests you added it for. Intentional?

It is used in every test it is declared, you got mistaken by the Phabricator diff that truncates the files.

Indeed. But it was also Firefox's fault for not updating its list of search matches after I unfolded these.

This revision was landed with ongoing or failed builds.Dec 13 2019, 3:01 PM

Closed by commit rDWAPPS50ab7013714f: api: Make next links in response headers contain full URLs (authored by anlambert). · Explain Why

This revision was automatically updated to reflect the committed changes.

anlambert added a commit: rDWAPPS50ab7013714f: api: Make next links in response headers contain full URLs.

Revision Contents
Changeset List

Path

Size

swh/

web/

api/

views/

5 lines

22 lines

7 lines

4 lines

tests/

api/

views/

test_content.py

9 lines

test_revision.py

8 lines

test_snapshot.py

10 lines

Diff 8658

View Options

swh/web/api/views/content.py

This file was added.

				# Copyright (C) 2020 The Software Heritage developers
				# See the AUTHORS file at the top-level directory of this distribution
				# License: GNU General Public License version 3, or any later version
				# See top-level LICENSE file for more information

				"""
				The purpose of this module is to display and to interact with the result of the
				scanner contained in the model.
				vlorentzUnsubmitted Not Done Inline Actions this group of imports should be after the other ones. See the second item here: https://www.python.org/dev/peps/pep-0008/#imports vlorentz: this group of imports should be after the other ones. See the second item here: https://www.

				The `sunburst` function generates a navigable sunburst chart from the
				directories information retrieved from the model. The chart displays for
				each directory the total number of files and the percentage of file known.

				The size of the directory is defined by the total number of contents whereas
				the color gradient is generated relying on the percentage of contents known.
				"""

				from typing import List, Dict, Tuple
				from pathlib import PosixPath

				from plotly.offline import offline # type: ignore
				import plotly.graph_objects as go # type: ignore
				import pandas as pd # type: ignore
				import numpy as np # type: ignore
				vlorentzUnsubmitted Not Done Inline Actions module docstring must be before imports vlorentz: module docstring must be before imports


				def build_hierarchical_df(
				dirs_dataframe: pd.DataFrame, levels: List[str],
				metrics_columns: List[str], root_name: str) -> pd.DataFrame:
				"""
				Build a hierarchy of levels for Sunburst or Treemap charts.
				vlorentzUnsubmitted Not Done Inline Actions Could you add a description of each argument? (and possibly change the name `df` to express what it contains) vlorentz: Could you add a description of each argument? (and possibly change the name `df` to express…
				vlorentzUnsubmitted Not Done Inline Actions renaming `df` to `dataframe` isn't more explicit about what it contains :/ vlorentz: renaming `df` to `dataframe` isn't more explicit about what it contains :/

				For each directory the new dataframe will have the following
				vlorentzUnsubmitted Not Done Inline Actions "For each directory level stored, the new dataframe will have"? vlorentz: "For each directory level stored, the new dataframe will have"?
				information:

				id: the directory name
				parent: the parent directory of id
				contents: the total number of contents of the directory id and
				vlorentzUnsubmitted Not Done Inline Actions Could you add an example of inputs, and what the outputs are? (like you did for `generate_df`) vlorentz: Could you add an example of inputs, and what the outputs are? (like you did for `generate_df`)
				the relative subdirectories
				known: the percentage of contents known relative to computed
				'contents'

				Example:
				Given the following dataframe:

				.. code-block:: none

				vlorentzUnsubmitted Not Done Inline Actions forgot this print vlorentz: forgot this print
				lev0 lev1 contents known
				'' '' 20 2 //root
				kernel kernel/subdirker 5 0
				telnet telnet/subdirtel 10 4

				The output hierarchical dataframe will be like the following:

				.. code-block:: none
				vlorentzUnsubmitted Not Done Inline Actions I think I understand what is going on here, but could you add a comment with an example of how the data is reshaped from `df` to `df_tree` step-by-step, for future readers? vlorentz: I think I understand what is going on here, but could you add a comment with an example of how…

				id parent contents known
				20 10.00
				vlorentzUnsubmitted Not Done Inline Actions The data from the first `df_tree`s will be copied into a new dataframe for each run of the loop. This is inefficient (quadratic complexity). You might want to build a list of dataframes and concatenate everything at the end. vlorentz: The data from the first `df_tree`s will be copied into a new dataframe for each run of the loop.
				kernel/subdirker kernel 5 0.00
				telnet/subdirtel telnet 10 40.00
				total 20 10.00
				kernel total 5 0.00
				telnet total 10 40.00
				total 35 17.14

				To create the hierarchical dataframe we need to iterate through
				the dataframe given in input relying on the number of levels.

				Based on the previous example we have to do two iterations:

				vlorentzUnsubmitted Not Done Inline Actions `dirs_path: List[PosixPath]` vlorentz: `dirs_path: List[PosixPath]`
				iteration 1
				vlorentzUnsubmitted Not Done Inline Actions also, would be nice to write in this docstring what data is in the dataframe vlorentz: also, would be nice to write in this docstring what data is in the dataframe
				The generated dataframe 'df_tree' will be:

				.. code-block:: none

				id parent contents known
				20 10.0
				kernel/subdirker kernel 5 0.0
				telnet/subdirtel telnet 10 40.0

				iteration 2
				The generated dataframe 'df_tree' will be:

				.. code-block:: none

				id parent contents known
				total 20 10.0
				kernel total 5 0.0
				vlorentzUnsubmitted Not Done Inline Actions You could rename it `generate_df_from_dirs`. And you can add a type annotation: `dirs: Dict[PosixPath, Tuple[int, int]]`. vlorentz: You could rename it `generate_df_from_dirs`. And you can add a type annotation: `dirs: Dict…
				telnet total 10 40.0

				Note that since we have reached the last level, the parent given
				to the directory id is the directory root.

				The 'total' row il computed by adding the number of contents of the
				dataframe given in input and the average of the contents known on
				vlorentzUnsubmitted Not Done Inline Actions Great! vlorentz: Great!
				the total number of contents.

				"""
				def compute_known_percentage(contents: pd.Series, known: pd.Series
				) -> pd.Series:
				"""This function compute the percentage of known contents and generate
				the new known column with the percentage values.

				It also assures that if there is no contents inside a directory
				the percentage is zero

				"""
				known_values = []
				for idx, content_val in enumerate(contents):
				if content_val == 0:
				known_values.append(0)
				else:
				percentage = known[idx] / contents[idx] * 100
				known_values.append(percentage)

				return pd.Series(np.array(known_values))

				complete_df = pd.DataFrame(columns=['id', 'parent', 'contents', 'known'])
				# revert the level order to start from the deepest
				levels = [level for level in reversed(levels)]
				contents_col = metrics_columns[0]
				vlorentzUnsubmitted Not Done Inline Actions `parts_list = list(dir_path.parts)` vlorentz: `parts_list = list(dir_path.parts)`
				known_col = metrics_columns[1]

				df_tree_list = []
				for i, level in enumerate(levels):
				df_tree = pd.DataFrame(columns=['id', 'parent', 'contents', 'known'])
				dfg = dirs_dataframe.groupby(levels[i:]).sum()
				dfg = dfg.reset_index()
				df_tree['id'] = dfg[level].copy()
				if i < len(levels) - 1:
				# copy the parent directories (one level above)
				df_tree['parent'] = dfg[levels[i+1]].copy()
				else:
				# last level reached
				df_tree['parent'] = root_name

				# copy the contents column
				df_tree['contents'] = dfg[contents_col]
				# compute the percentage relative to the contents
				df_tree['known'] = compute_known_percentage(
				dfg[contents_col], dfg[known_col])

				df_tree_list.append(df_tree)

				complete_df = complete_df.append(df_tree_list, ignore_index=True)

				# create the main parent
				total_contents = dirs_dataframe[contents_col].sum()
				total_known = dirs_dataframe[known_col].sum()
				total_avg = total_known / total_contents * 100

				total = pd.Series(dict(id=root_name, parent='',
				contents=total_contents,
				known=total_avg))

				complete_df = complete_df.append(total, ignore_index=True)

				return complete_df


				def compute_max_depth(dirs_path: List[PosixPath], root: PosixPath) -> int:
				"""Compute the maximum depth level of the given directory paths.

				Example: for `var/log/kernel/` the depth level is 3

				"""
				max_depth = 0
				for dir_path in dirs_path:
				if dir_path == root:
				continue

				dir_depth = len(dir_path.parts)
				if dir_depth > max_depth:
				max_depth = dir_depth

				return max_depth


				def generate_df_from_dirs(dirs: Dict[PosixPath, Tuple[int, int]],
				columns: List[str], root: PosixPath, max_depth: int
				) -> pd.DataFrame:
				"""Generate a dataframe from the directories given in input.

				Example:
				given the following directories as input

				.. code-block:: python

				dirs = {
				'/var/log/': (23, 2),
				'/var/log/kernel': (5, 0),
				'/var/log/telnet': (10, 3)
				}

				The generated dataframe will be:

				.. code-block:: none

				lev0 lev1 lev2 contents known
				'var' 'var/log' '' 23 2
				'var' 'var/log' 'var/log/kernel' 5 0
				'var' 'var/log' 'var/log/telnet' 10 3

				"""
				def get_parents(path: PosixPath):
				parts = path.parts[1:] if path.parts[0] == '/' else path.parts

				for i in range(1, len(parts)+1):
				yield '/'.join(parts[0:i])

				def get_dirs_array():
				for dir_path, contents_info in dirs.items():
				empty_lvl = max_depth - len(dir_path.parts)

				if dir_path == root:
				# ignore the root but store contents information
				yield ['']*(max_depth) + list(contents_info)
				else:
				yield list(get_parents(dir_path)) + \
				['']*empty_lvl + \
				list(contents_info)

				df = pd.DataFrame(np.array(
				[dir_array for dir_array in get_dirs_array()]), columns=columns)

				df['contents'] = pd.to_numeric(df['contents'])
				df['known'] = pd.to_numeric(df['known'])

				return df


				def sunburst(directories: Dict[PosixPath, Tuple[int, int]],
				root: PosixPath) -> None:
				"""Show the sunburst chart from the directories given in input.

				"""
				max_depth = compute_max_depth(list(directories.keys()), root)
				metrics_columns = ['contents', 'known']
				levels_columns = ['lev'+str(i) for i in range(max_depth)]

				df_columns = levels_columns + metrics_columns
				dirs_df = generate_df_from_dirs(directories, df_columns, root, max_depth)

				hierarchical_df = build_hierarchical_df(
				dirs_df, levels_columns, metrics_columns, str(root))
				known_avg = dirs_df['known'].sum() / dirs_df['contents'].sum()

				fig = go.Figure()
				fig.add_trace(go.Sunburst(
				labels=hierarchical_df['id'],
				parents=hierarchical_df['parent'],
				values=hierarchical_df['contents'],
				branchvalues='total',
				marker=dict(
				colors=hierarchical_df['known'],
				colorscale='RdBu',
				cmid=known_avg),
				hovertemplate='''<b>%{label}</b>
				<br>Files: %{value}
				<br>Known: <b>%{color:.2f}%</b>''',
				name=''
				))

				offline.plot(fig, filename='sunburst.html')

View Options

swh/web/api/views/origin.py

This file was added.

				# Copyright (C) 2020 The Software Heritage developers
				# See the AUTHORS file at the top-level directory of this distribution
				# License: GNU General Public License version 3, or any later version
				# See top-level LICENSE file for more information

				"""
				The purpose of this module is to display and to interact with the result of the
				scanner contained in the model.
				vlorentzUnsubmitted Not Done Inline Actions this group of imports should be after the other ones. See the second item here: https://www.python.org/dev/peps/pep-0008/#imports vlorentz: this group of imports should be after the other ones. See the second item here: https://www.

				The `sunburst` function generates a navigable sunburst chart from the
				directories information retrieved from the model. The chart displays for
				each directory the total number of files and the percentage of file known.

				The size of the directory is defined by the total number of contents whereas
				the color gradient is generated relying on the percentage of contents known.
				"""

				from typing import List, Dict, Tuple
				from pathlib import PosixPath

				from plotly.offline import offline # type: ignore
				import plotly.graph_objects as go # type: ignore
				import pandas as pd # type: ignore
				import numpy as np # type: ignore
				vlorentzUnsubmitted Not Done Inline Actions module docstring must be before imports vlorentz: module docstring must be before imports


				def build_hierarchical_df(
				dirs_dataframe: pd.DataFrame, levels: List[str],
				metrics_columns: List[str], root_name: str) -> pd.DataFrame:
				"""
				Build a hierarchy of levels for Sunburst or Treemap charts.
				vlorentzUnsubmitted Not Done Inline Actions Could you add a description of each argument? (and possibly change the name `df` to express what it contains) vlorentz: Could you add a description of each argument? (and possibly change the name `df` to express…
				vlorentzUnsubmitted Not Done Inline Actions renaming `df` to `dataframe` isn't more explicit about what it contains :/ vlorentz: renaming `df` to `dataframe` isn't more explicit about what it contains :/

				For each directory the new dataframe will have the following
				vlorentzUnsubmitted Not Done Inline Actions "For each directory level stored, the new dataframe will have"? vlorentz: "For each directory level stored, the new dataframe will have"?
				information:

				id: the directory name
				parent: the parent directory of id
				contents: the total number of contents of the directory id and
				vlorentzUnsubmitted Not Done Inline Actions Could you add an example of inputs, and what the outputs are? (like you did for `generate_df`) vlorentz: Could you add an example of inputs, and what the outputs are? (like you did for `generate_df`)
				the relative subdirectories
				known: the percentage of contents known relative to computed
				'contents'

				Example:
				Given the following dataframe:

				.. code-block:: none

				vlorentzUnsubmitted Not Done Inline Actions forgot this print vlorentz: forgot this print
				lev0 lev1 contents known
				'' '' 20 2 //root
				kernel kernel/subdirker 5 0
				telnet telnet/subdirtel 10 4

				The output hierarchical dataframe will be like the following:

				.. code-block:: none
				vlorentzUnsubmitted Not Done Inline Actions I think I understand what is going on here, but could you add a comment with an example of how the data is reshaped from `df` to `df_tree` step-by-step, for future readers? vlorentz: I think I understand what is going on here, but could you add a comment with an example of how…

				id parent contents known
				20 10.00
				vlorentzUnsubmitted Not Done Inline Actions The data from the first `df_tree`s will be copied into a new dataframe for each run of the loop. This is inefficient (quadratic complexity). You might want to build a list of dataframes and concatenate everything at the end. vlorentz: The data from the first `df_tree`s will be copied into a new dataframe for each run of the loop.
				kernel/subdirker kernel 5 0.00
				telnet/subdirtel telnet 10 40.00
				total 20 10.00
				kernel total 5 0.00
				telnet total 10 40.00
				total 35 17.14

				To create the hierarchical dataframe we need to iterate through
				the dataframe given in input relying on the number of levels.

				Based on the previous example we have to do two iterations:

				vlorentzUnsubmitted Not Done Inline Actions `dirs_path: List[PosixPath]` vlorentz: `dirs_path: List[PosixPath]`
				iteration 1
				vlorentzUnsubmitted Not Done Inline Actions also, would be nice to write in this docstring what data is in the dataframe vlorentz: also, would be nice to write in this docstring what data is in the dataframe
				The generated dataframe 'df_tree' will be:

				.. code-block:: none

				id parent contents known
				20 10.0
				kernel/subdirker kernel 5 0.0
				telnet/subdirtel telnet 10 40.0

				iteration 2
				The generated dataframe 'df_tree' will be:

				.. code-block:: none

				id parent contents known
				total 20 10.0
				kernel total 5 0.0
				vlorentzUnsubmitted Not Done Inline Actions You could rename it `generate_df_from_dirs`. And you can add a type annotation: `dirs: Dict[PosixPath, Tuple[int, int]]`. vlorentz: You could rename it `generate_df_from_dirs`. And you can add a type annotation: `dirs: Dict…
				telnet total 10 40.0

				Note that since we have reached the last level, the parent given
				to the directory id is the directory root.

				The 'total' row il computed by adding the number of contents of the
				dataframe given in input and the average of the contents known on
				vlorentzUnsubmitted Not Done Inline Actions Great! vlorentz: Great!
				the total number of contents.

				"""
				def compute_known_percentage(contents: pd.Series, known: pd.Series
				) -> pd.Series:
				"""This function compute the percentage of known contents and generate
				the new known column with the percentage values.

				It also assures that if there is no contents inside a directory
				the percentage is zero

				"""
				known_values = []
				for idx, content_val in enumerate(contents):
				if content_val == 0:
				known_values.append(0)
				else:
				percentage = known[idx] / contents[idx] * 100
				known_values.append(percentage)

				return pd.Series(np.array(known_values))

				complete_df = pd.DataFrame(columns=['id', 'parent', 'contents', 'known'])
				# revert the level order to start from the deepest
				levels = [level for level in reversed(levels)]
				contents_col = metrics_columns[0]
				vlorentzUnsubmitted Not Done Inline Actions `parts_list = list(dir_path.parts)` vlorentz: `parts_list = list(dir_path.parts)`
				known_col = metrics_columns[1]

				df_tree_list = []
				for i, level in enumerate(levels):
				df_tree = pd.DataFrame(columns=['id', 'parent', 'contents', 'known'])
				dfg = dirs_dataframe.groupby(levels[i:]).sum()
				dfg = dfg.reset_index()
				df_tree['id'] = dfg[level].copy()
				if i < len(levels) - 1:
				# copy the parent directories (one level above)
				df_tree['parent'] = dfg[levels[i+1]].copy()
				else:
				# last level reached
				df_tree['parent'] = root_name

				# copy the contents column
				df_tree['contents'] = dfg[contents_col]
				# compute the percentage relative to the contents
				df_tree['known'] = compute_known_percentage(
				dfg[contents_col], dfg[known_col])

				df_tree_list.append(df_tree)

				complete_df = complete_df.append(df_tree_list, ignore_index=True)

				# create the main parent
				total_contents = dirs_dataframe[contents_col].sum()
				total_known = dirs_dataframe[known_col].sum()
				total_avg = total_known / total_contents * 100

				total = pd.Series(dict(id=root_name, parent='',
				contents=total_contents,
				known=total_avg))

				complete_df = complete_df.append(total, ignore_index=True)

				return complete_df


				def compute_max_depth(dirs_path: List[PosixPath], root: PosixPath) -> int:
				"""Compute the maximum depth level of the given directory paths.

				Example: for `var/log/kernel/` the depth level is 3

				"""
				max_depth = 0
				for dir_path in dirs_path:
				if dir_path == root:
				continue

				dir_depth = len(dir_path.parts)
				if dir_depth > max_depth:
				max_depth = dir_depth

				return max_depth


				def generate_df_from_dirs(dirs: Dict[PosixPath, Tuple[int, int]],
				columns: List[str], root: PosixPath, max_depth: int
				) -> pd.DataFrame:
				"""Generate a dataframe from the directories given in input.

				Example:
				given the following directories as input

				.. code-block:: python

				dirs = {
				'/var/log/': (23, 2),
				'/var/log/kernel': (5, 0),
				'/var/log/telnet': (10, 3)
				}

				The generated dataframe will be:

				.. code-block:: none

				lev0 lev1 lev2 contents known
				'var' 'var/log' '' 23 2
				'var' 'var/log' 'var/log/kernel' 5 0
				'var' 'var/log' 'var/log/telnet' 10 3

				"""
				def get_parents(path: PosixPath):
				parts = path.parts[1:] if path.parts[0] == '/' else path.parts

				for i in range(1, len(parts)+1):
				yield '/'.join(parts[0:i])

				def get_dirs_array():
				for dir_path, contents_info in dirs.items():
				empty_lvl = max_depth - len(dir_path.parts)

				if dir_path == root:
				# ignore the root but store contents information
				yield ['']*(max_depth) + list(contents_info)
				else:
				yield list(get_parents(dir_path)) + \
				['']*empty_lvl + \
				list(contents_info)

				df = pd.DataFrame(np.array(
				[dir_array for dir_array in get_dirs_array()]), columns=columns)

				df['contents'] = pd.to_numeric(df['contents'])
				df['known'] = pd.to_numeric(df['known'])

				return df


				def sunburst(directories: Dict[PosixPath, Tuple[int, int]],
				root: PosixPath) -> None:
				"""Show the sunburst chart from the directories given in input.

				"""
				max_depth = compute_max_depth(list(directories.keys()), root)
				metrics_columns = ['contents', 'known']
				levels_columns = ['lev'+str(i) for i in range(max_depth)]

				df_columns = levels_columns + metrics_columns
				dirs_df = generate_df_from_dirs(directories, df_columns, root, max_depth)

				hierarchical_df = build_hierarchical_df(
				dirs_df, levels_columns, metrics_columns, str(root))
				known_avg = dirs_df['known'].sum() / dirs_df['contents'].sum()

				fig = go.Figure()
				fig.add_trace(go.Sunburst(
				labels=hierarchical_df['id'],
				parents=hierarchical_df['parent'],
				values=hierarchical_df['contents'],
				branchvalues='total',
				marker=dict(
				colors=hierarchical_df['known'],
				colorscale='RdBu',
				cmid=known_avg),
				hovertemplate='''<b>%{label}</b>
				<br>Files: %{value}
				<br>Known: <b>%{color:.2f}%</b>''',
				name=''
				))

				offline.plot(fig, filename='sunburst.html')

View Options

swh/web/api/views/revision.py

This file was added.

				# Copyright (C) 2020 The Software Heritage developers
				# See the AUTHORS file at the top-level directory of this distribution
				# License: GNU General Public License version 3, or any later version
				# See top-level LICENSE file for more information

				"""
				The purpose of this module is to display and to interact with the result of the
				scanner contained in the model.
				vlorentzUnsubmitted Not Done Inline Actions this group of imports should be after the other ones. See the second item here: https://www.python.org/dev/peps/pep-0008/#imports vlorentz: this group of imports should be after the other ones. See the second item here: https://www.

				The `sunburst` function generates a navigable sunburst chart from the
				directories information retrieved from the model. The chart displays for
				each directory the total number of files and the percentage of file known.

				The size of the directory is defined by the total number of contents whereas
				the color gradient is generated relying on the percentage of contents known.
				"""

				from typing import List, Dict, Tuple
				from pathlib import PosixPath

				from plotly.offline import offline # type: ignore
				import plotly.graph_objects as go # type: ignore
				import pandas as pd # type: ignore
				import numpy as np # type: ignore
				vlorentzUnsubmitted Not Done Inline Actions module docstring must be before imports vlorentz: module docstring must be before imports


				def build_hierarchical_df(
				dirs_dataframe: pd.DataFrame, levels: List[str],
				metrics_columns: List[str], root_name: str) -> pd.DataFrame:
				"""
				Build a hierarchy of levels for Sunburst or Treemap charts.
				vlorentzUnsubmitted Not Done Inline Actions Could you add a description of each argument? (and possibly change the name `df` to express what it contains) vlorentz: Could you add a description of each argument? (and possibly change the name `df` to express…
				vlorentzUnsubmitted Not Done Inline Actions renaming `df` to `dataframe` isn't more explicit about what it contains :/ vlorentz: renaming `df` to `dataframe` isn't more explicit about what it contains :/

				For each directory the new dataframe will have the following
				vlorentzUnsubmitted Not Done Inline Actions "For each directory level stored, the new dataframe will have"? vlorentz: "For each directory level stored, the new dataframe will have"?
				information:

				id: the directory name
				parent: the parent directory of id
				contents: the total number of contents of the directory id and
				vlorentzUnsubmitted Not Done Inline Actions Could you add an example of inputs, and what the outputs are? (like you did for `generate_df`) vlorentz: Could you add an example of inputs, and what the outputs are? (like you did for `generate_df`)
				the relative subdirectories
				known: the percentage of contents known relative to computed
				'contents'

				Example:
				Given the following dataframe:

				.. code-block:: none

				vlorentzUnsubmitted Not Done Inline Actions forgot this print vlorentz: forgot this print
				lev0 lev1 contents known
				'' '' 20 2 //root
				kernel kernel/subdirker 5 0
				telnet telnet/subdirtel 10 4

				The output hierarchical dataframe will be like the following:

				.. code-block:: none
				vlorentzUnsubmitted Not Done Inline Actions I think I understand what is going on here, but could you add a comment with an example of how the data is reshaped from `df` to `df_tree` step-by-step, for future readers? vlorentz: I think I understand what is going on here, but could you add a comment with an example of how…

				id parent contents known
				20 10.00
				vlorentzUnsubmitted Not Done Inline Actions The data from the first `df_tree`s will be copied into a new dataframe for each run of the loop. This is inefficient (quadratic complexity). You might want to build a list of dataframes and concatenate everything at the end. vlorentz: The data from the first `df_tree`s will be copied into a new dataframe for each run of the loop.
				kernel/subdirker kernel 5 0.00
				telnet/subdirtel telnet 10 40.00
				total 20 10.00
				kernel total 5 0.00
				telnet total 10 40.00
				total 35 17.14

				To create the hierarchical dataframe we need to iterate through
				the dataframe given in input relying on the number of levels.

				Based on the previous example we have to do two iterations:

				vlorentzUnsubmitted Not Done Inline Actions `dirs_path: List[PosixPath]` vlorentz: `dirs_path: List[PosixPath]`
				iteration 1
				vlorentzUnsubmitted Not Done Inline Actions also, would be nice to write in this docstring what data is in the dataframe vlorentz: also, would be nice to write in this docstring what data is in the dataframe
				The generated dataframe 'df_tree' will be:

				.. code-block:: none

				id parent contents known
				20 10.0
				kernel/subdirker kernel 5 0.0
				telnet/subdirtel telnet 10 40.0

				iteration 2
				The generated dataframe 'df_tree' will be:

				.. code-block:: none

				id parent contents known
				total 20 10.0
				kernel total 5 0.0
				vlorentzUnsubmitted Not Done Inline Actions You could rename it `generate_df_from_dirs`. And you can add a type annotation: `dirs: Dict[PosixPath, Tuple[int, int]]`. vlorentz: You could rename it `generate_df_from_dirs`. And you can add a type annotation: `dirs: Dict…
				telnet total 10 40.0

				Note that since we have reached the last level, the parent given
				to the directory id is the directory root.

				The 'total' row il computed by adding the number of contents of the
				dataframe given in input and the average of the contents known on
				vlorentzUnsubmitted Not Done Inline Actions Great! vlorentz: Great!
				the total number of contents.

				"""
				def compute_known_percentage(contents: pd.Series, known: pd.Series
				) -> pd.Series:
				"""This function compute the percentage of known contents and generate
				the new known column with the percentage values.

				It also assures that if there is no contents inside a directory
				the percentage is zero

				"""
				known_values = []
				for idx, content_val in enumerate(contents):
				if content_val == 0:
				known_values.append(0)
				else:
				percentage = known[idx] / contents[idx] * 100
				known_values.append(percentage)

				return pd.Series(np.array(known_values))

				complete_df = pd.DataFrame(columns=['id', 'parent', 'contents', 'known'])
				# revert the level order to start from the deepest
				levels = [level for level in reversed(levels)]
				contents_col = metrics_columns[0]
				vlorentzUnsubmitted Not Done Inline Actions `parts_list = list(dir_path.parts)` vlorentz: `parts_list = list(dir_path.parts)`
				known_col = metrics_columns[1]

				df_tree_list = []
				for i, level in enumerate(levels):
				df_tree = pd.DataFrame(columns=['id', 'parent', 'contents', 'known'])
				dfg = dirs_dataframe.groupby(levels[i:]).sum()
				dfg = dfg.reset_index()
				df_tree['id'] = dfg[level].copy()
				if i < len(levels) - 1:
				# copy the parent directories (one level above)
				df_tree['parent'] = dfg[levels[i+1]].copy()
				else:
				# last level reached
				df_tree['parent'] = root_name

				# copy the contents column
				df_tree['contents'] = dfg[contents_col]
				# compute the percentage relative to the contents
				df_tree['known'] = compute_known_percentage(
				dfg[contents_col], dfg[known_col])

				df_tree_list.append(df_tree)

				complete_df = complete_df.append(df_tree_list, ignore_index=True)

				# create the main parent
				total_contents = dirs_dataframe[contents_col].sum()
				total_known = dirs_dataframe[known_col].sum()
				total_avg = total_known / total_contents * 100

				total = pd.Series(dict(id=root_name, parent='',
				contents=total_contents,
				known=total_avg))

				complete_df = complete_df.append(total, ignore_index=True)

				return complete_df


				def compute_max_depth(dirs_path: List[PosixPath], root: PosixPath) -> int:
				"""Compute the maximum depth level of the given directory paths.

				Example: for `var/log/kernel/` the depth level is 3

				"""
				max_depth = 0
				for dir_path in dirs_path:
				if dir_path == root:
				continue

				dir_depth = len(dir_path.parts)
				if dir_depth > max_depth:
				max_depth = dir_depth

				return max_depth


				def generate_df_from_dirs(dirs: Dict[PosixPath, Tuple[int, int]],
				columns: List[str], root: PosixPath, max_depth: int
				) -> pd.DataFrame:
				"""Generate a dataframe from the directories given in input.

				Example:
				given the following directories as input

				.. code-block:: python

				dirs = {
				'/var/log/': (23, 2),
				'/var/log/kernel': (5, 0),
				'/var/log/telnet': (10, 3)
				}

				The generated dataframe will be:

				.. code-block:: none

				lev0 lev1 lev2 contents known
				'var' 'var/log' '' 23 2
				'var' 'var/log' 'var/log/kernel' 5 0
				'var' 'var/log' 'var/log/telnet' 10 3

				"""
				def get_parents(path: PosixPath):
				parts = path.parts[1:] if path.parts[0] == '/' else path.parts

				for i in range(1, len(parts)+1):
				yield '/'.join(parts[0:i])

				def get_dirs_array():
				for dir_path, contents_info in dirs.items():
				empty_lvl = max_depth - len(dir_path.parts)

				if dir_path == root:
				# ignore the root but store contents information
				yield ['']*(max_depth) + list(contents_info)
				else:
				yield list(get_parents(dir_path)) + \
				['']*empty_lvl + \
				list(contents_info)

				df = pd.DataFrame(np.array(
				[dir_array for dir_array in get_dirs_array()]), columns=columns)

				df['contents'] = pd.to_numeric(df['contents'])
				df['known'] = pd.to_numeric(df['known'])

				return df


				def sunburst(directories: Dict[PosixPath, Tuple[int, int]],
				root: PosixPath) -> None:
				"""Show the sunburst chart from the directories given in input.

				"""
				max_depth = compute_max_depth(list(directories.keys()), root)
				metrics_columns = ['contents', 'known']
				levels_columns = ['lev'+str(i) for i in range(max_depth)]

				df_columns = levels_columns + metrics_columns
				dirs_df = generate_df_from_dirs(directories, df_columns, root, max_depth)

				hierarchical_df = build_hierarchical_df(
				dirs_df, levels_columns, metrics_columns, str(root))
				known_avg = dirs_df['known'].sum() / dirs_df['contents'].sum()

				fig = go.Figure()
				fig.add_trace(go.Sunburst(
				labels=hierarchical_df['id'],
				parents=hierarchical_df['parent'],
				values=hierarchical_df['contents'],
				branchvalues='total',
				marker=dict(
				colors=hierarchical_df['known'],
				colorscale='RdBu',
				cmid=known_avg),
				hovertemplate='''<b>%{label}</b>
				<br>Files: %{value}
				<br>Known: <b>%{color:.2f}%</b>''',
				name=''
				))

				offline.plot(fig, filename='sunburst.html')

View Options

swh/web/api/views/snapshot.py

This file was added.

				# Copyright (C) 2020 The Software Heritage developers
				# See the AUTHORS file at the top-level directory of this distribution
				# License: GNU General Public License version 3, or any later version
				# See top-level LICENSE file for more information

				"""
				The purpose of this module is to display and to interact with the result of the
				scanner contained in the model.
				vlorentzUnsubmitted Not Done Inline Actions this group of imports should be after the other ones. See the second item here: https://www.python.org/dev/peps/pep-0008/#imports vlorentz: this group of imports should be after the other ones. See the second item here: https://www.

				The `sunburst` function generates a navigable sunburst chart from the
				directories information retrieved from the model. The chart displays for
				each directory the total number of files and the percentage of file known.

				The size of the directory is defined by the total number of contents whereas
				the color gradient is generated relying on the percentage of contents known.
				"""

				from typing import List, Dict, Tuple
				from pathlib import PosixPath

				from plotly.offline import offline # type: ignore
				import plotly.graph_objects as go # type: ignore
				import pandas as pd # type: ignore
				import numpy as np # type: ignore
				vlorentzUnsubmitted Not Done Inline Actions module docstring must be before imports vlorentz: module docstring must be before imports


				def build_hierarchical_df(
				dirs_dataframe: pd.DataFrame, levels: List[str],
				metrics_columns: List[str], root_name: str) -> pd.DataFrame:
				"""
				Build a hierarchy of levels for Sunburst or Treemap charts.
				vlorentzUnsubmitted Not Done Inline Actions Could you add a description of each argument? (and possibly change the name `df` to express what it contains) vlorentz: Could you add a description of each argument? (and possibly change the name `df` to express…
				vlorentzUnsubmitted Not Done Inline Actions renaming `df` to `dataframe` isn't more explicit about what it contains :/ vlorentz: renaming `df` to `dataframe` isn't more explicit about what it contains :/

				For each directory the new dataframe will have the following
				vlorentzUnsubmitted Not Done Inline Actions "For each directory level stored, the new dataframe will have"? vlorentz: "For each directory level stored, the new dataframe will have"?
				information:

				id: the directory name
				parent: the parent directory of id
				contents: the total number of contents of the directory id and
				vlorentzUnsubmitted Not Done Inline Actions Could you add an example of inputs, and what the outputs are? (like you did for `generate_df`) vlorentz: Could you add an example of inputs, and what the outputs are? (like you did for `generate_df`)
				the relative subdirectories
				known: the percentage of contents known relative to computed
				'contents'

				Example:
				Given the following dataframe:

				.. code-block:: none

				vlorentzUnsubmitted Not Done Inline Actions forgot this print vlorentz: forgot this print
				lev0 lev1 contents known
				'' '' 20 2 //root
				kernel kernel/subdirker 5 0
				telnet telnet/subdirtel 10 4

				The output hierarchical dataframe will be like the following:

				.. code-block:: none
				vlorentzUnsubmitted Not Done Inline Actions I think I understand what is going on here, but could you add a comment with an example of how the data is reshaped from `df` to `df_tree` step-by-step, for future readers? vlorentz: I think I understand what is going on here, but could you add a comment with an example of how…

				id parent contents known
				20 10.00
				vlorentzUnsubmitted Not Done Inline Actions The data from the first `df_tree`s will be copied into a new dataframe for each run of the loop. This is inefficient (quadratic complexity). You might want to build a list of dataframes and concatenate everything at the end. vlorentz: The data from the first `df_tree`s will be copied into a new dataframe for each run of the loop.
				kernel/subdirker kernel 5 0.00
				telnet/subdirtel telnet 10 40.00
				total 20 10.00
				kernel total 5 0.00
				telnet total 10 40.00
				total 35 17.14

				To create the hierarchical dataframe we need to iterate through
				the dataframe given in input relying on the number of levels.

				Based on the previous example we have to do two iterations:

				vlorentzUnsubmitted Not Done Inline Actions `dirs_path: List[PosixPath]` vlorentz: `dirs_path: List[PosixPath]`
				iteration 1
				vlorentzUnsubmitted Not Done Inline Actions also, would be nice to write in this docstring what data is in the dataframe vlorentz: also, would be nice to write in this docstring what data is in the dataframe
				The generated dataframe 'df_tree' will be:

				.. code-block:: none

				id parent contents known
				20 10.0
				kernel/subdirker kernel 5 0.0
				telnet/subdirtel telnet 10 40.0

				iteration 2
				The generated dataframe 'df_tree' will be:

				.. code-block:: none

				id parent contents known
				total 20 10.0
				kernel total 5 0.0
				vlorentzUnsubmitted Not Done Inline Actions You could rename it `generate_df_from_dirs`. And you can add a type annotation: `dirs: Dict[PosixPath, Tuple[int, int]]`. vlorentz: You could rename it `generate_df_from_dirs`. And you can add a type annotation: `dirs: Dict…
				telnet total 10 40.0

				Note that since we have reached the last level, the parent given
				to the directory id is the directory root.

				The 'total' row il computed by adding the number of contents of the
				dataframe given in input and the average of the contents known on
				vlorentzUnsubmitted Not Done Inline Actions Great! vlorentz: Great!
				the total number of contents.

				"""
				def compute_known_percentage(contents: pd.Series, known: pd.Series
				) -> pd.Series:
				"""This function compute the percentage of known contents and generate
				the new known column with the percentage values.

				It also assures that if there is no contents inside a directory
				the percentage is zero

				"""
				known_values = []
				for idx, content_val in enumerate(contents):
				if content_val == 0:
				known_values.append(0)
				else:
				percentage = known[idx] / contents[idx] * 100
				known_values.append(percentage)

				return pd.Series(np.array(known_values))

				complete_df = pd.DataFrame(columns=['id', 'parent', 'contents', 'known'])
				# revert the level order to start from the deepest
				levels = [level for level in reversed(levels)]
				contents_col = metrics_columns[0]
				vlorentzUnsubmitted Not Done Inline Actions `parts_list = list(dir_path.parts)` vlorentz: `parts_list = list(dir_path.parts)`
				known_col = metrics_columns[1]

				df_tree_list = []
				for i, level in enumerate(levels):
				df_tree = pd.DataFrame(columns=['id', 'parent', 'contents', 'known'])
				dfg = dirs_dataframe.groupby(levels[i:]).sum()
				dfg = dfg.reset_index()
				df_tree['id'] = dfg[level].copy()
				if i < len(levels) - 1:
				# copy the parent directories (one level above)
				df_tree['parent'] = dfg[levels[i+1]].copy()
				else:
				# last level reached
				df_tree['parent'] = root_name

				# copy the contents column
				df_tree['contents'] = dfg[contents_col]
				# compute the percentage relative to the contents
				df_tree['known'] = compute_known_percentage(
				dfg[contents_col], dfg[known_col])

				df_tree_list.append(df_tree)

				complete_df = complete_df.append(df_tree_list, ignore_index=True)

				# create the main parent
				total_contents = dirs_dataframe[contents_col].sum()
				total_known = dirs_dataframe[known_col].sum()
				total_avg = total_known / total_contents * 100

				total = pd.Series(dict(id=root_name, parent='',
				contents=total_contents,
				known=total_avg))

				complete_df = complete_df.append(total, ignore_index=True)

				return complete_df


				def compute_max_depth(dirs_path: List[PosixPath], root: PosixPath) -> int:
				"""Compute the maximum depth level of the given directory paths.

				Example: for `var/log/kernel/` the depth level is 3

				"""
				max_depth = 0
				for dir_path in dirs_path:
				if dir_path == root:
				continue

				dir_depth = len(dir_path.parts)
				if dir_depth > max_depth:
				max_depth = dir_depth

				return max_depth


				def generate_df_from_dirs(dirs: Dict[PosixPath, Tuple[int, int]],
				columns: List[str], root: PosixPath, max_depth: int
				) -> pd.DataFrame:
				"""Generate a dataframe from the directories given in input.

				Example:
				given the following directories as input

				.. code-block:: python

				dirs = {
				'/var/log/': (23, 2),
				'/var/log/kernel': (5, 0),
				'/var/log/telnet': (10, 3)
				}

				The generated dataframe will be:

				.. code-block:: none

				lev0 lev1 lev2 contents known
				'var' 'var/log' '' 23 2
				'var' 'var/log' 'var/log/kernel' 5 0
				'var' 'var/log' 'var/log/telnet' 10 3

				"""
				def get_parents(path: PosixPath):
				parts = path.parts[1:] if path.parts[0] == '/' else path.parts

				for i in range(1, len(parts)+1):
				yield '/'.join(parts[0:i])

				def get_dirs_array():
				for dir_path, contents_info in dirs.items():
				empty_lvl = max_depth - len(dir_path.parts)

				if dir_path == root:
				# ignore the root but store contents information
				yield ['']*(max_depth) + list(contents_info)
				else:
				yield list(get_parents(dir_path)) + \
				['']*empty_lvl + \
				list(contents_info)

				df = pd.DataFrame(np.array(
				[dir_array for dir_array in get_dirs_array()]), columns=columns)

				df['contents'] = pd.to_numeric(df['contents'])
				df['known'] = pd.to_numeric(df['known'])

				return df


				def sunburst(directories: Dict[PosixPath, Tuple[int, int]],
				root: PosixPath) -> None:
				"""Show the sunburst chart from the directories given in input.

				"""
				max_depth = compute_max_depth(list(directories.keys()), root)
				metrics_columns = ['contents', 'known']
				levels_columns = ['lev'+str(i) for i in range(max_depth)]

				df_columns = levels_columns + metrics_columns
				dirs_df = generate_df_from_dirs(directories, df_columns, root, max_depth)

				hierarchical_df = build_hierarchical_df(
				dirs_df, levels_columns, metrics_columns, str(root))
				known_avg = dirs_df['known'].sum() / dirs_df['contents'].sum()

				fig = go.Figure()
				fig.add_trace(go.Sunburst(
				labels=hierarchical_df['id'],
				parents=hierarchical_df['parent'],
				values=hierarchical_df['contents'],
				branchvalues='total',
				marker=dict(
				colors=hierarchical_df['known'],
				colorscale='RdBu',
				cmid=known_avg),
				hovertemplate='''<b>%{label}</b>
				<br>Files: %{value}
				<br>Known: <b>%{color:.2f}%</b>''',
				name=''
				))

				offline.plot(fig, filename='sunburst.html')

View Options

swh/web/tests/api/views/test_content.py

View Options

swh/web/tests/api/views/test_revision.py

View Options

swh/web/tests/api/views/test_snapshot.py

api: Make next links in response headers contain full URLsClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 8658

swh/web/api/views/content.py

swh/web/api/views/origin.py

swh/web/api/views/revision.py

swh/web/api/views/snapshot.py

swh/web/tests/api/views/test_content.py

swh/web/tests/api/views/test_revision.py

swh/web/tests/api/views/test_snapshot.py

api: Make next links in response headers contain full URLs
ClosedPublic
Actions

Revision Contents
Changeset List