Page MenuHomeSoftware Heritage
Differential D675

Start adding a Table abstraction.
AbandonedPublic
Actions

Authored by vlorentz on Nov 19 2018, 1:33 PM.

Details

Reviewers
douardda
Group Reviewers
Reviewers
Summary

This Diff introduces a new Table class, that acts as an abstraction over Python data structures, providing only features a regular database would.
For instance, it prevents stuff like modifying the database's content by manipulating an object that's already in it.

This is what I mentioned in last week's diff's comments (https://forge.softwareheritage.org/D645#inline-3404).

Note that this Diff only uses the Table for simple "tables". Snapshots, origins, and origin visits are more tricky (origin keys are two columns) and I did not completely figure out how to deal with them in a clean way.

Diff Detail

Repository
rDSTO Storage manager
Branch
in-mem-tables
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 2542
Build 3157: tox-on-jenkinsJenkins
Build 3156: arc lint + arc unit

Event Timeline

vlorentz created this revision.Nov 19 2018, 1:33 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptNov 19 2018, 1:33 PM
swh-public-ci added a subscriber: swh-public-ci.Nov 19 2018, 1:38 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/85/ for more details.

Harbormaster completed remote builds in B2448: Diff 2126.Nov 19 2018, 1:38 PM
vlorentz planned changes to this revision.Nov 19 2018, 2:16 PM
vlorentz updated this revision to Diff 2132.Nov 19 2018, 5:36 PM
  • Remove the freezing stuff, just don't allow lists and dicts.
swh-public-ci added a comment.Nov 19 2018, 5:41 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/88/ for more details.

Harbormaster completed remote builds in B2454: Diff 2132.Nov 19 2018, 5:41 PM
vlorentz updated this revision to Diff 2134.Nov 19 2018, 5:53 PM
  • Add some doc.
swh-public-ci added a comment.Nov 19 2018, 5:59 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/89/ for more details.

Harbormaster completed remote builds in B2456: Diff 2134.Nov 19 2018, 5:59 PM
vlorentz planned changes to this revision.Nov 20 2018, 10:46 AM
vlorentz updated this revision to Diff 2204.Nov 22 2018, 4:39 PM
  • Use 'attrs' instead of namedtuples.
Harbormaster failed remote builds in B2540: Diff 2204!Nov 22 2018, 4:40 PM
swh-public-ci added a comment.Nov 22 2018, 4:40 PM

Build has FAILED

Link to build: https://jenkins.softwareheritage.org/job/DSTO/job/tox/117/
See console output for more information: https://jenkins.softwareheritage.org/job/DSTO/job/tox/117/console

vlorentz updated this revision to Diff 2205.Nov 22 2018, 4:42 PM
  • Rebase
vlorentz updated this revision to Diff 2206.Nov 22 2018, 4:43 PM
  • Remove the doc update, that belongs in another diff.
swh-public-ci added a comment.Nov 22 2018, 4:47 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/118/ for more details.

Harbormaster completed remote builds in B2541: Diff 2205.Nov 22 2018, 4:48 PM
swh-public-ci added a comment.Nov 22 2018, 4:53 PM

Build is green
See https://jenkins.softwareheritage.org/job/DSTO/job/tox/119/ for more details.

Harbormaster completed remote builds in B2542: Diff 2206.Nov 22 2018, 4:53 PM
douardda requested changes to this revision.Dec 3 2018, 11:43 AM
douardda added a subscriber: douardda.

Before going further in this direction, I'd like we have a discussion so:

  • we agree to introduce the usage of a lib like attr,
  • and agree to use it everywhere we need somehow to describe models in the code.

I do not want to have 3 or 4 different ways of defining such models in our code base.

Also:

  • adding 200+ loc in an already rather big python module should be avaided: create a separated module with this model logics in, we could even argue that the Table class should live in its own module,
  • not having a single test neither for this new data model nor the Table class is not reasonable.
This revision now requires changes to proceed.Dec 3 2018, 11:43 AM
vlorentz added a comment.Dec 3 2018, 12:04 PM
In D675#15851, @douardda wrote:

Before going further in this direction, I'd like we have a discussion so:

  • we agree to introduce the usage of a lib like attr,
  • and agree to use it everywhere we need somehow to describe models in the code.

I do not want to have 3 or 4 different ways of defining such models in our code base.

Agreed

  • adding 200+ loc in an already rather big python module should be avaided: create a separated module with this model logics in, we could even argue that the Table class should live in its own module,

I propose to have it here only temporarily, and move it to swh.model once it's stabilized.

  • not having a single test neither for this new data model nor the Table class is not reasonable.

Existing tests already test it's working on valid data. I'll add tests for invalid data.

vlorentz abandoned this revision.May 6 2019, 11:54 AM

Revision Contents
Changeset List

PathSize
debian/
1 line
1 line
swh/
storage/
275 lines

Diff 2206

View Options

debian/control

Loading...
View Options

requirements.txt

Loading...
View Options

swh/storage/in_memory.py

Loading...
About · Developer information · Legal