Page MenuHomeSoftware Heritage
Differential D7463

Delay the unsubscribe to the end of handle_messages
ClosedPublic
Actions

Authored by douardda on Mar 29 2022, 5:44 PM.

Details

Reviewers
vlorentz
Group Reviewers
Reviewers
Commits
rDDATASETbe3c5da2bb44: Delay the unsubscribe to the end of handle_messages
Summary

to prevent some possible race condition leading to kafka errors like:

rd_kafka_assignment_partition_stopped: Assertion `rktp->rktp_started' failed

This could occur when, at the time of unsubscribing from a partition,
another partition is also depleted. Since the unsubscription consist in
resubscribing to all partitions except unsubscribes ones, we could try
to subscribe to such a depleted parition, leading to the error message
listed above.

Depends on D7462.

Diff Detail

Repository
rDDATASET Datasets
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

douardda created this revision.Mar 29 2022, 5:44 PM
Herald added a reviewer: Reviewers. · View Herald TranscriptMar 29 2022, 5:44 PM
douardda added a child revision: D7464: Improve the progress reporting look and feel.Mar 29 2022, 5:45 PM
swh-public-ci added a comment.Mar 29 2022, 5:47 PM

Build is green

Patch application report for D7463 (id=27048)

Could not rebase; Attempt merge onto 5a8a8a7847...

Updating 5a8a8a7..bc0a7ce
Fast-forward
 swh/dataset/exporters/orc.py    |  98 +++++++++++++++++++++++++++++++++------
 swh/dataset/journalprocessor.py |  22 ++++++++-
 swh/dataset/relational.py       |  10 ++++
 swh/dataset/test/test_orc.py    | 100 ++++++++++++++++++++++++++++++++++++----
 4 files changed, 205 insertions(+), 25 deletions(-)
Changes applied before test
commit bc0a7ce6986eb5242bd32054db13277ca0f1e1bb
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 25 15:52:26 2022 +0100

    Delay the unsubscribe to the end of handle_messages
    
    to prevent some possible race condition leading to kafka errors like:
    
      rd_kafka_assignment_partition_stopped: Assertion `rktp->rktp_started' failed
    
    This could occur when, at the time of unsubscribing from a partition,
    another partition is also depleted. Since the unsubscription consist in
    resubscribing to all partitions except unsubscribes ones, we could try
    to subscribe to such a depleted parition, leading to the error message
    listed above.

commit ebb5a89f95d73f52e87c456b872ec6c529d80fe3
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 25 15:43:18 2022 +0100

    Move exporter config entries in dedicated sections
    
    eg. orc exporter specific exporter config entries are now under the
    'orc' section, like:
    
      journal:
        brokers: [...]
    
      orc:
        remove_pull_requests: true
        max_rows:
          revision: 100000
          directory: 10000

commit e8ccb166a6aaa82f5917388f9b995c830499170a
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 23 16:35:52 2022 +0100

    Add support for limited row numbers in ORC files
    
    Make it possible to specify a maximum number of rows a table can store
    in a single ORC file. The limit can only be set on main tables for now
    (i.e. cannot be specified for tables like revision_history or
    directory_entry).
    
    This can be set by configuration only (no extra cli options).

commit fd3f9aa61de374655fd4bc4920d5047eb7d0c4ca
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:24:31 2022 +0100

    Add the raw_manifest column for revision, release and directory ORC files

commit 5c652bb058e2c1b59bafefd6817f392fdc171a20
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:22:47 2022 +0100

    Export revision extra headers in a dedicated ORC file

commit 45c8124b7a310963a868eb6602ea24e240d761e4
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 18 12:20:20 2022 +0100

    Add the type fields for revision and origin_visit_status ORC table

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/100/ for more details.

Harbormaster completed remote builds in B27963: Diff 27048.Mar 29 2022, 5:47 PM
douardda requested review of this revision.Mar 29 2022, 5:47 PM
vlorentz accepted this revision.Mar 29 2022, 6:02 PM
vlorentz added a subscriber: vlorentz.

thank you kafka

Please add a comment to explain this, though

This revision is now accepted and ready to land.Mar 29 2022, 6:02 PM
douardda updated this revision to Diff 27067.Mar 30 2022, 10:44 AM

rebase

swh-public-ci added a comment.Mar 30 2022, 10:45 AM

Build has FAILED

Patch application report for D7463 (id=27067)

Could not rebase; Attempt merge onto fd3f9aa61d...

Updating fd3f9aa..5849313
Fast-forward
 swh/dataset/exporters/orc.py    | 99 ++++++++++++++++++++++++++++++++---------
 swh/dataset/journalprocessor.py | 22 ++++++++-
 swh/dataset/relational.py       | 58 ++++++++++++++----------
 swh/dataset/test/test_orc.py    | 96 ++++++++++++++++++++++++++++++++++++---
 4 files changed, 222 insertions(+), 53 deletions(-)
Changes applied before test
commit 58493130ee7c13591d26069f91c3cd4ec1ccdc37
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 25 15:52:26 2022 +0100

    Delay the unsubscribe to the end of handle_messages
    
    to prevent some possible race condition leading to kafka errors like:
    
      rd_kafka_assignment_partition_stopped: Assertion `rktp->rktp_started' failed
    
    This could occur when, at the time of unsubscribing from a partition,
    another partition is also depleted. Since the unsubscription consist in
    resubscribing to all partitions except unsubscribes ones, we could try
    to subscribe to such a depleted parition, leading to the error message
    listed above.

commit c728c05e630cf02f35081e810279a5e5b24ebf98
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 25 15:43:18 2022 +0100

    Move exporter config entries in dedicated sections
    
    eg. orc exporter specific exporter config entries are now under the
    'orc' section, like:
    
      journal:
        brokers: [...]
    
      orc:
        remove_pull_requests: true
        max_rows:
          revision: 100000
          directory: 10000

commit 850ee3be47cf3b1e0ab53f8820cc5e4c86b94f38
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 23 16:35:52 2022 +0100

    Add support for limited row numbers in ORC files
    
    Make it possible to specify a maximum number of rows a table can store
    in a single ORC file. The limit can only be set on main tables for now
    (i.e. cannot be specified for tables like revision_history or
    directory_entry).
    
    This can be set by configuration only (no extra cli options).

Link to build: https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/106/
See console output for more information: https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/106/console

Harbormaster failed remote builds in B27983: Diff 27067!Mar 30 2022, 10:45 AM
douardda updated this revision to Diff 27076.Mar 30 2022, 11:09 AM

rebase

swh-public-ci added a comment.Mar 30 2022, 11:13 AM

Build is green

Patch application report for D7463 (id=27076)

Could not rebase; Attempt merge onto fd3f9aa61d...

Updating fd3f9aa..9f7c128
Fast-forward
 swh/dataset/exporters/orc.py    | 99 ++++++++++++++++++++++++++++++++---------
 swh/dataset/journalprocessor.py | 22 ++++++++-
 swh/dataset/relational.py       | 58 ++++++++++++++----------
 swh/dataset/test/test_orc.py    | 96 ++++++++++++++++++++++++++++++++++++---
 4 files changed, 222 insertions(+), 53 deletions(-)
Changes applied before test
commit 9f7c128eb8aba748e24eaa682269d74dcefb98c6
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 25 15:52:26 2022 +0100

    Delay the unsubscribe to the end of handle_messages
    
    to prevent some possible race condition leading to kafka errors like:
    
      rd_kafka_assignment_partition_stopped: Assertion `rktp->rktp_started' failed
    
    This could occur when, at the time of unsubscribing from a partition,
    another partition is also depleted. Since the unsubscription consist in
    resubscribing to all partitions except unsubscribes ones, we could try
    to subscribe to such a depleted parition, leading to the error message
    listed above.

commit e01daba4d601733a86ce7401fe54247908d03e5c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 25 15:43:18 2022 +0100

    Move exporter config entries in dedicated sections
    
    eg. orc exporter specific exporter config entries are now under the
    'orc' section, like:
    
      journal:
        brokers: [...]
    
      orc:
        remove_pull_requests: true
        max_rows:
          revision: 100000
          directory: 10000

commit 3df08fd71759487e963e6569c8dfd0c502b060de
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 23 16:35:52 2022 +0100

    Add support for limited row numbers in ORC files
    
    Make it possible to specify a maximum number of rows a table can store
    in a single ORC file. The limit can only be set on main tables for now
    (i.e. cannot be specified for tables like revision_history or
    directory_entry).
    
    This can be set by configuration only (no extra cli options).

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/111/ for more details.

Harbormaster completed remote builds in B27992: Diff 27076.Mar 30 2022, 11:13 AM
douardda updated this revision to Diff 27080.Mar 30 2022, 11:22 AM

Add a comment

swh-public-ci added a comment.Mar 30 2022, 11:25 AM

Build is green

Patch application report for D7463 (id=27080)

Could not rebase; Attempt merge onto fd3f9aa61d...

Updating fd3f9aa..be3c5da
Fast-forward
 swh/dataset/exporters/orc.py    | 99 ++++++++++++++++++++++++++++++++---------
 swh/dataset/journalprocessor.py | 30 ++++++++++++-
 swh/dataset/relational.py       | 58 ++++++++++++++----------
 swh/dataset/test/test_orc.py    | 96 ++++++++++++++++++++++++++++++++++++---
 4 files changed, 230 insertions(+), 53 deletions(-)
Changes applied before test
commit be3c5da2bb44c73fe614e93073727ffd50665a9c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 25 15:52:26 2022 +0100

    Delay the unsubscribe to the end of handle_messages
    
    to prevent some possible race condition leading to kafka errors like:
    
      rd_kafka_assignment_partition_stopped: Assertion `rktp->rktp_started' failed
    
    This could occur when, at the time of unsubscribing from a partition,
    another partition is also depleted. Since the unsubscription consist in
    resubscribing to all partitions except unsubscribes ones, we could try
    to subscribe to such a depleted parition, leading to the error message
    listed above.

commit e01daba4d601733a86ce7401fe54247908d03e5c
Author: David Douard <david.douard@sdfa3.org>
Date:   Fri Mar 25 15:43:18 2022 +0100

    Move exporter config entries in dedicated sections
    
    eg. orc exporter specific exporter config entries are now under the
    'orc' section, like:
    
      journal:
        brokers: [...]
    
      orc:
        remove_pull_requests: true
        max_rows:
          revision: 100000
          directory: 10000

commit 3df08fd71759487e963e6569c8dfd0c502b060de
Author: David Douard <david.douard@sdfa3.org>
Date:   Wed Mar 23 16:35:52 2022 +0100

    Add support for limited row numbers in ORC files
    
    Make it possible to specify a maximum number of rows a table can store
    in a single ORC file. The limit can only be set on main tables for now
    (i.e. cannot be specified for tables like revision_history or
    directory_entry).
    
    This can be set by configuration only (no extra cli options).

See https://jenkins.softwareheritage.org/job/DDATASET/job/tests-on-diff/114/ for more details.

Harbormaster completed remote builds in B27996: Diff 27080.Mar 30 2022, 11:26 AM
Closed by commit rDDATASETbe3c5da2bb44: Delay the unsubscribe to the end of handle_messages (authored by douardda). · Explain WhyMar 30 2022, 11:34 AM
This revision was automatically updated to reflect the committed changes.
douardda added a commit: rDDATASETbe3c5da2bb44: Delay the unsubscribe to the end of handle_messages.

Revision Contents
Changeset List

PathSize
swh/
dataset/
30 lines

Diff 27086

View Options

swh/dataset/journalprocessor.py

Loading...
About · Developer information · Legal