Page MenuHomeSoftware Heritage
Differential D7518

Add support for recursive multipart messages
ClosedPublic
Actions

Authored by vlorentz on Apr 7 2022, 11:44 AM.

Details

Reviewers
douardda
olasd
Group Reviewers
Reviewers
Commits
rDWAPPS120e48badec2: Add support for recursive multipart messages
Summary

Before this commit, parsing the test file would just return
and more plain text because it is the largest text part.

This uses message.get_payload() instead of message.walk(), because
message.walk() implements a bottom-up DFS; but with no easy way of
propagating information between nodes.

Depends on D7517.

Diff Detail

Repository
rDWAPPS Web applications
Branch
email
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 28210
Build 44167: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 44166: arc lint + arc unit

Event Timeline

vlorentz created this revision.Apr 7 2022, 11:44 AM
Herald added a reviewer: Reviewers. · View Herald TranscriptApr 7 2022, 11:44 AM
swh-public-ci added a comment.Apr 7 2022, 12:04 PM

Build is green

Patch application report for D7518 (id=27280)

Could not rebase; Attempt merge onto 17cd03607c...

Updating 17cd0360..9ae798a6
Fast-forward
 swh/web/inbound_email/utils.py                     | 87 ++++++++++++++--------
 .../{multipart.eml => multipart_alternative.eml}   |  0
 ...nly.eml => multipart_alternative_html_only.eml} |  0
 .../resources/multipart_alternative_recursive.eml  | 45 +++++++++++
 ...nly.eml => multipart_alternative_text_only.eml} |  0
 .../inbound_email/resources/multipart_mixed.eml    | 23 ++++++
 .../inbound_email/resources/multipart_mixed2.eml   | 25 +++++++
 .../resources/multipart_mixed_text_only.eml        | 36 +++++++++
 swh/web/tests/inbound_email/test_utils.py          | 40 ++++++++--
 9 files changed, 220 insertions(+), 36 deletions(-)
 rename swh/web/tests/inbound_email/resources/{multipart.eml => multipart_alternative.eml} (100%)
 rename swh/web/tests/inbound_email/resources/{multipart_html_only.eml => multipart_alternative_html_only.eml} (100%)
 create mode 100644 swh/web/tests/inbound_email/resources/multipart_alternative_recursive.eml
 rename swh/web/tests/inbound_email/resources/{multipart_text_only.eml => multipart_alternative_text_only.eml} (100%)
 create mode 100644 swh/web/tests/inbound_email/resources/multipart_mixed.eml
 create mode 100644 swh/web/tests/inbound_email/resources/multipart_mixed2.eml
 create mode 100644 swh/web/tests/inbound_email/resources/multipart_mixed_text_only.eml
Changes applied before test
commit 9ae798a6830b7194b5582a8c249bc0f24d57152a
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 7 11:40:38 2022 +0200

    Add support for recursive multipart messages
    
    Before this commit, parsing the test file would just return
    `and more plain text` because it is the largest text part.
    
    This uses `message.get_payload()` instead of `message.walk()`, because
    `message.walk()` implements a bottom-up DFS; but with no easy way of
    propagating information between nodes.

commit 4f5f426891391a1e72f61ce53978503bd219cd0c
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 7 10:56:08 2022 +0200

    Add support for multipart/mixed + better fallback for multipart/*

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1691/ for more details.

Harbormaster completed remote builds in B28210: Diff 27280.Apr 7 2022, 12:04 PM
vlorentz requested review of this revision.Apr 7 2022, 12:04 PM
vlorentz updated this revision to Diff 27285.Apr 7 2022, 12:05 PM

rebase + anonymize test data + add test for multipart/related

swh-public-ci added a comment.Apr 7 2022, 12:26 PM

Build is green

Patch application report for D7518 (id=27285)

Could not rebase; Attempt merge onto 17cd03607c...

Updating 17cd0360..ae9a460c
Fast-forward
 swh/web/inbound_email/utils.py                     | 87 ++++++++++++++--------
 .../{multipart.eml => multipart_alternative.eml}   |  0
 ...nly.eml => multipart_alternative_html_only.eml} |  0
 .../resources/multipart_alternative_recursive.eml  | 45 +++++++++++
 ...nly.eml => multipart_alternative_text_only.eml} |  0
 .../inbound_email/resources/multipart_mixed.eml    | 23 ++++++
 .../inbound_email/resources/multipart_mixed2.eml   | 25 +++++++
 .../resources/multipart_mixed_text_only.eml        | 36 +++++++++
 .../inbound_email/resources/multipart_related.eml  | 42 +++++++++++
 swh/web/tests/inbound_email/test_utils.py          | 49 ++++++++++--
 10 files changed, 271 insertions(+), 36 deletions(-)
 rename swh/web/tests/inbound_email/resources/{multipart.eml => multipart_alternative.eml} (100%)
 rename swh/web/tests/inbound_email/resources/{multipart_html_only.eml => multipart_alternative_html_only.eml} (100%)
 create mode 100644 swh/web/tests/inbound_email/resources/multipart_alternative_recursive.eml
 rename swh/web/tests/inbound_email/resources/{multipart_text_only.eml => multipart_alternative_text_only.eml} (100%)
 create mode 100644 swh/web/tests/inbound_email/resources/multipart_mixed.eml
 create mode 100644 swh/web/tests/inbound_email/resources/multipart_mixed2.eml
 create mode 100644 swh/web/tests/inbound_email/resources/multipart_mixed_text_only.eml
 create mode 100644 swh/web/tests/inbound_email/resources/multipart_related.eml
Changes applied before test
commit ae9a460cbaaf94e7e3c990355f2764f8afc89912
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 7 11:40:38 2022 +0200

    Add support for recursive multipart messages
    
    Before this commit, parsing the test file would just return
    `and more plain text` because it is the largest text part.
    
    This uses `message.get_payload()` instead of `message.walk()`, because
    `message.walk()` implements a bottom-up DFS; but with no easy way of
    propagating information between nodes.

commit 5444626178f72c89d08072f41716bf4427772c24
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 7 10:56:08 2022 +0200

    Add support for multipart/mixed + better fallback for multipart/*

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1693/ for more details.

Harbormaster completed remote builds in B28215: Diff 27285.Apr 7 2022, 12:26 PM
douardda accepted this revision.Apr 8 2022, 11:34 AM
douardda added a subscriber: douardda.

LGTM

This revision is now accepted and ready to land.Apr 8 2022, 11:34 AM
olasd accepted this revision.Apr 8 2022, 11:46 AM
olasd added a subscriber: olasd.

Thanks!

vlorentz updated this revision to Diff 27410.Apr 13 2022, 1:54 PM

rebase

This revision was landed with ongoing or failed builds.Apr 13 2022, 1:55 PM
Closed by commit rDWAPPS120e48badec2: Add support for recursive multipart messages (authored by vlorentz). · Explain Why
This revision was automatically updated to reflect the committed changes.
vlorentz added a commit: rDWAPPS120e48badec2: Add support for recursive multipart messages.
swh-public-ci added a comment.Apr 13 2022, 2:21 PM

Build is green

Patch application report for D7518 (id=27410)

Could not rebase; Attempt merge onto e5115fdbdf...

Updating e5115fdb..120e48ba
Fast-forward
 swh/web/inbound_email/utils.py                     | 87 ++++++++++++++--------
 .../{multipart.eml => multipart_alternative.eml}   |  0
 ...nly.eml => multipart_alternative_html_only.eml} |  0
 .../resources/multipart_alternative_recursive.eml  | 45 +++++++++++
 ...nly.eml => multipart_alternative_text_only.eml} |  0
 .../inbound_email/resources/multipart_mixed.eml    | 23 ++++++
 .../inbound_email/resources/multipart_mixed2.eml   | 25 +++++++
 .../resources/multipart_mixed_text_only.eml        | 36 +++++++++
 .../inbound_email/resources/multipart_related.eml  | 42 +++++++++++
 swh/web/tests/inbound_email/test_utils.py          | 49 ++++++++++--
 10 files changed, 271 insertions(+), 36 deletions(-)
 rename swh/web/tests/inbound_email/resources/{multipart.eml => multipart_alternative.eml} (100%)
 rename swh/web/tests/inbound_email/resources/{multipart_html_only.eml => multipart_alternative_html_only.eml} (100%)
 create mode 100644 swh/web/tests/inbound_email/resources/multipart_alternative_recursive.eml
 rename swh/web/tests/inbound_email/resources/{multipart_text_only.eml => multipart_alternative_text_only.eml} (100%)
 create mode 100644 swh/web/tests/inbound_email/resources/multipart_mixed.eml
 create mode 100644 swh/web/tests/inbound_email/resources/multipart_mixed2.eml
 create mode 100644 swh/web/tests/inbound_email/resources/multipart_mixed_text_only.eml
 create mode 100644 swh/web/tests/inbound_email/resources/multipart_related.eml
Changes applied before test
commit 120e48badec244b9820efbc0cc7c2c5af08831c9
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 7 11:40:38 2022 +0200

    Add support for recursive multipart messages
    
    Before this commit, parsing the test file would just return
    `and more plain text` because it is the largest text part.
    
    This uses `message.get_payload()` instead of `message.walk()`, because
    `message.walk()` implements a bottom-up DFS; but with no easy way of
    propagating information between nodes.

commit ae8b3148611bb44cd1b16126c49efc40097aca7f
Author: Valentin Lorentz <vlorentz@softwareheritage.org>
Date:   Thu Apr 7 10:56:08 2022 +0200

    Add support for multipart/mixed + better fallback for multipart/*

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1712/ for more details.

Harbormaster completed remote builds in B28425: Diff 27410.Apr 13 2022, 2:21 PM

Revision Contents
Changeset List

PathSizeCoverage (All)Coverage (Touched)
swh/
web/
inbound_email/
96 lines96%
Loading...
tests/
inbound_email/
resources/
45 lines--
6 lines100%
Loading...

Diff 27280

View Options

swh/web/inbound_email/utils.py

Loading...
View Options

swh/web/tests/inbound_email/resources/multipart_alternative_recursive.eml

Loading...
View Options

swh/web/tests/inbound_email/test_utils.py

Loading...
About · Developer information · Legal