Page MenuHomeSoftware Heritage

Hook up processing of inbound emails for add_forge_now
ClosedPublic

Authored by olasd on Mar 9 2022, 5:57 PM.

Details

Summary

Depends on D7799
Related to T3999

Test Plan

Added an integration test in cypress that checks email processing and that the message source link shows up properly.

Diff Detail

Repository
rDWAPPS Web applications
Branch
sprint-add-forge-now-inbound-emails
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 27430
Build 42926: Phabricator diff pipeline on jenkinsJenkins console · Jenkins
Build 42925: arc lint + arc unit

Event Timeline

Build is green

Patch application report for D7326 (id=26499)

Could not rebase; Attempt merge onto 51aa6fbc8e...

Merge made by the 'recursive' strategy.
 swh/web/add_forge_now/__init__.py                  |   0
 swh/web/add_forge_now/apps.py                      |  10 ++
 swh/web/add_forge_now/migrations/0001_initial.py   | 109 +++++++++++++++++++
 swh/web/add_forge_now/migrations/__init__.py       |   0
 swh/web/add_forge_now/models.py                    | 118 +++++++++++++++++++++
 swh/web/add_forge_now/signal_receivers.py          |  76 +++++++++++++
 swh/web/add_forge_now/tests/test_migration.py      |  62 +++++++++++
 swh/web/api/urls.py                                |   1 +
 swh/web/api/views/add_forge_now.py                 | 108 +++++++++++++++++++
 swh/web/config.py                                  |   1 +
 swh/web/inbound_email/__init__.py                  |   0
 swh/web/inbound_email/apps.py                      |  11 ++
 .../management/commands/process_inbound_email.py   |  73 +++++++++++++
 swh/web/inbound_email/signals.py                   |  36 +++++++
 swh/web/inbound_email/tests.py                     |   3 +
 swh/web/inbound_email/utils.py                     |  66 ++++++++++++
 swh/web/settings/common.py                         |   2 +
 swh/web/tests/api/views/test_add_forge_now.py      | 117 ++++++++++++++++++++
 swh/web/tests/inbound_email/test_utils.py          | 113 ++++++++++++++++++++
 swh/web/tests/utils.py                             |   4 +-
 20 files changed, 909 insertions(+), 1 deletion(-)
 create mode 100644 swh/web/add_forge_now/__init__.py
 create mode 100644 swh/web/add_forge_now/apps.py
 create mode 100644 swh/web/add_forge_now/migrations/0001_initial.py
 create mode 100644 swh/web/add_forge_now/migrations/__init__.py
 create mode 100644 swh/web/add_forge_now/models.py
 create mode 100644 swh/web/add_forge_now/signal_receivers.py
 create mode 100644 swh/web/add_forge_now/tests/test_migration.py
 create mode 100644 swh/web/api/views/add_forge_now.py
 create mode 100644 swh/web/inbound_email/__init__.py
 create mode 100644 swh/web/inbound_email/apps.py
 create mode 100644 swh/web/inbound_email/management/commands/process_inbound_email.py
 create mode 100644 swh/web/inbound_email/signals.py
 create mode 100644 swh/web/inbound_email/tests.py
 create mode 100644 swh/web/inbound_email/utils.py
 create mode 100644 swh/web/tests/api/views/test_add_forge_now.py
 create mode 100644 swh/web/tests/inbound_email/test_utils.py
Changes applied before test
commit 977ffa190ef93d2bcf8e2828fe72d863920705fc
Merge: 51aa6fbc 3804c203
Author: Jenkins user <jenkins@localhost>
Date:   Wed Mar 9 16:58:56 2022 +0000

    Merge branch 'diff-target' into HEAD

commit 3804c20354bd0cfdd259ce1d99df07a7ca3b9df0
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Mar 9 17:43:36 2022 +0100

    wip: hook up processing of inbound emails for add_forge_now

commit e49b4cee5906f89629d6eb89bbceacdd7ba91cc7
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Mar 9 17:48:45 2022 +0100

    inbound_email: add some logging statements for unhandled exceptions

commit 889a84771fce10d5483de6cd5991d4feee9d1234
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Mar 9 16:37:54 2022 +0100

    inbound_email: add utilities to extract and filter email recipients

commit 1c00cc8a08add62f30468a26b1d8784d51b3a2ec
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Mar 9 14:29:37 2022 +0100

    Bootstrap a common app for management of inbound emails

commit 130e9faa9bcab18bc3c76dc898546edb89f3f9b8
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Mar 8 15:23:50 2022 +0100

    api: Add endpoint to create an add-forge request
    
    Related to T3990

commit 03101208803501e5178f35d18d171e740ee4ca76
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Mar 8 11:34:44 2022 +0100

    add_forge_now: Bootstrap app and model

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1434/ for more details.

Build is green

Patch application report for D7326 (id=26549)

Could not rebase; Attempt merge onto cdcc1efdfb...

Merge made by the 'recursive' strategy.
 swh/web/add_forge_now/__init__.py                  |   0
 swh/web/add_forge_now/apps.py                      |  10 +
 swh/web/add_forge_now/migrations/0001_initial.py   | 109 +++++
 swh/web/add_forge_now/migrations/__init__.py       |   0
 swh/web/add_forge_now/models.py                    | 146 ++++++
 swh/web/add_forge_now/signal_receivers.py          |  76 ++++
 swh/web/add_forge_now/tests/test_migration.py      |  62 +++
 swh/web/api/urls.py                                |   1 +
 swh/web/api/views/add_forge_now.py                 | 352 +++++++++++++++
 swh/web/config.py                                  |   1 +
 swh/web/inbound_email/__init__.py                  |   0
 swh/web/inbound_email/apps.py                      |  11 +
 .../management/commands/process_inbound_email.py   |  73 +++
 swh/web/inbound_email/signals.py                   |  36 ++
 swh/web/inbound_email/utils.py                     |  63 +++
 swh/web/settings/common.py                         |   2 +
 swh/web/tests/add_forge_now/test_models.py         |  26 ++
 swh/web/tests/api/views/test_add_forge_now.py      | 489 +++++++++++++++++++++
 .../tests/inbound_email/test_management_command.py | 161 +++++++
 swh/web/tests/inbound_email/test_utils.py          | 113 +++++
 swh/web/tests/utils.py                             |  10 +-
 21 files changed, 1738 insertions(+), 3 deletions(-)
 create mode 100644 swh/web/add_forge_now/__init__.py
 create mode 100644 swh/web/add_forge_now/apps.py
 create mode 100644 swh/web/add_forge_now/migrations/0001_initial.py
 create mode 100644 swh/web/add_forge_now/migrations/__init__.py
 create mode 100644 swh/web/add_forge_now/models.py
 create mode 100644 swh/web/add_forge_now/signal_receivers.py
 create mode 100644 swh/web/add_forge_now/tests/test_migration.py
 create mode 100644 swh/web/api/views/add_forge_now.py
 create mode 100644 swh/web/inbound_email/__init__.py
 create mode 100644 swh/web/inbound_email/apps.py
 create mode 100644 swh/web/inbound_email/management/commands/process_inbound_email.py
 create mode 100644 swh/web/inbound_email/signals.py
 create mode 100644 swh/web/inbound_email/utils.py
 create mode 100644 swh/web/tests/add_forge_now/test_models.py
 create mode 100644 swh/web/tests/api/views/test_add_forge_now.py
 create mode 100644 swh/web/tests/inbound_email/test_management_command.py
 create mode 100644 swh/web/tests/inbound_email/test_utils.py
Changes applied before test
commit 176c43d04ee5ed9f1cf43214f510c5ddb6268d0b
Merge: cdcc1efd c60fc7c6
Author: Jenkins user <jenkins@localhost>
Date:   Mon Mar 14 12:37:32 2022 +0000

    Merge branch 'diff-target' into HEAD

commit c60fc7c6c8fe8d7e392f94533142b447ad70f435
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Mar 9 17:43:36 2022 +0100

    wip: hook up processing of inbound emails for add_forge_now

commit 531a8bd04ad6bc938a66565f0d146da08b172eeb
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Mar 9 16:37:54 2022 +0100

    inbound_email: add utilities to extract and filter email recipients

commit 99413db8665d649e7a598233e972f374d6856e93
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Wed Mar 9 14:29:37 2022 +0100

    Bootstrap a common app for management of inbound emails

commit 26748e56ecce5877fce3a215eba06a19ebb8342f
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Mar 10 14:09:36 2022 +0100

    api: Add endpoint to get details about an add-forge request
    
    Related to T4030

commit 294a95c711bc40622c6d718afa2535f377b90dfc
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Mar 9 16:30:06 2022 +0100

    api: Add endpoint to list add-forge requests
    
    Related to T4027

commit 1bf17d6a75fd7265b1da9c9e87220efa62869c93
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Wed Mar 9 16:28:03 2022 +0100

    api: Add endpoint to update an add-forge request
    
    Related to T4026

commit 130e9faa9bcab18bc3c76dc898546edb89f3f9b8
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Mar 8 15:23:50 2022 +0100

    api: Add endpoint to create an add-forge request
    
    Related to T3990

commit 03101208803501e5178f35d18d171e740ee4ca76
Author: Antoine R. Dumont (@ardumont) <ardumont@softwareheritage.org>
Date:   Tue Mar 8 11:34:44 2022 +0100

    add_forge_now: Bootstrap app and model

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1453/ for more details.

  • Restrict pytest-postgresql to < 4.0.0
  • inbound_email: split recipient matching logic out
  • inbound_email: add support for signed email addresses
  • add_forge_now: add inbound email address to model
  • inbound_email: add function to extract the plaintext from a mail
  • Add forge now: prepare fields for inbound email processing
  • Add forge now: Add Cc and Reply-To to forge administrator email
  • Add forge now: process inbound emails

Build was aborted

Patch application report for D7326 (id=27157)

Rebasing onto ad5add7d36...

Current branch diff-target is up to date.
Changes applied before test
commit 276076996b038406f51694768e41ec73165534d9
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 17:30:19 2022 +0200

    Add forge now: process inbound emails
    
    Add a signal receiver to process inbound emails to the address set up in
    config. The text part of the emails is parsed if available, and the raw
    message is saved as a last resort.

commit cdd0fb00bf4d0c67407caa46ac950314eff530ff
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 17:16:36 2022 +0200

    Add forge now: Add Cc and Reply-To to forge administrator email
    
    This sets Cc: and Reply-To to the inbound email for the add-forge-now feature.

commit 6c8b7fdb3970c5d804a905931d2d9286b0d7a57b
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 17:08:47 2022 +0200

    Add forge now: prepare fields for inbound email processing
    
    - add raw message field to request history table
    - add email as choice for the request history actor roles

commit 7d45ac4711455e84720f504f07d4f57285171bf4
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 17:04:30 2022 +0200

    inbound_email: add function to extract the plaintext from a mail
    
    This function decodes the html part if no text part is available.

commit a677e9d164e156b60f0281ab7ff20c84a237d102
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 10:38:38 2022 +0200

    add_forge_now: add inbound email address to model

commit e88291be19b8f52167c3c38804cabc0410578a8f
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Mar 31 16:01:56 2022 +0200

    inbound_email: add support for signed email addresses
    
    These utilities allow us to generate addresses of the form
    `<localpart>+<integer>.<signature>@<domain>`, where the integer is the
    primary key of a given object for which we want to track email
    exchanges. The signature prevents the addresses from being forged, that
    is, the addresses have to be explicitly generated and displayed by the
    web app to be discovered.
    
    The counterpart function retrieves all relevant email addresses from the
    list of recipients of an email message, and validates the signatures to
    recover the integer values that won't have been tampered with.
    
    These new utilities are expected to be used in the views and signal
    handlers pertaining to processing of inbound email messages.

commit f8fb7a04639729d6530f126ce1c7dd87ce82364c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 15:16:11 2022 +0200

    inbound_email: split recipient matching logic out
    
    This allows calling the function on a single recipient rather than on a
    whole message, when one isn't available.

commit 1864b158225d2ae23bc81ac73cb3966caa2c65a1
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 15:15:05 2022 +0200

    Restrict pytest-postgresql to < 4.0.0
    
    Other modules still need psycopg2 and pytest-postgresql 4 introduced a
    hard dependency on psycopg3.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1648/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1648/console

Build is green

Patch application report for D7326 (id=27157)

Rebasing onto ad5add7d36...

Current branch diff-target is up to date.
Changes applied before test
commit 276076996b038406f51694768e41ec73165534d9
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 17:30:19 2022 +0200

    Add forge now: process inbound emails
    
    Add a signal receiver to process inbound emails to the address set up in
    config. The text part of the emails is parsed if available, and the raw
    message is saved as a last resort.

commit cdd0fb00bf4d0c67407caa46ac950314eff530ff
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 17:16:36 2022 +0200

    Add forge now: Add Cc and Reply-To to forge administrator email
    
    This sets Cc: and Reply-To to the inbound email for the add-forge-now feature.

commit 6c8b7fdb3970c5d804a905931d2d9286b0d7a57b
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 17:08:47 2022 +0200

    Add forge now: prepare fields for inbound email processing
    
    - add raw message field to request history table
    - add email as choice for the request history actor roles

commit 7d45ac4711455e84720f504f07d4f57285171bf4
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 17:04:30 2022 +0200

    inbound_email: add function to extract the plaintext from a mail
    
    This function decodes the html part if no text part is available.

commit a677e9d164e156b60f0281ab7ff20c84a237d102
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 10:38:38 2022 +0200

    add_forge_now: add inbound email address to model

commit e88291be19b8f52167c3c38804cabc0410578a8f
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Mar 31 16:01:56 2022 +0200

    inbound_email: add support for signed email addresses
    
    These utilities allow us to generate addresses of the form
    `<localpart>+<integer>.<signature>@<domain>`, where the integer is the
    primary key of a given object for which we want to track email
    exchanges. The signature prevents the addresses from being forged, that
    is, the addresses have to be explicitly generated and displayed by the
    web app to be discovered.
    
    The counterpart function retrieves all relevant email addresses from the
    list of recipients of an email message, and validates the signatures to
    recover the integer values that won't have been tampered with.
    
    These new utilities are expected to be used in the views and signal
    handlers pertaining to processing of inbound email messages.

commit f8fb7a04639729d6530f126ce1c7dd87ce82364c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 15:16:11 2022 +0200

    inbound_email: split recipient matching logic out
    
    This allows calling the function on a single recipient rather than on a
    whole message, when one isn't available.

commit 1864b158225d2ae23bc81ac73cb3966caa2c65a1
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 15:15:05 2022 +0200

    Restrict pytest-postgresql to < 4.0.0
    
    Other modules still need psycopg2 and pytest-postgresql 4 introduced a
    hard dependency on psycopg3.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1649/ for more details.

Build has FAILED

Patch application report for D7326 (id=27238)

Could not rebase; Attempt merge onto 8126ea65db...

Merge made by the 'recursive' strategy.
 .pre-commit-config.yaml                            |   1 +
 MANIFEST.in                                        |   1 +
 assets/src/bundles/add_forge/request-dashboard.js  |   4 +-
 swh/web/add_forge_now/apps.py                      |   6 +
 .../migrations/0005_prepare_inbound_email.py       |  30 +++++
 swh/web/add_forge_now/models.py                    |   9 ++
 swh/web/add_forge_now/signal_receivers.py          |  70 +++++++++++
 swh/web/api/views/add_forge_now.py                 |   4 +-
 swh/web/config.py                                  |   1 +
 swh/web/inbound_email/utils.py                     | 126 ++++++++++++++++++-
 .../templates/add_forge_now/request-dashboard.html |   2 +-
 swh/web/tests/api/views/test_add_forge_now.py      |  16 +++
 swh/web/tests/inbound_email/__init__.py            |   0
 swh/web/tests/inbound_email/resources/__init__.py  |   0
 .../tests/inbound_email/resources/multipart.eml    |  24 ++++
 .../tests/inbound_email/resources/plaintext.eml    |  15 +++
 swh/web/tests/inbound_email/test_utils.py          | 133 +++++++++++++++++++++
 17 files changed, 438 insertions(+), 4 deletions(-)
 create mode 100644 swh/web/add_forge_now/migrations/0005_prepare_inbound_email.py
 create mode 100644 swh/web/add_forge_now/signal_receivers.py
 create mode 100644 swh/web/tests/inbound_email/__init__.py
 create mode 100644 swh/web/tests/inbound_email/resources/__init__.py
 create mode 100644 swh/web/tests/inbound_email/resources/multipart.eml
 create mode 100644 swh/web/tests/inbound_email/resources/plaintext.eml
Changes applied before test
commit 69154f9376857c9bd8f46d442696a6a9d0294a8f
Merge: 8126ea65 18c55d79
Author: Jenkins user <jenkins@localhost>
Date:   Tue Apr 5 15:51:28 2022 +0000

    Merge branch 'diff-target' into HEAD

commit 18c55d79c98f206b525a243b86a97c7d73df107e
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 17:30:19 2022 +0200

    Add forge now: process inbound emails
    
    Add a signal receiver to process inbound emails to the address set up in
    config. The text part of the emails is parsed if available, and the raw
    message is saved as a last resort.

commit ea50915c8caf95c45906f94d1b918247a7ca7998
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 17:16:36 2022 +0200

    Add forge now: Add Cc and Reply-To to forge administrator email
    
    This sets Cc: and Reply-To to the inbound email for the add-forge-now feature.

commit 65ef2fac94f634529f807b23c06246ea0f49d11c
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 17:08:47 2022 +0200

    Add forge now: prepare fields for inbound email processing
    
    - add raw message field to request history table
    - add email as choice for the request history actor roles

commit 7802e3b9d01cd177b08a615fe675531eec6d2238
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 10:38:38 2022 +0200

    add_forge_now: add inbound email address to model

commit b690064679fde54f912c8b77b2ea4ac064db19ac
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 17:04:30 2022 +0200

    inbound_email: add function to extract the plaintext from a mail
    
    This function uses the html part if no text part is available.

commit 620d14c60e2c3152bacf9ccaa1d23edbc4a5f98b
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Thu Mar 31 16:01:56 2022 +0200

    inbound_email: add support for signed email addresses
    
    These utilities allow us to generate addresses of the form
    `<localpart>+<integer>.<signature>@<domain>`, where the integer is the
    primary key of a given object for which we want to track email
    exchanges. The signature prevents the addresses from being forged, that
    is, the addresses have to be explicitly generated and displayed by the
    web app to be discovered.
    
    The counterpart function retrieves all relevant email addresses from the
    list of recipients of an email message, and validates the signatures to
    recover the integer values that won't have been tampered with.
    
    These new utilities are expected to be used in the views and signal
    handlers pertaining to processing of inbound email messages.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1684/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1684/console

Below is a diff to allow end to end testing with cypress with inbound email processing.
I only handle the mail sent by us to the forge administrator but simulating his response should be quite similar.

diff --git a/assets/src/bundles/add_forge/add-request-history-item.ejs b/assets/src/bundles/add_forge/add-request-history-item.ejs
index 57b43972..869e1786 100644
--- a/assets/src/bundles/add_forge/add-request-history-item.ejs
+++ b/assets/src/bundles/add_forge/add-request-history-item.ejs
@@ -19,7 +19,9 @@
     </div>
     <div id="collapse<%= index %>" class="collapse" aria-labelledby="historyItem<%= index %>" data-parent="#requestHistory">
       <div class="card-body">
-        <p><%= event.text %></p>
+        <%if (event.text) { %>
+          <pre><%= event.text %></pre>
+        <% } %>
         <%if (event.new_status !== null) { %>
           <p>
            <span>Status changed to:</span> <strong><%= swh.add_forge.formatRequestStatusName(event.new_status) %></strong>
diff --git a/cypress/integration/add-forge-now-request-dashboard.spec.js b/cypress/integration/add-forge-now-request-dashboard.spec.js
index 70bdaeca..2e8c88bb 100644
--- a/cypress/integration/add-forge-now-request-dashboard.spec.js
+++ b/cypress/integration/add-forge-now-request-dashboard.spec.js
@@ -6,29 +6,33 @@
  */
 
 let requestId;
+let requestInboundEmailAddress;
+
+const requestData = {
+  forge_type: 'bitbucket',
+  forge_url: 'test.example.com',
+  forge_contact_email: 'test@example.com',
+  forge_contact_name: 'test user',
+  submitter_forward_username: true,
+  forge_contact_comment: 'test comment'
+};
 
 function createDummyRequest(urls) {
   cy.task('db:add_forge_now:delete');
   cy.userLogin();
 
-  cy.getCookie('csrftoken').its('value').then((token) => {
+  return cy.getCookie('csrftoken').its('value').then((token) => {
     cy.request({
       method: 'POST',
       url: urls.api_1_add_forge_request_create(),
-      body: {
-        forge_type: 'bitbucket',
-        forge_url: 'test.example.com',
-        forge_contact_email: 'test@example.com',
-        forge_contact_name: 'test user',
-        submitter_forward_username: true,
-        forge_contact_comment: 'test comment'
-      },
+      body: requestData,
       headers: {
         'X-CSRFToken': token
       }
     }).then((response) => {
       // setting requestId from response
       requestId = response.body.id;
+      requestInboundEmailAddress = response.body.inbound_email_address;
       // logout the user
       cy.visit(urls.swh_web_homepage());
       cy.contains('a', 'logout').click();
@@ -36,6 +40,45 @@ function createDummyRequest(urls) {
   });
 }
 
+function genEmailSrc() {
+  return `Message-ID: <d5c43e75-2a11-250a-43e3-37034ae3904b@inria.fr>
+Date: Tue, 19 Apr 2022 14:00:56 +0200
+MIME-Version: 1.0
+User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101
+ Thunderbird/91.8.0
+To: ${requestData.forge_contact_email}
+Cc: ${requestInboundEmailAddress}
+Subject: [swh-add_forge_now] Request 1
+Content-Language: en-US
+From: Antoine Lambert <anlambert@softwareheritage.org>
+Content-Type: text/plain; charset=UTF-8; format=flowed
+Content-Transfer-Encoding: 7bit
+
+Dear forge administrator,
+
+The mission of Software Heritage is to collect, preserve and share all the
+publicly available source code (see https://www.softwareheritage.org for more
+information).
+
+We just received a request to add the forge hosted at https://test.cgit.org to the
+list of software origins that are archived, and it is our understanding that you
+are the contact person for this forge.
+
+In order to archive the forge contents, we will have to periodically pull the
+public repositories it contains and clone them into the
+Software Heritage archive.
+
+Would you be so kind as to reply to this message to acknowledge the reception
+of this email and let us know if there are any special steps we should take in
+order to properly archive the public repositories hosted on your infrastructure?
+
+Thank you in advance for your help.
+
+Kind regards,
+The Software Heritage team
+`;
+}
+
 describe('Test add forge now request dashboard load', function() {
 
   before(function() {
@@ -148,15 +191,16 @@ function populateAndSubmitForm() {
 describe('Test forge now request update', function() {
 
   beforeEach(function() {
-    createDummyRequest(this.Urls);
-
-    const url = this.Urls.add_forge_now_request_dashboard(requestId);
-    cy.adminLogin();
-    // intercept GET API on page load
-    cy.intercept(`${this.Urls.api_1_add_forge_request_get(requestId)}**`).as('forgeRequestGet');
-    // intercept update POST API
-    cy.intercept('POST', `${this.Urls.api_1_add_forge_request_update(requestId)}**`).as('forgeRequestUpdate');
-    cy.visit(url);
+    createDummyRequest(this.Urls).then(() => {
+
+      this.url = this.Urls.add_forge_now_request_dashboard(requestId);
+      cy.adminLogin();
+      // intercept GET API on page load
+      cy.intercept(`${this.Urls.api_1_add_forge_request_get(requestId)}**`).as('forgeRequestGet');
+      // intercept update POST API
+      cy.intercept('POST', `${this.Urls.api_1_add_forge_request_update(requestId)}**`).as('forgeRequestUpdate');
+      cy.visit(this.url);
+    });
   });
 
   it('should submit correct details', function() {
@@ -212,6 +256,20 @@ describe('Test forge now request update', function() {
       .should('have.length', 2);
   });
 
+  it('should update the dashboard after sending or receiving email', function() {
+    cy.wait('@forgeRequestGet');
+    cy.task('processAddForgeNowInboundEmail', genEmailSrc());
+    cy.visit(this.url);
+
+    cy.get('#requestHistory')
+      .children()
+      .should('have.length', 2);
+
+    cy.get('#requestHistory')
+      .children()
+      .should('contain', 'New status: Waiting for feedback');
+  });
+
   it('should show an error on API failure', function() {
     cy.intercept('POST',
                  `${this.Urls.api_1_add_forge_request_update(requestId)}**`,
diff --git a/cypress/plugins/index.js b/cypress/plugins/index.js
index fac4b035..a7f7146b 100644
--- a/cypress/plugins/index.js
+++ b/cypress/plugins/index.js
@@ -6,6 +6,7 @@
  */
 
 const axios = require('axios');
+const {execFileSync} = require('child_process');
 const fs = require('fs');
 const sqlite3 = require('sqlite3').verbose();
 
@@ -159,6 +160,16 @@ module.exports = (on, config) => {
       });
       db.close();
       return true;
+    },
+    processAddForgeNowInboundEmail(emailSrc) {
+      try {
+        execFileSync('django-admin',
+                     ['process_inbound_email', '--settings=swh.web.settings.tests'],
+                     {input: emailSrc});
+        return true;
+      } catch (_) {
+        return false;
+      }
     }
   });
   return config;
swh/web/add_forge_now/signal_receivers.py
52

We should decode bytes message to string for proper display in the Web UI.

text=get_message_plaintext(message).decode('utf-8', errors="replace")
60

You need to use the [] operator to get enum by name:

>>> from swh.web.add_forge_now.models import RequestStatus
>>> RequestStatus("PENDING")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/enum.py", line 360, in __call__
    return cls.__new__(cls, value)
  File "/usr/lib/python3.9/enum.py", line 677, in __new__
    raise ve_exc
ValueError: 'PENDING' is not a valid RequestStatus
>>> RequestStatus["PENDING"]
<RequestStatus.PENDING: 'Pending'>

Rebase on top of D7799.

  • Use and tweak @anlambert's suggestions to properly test email processing in cypress.
  • Add tests that the message source endpoint works as expected
olasd published this revision for review.May 10 2022, 4:18 PM
olasd retitled this revision from wip: hook up processing of inbound emails for add_forge_now to Hook up processing of inbound emails for add_forge_now.
olasd edited the summary of this revision. (Show Details)
olasd edited the test plan for this revision. (Show Details)

Build was aborted

Patch application report for D7326 (id=28195)

Rebasing onto 5d37baf27e...

Current branch diff-target is up to date.
Changes applied before test
commit 7650e3e36a61f98ea6eb7281c762f9dcb54d99e7
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue May 10 15:43:22 2022 +0200

    add forge now: only catch expected exceptions in request dashboard
    
    The unqualified catch would hide all exceptions instead of only setting
    an error message when the request itself fails.

commit 4179f3d5510755604a0b8634018efb8c1a1a2011
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 17:30:19 2022 +0200

    Add forge now: process inbound emails
    
    Add a signal receiver to process inbound emails to the address set up in
    config. The text part of the emails is parsed if available, and the raw
    message is saved as a last resort.

commit ecf5d88828036f208a8fb63f22f58fb095100cf1
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Fri Apr 1 17:16:36 2022 +0200

    Add forge now: set Cc and Reply-To for admin emails
    
    This sets the Cc and Reply-To headers to the email address for inbound
    message processing, when sending an email to the forge administrator.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1822/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1822/console

Build is green

Patch application report for D7326 (id=28196)

Rebasing onto 5d37baf27e...

Current branch diff-target is up to date.
Changes applied before test
commit 080c988fe92067d4c82a56884e37eb79f719d5e4
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue May 10 16:21:10 2022 +0200

    Add forge now: only catch expected exceptions in request dashboard
    
    The unqualified catch would hide all exceptions instead of only setting
    an error message when the request itself fails.

commit dc5d570ea30acbcf6e8b1b2a6a8dae9b80a19a9f
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue May 10 16:20:57 2022 +0200

    Add forge now: process inbound emails
    
    Add a signal receiver to process inbound emails to the address set up in
    config. The text part of the emails is parsed if available, and the raw
    message is saved as a last resort.

commit f80541830b617ff48df8360dda5046306c6bd165
Author: Nicolas Dandrimont <nicolas@dandrimont.eu>
Date:   Tue May 10 16:20:46 2022 +0200

    Add forge now: set Cc and Reply-To for admin emails
    
    This sets the Cc and Reply-To headers to the email address for inbound
    message processing, when sending an email to the forge administrator.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1823/ for more details.

This revision is now accepted and ready to land.May 10 2022, 5:07 PM