Differential D7197

browse/utils: Robustify content encoding detection
ClosedPublic
Actions

Authored by anlambert on Feb 17 2022, 5:07 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Commits

rDWAPPSd9944bdd56c4: browse/utils: Robustify content encoding detection

Summary

When attempting to re-encode non UTF-8 textual content, use chardet
to find the encoding first and use it if the detection confidence
is really high.

Previously some encoding like SHIFT_JIS (for japanese language) were
not correctly detected and thus content were badly rendered in the
browse Web UI (see example).

Diff Detail

Repository

rDWAPPS Web applications

Branch

robustify-content-encoding-detection

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 26971
Build 42170: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 42169: arc lint + arc unit

Event Timeline

anlambert created this revision.Feb 17 2022, 5:07 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 17 2022, 5:07 PM

Build is green

Patch application report for D7197 (id=26100)

Rebasing onto d858c9b457...

Current branch diff-target is up to date.

Changes applied before test

commit d9944bdd56c4df9c6a4c614cea07b16a0e33728c
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Feb 17 16:57:29 2022 +0100

    browse/utils: Robustify content encoding detection
    
    When attempting to re-encode non UTF-8 textual content, use chardet
    to find the encoding first and use it if the detection confidence
    is really high.
    
    Previously some encoding like SHIFT_JIS (for japanese language) were
    not correctly detected and thus content were badly rendered in the
    browse Web UI.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1389/ for more details.

Harbormaster completed remote builds in B26971: Diff 26100.Feb 17 2022, 5:23 PM

anlambert requested review of this revision.Feb 17 2022, 5:23 PM

ardumont accepted this revision.Feb 17 2022, 6:44 PM

This revision is now accepted and ready to land.Feb 17 2022, 6:44 PM

Closed by commit rDWAPPSd9944bdd56c4: browse/utils: Robustify content encoding detection (authored by anlambert). · Explain WhyFeb 18 2022, 11:06 AM

This revision was automatically updated to reflect the committed changes.

anlambert added a commit: rDWAPPSd9944bdd56c4: browse/utils: Robustify content encoding detection.