Differential D7197

browse/utils: Robustify content encoding detection
ClosedPublic
Actions

Authored by anlambert on Feb 17 2022, 5:07 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Commits

rDWAPPSd9944bdd56c4: browse/utils: Robustify content encoding detection

Summary

When attempting to re-encode non UTF-8 textual content, use chardet
to find the encoding first and use it if the detection confidence
is really high.

Previously some encoding like SHIFT_JIS (for japanese language) were
not correctly detected and thus content were badly rendered in the
browse Web UI (see example).

Diff Detail

Repository

rDWAPPS Web applications

Lint

Automatic diff as part of commit; lint not applicable.

Unit

Automatic diff as part of commit; unit tests not applicable.

Event Timeline

anlambert created this revision.Feb 17 2022, 5:07 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptFeb 17 2022, 5:07 PM

Build is green

Patch application report for D7197 (id=26100)

Rebasing onto d858c9b457...

Current branch diff-target is up to date.

Changes applied before test

commit d9944bdd56c4df9c6a4c614cea07b16a0e33728c
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Thu Feb 17 16:57:29 2022 +0100

    browse/utils: Robustify content encoding detection
    
    When attempting to re-encode non UTF-8 textual content, use chardet
    to find the encoding first and use it if the detection confidence
    is really high.
    
    Previously some encoding like SHIFT_JIS (for japanese language) were
    not correctly detected and thus content were badly rendered in the
    browse Web UI.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1389/ for more details.

Harbormaster completed remote builds in B26971: Diff 26100.Feb 17 2022, 5:23 PM

anlambert requested review of this revision.Feb 17 2022, 5:23 PM

ardumont accepted this revision.Feb 17 2022, 6:44 PM

This revision is now accepted and ready to land.Feb 17 2022, 6:44 PM

Closed by commit rDWAPPSd9944bdd56c4: browse/utils: Robustify content encoding detection (authored by anlambert). · Explain WhyFeb 18 2022, 11:06 AM

This revision was automatically updated to reflect the committed changes.

anlambert added a commit: rDWAPPSd9944bdd56c4: browse/utils: Robustify content encoding detection.