assets/highlightjs: Add new languages to highlight and fix some issues
ClosedPublic
Actions

Authored by anlambert on Jan 25 2022, 3:23 PM.

Details

Reviewers

ardumont

Group Reviewers

Reviewers

Commits

rDWAPPS7ea3aab33ed8: assets/highlightjs: Add new languages to highlight and fix some issues
rDWAPPSbce990970824: webpack/weblabels-plugin: Read package.json file only when required

Summary

Add support for new programming languages to highlight in the webapp
by integrating highlight.js language plugins in the highlightjs chunk
generated by webpack.

Fix issue in Python code that was preventing to map a mimetype to a
language through its aliases.

The following new languages will be highlighted once these changes deployed:

Diff Detail

Repository

rDWAPPS Web applications

Branch

highlightjs-more-languages

Lint

No Linters Available

Unit

No Unit Test Coverage

Build Status

Buildable 26328
Build 41167: Phabricator diff pipeline on jenkins	Jenkins console · Jenkins
Build 41166: arc lint + arc unit

Unit TestsFailed
View All

	Time	Test
	163 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.web.tests.browse.views.test_content::Tests / Python tests / test_content_view_no_highlight @pytest.fixture(scope="function") def content_application_no_highlight(): """Fixture returning a random textual content with mimetype
	10,635 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.web.tests.test_random_fixtures::Tests / Python tests / test_random_fixtures def test_random_fixtures(): """Check random fixture values will be different when random seed is not explicitly provided.
	10,293 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.web.tests.test_random_fixtures::Tests / Python tests / test_random_fixtures_with_seed def test_random_fixtures_with_seed(): """Check random fixture values will be the same when random seed is explicitly provided through a custom pytest option.
	572 ms	Jenkins > should have no selected language in dropdown::Tests / Cypress tests / Run cypress tests / Test Content Language Select When Language is not detected should have no selected language in dropdown AssertionError: expected 'plaintext' to equal null at Context.eval (http://localhost:5004/__cypress/tests?p=cypress/integration/language-select.spec.js:340:16)
	221 ms	Jenkins > .tox.py3.lib.python3.7.site-packages.swh.web.tests.admin.test_deposit::Tests / Python tests / test_deposit_admin_view_available_for_staff_user
		View Full Test Results (4 Failed · 814 Passed · 5 Skipped)

Event Timeline

anlambert created this revision.Jan 25 2022, 3:23 PM

Herald added a reviewer: Reviewers. · View Herald TranscriptJan 25 2022, 3:23 PM

anlambert added inline comments.Jan 25 2022, 3:25 PM

assets/config/webpack-plugins/generate-weblabels-webpack-plugin/index.js
155–164	This change lives in a separate commit.

Build has FAILED

Patch application report for D7032 (id=25496)

Rebasing onto 1f6bf0c1f5...

Current branch diff-target is up to date.

Changes applied before test

commit c7e7d5c76918040cbefe56e5c566f5ad6537811e
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Jan 25 14:36:30 2022 +0100

    assets/highlightjs: Add new languages to highlight and fix some issues
    
    Add support for new programming languages to highlight in the webapp
    by integrating highlight.js language plugins in the highlightjs chunk
    generated by webpack.
    
    Fix issue in Python code that was preventing to map a mimetype to a
    language through its aliases.

commit bce990970824d00ec818fdfb677df54652976a3b
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Jan 25 14:37:55 2022 +0100

    webpack/weblabels-plugin: Read package.json file only when required
    
    Some module folders do not have package.json file in it but licenses can
    still be overridden using the licenseOverride option of the plugin.
    
    Previously, package.json file was attempted to be read regardless of
    possible license override, which was causing error.

Link to build: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1301/
See console output for more information: https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1301/console

Harbormaster returned this revision to the author for changes because remote builds failed.Jan 25 2022, 3:40 PM

Harbormaster failed remote builds in B26328: Diff 25496!

Fix tests

Build is green

Patch application report for D7032 (id=25506)

Rebasing onto 1f6bf0c1f5...

Current branch diff-target is up to date.

Changes applied before test

commit 7ea3aab33ed8f7c9a748f6dfa85a87645aa6079e
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Jan 25 14:36:30 2022 +0100

    assets/highlightjs: Add new languages to highlight and fix some issues
    
    Add support for new programming languages to highlight in the webapp
    by integrating highlight.js language plugins in the highlightjs chunk
    generated by webpack.
    
    Fix issue in Python code that was preventing to map a mimetype to a
    language through its aliases.

commit bce990970824d00ec818fdfb677df54652976a3b
Author: Antoine Lambert <anlambert@softwareheritage.org>
Date:   Tue Jan 25 14:37:55 2022 +0100

    webpack/weblabels-plugin: Read package.json file only when required
    
    Some module folders do not have package.json file in it but licenses can
    still be overridden using the licenseOverride option of the plugin.
    
    Previously, package.json file was attempted to be read regardless of
    possible license override, which was causing error.

See https://jenkins.softwareheritage.org/job/DWAPPS/job/tests-on-diff/1303/ for more details.

Harbormaster completed remote builds in B26337: Diff 25506.Jan 25 2022, 4:40 PM

anlambert requested review of this revision.Jan 25 2022, 4:40 PM

anlambert added a child revision: D7037: package.json: Upgrade dependencies.Jan 25 2022, 4:46 PM

lgtm

couple of question/suggestion inline.

assets/config/webpack-plugins/dump-highlightjs-languages-data-plugin.js
49	can't we do a for loop or something? (and another for the registerLanguage stanza).
swh/web/common/highlightjs.py
31	why the pop though? I see some mapping about it later on.

This revision is now accepted and ready to land.Jan 26 2022, 9:31 AM

anlambert added inline comments.Jan 26 2022, 11:18 AM

assets/config/webpack-plugins/dump-highlightjs-languages-data-plugin.js
49	Maybe here with the require syntax but not in the `highlightjs.js` file with the import syntax. So I will keep it that way.
swh/web/common/highlightjs.py
31	Because it clashes with the `plaintext` language. Nevertheless, `robots.txt` files will be correctly highlighted thanks to the `_filename_to_hljs_language` dict.