rebase

grammar.js: Allow using '-' with sort_by options

• query_language: Segregate sort_by and limit from filters
• Add tests for the change mentioned above

query_language/grammar.js: Improve function comments

• query_language: Add test-sitter tests for escaping keywords in filter values
  • Origins with ' and " inside filter values
  • Origins with 'and' and 'or' inside filter values

• grammar.js: Add some comments to improve readability
• Add support for escaping " and ' from the filter values
• Improve and break bulky tree-sitter tests into smaller tests (for readability)
• setup.py: Use super().run() instead of build.run(self)
• Squash commits

• Rebase

• Changes suggested by @zack
• Squash commits

• docs/query-language: Use 'and' and 'or'
• Add details and examples for precedences
• Fix typos

• query_language/grammar.js: Include Z in date regex
• query_language: Add precedence and improve field names
• query_language: Add support for freely using brackets
• README: Add emsdk setup instructions
• setup.py: Add commands for build steps
• query_language/grammar.js: Remove redundancies using functions

• docs: Update query-language specs

Some aspects of the query language that I'm still thinking about :

Can we have some documentation of the query language, included in this diff?

• Add newline at the end of package.json and sample_query

• Improve Makefile and README
• Improve tree-sitter's native test development workflow

• Use locally installed emscripten instead of docker

• Inspecting jenkins build with echo

• Inspect docker container inside builds

• Use docker for emsdk

• Use sh instead of source

• Use emsdk for building .wasm

• Specify directory for build-wasm

Sorry for the mess (so many failed Jenkins builds), I couldn't reproduce the errors on my local machine.

• fix typo

• code to inspect jenkins builds

• Try fixing build failures

• Try fixing errors

Improve build process for .so and .wasm files :

• Makefile.local: Add command inter-dependencies and introduce new commands
• package.json: Use --no-bindings in generate and introduce new scripts to build .wasm files
• setup.py: Make os.system calls configurable with environment variables, build .so and .wasm files and move to static/

• Polish the code

• Install tree-sitter-cli (NodeJS) during builds

• Generate parser before building swh_ql.so

• Fix installation/build errors

• Generate swh_ql.so at builds

• Fix failing build ( because of data_files )

• Move parser to search_language dir
• Introduce Makefile.local and add TreeSitter related commands
• Set data_files of setup.py to 'generated/search_ql.so'

• Add newline at the end of files

• Add test for sort_by : ["date_created"]
• Deduplicate calculation of some variables in _get_sorting_key
• Use iso8601 library to validate date format in instrinsic_metadata fields

string_content and escape_sequence have been adapted from JSON Treesitter grammar

For input

url : "github.com/django/Django" metadata : something qewq
with_visit : true with_visit : false
nb_visits >= 0  nb_visits = 10 nb_visits != 256 nb_visits < 1000 
sort_by : ["nb_visits", "last_revision_date", last_release_date]
last_release_date < 2001-02-13 15:54:21
licenses in ["MIT","BSD X","Apache"]

• I highly recommend you to generate/visualise the corresponding Railroad diagram with https://www.bottlecaps.de/rr/ui
• Checkout P1091 for the Treesitter implementation and some example queries.

• origin_update: Document rejection of metadata date fields if not parsable

• elasticsearch.py: Use "linient: true"
• origin_search: Validate intrinsic_metadata date field format before storing
• test_search: Fix failing tests

In D5964#153292, @vlorentz wrote:

Can you either add tests, or deduplicate this code so we don't need to test every field?

• Move get_expansion to utils.py
• Add tests filters as well as sorting options
• Polish existing code

Add commit body

• Squash
• Minor polishes

• origin_search: Polish the code with get_expansion and other methods

• Fix failing doctest
• Add commit description

• Improve doctest for _nested_get
• Squash

• in_memory: Allow list of licenses and programmingLanguages

• test_in_memory: Add test for _nested_get

• in_memory: Use expanded instrinsic_metadata

• test_search: Test for search on multiple instrinsic_metadata ields

• Use analyzer on the list of licenses and langauges

• origin_search: Allow search for multiple licenses and languages at once
• test_search: Separate tests for programming_language and license

• origin_search: Expose language and license from instrinsic_metadata
• test_search: Add test for language and license

If the only issue is the slowdown, can we keep them nested for now, and benchmark later to see if un-nesting is worth it?

@vlorentz, just in case you missed it, the values don't get duplicated. I'm popping them out of instrinsic_metdadata.
I was trying to avoid nested documents as I've read at many places that they slow down searches. (not sure by how much)