Page MenuHomeSoftware Heritage
Create Task
Maniphest T1523

Search tools on metadata
Closed, MigratedEdits Locked
Actions

Description

For now, metadata are only used for fulltext/keyword search. There is a lot more we can do. This meta-task will track ideas and progress of their design.

Ideas:

  • Better full-text search (for now, it's keyword-based)
  • Search per field
    • Single field
    • Combining fields
  • Filter results to a subset of metadata file types ("mappings")
  • Stats on frequency of values for each field (e.g., license, ...)
    • on all metadata
    • on a subset of metadata (filtered by a search query)
  • Metadata-only search (right now it's always in OR with URL-based search)

Related Objects
Search...

StatusAssignedTask
 Migratedgitlab-migrationT1523 Search tools on metadata
 Migratedgitlab-migrationT1910 Redesign origin search using a dedicated component (swh-search)
 Migratedgitlab-migrationT2052 Publish swh-search on PyPI
 Migratedgitlab-migrationT2182 Switch production swh-web to use swh-search instead of postgresql search.
 Migratedgitlab-migrationT2183 Switch webapp0 to use swh-search instead of postgresql search.
 Migratedgitlab-migrationT2167 Deploy swh-search
 Migratedgitlab-migrationT2174 Add debian package for swh-search
 Migratedgitlab-migrationT2184 Replay origins to ElasticSearch's "origin" index
 Migratedgitlab-migrationT2497 Create an ElasticSearch cluster tuned for origin/metadata search
 Migratedgitlab-migrationT2590 Finish the indexer -> swh-search pipeline
 Migratedgitlab-migrationT2651 Make the indexer-storage publish its rows to Kafka
 Migratedgitlab-migrationT2652 Make the indexer-storage interface use attr classes instead of dicts
 Migratedgitlab-migrationT2816 Enable the journal-writer for the swh-idx-storage in staging
 Migratedgitlab-migrationT2817 Enable the swh-search environment in staging
 Migratedgitlab-migrationT2876 metadata indexation : ES' dynamic mapping creation fails for field values that are of varying types
 Migratedgitlab-migrationT2904 Create a new production webapp using the frozen index on the staging ES
 Migratedgitlab-migrationT2905 Deploy swh-search for production
 Migratedgitlab-migrationT2936 Update the swh-search journal client to only set "has_visit" on "full" status of the visit
 Migratedgitlab-migrationT2944 Deploy swh-search v0.4.1
 Migratedgitlab-migrationT3037 Reschedule origin-intrinsic-metadata tasks for all origins
 Migratedgitlab-migrationT2780 Enable the journal-writer for the swh-idx-storage in production
 Migratedgitlab-migrationT3040 [production] Enable swh-search's journal-client for indexed objects
 Migratedgitlab-migrationT3041 [production] Provision enough space for the search ES cluster to ingest all intrinsic metadata
 Migratedgitlab-migrationT3373 Metadata search is failing due to a boolean field in the mapping of the metadata fields
 Migratedgitlab-migrationT3391 [swh-search] Deploy v0.9.0 on staging and execute a full origin and metadata reindexation
 Migratedgitlab-migrationT3392 [staging] Properly recreate the origin_intrinsic_metadata topic
 Migratedgitlab-migrationT3398 [swh-search] Deploy v0.9.0 on production and execute a full origin and metadata reindexation
 Migratedgitlab-migrationT3047 Enable to search in origin metadata with swh-search in webapp
 Migratedgitlab-migrationT3058 Metadata search is failing with "failed to parse date field"
 Migratedgitlab-migrationT3060 Deploy swh-search v0.6.0 in **staging**
 Migratedgitlab-migrationT3076 [swh-search] Improve the index/mapping migration process
 Migratedgitlab-migrationT3083 Deploy swh-search v0.7.0/v0.7.1
 Migratedgitlab-migrationT2254 textual search language for the Web UI

Event Timeline

vlorentz triaged this task as Normal priority.Feb 8 2019, 1:09 PM
vlorentz created this task.
vlorentz claimed this task.Feb 11 2019, 11:40 AM
vlorentz added a project: Restricted Project.
zack updated the task description. (Show Details)Feb 23 2019, 2:13 PM

I've added an item to the above list (metadata-only search); I think the ideal UI would be a single form with two checkboxes under it, one enabling URL-based search (enabled by default), one enabling metadata-based search (disabled by default).

Relatedly, I don't understand what the "stats" items do in this list, they don't seem to be related to metadata-based *search*.

vlorentz added a comment.Feb 25 2019, 9:49 AM

right now it's always in OR with URL-based search

Are you sure? Looking at the code, it seems to me it searches either in URL or in metadata, but not both at the same time.

zack added a comment.Feb 25 2019, 9:57 AM

That's the impression i got from testing. Either way, the current UI & semantics are bad, the proposed ones would be much better.

vlorentz added a comment.Feb 25 2019, 10:20 AM

That's the impression i got from testing.

The URL is often part of the metadata, in the codeRepository key.

Relatedly, I don't understand what the "stats" items do in this list, they don't seem to be related to metadata-based *search*.

The second item ("on a subset of metadata") involves search

vlorentz removed vlorentz as the assignee of this task.Aug 20 2019, 1:47 PM
vlorentz added a subtask: T1910: Redesign origin search using a dedicated component (swh-search).Aug 21 2019, 2:34 PM
vlorentz changed the status of subtask T1910: Redesign origin search using a dedicated component (swh-search) from Open to Work in Progress.Dec 20 2019, 3:26 PM
vlorentz renamed this task from Search tools on metadata (meta task) to Search tools on metadata.Jan 22 2020, 4:23 PM
vlorentz added a project: meta-task.
vlorentz added a subtask: T2254: textual search language for the Web UI.Jan 29 2020, 1:36 PM
moranegg moved this task from Backlog to Implementation on the Metadata workflow board.Sep 18 2020, 1:50 PM
KShivendu added a subscriber: KShivendu.Feb 23 2021, 11:49 AM
KShivendu added a comment.Feb 23 2021, 1:41 PM

Hi everyone, I am new here. I am excited by the idea of preserving open-source code and would love to contribute this year during GSoC. I do have a decent knowledge of Elasticsearch, Django, and Web Development in general.

I want to work on the metadata search engine idea as mentioned here. But I noticed that the GSoC page for 2020 is exactly the same as that of 2021 (except the banner). So I wanted to ask if this project can be a part of GSoC this year?

Cheers!
Kumar Shivendu

vlorentz added a comment.Feb 23 2021, 2:50 PM

@KShivendu Thanks for your interest!

Yes it can be part of GSoC this year, we added it earlier this month (history), it's only shown on the GSoC 2020 page because it is automatically generated from the same list of projects

vlorentz closed subtask T1910: Redesign origin search using a dedicated component (swh-search) as Resolved.Mar 25 2021, 11:16 AM
hack3r-0m added a subscriber: hack3r-0m.Apr 7 2021, 7:59 AM
hack3r-0m added a comment.Apr 7 2021, 8:22 AM

hi, I would like to get familiar with this, as mentioned in the wiki ,

By the time GSoC starts, it will be implemented by a very small Python service (under 100 lines of code) backed by ElasticSearch.

can anybody point me to the relevant repo?

ardumont added a subscriber: ardumont.Apr 7 2021, 8:53 AM

can anybody point me to the relevant repo?

Most likely, swh-search [1]

[1] https://forge.softwareheritage.org/source/swh-search/

gitlab-migration changed the status of subtask T1910: Redesign origin search using a dedicated component (swh-search) from Resolved to Migrated.Jan 8 2023, 9:59 PM
gitlab-migration closed this task as Migrated.Jan 8 2023, 10:21 PM
gitlab-migration claimed this task.
gitlab-migration added a subscriber: gitlab-migration.

This task has been migrated to GitLab.

gitlab-migration closed subtask T2254: textual search language for the Web UI as Migrated.Jan 8 2023, 10:22 PM
About · Developer information · Legal