diff --git a/docs/faq/index.rst b/docs/faq/index.rst index a34932f..c4e4485 100644 --- a/docs/faq/index.rst +++ b/docs/faq/index.rst @@ -1,254 +1,259 @@ .. _faq: Frequently Asked Questions ************************** +.. contents:: + :depth: 3 + :local: +.. + .. _faq_prerequisites: Prerequisites for code contributions ==================================== What are the Skills required to be a code contributor? ------------------------------------------------------ It depends on what area you want to work on. The `internships`_ postings list specific skills required. Generally, only Python and basic Git knowledge are required. Feel free to contact us on one of the `development channels `__ for details. What are the minimum system requirements (hardware/software) to run SWH locally? -------------------------------------------------------------------------------- Python3.7 or newer is required. See the :ref:`developer setup documentation ` for more details. .. _faq_getting_started: Getting Started =============== What are the must read docs before I start contributing? -------------------------------------------------------- We recommend you read the top links listed at from the :ref:`documentation home page ` in order: getting started, contributing, and architecture, as well as the data model. Where can I see the getting started guide for developers? --------------------------------------------------------- See :ref:`getting-started` tutorial. How do I find an easy ticket to get started? -------------------------------------------- We maintain a `list of easy tickets `__ to work on, see the `Easy hacks page `__ for more details. I am skilled in one specific technology, can I find tickets requiring that skill? --------------------------------------------------------------------------------- Unfortunately, not at the moment. But you can look at the `internships`_ list to look for something matching this skill, and this may allow you to find topics to search for in the `bug tracking system`_. Either way, feel free to contact our developers through any of the `development channels`_, we would love to work with you. Where should I ask for technical help? -------------------------------------- You can choose one of the following: * `development channels`_ * `contact form`_ for any enquiries .. _faq_run_swh: Running an SWH instance locally =============================== I have SWH stack running in my local. How do I get some initial data to play around? ------------------------------------------------------------------------------------ You can setup a job on your local machine, for this you can :ref:`schedule a listing task ` for example. Doing so on small forge, will allow you to load some repositories. Or you can also trigger directly :ref:`loading from the cli `. I have a SWH stack running in local, How do I setup a lister/loader job? ------------------------------------------------------------------------ See the :ref:`"Managing tasks" chapter ` in the Docker environment documentation. How can I create a user in my local instance? --------------------------------------------- We cannot right now. Stay either anonymous or use the user "test" (password "test") or the user ambassador (password "ambassador"). Should I run/test the web app in any particular browser? -------------------------------------------------------- We expect the web app to work on all major browsers. It uses mostly straightforward HTML/CSS and a little Javascript for search and source code highlighting, so testing in a single browser is usually enough. .. _faq_dataset: Getting sample datasets ======================= Is there a way to connect to SWH archived (production) database from my local machine? -------------------------------------------------------------------------------------- We provide the archive as a dataset on public clouds, see the :ref:`swh-dataset documentation `. We can also provide read access to one of the main databases on request, `contact us`_. .. _faq_error_bugs: Errors and bugs =============== I found a bug/improvement in the system, where should I report it? ------------------------------------------------------------------ Please report it on our `bug tracking system`_. First create an account, then create a bug report using the "Create task" button. You should get some feedback within a week (at least someone triaging your issue). If not, `get in touch with us `_ to make sure we did not miss it. .. _faq_legal: Legal matters ============= Do I need to sign a form to contribute code? -------------------------------------------- Yes, on your first diff, you will have to sign such document. As long as it's not signed, your diff content won't be visible. Will my name be added to a CONTRIBUTORS file? --------------------------------------------- You will be asked during review to add yourself. .. _faq_code_review: Code Review =========== I found a straightforward typo fix, should my fix go through the entire code review process? -------------------------------------------------------------------------------------------- You are welcome to drop us a message at one of the `development channels`_, we will pick it up and fix it so you don't have to follow the whole :ref:`code review process `. What tests I should run before committing the code? --------------------------------------------------- Mostly run `tox` (or `pytest`) to run the unit tests suite. When you will propose a patch in our forge, the continuous integration factory will trigger a build (using `tox` as well). I am getting errors while trying to commit. What is going wrong? ---------------------------------------------------------------- Ensure you followed the proper guide to :ref:`setup your environment ` and try again. If the error persists, you are welcome to drop us a message at one of the `development channels`_ Is there a format/guideline for writing commit messages? -------------------------------------------------------- See the :ref:`git-style-guide` Is there some recommended git branching strategy? ------------------------------------------------- It's left at the developer's discretion. Mostly people hack on their feature, then propose a diff from a git branch or directly from the master branch. There is no imperative. The only imperative is that for a feature to be packaged and deployed, it needs to land first in the master branch. how should I document the code I contribute to SWH? --------------------------------------------------- Any new feature should include documentation in the form of comments and/or docstrings. Ideally, they should also be documented in plain English in the repository's `docs/` folder if relevant to a single package, or in the main `swh-docs` repository if it is a transversal feature. .. _faq_api: Software Heritage API ===================== How do I generate API usage credentials? ---------------------------------------- See the :ref:`Authentication guide `. Is there a page where I can see all the API endpoints? ------------------------------------------------------ See the `API endpoint listing page`_. What are the usage limits for SWH APIs? --------------------------------------- Maximum number of permitted requests per hour: * 120 for anonymous users * 1200 for authenticated users It's described in the `rate limit documentation page`_. .. It's temporarily here but it should be moved into its own sphinx instance at some point in the future. .. _faq_sysadm: System Administration ===================== How does SWH release? --------------------- Release is mostly done: - first in docker (somewhat as part of the development process) - secondly packaged and deployed on staging (mostly) - thirdly the same package is deployed on production Is there a release cycle? ------------------------- When a functionality is ready (tests ok, landed in master, docker run ok), the module is tagged. The tag is pushed. This triggers a packaging build process. When the package is ready, depending on the module [1], sysadms deploy the package with the help of puppet. [1] swh-web module is mostly automatic. Other modules are not yet automatic as some internal state migration (dbs) often enters the release cycle and due to the data volume, that may need human intervention. .. _API endpoint listing page: https://archive.softwareheritage.org/api/1/ .. _rate limit documentation page: https://archive.softwareheritage.org/api/#rate-limiting .. _bug tracking system: https://forge.softwareheritage.org/ .. _contact form: https://www.softwareheritage.org/contact/ .. _contact us: https://www.softwareheritage.org/contact/ .. _development channels: https://www.softwareheritage.org/community/developers/ .. _internships: https://wiki.softwareheritage.org/wiki/Internships diff --git a/docs/index.rst b/docs/index.rst index 42f95b4..4f94193 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,242 +1,235 @@ .. _swh-docs: Software Heritage - Development Documentation ============================================= Getting started --------------- * :ref:`getting-started` → deploy a local copy of the Software Heritage software stack in less than 5 minutes, or * :ref:`developer-setup` → get a working development setup that allows to hack on the Software Heritage software stack +* :ref:`faq` Contributing ------------ * :ref:`patch-submission` → learn how to submit your patches to the Software Heritage codebase * :ref:`code-review` → rules and guidelines to review code in Software Heritage * :ref:`python-style-guide` → how to format the Python code you write Architecture ------------ * :ref:`architecture-overview` → get a glimpse of the Software Heritage software architecture * :ref:`mirror` → learn what a Software Heritage mirror is and how to set up one * :ref:`Metadata workflow ` → learn how Software Heritage stores and handles metadata * :ref:`Keycloak ` → learn how to use Keycloak, the authentication system used by |swh|'s web interface and public APIs Data Model and Specifications ----------------------------- * :ref:`persistent-identifiers` Specifications of the SoftWare Heritage persistent IDentifiers (SWHID). * :ref:`data-model` Documentation of the main |swh| archive data model. * :ref:`journal-specs` Documentation of the Kafka journal of the |swh| archive. Tutorials --------- * :ref:`testing-guide` * :doc:`/tutorials/issue-debugging-monitoring` * :ref:`Listing the content of your favorite forge ` and :ref:`running a lister in Docker ` * :ref:`Add a new swh package ` * :ref:`doc-contribution` -Frequently Asked Questions --------------------------- - -.. toctree:: - :maxdepth: 2 - - faq/index - Roadmap ------- * :ref:`roadmap-2021` Engineering ----------- * :ref:`infrastructure` Components ---------- Here is brief overview of the most relevant software components in the Software Heritage stack, in alphabetical order. For a better introduction to the architecture, see the :ref:`architecture-overview`, which presents each of them in a didactical order. Each component name is linked to the development documentation of the corresponding Python module. :ref:`swh.auth ` low-level library used by modules needing keycloak authentication :ref:`swh.core ` low-level utilities and helpers used by almost all other modules in the stack :ref:`swh.counters ` service providing efficient estimates of the number of objects in the SWH archive, using Redis's Hyperloglog :ref:`swh.dataset ` public datasets and periodic data dumps of the archive released by Software Heritage :ref:`swh.deposit ` push-based deposit of software artifacts to the archive swh.docs developer documentation (used to generate this doc you are reading) :ref:`swh.fuse ` Virtual file system to browse the Software Heritage archive, based on `FUSE `_ :ref:`swh.graph ` Fast, compressed, in-memory representation of the archive, with tooling to generate and query it. :ref:`swh.indexer ` tools and workers used to crawl the content of the archive and extract derived information from any artifact stored in it :ref:`swh.journal ` persistent logger of changes to the archive, with publish-subscribe support :ref:`swh.lister ` collection of listers for all sorts of source code hosting and distribution places (forges, distributions, package managers, etc.) :ref:`swh.loader-core ` low-level loading utilities and helpers used by all other loaders :ref:`swh.loader-git ` loader for `Git `_ repositories :ref:`swh.loader-mercurial ` loader for `Mercurial `_ repositories :ref:`swh.loader-svn ` loader for `Subversion `_ repositories :ref:`swh.loader-cvs ` loader for `CVS `_ repositories :ref:`swh.model ` implementation of the :ref:`data-model` to archive source code artifacts :ref:`swh.objstorage ` content-addressable object storage :ref:`swh.objstorage.replayer ` Object storage replication tool :ref:`swh.scanner ` source code scanner to analyze code bases and compare them with source code artifacts archived by Software Heritage :ref:`swh.scheduler ` task manager for asynchronous/delayed tasks, used for recurrent (e.g., listing a forge, loading new stuff from a Git repository) and one-off activities (e.g., loading a specific version of a source package) :ref:`swh.search ` search engine for the archive :ref:`swh.storage ` abstraction layer over the archive, allowing to access all stored source code artifacts as well as their metadata :ref:`swh.vault ` implementation of the vault service, allowing to retrieve parts of the archive as self-contained bundles (e.g., individual releases, entire repository snapshots, etc.) :ref:`swh.web ` Web application(s) to browse the archive, for both interactive (HTML UI) and mechanized (REST API) use :ref:`swh.web.client ` Python client for :ref:`swh.web ` Dependencies ------------ The dependency relationships among the various modules are depicted below. .. _py-deps-swh: .. figure:: images/py-deps-swh.svg :width: 1024px :align: center Dependencies among top-level Python modules (click to zoom). Archive ------- * :ref:`Archive ChangeLog `: notable changes to the archive over time Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * `URLs index `_ * :ref:`search` * :ref:`glossary` .. ensure sphinx does not complain about index files not being included .. toctree:: :maxdepth: 2 :caption: Contents: :titlesonly: :hidden: getting-started/index architecture/index contributing/index tutorials/index faq/index roadmap/roadmap-2021.rst infrastructure/index swh.auth swh.core swh.counters swh.dataset swh.deposit swh.fuse swh.graph swh.indexer swh.journal swh.lister swh.loader swh.model swh.objstorage swh.objstorage.replayer swh.scanner swh.scheduler swh.search swh.storage swh.vault swh.web swh.web.client archive-changelog journal Python modules autodocumentation diff --git a/user/faq/index.rst b/user/faq/index.rst index 7bab05a..e80fa38 100644 --- a/user/faq/index.rst +++ b/user/faq/index.rst @@ -1,81 +1,86 @@ .. _faq: Frequently Asked Questions ************************** +.. contents:: + :depth: 3 + :local: +.. + .. _faq_roadmap: Roadmap ======= Where can I find the SWH roadmap? --------------------------------- The roadmap is accessible on the :ref:`development docs `. .. _faq_savecodenow: Save Code Now ============= What file formats are supported with the save code now? ------------------------------------------------------- The save code now is intended publicly for code repositories with git, mercurial or svn version control systems. For authenticated user with role ambassador, it is possible to do save code now requests on zip or tarballs. Is there a license existence check (the one included in the source code)? ------------------------------------------------------------------------- No. How are Save code now requests handled? --------------------------------------- `Save code now `__ requests on known forges for origins are scheduled as soon as possible. Unknown origins are put in a moderation queue waiting for human vetting (Ambassadors or staff). what file formats are supported with the save code now? ------------------------------------------------------- The "save code now" supports public code repositories with git, mercurial or svn version control systems. Ambassadors, however can also deposits for a given origins multiple artifacts (zip, tarballs). My "Save code now" request is stuck in pending status. what should I do? ------------------------------------------------------------------------ If the repository you want to save is already in the authorized list (e.g. GitLab, GitHub, ...), the repositories will be saved without approval, so this should not last more than a few hours. If it's not the case, requests should be approved within a few days (minus French bank holidays), and loaded in a few hours. If your repository is still pending after this time, this is most likely a bug. `Get in touch with us `__ to check whether we are aware of this potential issue or are working on it. .. _faq_search: Search ====== I cannot find all my "releases" in a git repository archived in Software Heritage, what should I do? ---------------------------------------------------------------------------------------------------- Do not worry, the repository has been saved in full. What you are witnessing is just a terminological difference between what platforms like GitHub calls “releases” (any non annotated git tag) and what we call “releases” (a node in the Merkle tree, which corresponds to a git annotated tag). Let’s say your “release” is FinalSubmission. If you click on the branch dropdown menu on the Software Heritage Web interface you’ll find what you are looking listed as “refs/tags/FinalSubmission”. If you want a “release” to appear in our web interface you should create your tags using “git tag -a”, instead of simply “git tag” (and then archive your repository again). How can I search the SWH archive? is it possible to search over metadata? ------------------------------------------------------------------------- At the moment searching is possible using the url of a repository, package or deposit (a.k.a the origin of the source code). You can use the checkbox "search in metadata (instead of URL)" to search over intrinsic metadata. diff --git a/user/index.rst b/user/index.rst index eb5e7f3..75bbe39 100644 --- a/user/index.rst +++ b/user/index.rst @@ -1,44 +1,35 @@ .. _swh-docs: .. Software Heritage documentation master file, created by sphinx-quickstart on Thu Jun 17 11:23:12 2021. You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. Software Heritage User documentation ==================================== Getting started --------------- - -... +* :ref:`faq` Tutorials --------- ... -Frequently Asked Questions ---------------------------- - -.. toctree:: - :maxdepth: 2 - - faq/index - Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * :ref:`search` .. toctree:: :maxdepth: 2 :caption: Contents: :titlesonly: :hidden: faq/index