diff --git a/docs/faq/faq.rst b/docs/faq/faq.rst deleted file mode 100644 index 662a7df..0000000 --- a/docs/faq/faq.rst +++ /dev/null @@ -1,247 +0,0 @@ -.. _faq_prerequisites: - -Prerequisites -============= - -What are the Skills required to be a code contributor? ------------------------------------------------------- - -It depends on what area you want to work on. `Internships -`__ postings list specific skills -required. Generally, only Python and basic Git knowledge are required. Feel free to -contact us on one of the `development channels -`__ for details. - -What are the minimum system requirements (hardware/software) to run SWH locally? --------------------------------------------------------------------------------- - -Python3.7 or newer is required. See the `developer setup documentation -`__ for -more details. - - -.. _faq_getting_started: - -Getting Started -================ - -What are the must read docs before I start contributing? --------------------------------------------------------- - -We recommend you read the top links listed at from the `documentation home page -`__ in order: getting started, -contributing, and architecture, as well as the data model. - -Where can I see the getting started guide for developers? ---------------------------------------------------------- - -See :ref:`getting-started` tutorial. - -How do I find an easy ticket to get started? --------------------------------------------- - -We maintain a `list of easy tickets -`__ to work on, see -the `Easy hacks page `__ for more -details. - -I am skilled in one specific technology, can I find tickets requiring that skill? ---------------------------------------------------------------------------------- - -Unfortunately, not at the moment. But you can look at the `Internship list -`__ to look for something matching -this skill, and this may allow you to find topics to search for in the `bug tracking -system `__. - -Either way, feel free to `contact our developers -`__, we would love to work with -you. - -Where should I ask for technical help? --------------------------------------- - -You can choose one of the following: - -* `development channels `__ -* `contact form for any enquiries `__ - -.. _faq_run_swh: - -Running SWH instance locally -============================ - -I have SWH stack running in my local. How do I get some initial data to play around? ------------------------------------------------------------------------------------- - -You can setup a job on your local machine, for this you can `schedule a listing task -`__ -for example. Doing so on small forge, will allow you to load some repositories. - -Or you can also trigger directly `loading from the cli -`__. - -I have a SWH stack running in local, How do I setup a lister/loader job? ------------------------------------------------------------------------- - -See the `"Managing tasks" chapter -`__ -in the Docker environment documentation. - -How can I create a user in my local instance? ---------------------------------------------- - -We cannot right now. Stay either anonymous or use the user "test" (password "test") or -the user ambassador (password "ambassador"). - -Should I run/test the web app in any particular browser? --------------------------------------------------------- - -We expect the web app to work on all major browsers. It uses mostly straightforward -HTML/CSS and a little Javascript for search and source code highlighting, so testing in -a single browser is usually enough. - -.. _faq_dataset: - -Dataset -======= - -Is there a way to connect to SWH archived (production) database from my local machine? --------------------------------------------------------------------------------------- - -We provide the archive as a dataset on public clouds, see the `swh-dataset -documentation `__. We can -also provide read access to one of the main databases on request `contact -us `__. - -.. _faq_error_bugs: - -Error and bugs -============== - -I found a bug/improvement in the system, where should I report it? ------------------------------------------------------------------- - -Please report it on our `bug tracking system `__. -First create an account, then create a bug report using the "Create task" button. You -should get some feedback within a week (at least someone triaging your issue). If not, -`get in touch with us `__ to -make sure we did not miss it. - -.. _faq_legal: - -Legal -===== - -Do I need to sign a form to contribute code? --------------------------------------------- - -Yes, on your first diff, you will have to sign such document. -As long as it's not signed, your diff content won't be visible. - -Will my name be added to a CONTRIBUTORS file? ---------------------------------------------- - -You will be asked during review to add yourself. - -.. _faq_code_review: - -Code Review -=========== - -I found a straightforward typo fix, should my fix go through the entire code review process? --------------------------------------------------------------------------------------------- - -You are welcome to drop us a message at one of the `development -channels `__, we will pick it up -and fix it so you don't have to follow the whole `code review -process `__. - -What tests I should run before committing the code? ---------------------------------------------------- - -Mostly run `tox` (or `pytest`) to run the unit tests suite. When you will propose a -patch in our forge, the continuous integration factory will trigger a build (using `tox` -as well). - -I am getting errors while trying to commit. What is going wrong? ----------------------------------------------------------------- - -Ensure you followed the proper guide to `setup your -environment `__ -and try again. If the error persists, you are welcome to drop us a message at one of the -`development channels `__ - -Is there a format/guideline for writing commit messages? --------------------------------------------------------- - -See the `Git style guide `__ - -Is there some recommended git branching strategy? -------------------------------------------------- - -It's left at the developer's discretion. Mostly people hack on their feature, then -propose a diff from a git branch or directly from the master branch. There is no -imperative. The only imperative is that for a feature to be packaged and deployed, it -needs to land first in the master branch. - -how should I document the code I contribute to SWH? ---------------------------------------------------- - -Any new feature should include documentation in the form of comments and/or docstrings. -Ideally, they should also be documented in plain English in the repository's `docs/` -folder if relevant to a single package, or in the main `swh-docs` repository if it is a -transversal feature. - -.. _faq_api: - -API -=== - -How do I generate API usage credentials? ----------------------------------------- - -See the `Authentication guide `__. - -Is there a page where I can see all the API endpoints? ------------------------------------------------------- - -See the `API endpoint listing page `__. - -What are the usage limits for SWH APIs? ---------------------------------------- - -Maximum number of permitted requests per hour: - -* 120 for anonymous users -* 1200 for authenticated users - -It's described in the `rate limit documentation page -`__. - -.. It's temporarily here but it should be moved into its own sphinx instance at some - point in the future. - -.. _faq_sysadm: - -System Administration -===================== - -How does SWH release? ---------------------- - -Release is mostly done: -- first in docker (somewhat as part of the development process) -- secondly packaged and deployed on staging (mostly) -- thirdly the same package is deployed on production - -Is there a release cycle? -------------------------- - -When a functionality is ready (tests ok, landed in master, docker run ok), the module is -tagged. The tag is pushed. This triggers a packaging build process. When the package is -ready, depending on the module [1], sysadms deploy the package with the help of puppet. - -[1] swh-web module is mostly automatic. Other modules are not yet automatic as some -internal state migration (dbs) often enters the release cycle and due to the data -volume, that may need human intervention. - diff --git a/docs/faq/index.rst b/docs/faq/index.rst index 8a0855c..9c586a0 100644 --- a/docs/faq/index.rst +++ b/docs/faq/index.rst @@ -1,10 +1,252 @@ .. _faq: Frequently Asked Questions -========================== +************************** -.. toctree:: - :maxdepth: 2 - :titlesonly: +.. _faq_prerequisites: + +Prerequisites for code contributions +==================================== + +What are the Skills required to be a code contributor? +------------------------------------------------------ + +It depends on what area you want to work on. `Internships +`__ postings list specific skills +required. Generally, only Python and basic Git knowledge are required. Feel free to +contact us on one of the `development channels +`__ for details. + +What are the minimum system requirements (hardware/software) to run SWH locally? +-------------------------------------------------------------------------------- + +Python3.7 or newer is required. See the `developer setup documentation +`__ for +more details. + + +.. _faq_getting_started: + +Getting Started +=============== + +What are the must read docs before I start contributing? +-------------------------------------------------------- + +We recommend you read the top links listed at from the `documentation home page +`__ in order: getting started, +contributing, and architecture, as well as the data model. + +Where can I see the getting started guide for developers? +--------------------------------------------------------- + +See :ref:`getting-started` tutorial. + +How do I find an easy ticket to get started? +-------------------------------------------- + +We maintain a `list of easy tickets +`__ to work on, see +the `Easy hacks page `__ for more +details. + +I am skilled in one specific technology, can I find tickets requiring that skill? +--------------------------------------------------------------------------------- + +Unfortunately, not at the moment. But you can look at the `Internship list +`__ to look for something matching +this skill, and this may allow you to find topics to search for in the `bug tracking +system `__. + +Either way, feel free to `contact our developers +`__, we would love to work with +you. + +Where should I ask for technical help? +-------------------------------------- + +You can choose one of the following: + +* `development channels `__ +* `contact form for any enquiries `__ + +.. _faq_run_swh: + +Running an SWH instance locally +=============================== + +I have SWH stack running in my local. How do I get some initial data to play around? +------------------------------------------------------------------------------------ + +You can setup a job on your local machine, for this you can `schedule a listing task +`__ +for example. Doing so on small forge, will allow you to load some repositories. + +Or you can also trigger directly `loading from the cli +`__. + +I have a SWH stack running in local, How do I setup a lister/loader job? +------------------------------------------------------------------------ + +See the `"Managing tasks" chapter +`__ +in the Docker environment documentation. + +How can I create a user in my local instance? +--------------------------------------------- + +We cannot right now. Stay either anonymous or use the user "test" (password "test") or +the user ambassador (password "ambassador"). + +Should I run/test the web app in any particular browser? +-------------------------------------------------------- + +We expect the web app to work on all major browsers. It uses mostly straightforward +HTML/CSS and a little Javascript for search and source code highlighting, so testing in +a single browser is usually enough. + +.. _faq_dataset: + +Getting sample datasets +======================= + +Is there a way to connect to SWH archived (production) database from my local machine? +-------------------------------------------------------------------------------------- + +We provide the archive as a dataset on public clouds, see the `swh-dataset +documentation `__. We can +also provide read access to one of the main databases on request `contact +us `__. + +.. _faq_error_bugs: + +Errors and bugs +=============== + +I found a bug/improvement in the system, where should I report it? +------------------------------------------------------------------ + +Please report it on our `bug tracking system `__. +First create an account, then create a bug report using the "Create task" button. You +should get some feedback within a week (at least someone triaging your issue). If not, +`get in touch with us `__ to +make sure we did not miss it. + +.. _faq_legal: + +Legal matters +============= + +Do I need to sign a form to contribute code? +-------------------------------------------- + +Yes, on your first diff, you will have to sign such document. +As long as it's not signed, your diff content won't be visible. + +Will my name be added to a CONTRIBUTORS file? +--------------------------------------------- + +You will be asked during review to add yourself. + +.. _faq_code_review: + +Code Review +=========== + +I found a straightforward typo fix, should my fix go through the entire code review process? +-------------------------------------------------------------------------------------------- + +You are welcome to drop us a message at one of the `development +channels `__, we will pick it up +and fix it so you don't have to follow the whole `code review +process `__. + +What tests I should run before committing the code? +--------------------------------------------------- + +Mostly run `tox` (or `pytest`) to run the unit tests suite. When you will propose a +patch in our forge, the continuous integration factory will trigger a build (using `tox` +as well). + +I am getting errors while trying to commit. What is going wrong? +---------------------------------------------------------------- + +Ensure you followed the proper guide to `setup your +environment `__ +and try again. If the error persists, you are welcome to drop us a message at one of the +`development channels `__ + +Is there a format/guideline for writing commit messages? +-------------------------------------------------------- + +See the `Git style guide `__ + +Is there some recommended git branching strategy? +------------------------------------------------- + +It's left at the developer's discretion. Mostly people hack on their feature, then +propose a diff from a git branch or directly from the master branch. There is no +imperative. The only imperative is that for a feature to be packaged and deployed, it +needs to land first in the master branch. + +how should I document the code I contribute to SWH? +--------------------------------------------------- + +Any new feature should include documentation in the form of comments and/or docstrings. +Ideally, they should also be documented in plain English in the repository's `docs/` +folder if relevant to a single package, or in the main `swh-docs` repository if it is a +transversal feature. + +.. _faq_api: + +Software Heritage API +===================== + +How do I generate API usage credentials? +---------------------------------------- + +See the `Authentication guide `__. + +Is there a page where I can see all the API endpoints? +------------------------------------------------------ + +See the `API endpoint listing page `__. + +What are the usage limits for SWH APIs? +--------------------------------------- + +Maximum number of permitted requests per hour: + +* 120 for anonymous users +* 1200 for authenticated users + +It's described in the `rate limit documentation page +`__. + +.. It's temporarily here but it should be moved into its own sphinx instance at some + point in the future. + +.. _faq_sysadm: + +System Administration +===================== + +How does SWH release? +--------------------- + +Release is mostly done: +- first in docker (somewhat as part of the development process) +- secondly packaged and deployed on staging (mostly) +- thirdly the same package is deployed on production + +Is there a release cycle? +------------------------- + +When a functionality is ready (tests ok, landed in master, docker run ok), the module is +tagged. The tag is pushed. This triggers a packaging build process. When the package is +ready, depending on the module [1], sysadms deploy the package with the help of puppet. + +[1] swh-web module is mostly automatic. Other modules are not yet automatic as some +internal state migration (dbs) often enters the release cycle and due to the data +volume, that may need human intervention. - faq diff --git a/docs/index.rst b/docs/index.rst index 4e07a20..73db53f 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,240 +1,235 @@ Software Heritage - Development Documentation ============================================= Getting started --------------- * :ref:`getting-started` → deploy a local copy of the Software Heritage software stack in less than 5 minutes, or * :ref:`developer-setup` → get a working development setup that allows to hack on the Software Heritage software stack Contributing ------------ * :ref:`patch-submission` → learn how to submit your patches to the Software Heritage codebase * :ref:`code-review` → rules and guidelines to review code in Software Heritage * :ref:`python-style-guide` → how to format the Python code you write Architecture ------------ * :ref:`architecture-overview` → get a glimpse of the Software Heritage software architecture * :ref:`mirror` → learn what a Software Heritage mirror is and how to set up one * :ref:`Keycloak ` → learn how to use Keycloak, the authentication system used by |swh|'s web interface and public APIs Data Model and Specifications ----------------------------- * :ref:`persistent-identifiers` Specifications of the SoftWare Heritage persistent IDentifiers (SWHID). * :ref:`data-model` Documentation of the main |swh| archive data model. * :ref:`journal-specs` Documentation of the Kafka journal of the |swh| archive. Tutorials --------- * :ref:`testing-guide` * :doc:`/tutorials/issue-debugging-monitoring` * :ref:`Listing the content of your favorite forge ` and :ref:`running a lister in Docker ` Frequently Asked Questions -------------------------- -* :ref:`faq_prerequisites` Prerequisites to be a contributor -* :ref:`faq_getting_started` Starter kit to contribute -* :ref:`faq_run_swh` Starter kit to run your local swh instance -* :ref:`faq_dataset` Getting some sample dataset -* :ref:`faq_error_bugs` Stuck somewhere -* :ref:`faq_legal` Legal and contributions -* :ref:`faq_code_review` Related to Code review -* :ref:`faq_api` General questions about the SWH API -* :ref:`faq_sysadm` System Administration related questions +.. toctree:: + :maxdepth: 2 + + faq/index Roadmap ------- * :ref:`roadmap-2021` Engineering ----------- * :ref:`infrastructure` Components ---------- Here is brief overview of the most relevant software components in the Software Heritage stack, in alphabetical order. For a better introduction to the architecture, see the :ref:`architecture-overview`, which presents each of them in a didactical order. Each component name is linked to the development documentation of the corresponding Python module. :ref:`swh.auth ` low-level library used by modules needing keycloak authentication :ref:`swh.core ` low-level utilities and helpers used by almost all other modules in the stack :ref:`swh.counters ` service providing efficient estimates of the number of objects in the SWH archive, using Redis's Hyperloglog :ref:`swh.dataset ` public datasets and periodic data dumps of the archive released by Software Heritage :ref:`swh.deposit ` push-based deposit of software artifacts to the archive swh.docs developer documentation (used to generate this doc you are reading) :ref:`swh.fuse ` Virtual file system to browse the Software Heritage archive, based on `FUSE `_ :ref:`swh.graph ` Fast, compressed, in-memory representation of the archive, with tooling to generate and query it. :ref:`swh.indexer ` tools and workers used to crawl the content of the archive and extract derived information from any artifact stored in it :ref:`swh.journal ` persistent logger of changes to the archive, with publish-subscribe support :ref:`swh.lister ` collection of listers for all sorts of source code hosting and distribution places (forges, distributions, package managers, etc.) :ref:`swh.loader-core ` low-level loading utilities and helpers used by all other loaders :ref:`swh.loader-git ` loader for `Git `_ repositories :ref:`swh.loader-mercurial ` loader for `Mercurial `_ repositories :ref:`swh.loader-svn ` loader for `Subversion `_ repositories :ref:`swh.model ` implementation of the :ref:`data-model` to archive source code artifacts :ref:`swh.objstorage ` content-addressable object storage :ref:`swh.objstorage.replayer ` Object storage replication tool :ref:`swh.scanner ` source code scanner to analyze code bases and compare them with source code artifacts archived by Software Heritage :ref:`swh.scheduler ` task manager for asynchronous/delayed tasks, used for recurrent (e.g., listing a forge, loading new stuff from a Git repository) and one-off activities (e.g., loading a specific version of a source package) :ref:`swh.search ` search engine for the archive :ref:`swh.storage ` abstraction layer over the archive, allowing to access all stored source code artifacts as well as their metadata :ref:`swh.vault ` implementation of the vault service, allowing to retrieve parts of the archive as self-contained bundles (e.g., individual releases, entire repository snapshots, etc.) :ref:`swh.web ` Web application(s) to browse the archive, for both interactive (HTML UI) and mechanized (REST API) use :ref:`swh.web.client ` Python client for :ref:`swh.web ` Dependencies ------------ The dependency relationships among the various modules are depicted below. .. _py-deps-swh: .. figure:: images/py-deps-swh.svg :width: 1024px :align: center Dependencies among top-level Python modules (click to zoom). Archive ------- * :ref:`Archive ChangeLog `: notable changes to the archive over time Indices and tables ================== * :ref:`genindex` * :ref:`modindex` * `URLs index `_ * :ref:`search` * :ref:`glossary` .. ensure sphinx does not complain about index files not being included .. toctree:: :maxdepth: 2 :caption: Contents: :titlesonly: :hidden: getting-started/index architecture/index contributing/index tutorials/index faq/index roadmap/roadmap-2021.rst infrastructure/index swh.auth swh.core swh.counters swh.dataset swh.deposit swh.fuse swh.graph swh.indexer swh.journal swh.lister swh.loader swh.model swh.objstorage swh.objstorage.replayer swh.scanner swh.scheduler swh.search swh.storage swh.vault swh.web swh.web.client archive-changelog journal Python modules autodocumentation