diff options
Diffstat (limited to 'certbot/docs/contributing.rst')
| -rw-r--r-- | certbot/docs/contributing.rst | 613 |
1 files changed, 613 insertions, 0 deletions
diff --git a/certbot/docs/contributing.rst b/certbot/docs/contributing.rst new file mode 100644 index 000000000..25d832761 --- /dev/null +++ b/certbot/docs/contributing.rst @@ -0,0 +1,613 @@ +=============== +Developer Guide +=============== + +.. contents:: Table of Contents + :local: + + +.. _getting_started: + +Getting Started +=============== + +Certbot has the same :ref:`system requirements <system_requirements>` when set +up for development. While the section below will help you install Certbot and +its dependencies, Certbot needs to be run on a UNIX-like OS so if you're using +Windows, you'll need to set up a (virtual) machine running an OS such as Linux +and continue with these instructions on that UNIX-like OS. + +.. _local copy: + +Running a local copy of the client +---------------------------------- + +Running the client in developer mode from your local tree is a little different +than running Certbot as a user. To get set up, clone our git repository by +running: + +.. code-block:: shell + + git clone https://github.com/certbot/certbot + +If you're on macOS, we recommend you skip the rest of this section and instead +run Certbot in Docker. You can find instructions for how to do this :ref:`here +<docker-dev>`. If you're running on Linux, you can run the following commands to +install dependencies and set up a virtual environment where you can run +Certbot. + +Install the OS system dependencies required to run Certbot. + +.. code-block:: shell + + # For APT-based distributions (e.g. Debian, Ubuntu ...) + sudo apt update + sudo apt install python3-dev python3-venv gcc libaugeas0 libssl-dev \ + libffi-dev ca-certificates openssl + # For RPM-based distributions (e.g. Fedora, CentOS ...) + # NB1: old distributions will use yum instead of dnf + # NB2: RHEL-based distributions use python3X-devel instead of python3-devel (e.g. python36-devel) + sudo dnf install python3-devel gcc augeas-libs openssl-devel libffi-devel \ + redhat-rpm-config ca-certificates openssl + +Set up the Python virtual environment that will host your Certbot local instance. + +.. code-block:: shell + + cd certbot + python tools/venv3.py + +.. note:: You may need to repeat this when + Certbot's dependencies change or when a new plugin is introduced. + +You can now run the copy of Certbot from git either by executing +``venv3/bin/certbot``, or by activating the virtual environment. You can do the +latter by running: + +.. code-block:: shell + + source venv3/bin/activate + +After running this command, ``certbot`` and development tools like ``ipdb``, +``ipython``, ``pytest``, and ``tox`` are available in the shell where you ran +the command. These tools are installed in the virtual environment and are kept +separate from your global Python installation. This works by setting +environment variables so the right executables are found and Python can pull in +the versions of various packages needed by Certbot. More information can be +found in the `virtualenv docs`_. + +.. _`virtualenv docs`: https://virtualenv.pypa.io + +Find issues to work on +---------------------- + +You can find the open issues in the `github issue tracker`_. Comparatively +easy ones are marked `good first issue`_. If you're starting work on +something, post a comment to let others know and seek feedback on your plan +where appropriate. + +Once you've got a working branch, you can open a pull request. All changes in +your pull request must have thorough unit test coverage, pass our +tests, and be compliant with the :ref:`coding style <coding-style>`. + +.. _github issue tracker: https://github.com/certbot/certbot/issues +.. _good first issue: https://github.com/certbot/certbot/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22 + +.. _testing: + +Testing +------- + +You can test your code in several ways: + +- running the `automated unit`_ tests, +- running the `automated integration`_ tests +- running an *ad hoc* `manual integration`_ test + +.. _automated unit: + +Running automated unit tests +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +When you are working in a file ``foo.py``, there should also be a file ``foo_test.py`` +either in the same directory as ``foo.py`` or in the ``tests`` subdirectory +(if there isn't, make one). While you are working on your code and tests, run +``python foo_test.py`` to run the relevant tests. + +For debugging, we recommend putting +``import ipdb; ipdb.set_trace()`` statements inside the source code. + +Once you are done with your code changes, and the tests in ``foo_test.py`` pass, +run all of the unittests for Certbot with ``tox -e py27`` (this uses Python +2.7). + +Once all the unittests pass, check for sufficient test coverage using ``tox -e +py27-cover``, and then check for code style with ``tox -e lint`` (all files) or +``pylint --rcfile=.pylintrc path/to/file.py`` (single file at a time). + +Once all of the above is successful, you may run the full test suite using +``tox --skip-missing-interpreters``. We recommend running the commands above +first, because running all tests like this is very slow, and the large amount +of output can make it hard to find specific failures when they happen. + +.. warning:: The full test suite may attempt to modify your system's Apache + config if your user has sudo permissions, so it should not be run on a + production Apache server. + +.. _automated integration: + +Running automated integration tests +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Generally it is sufficient to open a pull request and let Github and Travis run +integration tests for you. However, you may want to run them locally before submitting +your pull request. You need Docker and docker-compose installed and working. + +The tox environment `integration` will setup `Pebble`_, the Let's Encrypt ACME CA server +for integration testing, then launch the Certbot integration tests. + +With a user allowed to access your local Docker daemon, run: + +.. code-block:: shell + + tox -e integration + +Tests will be run using pytest. A test report and a code coverage report will be +displayed at the end of the integration tests execution. + +.. _Pebble: https://github.com/letsencrypt/pebble + +.. _manual integration: + +Running manual integration tests +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +You can also manually execute Certbot against a local instance of the `Pebble`_ ACME server. +This is useful to verify that the modifications done to the code makes Certbot behave as expected. + +To do so you need: + +- Docker installed, and a user with access to the Docker client, +- an available `local copy`_ of Certbot. + +The virtual environment set up with `python tools/venv.py` contains two commands +that can be used once the virtual environment is activated: + +.. code-block:: shell + + run_acme_server + +- Starts a local instance of Pebble and runs in the foreground printing its logs. +- Press CTRL+C to stop this instance. +- This instance is configured to validate challenges against certbot executed locally. + +.. code-block:: shell + + certbot_test [ARGS...] + +- Execute certbot with the provided arguments and other arguments useful for testing purposes, + such as: verbose output, full tracebacks in case Certbot crashes, *etc.* +- Execution is preconfigured to interact with the Pebble CA started with ``run_acme_server``. +- Any arguments can be passed as they would be to Certbot (eg. ``certbot_test certonly -d test.example.com``). + +Here is a typical workflow to verify that Certbot successfully issued a certificate +using an HTTP-01 challenge on a machine with Python 3: + +.. code-block:: shell + + python tools/venv3.py + source venv3/bin/activate + run_acme_server & + certbot_test certonly --standalone -d test.example.com + # To stop Pebble, launch `fg` to get back the background job, then press CTRL+C + +Running tests in CI +~~~~~~~~~~~~~~~~~~~ + +Certbot uses both Azure Pipelines and Travis to run continuous integration +tests. If you are using our Azure and Travis setup, a branch whose name starts +with `test-` will run all Azure and Travis tests on that branch. If the branch +name starts with `azure-test-`, it will run all of our Azure tests and none of +our Travis tests. If the branch stats with `travis-test-`, only our Travis +tests will be run. + +Code components and layout +========================== + +The following components of the Certbot repository are distributed to users: + +acme + contains all protocol specific code +certbot + main client code +certbot-apache and certbot-nginx + client code to configure specific web servers +certbot-dns-* + client code to configure DNS providers +certbot-auto and letsencrypt-auto + shell scripts to install Certbot and its dependencies on UNIX systems +windows installer + Installs Certbot on Windows and is built using the files in windows-installer/ + +Plugin-architecture +------------------- + +Certbot has a plugin architecture to facilitate support for +different webservers, other TLS servers, and operating systems. +The interfaces available for plugins to implement are defined in +`interfaces.py`_ and `plugins/common.py`_. + +The main two plugin interfaces are `~certbot.interfaces.IAuthenticator`, which +implements various ways of proving domain control to a certificate authority, +and `~certbot.interfaces.IInstaller`, which configures a server to use a +certificate once it is issued. Some plugins, like the built-in Apache and Nginx +plugins, implement both interfaces and perform both tasks. Others, like the +built-in Standalone authenticator, implement just one interface. + +There are also `~certbot.interfaces.IDisplay` plugins, +which can change how prompts are displayed to a user. + +.. _interfaces.py: https://github.com/certbot/certbot/blob/master/certbot/interfaces.py +.. _plugins/common.py: https://github.com/certbot/certbot/blob/master/certbot/plugins/common.py#L34 + + +Authenticators +-------------- + +Authenticators are plugins that prove control of a domain name by solving a +challenge provided by the ACME server. ACME currently defines several types of +challenges: HTTP, TLS-ALPN, and DNS, represented by classes in `acme.challenges`. +An authenticator plugin should implement support for at least one challenge type. + +An Authenticator indicates which challenges it supports by implementing +`get_chall_pref(domain)` to return a sorted list of challenge types in +preference order. + +An Authenticator must also implement `perform(achalls)`, which "performs" a list +of challenges by, for instance, provisioning a file on an HTTP server, or +setting a TXT record in DNS. Once all challenges have succeeded or failed, +Certbot will call the plugin's `cleanup(achalls)` method to remove any files or +DNS records that were needed only during authentication. + +Installer +--------- + +Installers plugins exist to actually setup the certificate in a server, +possibly tweak the security configuration to make it more correct and secure +(Fix some mixed content problems, turn on HSTS, redirect to HTTPS, etc). +Installer plugins tell the main client about their abilities to do the latter +via the :meth:`~.IInstaller.supported_enhancements` call. We currently +have two Installers in the tree, the `~.ApacheConfigurator`. and the +`~.NginxConfigurator`. External projects have made some progress toward +support for IIS, Icecast and Plesk. + +Installers and Authenticators will oftentimes be the same class/object +(because for instance both tasks can be performed by a webserver like nginx) +though this is not always the case (the standalone plugin is an authenticator +that listens on port 80, but it cannot install certs; a postfix plugin would +be an installer but not an authenticator). + +Installers and Authenticators are kept separate because +it should be possible to use the `~.StandaloneAuthenticator` (it sets +up its own Python server to perform challenges) with a program that +cannot solve challenges itself (Such as MTA installers). + + +Installer Development +--------------------- + +There are a few existing classes that may be beneficial while +developing a new `~certbot.interfaces.IInstaller`. +Installers aimed to reconfigure UNIX servers may use Augeas for +configuration parsing and can inherit from `~.AugeasConfigurator` class +to handle much of the interface. Installers that are unable to use +Augeas may still find the `~.Reverter` class helpful in handling +configuration checkpoints and rollback. + + +.. _dev-plugin: + +Writing your own plugin +~~~~~~~~~~~~~~~~~~~~~~~ + +.. note:: The Certbot team is not currently accepting any new DNS plugins + because we want to rethink our approach to the challenge and resolve some + issues like `#6464 <https://github.com/certbot/certbot/issues/6464>`_, + `#6503 <https://github.com/certbot/certbot/issues/6503>`_, and `#6504 + <https://github.com/certbot/certbot/issues/6504>`_ first. + + In the meantime, you're welcome to release it as a third-party plugin. See + `certbot-dns-ispconfig <https://github.com/m42e/certbot-dns-ispconfig>`_ + for one example of that. + +Certbot client supports dynamic discovery of plugins through the +`setuptools entry points`_ using the `certbot.plugins` group. This +way you can, for example, create a custom implementation of +`~certbot.interfaces.IAuthenticator` or the +`~certbot.interfaces.IInstaller` without having to merge it +with the core upstream source code. An example is provided in +``examples/plugins/`` directory. + +While developing, you can install your plugin into a Certbot development +virtualenv like this: + +.. code-block:: shell + + . venv/bin/activate + pip install -e examples/plugins/ + certbot_test plugins + +Your plugin should show up in the output of the last command. If not, +it was not installed properly. + +Once you've finished your plugin and published it, you can have your +users install it system-wide with `pip install`. Note that this will +only work for users who have Certbot installed from OS packages or via +pip. Users who run `certbot-auto` are currently unable to use third-party +plugins. It's technically possible to install third-party plugins into +the virtualenv used by `certbot-auto`, but they will be wiped away when +`certbot-auto` upgrades. + +.. _`setuptools entry points`: + http://setuptools.readthedocs.io/en/latest/pkg_resources.html#entry-points + +.. _coding-style: + +Coding style +============ + +Please: + +1. **Be consistent with the rest of the code**. + +2. Read `PEP 8 - Style Guide for Python Code`_. + +3. Follow the `Google Python Style Guide`_, with the exception that we + use `Sphinx-style`_ documentation:: + + def foo(arg): + """Short description. + + :param int arg: Some number. + + :returns: Argument + :rtype: int + + """ + return arg + +4. Remember to use ``pylint``. + +.. _Google Python Style Guide: + https://google.github.io/styleguide/pyguide.html +.. _Sphinx-style: http://sphinx-doc.org/ +.. _PEP 8 - Style Guide for Python Code: + https://www.python.org/dev/peps/pep-0008 + +Use ``certbot.compat.os`` instead of ``os`` +=========================================== + + +Python's standard library ``os`` module lacks full support for several Windows +security features about file permissions (eg. DACLs). However several files +handled by Certbot (eg. private keys) need strongly restricted access +on both Linux and Windows. + +To help with this, the ``certbot.compat.os`` module wraps the standard +``os`` module, and forbids usage of methods that lack support for these Windows +security features. + +As a developer, when working on Certbot or its plugins, you must use ``certbot.compat.os`` +in every place you would need ``os`` (eg. ``from certbot.compat import os`` instead of +``import os``). Otherwise the tests will fail when your PR is submitted. + +.. _type annotations: + +Mypy type annotations +===================== + +Certbot uses the `mypy`_ static type checker. Python 3 natively supports official type annotations, +which can then be tested for consistency using mypy. Python 2 doesn’t, but type annotations can +be `added in comments`_. Mypy does some type checks even without type annotations; we can find +bugs in Certbot even without a fully annotated codebase. + +Certbot supports both Python 2 and 3, so we’re using Python 2-style annotations. + +Zulip wrote a `great guide`_ to using mypy. It’s useful, but you don’t have to read the whole thing +to start contributing to Certbot. + +To run mypy on Certbot, use ``tox -e mypy`` on a machine that has Python 3 installed. + +Note that instead of just importing ``typing``, due to packaging issues, in Certbot we import from +``acme.magic_typing`` and have to add some comments for pylint like this: + +.. code-block:: python + + from acme.magic_typing import Dict + +Also note that OpenSSL, which we rely on, has type definitions for crypto but not SSL. We use both. +Those imports should look like this: + +.. code-block:: python + + from OpenSSL import crypto + from OpenSSL import SSL # type: ignore # https://github.com/python/typeshed/issues/2052 + +.. _mypy: https://mypy.readthedocs.io +.. _added in comments: https://mypy.readthedocs.io/en/latest/cheat_sheet.html +.. _great guide: https://blog.zulip.org/2016/10/13/static-types-in-python-oh-mypy/ + +Submitting a pull request +========================= + +Steps: + +1. Write your code! When doing this, you should add :ref:`mypy type annotations + <type annotations>` for any functions you add or modify. You can check that + you've done this correctly by running ``tox -e mypy`` on a machine that has + Python 3 installed. +2. Make sure your environment is set up properly and that you're in your + virtualenv. You can do this by following the instructions in the + :ref:`Getting Started <getting_started>` section. +3. Run ``tox -e lint`` to check for pylint errors. Fix any errors. +4. Run ``tox --skip-missing-interpreters`` to run the entire test suite + including coverage. The ``--skip-missing-interpreters`` argument ignores + missing versions of Python needed for running the tests. Fix any errors. +5. Submit the PR. Once your PR is open, please do not force push to the branch + containing your pull request to squash or amend commits. We use `squash + merges <https://github.com/blog/2141-squash-your-commits>`_ on PRs and + rewriting commits makes changes harder to track between reviews. +6. Did your tests pass on Travis? If they didn't, fix any errors. + +.. _ask for help: + +Asking for help +=============== + +If you have any questions while working on a Certbot issue, don't hesitate to +ask for help! You can do this in the Certbot channel in EFF's Mattermost +instance for its open source projects as described below. + +You can get involved with several of EFF's software projects such as Certbot at +the `EFF Open Source Contributor Chat Platform +<https://opensource.eff.org/signup_user_complete/?id=6iqur37ucfrctfswrs14iscobw>`_. +By signing up for the EFF Open Source Contributor Chat Platform, you consent to +share your personal information with the Electronic Frontier Foundation, which +is the operator and data controller for this platform. The channels will be +available both to EFF, and to other users of EFFOSCCP, who may use or disclose +information in these channels outside of EFFOSCCP. EFF will use your +information, according to the `Privacy Policy <https://www.eff.org/policy>`_, +to further the mission of EFF, including hosting and moderating the discussions +on this platform. + +Use of EFFOSCCP is subject to the `EFF Code of Conduct +<https://www.eff.org/pages/eppcode>`_. When investigating an alleged Code of +Conduct violation, EFF may review discussion channels or direct messages. + +Updating certbot-auto and letsencrypt-auto +========================================== + +.. note:: We are currently only accepting changes to certbot-auto that fix + regressions on platforms where certbot-auto is the recommended installation + method at https://certbot.eff.org/instructions. If you are unsure if a change + you want to make qualifies, don't hesitate to `ask for help`_! + +Updating the scripts +-------------------- +Developers should *not* modify the ``certbot-auto`` and ``letsencrypt-auto`` files +in the root directory of the repository. Rather, modify the +``letsencrypt-auto.template`` and associated platform-specific shell scripts in +the ``letsencrypt-auto-source`` and +``letsencrypt-auto-source/pieces/bootstrappers`` directory, respectively. + +Building letsencrypt-auto-source/letsencrypt-auto +------------------------------------------------- +Once changes to any of the aforementioned files have been made, the +``letsencrypt-auto-source/letsencrypt-auto`` script should be updated. In lieu of +manually updating this script, run the build script, which lives at +``letsencrypt-auto-source/build.py``: + +.. code-block:: shell + + python letsencrypt-auto-source/build.py + +Running ``build.py`` will update the ``letsencrypt-auto-source/letsencrypt-auto`` +script. Note that the ``certbot-auto`` and ``letsencrypt-auto`` scripts in the root +directory of the repository will remain **unchanged** after this script is run. +Your changes will be propagated to these files during the next release of +Certbot. + +Opening a PR +------------ +When opening a PR, ensure that the following files are committed: + +1. ``letsencrypt-auto-source/letsencrypt-auto.template`` and + ``letsencrypt-auto-source/pieces/bootstrappers/*`` +2. ``letsencrypt-auto-source/letsencrypt-auto`` (generated by ``build.py``) + +It might also be a good idea to double check that **no** changes were +inadvertently made to the ``certbot-auto`` or ``letsencrypt-auto`` scripts in the +root of the repository. These scripts will be updated by the core developers +during the next release. + + +Updating the documentation +========================== + +Many of the packages in the Certbot repository have documentation in a +``docs/`` directory. This directory is located under the top level directory +for the package. For instance, Certbot's documentation is under +``certbot/docs``. + +To build the documentation of a package, make sure you have followed the +instructions to set up a `local copy`_ of Certbot including activating the +virtual environment. After that, ``cd`` to the docs directory you want to build +and run the command: + +.. code-block:: shell + + make clean html + +This would generate the HTML documentation in ``_build/html`` in your current +``docs/`` directory. + +.. _docker-dev: + +Running the client with Docker +============================== + +You can use Docker Compose to quickly set up an environment for running and +testing Certbot. To install Docker Compose, follow the instructions at +https://docs.docker.com/compose/install/. + +.. note:: Linux users can simply run ``pip install docker-compose`` to get + Docker Compose after installing Docker Engine and activating your shell as + described in the :ref:`Getting Started <getting_started>` section. + +Now you can develop on your host machine, but run Certbot and test your changes +in Docker. When using ``docker-compose`` make sure you are inside your clone of +the Certbot repository. As an example, you can run the following command to +check for linting errors:: + + docker-compose run --rm --service-ports development bash -c 'tox -e lint' + +You can also leave a terminal open running a shell in the Docker container and +modify Certbot code in another window. The Certbot repo on your host machine is +mounted inside of the container so any changes you make immediately take +effect. To do this, run:: + + docker-compose run --rm --service-ports development bash + +Now running the check for linting errors described above is as easy as:: + + tox -e lint + +.. _prerequisites: + +Notes on OS dependencies +======================== + +OS-level dependencies can be installed like so: + +.. code-block:: shell + + ./certbot-auto --debug --os-packages-only + +In general... + +* ``sudo`` is required as a suggested way of running privileged process +* `Python`_ 2.7 or 3.5+ is required +* `Augeas`_ is required for the Python bindings +* ``virtualenv`` is used for managing other Python library dependencies + +.. _Python: https://wiki.python.org/moin/BeginnersGuide/Download +.. _Augeas: http://augeas.net/ +.. _Virtualenv: https://virtualenv.pypa.io + + +FreeBSD +------- + +FreeBSD by default uses ``tcsh``. In order to activate virtualenv (see +above), you will need a compatible shell, e.g. ``pkg install bash && +bash``. |