[Unix] Create a framework for certbot integration tests: PART 1 (#6578)

* First part

* Several optimizations about the docker env setup

* Documentation

* Various corrections and documentation. Add acme and certbot explicitly as dependencies of certbot-ci.

* Correct a variable misinterpreted as a pytest hook

* Correct strict parsing option on pebble

* Refactor acme setup to be executed from pytest hooks.

* Pass TRAVIS env variable to trigger specific xdist logic

* Retrigger build.

* Work in progress

* Config operational

* Propagate to xdist

* Corrections on acme and misc

* Correct subnet for pebble

* Remove gobetween, as tls-sni challenges are not tested anymore.

* Improve pebble setup. Reduce LOC.

* Update acme.py

* Optimize acme ca setup, with less temporary assets

* Silent setup

* Clean code

* Remove unused workspace

* Use default network driver

* Remove bridge

* Update package documentation

* Remove rerun capability for integration tests, not needed.

* Add documentation

* Variable for all ports and subnets used by the stack

* Update certbot-ci/certbot_integration_tests/conftest.py

Co-Authored-By: adferrand <adferrand@users.noreply.github.com>

* Update certbot-ci/certbot_integration_tests/utils/acme.py

Co-Authored-By: adferrand <adferrand@users.noreply.github.com>

* Update certbot-ci/certbot_integration_tests/utils/misc.py

Co-Authored-By: adferrand <adferrand@users.noreply.github.com>

* Update tox.ini

Co-Authored-By: adferrand <adferrand@users.noreply.github.com>

* Update certbot-ci/certbot_integration_tests/utils/misc.py

Co-Authored-By: adferrand <adferrand@users.noreply.github.com>

* Update certbot-ci/certbot_integration_tests/utils/acme.py

Co-Authored-By: adferrand <adferrand@users.noreply.github.com>

* Update certbot-ci/certbot_integration_tests/utils/acme.py

Co-Authored-By: adferrand <adferrand@users.noreply.github.com>

* Update certbot-ci/certbot_integration_tests/conftest.py

Co-Authored-By: adferrand <adferrand@users.noreply.github.com>

* Rename to acme_server

* Add comment

* Refactor in a unique context fixture

* Remove the need of CERTBOT_ACME_XDIST environment variable

* Remove nonstrict/strict options in pebble

* Clean dependencies

* Clean tox

* Change function name

* Add comment about coveragerc specificities

* Change a comment.

* Update setup.py

* Update conftest.py

* Use the production-ready docker-compose.yml file for Pebble

* New style class

* Tune pebble to have a stable test environment

* Pin a dependency

1 files changed, 92 insertions, 0 deletions

diff --git a/certbot-ci/certbot_integration_tests/conftest.py b/certbot-ci/certbot_integration_tests/conftest.py
new file mode 100644
index 000000000..892c16266
--- /dev/null
+++ b/certbot-ci/certbot_integration_tests/conftest.py
@@ -0,0 +1,92 @@
+"""
+General conftest for pytest execution of all integration tests lying
+in the certbot_integration tests package.
+As stated by pytest documentation, conftest module is used to set on
+for a directory a specific configuration using built-in pytest hooks.
+
+See https://docs.pytest.org/en/latest/reference.html#hook-reference
+"""
+import contextlib
+import sys
+import subprocess
+
+from certbot_integration_tests.utils import acme_server as acme_lib
+
+
+def pytest_addoption(parser):
+    """
+    Standard pytest hook to add options to the pytest parser.
+    :param parser: current pytest parser that will be used on the CLI
+    """
+    parser.addoption('--acme-server', default='pebble',
+                     choices=['boulder-v1', 'boulder-v2', 'pebble'],
+                     help='select the ACME server to use (boulder-v1, boulder-v2, '
+                          'pebble), defaulting to pebble')
+
+
+def pytest_configure(config):
+    """
+    Standard pytest hook used to add a configuration logic for each node of a pytest run.
+    :param config: the current pytest configuration
+    """
+    if not hasattr(config, 'slaveinput'):  # If true, this is the primary node
+        with _print_on_err():
+            config.acme_xdist = _setup_primary_node(config)
+
+
+def pytest_configure_node(node):
+    """
+    Standard pytest-xdist hook used to configure a worker node.
+    :param node: current worker node
+    """
+    node.slaveinput['acme_xdist'] = node.config.acme_xdist
+
+
+@contextlib.contextmanager
+def _print_on_err():
+    """
+    During pytest-xdist setup, stdout is used for nodes communication, so print is useless.
+    However, stderr is still available. This context manager transfers stdout to stderr
+    for the duration of the context, allowing to display prints to the user.
+    """
+    old_stdout = sys.stdout
+    sys.stdout = sys.stderr
+    try:
+        yield
+    finally:
+        sys.stdout = old_stdout
+
+
+def _setup_primary_node(config):
+    """
+    Setup the environment for integration tests.
+    Will:
+        - check runtime compatiblity (Docker, docker-compose, Nginx)
+        - create a temporary workspace and the persistent GIT repositories space
+        - configure and start paralleled ACME CA servers using Docker
+        - transfer ACME CA servers configurations to pytest nodes using env variables
+    :param config: Configuration of the pytest primary node
+    """
+    # Check for runtime compatibility: some tools are required to be available in PATH
+    try:
+        subprocess.check_output(['docker', '-v'], stderr=subprocess.STDOUT)
+    except (subprocess.CalledProcessError, OSError):
+        raise ValueError('Error: docker is required in PATH to launch the integration tests, '
+                         'but is not installed or not available for current user.')
+
+    try:
+        subprocess.check_output(['docker-compose', '-v'], stderr=subprocess.STDOUT)
+    except (subprocess.CalledProcessError, OSError):
+        raise ValueError('Error: docker-compose is required in PATH to launch the integration tests, '
+                         'but is not installed or not available for current user.')
+
+    # Parameter numprocesses is added to option by pytest-xdist
+    workers = ['primary'] if not config.option.numprocesses\
+        else ['gw{0}'.format(i) for i in range(config.option.numprocesses)]
+
+    # By calling setup_acme_server we ensure that all necessary acme server instances will be
+    # fully started. This runtime is reflected by the acme_xdist returned.
+    acme_xdist = acme_lib.setup_acme_server(config.option.acme_server, workers)
+    print('ACME xdist config:\n{0}'.format(acme_xdist))
+
+    return acme_xdist