Make a test farm tests package (#8821)

Fixes https://github.com/certbot/certbot/issues/8781.

This PR makes our test farm tests into a normal package so it and its dependencies can be tracked and installed like our other packages.

Other noteworthy changes in this PR:

* Rather than continuing to place logs in your CWD, they're placed in a temporary directory that is printed to the terminal.
*  `tests/letstest/auto_targets.yaml` was deleted rather than renamed because the file is no longer used.

* make a letstest package

* remove deleted deps

* fix letstest install

* add __init__.py

* call main

* Explicitly mention activating venv

* rerename file

* fix version.py path

* clarify "this"

* Use >= instead of caret requirement

11 files changed, 1130 insertions, 0 deletions

diff --git a/letstest/README.md b/letstest/README.md
new file mode 100644
index 000000000..c569d1e8f
--- /dev/null
+++ b/letstest/README.md
@@ -0,0 +1,65 @@
+# letstest
+Simple AWS testfarm scripts for certbot client testing
+
+- Launches EC2 instances with a given list of AMIs for different distros
+- Copies certbot repo and puts it on the instances
+- Runs certbot tests (bash scripts) on all of these
+- Logs execution and success/fail for debugging
+
+## Notes
+  - Some AWS images, e.g. official CentOS and FreeBSD images
+    require acceptance of user terms on the AWS marketplace
+    website.  This can't be automated.
+  - AWS EC2 has a default limit of 20 t2/t1 instances, if more
+    are needed, they need to be requested via online webform.
+
+## Installation and configuration
+
+This package is installed in the Certbot development environment that is
+created by following the instructions at
+https://certbot.eff.org/docs/contributing.html#running-a-local-copy-of-the-client.
+
+After activating that virtual environment, you can then configure AWS
+credentials and create a key by running:
+```
+>aws configure --profile <profile name>
+[interactive: enter secrets for IAM role]
+>aws ec2 create-key-pair --profile <profile name> --key-name <key name> --query 'KeyMaterial' --output text > whatever/path/you/want.pem
+```
+Note: whatever you pick for `<key name>` will be shown to other users with AWS access.
+
+When prompted for a default region name, enter: `us-east-1`.
+
+## Usage
+To run tests, activate the virtual environment you created above and from this directory run:
+```
+>letstest targets/targets.yaml /path/to/your/key.pem <profile name> scripts/<test to run>
+```
+
+You can only run up to two tests at once. The following error is often indicative of there being too many AWS instances running on our account:
+```
+NameError: name 'instances' is not defined
+```
+
+If you see this, you can run the following command to shut down all running instances:
+```
+aws ec2 terminate-instances --profile <profile name> --instance-ids $(aws ec2 describe-instances --profile <profile name> | grep <key name> | cut -f8)
+```
+
+It will take a minute for these instances to shut down and become available again. Running this will invalidate any in progress tests.
+
+A temporary directory whose name is output by the tests is also created with a log file from each instance of the test and a file named "results" containing the output above.
+The tests take quite a while to run.
+
+## Scripts
+Example scripts are in the 'scripts' directory, these are just bash scripts that have a few parameters passed
+to them at runtime via environment variables.  test_apache2.sh is a useful reference.
+
+test_apache2 runs the dev venv and does local tests.
+
+See:
+- https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
+- https://docs.aws.amazon.com/cli/latest/userguide/cli-ec2-keypairs.html
+
+Main repos:
+- https://github.com/letsencrypt/letsencrypt
diff --git a/letstest/letstest/__init__.py b/letstest/letstest/__init__.py
new file mode 100644
index 000000000..e69de29bb
--- /dev/null
+++ b/letstest/letstest/__init__.py
diff --git a/letstest/letstest/multitester.py b/letstest/letstest/multitester.py
new file mode 100644
index 000000000..a56bf0f37
--- /dev/null
+++ b/letstest/letstest/multitester.py
@@ -0,0 +1,526 @@
+"""
+Certbot Integration Test Tool
+
+- Launches EC2 instances with a given list of AMIs for different distros
+- Copies certbot repo and puts it on the instances
+- Runs certbot tests (bash scripts) on all of these
+- Logs execution and success/fail for debugging
+
+Notes:
+  - Some AWS images, e.g. official CentOS and FreeBSD images
+    require acceptance of user terms on the AWS marketplace
+    website.  This can't be automated.
+  - AWS EC2 has a default limit of 20 t2/t1 instances, if more
+    are needed, they need to be requested via online webform.
+
+Usage:
+  - Requires AWS IAM secrets to be set up with aws cli
+  - Requires an AWS associated keyfile <keyname>.pem
+
+>aws configure --profile HappyHacker
+[interactive: enter secrets for IAM role]
+>aws ec2 create-key-pair --profile HappyHacker --key-name MyKeyPair \
+ --query 'KeyMaterial' --output text > MyKeyPair.pem
+then:
+>letstest targets/targets.yaml MyKeyPair.pem HappyHacker scripts/test_sdists.sh
+see:
+  https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
+  https://docs.aws.amazon.com/cli/latest/userguide/cli-ec2-keypairs.html
+"""
+import argparse
+import multiprocessing as mp
+from multiprocessing import Manager
+import os
+import socket
+import sys
+import tempfile
+import time
+import traceback
+import urllib.error as urllib_error
+import urllib.request as urllib_request
+
+import boto3
+from botocore.exceptions import ClientError
+import yaml
+
+from fabric import Config
+from fabric import Connection
+
+# Command line parser
+#-------------------------------------------------------------------------------
+parser = argparse.ArgumentParser(description='Builds EC2 cluster for testing.')
+parser.add_argument('config_file',
+                    help='yaml configuration file for AWS server cluster')
+parser.add_argument('key_file',
+                    help='key file (<keyname>.pem) for AWS')
+parser.add_argument('aws_profile',
+                    help='profile for AWS (i.e. as in ~/.aws/certificates)')
+parser.add_argument('test_script',
+                    default='test_sdists.sh',
+                    help='path of bash script in to deploy and run')
+parser.add_argument('--repo',
+                    default='https://github.com/letsencrypt/letsencrypt.git',
+                    help='certbot git repo to use')
+parser.add_argument('--branch',
+                    default='~',
+                    help='certbot git branch to trial')
+parser.add_argument('--pull_request',
+                    default='~',
+                    help='letsencrypt/letsencrypt pull request to trial')
+parser.add_argument('--merge_master',
+                    action='store_true',
+                    help="if set merges PR into master branch of letsencrypt/letsencrypt")
+parser.add_argument('--saveinstances',
+                    action='store_true',
+                    help="don't kill EC2 instances after run, useful for debugging")
+parser.add_argument('--alt_pip',
+                    default='',
+                    help="server from which to pull candidate release packages")
+cl_args = parser.parse_args()
+
+# Credential Variables
+#-------------------------------------------------------------------------------
+# assumes naming: <key_filename> = <keyname>.pem
+KEYFILE = cl_args.key_file
+KEYNAME = os.path.split(cl_args.key_file)[1].split('.pem')[0]
+PROFILE = None if cl_args.aws_profile == 'SET_BY_ENV' else cl_args.aws_profile
+
+# Globals
+#-------------------------------------------------------------------------------
+SECURITY_GROUP_NAME = 'certbot-security-group'
+SENTINEL = None #queue kill signal
+SUBNET_NAME = 'certbot-subnet'
+
+class Status:
+    """Possible statuses of client tests."""
+    PASS = 'pass'
+    FAIL = 'fail'
+
+# Boto3/AWS automation functions
+#-------------------------------------------------------------------------------
+def should_use_subnet(subnet):
+    """Should we use the given subnet for these tests?
+
+    We should if it is the default subnet for the availability zone or the
+    subnet is named "certbot-subnet".
+
+    """
+    if not subnet.map_public_ip_on_launch:
+        return False
+    if subnet.default_for_az:
+        return True
+    for tag in subnet.tags:
+        if tag['Key'] == 'Name' and tag['Value'] == SUBNET_NAME:
+            return True
+    return False
+
+def make_security_group(vpc):
+    """Creates a security group in the given VPC."""
+    # will fail if security group of GroupName already exists
+    # cannot have duplicate SGs of the same name
+    mysg = vpc.create_security_group(GroupName=SECURITY_GROUP_NAME,
+                                     Description='security group for automated testing')
+    mysg.authorize_ingress(IpProtocol="tcp", CidrIp="0.0.0.0/0", FromPort=22, ToPort=22)
+    # for mosh
+    mysg.authorize_ingress(IpProtocol="udp", CidrIp="0.0.0.0/0", FromPort=60000, ToPort=61000)
+    return mysg
+
+def make_instance(ec2_client,
+                  instance_name,
+                  ami_id,
+                  keyname,
+                  security_group_id,
+                  subnet_id,
+                  self_destruct,
+                  machine_type='t2.micro'):
+    """Creates an instance using the given parameters.
+
+    If self_destruct is True, the instance will be configured to shutdown after
+    1 hour and to terminate itself on shutdown.
+
+    """
+    block_device_mappings = _get_block_device_mappings(ec2_client, ami_id)
+    tags = [{'Key': 'Name', 'Value': instance_name}]
+    tag_spec = [{'ResourceType': 'instance', 'Tags': tags}]
+    kwargs = {
+        'BlockDeviceMappings': block_device_mappings,
+        'ImageId': ami_id,
+        'SecurityGroupIds': [security_group_id],
+        'SubnetId': subnet_id,
+        'KeyName': keyname,
+        'MinCount': 1,
+        'MaxCount': 1,
+        'InstanceType': machine_type,
+        'TagSpecifications': tag_spec
+    }
+    if self_destruct:
+            kwargs['InstanceInitiatedShutdownBehavior'] = 'terminate'
+            kwargs['UserData'] = '#!/bin/bash\nshutdown -P +60\n'
+    return ec2_client.create_instances(**kwargs)[0]
+
+def _get_block_device_mappings(ec2_client, ami_id):
+    """Returns the list of block device mappings to ensure cleanup.
+
+    This list sets connected EBS volumes to be deleted when the EC2
+    instance is terminated.
+
+    """
+    # Not all devices use EBS, but the default value for DeleteOnTermination
+    # when the device does use EBS is true. See:
+    # * https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-blockdev-mapping.html
+    # * https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-blockdev-template.html
+    return [{'DeviceName': mapping['DeviceName'],
+             'Ebs': {'DeleteOnTermination': True}}
+            for mapping in ec2_client.Image(ami_id).block_device_mappings
+            if not mapping.get('Ebs', {}).get('DeleteOnTermination', True)]
+
+
+# Helper Routines
+#-------------------------------------------------------------------------------
+def block_until_ssh_open(ipstring, wait_time=10, timeout=120):
+    "Blocks until server at ipstring has an open port 22"
+    reached = False
+    t_elapsed = 0
+    while not reached and t_elapsed < timeout:
+        try:
+            sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+            sock.connect((ipstring, 22))
+            reached = True
+        except socket.error as err:
+            time.sleep(wait_time)
+            t_elapsed += wait_time
+    sock.close()
+
+def block_until_instance_ready(booting_instance, extra_wait_time=20):
+    "Blocks booting_instance until AWS EC2 instance is ready to accept SSH connections"
+    booting_instance.wait_until_running()
+    # The instance needs to be reloaded to update its local attributes. See
+    # https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Instance.reload.
+    booting_instance.reload()
+    # After waiting for the instance to be running and reloading the instance
+    # state, we should have an IP address.
+    assert booting_instance.public_ip_address is not None
+    block_until_ssh_open(booting_instance.public_ip_address)
+    time.sleep(extra_wait_time)
+    return booting_instance
+
+
+# Fabric Routines
+#-------------------------------------------------------------------------------
+def local_git_clone(local_cxn, repo_url, log_dir):
+    """clones master of repo_url"""
+    local_cxn.local('cd %s && if [ -d letsencrypt ]; then rm -rf letsencrypt; fi' % log_dir)
+    local_cxn.local('cd %s && git clone %s letsencrypt'% (log_dir, repo_url))
+    local_cxn.local('cd %s && tar czf le.tar.gz letsencrypt'% log_dir)
+
+def local_git_branch(local_cxn, repo_url, branch_name, log_dir):
+    """clones branch <branch_name> of repo_url"""
+    local_cxn.local('cd %s && if [ -d letsencrypt ]; then rm -rf letsencrypt; fi' % log_dir)
+    local_cxn.local('cd %s && git clone %s letsencrypt --branch %s --single-branch'%
+        (log_dir, repo_url, branch_name))
+    local_cxn.local('cd %s && tar czf le.tar.gz letsencrypt' % log_dir)
+
+def local_git_PR(local_cxn, repo_url, PRnumstr, log_dir, merge_master=True):
+    """clones specified pull request from repo_url and optionally merges into master"""
+    local_cxn.local('cd %s && if [ -d letsencrypt ]; then rm -rf letsencrypt; fi' % log_dir)
+    local_cxn.local('cd %s && git clone %s letsencrypt' % (log_dir, repo_url))
+    local_cxn.local('cd %s && cd letsencrypt && '
+        'git fetch origin pull/%s/head:lePRtest' % (log_dir, PRnumstr))
+    local_cxn.local('cd %s && cd letsencrypt && git checkout lePRtest' % log_dir)
+    if merge_master:
+        local_cxn.local('cd %s && cd letsencrypt && git remote update origin' % log_dir)
+        local_cxn.local('cd %s && cd letsencrypt && '
+            'git merge origin/master -m "testmerge"' % log_dir)
+    local_cxn.local('cd %s && tar czf le.tar.gz letsencrypt' % log_dir)
+
+def local_repo_to_remote(cxn, log_dir):
+    """copies local tarball of repo to remote"""
+    filename = 'le.tar.gz'
+    local_path = os.path.join(log_dir, filename)
+    cxn.put(local=local_path, remote='')
+    cxn.run('tar xzf %s' % filename)
+
+def local_repo_clean(local_cxn, log_dir):
+    """delete tarball"""
+    filename = 'le.tar.gz'
+    local_path = os.path.join(log_dir, filename)
+    local_cxn.local('rm %s' % local_path)
+
+def deploy_script(cxn, scriptpath, *args):
+    """copies to remote and executes local script"""
+    cxn.put(local=scriptpath, remote='', preserve_mode=True)
+    scriptfile = os.path.split(scriptpath)[1]
+    args_str = ' '.join(args)
+    cxn.run('./'+scriptfile+' '+args_str)
+
+def install_and_launch_certbot(cxn, instance, target, log_dir):
+    local_repo_to_remote(cxn, log_dir)
+    # This needs to be like this, I promise. 1) The env argument to run doesn't work.
+    # See https://github.com/fabric/fabric/issues/1744. 2) prefix() sticks an && between
+    # the commands, so it needs to be exports rather than no &&s in between for the script subshell.
+    with cxn.prefix('export PUBLIC_IP=%s && export PRIVATE_IP=%s && '
+                    'export PUBLIC_HOSTNAME=%s && export PIP_EXTRA_INDEX_URL=%s && '
+                    'export OS_TYPE=%s' %
+                    (instance.public_ip_address,
+                    instance.private_ip_address,
+                    instance.public_dns_name,
+                    cl_args.alt_pip,
+                    target['type'])):
+        deploy_script(cxn, cl_args.test_script)
+
+def grab_certbot_log(cxn):
+    "grabs letsencrypt.log via cat into logged stdout"
+    cxn.sudo('/bin/bash -l -i -c \'if [ -f "/var/log/letsencrypt/letsencrypt.log" ]; then ' +
+        'cat "/var/log/letsencrypt/letsencrypt.log"; else echo "[novarlog]"; fi\'')
+    # fallback file if /var/log is unwriteable...? correct?
+    cxn.sudo('/bin/bash -l -i -c \'if [ -f ./certbot.log ]; then ' +
+        'cat ./certbot.log; else echo "[nolocallog]"; fi\'')
+
+
+def create_client_instance(ec2_client, target, security_group_id, subnet_id, self_destruct):
+    """Create a single client instance for running tests."""
+    if 'machine_type' in target:
+        machine_type = target['machine_type']
+    elif target['virt'] == 'hvm':
+        machine_type = 't2.medium'
+    else:
+        # 32 bit systems
+        machine_type = 'c1.medium'
+    name = 'le-%s'%target['name']
+    print(name, end=" ")
+    return make_instance(ec2_client,
+                         name,
+                         target['ami'],
+                         KEYNAME,
+                         machine_type=machine_type,
+                         security_group_id=security_group_id,
+                         subnet_id=subnet_id,
+                         self_destruct=self_destruct)
+
+
+def test_client_process(fab_config, inqueue, outqueue, log_dir):
+    cur_proc = mp.current_process()
+    for inreq in iter(inqueue.get, SENTINEL):
+        ii, instance_id, target = inreq
+
+        # Each client process is given its own session due to the suggestion at
+        # https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html?highlight=multithreading#multithreading-multiprocessing.
+        aws_session = boto3.session.Session(profile_name=PROFILE)
+        ec2_client = aws_session.resource('ec2')
+        instance = ec2_client.Instance(id=instance_id)
+
+        #save all stdout to log file
+        sys.stdout = open(log_dir+'/'+'%d_%s.log'%(ii,target['name']), 'w')
+
+        print("[%s : client %d %s %s]" % (cur_proc.name, ii, target['ami'], target['name']))
+        instance = block_until_instance_ready(instance)
+        print("server %s at %s"%(instance, instance.public_ip_address))
+        host_string = "%s@%s"%(target['user'], instance.public_ip_address)
+        print(host_string)
+
+        with Connection(host_string, config=fab_config) as cxn:
+            try:
+                install_and_launch_certbot(cxn, instance, target, log_dir)
+                outqueue.put((ii, target, Status.PASS))
+                print("%s - %s SUCCESS"%(target['ami'], target['name']))
+            except:
+                outqueue.put((ii, target, Status.FAIL))
+                print("%s - %s FAIL"%(target['ami'], target['name']))
+                traceback.print_exc(file=sys.stdout)
+                pass
+
+            # append server certbot.log to each per-machine output log
+            print("\n\ncertbot.log\n" + "-"*80 + "\n")
+            try:
+                grab_certbot_log(cxn)
+            except:
+                print("log fail\n")
+                traceback.print_exc(file=sys.stdout)
+                pass
+
+
+def cleanup(cl_args, instances, targetlist, log_dir):
+    print('Logs in ', log_dir)
+    # If lengths of instances and targetlist aren't equal, instances failed to
+    # start before running tests so leaving instances running for debugging
+    # isn't very useful. Let's cleanup after ourselves instead.
+    if len(instances) != len(targetlist) or not cl_args.saveinstances:
+        print('Terminating EC2 Instances')
+        for instance in instances:
+            instance.terminate()
+    else:
+        # print login information for the boxes for debugging
+        for ii, target in enumerate(targetlist):
+            print(target['name'],
+                  target['ami'],
+                  "%s@%s"%(target['user'], instances[ii].public_ip_address))
+
+
+def main():
+    # Fabric library controlled through global env parameters
+    fab_config = Config(overrides={
+        "connect_kwargs": {
+            "key_filename": [KEYFILE], # https://github.com/fabric/fabric/issues/2007
+        },
+        "run": {
+            "echo": True,
+            "pty": True,
+        },
+        "timeouts": {
+            "connect": 10,
+        },
+    })
+    # no network connection, so don't worry about closing this one.
+    local_cxn = Connection('localhost', config=fab_config)
+
+    # Set up local copy of git repo
+    #-------------------------------------------------------------------------------
+    log_dir = tempfile.mkdtemp()  # points to logging / working directory
+    print("Local dir for test repo and logs: %s"%log_dir)
+
+    try:
+        # figure out what git object to test and locally create it in log_dir
+        print("Making local git repo")
+        if cl_args.pull_request != '~':
+            print('Testing PR %s ' % cl_args.pull_request,
+                  "MERGING into master" if cl_args.merge_master else "")
+            local_git_PR(local_cxn, cl_args.repo, cl_args.pull_request, log_dir,
+                         cl_args.merge_master)
+        elif cl_args.branch != '~':
+            print('Testing branch %s of %s' % (cl_args.branch, cl_args.repo))
+            local_git_branch(local_cxn, cl_args.repo, cl_args.branch, log_dir)
+        else:
+            print('Testing current branch of %s' % cl_args.repo, log_dir)
+            local_git_clone(local_cxn, cl_args.repo, log_dir)
+    except BaseException:
+        print("FAIL: trouble with git repo")
+        traceback.print_exc()
+        exit(1)
+
+
+    # Set up EC2 instances
+    #-------------------------------------------------------------------------------
+    configdata = yaml.safe_load(open(cl_args.config_file, 'r'))
+    targetlist = configdata['targets']
+    print('Testing against these images: [%d total]'%len(targetlist))
+    for target in targetlist:
+        print(target['ami'], target['name'])
+
+    print("Connecting to EC2 using\n profile %s\n keyname %s\n keyfile %s"%(PROFILE, KEYNAME, KEYFILE))
+    aws_session = boto3.session.Session(profile_name=PROFILE)
+    ec2_client = aws_session.resource('ec2')
+
+    print("Determining Subnet")
+    for subnet in ec2_client.subnets.all():
+        if should_use_subnet(subnet):
+            subnet_id = subnet.id
+            vpc_id = subnet.vpc.id
+            break
+    else:
+        print("No usable subnet exists!")
+        print("Please create a VPC with a subnet named {0}".format(SUBNET_NAME))
+        print("that maps public IPv4 addresses to instances launched in the subnet.")
+        sys.exit(1)
+
+    print("Making Security Group")
+    vpc = ec2_client.Vpc(vpc_id)
+    sg_exists = False
+    for sg in vpc.security_groups.all():
+        if sg.group_name == SECURITY_GROUP_NAME:
+            security_group_id = sg.id
+            sg_exists = True
+            print("  %s already exists"%SECURITY_GROUP_NAME)
+    if not sg_exists:
+        security_group_id = make_security_group(vpc).id
+        time.sleep(30)
+
+    instances = []
+    try:
+        print("Creating instances: ", end="")
+        # If we want to preserve instances, do not have them self-destruct.
+        self_destruct = not cl_args.saveinstances
+        for target in targetlist:
+            instances.append(
+                create_client_instance(ec2_client, target,
+                                       security_group_id, subnet_id,
+                                       self_destruct)
+            )
+        print()
+
+        # Install and launch client scripts in parallel
+        #-------------------------------------------------------------------------------
+        print("Uploading and running test script in parallel: %s"%cl_args.test_script)
+        print("Output routed to log files in %s"%log_dir)
+        # (Advice: always use Manager.Queue, never regular multiprocessing.Queue
+        # the latter has implementation flaws that deadlock it in some circumstances)
+        manager = Manager()
+        outqueue = manager.Queue()
+        inqueue = manager.Queue()
+
+        # launch as many processes as clients to test
+        num_processes = len(targetlist)
+        jobs = [] #keep a reference to current procs
+
+
+        # initiate process execution
+        client_process_args=(fab_config, inqueue, outqueue, log_dir)
+        for i in range(num_processes):
+            p = mp.Process(target=test_client_process, args=client_process_args)
+            jobs.append(p)
+            p.daemon = True  # kills subprocesses if parent is killed
+            p.start()
+
+        # fill up work queue
+        for ii, target in enumerate(targetlist):
+            inqueue.put((ii, instances[ii].id, target))
+
+        # add SENTINELs to end client processes
+        for i in range(num_processes):
+            inqueue.put(SENTINEL)
+        print('Waiting on client processes', end='')
+        for p in jobs:
+            while p.is_alive():
+                p.join(5 * 60)
+                # Regularly print output to keep Travis happy
+                print('.', end='')
+                sys.stdout.flush()
+        print()
+        # add SENTINEL to output queue
+        outqueue.put(SENTINEL)
+
+        # clean up
+        local_repo_clean(local_cxn, log_dir)
+
+        # print and save summary results
+        results_file = open(log_dir+'/results', 'w')
+        outputs = [outq for outq in iter(outqueue.get, SENTINEL)]
+        outputs.sort(key=lambda x: x[0])
+        failed = False
+        results_msg = ""
+        for outq in outputs:
+            ii, target, status = outq
+            if status == Status.FAIL:
+                failed = True
+                with open(log_dir+'/'+'%d_%s.log'%(ii,target['name']), 'r') as f:
+                    print(target['name'] + " test failed. Test log:")
+                    print(f.read())
+            results_msg = results_msg + '%d %s %s\n'%(ii, target['name'], status)
+            results_file.write('%d %s %s\n'%(ii, target['name'], status))
+        print(results_msg)
+        if len(outputs) != num_processes:
+            failed = True
+            failure_message = 'FAILURE: Some target machines failed to run and were not tested. ' +\
+                'Tests should be rerun.'
+            print(failure_message)
+            results_file.write(failure_message + '\n')
+        results_file.close()
+
+        if failed:
+            sys.exit(1)
+
+    finally:
+        cleanup(cl_args, instances, targetlist, log_dir)
+
+
+if __name__ == '__main__':
+    main()
diff --git a/letstest/scripts/bootstrap_os_packages.sh b/letstest/scripts/bootstrap_os_packages.sh
new file mode 100755
index 000000000..3f4c6e30e
--- /dev/null
+++ b/letstest/scripts/bootstrap_os_packages.sh
@@ -0,0 +1,140 @@
+#!/bin/sh
+#
+# Install OS dependencies for test farm tests.
+
+set -ex  # Work even if somebody does "sh thisscript.sh".
+
+error() {
+    echo "$@"
+}
+
+if command -v command > /dev/null 2>&1 ; then
+  export EXISTS="command -v"
+elif which which > /dev/null 2>&1 ; then
+  export EXISTS="which"
+else
+  error "Cannot find command nor which... please install one!"
+  exit 1
+fi
+
+# Sets LE_PYTHON to Python version string and PYVER to the first two
+# digits of the python version.
+DeterminePythonVersion() {
+  # If no Python is found, PYVER is set to 0.
+  for LE_PYTHON in python3 python2.7 python27 python2 python; do
+    # Break (while keeping the LE_PYTHON value) if found.
+    $EXISTS "$LE_PYTHON" > /dev/null && break
+  done
+  if [ "$?" != "0" ]; then
+    PYVER=0
+    return 0
+  fi
+
+  PYVER=$("$LE_PYTHON" -V 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//')
+}
+
+BootstrapDebCommon() {
+  sudo apt-get update || error apt-get update hit problems but continuing anyway...
+
+  sudo apt-get install -y --no-install-recommends \
+    python3 \
+    python3-dev \
+    python3-venv \
+    gcc \
+    libaugeas0 \
+    libssl-dev \
+    openssl \
+    libffi-dev \
+    ca-certificates \
+    build-essential \
+    curl \
+    make # needed on debian 9 arm64 which doesn't have a python3 pynacl wheel
+
+  # make sure rust isn't installed by the package manager
+  if ! sudo apt-get remove -y rustc; then
+    error "Could not remove existing rust. Aborting bootstrap!"
+    exit 1
+  fi
+
+  # Install rust for cryptography (needed on Debian)
+  curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y
+  . $HOME/.cargo/env
+}
+
+# Sets TOOL to the name of the package manager
+InitializeRPMCommonBase() {
+  if type dnf 2>/dev/null
+  then
+    TOOL=dnf
+  elif type yum 2>/dev/null
+  then
+    TOOL=yum
+
+  else
+    error "Neither yum nor dnf found. Aborting bootstrap!"
+    exit 1
+  fi
+
+}
+
+BootstrapRpmCommonBase() {
+  # Arguments: whitespace-delimited python packages to install
+
+  InitializeRPMCommonBase
+
+  pkgs="
+    gcc
+    augeas-libs
+    openssl
+    openssl-devel
+    libffi-devel
+    redhat-rpm-config
+    ca-certificates
+    cargo
+  "
+
+  # Add the python packages
+  pkgs="$pkgs
+    $1
+  "
+
+  if $TOOL list installed "httpd" >/dev/null 2>&1; then
+    pkgs="$pkgs
+      mod_ssl
+    "
+  fi
+
+  if ! sudo $TOOL install -y $pkgs; then
+    error "Could not install OS dependencies. Aborting bootstrap!"
+    exit 1
+  fi
+}
+
+BootstrapRpmPython3() {
+  InitializeRPMCommonBase
+
+  python_pkgs="python3
+    python3-devel
+  "
+
+  if ! sudo $TOOL list 'python3*-devel' >/dev/null 2>&1; then
+    sudo yum-config-manager --enable rhui-REGION-rhel-server-extras rhui-REGION-rhel-server-optional
+  fi
+
+  BootstrapRpmCommonBase "$python_pkgs"
+}
+
+# Set Bootstrap to the function that installs OS dependencies on this system.
+if [ -f /etc/debian_version ]; then
+  Bootstrap() {
+    BootstrapDebCommon
+  }
+elif [ -f /etc/redhat-release ]; then
+  DeterminePythonVersion
+  Bootstrap() {
+    BootstrapRpmPython3
+  }
+
+fi
+
+Bootstrap
diff --git a/letstest/scripts/test_apache2.sh b/letstest/scripts/test_apache2.sh
new file mode 100755
index 000000000..9d9ca6c12
--- /dev/null
+++ b/letstest/scripts/test_apache2.sh
@@ -0,0 +1,137 @@
+#!/bin/bash -x
+
+# $OS_TYPE $PUBLIC_IP $PRIVATE_IP $PUBLIC_HOSTNAME $BOULDER_URL
+# are dynamically set at execution
+
+if [ "$OS_TYPE" = "ubuntu" ]
+then
+    CONFFILE=/etc/apache2/sites-available/000-default.conf
+    sudo apt-get update
+    sudo apt-get -y --no-upgrade install apache2 curl
+    sudo apt-get -y install realpath # needed for test-apache-conf
+    # For apache 2.4, set up ServerName
+    sudo sed -i '/ServerName/ s/#ServerName/ServerName/' $CONFFILE
+    sudo sed -i '/ServerName/ s/www.example.com/'$PUBLIC_HOSTNAME'/' $CONFFILE
+    if [ $(python3 -V 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//') -lt 36 ]
+    then
+        # Upgrade python version using pyenv because py3.5 is deprecated
+        # Don't upgrade if it's already 3.8 because pyenv doesn't work great on arm, and
+        # our arm representative happens to be ubuntu20, which already has a perfectly
+        # good version of python.
+        sudo apt-get install -y make gcc build-essential libssl-dev zlib1g-dev libbz2-dev \
+          libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \
+          xz-utils tk-dev libffi-dev liblzma-dev python-openssl git # pyenv deps
+        curl https://pyenv.run | bash
+        export PATH="~/.pyenv/bin:$PATH"
+        pyenv init -
+        pyenv virtualenv-init -
+        pyenv install 3.8.5
+        pyenv global 3.8.5
+        # you do, in fact need to run these again, exactly like this.
+        eval "$(pyenv init -)"
+        eval "$(pyenv virtualenv-init -)"
+    fi
+elif [ "$OS_TYPE" = "centos" ]
+then
+    CONFFILE=/etc/httpd/conf/httpd.conf
+    sudo setenforce 0 || true #disable selinux
+    sudo yum -y install httpd
+    sudo yum -y install nghttp2 || echo this is probably ok but see https://bugzilla.redhat.com/show_bug.cgi?id=1358875
+    sudo service httpd start
+    sudo mkdir -p /var/www/$PUBLIC_HOSTNAME/public_html
+    sudo chmod -R oug+rwx /var/www
+    sudo chmod -R oug+rw /etc/httpd
+    sudo echo '<html><head><title>foo</title></head><body>bar</body></html>' > /var/www/$PUBLIC_HOSTNAME/public_html/index.html
+    sudo mkdir /etc/httpd/sites-available #certbot requires this...
+    sudo mkdir /etc/httpd/sites-enabled #certbot requires this...
+    #sudo echo "IncludeOptional sites-enabled/*.conf" >> /etc/httpd/conf/httpd.conf
+    sudo echo """
+<VirtualHost *:80>
+    ServerName $PUBLIC_HOSTNAME
+    DocumentRoot /var/www/$PUBLIC_HOSTNAME/public_html
+    ErrorLog /var/www/$PUBLIC_HOSTNAME/error.log
+    CustomLog /var/www/$PUBLIC_HOSTNAME/requests.log combined
+</VirtualHost>""" >> /etc/httpd/conf.d/$PUBLIC_HOSTNAME.conf
+    #sudo cp /etc/httpd/sites-available/$PUBLIC_HOSTNAME.conf /etc/httpd/sites-enabled/
+fi
+
+# Run certbot-apache2.
+cd letsencrypt
+
+echo "Bootstrapping dependencies..."
+sudo letstest/scripts/bootstrap_os_packages.sh
+if [ $? -ne 0 ] ; then
+    exit 1
+fi
+
+tools/venv.py -e acme[dev] -e certbot[dev,docs] -e certbot-apache -e certbot-ci
+PEBBLE_LOGS="acme_server.log"
+PEBBLE_URL="https://localhost:14000/dir"
+# We configure Pebble to use port 80 for http-01 validation rather than an
+# alternate port because:
+#   1) It allows us to test with Apache configurations that are more realistic
+#   and closer to the default configuration on various OSes.
+#   2) As of writing this, Certbot's Apache plugin requires there to be an
+#   existing virtual host for the port used for http-01 validation.
+venv/bin/run_acme_server --http-01-port 80 > "${PEBBLE_LOGS}" 2>&1 &
+
+DumpPebbleLogs() {
+    if [ -f "${PEBBLE_LOGS}" ] ; then
+        echo "Pebble's logs were:"
+        cat "${PEBBLE_LOGS}"
+    fi
+}
+
+for n in $(seq 1 150) ; do
+    if curl --insecure "${PEBBLE_URL}" 2>/dev/null; then
+        break
+    else
+        echo "waiting for pebble"
+        sleep 1
+    fi
+done
+if ! curl --insecure "${PEBBLE_URL}" 2>/dev/null; then
+  echo "timed out waiting for pebble to start"
+  DumpPebbleLogs
+  exit 1
+fi
+
+sudo "venv/bin/certbot" -v --debug --text --agree-tos --no-verify-ssl \
+                   --renew-by-default --redirect --register-unsafely-without-email \
+                   --domain "${PUBLIC_HOSTNAME}" --server "${PEBBLE_URL}"
+if [ $? -ne 0 ] ; then
+    FAIL=1
+fi
+
+# Check that ssl_module detection is working on various systems
+if [ "$OS_TYPE" = "ubuntu" ] ; then
+    MOD_SSL_LOCATION="/usr/lib/apache2/modules/mod_ssl.so"
+    APACHE_NAME=apache2ctl
+elif [ "$OS_TYPE" = "centos" ]; then
+    MOD_SSL_LOCATION="/etc/httpd/modules/mod_ssl.so"
+    APACHE_NAME=httpd
+fi
+OPENSSL_VERSION=$(strings "$MOD_SSL_LOCATION" | egrep -o -m1 '^OpenSSL ([0-9]\.[^ ]+) ' | tail -c +9)
+APACHE_VERSION=$(sudo $APACHE_NAME -v | egrep -o 'Apache/([0-9]\.[^ ]+)' | tail -c +8)
+"venv/bin/python" letstest/scripts/test_openssl_version.py "$OPENSSL_VERSION" "$APACHE_VERSION"
+if [ $? -ne 0 ] ; then
+    FAIL=1
+fi
+
+
+if [ "$OS_TYPE" = "ubuntu" ] ; then
+    export SERVER="${PEBBLE_URL}"
+    "venv/bin/tox" -e apacheconftest
+else
+    echo Not running hackish apache tests on $OS_TYPE
+fi
+
+if [ $? -ne 0 ] ; then
+    FAIL=1
+fi
+
+# return error if any of the subtests failed
+if [ "$FAIL" = 1 ] ; then
+    DumpPebbleLogs
+    exit 1
+fi
diff --git a/letstest/scripts/test_openssl_version.py b/letstest/scripts/test_openssl_version.py
new file mode 100644
index 000000000..c55441c5d
--- /dev/null
+++ b/letstest/scripts/test_openssl_version.py
@@ -0,0 +1,30 @@
+#!/usr/bin/env python
+# Test script for OpenSSL version checking
+from distutils.version import LooseVersion
+import sys
+
+
+def main(openssl_version, apache_version):
+    if not openssl_version.strip():
+        raise Exception("No OpenSSL version found.")
+    if not apache_version.strip():
+        raise Exception("No Apache version found.")
+    conf_file_location = "/etc/letsencrypt/options-ssl-apache.conf"
+    with open(conf_file_location) as f:
+        contents = f.read()
+    if LooseVersion(apache_version.strip()) < LooseVersion('2.4.11') or \
+        LooseVersion(openssl_version.strip()) < LooseVersion('1.0.2l'):
+        # should be old version
+        # assert SSLSessionTickets not in conf file
+        if "SSLSessionTickets" in contents:
+            raise Exception("Apache or OpenSSL version is too old, "
+                "but SSLSessionTickets is set.")
+    else:
+        # should be current version
+        # assert SSLSessionTickets in conf file
+        if "SSLSessionTickets" not in contents:
+            raise Exception("Apache and OpenSSL versions are sufficiently new, "
+                "but SSLSessionTickets is not set.")
+
+if __name__ == '__main__':
+    main(*sys.argv[1:])
diff --git a/letstest/scripts/test_sdists.sh b/letstest/scripts/test_sdists.sh
new file mode 100755
index 000000000..562169524
--- /dev/null
+++ b/letstest/scripts/test_sdists.sh
@@ -0,0 +1,53 @@
+#!/bin/sh -xe
+
+cd letsencrypt
+
+BOOTSTRAP_SCRIPT="letstest/scripts/bootstrap_os_packages.sh"
+VENV_PATH=venv
+
+# install OS packages
+. $BOOTSTRAP_SCRIPT
+
+# setup venv
+python3 -m venv $VENV_PATH
+$VENV_PATH/bin/python3 tools/pipstrap.py
+. "$VENV_PATH/bin/activate"
+# pytest is needed to run tests on our packages so we install a pinned version here.
+tools/pip_install.py pytest
+
+# setup constraints
+TEMP_DIR=$(mktemp -d)
+CONSTRAINTS="$TEMP_DIR/constraints.txt"
+cp tools/requirements.txt "$CONSTRAINTS"
+
+# We pin cryptography to 3.1.1 and pyopenssl to 19.1.0 specifically for CentOS 7 / RHEL 7
+# because these systems ship only with OpenSSL 1.0.2, and this OpenSSL version support has been
+# dropped on cryptography>=3.2 and pyopenssl>=20.0.0.
+# Using this old version of OpenSSL would break the cryptography and pyopenssl wheels builds.
+if [ -f /etc/redhat-release ] && [ "$(. /etc/os-release 2> /dev/null && echo "$VERSION_ID" | cut -d '.' -f1)" -eq 7 ]; then
+  sed -i 's|cryptography==.*|cryptography==3.1.1|g' "$CONSTRAINTS"
+  sed -i 's|pyopenssl==.*|pyopenssl==19.1.0|g' "$CONSTRAINTS"
+fi
+
+
+PLUGINS="certbot-apache certbot-nginx"
+# build sdists
+for pkg_dir in acme certbot $PLUGINS; do
+    cd $pkg_dir
+    python setup.py clean
+    rm -rf build dist
+    python setup.py sdist
+    mv dist/* $TEMP_DIR
+    cd -
+done
+
+VERSION=$(python letstest/scripts/version.py)
+# test sdists
+cd $TEMP_DIR
+for pkg in acme certbot $PLUGINS; do
+    tar -xvf "$pkg-$VERSION.tar.gz"
+    cd "$pkg-$VERSION"
+    PIP_CONSTRAINT=../constraints.txt PIP_NO_BINARY=:all: pip install .
+    python -m pytest
+    cd -
+done
diff --git a/letstest/scripts/version.py b/letstest/scripts/version.py
new file mode 100755
index 000000000..6e538b032
--- /dev/null
+++ b/letstest/scripts/version.py
@@ -0,0 +1,28 @@
+#!/usr/bin/env python
+"""Get the current Certbot version number.
+
+Provides a simple utility for determining the Certbot version number
+
+"""
+from __future__ import print_function
+from os.path import abspath, dirname, join
+import re
+
+
+def certbot_version(letstest_scripts_dir):
+    """Return the version number stamped in certbot/__init__.py."""
+    return re.search('''^__version__ = ['"](.+)['"].*''',
+                     file_contents(join(dirname(dirname(letstest_scripts_dir)),
+                                        'certbot',
+                                        'certbot',
+                                        '__init__.py')),
+                     re.M).group(1)
+
+
+def file_contents(path):
+    with open(path) as file:
+        return file.read()
+
+
+if __name__ == '__main__':
+    print(certbot_version(dirname(abspath(__file__))))
diff --git a/letstest/setup.py b/letstest/setup.py
new file mode 100644
index 000000000..a552cf920
--- /dev/null
+++ b/letstest/setup.py
@@ -0,0 +1,45 @@
+from setuptools import find_packages
+from setuptools import setup
+
+setup(
+    name='letstest',
+    version='1.0',
+    description='Test Certbot on different AWS images',
+    url='https://github.com/certbot/certbot',
+    author='Certbot Project',
+    author_email='certbot-dev@eff.org',
+    license='Apache License 2.0',
+    python_requires='>=3.6',
+    classifiers=[
+        'Development Status :: 5 - Production/Stable',
+        'Intended Audience :: Developers',
+        'License :: OSI Approved :: Apache Software License',
+        'Programming Language :: Python',
+        'Programming Language :: Python :: 3',
+        'Programming Language :: Python :: 3.6',
+        'Programming Language :: Python :: 3.7',
+        'Programming Language :: Python :: 3.8',
+        'Programming Language :: Python :: 3.9',
+        'Topic :: Internet :: WWW/HTTP',
+        'Topic :: Security',
+    ],
+
+    packages=find_packages(),
+    include_package_data=True,
+    install_requires=[
+        # awscli isn't required by the tests themselves, but it is a useful
+        # tool to have when using these tests to generate keys and control
+        # running instances so the dependency is declared here for convenience.
+        'awscli',
+        'boto3',
+        'botocore',
+        # The API from Fabric 2.0+ is used instead of the 1.0 API.
+        'fabric>=2',
+        'pyyaml',
+    ],
+    entry_points={
+        'console_scripts': [
+            'letstest=letstest.multitester:main',
+        ],
+    }
+)
diff --git a/letstest/targets/apache2_targets.yaml b/letstest/targets/apache2_targets.yaml
new file mode 100644
index 000000000..2663782ce
--- /dev/null
+++ b/letstest/targets/apache2_targets.yaml
@@ -0,0 +1,47 @@
+# These images are located in us-east-1.
+#
+# All machines must currently use x86_64 since Pebble does not currently
+# publish images for other architectures.
+
+targets:
+  #-----------------------------------------------------------------------------
+  #Ubuntu
+  - ami: ami-0f2e2c076f4c2f941
+    name: ubuntu20.10
+    type: ubuntu
+    virt: hvm
+    user: ubuntu
+  - ami: ami-0758470213bdd23b1
+    name: ubuntu20.04
+    type: ubuntu
+    virt: hvm
+    user: ubuntu
+  - ami: ami-095192256fe1477ad
+    name: ubuntu18.04LTS
+    type: ubuntu
+    virt: hvm
+    user: ubuntu
+  - ami: ami-09677e0a6b14905b0
+    name: ubuntu16.04LTS
+    type: ubuntu
+    virt: hvm
+    user: ubuntu
+  #-----------------------------------------------------------------------------
+  # Debian
+  - ami: ami-01db78123b2b99496
+    name: debian10
+    type: ubuntu
+    virt: hvm
+    user: admin
+  - ami: ami-003f19e0e687de1cd
+    name: debian9
+    type: ubuntu
+    virt: hvm
+    user: admin
+  #-----------------------------------------------------------------------------
+  # CentOS
+  - ami: ami-9887c6e7
+    name: centos7
+    type: centos
+    virt: hvm
+    user: centos
diff --git a/letstest/targets/targets.yaml b/letstest/targets/targets.yaml
new file mode 100644
index 000000000..97c775f6c
--- /dev/null
+++ b/letstest/targets/targets.yaml
@@ -0,0 +1,59 @@
+# These images are located in us-east-1.
+
+targets:
+  #-----------------------------------------------------------------------------
+  #Ubuntu
+  - ami: ami-0f2e2c076f4c2f941
+    name: ubuntu20.10
+    type: ubuntu
+    virt: hvm
+    user: ubuntu
+  - ami: ami-0758470213bdd23b1
+    name: ubuntu20.04
+    type: ubuntu
+    virt: hvm
+    user: ubuntu
+  - ami: ami-095192256fe1477ad
+    name: ubuntu18.04LTS
+    type: ubuntu
+    virt: hvm
+    user: ubuntu
+  #-----------------------------------------------------------------------------
+  # Debian
+  - ami: ami-01db78123b2b99496
+    name: debian10
+    type: ubuntu
+    virt: hvm
+    user: admin
+  - ami: ami-0dcd54b7d2fff584f
+    name: debian10_arm64
+    type: ubuntu
+    virt: hvm
+    user: admin
+    machine_type: a1.medium
+  #-----------------------------------------------------------------------------
+  # Other Redhat Distros
+  - ami: ami-0916c408cb02e310b
+    name: RHEL7
+    type: centos
+    virt: hvm
+    user: ec2-user
+  - ami: ami-0c322300a1dd5dc79
+    name: RHEL8
+    type: centos
+    virt: hvm
+    user: ec2-user
+  #-----------------------------------------------------------------------------
+  # CentOS
+  # These Marketplace AMIs must, irritatingly, have their terms manually
+  # agreed to on the AWS marketplace site for any new AWS account using them...
+  - ami: ami-9887c6e7
+    name: centos7
+    type: centos
+    virt: hvm
+    user: centos
+  - ami: ami-01ca03df4a6012157
+    name: centos8
+    type: centos
+    virt: hvm
+    user: centos