diff options
author | Brad Warren <bmw@users.noreply.github.com> | 2021-05-04 03:42:30 +0300 |
---|---|---|
committer | GitHub <noreply@github.com> | 2021-05-04 03:42:30 +0300 |
commit | dd0e590de3095097832051756d13850bf1561d8e (patch) | |
tree | 83f68fc2005d2669eac252fb2731faf956a9f68d /letstest | |
parent | d3d9a05826af027b31de5cb60948dafbf6597873 (diff) |
Make a test farm tests package (#8821)
Fixes https://github.com/certbot/certbot/issues/8781.
This PR makes our test farm tests into a normal package so it and its dependencies can be tracked and installed like our other packages.
Other noteworthy changes in this PR:
* Rather than continuing to place logs in your CWD, they're placed in a temporary directory that is printed to the terminal.
* `tests/letstest/auto_targets.yaml` was deleted rather than renamed because the file is no longer used.
* make a letstest package
* remove deleted deps
* fix letstest install
* add __init__.py
* call main
* Explicitly mention activating venv
* rerename file
* fix version.py path
* clarify "this"
* Use >= instead of caret requirement
Diffstat (limited to 'letstest')
-rw-r--r-- | letstest/README.md | 65 | ||||
-rw-r--r-- | letstest/letstest/__init__.py | 0 | ||||
-rw-r--r-- | letstest/letstest/multitester.py | 526 | ||||
-rwxr-xr-x | letstest/scripts/bootstrap_os_packages.sh | 140 | ||||
-rwxr-xr-x | letstest/scripts/test_apache2.sh | 137 | ||||
-rw-r--r-- | letstest/scripts/test_openssl_version.py | 30 | ||||
-rwxr-xr-x | letstest/scripts/test_sdists.sh | 53 | ||||
-rwxr-xr-x | letstest/scripts/version.py | 28 | ||||
-rw-r--r-- | letstest/setup.py | 45 | ||||
-rw-r--r-- | letstest/targets/apache2_targets.yaml | 47 | ||||
-rw-r--r-- | letstest/targets/targets.yaml | 59 |
11 files changed, 1130 insertions, 0 deletions
diff --git a/letstest/README.md b/letstest/README.md new file mode 100644 index 000000000..c569d1e8f --- /dev/null +++ b/letstest/README.md @@ -0,0 +1,65 @@ +# letstest +Simple AWS testfarm scripts for certbot client testing + +- Launches EC2 instances with a given list of AMIs for different distros +- Copies certbot repo and puts it on the instances +- Runs certbot tests (bash scripts) on all of these +- Logs execution and success/fail for debugging + +## Notes + - Some AWS images, e.g. official CentOS and FreeBSD images + require acceptance of user terms on the AWS marketplace + website. This can't be automated. + - AWS EC2 has a default limit of 20 t2/t1 instances, if more + are needed, they need to be requested via online webform. + +## Installation and configuration + +This package is installed in the Certbot development environment that is +created by following the instructions at +https://certbot.eff.org/docs/contributing.html#running-a-local-copy-of-the-client. + +After activating that virtual environment, you can then configure AWS +credentials and create a key by running: +``` +>aws configure --profile <profile name> +[interactive: enter secrets for IAM role] +>aws ec2 create-key-pair --profile <profile name> --key-name <key name> --query 'KeyMaterial' --output text > whatever/path/you/want.pem +``` +Note: whatever you pick for `<key name>` will be shown to other users with AWS access. + +When prompted for a default region name, enter: `us-east-1`. + +## Usage +To run tests, activate the virtual environment you created above and from this directory run: +``` +>letstest targets/targets.yaml /path/to/your/key.pem <profile name> scripts/<test to run> +``` + +You can only run up to two tests at once. The following error is often indicative of there being too many AWS instances running on our account: +``` +NameError: name 'instances' is not defined +``` + +If you see this, you can run the following command to shut down all running instances: +``` +aws ec2 terminate-instances --profile <profile name> --instance-ids $(aws ec2 describe-instances --profile <profile name> | grep <key name> | cut -f8) +``` + +It will take a minute for these instances to shut down and become available again. Running this will invalidate any in progress tests. + +A temporary directory whose name is output by the tests is also created with a log file from each instance of the test and a file named "results" containing the output above. +The tests take quite a while to run. + +## Scripts +Example scripts are in the 'scripts' directory, these are just bash scripts that have a few parameters passed +to them at runtime via environment variables. test_apache2.sh is a useful reference. + +test_apache2 runs the dev venv and does local tests. + +See: +- https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html +- https://docs.aws.amazon.com/cli/latest/userguide/cli-ec2-keypairs.html + +Main repos: +- https://github.com/letsencrypt/letsencrypt diff --git a/letstest/letstest/__init__.py b/letstest/letstest/__init__.py new file mode 100644 index 000000000..e69de29bb --- /dev/null +++ b/letstest/letstest/__init__.py diff --git a/letstest/letstest/multitester.py b/letstest/letstest/multitester.py new file mode 100644 index 000000000..a56bf0f37 --- /dev/null +++ b/letstest/letstest/multitester.py @@ -0,0 +1,526 @@ +""" +Certbot Integration Test Tool + +- Launches EC2 instances with a given list of AMIs for different distros +- Copies certbot repo and puts it on the instances +- Runs certbot tests (bash scripts) on all of these +- Logs execution and success/fail for debugging + +Notes: + - Some AWS images, e.g. official CentOS and FreeBSD images + require acceptance of user terms on the AWS marketplace + website. This can't be automated. + - AWS EC2 has a default limit of 20 t2/t1 instances, if more + are needed, they need to be requested via online webform. + +Usage: + - Requires AWS IAM secrets to be set up with aws cli + - Requires an AWS associated keyfile <keyname>.pem + +>aws configure --profile HappyHacker +[interactive: enter secrets for IAM role] +>aws ec2 create-key-pair --profile HappyHacker --key-name MyKeyPair \ + --query 'KeyMaterial' --output text > MyKeyPair.pem +then: +>letstest targets/targets.yaml MyKeyPair.pem HappyHacker scripts/test_sdists.sh +see: + https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html + https://docs.aws.amazon.com/cli/latest/userguide/cli-ec2-keypairs.html +""" +import argparse +import multiprocessing as mp +from multiprocessing import Manager +import os +import socket +import sys +import tempfile +import time +import traceback +import urllib.error as urllib_error +import urllib.request as urllib_request + +import boto3 +from botocore.exceptions import ClientError +import yaml + +from fabric import Config +from fabric import Connection + +# Command line parser +#------------------------------------------------------------------------------- +parser = argparse.ArgumentParser(description='Builds EC2 cluster for testing.') +parser.add_argument('config_file', + help='yaml configuration file for AWS server cluster') +parser.add_argument('key_file', + help='key file (<keyname>.pem) for AWS') +parser.add_argument('aws_profile', + help='profile for AWS (i.e. as in ~/.aws/certificates)') +parser.add_argument('test_script', + default='test_sdists.sh', + help='path of bash script in to deploy and run') +parser.add_argument('--repo', + default='https://github.com/letsencrypt/letsencrypt.git', + help='certbot git repo to use') +parser.add_argument('--branch', + default='~', + help='certbot git branch to trial') +parser.add_argument('--pull_request', + default='~', + help='letsencrypt/letsencrypt pull request to trial') +parser.add_argument('--merge_master', + action='store_true', + help="if set merges PR into master branch of letsencrypt/letsencrypt") +parser.add_argument('--saveinstances', + action='store_true', + help="don't kill EC2 instances after run, useful for debugging") +parser.add_argument('--alt_pip', + default='', + help="server from which to pull candidate release packages") +cl_args = parser.parse_args() + +# Credential Variables +#------------------------------------------------------------------------------- +# assumes naming: <key_filename> = <keyname>.pem +KEYFILE = cl_args.key_file +KEYNAME = os.path.split(cl_args.key_file)[1].split('.pem')[0] +PROFILE = None if cl_args.aws_profile == 'SET_BY_ENV' else cl_args.aws_profile + +# Globals +#------------------------------------------------------------------------------- +SECURITY_GROUP_NAME = 'certbot-security-group' +SENTINEL = None #queue kill signal +SUBNET_NAME = 'certbot-subnet' + +class Status: + """Possible statuses of client tests.""" + PASS = 'pass' + FAIL = 'fail' + +# Boto3/AWS automation functions +#------------------------------------------------------------------------------- +def should_use_subnet(subnet): + """Should we use the given subnet for these tests? + + We should if it is the default subnet for the availability zone or the + subnet is named "certbot-subnet". + + """ + if not subnet.map_public_ip_on_launch: + return False + if subnet.default_for_az: + return True + for tag in subnet.tags: + if tag['Key'] == 'Name' and tag['Value'] == SUBNET_NAME: + return True + return False + +def make_security_group(vpc): + """Creates a security group in the given VPC.""" + # will fail if security group of GroupName already exists + # cannot have duplicate SGs of the same name + mysg = vpc.create_security_group(GroupName=SECURITY_GROUP_NAME, + Description='security group for automated testing') + mysg.authorize_ingress(IpProtocol="tcp", CidrIp="0.0.0.0/0", FromPort=22, ToPort=22) + # for mosh + mysg.authorize_ingress(IpProtocol="udp", CidrIp="0.0.0.0/0", FromPort=60000, ToPort=61000) + return mysg + +def make_instance(ec2_client, + instance_name, + ami_id, + keyname, + security_group_id, + subnet_id, + self_destruct, + machine_type='t2.micro'): + """Creates an instance using the given parameters. + + If self_destruct is True, the instance will be configured to shutdown after + 1 hour and to terminate itself on shutdown. + + """ + block_device_mappings = _get_block_device_mappings(ec2_client, ami_id) + tags = [{'Key': 'Name', 'Value': instance_name}] + tag_spec = [{'ResourceType': 'instance', 'Tags': tags}] + kwargs = { + 'BlockDeviceMappings': block_device_mappings, + 'ImageId': ami_id, + 'SecurityGroupIds': [security_group_id], + 'SubnetId': subnet_id, + 'KeyName': keyname, + 'MinCount': 1, + 'MaxCount': 1, + 'InstanceType': machine_type, + 'TagSpecifications': tag_spec + } + if self_destruct: + kwargs['InstanceInitiatedShutdownBehavior'] = 'terminate' + kwargs['UserData'] = '#!/bin/bash\nshutdown -P +60\n' + return ec2_client.create_instances(**kwargs)[0] + +def _get_block_device_mappings(ec2_client, ami_id): + """Returns the list of block device mappings to ensure cleanup. + + This list sets connected EBS volumes to be deleted when the EC2 + instance is terminated. + + """ + # Not all devices use EBS, but the default value for DeleteOnTermination + # when the device does use EBS is true. See: + # * https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-blockdev-mapping.html + # * https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-ec2-blockdev-template.html + return [{'DeviceName': mapping['DeviceName'], + 'Ebs': {'DeleteOnTermination': True}} + for mapping in ec2_client.Image(ami_id).block_device_mappings + if not mapping.get('Ebs', {}).get('DeleteOnTermination', True)] + + +# Helper Routines +#------------------------------------------------------------------------------- +def block_until_ssh_open(ipstring, wait_time=10, timeout=120): + "Blocks until server at ipstring has an open port 22" + reached = False + t_elapsed = 0 + while not reached and t_elapsed < timeout: + try: + sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) + sock.connect((ipstring, 22)) + reached = True + except socket.error as err: + time.sleep(wait_time) + t_elapsed += wait_time + sock.close() + +def block_until_instance_ready(booting_instance, extra_wait_time=20): + "Blocks booting_instance until AWS EC2 instance is ready to accept SSH connections" + booting_instance.wait_until_running() + # The instance needs to be reloaded to update its local attributes. See + # https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2.html#EC2.Instance.reload. + booting_instance.reload() + # After waiting for the instance to be running and reloading the instance + # state, we should have an IP address. + assert booting_instance.public_ip_address is not None + block_until_ssh_open(booting_instance.public_ip_address) + time.sleep(extra_wait_time) + return booting_instance + + +# Fabric Routines +#------------------------------------------------------------------------------- +def local_git_clone(local_cxn, repo_url, log_dir): + """clones master of repo_url""" + local_cxn.local('cd %s && if [ -d letsencrypt ]; then rm -rf letsencrypt; fi' % log_dir) + local_cxn.local('cd %s && git clone %s letsencrypt'% (log_dir, repo_url)) + local_cxn.local('cd %s && tar czf le.tar.gz letsencrypt'% log_dir) + +def local_git_branch(local_cxn, repo_url, branch_name, log_dir): + """clones branch <branch_name> of repo_url""" + local_cxn.local('cd %s && if [ -d letsencrypt ]; then rm -rf letsencrypt; fi' % log_dir) + local_cxn.local('cd %s && git clone %s letsencrypt --branch %s --single-branch'% + (log_dir, repo_url, branch_name)) + local_cxn.local('cd %s && tar czf le.tar.gz letsencrypt' % log_dir) + +def local_git_PR(local_cxn, repo_url, PRnumstr, log_dir, merge_master=True): + """clones specified pull request from repo_url and optionally merges into master""" + local_cxn.local('cd %s && if [ -d letsencrypt ]; then rm -rf letsencrypt; fi' % log_dir) + local_cxn.local('cd %s && git clone %s letsencrypt' % (log_dir, repo_url)) + local_cxn.local('cd %s && cd letsencrypt && ' + 'git fetch origin pull/%s/head:lePRtest' % (log_dir, PRnumstr)) + local_cxn.local('cd %s && cd letsencrypt && git checkout lePRtest' % log_dir) + if merge_master: + local_cxn.local('cd %s && cd letsencrypt && git remote update origin' % log_dir) + local_cxn.local('cd %s && cd letsencrypt && ' + 'git merge origin/master -m "testmerge"' % log_dir) + local_cxn.local('cd %s && tar czf le.tar.gz letsencrypt' % log_dir) + +def local_repo_to_remote(cxn, log_dir): + """copies local tarball of repo to remote""" + filename = 'le.tar.gz' + local_path = os.path.join(log_dir, filename) + cxn.put(local=local_path, remote='') + cxn.run('tar xzf %s' % filename) + +def local_repo_clean(local_cxn, log_dir): + """delete tarball""" + filename = 'le.tar.gz' + local_path = os.path.join(log_dir, filename) + local_cxn.local('rm %s' % local_path) + +def deploy_script(cxn, scriptpath, *args): + """copies to remote and executes local script""" + cxn.put(local=scriptpath, remote='', preserve_mode=True) + scriptfile = os.path.split(scriptpath)[1] + args_str = ' '.join(args) + cxn.run('./'+scriptfile+' '+args_str) + +def install_and_launch_certbot(cxn, instance, target, log_dir): + local_repo_to_remote(cxn, log_dir) + # This needs to be like this, I promise. 1) The env argument to run doesn't work. + # See https://github.com/fabric/fabric/issues/1744. 2) prefix() sticks an && between + # the commands, so it needs to be exports rather than no &&s in between for the script subshell. + with cxn.prefix('export PUBLIC_IP=%s && export PRIVATE_IP=%s && ' + 'export PUBLIC_HOSTNAME=%s && export PIP_EXTRA_INDEX_URL=%s && ' + 'export OS_TYPE=%s' % + (instance.public_ip_address, + instance.private_ip_address, + instance.public_dns_name, + cl_args.alt_pip, + target['type'])): + deploy_script(cxn, cl_args.test_script) + +def grab_certbot_log(cxn): + "grabs letsencrypt.log via cat into logged stdout" + cxn.sudo('/bin/bash -l -i -c \'if [ -f "/var/log/letsencrypt/letsencrypt.log" ]; then ' + + 'cat "/var/log/letsencrypt/letsencrypt.log"; else echo "[novarlog]"; fi\'') + # fallback file if /var/log is unwriteable...? correct? + cxn.sudo('/bin/bash -l -i -c \'if [ -f ./certbot.log ]; then ' + + 'cat ./certbot.log; else echo "[nolocallog]"; fi\'') + + +def create_client_instance(ec2_client, target, security_group_id, subnet_id, self_destruct): + """Create a single client instance for running tests.""" + if 'machine_type' in target: + machine_type = target['machine_type'] + elif target['virt'] == 'hvm': + machine_type = 't2.medium' + else: + # 32 bit systems + machine_type = 'c1.medium' + name = 'le-%s'%target['name'] + print(name, end=" ") + return make_instance(ec2_client, + name, + target['ami'], + KEYNAME, + machine_type=machine_type, + security_group_id=security_group_id, + subnet_id=subnet_id, + self_destruct=self_destruct) + + +def test_client_process(fab_config, inqueue, outqueue, log_dir): + cur_proc = mp.current_process() + for inreq in iter(inqueue.get, SENTINEL): + ii, instance_id, target = inreq + + # Each client process is given its own session due to the suggestion at + # https://boto3.amazonaws.com/v1/documentation/api/latest/guide/resources.html?highlight=multithreading#multithreading-multiprocessing. + aws_session = boto3.session.Session(profile_name=PROFILE) + ec2_client = aws_session.resource('ec2') + instance = ec2_client.Instance(id=instance_id) + + #save all stdout to log file + sys.stdout = open(log_dir+'/'+'%d_%s.log'%(ii,target['name']), 'w') + + print("[%s : client %d %s %s]" % (cur_proc.name, ii, target['ami'], target['name'])) + instance = block_until_instance_ready(instance) + print("server %s at %s"%(instance, instance.public_ip_address)) + host_string = "%s@%s"%(target['user'], instance.public_ip_address) + print(host_string) + + with Connection(host_string, config=fab_config) as cxn: + try: + install_and_launch_certbot(cxn, instance, target, log_dir) + outqueue.put((ii, target, Status.PASS)) + print("%s - %s SUCCESS"%(target['ami'], target['name'])) + except: + outqueue.put((ii, target, Status.FAIL)) + print("%s - %s FAIL"%(target['ami'], target['name'])) + traceback.print_exc(file=sys.stdout) + pass + + # append server certbot.log to each per-machine output log + print("\n\ncertbot.log\n" + "-"*80 + "\n") + try: + grab_certbot_log(cxn) + except: + print("log fail\n") + traceback.print_exc(file=sys.stdout) + pass + + +def cleanup(cl_args, instances, targetlist, log_dir): + print('Logs in ', log_dir) + # If lengths of instances and targetlist aren't equal, instances failed to + # start before running tests so leaving instances running for debugging + # isn't very useful. Let's cleanup after ourselves instead. + if len(instances) != len(targetlist) or not cl_args.saveinstances: + print('Terminating EC2 Instances') + for instance in instances: + instance.terminate() + else: + # print login information for the boxes for debugging + for ii, target in enumerate(targetlist): + print(target['name'], + target['ami'], + "%s@%s"%(target['user'], instances[ii].public_ip_address)) + + +def main(): + # Fabric library controlled through global env parameters + fab_config = Config(overrides={ + "connect_kwargs": { + "key_filename": [KEYFILE], # https://github.com/fabric/fabric/issues/2007 + }, + "run": { + "echo": True, + "pty": True, + }, + "timeouts": { + "connect": 10, + }, + }) + # no network connection, so don't worry about closing this one. + local_cxn = Connection('localhost', config=fab_config) + + # Set up local copy of git repo + #------------------------------------------------------------------------------- + log_dir = tempfile.mkdtemp() # points to logging / working directory + print("Local dir for test repo and logs: %s"%log_dir) + + try: + # figure out what git object to test and locally create it in log_dir + print("Making local git repo") + if cl_args.pull_request != '~': + print('Testing PR %s ' % cl_args.pull_request, + "MERGING into master" if cl_args.merge_master else "") + local_git_PR(local_cxn, cl_args.repo, cl_args.pull_request, log_dir, + cl_args.merge_master) + elif cl_args.branch != '~': + print('Testing branch %s of %s' % (cl_args.branch, cl_args.repo)) + local_git_branch(local_cxn, cl_args.repo, cl_args.branch, log_dir) + else: + print('Testing current branch of %s' % cl_args.repo, log_dir) + local_git_clone(local_cxn, cl_args.repo, log_dir) + except BaseException: + print("FAIL: trouble with git repo") + traceback.print_exc() + exit(1) + + + # Set up EC2 instances + #------------------------------------------------------------------------------- + configdata = yaml.safe_load(open(cl_args.config_file, 'r')) + targetlist = configdata['targets'] + print('Testing against these images: [%d total]'%len(targetlist)) + for target in targetlist: + print(target['ami'], target['name']) + + print("Connecting to EC2 using\n profile %s\n keyname %s\n keyfile %s"%(PROFILE, KEYNAME, KEYFILE)) + aws_session = boto3.session.Session(profile_name=PROFILE) + ec2_client = aws_session.resource('ec2') + + print("Determining Subnet") + for subnet in ec2_client.subnets.all(): + if should_use_subnet(subnet): + subnet_id = subnet.id + vpc_id = subnet.vpc.id + break + else: + print("No usable subnet exists!") + print("Please create a VPC with a subnet named {0}".format(SUBNET_NAME)) + print("that maps public IPv4 addresses to instances launched in the subnet.") + sys.exit(1) + + print("Making Security Group") + vpc = ec2_client.Vpc(vpc_id) + sg_exists = False + for sg in vpc.security_groups.all(): + if sg.group_name == SECURITY_GROUP_NAME: + security_group_id = sg.id + sg_exists = True + print(" %s already exists"%SECURITY_GROUP_NAME) + if not sg_exists: + security_group_id = make_security_group(vpc).id + time.sleep(30) + + instances = [] + try: + print("Creating instances: ", end="") + # If we want to preserve instances, do not have them self-destruct. + self_destruct = not cl_args.saveinstances + for target in targetlist: + instances.append( + create_client_instance(ec2_client, target, + security_group_id, subnet_id, + self_destruct) + ) + print() + + # Install and launch client scripts in parallel + #------------------------------------------------------------------------------- + print("Uploading and running test script in parallel: %s"%cl_args.test_script) + print("Output routed to log files in %s"%log_dir) + # (Advice: always use Manager.Queue, never regular multiprocessing.Queue + # the latter has implementation flaws that deadlock it in some circumstances) + manager = Manager() + outqueue = manager.Queue() + inqueue = manager.Queue() + + # launch as many processes as clients to test + num_processes = len(targetlist) + jobs = [] #keep a reference to current procs + + + # initiate process execution + client_process_args=(fab_config, inqueue, outqueue, log_dir) + for i in range(num_processes): + p = mp.Process(target=test_client_process, args=client_process_args) + jobs.append(p) + p.daemon = True # kills subprocesses if parent is killed + p.start() + + # fill up work queue + for ii, target in enumerate(targetlist): + inqueue.put((ii, instances[ii].id, target)) + + # add SENTINELs to end client processes + for i in range(num_processes): + inqueue.put(SENTINEL) + print('Waiting on client processes', end='') + for p in jobs: + while p.is_alive(): + p.join(5 * 60) + # Regularly print output to keep Travis happy + print('.', end='') + sys.stdout.flush() + print() + # add SENTINEL to output queue + outqueue.put(SENTINEL) + + # clean up + local_repo_clean(local_cxn, log_dir) + + # print and save summary results + results_file = open(log_dir+'/results', 'w') + outputs = [outq for outq in iter(outqueue.get, SENTINEL)] + outputs.sort(key=lambda x: x[0]) + failed = False + results_msg = "" + for outq in outputs: + ii, target, status = outq + if status == Status.FAIL: + failed = True + with open(log_dir+'/'+'%d_%s.log'%(ii,target['name']), 'r') as f: + print(target['name'] + " test failed. Test log:") + print(f.read()) + results_msg = results_msg + '%d %s %s\n'%(ii, target['name'], status) + results_file.write('%d %s %s\n'%(ii, target['name'], status)) + print(results_msg) + if len(outputs) != num_processes: + failed = True + failure_message = 'FAILURE: Some target machines failed to run and were not tested. ' +\ + 'Tests should be rerun.' + print(failure_message) + results_file.write(failure_message + '\n') + results_file.close() + + if failed: + sys.exit(1) + + finally: + cleanup(cl_args, instances, targetlist, log_dir) + + +if __name__ == '__main__': + main() diff --git a/letstest/scripts/bootstrap_os_packages.sh b/letstest/scripts/bootstrap_os_packages.sh new file mode 100755 index 000000000..3f4c6e30e --- /dev/null +++ b/letstest/scripts/bootstrap_os_packages.sh @@ -0,0 +1,140 @@ +#!/bin/sh +# +# Install OS dependencies for test farm tests. + +set -ex # Work even if somebody does "sh thisscript.sh". + +error() { + echo "$@" +} + +if command -v command > /dev/null 2>&1 ; then + export EXISTS="command -v" +elif which which > /dev/null 2>&1 ; then + export EXISTS="which" +else + error "Cannot find command nor which... please install one!" + exit 1 +fi + +# Sets LE_PYTHON to Python version string and PYVER to the first two +# digits of the python version. +DeterminePythonVersion() { + # If no Python is found, PYVER is set to 0. + for LE_PYTHON in python3 python2.7 python27 python2 python; do + # Break (while keeping the LE_PYTHON value) if found. + $EXISTS "$LE_PYTHON" > /dev/null && break + done + if [ "$?" != "0" ]; then + PYVER=0 + return 0 + fi + + PYVER=$("$LE_PYTHON" -V 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//') +} + +BootstrapDebCommon() { + sudo apt-get update || error apt-get update hit problems but continuing anyway... + + sudo apt-get install -y --no-install-recommends \ + python3 \ + python3-dev \ + python3-venv \ + gcc \ + libaugeas0 \ + libssl-dev \ + openssl \ + libffi-dev \ + ca-certificates \ + build-essential \ + curl \ + make # needed on debian 9 arm64 which doesn't have a python3 pynacl wheel + + # make sure rust isn't installed by the package manager + if ! sudo apt-get remove -y rustc; then + error "Could not remove existing rust. Aborting bootstrap!" + exit 1 + fi + + # Install rust for cryptography (needed on Debian) + curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y + . $HOME/.cargo/env +} + +# Sets TOOL to the name of the package manager +InitializeRPMCommonBase() { + if type dnf 2>/dev/null + then + TOOL=dnf + elif type yum 2>/dev/null + then + TOOL=yum + + else + error "Neither yum nor dnf found. Aborting bootstrap!" + exit 1 + fi + +} + +BootstrapRpmCommonBase() { + # Arguments: whitespace-delimited python packages to install + + InitializeRPMCommonBase + + pkgs=" + gcc + augeas-libs + openssl + openssl-devel + libffi-devel + redhat-rpm-config + ca-certificates + cargo + " + + # Add the python packages + pkgs="$pkgs + $1 + " + + if $TOOL list installed "httpd" >/dev/null 2>&1; then + pkgs="$pkgs + mod_ssl + " + fi + + if ! sudo $TOOL install -y $pkgs; then + error "Could not install OS dependencies. Aborting bootstrap!" + exit 1 + fi +} + +BootstrapRpmPython3() { + InitializeRPMCommonBase + + python_pkgs="python3 + python3-devel + " + + if ! sudo $TOOL list 'python3*-devel' >/dev/null 2>&1; then + sudo yum-config-manager --enable rhui-REGION-rhel-server-extras rhui-REGION-rhel-server-optional + fi + + BootstrapRpmCommonBase "$python_pkgs" +} + +# Set Bootstrap to the function that installs OS dependencies on this system. +if [ -f /etc/debian_version ]; then + Bootstrap() { + BootstrapDebCommon + } +elif [ -f /etc/redhat-release ]; then + DeterminePythonVersion + Bootstrap() { + BootstrapRpmPython3 + } + +fi + +Bootstrap diff --git a/letstest/scripts/test_apache2.sh b/letstest/scripts/test_apache2.sh new file mode 100755 index 000000000..9d9ca6c12 --- /dev/null +++ b/letstest/scripts/test_apache2.sh @@ -0,0 +1,137 @@ +#!/bin/bash -x + +# $OS_TYPE $PUBLIC_IP $PRIVATE_IP $PUBLIC_HOSTNAME $BOULDER_URL +# are dynamically set at execution + +if [ "$OS_TYPE" = "ubuntu" ] +then + CONFFILE=/etc/apache2/sites-available/000-default.conf + sudo apt-get update + sudo apt-get -y --no-upgrade install apache2 curl + sudo apt-get -y install realpath # needed for test-apache-conf + # For apache 2.4, set up ServerName + sudo sed -i '/ServerName/ s/#ServerName/ServerName/' $CONFFILE + sudo sed -i '/ServerName/ s/www.example.com/'$PUBLIC_HOSTNAME'/' $CONFFILE + if [ $(python3 -V 2>&1 | cut -d" " -f 2 | cut -d. -f1,2 | sed 's/\.//') -lt 36 ] + then + # Upgrade python version using pyenv because py3.5 is deprecated + # Don't upgrade if it's already 3.8 because pyenv doesn't work great on arm, and + # our arm representative happens to be ubuntu20, which already has a perfectly + # good version of python. + sudo apt-get install -y make gcc build-essential libssl-dev zlib1g-dev libbz2-dev \ + libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev \ + xz-utils tk-dev libffi-dev liblzma-dev python-openssl git # pyenv deps + curl https://pyenv.run | bash + export PATH="~/.pyenv/bin:$PATH" + pyenv init - + pyenv virtualenv-init - + pyenv install 3.8.5 + pyenv global 3.8.5 + # you do, in fact need to run these again, exactly like this. + eval "$(pyenv init -)" + eval "$(pyenv virtualenv-init -)" + fi +elif [ "$OS_TYPE" = "centos" ] +then + CONFFILE=/etc/httpd/conf/httpd.conf + sudo setenforce 0 || true #disable selinux + sudo yum -y install httpd + sudo yum -y install nghttp2 || echo this is probably ok but see https://bugzilla.redhat.com/show_bug.cgi?id=1358875 + sudo service httpd start + sudo mkdir -p /var/www/$PUBLIC_HOSTNAME/public_html + sudo chmod -R oug+rwx /var/www + sudo chmod -R oug+rw /etc/httpd + sudo echo '<html><head><title>foo</title></head><body>bar</body></html>' > /var/www/$PUBLIC_HOSTNAME/public_html/index.html + sudo mkdir /etc/httpd/sites-available #certbot requires this... + sudo mkdir /etc/httpd/sites-enabled #certbot requires this... + #sudo echo "IncludeOptional sites-enabled/*.conf" >> /etc/httpd/conf/httpd.conf + sudo echo """ +<VirtualHost *:80> + ServerName $PUBLIC_HOSTNAME + DocumentRoot /var/www/$PUBLIC_HOSTNAME/public_html + ErrorLog /var/www/$PUBLIC_HOSTNAME/error.log + CustomLog /var/www/$PUBLIC_HOSTNAME/requests.log combined +</VirtualHost>""" >> /etc/httpd/conf.d/$PUBLIC_HOSTNAME.conf + #sudo cp /etc/httpd/sites-available/$PUBLIC_HOSTNAME.conf /etc/httpd/sites-enabled/ +fi + +# Run certbot-apache2. +cd letsencrypt + +echo "Bootstrapping dependencies..." +sudo letstest/scripts/bootstrap_os_packages.sh +if [ $? -ne 0 ] ; then + exit 1 +fi + +tools/venv.py -e acme[dev] -e certbot[dev,docs] -e certbot-apache -e certbot-ci +PEBBLE_LOGS="acme_server.log" +PEBBLE_URL="https://localhost:14000/dir" +# We configure Pebble to use port 80 for http-01 validation rather than an +# alternate port because: +# 1) It allows us to test with Apache configurations that are more realistic +# and closer to the default configuration on various OSes. +# 2) As of writing this, Certbot's Apache plugin requires there to be an +# existing virtual host for the port used for http-01 validation. +venv/bin/run_acme_server --http-01-port 80 > "${PEBBLE_LOGS}" 2>&1 & + +DumpPebbleLogs() { + if [ -f "${PEBBLE_LOGS}" ] ; then + echo "Pebble's logs were:" + cat "${PEBBLE_LOGS}" + fi +} + +for n in $(seq 1 150) ; do + if curl --insecure "${PEBBLE_URL}" 2>/dev/null; then + break + else + echo "waiting for pebble" + sleep 1 + fi +done +if ! curl --insecure "${PEBBLE_URL}" 2>/dev/null; then + echo "timed out waiting for pebble to start" + DumpPebbleLogs + exit 1 +fi + +sudo "venv/bin/certbot" -v --debug --text --agree-tos --no-verify-ssl \ + --renew-by-default --redirect --register-unsafely-without-email \ + --domain "${PUBLIC_HOSTNAME}" --server "${PEBBLE_URL}" +if [ $? -ne 0 ] ; then + FAIL=1 +fi + +# Check that ssl_module detection is working on various systems +if [ "$OS_TYPE" = "ubuntu" ] ; then + MOD_SSL_LOCATION="/usr/lib/apache2/modules/mod_ssl.so" + APACHE_NAME=apache2ctl +elif [ "$OS_TYPE" = "centos" ]; then + MOD_SSL_LOCATION="/etc/httpd/modules/mod_ssl.so" + APACHE_NAME=httpd +fi +OPENSSL_VERSION=$(strings "$MOD_SSL_LOCATION" | egrep -o -m1 '^OpenSSL ([0-9]\.[^ ]+) ' | tail -c +9) +APACHE_VERSION=$(sudo $APACHE_NAME -v | egrep -o 'Apache/([0-9]\.[^ ]+)' | tail -c +8) +"venv/bin/python" letstest/scripts/test_openssl_version.py "$OPENSSL_VERSION" "$APACHE_VERSION" +if [ $? -ne 0 ] ; then + FAIL=1 +fi + + +if [ "$OS_TYPE" = "ubuntu" ] ; then + export SERVER="${PEBBLE_URL}" + "venv/bin/tox" -e apacheconftest +else + echo Not running hackish apache tests on $OS_TYPE +fi + +if [ $? -ne 0 ] ; then + FAIL=1 +fi + +# return error if any of the subtests failed +if [ "$FAIL" = 1 ] ; then + DumpPebbleLogs + exit 1 +fi diff --git a/letstest/scripts/test_openssl_version.py b/letstest/scripts/test_openssl_version.py new file mode 100644 index 000000000..c55441c5d --- /dev/null +++ b/letstest/scripts/test_openssl_version.py @@ -0,0 +1,30 @@ +#!/usr/bin/env python +# Test script for OpenSSL version checking +from distutils.version import LooseVersion +import sys + + +def main(openssl_version, apache_version): + if not openssl_version.strip(): + raise Exception("No OpenSSL version found.") + if not apache_version.strip(): + raise Exception("No Apache version found.") + conf_file_location = "/etc/letsencrypt/options-ssl-apache.conf" + with open(conf_file_location) as f: + contents = f.read() + if LooseVersion(apache_version.strip()) < LooseVersion('2.4.11') or \ + LooseVersion(openssl_version.strip()) < LooseVersion('1.0.2l'): + # should be old version + # assert SSLSessionTickets not in conf file + if "SSLSessionTickets" in contents: + raise Exception("Apache or OpenSSL version is too old, " + "but SSLSessionTickets is set.") + else: + # should be current version + # assert SSLSessionTickets in conf file + if "SSLSessionTickets" not in contents: + raise Exception("Apache and OpenSSL versions are sufficiently new, " + "but SSLSessionTickets is not set.") + +if __name__ == '__main__': + main(*sys.argv[1:]) diff --git a/letstest/scripts/test_sdists.sh b/letstest/scripts/test_sdists.sh new file mode 100755 index 000000000..562169524 --- /dev/null +++ b/letstest/scripts/test_sdists.sh @@ -0,0 +1,53 @@ +#!/bin/sh -xe + +cd letsencrypt + +BOOTSTRAP_SCRIPT="letstest/scripts/bootstrap_os_packages.sh" +VENV_PATH=venv + +# install OS packages +. $BOOTSTRAP_SCRIPT + +# setup venv +python3 -m venv $VENV_PATH +$VENV_PATH/bin/python3 tools/pipstrap.py +. "$VENV_PATH/bin/activate" +# pytest is needed to run tests on our packages so we install a pinned version here. +tools/pip_install.py pytest + +# setup constraints +TEMP_DIR=$(mktemp -d) +CONSTRAINTS="$TEMP_DIR/constraints.txt" +cp tools/requirements.txt "$CONSTRAINTS" + +# We pin cryptography to 3.1.1 and pyopenssl to 19.1.0 specifically for CentOS 7 / RHEL 7 +# because these systems ship only with OpenSSL 1.0.2, and this OpenSSL version support has been +# dropped on cryptography>=3.2 and pyopenssl>=20.0.0. +# Using this old version of OpenSSL would break the cryptography and pyopenssl wheels builds. +if [ -f /etc/redhat-release ] && [ "$(. /etc/os-release 2> /dev/null && echo "$VERSION_ID" | cut -d '.' -f1)" -eq 7 ]; then + sed -i 's|cryptography==.*|cryptography==3.1.1|g' "$CONSTRAINTS" + sed -i 's|pyopenssl==.*|pyopenssl==19.1.0|g' "$CONSTRAINTS" +fi + + +PLUGINS="certbot-apache certbot-nginx" +# build sdists +for pkg_dir in acme certbot $PLUGINS; do + cd $pkg_dir + python setup.py clean + rm -rf build dist + python setup.py sdist + mv dist/* $TEMP_DIR + cd - +done + +VERSION=$(python letstest/scripts/version.py) +# test sdists +cd $TEMP_DIR +for pkg in acme certbot $PLUGINS; do + tar -xvf "$pkg-$VERSION.tar.gz" + cd "$pkg-$VERSION" + PIP_CONSTRAINT=../constraints.txt PIP_NO_BINARY=:all: pip install . + python -m pytest + cd - +done diff --git a/letstest/scripts/version.py b/letstest/scripts/version.py new file mode 100755 index 000000000..6e538b032 --- /dev/null +++ b/letstest/scripts/version.py @@ -0,0 +1,28 @@ +#!/usr/bin/env python +"""Get the current Certbot version number. + +Provides a simple utility for determining the Certbot version number + +""" +from __future__ import print_function +from os.path import abspath, dirname, join +import re + + +def certbot_version(letstest_scripts_dir): + """Return the version number stamped in certbot/__init__.py.""" + return re.search('''^__version__ = ['"](.+)['"].*''', + file_contents(join(dirname(dirname(letstest_scripts_dir)), + 'certbot', + 'certbot', + '__init__.py')), + re.M).group(1) + + +def file_contents(path): + with open(path) as file: + return file.read() + + +if __name__ == '__main__': + print(certbot_version(dirname(abspath(__file__)))) diff --git a/letstest/setup.py b/letstest/setup.py new file mode 100644 index 000000000..a552cf920 --- /dev/null +++ b/letstest/setup.py @@ -0,0 +1,45 @@ +from setuptools import find_packages +from setuptools import setup + +setup( + name='letstest', + version='1.0', + description='Test Certbot on different AWS images', + url='https://github.com/certbot/certbot', + author='Certbot Project', + author_email='certbot-dev@eff.org', + license='Apache License 2.0', + python_requires='>=3.6', + classifiers=[ + 'Development Status :: 5 - Production/Stable', + 'Intended Audience :: Developers', + 'License :: OSI Approved :: Apache Software License', + 'Programming Language :: Python', + 'Programming Language :: Python :: 3', + 'Programming Language :: Python :: 3.6', + 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', + 'Programming Language :: Python :: 3.9', + 'Topic :: Internet :: WWW/HTTP', + 'Topic :: Security', + ], + + packages=find_packages(), + include_package_data=True, + install_requires=[ + # awscli isn't required by the tests themselves, but it is a useful + # tool to have when using these tests to generate keys and control + # running instances so the dependency is declared here for convenience. + 'awscli', + 'boto3', + 'botocore', + # The API from Fabric 2.0+ is used instead of the 1.0 API. + 'fabric>=2', + 'pyyaml', + ], + entry_points={ + 'console_scripts': [ + 'letstest=letstest.multitester:main', + ], + } +) diff --git a/letstest/targets/apache2_targets.yaml b/letstest/targets/apache2_targets.yaml new file mode 100644 index 000000000..2663782ce --- /dev/null +++ b/letstest/targets/apache2_targets.yaml @@ -0,0 +1,47 @@ +# These images are located in us-east-1. +# +# All machines must currently use x86_64 since Pebble does not currently +# publish images for other architectures. + +targets: + #----------------------------------------------------------------------------- + #Ubuntu + - ami: ami-0f2e2c076f4c2f941 + name: ubuntu20.10 + type: ubuntu + virt: hvm + user: ubuntu + - ami: ami-0758470213bdd23b1 + name: ubuntu20.04 + type: ubuntu + virt: hvm + user: ubuntu + - ami: ami-095192256fe1477ad + name: ubuntu18.04LTS + type: ubuntu + virt: hvm + user: ubuntu + - ami: ami-09677e0a6b14905b0 + name: ubuntu16.04LTS + type: ubuntu + virt: hvm + user: ubuntu + #----------------------------------------------------------------------------- + # Debian + - ami: ami-01db78123b2b99496 + name: debian10 + type: ubuntu + virt: hvm + user: admin + - ami: ami-003f19e0e687de1cd + name: debian9 + type: ubuntu + virt: hvm + user: admin + #----------------------------------------------------------------------------- + # CentOS + - ami: ami-9887c6e7 + name: centos7 + type: centos + virt: hvm + user: centos diff --git a/letstest/targets/targets.yaml b/letstest/targets/targets.yaml new file mode 100644 index 000000000..97c775f6c --- /dev/null +++ b/letstest/targets/targets.yaml @@ -0,0 +1,59 @@ +# These images are located in us-east-1. + +targets: + #----------------------------------------------------------------------------- + #Ubuntu + - ami: ami-0f2e2c076f4c2f941 + name: ubuntu20.10 + type: ubuntu + virt: hvm + user: ubuntu + - ami: ami-0758470213bdd23b1 + name: ubuntu20.04 + type: ubuntu + virt: hvm + user: ubuntu + - ami: ami-095192256fe1477ad + name: ubuntu18.04LTS + type: ubuntu + virt: hvm + user: ubuntu + #----------------------------------------------------------------------------- + # Debian + - ami: ami-01db78123b2b99496 + name: debian10 + type: ubuntu + virt: hvm + user: admin + - ami: ami-0dcd54b7d2fff584f + name: debian10_arm64 + type: ubuntu + virt: hvm + user: admin + machine_type: a1.medium + #----------------------------------------------------------------------------- + # Other Redhat Distros + - ami: ami-0916c408cb02e310b + name: RHEL7 + type: centos + virt: hvm + user: ec2-user + - ami: ami-0c322300a1dd5dc79 + name: RHEL8 + type: centos + virt: hvm + user: ec2-user + #----------------------------------------------------------------------------- + # CentOS + # These Marketplace AMIs must, irritatingly, have their terms manually + # agreed to on the AWS marketplace site for any new AWS account using them... + - ami: ami-9887c6e7 + name: centos7 + type: centos + virt: hvm + user: centos + - ami: ami-01ca03df4a6012157 + name: centos8 + type: centos + virt: hvm + user: centos |