Move compatibility tests off of certbot-auto and Python 2 (#8248)

Fixes https://github.com/certbot/certbot/issues/8162.

I had to update the base of the Dockerfile to get a new enough version of Python 3. I also simplified things a lot and removed a lot of the comments that were essentially just describing how Dockerfiles work.

The most complicated changes here are in `testdata`. You can find a diff of the changes to `nginx.tar.gz` at https://gist.github.com/c7727db0cecf3f15f02439f085c73848.

The first problem was that there were some complaints from the new Apache/nginx/OpenSSL version about the 1024 bit RSA key so I updated `empty_cert.pem` both inside and outside of the tarball as well as the corresponding private key in the tarball to use a 2048 bit key.

The 2nd problem is trickier to understand. If you look at the output from nginx after loading the config from `lots/` you'll see it complaining about conflicting `server_name` directives for the directives I deleted. See https://dev.azure.com/certbot/certbot/_build/results?buildId=2578&view=logs&j=250aa146-b243-5f8f-bf86-17a529c9fb7e&t=9baa2014-9673-5e78-8f4f-7a463caf2bfa&l=1516.

After switching the tests to Python 3, tests on that domain started failing. What I believe to be happening is we were just lucky these tests were passing to begin with. In both the Apache and Nginx plugin, if there are conflicting virtual hosts like this, we just arbitrarily pick one. The relevant code here for nginx is https://github.com/certbot/certbot/blob/575092d6030330ed8379babaa4cbbfe43e7bf721/certbot-nginx/certbot_nginx/_internal/configurator.py#L455

I played around with a debugger and confirmed that before I removed the conflicting server names, there were two exact matches for the domain we were searching for here.

I think all that's going on is with the switch to Python 3, the vhost we happen to choose changes and "breaks" the test. I suspect this to be due to something like getting values out of a dict somewhere where the order of items in a dict while iterating over it is different between Python 2 and 3. I didn't track where this difference happens down, but I personally don't think it's a good use of time since I think the real problem here is that the nginx config being tested was invalid with conflicting `server` blocks.

I removed all references to the `server_name` causing conflicts in that nginx configuration because both server blocks had other domains that are being tested, but I could add either back if you prefer. You can see the `nginx_compat` test passing with these changes at https://dev.azure.com/certbot/certbot/_build/results?buildId=2587&view=logs&j=250aa146-b243-5f8f-bf86-17a529c9fb7e.

* update Dockerfile

* Fix apache_compat on py3.

* Update empty_cert.pem.

The command used here was `openssl req -key
certbot/certbot/tests/testdata/rsa2048_key.pem -new -subj '/CN=example.com'
-x509 >
certbot-compatibility-test/certbot_compatibility_test/testdata/empty_cert.pem`.

* update nginx.tar.gz

* Remove conflicting server_names

4 files changed, 30 insertions, 53 deletions

diff --git a/certbot-compatibility-test/Dockerfile b/certbot-compatibility-test/Dockerfile
index a6a0c93db..f66e4c945 100644
--- a/certbot-compatibility-test/Dockerfile
+++ b/certbot-compatibility-test/Dockerfile
@@ -1,47 +1,18 @@
-FROM debian:stretch
+FROM debian:buster
 MAINTAINER Brad Warren <bmw@eff.org>
 
-# no need to mkdir anything:
-# https://docs.docker.com/reference/builder/#copy
-# If <dest> doesn't exist, it is created along with all missing
-# directories in its path.
+RUN apt-get update && \
+    apt install python3-dev python3-venv gcc libaugeas0 libssl-dev \
+                 libffi-dev ca-certificates openssl -y
 
-# TODO: Install non-default Python versions for tox.
-# TODO: Install Apache/Nginx for plugin development.
-COPY letsencrypt-auto-source /opt/certbot/src/letsencrypt-auto-source
-RUN /opt/certbot/src/letsencrypt-auto-source/letsencrypt-auto --os-packages-only
+WORKDIR /opt/certbot/src
 
-# the above is not likely to change, so by putting it further up the
-# Dockerfile we make sure we cache as much as possible
+# We copy all contents of the build directory to allow us to easily use
+# things like tools/venv3.py which expects all of our packages to be available.
+COPY . .
 
-COPY certbot/setup.py certbot/README.rst certbot/CHANGELOG.md certbot/MANIFEST.in linter_plugin.py tox.cover.py tox.ini .pylintrc /opt/certbot/src/
-
-# all above files are necessary for setup.py, however, package source
-# code directory has to be copied separately to a subdirectory...
-# https://docs.docker.com/reference/builder/#copy: "If <src> is a
-# directory, the entire contents of the directory are copied,
-# including filesystem metadata. Note: The directory itself is not
-# copied, just its contents." Order again matters, three files are far
-# more likely to be cached than the whole project directory
-
-COPY certbot /opt/certbot/src/certbot/
-COPY acme /opt/certbot/src/acme/
-COPY certbot-apache /opt/certbot/src/certbot-apache/
-COPY certbot-nginx /opt/certbot/src/certbot-nginx/
-COPY certbot-compatibility-test /opt/certbot/src/certbot-compatibility-test/
-COPY tools /opt/certbot/src/tools
-
-RUN VIRTUALENV_NO_DOWNLOAD=1 virtualenv -p python2 /opt/certbot/venv && \
-    /opt/certbot/venv/bin/pip install -U setuptools && \
-    /opt/certbot/venv/bin/pip install -U pip
-ENV PATH /opt/certbot/venv/bin:$PATH
-RUN /opt/certbot/venv/bin/python \
-    /opt/certbot/src/tools/pip_install_editable.py \
-    /opt/certbot/src/acme \
-    /opt/certbot/src/certbot \
-    /opt/certbot/src/certbot-apache \
-    /opt/certbot/src/certbot-nginx \
-    /opt/certbot/src/certbot-compatibility-test
+RUN tools/venv3.py
+ENV PATH /opt/certbot/src/venv3/bin:$PATH
 
 # install in editable mode (-e) to save space: it's not possible to
 # "rm -rf /opt/certbot/src" (it's stays in the underlaying image);
diff --git a/certbot-compatibility-test/certbot_compatibility_test/test_driver.py b/certbot-compatibility-test/certbot_compatibility_test/test_driver.py
index 5140dc8ea..f11b9fdf8 100644
--- a/certbot-compatibility-test/certbot_compatibility_test/test_driver.py
+++ b/certbot-compatibility-test/certbot_compatibility_test/test_driver.py
@@ -102,8 +102,10 @@ def _create_achalls(plugin):
         prefs = plugin.get_chall_pref(domain)
         for chall_type in prefs:
             if chall_type == challenges.HTTP01:
+                # challenges.HTTP01.TOKEN_SIZE is a float but os.urandom
+                # expects an integer.
                 chall = challenges.HTTP01(
-                    token=os.urandom(challenges.HTTP01.TOKEN_SIZE))
+                    token=os.urandom(int(challenges.HTTP01.TOKEN_SIZE)))
                 challb = acme_util.chall_to_challb(
                     chall, messages.STATUS_PENDING)
                 achall = achallenges.KeyAuthorizationAnnotatedChallenge(
@@ -137,7 +139,7 @@ def test_deploy_cert(plugin, temp_dir, domains):
     """Tests deploy_cert returning True if the tests are successful"""
     cert = crypto_util.gen_ss_cert(util.KEY, domains)
     cert_path = os.path.join(temp_dir, "cert.pem")
-    with open(cert_path, "w") as f:
+    with open(cert_path, "wb") as f:
         f.write(OpenSSL.crypto.dump_certificate(
             OpenSSL.crypto.FILETYPE_PEM, cert))
 
@@ -273,7 +275,7 @@ def _dirs_are_unequal(dir1, dir2):
             logger.error(str(dircmp.diff_files))
             return True
 
-        for subdir in dircmp.subdirs.itervalues():
+        for subdir in dircmp.subdirs.values():
             dircmps.append(subdir)
 
     return False
diff --git a/certbot-compatibility-test/certbot_compatibility_test/testdata/empty_cert.pem b/certbot-compatibility-test/certbot_compatibility_test/testdata/empty_cert.pem
index 4ea812a87..9404a4437 100644
--- a/certbot-compatibility-test/certbot_compatibility_test/testdata/empty_cert.pem
+++ b/certbot-compatibility-test/certbot_compatibility_test/testdata/empty_cert.pem
@@ -1,13 +1,17 @@
 -----BEGIN CERTIFICATE-----
-MIICATCCAWoCCQCvMbKu4FHZ6zANBgkqhkiG9w0BAQsFADBFMQswCQYDVQQGEwJB
-VTETMBEGA1UECAwKU29tZS1TdGF0ZTEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0
-cyBQdHkgTHRkMB4XDTE1MDcyMzIzMjc1MFoXDTE2MDcyMjIzMjc1MFowRTELMAkG
-A1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGEludGVybmV0
-IFdpZGdpdHMgUHR5IEx0ZDCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAws3o
-y46PMLM9Gr68pbex0MhdPr7Cq4rRe9BBpnOuHFdF35Ak0aPrzFwVzLlGOir94U11
-e5JYJDWJi+4FwLBRkOAfanjJ5GJ9BnEHSOdbtO+sv9uhbt+7iYOOUOngKSiJyUrM
-i1THAE+B1CenxZ1KHRQCke708zkK8jVuxLeIAOMCAwEAATANBgkqhkiG9w0BAQsF
-AAOBgQCC3LUP3MHk+IBmwHHZAZCX+6p4lop9SP6y6rDpWgnqEEeb9oFleHi2Rvzq
-7gxl6nS5AsaSzfAygJ3zWKTwVAZyU4GOQ8QTK+nHk3+LO1X4cDbUlQfm5+YuwKDa
-4LFKeovmrK6BiMLIc1J+MxUjLfCeVHYSdkZULTVXue0zif0BUA==
+MIICqDCCAZACCQCRC1UKg2WfRTANBgkqhkiG9w0BAQsFADAWMRQwEgYDVQQDDAtl
+eGFtcGxlLmNvbTAeFw0yMDA4MTkyMzM5MjdaFw0yMDA5MTgyMzM5MjdaMBYxFDAS
+BgNVBAMMC2V4YW1wbGUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
+AQEA5tViHnJx4y+BbCb8Qz9uxsnqp1ynONR7ET/XL+M/jQ4xPeJg4L2uZ3YnogPc
+WdEoey17WXBg3KRqKfg+7PqIdGqVeonSCfXhD1HoGJRsThSUJ2fK3uoQ+zGgJTWR
+FYWa8Cb6xsuq0xaYtw2jaJBp+697Np60PWs4pY5FkadT50wZ0TYDnYt3NSAdn+Pt
+j3cpI4ocZZ2FLiOFn+UFOaRcetGtpnU1QwvmygD9tiL7kJ55B4CWGEv6DMRQk/UE
+eMUETzse1NkVlaxQ1TCd5iAfBTluiV30EpmmWa+OsXJWxCK+EEOkXD1r3CdXAldY
+nRYxJrn4udrFe69QX95wiRZNXwIDAQABMA0GCSqGSIb3DQEBCwUAA4IBAQCJvtDC
+875CK7SKNf006gSciXsNPNSVORGPjc/5OQ23baK4iPhxftI4LGZN8773N14jWp3E
+QnQLL1gZ9/G+98SlI5lm97a4m4XZyNaULbmQwRKgI22H0F1AWbvsG0SppjnhVlJ+
+93ZUqSQBXgbXelFHSsNfk1AB6Kvo6+UvS8s0vkz7SfkPOZGx0b+3RJSJZnZHvYih
+ggudN/jJggSgRrb+F6lpaelJE9pZsznJFb9R7mFI33AGBpQWV4r3p1ZbM1vGMqGc
+4PGBzDzi28BhLBplSOPZZxqRiINQzGiQ5T2SfN06usr7EafFr6+7YKNhgrCdlVjU
+thzJ5MgHZgALNXsh
 -----END CERTIFICATE-----
diff --git a/certbot-compatibility-test/certbot_compatibility_test/testdata/nginx.tar.gz b/certbot-compatibility-test/certbot_compatibility_test/testdata/nginx.tar.gz
index 2f06add17..4234ddf73 100644
--- a/certbot-compatibility-test/certbot_compatibility_test/testdata/nginx.tar.gz
+++ b/certbot-compatibility-test/certbot_compatibility_test/testdata/nginx.tar.gz