| Age | Commit message (Collapse) | Author |
|---|
|
|
|
* Remove Windows 2016 environment, generate 64 bit installer
* Add note to changelog
* Use win_amd64 as installer suffix
* Bump PYTHON_BITNESS to 64
* Require 64 bit Windows for the installer_build job
* Update certbot install path
* update windows test name
* Base installer suffix on PYTHON_BITNESS again
* Update changelog to request users uninstall old version
|
|
* Remove deprecation warnings
* update ci
* update setup.py files
* update changelog
* update pinnings
* update requests and friends
* update setuptools pin
* update setuptools pin pt2
* update pytz
* upgrade pyparsing
* upgrade boto deps
* update deps and docs
* update pyproject.toml comment
* remove trailets pin
* remove explicit PYTHON_VERSION
|
|
Fixes https://github.com/certbot/certbot/issues/9058.
The changes to the CI config are equivalent to the ones made in https://github.com/certbot/certbot/pull/8460.
Other than ignoring some warnings raised by botocore, the main additional work that had to be done here was switching away from using `distutils.version.LooseVersion` since the entire `distutils` module was deprecated in Python 3.10. To do that, I took a few different approaches:
* If the version strings being parsed are from Python packages such as Certbot or setuptools, I switched to using [pkg_resources.parse_version](https://setuptools.pypa.io/en/latest/pkg_resources.html#parsing-utilities) from `setuptools`. This functionality has been available since [setuptools 8.0 from 2014](https://setuptools.pypa.io/en/latest/history.html#id865).
* If the version strings being parsed are not from Python packages, I added code equivalent to `distutils.version.LooseVersion` in `certbot.util.parse_loose_version`.
* The code for `CERTBOT_PIP_NO_BINARY` can be completely removed since that variable isn't used or referenced anywhere in this repo.
* add python 3.10 support
* make some version changes
* don't use looseversion in setup.py
* switch to pkg_resources
* deprecate get_strict_version
* fix route53 tests
* remove unused CERTBOT_PIP_NO_BINARY code
* stop using distutils in letstest
* add unit tests
* more changelog entries
|
|
It seems that all required pre-compiled wheels to install Certbot on Python 3.9 on Windows are present.
This PR upgrades Windows tests on Python 3.9 and repackages the installer on this version of Python.
|
|
Adds a Pynsist extra_preamble in the Certbot entry_point for the
Windows installer, ensuring the flag is always set.
|
|
conflicts (#8836)
Fixes #8824
This PR makes the installer first delete (if exist) the previous `pkg` directory in the Certbot installation in order to avoid dependencies conflicts when a new version of Certbot (with new versions of dependencies) is intaller other an existing one.
I took the simplest approach here, which is to delete specifically the directories known to create conflicts, instead of more complex approaches that involve to factor in some way the complete uninstaller logic. This is because the complexity added without a clear improvement does not worth it in my opinion. More specifically:
* factorizing in some way the uninstaller section in the NSIS template make the installer use any potential new logic of a new installation of Certbot instead of the one applying for the current installation, and may create unexpected errors during installation or at runtime
* calling the existing `uninstaller.exe` would be better, but I could not find a proper way to let NSIS wait for the actual end of the uninstall logic, and again may create unexpected errors during installation or at runtime
* Cleanup Certbot pkg dir before installing to avoid dependencies conflicts
* Add a changelog
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
|
|
client-dev@letsencrypt.org is no longer used by the Certbot team so this PR updates the email address in our packages to our current mailing list.
|
|
Over the weekend, Python released new versions of Python 3.8 and Python 3.9 partially in response to the OpenSSL CVEs discussed at https://github.com/certbot/certbot/pull/8741#issuecomment-809644789. You can see this mentioned in their changelog at https://docs.python.org/release/3.8.9/whatsnew/changelog.html#build.
This PR updates the windows installer to use that new release so all of our distribution methods that contain their own copy of OpenSSL are patched for the release tomorrow.
You can see tests passing with this change at https://dev.azure.com/certbot/certbot/_build/results?buildId=3751&view=results. You can see Python 3.8.9 being downloaded instead of an older version at https://dev.azure.com/certbot/certbot/_build/results?buildId=3751&view=logs&j=ad29f110-3cce-5317-4ef2-0a692ae1dee7&t=901eeead-396c-5477-aba2-f402fdcfb885&l=1055.
|
|
|
|
In #8649 we added some code to trick pynsist and make it understand that `abi3` wheels for Windows are forward compatible, meaning that the cryptography wheel tagged `cp36-abi3` is in fact compatible with Python 3.6+, and not only Python 3.6.
Since pynsist 2.7 the tool now understand `abi3` wheels properly, and this trick is not needed anymore.
Please note that despite modifying the pynsist pinning in `dev_constraints.txt`, it will have no effect since pynsist currently escape the pinning system. This is handled in https://github.com/certbot/certbot/pull/8749.
|
|
* Pin pynsist
* Update dependencies
* Set windows installer a proper python project
* Optimize usage of the venvs
* Add windows-installer when venv is set up
* Fix call
* Remove env marker
|
|
* add initial pyproject.toml
* add extra dependencies
* add simple bash script
* polish
* reuse pipstrap
* add requirements.txt
* temporarily remove hashin dep
* Switch to requirements.txt
* remove hashin check
* update requirements.txt again
* remove unnecessary merge
* pin back augeas
* unpin cryptography
* simplify pywin32 pinning
* update comment
* pin back pytest and pylint
* pin back pytest-forked
* pin back coverage
* update script comments
* fix pyopenssl case
* add minimum poetry version
* run pin.sh
|
|
cryptography (#8649)
* Refactor to not depend on certbot-auto dependencies pinning anymore
* Update constraints
* Replaces references
* Upgrade AWS dependencies pinning
* Fix script
* Fix Windows installer builds
* Fixing sdists letstest script
* Pin cryptography on 3.1.1 specifically for RHEL/CentOS 7 to avoid build failures during test_sdists test.
* Finish fix
* Fix VERSION_ID in RHEL 7
|
|
Fixes https://github.com/certbot/certbot/issues/8681. https://python-security.readthedocs.io/vuln/ctypes-buffer-overflow-pycarg_repr.html is the best resource I found linking to the original Python bug, when each Python branch was fixed, etc.
|
|
Now that we have a new pipstrap script with recent version of pip, dependencies for Windows can be resolved correctly on Python 3.8.
This PR enables tests on Python 3.8, and package Certbot for Windows on Python 3.8 also. I do not move up to Python 3.9 since some dependencies (`cryptography`, `pynacl`) do not provide wheels for Python 3.9 yet on Windows, which would require a complete C++ build system to compile them.
* Enable windows tests on Python 3.8 and package it on Python 3.8 also.
* Upgrade pynsist, nsis and pywin32, remove old workarounds
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
|
|
Fixes #8256
First let's sum up the problem to solve. We disabled the build isolation available in pip>=19 because it could potential break certbot build without a control on our side. Basically builds are not reproductible. Indeed the build isolation triggers build of PEP-517 enabled transitive dependencies (like `cryptography`) with the build dependencies defined in their `pyproject.toml`. For `cryptography` in particular these requirements include `setuptools>=40.6.0`, and quite logically pip will install the latest version of `setuptools` for the build. And when `setuptools` broke with the version 50, our build did the same.
But disabling the build isolation is not a long term solution, as more and more project will migrate on this approach and it basically provides a lot of benefit in how dependencies are built.
The ideal solution would be to be able to apply version constraints on our side on the build dependencies, in order to pin `setuptools` for instance, and decide precisely when we upgrade to a newer version. However for now pip does not provide a mechanism for that (like a `--build-constraint` flag or propagation of existing `--constraint` flag).
Until I saw https://github.com/pypa/pip/issues/9081 and https://github.com/pypa/pip/issues/8439.
Apart the fact that https://github.com/pypa/pip/issues/9081 shows that pip maintainers are working on this issue, it explains how pip works regarding PEP-517 and infers which workaround can be used to still pin the build dependencies. It turns out that pip invokes itself in each build isolation to install the build dependencies. It means that even if some flags (like `--constraint`) are not explicitly passed to the pip sub call, the global environment remains, in particular the environment variables.
Thus it is known that every pip flag can alternatively be set by environment variable using the following pattern for the variable name: `PIP_[FLAG_NAME_UPPERCASE]`. So for `--constraint`, it is `PIP_CONSTRAINT`. And so you can pass a constraint file to the pip sub call through that mechanism.
I made some tests with a constraint file containing pinning for `setuptools`: indeed under isolation zone, the constraint file has been honored and the provided pinned version has been used to build the dependencies (I tested it with `cryptography`).
Finally this PR takes advantage of this mechanism, by setting `PIP_CONSTRAINT` to `pip_install`, the snap building process, the Dockerfiles and the windows installer building process.
I also extracted out the requirements of the new `pipstrap.py` to be reusable in these various build processes.
* Use workaround to fix build requirements in build isolation, and renable build isolation
* Clean imports in pipstrap
* Externalize pipstrap reqs to be reusable
* Inject pipstrap constraints during pip_install
* Update docker build
* Update snapcraft build
* Prepare installer build
* Fix pipstrap constraints in snap build
* Add back --no-build-cache option in Docker images build
* Update snap/snapcraft.yaml
* Use proper flags with pip
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
|
|
* Add a new, simplified version of pipstrap.
* Use tools/pipstrap.py
* Uncomment code
* Refactor pip_install.py and provide hashes.
* Fix test_sdists.sh.
* Make code work on Python 2.
* Call strip_hashes.py using Python 3.
* Pin the oldest version of httplib2 used in distros
* Strip enum34 dependency.
* Remove pip pinnings from dev_constraints.txt
* Correct pipstrap docstring.
* Don't set working_dir twice.
* Add comments
|
|
|
|
* Improve github release creation process
* Comment file
* Update tools/create_github_release.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
* run chmod +x on tools/create_github_release.py
* Add description of create github release method
* remove references to unnecessary azure credential
* remove unnecessary import
* Add reminders to update other file to definitions in .azure-pipelines
* Raise an error if we fail to fetch the artifact from azure
* Create github release as a draft, upload artifacts, then un-draft, for hooks to be run at the right point
* get the version number from the release
* add new packages to dev3_extras so they're installed by tools/venv3.py
* remove unnecessary import
* fun fact: tempdirs behave differently when used as a context manager
* Move comment to construct.py
Co-authored-by: Brad Warren <bmw@users.noreply.github.com>
|
|
* Fix dangerous default argument
* Remove unused imports
* Remove unnecessary comprehension
* Use literal syntax to create data structure
* Use literal syntax instead of function calls to create data structure
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
|
|
* Add --allow-downgrade to chocolatey command.
* Upgrade tests to use Python 3.8.1.
|
|
* Isort execution
* Fix pylint, adapt coverage
* New isort
* Fix magic_typing lint
* Second round
* Fix pylint
* Third round. Store isort configuration
* Fix latest mistakes
* Other fixes
* Add newline
* Fix lint errors
|
|
Current version of pywin32 used in certbot (225) does not have wheels available for Python 3.8. Installing certbot for development in this case requires to build from source. On Windows, this implies a Visual Studio C++ environment up and ready, which is absolutely not fun.
Let's upgrade to pywin32 227, that provides these wheels for all Python versions from 3.5 up to current dev status of 3.9.
|
|
Clean up some places missed by #7544.
Found this when running test farm tests. They were working as of 5d90544, and I will truly shocked if subsequent changes (all to the windows installer) made them stop working.
* Release script needs to target new CHANGELOG location
* Clean up various other CHANGELOG path references
* Update windows paths for new certbot location
* Add certbot to packages list for windows installer
|
|
packages (#7544)
Summary of changes in this PR:
- Refactor files involved in the `certbot` module to be of a similar structure to every other package; that is, inside a directory inside the main repo root (see below).
- Make repo root README symlink to `certbot` README.
- Pull tests outside of the distributed module.
- Make `certbot/tests` not be a module so that `certbot` isn't added to Python's path for module discovery.
- Remove `--pyargs` from test calls, and make sure to call tests from repo root since without `--pyargs`, `pytest` takes directory names rather than package names as arguments.
- Replace mentions of `.` with `certbot` when referring to packages to install, usually editably.
- Clean up some unused code around executing tests in a different directory.
- Create public shim around main and make that the entry point.
New directory structure summary:
```
repo root ("certbot", probably, but for clarity all files I mention are relative to here)
├── certbot
│ ├── setup.py
│ ├── certbot
│ │ ├── __init__.py
│ │ ├── achallenges.py
│ │ ├── _internal
│ │ │ ├── __init__.py
│ │ │ ├── account.py
│ │ │ ├── ...
│ │ ├── ...
│ ├── tests
│ │ ├── account_test.py
│ │ ├── display
│ │ │ ├── __init__.py
│ │ │ ├── ...
│ │ ├── ... # note no __init__.py at this level
│ ├── ...
├── acme
│ ├── ...
├── certbot-apache
│ ├── ...
├── ...
```
* refactor certbot/ and certbot/tests/ to use the same structure as the other packages
* git grep -lE "\-e(\s+)\." | xargs sed -i -E "s/\-e(\s+)\./-e certbot/g"
* git grep -lE "\.\[dev\]" | xargs sed -i -E "s/\.\[dev\]/certbot[dev]/g"
* git grep -lE "\.\[dev3\]" | xargs sed -i -E "s/\.\[dev3\]/certbot[dev3]/g"
* Remove replacement of certbot into . in install_and_test.py
* copy license back out to main folder
* remove linter_plugin.py and CONTRIBUTING.md from certbot/MANIFEST.in because these files are not under certbot/
* Move README back into main folder, and make the version inside certbot/ a symlink
* symlink certbot READMEs the other way around
* move testdata into the public api certbot zone
* update source_paths in tox.ini to certbot/certbot to find the right subfolder for tests
* certbot version has been bumped down a directory level
* make certbot tests directory not a package and import sibling as module
* Remove unused script cruft
* change . to certbot in test_sdists
* remove outdated comment referencing a command that doesn't work
* Install instructions should reference an existing file
* update file paths in Dockerfile
* some package named in tox.ini were manually specified, change those to certbot
* new directory format doesn't work easily with pyargs according to http://doc.pytest.org/en/latest/goodpractices.html#tests-as-part-of-application-code
* remove other instance of pyargs
* fix up some references in _release.sh by searching for ' . ' and manual check
* another stray . in tox.ini
* fix paths in tools/_release.sh
* Remove final --pyargs call, and now-unnecessary call to modules instead of local files, since that's fixed by certbot's code being one layer deeper
* Create public shim around main and make that the entry point
* without pyargs, tests cannot be run from an empty directory
* Remove cruft for running certbot directly from main
* Have main shim take real arg
* add docs/api file for main, and fix up main comment
* Update certbot/docs/install.rst
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Fix comments in readthedocs requirements files to refer to current package
* Update .[docs] reference in contributing.rst
* Move plugins tests to certbot tests directory
* add certbot tests to MANIFEST.in so packagers can run python setup.py test
* move examples directory inside certbot/
* Move CHANGELOG into certbot, and create a top-level symlink
* Remove unused sys and logging from main shim
* nginx http01 test no longer relies on certbot plugins common test
|
|
renew task (#7537)
While coding for #7536, I ran into another issue. It appears that Certbot logs generated during the scheduled task execution have wrong permissions that make them almost unusable: they do not have an owner, and their ACL contains nonsense values (non existant accounts name).
The class `logging.handler.RotatingFileHandler` is responsible for these logs, and become mad when it is in a Python process run under a scheduled task owned by `SYSTEM`. This is precisely our case here.
This PR avoids (but not fix) the issue, by changing the owner of the scheduled task from `SYSTEM` to the `Administrators` group, that appears to work fine.
* Use Administrators group instead of SYSTEM to run the certbot renew task
|
|
* Create _internal package for Certbot's non-public modules
* Move account.py to _internal
* Move auth_handler.py to _internal
* Move cert_manager.py to _internal
* Move client.py to _internal
* Move error_handler.py to _internal
* Move lock.py to _internal
* Move main.py to _internal
* Move notify.py to _internal
* Move ocsp.py to _internal
* Move renewal.py to _internal
* Move reporter.py to _internal
* Move storage.py to _internal
* Move updater.py to _internal
* update apache and nginx oldest requirements
* Keep the lock file as certbot.lock
* nginx oldest tests still need to rely on newer certbot
* python doesn't have good dependency resolution, so specify the transitive dependency
* update required minimum versions in nginx setup.py
|
|
This PR uses pipstrap to bootstrap the venv used to build Windows installers. This effectively pin all build dependencies, since pynsist is already installed through pip_install.py script.
* Use pipstrap
* Pin also NSIS version
|
|
* Consume constraints file
* Independent pywin32 dependency definition in setup.py and construct.py
|
|
This PR creates a pipeline triggered on tag push matching v0.* (eg. v0.40.0).
Once triggered, this pipeline will build the windows installer, and run integration tests on it, like for the pipeline run nightly.
I also add a simple script to extract from CHANGELOG.md file to extract the relevant part to put it in the body of the GitHub release. I believe it makes things nicer.
* Create release pipeline
* Relax condition on tags
* Put beta keyword
* Update job name
* Fix release pipeline
|
|
This PR defines pipelines that can be run on Azure Pipelines. Currently there are two:
* `.azure-pipelines/main.yml` is the main one, executed on PRs for master, and pushes to master,
* `.azure-pipelines/advanced.yml` add installer testing on top of the main pipeline, and is executed for `test-*` branches, release branches, and nightly run for master.
These two pipelines covers all existing stuff done by AppVeyor currently, and so AppVeyor can be decommissioned once Azure Pipelines is operational.
You can see working pipeline in my fork:
* a PR for `master` (so using main pipeline): https://github.com/adferrand/certbot/pull/65
* a PR for `test-something` (so using advanced pipeline): https://github.com/adferrand/certbot/pull/66
* uploaded coverage from Azure Pipelines: https://codecov.io/gh/adferrand/certbot/commit/499aa2cbf25e1e0ab4c93ab64057db92dfec0fba/build
Once this PR is merged, we need to enable Azure Pipelines for Certbot. Instructions are written in `azure-pipelines/INSTALL.md`. This document also references all access rights required to Azure Pipelines onto GitHub to make the CI process work.
Future work for future PRs:
* create a CD pipeline for the releases that will push the installer to GitHub releases
* implement a solution to generate notification on IRC or Mattermost when a nightly build fails
* Define pipelines
* Update locations
* Update nightly
* Use x86
* Update nightly.yml for Azure Pipelines
* Run script
* Use script
* Update install
* Use local installation
* Register warnings
* Fix pywin32 loading
* Clean context
* Enable coverage publication
* Consume codecov token
* Document installation
* Update tool to upload coverage
* Prepare pipeline artifacts
* Update artifact ignore
* Protect against codecov failures
* Add a comment about codecov
* Add a comment on RW access asked by Azure
* Add instructions
* Rename pipeline file
* Update instructions
* Update .azure-pipelines/templates/tests-suite.yml
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update .azure-pipelines/INSTALL.md
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Modified scheduled pipeline
* Add comment
* Remove dynamic version-based installer name
|
|
This PR implements the item "register a scheduled task for certificate renewal" from the list of requirements described in #7365.
This PR adds required instructions in the NSIS installer for Certbot to create a task, named "Certbot Renew Task" in the Windows Scheduler. This task is run twice a day, to execute the command certbot renew and keep the certificates up-to-date.
Uninstalling Certbot will also remove this scheduled task.
* Implementation
* Corrections
* Update template.nsi
* Improve scripts
* Add a random delay of 12 hours
* Synchronize template against default one in pynsist 2.4
* Clean config of scheduled task
* Install only in AllUsers mode
* Add comments
* Remove the logic of single user install
|
|
This PR is the first step to create an official distribution channel of Certbot for Windows. It consists essentially in creating a proper Certbot Windows installer.
Usually distributing an application requires, in a way or another, to stabilize the application logic and its dependencies around a given version. On Windows, this usually takes the form of a freezed application, that vendors its dependencies into a single executable.
There are two well-known solutions to create an executable shipping a Python application on Windows: [py2exe](http://www.py2exe.org/) and [pyinstaller](https://www.pyinstaller.org/). However these solutions create self-executable `.EXE` files: you run the `.EXE` file that launches immediately the software.
This is not a end-user solution. Indeed when a Windows user wants to install a piece of software, he expects to find and download an installer. When run the installer would interface with Windows to setup configuration entries in the Registry, update the environment variable, add shortcuts in the Start Menu, and declare a uninstaller entry into the Uninstaller Manager. Quite similarly, this is what you would get from a `.deb` or `.rpm` package.
A solution that builds proper installers is [pynsis](https://pynsist.readthedocs.io/en/latest/). It is a Python project that constructs installers for Python software using [NSIS](https://sourceforge.net/projects/nsis/), the most known free Windows installer builder solution.
This PR uses pynsist to build a Windows installer. The Python script to launch the installer build is `.\windows-installer\construct.py`. Once finished, the installer is located in `.\windows-installer\build\nsis`.
This installer will do the following operations during the installation:
* copy in the install path a full python distribution used exclusively for Certbot
* copy all Python requirements gathered from the `setup.py` of relevant certbot projects
* copy `certbot` and `acme`
* pre-build python binary assets
* register the existence of the application correctly in Windows Registry
* prepare a procedure to uninstall Certbot
* and of course, expose `certbot` executable to the Windows command line, like on Linux, to be able to launch it as any CLI application from Batch or Powershell
This installer support updates: downloading a new version of it and running it on a Windows with existing installation of Certbot will replace it with the new version.
Future capabilities not included in this PR:
* auto-update of Certbot when a new release is available
* online documentation for Windows
* register a scheduled task for certificate renewal
* installer distribution (continuous deployment + distribution channels)
* method to check the downloaded installer is untampered
* Setup config
* Fix shortcut
* Various improvments
* Update windows-installer/construct.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Split into several method
* Change installer name
* Remove DNS plugins for now
* Add a comment about administrator privileges
* Update welcome
* Control python version
* Control bitness
* Update windows-installer/construct.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update windows-installer/construct.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
* Update windows-installer/construct.py
Co-Authored-By: Brad Warren <bmw@users.noreply.github.com>
|
