diff options
| author | Adam Warner <me@adamwarner.co.uk> | 2020-11-28 22:05:01 +0300 |
|---|---|---|
| committer | GitHub <noreply@github.com> | 2020-11-28 22:05:01 +0300 |
| commit | fee1b8b736a402c8b72a68d6f2449e9061477dee (patch) | |
| tree | a627be452b66b2683369bf8483f6a52396ac8f71 | |
| parent | 6b536b7428a1f57ff34ddc444ded6d3a62b00a38 (diff) | |
| parent | f13e0dacde6f7cc0e52fb5b8075444977c1de239 (diff) | |
Merge pull request #3867 from pi-hole/release/v5.2v5.2
Pi-hole Core v5.2
32 files changed, 839 insertions, 263 deletions
diff --git a/.stickler.yml b/.stickler.yml index ab98025e..8a2a1ce9 100644 --- a/.stickler.yml +++ b/.stickler.yml @@ -3,3 +3,4 @@ linters: shell: bash phpcs: flake8: + max-line-length: 120 diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 6992cce5..0dd22b42 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,8 +4,8 @@ Please read and understand the contribution guide before creating an issue or pu ## Etiquette -- Our goal for Pi-hole is **stability before features**. This means we focus on squashing critical bugs before adding new features. Often, we can do both in tandem, but bugs will take priority over a new feature. -- Pi-hole is open source and [powered by donations](https://pi-hole.net/donate/), and as such, we give our **free time** to build, maintain, and **provide user support** for this project. It would be extremely unfair for us to suffer abuse or anger for our hard work, so please take a moment to consider that. +- Our goal for Pi-hole is **stability before features**. This means we focus on squashing critical bugs before adding new features. Often, we can do both in tandem, but bugs will take priority over a new feature. +- Pi-hole is open source and [powered by donations](https://pi-hole.net/donate/), and as such, we give our **free time** to build, maintain, and **provide user support** for this project. It would be extremely unfair for us to suffer abuse or anger for our hard work, so please take a moment to consider that. - Please be considerate towards the developers and other users when raising issues or presenting pull requests. - Respect our decision(s), and do not be upset or abusive if your submission is not used. @@ -26,7 +26,7 @@ When requesting or submitting new features, first consider whether it might be u - Check the codebase to ensure that your feature doesn't already exist. - Check the pull requests to ensure that another person hasn't already submitted the feature or fix. -- Read and understand the [DCO guidelines](https://github.com/pi-hole/pi-hole/wiki/Contributing-to-the-project) for the project. +- Read and understand the [DCO guidelines](https://docs.pi-hole.net/guides/github/contributing/) for the project. ## Technical Requirements @@ -36,3 +36,77 @@ When requesting or submitting new features, first consider whether it might be u - Commit Unix line endings. - Please use the Pi-hole brand: **Pi-hole** (Take a special look at the capitalized 'P' and a low 'h' with a hyphen) - (Optional fun) keep to the theme of Star Trek/black holes/gravity. + +## Forking and Cloning from GitHub to GitHub + +1. Fork <https://github.com/pi-hole/pi-hole/> to a repo under a namespace you control, or have permission to use, for example: `https://github.com/<your_namespace>/<your_repo_name>/`. You can do this from the github.com website. +2. Clone `https://github.com/<your_namespace>/<your_repo_name>/` with the tool of you choice. +3. To keep your fork in sync with our repo, add an upstream remote for pi-hole/pi-hole to your repo. + + ```bash + git remote add upstream https://github.com/pi-hole/pi-hole.git + ``` + +4. Checkout the `development` branch from your fork `https://github.com/<your_namespace>/<your_repo_name>/`. +5. Create a topic/branch, based on the `development` branch code. *Bonus fun to keep to the theme of Star Trek/black holes/gravity.* +6. Make your changes and commit to your topic branch in your repo. +7. Rebase your commits and squash any insignificant commits. See the notes below for an example. +8. Merge `development` your branch and fix any conflicts. +9. Open a Pull Request to merge your topic branch into our repo's `development` branch. + +- Keep in mind the technical requirements from above. + +## Forking and Cloning from GitHub to other code hosting sites + +- Forking is a GitHub concept and cannot be done from GitHub to other git-based code hosting sites. However, those sites may be able to mirror a GitHub repo. + +1. To contribute from another code hosting site, you must first complete the steps above to fork our repo to a GitHub namespace you have permission to use, for example: `https://github.com/<your_namespace>/<your_repo_name>/`. +2. Create a repo in your code hosting site, for example: `https://gitlab.com/<your_namespace>/<your_repo_name>/` +3. Follow the instructions from your code hosting site to create a mirror between `https://github.com/<your_namespace>/<your_repo_name>/` and `https://gitlab.com/<your_namespace>/<your_repo_name>/`. +4. When you are ready to create a Pull Request (PR), follow the steps `(starting at step #6)` from [Forking and Cloning from GitHub to GitHub](#forking-and-cloning-from-github-to-github) and create the PR from `https://github.com/<your_namespace>/<your_repo_name>/`. + +## Notes for squashing commits with rebase + +- To rebase your commits and squash previous commits, you can use: + + ```bash + git rebase -i your_topic_branch~(number of commits to combine) + ``` + +- For more details visit [gitready.com](http://gitready.com/advanced/2009/02/10/squashing-commits-with-rebase.html) + +1. The following would combine the last four commits in the branch `mytopic`. + + ```bash + git rebase -i mytopic~4 + ``` + +2. An editor window opens with the most recent commits indicated: (edit the commands to the left of the commit ID) + + ```gitattributes + pick 9dff55b2 existing commit comments + squash ebb1a730 existing commit comments + squash 07cc5b50 existing commit comments + reword 9dff55b2 existing commit comments + ``` + +3. Save and close the editor. The next editor window opens: (edit the new commit message). *If you select reword for a commit, an additional editor window will open for you to edit the comment.* + + ```bash + new commit comments + Signed-off-by: yourname <your email address> + ``` + +4. Save and close the editor for the rebase process to execute. The terminal output should say something like the following: + + ```bash + Successfully rebased and updated refs/heads/mytopic. + ``` + +5. Once you have a successful rebase, and before you sync your local clone, you have to force push origin to update your repo: + + ```bash + git push -f origin + ``` + +6. Continue on from step #7 from [Forking and Cloning from GitHub to GitHub](#forking-and-cloning-from-github-to-github) @@ -13,7 +13,7 @@ The Pi-hole® is a [DNS sinkhole](https://en.wikipedia.org/wiki/DNS_Sinkhole) that protects your devices from unwanted content, without installing any client-side software. -- **Easy-to-install**: our versatile installer walks you through the process, and [takes less than ten minutes](https://www.youtube.com/watch?v=vKWjx1AQYgs) +- **Easy-to-install**: our versatile installer walks you through the process, and takes less than ten minutes - **Resolute**: content is blocked in _non-browser locations_, such as ad-laden mobile apps and smart TVs - **Responsive**: seamlessly speeds up the feel of everyday browsing by caching DNS queries - **Lightweight**: runs smoothly with [minimal hardware and software requirements](https://docs.pi-hole.net/main/prerequisites/) @@ -162,4 +162,4 @@ Some of the statistics you can integrate include: - Queries cached - Unique clients -The API can be accessed via [`telnet`](https://github.com/pi-hole/FTL), the Web (`admin/api.php`) and Command Line (`pihole -c -j`). You can out find [more details over here](https://discourse.pi-hole.net/t/pi-hole-api/1863). +The API can be accessed via [`telnet`](https://github.com/pi-hole/FTL), the Web (`admin/api.php`) and Command Line (`pihole -c -j`). You can find out [more details over here](https://discourse.pi-hole.net/t/pi-hole-api/1863). diff --git a/advanced/01-pihole.conf b/advanced/01-pihole.conf index 2c8b3749..e243e91a 100644 --- a/advanced/01-pihole.conf +++ b/advanced/01-pihole.conf @@ -34,7 +34,7 @@ server=@DNS2@ interface=@INT@ -cache-size=10000 +cache-size=@CACHE_SIZE@ log-queries log-facility=/var/log/pihole.log diff --git a/advanced/Scripts/chronometer.sh b/advanced/Scripts/chronometer.sh index 98f43c3f..37fd5177 100755 --- a/advanced/Scripts/chronometer.sh +++ b/advanced/Scripts/chronometer.sh @@ -236,7 +236,7 @@ get_sys_stats() { sys_name=$(hostname) - [[ -n "$TEMPERATUREUNIT" ]] && temp_unit="$TEMPERATUREUNIT" || temp_unit="c" + [[ -n "$TEMPERATUREUNIT" ]] && temp_unit="${TEMPERATUREUNIT^^}" || temp_unit="C" # Get storage stats for partition mounted on / read -r -a disk_raw <<< "$(df -B1 / 2> /dev/null | awk 'END{ print $3,$2,$5 }')" diff --git a/advanced/Scripts/database_migration/gravity-db.sh b/advanced/Scripts/database_migration/gravity-db.sh index 70090a3b..282ea07b 100644 --- a/advanced/Scripts/database_migration/gravity-db.sh +++ b/advanced/Scripts/database_migration/gravity-db.sh @@ -110,4 +110,10 @@ upgrade_gravityDB(){ sqlite3 "${database}" < "${scriptPath}/11_to_12.sql" version=12 fi + if [[ "$version" == "12" ]]; then + # Add column date_updated to alist table + echo -e " ${INFO} Upgrading gravity database from version 12 to 13" + sqlite3 "${database}" < "${scriptPath}/12_to_13.sql" + version=13 + fi } diff --git a/advanced/Scripts/database_migration/gravity/12_to_13.sql b/advanced/Scripts/database_migration/gravity/12_to_13.sql new file mode 100644 index 00000000..d16791d6 --- /dev/null +++ b/advanced/Scripts/database_migration/gravity/12_to_13.sql @@ -0,0 +1,18 @@ +.timeout 30000 + +PRAGMA FOREIGN_KEYS=OFF; + +BEGIN TRANSACTION; + +ALTER TABLE adlist ADD COLUMN date_updated INTEGER; + +DROP TRIGGER tr_adlist_update; + +CREATE TRIGGER tr_adlist_update AFTER UPDATE OF address,enabled,comment ON adlist + BEGIN + UPDATE adlist SET date_modified = (cast(strftime('%s', 'now') as int)) WHERE id = NEW.id; + END; + +UPDATE info SET value = 13 WHERE property = 'version'; + +COMMIT;
\ No newline at end of file diff --git a/advanced/Scripts/list.sh b/advanced/Scripts/list.sh index 77a5dece..16bb2001 100755 --- a/advanced/Scripts/list.sh +++ b/advanced/Scripts/list.sh @@ -231,7 +231,15 @@ Displaylist() { } NukeList() { - sqlite3 "${gravityDBfile}" "DELETE FROM domainlist WHERE type = ${typeId};" + count=$(sqlite3 "${gravityDBfile}" "SELECT COUNT(1) FROM domainlist WHERE type = ${typeId};") + listname="$(GetListnameFromTypeId "${typeId}")" + if [ "$count" -gt 0 ];then + sqlite3 "${gravityDBfile}" "DELETE FROM domainlist WHERE type = ${typeId};" + echo " ${TICK} Removed ${count} domain(s) from the ${listname}" + else + echo " ${INFO} ${listname} already empty. Nothing to do!" + fi + exit 0; } GetComment() { diff --git a/advanced/Scripts/piholeDebug.sh b/advanced/Scripts/piholeDebug.sh index 92d6dad7..5c2b5f1c 100755 --- a/advanced/Scripts/piholeDebug.sh +++ b/advanced/Scripts/piholeDebug.sh @@ -48,6 +48,7 @@ FAQ_UPDATE_PI_HOLE="${COL_CYAN}https://discourse.pi-hole.net/t/how-do-i-update-p FAQ_CHECKOUT_COMMAND="${COL_CYAN}https://discourse.pi-hole.net/t/the-pihole-command-with-examples/738#checkout${COL_NC}" FAQ_HARDWARE_REQUIREMENTS="${COL_CYAN}https://docs.pi-hole.net/main/prerequisites/${COL_NC}" FAQ_HARDWARE_REQUIREMENTS_PORTS="${COL_CYAN}https://docs.pi-hole.net/main/prerequisites/#ports${COL_NC}" +FAQ_HARDWARE_REQUIREMENTS_FIREWALLD="${COL_CYAN}https://docs.pi-hole.net/main/prerequisites/#firewalld${COL_NC}" FAQ_GATEWAY="${COL_CYAN}https://discourse.pi-hole.net/t/why-is-a-default-gateway-important-for-pi-hole/3546${COL_NC}" FAQ_ULA="${COL_CYAN}https://discourse.pi-hole.net/t/use-ipv6-ula-addresses-for-pi-hole/2127${COL_NC}" FAQ_FTL_COMPATIBILITY="${COL_CYAN}https://github.com/pi-hole/FTL#compatibility-list${COL_NC}" @@ -124,6 +125,8 @@ get_ftl_conf_value() { PIHOLE_GRAVITY_DB_FILE="$(get_ftl_conf_value "GRAVITYDB" "${PIHOLE_DIRECTORY}/gravity.db")" +PIHOLE_FTL_DB_FILE="$(get_ftl_conf_value "DBFILE" "${PIHOLE_DIRECTORY}/pihole-FTL.db")" + PIHOLE_COMMAND="${BIN_DIRECTORY}/pihole" PIHOLE_COLTABLE_FILE="${BIN_DIRECTORY}/COL_TABLE" @@ -396,49 +399,54 @@ check_critical_program_versions() { os_check() { # This function gets a list of supported OS versions from a TXT record at versions.pi-hole.net # and determines whether or not the script is running on one of those systems - local remote_os_domain valid_os valid_version detected_os_pretty detected_os detected_version + local remote_os_domain valid_os valid_version detected_os detected_version cmdResult digReturnCode response remote_os_domain="versions.pi-hole.net" - valid_os=false - valid_version=false - detected_os_pretty=$(cat /etc/*release | grep PRETTY_NAME | cut -d '=' -f2- | tr -d '"') - detected_os="${detected_os_pretty%% *}" - detected_version=$(cat /etc/*release | grep VERSION_ID | cut -d '=' -f2- | tr -d '"') + detected_os=$(grep "\bID\b" /etc/os-release | cut -d '=' -f2 | tr -d '"') + detected_version=$(grep VERSION_ID /etc/os-release | cut -d '=' -f2 | tr -d '"') + + cmdResult="$(dig +short -t txt ${remote_os_domain} @ns1.pi-hole.net 2>&1; echo $?)" + #Get the return code of the previous command (last line) + digReturnCode="${cmdResult##*$'\n'}" - IFS=" " read -r -a supportedOS < <(dig +short -t txt ${remote_os_domain} | tr -d '"') + # Extract dig response + response="${cmdResult%%$'\n'*}" - for i in "${supportedOS[@]}" + IFS=" " read -r -a supportedOS < <(echo "${response}" | tr -d '"') + for distro_and_versions in "${supportedOS[@]}" do - os_part=$(echo "$i" | cut -d '=' -f1) - versions_part=$(echo "$i" | cut -d '=' -f2-) - - if [[ "${detected_os}" =~ ${os_part} ]]; then - valid_os=true - IFS="," read -r -a supportedVer <<<"${versions_part}" - for x in "${supportedVer[@]}" - do - if [[ "${detected_version}" =~ $x ]];then - valid_version=true - break - fi - done - break + distro_part="${distro_and_versions%%=*}" + versions_part="${distro_and_versions##*=}" + + if [[ "${detected_os^^}" =~ ${distro_part^^} ]]; then + valid_os=true + IFS="," read -r -a supportedVer <<<"${versions_part}" + for version in "${supportedVer[@]}" + do + if [[ "${detected_version}" =~ $version ]]; then + valid_version=true + break + fi + done + break fi done - # Display findings back to the user + log_write "${INFO} dig return code: ${digReturnCode}" + log_write "${INFO} dig response: ${response}" + if [ "$valid_os" = true ]; then - log_write "${TICK} Distro: ${COL_GREEN}${detected_os}${COL_NC}" + log_write "${TICK} Distro: ${COL_GREEN}${detected_os^}${COL_NC}" if [ "$valid_version" = true ]; then log_write "${TICK} Version: ${COL_GREEN}${detected_version}${COL_NC}" else log_write "${CROSS} Version: ${COL_RED}${detected_version}${COL_NC}" - log_write "${CROSS} Error: ${COL_RED}${detected_os} is supported but version ${detected_version} is currently unsupported (${FAQ_HARDWARE_REQUIREMENTS})${COL_NC}" + log_write "${CROSS} Error: ${COL_RED}${detected_os^} is supported but version ${detected_version} is currently unsupported (${FAQ_HARDWARE_REQUIREMENTS})${COL_NC}" fi else - log_write "${CROSS} Distro: ${COL_RED}${detected_os}${COL_NC}" - log_write "${CROSS} Error: ${COL_RED}${detected_os} is not a supported distro (${FAQ_HARDWARE_REQUIREMENTS})${COL_NC}" + log_write "${CROSS} Distro: ${COL_RED}${detected_os^}${COL_NC}" + log_write "${CROSS} Error: ${COL_RED}${detected_os^} is not a supported distro (${FAQ_HARDWARE_REQUIREMENTS})${COL_NC}" fi } @@ -488,6 +496,58 @@ check_selinux() { fi } +check_firewalld() { + # FirewallD ships by default on Fedora/CentOS/RHEL and enabled upon clean install + # FirewallD is not configured by the installer and is the responsibility of the user + echo_current_diagnostic "FirewallD" + # Check if FirewallD service is enabled + if command -v systemctl &> /dev/null; then + # get its status via systemctl + local firewalld_status + firewalld_status=$(systemctl is-active firewalld) + log_write "${INFO} ${COL_GREEN}Firewalld service ${firewalld_status}${COL_NC}"; + if [ "${firewalld_status}" == "active" ]; then + # test common required service ports + local firewalld_enabled_services + firewalld_enabled_services=$(firewall-cmd --list-services) + local firewalld_expected_services=("http" "dns" "dhcp" "dhcpv6") + for i in "${firewalld_expected_services[@]}"; do + if [[ "${firewalld_enabled_services}" =~ ${i} ]]; then + log_write "${TICK} ${COL_GREEN} Allow Service: ${i}${COL_NC}"; + else + log_write "${CROSS} ${COL_RED} Allow Service: ${i}${COL_NC} (${FAQ_HARDWARE_REQUIREMENTS_FIREWALLD})" + fi + done + # check for custom FTL FirewallD zone + local firewalld_zones + firewalld_zones=$(firewall-cmd --get-zones) + if [[ "${firewalld_zones}" =~ "ftl" ]]; then + log_write "${TICK} ${COL_GREEN}FTL Custom Zone Detected${COL_NC}"; + # check FTL custom zone interface: lo + local firewalld_ftl_zone_interfaces + firewalld_ftl_zone_interfaces=$(firewall-cmd --zone=ftl --list-interfaces) + if [[ "${firewalld_ftl_zone_interfaces}" =~ "lo" ]]; then + log_write "${TICK} ${COL_GREEN} Local Interface Detected${COL_NC}"; + else + log_write "${CROSS} ${COL_RED} Local Interface Not Detected${COL_NC} (${FAQ_HARDWARE_REQUIREMENTS_FIREWALLD})" + fi + # check FTL custom zone port: 4711 + local firewalld_ftl_zone_ports + firewalld_ftl_zone_ports=$(firewall-cmd --zone=ftl --list-ports) + if [[ "${firewalld_ftl_zone_ports}" =~ "4711/tcp" ]]; then + log_write "${TICK} ${COL_GREEN} FTL Port 4711/tcp Detected${COL_NC}"; + else + log_write "${CROSS} ${COL_RED} FTL Port 4711/tcp Not Detected${COL_NC} (${FAQ_HARDWARE_REQUIREMENTS_FIREWALLD})" + fi + else + log_write "${CROSS} ${COL_RED}FTL Custom Zone Not Detected${COL_NC} (${FAQ_HARDWARE_REQUIREMENTS_FIREWALLD})" + fi + fi + else + log_write "${TICK} ${COL_GREEN}Firewalld service not detected${COL_NC}"; + fi +} + processor_check() { echo_current_diagnostic "Processor" # Store the processor type in a variable @@ -888,6 +948,18 @@ process_status(){ done } +ftl_full_status(){ + # if using systemd print the full status of pihole-FTL + echo_current_diagnostic "Pi-hole-FTL full status" + local FTL_status + if command -v systemctl &> /dev/null; then + FTL_status=$(systemctl status --full --no-pager pihole-FTL.service) + log_write " ${FTL_status}" + else + log_write "${INFO} systemctl: command not found" + fi +} + make_array_from_file() { local filename="${1}" # The second argument can put a limit on how many line should be read from the file @@ -1021,8 +1093,8 @@ list_files_in_dir() { log_write "\\n${COL_GREEN}$(ls -ld "${dir_to_parse}"/"${each_file}")${COL_NC}" # Check if the file we want to view has a limit (because sometimes we just need a little bit of info from the file, not the entire thing) case "${dir_to_parse}/${each_file}" in - # If it's Web server error log, just give the first 25 lines - "${PIHOLE_WEB_SERVER_ERROR_LOG_FILE}") make_array_from_file "${dir_to_parse}/${each_file}" 25 + # If it's Web server error log, give the first and last 25 lines + "${PIHOLE_WEB_SERVER_ERROR_LOG_FILE}") head_tail_log "${dir_to_parse}/${each_file}" 25 ;; # Same for the FTL log "${PIHOLE_FTL_LOG}") head_tail_log "${dir_to_parse}/${each_file}" 35 @@ -1112,6 +1184,46 @@ show_db_entries() { IFS="$OLD_IFS" } +show_FTL_db_entries() { + local title="${1}" + local query="${2}" + local widths="${3}" + + echo_current_diagnostic "${title}" + + OLD_IFS="$IFS" + IFS=$'\r\n' + local entries=() + mapfile -t entries < <(\ + sqlite3 "${PIHOLE_FTL_DB_FILE}" \ + -cmd ".headers on" \ + -cmd ".mode column" \ + -cmd ".width ${widths}" \ + "${query}"\ + ) + + for line in "${entries[@]}"; do + log_write " ${line}" + done + + IFS="$OLD_IFS" +} + +check_dhcp_servers() { + echo_current_diagnostic "Discovering active DHCP servers (takes 10 seconds)" + + OLD_IFS="$IFS" + IFS=$'\n' + local entries=() + mapfile -t entries < <(pihole-FTL dhcp-discover) + + for line in "${entries[@]}"; do + log_write " ${line}" + done + + IFS="$OLD_IFS" +} + show_groups() { show_db_entries "Groups" "SELECT id,CASE enabled WHEN '0' THEN ' 0' WHEN '1' THEN ' 1' ELSE enabled END enabled,name,datetime(date_added,'unixepoch','localtime') date_added,datetime(date_modified,'unixepoch','localtime') date_modified,description FROM \"group\"" "4 7 50 19 19 50" } @@ -1128,6 +1240,10 @@ show_clients() { show_db_entries "Clients" "SELECT id,GROUP_CONCAT(client_by_group.group_id) group_ids,ip,datetime(date_added,'unixepoch','localtime') date_added,datetime(date_modified,'unixepoch','localtime') date_modified,comment FROM client LEFT JOIN client_by_group ON client.id = client_by_group.client_id GROUP BY id;" "4 12 100 19 19 50" } +show_messages() { + show_FTL_db_entries "Pi-hole diagnosis messages" "SELECT id,datetime(timestamp,'unixepoch','localtime') timestamp,type,message,blob1,blob2,blob3,blob4,blob5 FROM message;" "4 19 20 60 20 20 20 20 20" +} + analyze_gravity_list() { echo_current_diagnostic "Gravity List and Database" @@ -1300,10 +1416,13 @@ check_component_versions check_critical_program_versions diagnose_operating_system check_selinux +check_firewalld processor_check check_networking check_name_resolution +check_dhcp_servers process_status +ftl_full_status parse_setup_vars check_x_headers analyze_gravity_list @@ -1312,6 +1431,7 @@ show_domainlist show_clients show_adlists show_content_of_pihole_files +show_messages parse_locale analyze_pihole_log copy_to_debug_log diff --git a/advanced/Scripts/webpage.sh b/advanced/Scripts/webpage.sh index 1f7cc728..15471ad4 100755 --- a/advanced/Scripts/webpage.sh +++ b/advanced/Scripts/webpage.sh @@ -44,7 +44,7 @@ Options: -e, email Set an administrative contact address for the Block Page -h, --help Show this help dialog -i, interface Specify dnsmasq's interface listening behavior - -l, privacylevel Set privacy level (0 = lowest, 4 = highest)" + -l, privacylevel Set privacy level (0 = lowest, 3 = highest)" exit 0 } @@ -167,9 +167,11 @@ ProcessDNSSettings() { fi delete_dnsmasq_setting "domain-needed" + delete_dnsmasq_setting "expand-hosts" if [[ "${DNS_FQDN_REQUIRED}" == true ]]; then add_dnsmasq_setting "domain-needed" + add_dnsmasq_setting "expand-hosts" fi delete_dnsmasq_setting "bogus-priv" @@ -224,17 +226,20 @@ trust-anchor=.,20326,8,2,E06D44B80B8F1D39A95C0B0D7C65D08458E880409BBC68345710423 REV_SERVER_TARGET="${CONDITIONAL_FORWARDING_IP}" add_setting "REV_SERVER_TARGET" "${REV_SERVER_TARGET}" + REV_SERVER_CIDR="${CONDITIONAL_FORWARDING_REVERSE}" + if [ -z "${REV_SERVER_CIDR}" ]; then + # Convert existing input to /24 subnet (preserves legacy behavior) + # This sed converts "192.168.1.2" to "192.168.1.0/24" + # shellcheck disable=2001 + REV_SERVER_CIDR="$(sed "s+\\.[0-9]*$+\\.0/24+" <<< "${REV_SERVER_TARGET}")" + fi + add_setting "REV_SERVER_CIDR" "${REV_SERVER_CIDR}" + # Remove obsolete settings from setupVars.conf delete_setting "CONDITIONAL_FORWARDING" delete_setting "CONDITIONAL_FORWARDING_REVERSE" delete_setting "CONDITIONAL_FORWARDING_DOMAIN" delete_setting "CONDITIONAL_FORWARDING_IP" - - # Convert existing input to /24 subnet (preserves legacy behavior) - # This sed converts "192.168.1.2" to "192.168.1.0/24" - # shellcheck disable=2001 - REV_SERVER_CIDR="$(sed "s+\\.[0-9]*$+\\.0/24+" <<< "${REV_SERVER_TARGET}")" - add_setting "REV_SERVER_CIDR" "${REV_SERVER_CIDR}" fi if [[ "${REV_SERVER}" == true ]]; then @@ -370,6 +375,9 @@ dhcp-leasefile=/etc/pihole/dhcp.leases if [[ "${PIHOLE_DOMAIN}" != "none" ]]; then echo "domain=${PIHOLE_DOMAIN}" >> "${dhcpconfig}" + if [[ "${DNS_FQDN_REQUIRED}" == true ]]; then + echo "local=/${PIHOLE_DOMAIN}/" >> "${dhcpconfig}" + fi fi # Sourced from setupVars @@ -633,8 +641,8 @@ clearAudit() } SetPrivacyLevel() { - # Set privacy level. Minimum is 0, maximum is 4 - if [ "${args[2]}" -ge 0 ] && [ "${args[2]}" -le 4 ]; then + # Set privacy level. Minimum is 0, maximum is 3 + if [ "${args[2]}" -ge 0 ] && [ "${args[2]}" -le 3 ]; then changeFTLsetting "PRIVACYLEVEL" "${args[2]}" pihole restartdns reload-lists fi diff --git a/advanced/Templates/gravity.db.sql b/advanced/Templates/gravity.db.sql index fcf3489b..2aa7e8f8 100644 --- a/advanced/Templates/gravity.db.sql +++ b/advanced/Templates/gravity.db.sql @@ -31,7 +31,8 @@ CREATE TABLE adlist enabled BOOLEAN NOT NULL DEFAULT 1, date_added INTEGER NOT NULL DEFAULT (cast(strftime('%s', 'now') as int)), date_modified INTEGER NOT NULL DEFAULT (cast(strftime('%s', 'now') as int)), - comment TEXT + comment TEXT, + date_updated INTEGER ); CREATE TABLE adlist_by_group @@ -53,7 +54,7 @@ CREATE TABLE info value TEXT NOT NULL ); -INSERT INTO "info" VALUES('version','12'); +INSERT INTO "info" VALUES('version','13'); CREATE TABLE domain_audit ( @@ -85,9 +86,9 @@ CREATE TABLE client_by_group PRIMARY KEY (client_id, group_id) ); -CREATE TRIGGER tr_adlist_update AFTER UPDATE ON adlist +CREATE TRIGGER tr_adlist_update AFTER UPDATE OF address,enabled,comment ON adlist BEGIN - UPDATE adlist SET date_modified = (cast(strftime('%s', 'now') as int)) WHERE address = NEW.address; + UPDATE adlist SET date_modified = (cast(strftime('%s', 'now') as int)) WHERE id = NEW.id; END; CREATE TRIGGER tr_client_update AFTER UPDATE ON client diff --git a/advanced/bash-completion/pihole b/advanced/bash-completion/pihole index 88282b02..25208a35 100644 --- a/advanced/bash-completion/pihole +++ b/advanced/bash-completion/pihole @@ -56,11 +56,11 @@ _pihole() { ;; "privacylevel") if ( [[ "$prev2" == "admin" ]] || [[ "$prev2" == "-a" ]] ); then - opts_privacy="0 1 2 3 4" + opts_privacy="0 1 2 3" COMPREPLY=( $(compgen -W "${opts_privacy}" -- ${cur}) ) - else + else return 1 - fi + fi ;; "core"|"admin"|"ftl") if [[ "$prev2" == "checkout" ]]; then diff --git a/advanced/index.php b/advanced/index.php index 4356f0b0..ca8c35e4 100644 --- a/advanced/index.php +++ b/advanced/index.php @@ -24,7 +24,7 @@ unset($setupVars); $landPage = "../landing.php"; // Define array for hostnames to be accepted as self address for splash page -$authorizedHosts = []; +$authorizedHosts = [ "localhost" ]; if (!empty($_SERVER["FQDN"])) { // If setenv.add-environment = ("fqdn" => "true") is configured in lighttpd, // append $serverName to $authorizedHosts @@ -55,7 +55,16 @@ if ($serverName === "pi.hole" // Redirect to Web Interface exit(header("Location: /admin")); } elseif (filter_var($serverName, FILTER_VALIDATE_IP) || in_array($serverName, $authorizedHosts)) { - // Set Splash Page output + // When directly browsing via IP or authorized hostname + // Render splash/landing page based off presence of $landPage file + // Unset variables so as to not be included in $landPage or $splashPage + unset($serverName, $svPasswd, $svEmail, $authorizedHosts, $validExtTypes, $currentUrlExt, $viewPort); + // If $landPage file is present + if (is_file(getcwd()."/$landPage")) { + include $landPage; + exit(); + } + // If $landPage file was not present, Set Splash Page output $splashPage = " <!doctype html> <html lang='en'> @@ -74,15 +83,7 @@ if ($serverName === "pi.hole" </body> </html> "; - - // Set splash/landing page based off presence of $landPage - $renderPage = is_file(getcwd()."/$landPage") ? include $landPage : "$splashPage"; - - // Unset variables so as to not be included in $landPage - unset($serverName, $svPasswd, $svEmail, $authorizedHosts, $validExtTypes, $currentUrlExt, $viewPort); - - // Render splash/landing page when directly browsing via IP or authorized hostname - exit($renderPage); + exit($splashPage); } elseif ($currentUrlExt === "js") { // Serve Pi-hole JavaScript for blocked domains requesting JS exit(setHeader("js").'var x = "Pi-hole: A black hole for Internet advertisements."'); @@ -305,7 +306,7 @@ setHeader(); </p> </div> <div class="aboutLink"> - <a class="linkPH" href="https://github.com/pi-hole/pi-hole/wiki/What-is-Pi-hole%3F-A-simple-explanation"><?php //About PH ?></a> + <a class="linkPH" href="https://docs.pi-hole.net/"><?php //About PH ?></a> <?php if (!empty($svEmail)) echo '<a class="linkEmail" href="mailto:'.$svEmail.'"></a>'; ?> </div> </div> diff --git a/automated install/basic-install.sh b/automated install/basic-install.sh index ac00bf2b..21e699c7 100755 --- a/automated install/basic-install.sh +++ b/automated install/basic-install.sh @@ -21,6 +21,10 @@ # instead of continuing the installation with something broken set -e +# Set PATH to a usual default to assure that all basic commands are available. +# When using "su" an uncomplete PATH could be passed: https://github.com/pi-hole/pi-hole/issues/3209 +export PATH+=':/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' + ######## VARIABLES ######### # For better maintainability, we store as much information that can change in variables # This allows us to make a change in one place that can propagate to all instances of the variable @@ -31,13 +35,13 @@ set -e # List of supported DNS servers DNS_SERVERS=$(cat << EOM Google (ECS);8.8.8.8;8.8.4.4;2001:4860:4860:0:0:0:0:8888;2001:4860:4860:0:0:0:0:8844 -OpenDNS (ECS);208.67.222.222;208.67.220.220;2620:119:35::35;2620:119:53::53 +OpenDNS (ECS, DNSSEC);208.67.222.222;208.67.220.220;2620:119:35::35;2620:119:53::53 Level3;4.2.2.1;4.2.2.2;; Comodo;8.26.56.26;8.20.247.20;; DNS.WATCH;84.200.69.80;84.200.70.40;2001:1608:10:25:0:0:1c04:b12f;2001:1608:10:25:0:0:9249:d69b Quad9 (filtered, DNSSEC);9.9.9.9;149.112.112.112;2620:fe::fe;2620:fe::9 Quad9 (unfiltered, no DNSSEC);9.9.9.10;149.112.112.10;2620:fe::10;2620:fe::fe:10 -Quad9 (filtered + ECS);9.9.9.11;149.112.112.11;2620:fe::11; +Quad9 (filtered + ECS);9.9.9.11;149.112.112.11;2620:fe::11;2620:fe::fe:11 Cloudflare;1.1.1.1;1.0.0.1;2606:4700:4700::1111;2606:4700:4700::1001 EOM ) @@ -67,7 +71,9 @@ PI_HOLE_INSTALL_DIR="/opt/pihole" PI_HOLE_CONFIG_DIR="/etc/pihole" PI_HOLE_BIN_DIR="/usr/local/bin" PI_HOLE_BLOCKPAGE_DIR="${webroot}/pihole" -useUpdateVars=false +if [ -z "$useUpdateVars" ]; then + useUpdateVars=false +fi adlistFile="/etc/pihole/adlists.list" # Pi-hole needs an IP address; to begin, these variables are empty since we don't know what the IP is until @@ -78,6 +84,7 @@ IPV6_ADDRESS=${IPV6_ADDRESS} QUERY_LOGGING=true INSTALL_WEB_INTERFACE=true PRIVACY_LEVEL=0 +CACHE_SIZE=10000 if [ -z "${USER}" ]; then USER="$(id -un)" @@ -106,7 +113,6 @@ c=$(( c < 70 ? 70 : c )) ######## Undocumented Flags. Shhh ######## # These are undocumented flags; some of which we can use when repairing an installation # The runUnattended flag is one example of this -skipSpaceCheck=false reconfigure=false runUnattended=false INSTALL_WEB_SERVER=true @@ -114,7 +120,6 @@ INSTALL_WEB_SERVER=true for var in "$@"; do case "$var" in "--reconfigure" ) reconfigure=true;; - "--i_do_not_follow_recommendations" ) skipSpaceCheck=true;; "--unattended" ) runUnattended=true;; "--disable-install-webserver" ) INSTALL_WEB_SERVER=false;; esac @@ -178,59 +183,85 @@ os_check() { if [ "$PIHOLE_SKIP_OS_CHECK" != true ]; then # This function gets a list of supported OS versions from a TXT record at versions.pi-hole.net # and determines whether or not the script is running on one of those systems - local remote_os_domain valid_os valid_version detected_os_pretty detected_os detected_version display_warning + local remote_os_domain valid_os valid_version valid_response detected_os detected_version display_warning cmdResult digReturnCode response remote_os_domain="versions.pi-hole.net" - valid_os=false - valid_version=false - display_warning=true - detected_os_pretty=$(cat /etc/*release | grep PRETTY_NAME | cut -d '=' -f2- | tr -d '"') - detected_os="${detected_os_pretty%% *}" - detected_version=$(cat /etc/*release | grep VERSION_ID | cut -d '=' -f2- | tr -d '"') + detected_os=$(grep "\bID\b" /etc/os-release | cut -d '=' -f2 | tr -d '"') + detected_version=$(grep VERSION_ID /etc/os-release | cut -d '=' -f2 | tr -d '"') - IFS=" " read -r -a supportedOS < <(dig +short -t txt ${remote_os_domain} @ns1.pi-hole.net | tr -d '"') + cmdResult="$(dig +short -t txt ${remote_os_domain} @ns1.pi-hole.net 2>&1; echo $?)" + #Get the return code of the previous command (last line) + digReturnCode="${cmdResult##*$'\n'}" - if [ ${#supportedOS[@]} -eq 0 ]; then - printf " %b %bRetrieval of supported OS failed. Please contact support. %b\\n" "${CROSS}" "${COL_LIGHT_RED}" "${COL_NC}" - exit 1 + if [ ! "${digReturnCode}" == "0" ]; then + valid_response=false else - for i in "${supportedOS[@]}" - do - os_part=$(echo "$i" | cut -d '=' -f1) - versions_part=$(echo "$i" | cut -d '=' -f2-) - - if [[ "${detected_os}" =~ ${os_part} ]]; then - valid_os=true - IFS="," read -r -a supportedVer <<<"${versions_part}" - for x in "${supportedVer[@]}" - do - if [[ "${detected_version}" =~ $x ]];then - valid_version=true + # Dig returned 0 code, so get the actual response, and loop through it to determine if the detected variables above are valid + response="${cmdResult%%$'\n'*}" + # If the value of ${result} is a single 0, then this is the return code, not the response. Response is blank + if [ "${response}" == 0 ]; then + valid_response=false + fi + + IFS=" " read -r -a supportedOS < <(echo "${response}" | tr -d '"') + for distro_and_versions in "${supportedOS[@]}" + do + distro_part="${distro_and_versions%%=*}" + versions_part="${distro_and_versions##*=}" + + if [[ "${detected_os^^}" =~ ${distro_part^^} ]]; then + valid_os=true + IFS="," read -r -a supportedVer <<<"${versions_part}" + for version in "${supportedVer[@]}" + do + if [[ "${detected_version}" =~ $version ]]; then + valid_version=true + break + fi + done break - fi - done - break - fi + fi done - fi + fi - if [ "$valid_os" = true ] && [ "$valid_version" = true ]; then + if [ "$valid_os" = true ] && [ "$valid_version" = true ] && [ ! "$valid_response" = false ]; then display_warning=false fi - if [ "$display_warning" = true ]; then - printf " %b %bUnsupported OS detected: %s%b\\n" "${CROSS}" "${COL_LIGHT_RED}" "${detected_os_pretty}" "${COL_NC}" - printf " https://docs.pi-hole.net/main/prerequesites/#supported-operating-systems\\n" + if [ "$display_warning" != false ]; then + if [ "$valid_response" = false ]; then + + if [ "${digReturnCode}" -eq 0 ]; then + errStr="dig succeeded, but response was blank. Please contact support" + else + errStr="dig failed with return code ${digReturnCode}" + fi + printf " %b %bRetrieval of supported OS list failed. %s. %b\\n" "${CROSS}" "${COL_LIGHT_RED}" "${errStr}" "${COL_NC}" + printf " %bUnable to determine if the detected OS (%s %s) is supported%b\\n" "${COL_LIGHT_RED}" "${detected_os^}" "${detected_version}" "${COL_NC}" + printf " Possible causes for this include:\\n" + printf " - Firewall blocking certain DNS lookups from Pi-hole device\\n" + printf " - ns1.pi-hole.net being blocked (required to obtain TXT record from versions.pi-hole.net containing supported operating systems)\\n" + printf " - Other internet connectivity issues\\n" + else + printf " %b %bUnsupported OS detected: %s %s%b\\n" "${CROSS}" "${COL_LIGHT_RED}" "${detected_os^}" "${detected_version}" "${COL_NC}" + printf " If you are seeing this message and you do have a supported OS, please contact support.\\n" + fi + printf "\\n" + printf " %bhttps://docs.pi-hole.net/main/prerequesites/#supported-operating-systems%b\\n" "${COL_LIGHT_GREEN}" "${COL_NC}" + printf "\\n" + printf " If you wish to attempt to continue anyway, you can try one of the following commands to skip this check:\\n" printf "\\n" printf " e.g: If you are seeing this message on a fresh install, you can run:\\n" - printf " 'curl -sSL https://install.pi-hole.net | PIHOLE_SKIP_OS_CHECK=true sudo -E bash'\\n" + printf " %bcurl -sSL https://install.pi-hole.net | PIHOLE_SKIP_OS_CHECK=true sudo -E bash%b\\n" "${COL_LIGHT_GREEN}" "${COL_NC}" printf "\\n" printf " If you are seeing this message after having run pihole -up:\\n" - printf " 'PIHOLE_SKIP_OS_CHECK=true sudo -E pihole -r'\\n" + printf " %bPIHOLE_SKIP_OS_CHECK=true sudo -E pihole -r%b\\n" "${COL_LIGHT_GREEN}" "${COL_NC}" printf " (In this case, your previous run of pihole -up will have already updated the local repository)\\n" printf "\\n" + printf " It is possible that the installation will still fail at this stage due to an unsupported configuration.\\n" printf " If that is the case, you can feel free to ask the community on Discourse with the %bCommunity Help%b category:\\n" "${COL_LIGHT_RED}" "${COL_NC}" - printf " https://discourse.pi-hole.net/c/bugs-problems-issues/community-help/\\n" + printf " %bhttps://discourse.pi-hole.net/c/bugs-problems-issues/community-help/%b\\n" "${COL_LIGHT_GREEN}" "${COL_NC}" + printf "\\n" exit 1 else @@ -325,7 +356,7 @@ if is_command apt-get ; then PIHOLE_DEPS=(cron curl iputils-ping lsof netcat psmisc sudo unzip wget idn2 sqlite3 libcap2-bin dns-root-data libcap2) # The Web dashboard has some that also need to be installed # It's useful to separate the two since our repos are also setup as "Core" code and "Web" code - PIHOLE_WEB_DEPS=(lighttpd "${phpVer}-common" "${phpVer}-cgi" "${phpVer}-${phpSqlite}" "${phpVer}-xml" "${phpVer}-intl") + PIHOLE_WEB_DEPS=(lighttpd "${phpVer}-common" "${phpVer}-cgi" "${phpVer}-${phpSqlite}" "${phpVer}-xml" "${phpVer}-json" "${phpVer}-intl") # The Web server user, LIGHTTPD_USER="www-data" # group, @@ -656,53 +687,6 @@ welcomeDialogs() { In the next section, you can choose to use your current network settings (DHCP) or to manually edit them." "${r}" "${c}" } -# We need to make sure there is enough space before installing, so there is a function to check this -verifyFreeDiskSpace() { - # 50MB is the minimum space needed (45MB install (includes web admin bootstrap/jquery libraries etc) + 5MB one day of logs.) - # - Fourdee: Local ensures the variable is only created, and accessible within this function/void. Generally considered a "good" coding practice for non-global variables. - local str="Disk space check" - # Required space in KB - local required_free_kilobytes=51200 - # Calculate existing free space on this machine - local existing_free_kilobytes - existing_free_kilobytes=$(df -Pk | grep -m1 '\/$' | awk '{print $4}') - - # If the existing space is not an integer, - if ! [[ "${existing_free_kilobytes}" =~ ^([0-9])+$ ]]; then - # show an error that we can't determine the free space - printf " %b %s\\n" "${CROSS}" "${str}" - printf " %b Unknown free disk space! \\n" "${INFO}" - printf " We were unable to determine available free disk space on this system.\\n" - printf " You may override this check, however, it is not recommended.\\n" - printf " The option '%b--i_do_not_follow_recommendations%b' can override this.\\n" "${COL_LIGHT_RED}" "${COL_NC}" - printf " e.g: curl -sSL https://install.pi-hole.net | bash /dev/stdin %b<option>%b\\n" "${COL_LIGHT_RED}" "${COL_NC}" - # exit with an error code - exit 1 - # If there is insufficient free disk space, - elif [[ "${existing_free_kilobytes}" -lt "${required_free_kilobytes}" ]]; then - # show an error message - printf " %b %s\\n" "${CROSS}" "${str}" - printf " %b Your system disk appears to only have %s KB free\\n" "${INFO}" "${existing_free_kilobytes}" - printf " It is recommended to have a minimum of %s KB to run the Pi-hole\\n" "${required_free_kilobytes}" - # if the vcgencmd command exists, - if is_command vcgencmd ; then - # it's probably a Raspbian install, so show a message about expanding the filesystem - printf " If this is a new install you may need to expand your disk\\n" - printf " Run 'sudo raspi-config', and choose the 'expand file system' option\\n" - printf " After rebooting, run this installation again\\n" - printf " e.g: curl -sSL https://install.pi-hole.net | bash\\n" - fi - # Show there is not enough free space - printf "\\n %bInsufficient free space, exiting...%b\\n" "${COL_LIGHT_RED}" "${COL_NC}" - # and exit with an error - exit 1 - # Otherwise, - else - # Show that we're running a disk space check - printf " %b %s\\n" "${TICK}" "${str}" - fi -} - # A function that let's the user pick an interface to use with Pi-hole chooseInterface() { # Turn the available interfaces into an array so it can be used with a whiptail dialog @@ -1229,7 +1213,6 @@ setPrivacyLevel() { "1" "Hide domains" off "2" "Hide domains and clients" off "3" "Anonymous mode" off - "4" "Disabled statistics" off ) # Get the user's choice @@ -1262,13 +1245,18 @@ setAdminFlag() { printf " %b Web Interface Off\\n" "${INFO}" # or false INSTALL_WEB_INTERFACE=false + # Deselect the web server as well, since it is obsolete then + INSTALL_WEB_SERVER=false ;; esac - # Request user to install web server, if --disable-install-webserver has not been used (INSTALL_WEB_SERVER=true is default). + # Request user to install web server, if it has not been deselected before (INSTALL_WEB_SERVER=true is default). if [[ "${INSTALL_WEB_SERVER}" == true ]]; then - WebToggleCommand=(whiptail --separate-output --radiolist "Do you wish to install the web server (lighttpd)?\\n\\nNB: If you disable this, and, do not have an existing webserver installed, the web interface will not function." "${r}" "${c}" 6) - # with the default being enabled + # Get list of required PHP modules, excluding base package (common) and handler (cgi) + local i php_modules + for i in "${PIHOLE_WEB_DEPS[@]}"; do [[ $i == 'php'* && $i != *'-common' && $i != *'-cgi' ]] && php_modules+=" ${i#*-}"; done + WebToggleCommand=(whiptail --separate-output --radiolist "Do you wish to install the web server (lighttpd) and required PHP modules?\\n\\nNB: If you disable this, and, do not have an existing web server and required PHP modules (${php_modules# }) installed, the web interface will not function. Additionally the web server user needs to be member of the \"pihole\" group for full functionality." "${r}" "${c}" 6) + # Enable as default and recommended option WebChooseOptions=("On (Recommended)" "" on Off "" off) WebChoices=$("${WebToggleCommand[@]}" "${WebChooseOptions[@]}" 2>&1 >/dev/tty) || (printf " %bCancel was selected, exiting installer%b\\n" "${COL_LIGHT_RED}" "${COL_NC}" && exit 1) @@ -1302,9 +1290,7 @@ chooseBlocklists() { # In a variable, show the choices available; exit if Cancel is selected choices=$("${cmd[@]}" "${options[@]}" 2>&1 >/dev/tty) || { printf " %bCancel was selected, exiting installer%b\\n" "${COL_LIGHT_RED}" "${COL_NC}"; rm "${adlistFile}" ;exit 1; } - # create empty adlist file if no list was selected - : > "${adlistFile}" - # For each choice available + # For each choice available, for choice in ${choices} do appendToListsFile "${choice}" @@ -1333,8 +1319,6 @@ installDefaultBlocklists() { fi appendToListsFile StevenBlack appendToListsFile MalwareDom - appendToListsFile DisconTrack - appendToListsFile DisconAd } # Check if /etc/dnsmasq.conf is from pi-hole. If so replace with an original and install new in .d directory @@ -1402,6 +1386,9 @@ version_check_dnsmasq() { sed -i '/^server=@DNS2@/d' "${dnsmasq_pihole_01_location}" fi + # Set the cache size + sed -i "s/@CACHE_SIZE@/$CACHE_SIZE/" ${dnsmasq_pihole_01_location} + # sed -i 's/^#conf-dir=\/etc\/dnsmasq.d$/conf-dir=\/etc\/dnsmasq.d/' "${dnsmasq_conf}" @@ -1490,6 +1477,15 @@ installConfigs() { return 1 fi fi + + # Install empty custom.list file if it does not exist + if [[ ! -r "${PI_HOLE_CONFIG_DIR}/custom.list" ]]; then + if ! install -o root -m 644 /dev/null "${PI_HOLE_CONFIG_DIR}/custom.list" &>/dev/null; then + printf " %bError: Unable to initialize configuration file %s/custom.list\\n" "${COL_LIGHT_RED}" "${PI_HOLE_CONFIG_DIR}" + return 1 + fi + fi + # If the user chose to install the dashboard, if [[ "${INSTALL_WEB_SERVER}" == true ]]; then # and if the Web server conf directory does not exist, @@ -1917,7 +1913,7 @@ finalExports() { # If the setup variable file exists, if [[ -e "${setupVars}" ]]; then # update the variables in the file - sed -i.update.bak '/PIHOLE_INTERFACE/d;/IPV4_ADDRESS/d;/IPV6_ADDRESS/d;/PIHOLE_DNS_1/d;/PIHOLE_DNS_2/d;/QUERY_LOGGING/d;/INSTALL_WEB_SERVER/d;/INSTALL_WEB_INTERFACE/d;/LIGHTTPD_ENABLED/d;' "${setupVars}" + sed -i.update.bak '/PIHOLE_INTERFACE/d;/IPV4_ADDRESS/d;/IPV6_ADDRESS/d;/PIHOLE_DNS_1/d;/PIHOLE_DNS_2/d;/QUERY_LOGGING/d;/INSTALL_WEB_SERVER/d;/INSTALL_WEB_INTERFACE/d;/LIGHTTPD_ENABLED/d;/CACHE_SIZE/d;' "${setupVars}" fi # echo the information to the user { @@ -1930,6 +1926,7 @@ finalExports() { echo "INSTALL_WEB_SERVER=${INSTALL_WEB_SERVER}" echo "INSTALL_WEB_INTERFACE=${INSTALL_WEB_INTERFACE}" echo "LIGHTTPD_ENABLED=${LIGHTTPD_ENABLED}" + echo "CACHE_SIZE=${CACHE_SIZE}" }>> "${setupVars}" chmod 644 "${setupVars}" @@ -2004,9 +2001,9 @@ installPihole() { # Set the owner and permissions chown ${LIGHTTPD_USER}:${LIGHTTPD_GROUP} ${webroot} chmod 0775 ${webroot} - # Repair permissions if /var/www/html is not world readable + # Repair permissions if webroot is not world readable chmod a+rx /var/www - chmod a+rx /var/www/html + chmod a+rx ${webroot} # Give lighttpd access to the pihole group so the web interface can # manage the gravity.db database usermod -a -G pihole ${LIGHTTPD_USER} @@ -2091,8 +2088,13 @@ checkSelinux() { if [[ "${SELINUX_ENFORCING}" -eq 1 ]] && [[ -z "${PIHOLE_SELINUX}" ]]; then printf " Pi-hole does not provide an SELinux policy as the required changes modify the security of your system.\\n" printf " Please refer to https://wiki.centos.org/HowTos/SELinux if SELinux is required for your deployment.\\n" + printf " This check can be skipped by setting the environment variable %bPIHOLE_SELINUX%b to %btrue%b\\n" "${COL_LIGHT_RED}" "${COL_NC}" "${COL_LIGHT_RED}" "${COL_NC}" + printf " e.g: export PIHOLE_SELINUX=true\\n" + printf " By setting this variable to true you acknowledge there may be issues with Pi-hole during or after the install\\n" printf "\\n %bSELinux Enforcing detected, exiting installer%b\\n" "${COL_LIGHT_RED}" "${COL_NC}"; exit 1; + elif [[ "${SELINUX_ENFORCING}" -eq 1 ]] && [[ -n "${PIHOLE_SELINUX}" ]]; then + printf " %b %bSELinux Enforcing detected%b. PIHOLE_SELINUX env variable set - installer will continue\\n" "${INFO}" "${COL_LIGHT_RED}" "${COL_NC}" fi } @@ -2394,7 +2396,7 @@ get_binary_name() { local l_binary - local str="Detecting architecture" + local str="Detecting processor" printf " %b %s..." "${INFO}" "${str}" # If the machine is arm or aarch if [[ "${machine}" == "arm"* || "${machine}" == *"aarch"* ]]; then @@ -2407,48 +2409,58 @@ get_binary_name() { lib=$(ldd /bin/ls | grep -E '^\s*/lib' | awk '{ print $1 }') # if [[ "${lib}" == "/lib/ld-linux-aarch64.so.1" ]]; then - printf "%b %b Detected ARM-aarch64 architecture\\n" "${OVER}" "${TICK}" + printf "%b %b Detected AArch64 (64 Bit ARM) processor\\n" "${OVER}" "${TICK}" # set the binary to be used l_binary="pihole-FTL-aarch64-linux-gnu" # elif [[ "${lib}" == "/lib/ld-linux-armhf.so.3" ]]; then - # - if [[ "${rev}" -gt 6 ]]; then - printf "%b %b Detected ARM-hf architecture (armv7+)\\n" "${OVER}" "${TICK}" + # Hard-float available: Use gnueabihf binaries + # If ARMv8 or higher is found (e.g., BCM2837 as found in Raspberry Pi Model 3B) + if [[ "${rev}" -gt 7 ]]; then + printf "%b %b Detected ARMv8 (or newer) processor\\n" "${OVER}" "${TICK}" # set the binary to be used - l_binary="pihole-FTL-arm-linux-gnueabihf" - # Otherwise, + l_binary="pihole-FTL-armv8-linux-gnueabihf" + # Otherwise, if ARMv7 is found (e.g., BCM2836 as found in Raspberry Pi Model 2) + elif [[ "${rev}" -eq 7 ]]; then + printf "%b %b Detected ARMv7 processor (with hard-float support)\\n" "${OVER}" "${TICK}" + # set the binary to be used + l_binary="pihole-FTL-armv7-linux-gnueabihf" + # Otherwise, use the ARMv6 binary (e.g., BCM2835 as found in Raspberry Pi Zero and Model 1) else - printf "%b %b Detected ARM-hf architecture (armv6 or lower) Using ARM binary\\n" "${OVER}" "${TICK}" + printf "%b %b Detected ARMv6 processor (with hard-float support)\\n" "${OVER}" "${TICK}" # set the binary to be used - l_binary="pihole-FTL-arm-linux-gnueabi" + l_binary="pihole-FTL-armv6-linux-gnueabihf" fi else - if [[ -f "/.dockerenv" ]]; then - printf "%b %b Detected ARM architecture in docker\\n" "${OVER}" "${TICK}" + # No hard-float support found: Use gnueabi binaries + # Use the ARMv4-compliant binary only if we detected an ARMv4T core + if [[ "${rev}" -eq 4 ]]; then + printf "%b %b Detected ARMv4 processor\\n" "${OVER}" "${TICK}" # set the binary to be used - l_binary="pihole-FTL-armel-native" + l_binary="pihole-FTL-armv4-linux-gnueabi" + # Otherwise, use the ARMv5 binary. To date (end of 2020), all modern ARM processors + # are backwards-compatible to the ARMv5 else - printf "%b %b Detected ARM architecture\\n" "${OVER}" "${TICK}" + printf "%b %b Detected ARMv5 (or newer) processor\\n" "${OVER}" "${TICK}" # set the binary to be used - l_binary="pihole-FTL-arm-linux-gnueabi" + l_binary="pihole-FTL-armv5-linux-gnueabi" fi fi elif [[ "${machine}" == "x86_64" ]]; then - # This gives the architecture of packages dpkg installs (for example, "i386") + # This gives the processor of packages dpkg installs (for example, "i386") local dpkgarch - dpkgarch=$(dpkg --print-architecture 2> /dev/null || true) + dpkgarch=$(dpkg --print-processor 2> /dev/null || true) # Special case: This is a 32 bit OS, installed on a 64 bit machine - # -> change machine architecture to download the 32 bit executable + # -> change machine processor to download the 32 bit executable # We only check this for Debian-based systems as this has been an issue # in the past (see https://github.com/pi-hole/pi-hole/pull/2004) if [[ "${dpkgarch}" == "i386" ]]; then - printf "%b %b Detected 32bit (i686) architecture\\n" "${OVER}" "${TICK}" + printf "%b %b Detected 32bit (i686) processor\\n" "${OVER}" "${TICK}" l_binary="pihole-FTL-linux-x86_32" else # 64bit - printf "%b %b Detected x86_64 architecture\\n" "${OVER}" "${TICK}" + printf "%b %b Detected x86_64 processor\\n" "${OVER}" "${TICK}" # set the binary to be used l_binary="pihole-FTL-linux-x86_64" fi @@ -2456,10 +2468,10 @@ get_binary_name() { # Something else - we try to use 32bit executable and warn the user if [[ ! "${machine}" == "i686" ]]; then printf "%b %b %s...\\n" "${OVER}" "${CROSS}" "${str}" - printf " %b %bNot able to detect architecture (unknown: %s), trying 32bit executable%b\\n" "${INFO}" "${COL_LIGHT_RED}" "${machine}" "${COL_NC}" + printf " %b %bNot able to detect processor (unknown: %s), trying x86 (32bit) executable%b\\n" "${INFO}" "${COL_LIGHT_RED}" "${machine}" "${COL_NC}" printf " %b Contact Pi-hole Support if you experience issues (e.g: FTL not running)\\n" "${INFO}" else - printf "%b %b Detected 32bit (i686) architecture\\n" "${OVER}" "${TICK}" + printf "%b %b Detected 32bit (i686) processor\\n" "${OVER}" "${TICK}" fi l_binary="pihole-FTL-linux-x86_32" fi @@ -2603,7 +2615,7 @@ main() { # Otherwise, else # They do not have enough privileges, so let the user know - printf " %b %s\\n" "${CROSS}" "${str}" + printf " %b %s\\n" "${INFO}" "${str}" printf " %b %bScript called with non-root privileges%b\\n" "${INFO}" "${COL_LIGHT_RED}" "${COL_NC}" printf " The Pi-hole requires elevated privileges to install and run\\n" printf " Please check the installer for any concerns regarding this requirement\\n" @@ -2613,8 +2625,16 @@ main() { # If the sudo command exists, if is_command sudo ; then printf "%b %b Sudo utility check\\n" "${OVER}" "${TICK}" - # Download the install script and run it with admin rights - exec curl -sSL https://raw.githubusercontent.com/pi-hole/pi-hole/master/automated%20install/basic-install.sh | sudo bash "$@" + + # when run via curl piping + if [[ "$0" == "bash" ]]; then + # Download the install script and run it with admin rights + exec curl -sSL https://raw.githubusercontent.com/pi-hole/pi-hole/master/automated%20install/basic-install.sh | sudo bash "$@" + else + # when run via calling local bash script + exec sudo bash "$0" "$@" + fi + exit $? # Otherwise, else @@ -2646,13 +2666,6 @@ main() { fi # Start the installer - # Verify there is enough disk space for the install - if [[ "${skipSpaceCheck}" == true ]]; then - printf " %b Skipping free disk space verification\\n" "${INFO}" - else - verifyFreeDiskSpace - fi - # Notify user of package availability notify_package_updates_available @@ -39,7 +39,6 @@ gravityDBfile="${piholeDir}/gravity.db" gravityTEMPfile="${piholeDir}/gravity_temp.db" gravityDBschema="${piholeGitDir}/advanced/Templates/gravity.db.sql" gravityDBcopy="${piholeGitDir}/advanced/Templates/gravity_copy.sql" -optimize_database=false domainsExtension="domains" @@ -207,6 +206,17 @@ database_table_from_file() { echo -e " ${CROSS} Unable to remove ${tmpFile}" } +# Update timestamp of last update of this list. We store this in the "old" database as all values in the new database will later be overwritten +database_adlist_updated() { + output=$( { printf ".timeout 30000\\nUPDATE adlist SET date_updated = (cast(strftime('%%s', 'now') as int)) WHERE id = %i;\\n" "${1}" | sqlite3 "${gravityDBfile}"; } 2>&1 ) + status="$?" + + if [[ "${status}" -ne 0 ]]; then + echo -e "\\n ${CROSS} Unable to update timestamp of adlist with ID ${1} in database ${gravityDBfile}\\n ${output}" + gravity_Cleanup "error" + fi +} + # Migrate pre-v5.0 list files to database-based Pi-hole versions migrate_to_database() { # Create database file only if not present @@ -334,7 +344,7 @@ gravity_DownloadBlocklists() { return 1 fi - local url domain agent cmd_ext str target + local url domain agent cmd_ext str target compression echo "" # Prepare new gravity database @@ -353,13 +363,24 @@ gravity_DownloadBlocklists() { target="$(mktemp -p "/tmp" --suffix=".gravity")" + # Use compression to reduce the amount of data that is transfered + # between the Pi-hole and the ad list provider. Use this feature + # only if it is supported by the locally available version of curl + if curl -V | grep -q "Features:.* libz"; then + compression="--compressed" + echo -e " ${INFO} Using libz compression\n" + else + compression="" + echo -e " ${INFO} Libz compression not available\n" + fi # Loop through $sources and download each one for ((i = 0; i < "${#sources[@]}"; i++)); do url="${sources[$i]}" domain="${sourceDomains[$i]}" + id="${sourceIDs[$i]}" # Save the file as list.#.domain - saveLocation="${piholeDir}/list.${i}.${domain}.${domainsExtension}" + saveLocation="${piholeDir}/list.${id}.${domain}.${domainsExtension}" activeDomains[$i]="${saveLocation}" # Default user-agent (for Cloudflare's Browser Integrity Check: https://support.cloudflare.com/hc/en-us/articles/200170086-What-does-the-Browser-Integrity-Check-do-) @@ -378,7 +399,7 @@ gravity_DownloadBlocklists() { if [[ "${url}" =~ ${regex} ]]; then echo -e " ${CROSS} Invalid Target" else - gravity_DownloadBlocklistFromUrl "${url}" "${cmd_ext}" "${agent}" "${sourceIDs[$i]}" "${saveLocation}" "${target}" + gravity_DownloadBlocklistFromUrl "${url}" "${cmd_ext}" "${agent}" "${sourceIDs[$i]}" "${saveLocation}" "${target}" "${compression}" fi echo "" done @@ -453,7 +474,7 @@ parseList() { # Download specified URL and perform checks on HTTP status and file content gravity_DownloadBlocklistFromUrl() { - local url="${1}" cmd_ext="${2}" agent="${3}" adlistID="${4}" saveLocation="${5}" target="${6}" + local url="${1}" cmd_ext="${2}" agent="${3}" adlistID="${4}" saveLocation="${5}" target="${6}" compression="${7}" local heisenbergCompensator="" patternBuffer str httpCode success="" # Create temp file to store content on disk instead of RAM @@ -502,8 +523,9 @@ gravity_DownloadBlocklistFromUrl() { echo -ne " ${INFO} ${str} Pending..." cmd_ext="--resolve $domain:$port:$ip $cmd_ext" fi + # shellcheck disable=SC2086 - httpCode=$(curl -s -L ${cmd_ext} ${heisenbergCompensator} -w "%{http_code}" -A "${agent}" "${url}" -o "${patternBuffer}" 2> /dev/null) + httpCode=$(curl -s -L ${compression} ${cmd_ext} ${heisenbergCompensator} -w "%{http_code}" -A "${agent}" "${url}" -o "${patternBuffer}" 2> /dev/null) case $url in # Did we "download" a local file? @@ -543,6 +565,8 @@ gravity_DownloadBlocklistFromUrl() { gravity_ParseFileIntoDomains "${patternBuffer}" "${saveLocation}" # Add domains to database table file parseList "${adlistID}" "${saveLocation}" "${target}" + # Update date_updated field in gravity database table + database_adlist_updated "${adlistID}" else # Fall back to previously cached list if $patternBuffer is empty echo -e " ${INFO} Received empty file: ${COL_LIGHT_GREEN}using previously cached list${COL_NC}" @@ -728,21 +752,6 @@ gravity_Cleanup() { echo -e "${OVER} ${TICK} ${str}" - if ${optimize_database} ; then - str="Optimizing domains database" - echo -ne " ${INFO} ${str}..." - # Run VACUUM command on database to optimize it - output=$( { sqlite3 "${gravityDBfile}" "VACUUM;"; } 2>&1 ) - status="$?" - - if [[ "${status}" -ne 0 ]]; then - echo -e "\\n ${CROSS} Unable to optimize gravity database ${gravityDBfile}\\n ${output}" - error="error" - else - echo -e "${OVER} ${TICK} ${str}" - fi - fi - # Only restart DNS service if offline if ! pgrep pihole-FTL &> /dev/null; then "${PIHOLE_COMMAND}" restartdns @@ -769,7 +778,6 @@ Options: for var in "$@"; do case "${var}" in "-f" | "--force" ) forceDelete=true;; - "-o" | "--optimize" ) optimize_database=true;; "-r" | "--recreate" ) recreate_database=true;; "-h" | "--help" ) helpFunc;; esac diff --git a/manpages/pihole-FTL.conf.5 b/manpages/pihole-FTL.conf.5 index 218f5e7a..5cb0481e 100644 --- a/manpages/pihole-FTL.conf.5 +++ b/manpages/pihole-FTL.conf.5 @@ -64,7 +64,7 @@ pihole-FTL.conf - FTL's config file On which port should FTL be listening? .br -\fBPRIVACYLEVEL=0|1|2|3|4\fR +\fBPRIVACYLEVEL=0|1|2|3\fR .br Which privacy level is used? .br @@ -76,8 +76,6 @@ pihole-FTL.conf - FTL's config file .br 3 - anonymous mode (hide everything) .br - 4 - disable all statistics -.br \fBIGNORE_LOCALHOST=no|yes\fR .br diff --git a/manpages/pihole.8 b/manpages/pihole.8 index e528bcde..a57eb05e 100644 --- a/manpages/pihole.8 +++ b/manpages/pihole.8 @@ -139,7 +139,7 @@ Available commands and options: -i, interface Specify dnsmasq's interface listening behavior .br -l, privacylevel <level> Set privacy level - (0 = lowest, 4 = highest) + (0 = lowest, 3 = highest) .br \fB-c, chronometer\fR [options] @@ -249,16 +249,47 @@ Options: echo -e "${OVER} ${TICK} ${str}" } +analyze_ports() { + # FTL is listening at least on at least one port when this + # function is getting called + echo -e " ${TICK} DNS service is listening" + # Check individual address family/protocol combinations + # For a healthy Pi-hole, they should all be up (nothing printed) + if grep -q "IPv4.*UDP" <<< "${1}"; then + echo -e " ${TICK} UDP (IPv4)" + else + echo -e " ${CROSS} UDP (IPv4)" + fi + if grep -q "IPv4.*TCP" <<< "${1}"; then + echo -e " ${TICK} TCP (IPv4)" + else + echo -e " ${CROSS} TCP (IPv4)" + fi + if grep -q "IPv6.*UDP" <<< "${1}"; then + echo -e " ${TICK} UDP (IPv6)" + else + echo -e " ${CROSS} UDP (IPv6)" + fi + if grep -q "IPv6.*TCP" <<< "${1}"; then + echo -e " ${TICK} TCP (IPv6)" + else + echo -e " ${CROSS} TCP (IPv6)" + fi + echo "" +} + statusFunc() { - # Determine if service is running on port 53 (Cr: https://superuser.com/a/806331) - if (echo > /dev/tcp/127.0.0.1/53) >/dev/null 2>&1; then + # Determine if there is a pihole service is listening on port 53 + local listening + listening="$(lsof -Pni:53)" + if grep -q "pihole" <<< "${listening}"; then if [[ "${1}" != "web" ]]; then - echo -e " ${TICK} DNS service is running" + analyze_ports "${listening}" fi else case "${1}" in "web") echo "-1";; - *) echo -e " ${CROSS} DNS service is NOT running";; + *) echo -e " ${CROSS} DNS service is NOT listening";; esac return 0 fi @@ -268,13 +299,13 @@ statusFunc() { # A config is commented out case "${1}" in "web") echo 0;; - *) echo -e " ${CROSS} Pi-hole blocking is Disabled";; + *) echo -e " ${CROSS} Pi-hole blocking is disabled";; esac elif grep -q "BLOCKING_ENABLED=true" /etc/pihole/setupVars.conf; then # Configs are set case "${1}" in "web") echo 1;; - *) echo -e " ${TICK} Pi-hole blocking is Enabled";; + *) echo -e " ${TICK} Pi-hole blocking is enabled";; esac else # No configs were found @@ -387,7 +418,7 @@ Whitelist/Blacklist Options: Debugging Options: -d, debug Start a debugging session - Add '-a' to enable automated debugging + Add '-a' to automatically upload the log to tricorder.pi-hole.net -f, flush Flush the Pi-hole log -r, reconfigure Reconfigure or Repair Pi-hole subsystems -t, tail View the live output of the Pi-hole log diff --git a/test/centos.Dockerfile b/test/centos_7.Dockerfile index 00543b67..00543b67 100644 --- a/test/centos.Dockerfile +++ b/test/centos_7.Dockerfile diff --git a/test/fedora.Dockerfile b/test/centos_8.Dockerfile index a06fc007..7444551b 100644 --- a/test/fedora.Dockerfile +++ b/test/centos_8.Dockerfile @@ -1,4 +1,4 @@ -FROM fedora:30 +FROM centos:8 ENV GITDIR /etc/.pihole ENV SCRIPTDIR /opt/pihole diff --git a/test/conftest.py b/test/conftest.py index 5b8be41e..cda861c1 100644 --- a/test/conftest.py +++ b/test/conftest.py @@ -69,7 +69,17 @@ def args(request): return '-t -d' -@pytest.fixture(params=['debian', 'centos', 'fedora']) +@pytest.fixture(params=[ + 'debian_9', + 'debian_10', + 'centos_7', + 'centos_8', + 'fedora_31', + 'fedora_32', + 'ubuntu_16', + 'ubuntu_18', + 'ubuntu_20' +]) def tag(request): ''' consumed by image to make the test matrix diff --git a/test/debian.Dockerfile b/test/debian_10.Dockerfile index 66436f1a..9b72fc6a 100644 --- a/test/debian.Dockerfile +++ b/test/debian_10.Dockerfile @@ -1,4 +1,4 @@ -FROM buildpack-deps:jessie-scm +FROM buildpack-deps:buster-scm ENV GITDIR /etc/.pihole ENV SCRIPTDIR /opt/pihole diff --git a/test/debian_9.Dockerfile b/test/debian_9.Dockerfile new file mode 100644 index 00000000..d6609ba3 --- /dev/null +++ b/test/debian_9.Dockerfile @@ -0,0 +1,16 @@ +FROM buildpack-deps:stretch-scm + +ENV GITDIR /etc/.pihole +ENV SCRIPTDIR /opt/pihole + +RUN mkdir -p $GITDIR $SCRIPTDIR /etc/pihole +ADD . $GITDIR +RUN cp $GITDIR/advanced/Scripts/*.sh $GITDIR/gravity.sh $GITDIR/pihole $GITDIR/automated\ install/*.sh $SCRIPTDIR/ +ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$SCRIPTDIR + +RUN true && \ + chmod +x $SCRIPTDIR/* + +ENV PH_TEST true + +#sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \ diff --git a/test/fedora_31.Dockerfile b/test/fedora_31.Dockerfile new file mode 100644 index 00000000..02dcb733 --- /dev/null +++ b/test/fedora_31.Dockerfile @@ -0,0 +1,16 @@ +FROM fedora:31 + +ENV GITDIR /etc/.pihole +ENV SCRIPTDIR /opt/pihole + +RUN mkdir -p $GITDIR $SCRIPTDIR /etc/pihole +ADD . $GITDIR +RUN cp $GITDIR/advanced/Scripts/*.sh $GITDIR/gravity.sh $GITDIR/pihole $GITDIR/automated\ install/*.sh $SCRIPTDIR/ +ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$SCRIPTDIR + +RUN true && \ + chmod +x $SCRIPTDIR/* + +ENV PH_TEST true + +#sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \ diff --git a/test/fedora_32.Dockerfile b/test/fedora_32.Dockerfile new file mode 100644 index 00000000..869efb2b --- /dev/null +++ b/test/fedora_32.Dockerfile @@ -0,0 +1,16 @@ +FROM fedora:32 + +ENV GITDIR /etc/.pihole +ENV SCRIPTDIR /opt/pihole + +RUN mkdir -p $GITDIR $SCRIPTDIR /etc/pihole +ADD . $GITDIR +RUN cp $GITDIR/advanced/Scripts/*.sh $GITDIR/gravity.sh $GITDIR/pihole $GITDIR/automated\ install/*.sh $SCRIPTDIR/ +ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$SCRIPTDIR + +RUN true && \ + chmod +x $SCRIPTDIR/* + +ENV PH_TEST true + +#sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \ diff --git a/test/test_000_build_containers.py b/test/test_000_build_containers.py index bca67989..3033249f 100644 --- a/test/test_000_build_containers.py +++ b/test/test_000_build_containers.py @@ -8,11 +8,17 @@ run_local = testinfra.get_backend( @pytest.mark.parametrize("image,tag", [ - ('test/debian.Dockerfile', 'pytest_pihole:debian'), - ('test/centos.Dockerfile', 'pytest_pihole:centos'), - ('test/fedora.Dockerfile', 'pytest_pihole:fedora'), + ('test/debian_9.Dockerfile', 'pytest_pihole:debian_9'), + ('test/debian_10.Dockerfile', 'pytest_pihole:debian_10'), + ('test/centos_7.Dockerfile', 'pytest_pihole:centos_7'), + ('test/centos_8.Dockerfile', 'pytest_pihole:centos_8'), + ('test/fedora_31.Dockerfile', 'pytest_pihole:fedora_31'), + ('test/fedora_32.Dockerfile', 'pytest_pihole:fedora_32'), + ('test/ubuntu_16.Dockerfile', 'pytest_pihole:ubuntu_16'), + ('test/ubuntu_18.Dockerfile', 'pytest_pihole:ubuntu_18'), + ('test/ubuntu_20.Dockerfile', 'pytest_pihole:ubuntu_20'), ]) -# mark as 'build_stage' so we can ensure images are build first when tests +# mark as 'build_stage' so we can ensure images are built first when tests # are executed in parallel. (not required when tests are executed serially) @pytest.mark.build_stage def test_build_pihole_image(image, tag): diff --git a/test/test_automated_install.py b/test/test_automated_install.py index 7351beb5..f6b5a87e 100644 --- a/test/test_automated_install.py +++ b/test/test_automated_install.py @@ -187,7 +187,55 @@ def test_FTL_detect_aarch64_no_errors(Pihole): ''') expected_stdout = info_box + ' FTL Checks...' assert expected_stdout in detectPlatform.stdout - expected_stdout = tick_box + ' Detected ARM-aarch64 architecture' + expected_stdout = tick_box + ' Detected AArch64 (64 Bit ARM) processor' + assert expected_stdout in detectPlatform.stdout + expected_stdout = tick_box + ' Downloading and Installing FTL' + assert expected_stdout in detectPlatform.stdout + + +def test_FTL_detect_armv4t_no_errors(Pihole): + ''' + confirms only armv4t package is downloaded for FTL engine + ''' + # mock uname to return armv4t platform + mock_command('uname', {'-m': ('armv4t', '0')}, Pihole) + # mock ldd to respond with ld-linux shared library + mock_command('ldd', {'/bin/ls': ('/lib/ld-linux.so.3', '0')}, Pihole) + detectPlatform = Pihole.run(''' + source /opt/pihole/basic-install.sh + create_pihole_user + funcOutput=$(get_binary_name) + binary="pihole-FTL${funcOutput##*pihole-FTL}" + theRest="${funcOutput%pihole-FTL*}" + FTLdetect "${binary}" "${theRest}" + ''') + expected_stdout = info_box + ' FTL Checks...' + assert expected_stdout in detectPlatform.stdout + expected_stdout = tick_box + (' Detected ARMv4 processor') + assert expected_stdout in detectPlatform.stdout + expected_stdout = tick_box + ' Downloading and Installing FTL' + assert expected_stdout in detectPlatform.stdout + + +def test_FTL_detect_armv5te_no_errors(Pihole): + ''' + confirms only armv5te package is downloaded for FTL engine + ''' + # mock uname to return armv5te platform + mock_command('uname', {'-m': ('armv5te', '0')}, Pihole) + # mock ldd to respond with ld-linux shared library + mock_command('ldd', {'/bin/ls': ('/lib/ld-linux.so.3', '0')}, Pihole) + detectPlatform = Pihole.run(''' + source /opt/pihole/basic-install.sh + create_pihole_user + funcOutput=$(get_binary_name) + binary="pihole-FTL${funcOutput##*pihole-FTL}" + theRest="${funcOutput%pihole-FTL*}" + FTLdetect "${binary}" "${theRest}" + ''') + expected_stdout = info_box + ' FTL Checks...' + assert expected_stdout in detectPlatform.stdout + expected_stdout = tick_box + (' Detected ARMv5 (or newer) processor') assert expected_stdout in detectPlatform.stdout expected_stdout = tick_box + ' Downloading and Installing FTL' assert expected_stdout in detectPlatform.stdout @@ -199,7 +247,7 @@ def test_FTL_detect_armv6l_no_errors(Pihole): ''' # mock uname to return armv6l platform mock_command('uname', {'-m': ('armv6l', '0')}, Pihole) - # mock ldd to respond with aarch64 shared library + # mock ldd to respond with ld-linux-armhf shared library mock_command('ldd', {'/bin/ls': ('/lib/ld-linux-armhf.so.3', '0')}, Pihole) detectPlatform = Pihole.run(''' source /opt/pihole/basic-install.sh @@ -211,8 +259,8 @@ def test_FTL_detect_armv6l_no_errors(Pihole): ''') expected_stdout = info_box + ' FTL Checks...' assert expected_stdout in detectPlatform.stdout - expected_stdout = tick_box + (' Detected ARM-hf architecture ' - '(armv6 or lower)') + expected_stdout = tick_box + (' Detected ARMv6 processor ' + '(with hard-float support)') assert expected_stdout in detectPlatform.stdout expected_stdout = tick_box + ' Downloading and Installing FTL' assert expected_stdout in detectPlatform.stdout @@ -224,7 +272,7 @@ def test_FTL_detect_armv7l_no_errors(Pihole): ''' # mock uname to return armv7l platform mock_command('uname', {'-m': ('armv7l', '0')}, Pihole) - # mock ldd to respond with aarch64 shared library + # mock ldd to respond with ld-linux-armhf shared library mock_command('ldd', {'/bin/ls': ('/lib/ld-linux-armhf.so.3', '0')}, Pihole) detectPlatform = Pihole.run(''' source /opt/pihole/basic-install.sh @@ -236,7 +284,32 @@ def test_FTL_detect_armv7l_no_errors(Pihole): ''') expected_stdout = info_box + ' FTL Checks...' assert expected_stdout in detectPlatform.stdout - expected_stdout = tick_box + ' Detected ARM-hf architecture (armv7+)' + expected_stdout = tick_box + (' Detected ARMv7 processor ' + '(with hard-float support)') + assert expected_stdout in detectPlatform.stdout + expected_stdout = tick_box + ' Downloading and Installing FTL' + assert expected_stdout in detectPlatform.stdout + + +def test_FTL_detect_armv8a_no_errors(Pihole): + ''' + confirms only armv8a package is downloaded for FTL engine + ''' + # mock uname to return armv8a platform + mock_command('uname', {'-m': ('armv8a', '0')}, Pihole) + # mock ldd to respond with ld-linux-armhf shared library + mock_command('ldd', {'/bin/ls': ('/lib/ld-linux-armhf.so.3', '0')}, Pihole) + detectPlatform = Pihole.run(''' + source /opt/pihole/basic-install.sh + create_pihole_user + funcOutput=$(get_binary_name) + binary="pihole-FTL${funcOutput##*pihole-FTL}" + theRest="${funcOutput%pihole-FTL*}" + FTLdetect "${binary}" "${theRest}" + ''') + expected_stdout = info_box + ' FTL Checks...' + assert expected_stdout in detectPlatform.stdout + expected_stdout = tick_box + ' Detected ARMv8 (or newer) processor' assert expected_stdout in detectPlatform.stdout expected_stdout = tick_box + ' Downloading and Installing FTL' assert expected_stdout in detectPlatform.stdout @@ -256,7 +329,7 @@ def test_FTL_detect_x86_64_no_errors(Pihole): ''') expected_stdout = info_box + ' FTL Checks...' assert expected_stdout in detectPlatform.stdout - expected_stdout = tick_box + ' Detected x86_64 architecture' + expected_stdout = tick_box + ' Detected x86_64 processor' assert expected_stdout in detectPlatform.stdout expected_stdout = tick_box + ' Downloading and Installing FTL' assert expected_stdout in detectPlatform.stdout @@ -274,7 +347,7 @@ def test_FTL_detect_unknown_no_errors(Pihole): theRest="${funcOutput%pihole-FTL*}" FTLdetect "${binary}" "${theRest}" ''') - expected_stdout = 'Not able to detect architecture (unknown: mips)' + expected_stdout = 'Not able to detect processor (unknown: mips)' assert expected_stdout in detectPlatform.stdout @@ -488,3 +561,37 @@ def test_validate_ip_invalid_letters(Pihole): ''') assert output.rc == 1 + + +def test_os_check_fails(Pihole): + ''' Confirms install fails on unsupported OS ''' + Pihole.run(''' + source /opt/pihole/basic-install.sh + distro_check + install_dependent_packages ${INSTALLER_DEPS[@]} + cat <<EOT > /etc/os-release + ID=UnsupportedOS + VERSION_ID="2" + EOT + ''') + detectOS = Pihole.run('''t + source /opt/pihole/basic-install.sh + os_check + ''') + expected_stdout = 'Unsupported OS detected: UnsupportedOS' + assert expected_stdout in detectOS.stdout + + +def test_os_check_passes(Pihole): + ''' Confirms OS meets the requirements ''' + Pihole.run(''' + source /opt/pihole/basic-install.sh + distro_check + install_dependent_packages ${INSTALLER_DEPS[@]} + ''') + detectOS = Pihole.run(''' + source /opt/pihole/basic-install.sh + os_check + ''') + expected_stdout = 'Supported OS detected' + assert expected_stdout in detectOS.stdout diff --git a/test/test_centos_fedora_support.py b/test/test_centos_fedora_support.py index 4b405920..8dd0ee5f 100644 --- a/test/test_centos_fedora_support.py +++ b/test/test_centos_fedora_support.py @@ -23,7 +23,7 @@ def mock_selinux_config(state, Pihole): '''.format(state=state.lower())) -@pytest.mark.parametrize("tag", [('centos'), ('fedora'), ]) +@pytest.mark.parametrize("tag", [('centos_7'), ('centos_8'), ('fedora_31'), ('fedora_32'), ]) def test_selinux_enforcing_exit(Pihole): ''' confirms installer prompts to exit when SELinux is Enforcing by default @@ -40,7 +40,7 @@ def test_selinux_enforcing_exit(Pihole): assert check_selinux.rc == 1 -@pytest.mark.parametrize("tag", [('centos'), ('fedora'), ]) +@pytest.mark.parametrize("tag", [('centos_7'), ('centos_8'), ('fedora_31'), ('fedora_32'), ]) def test_selinux_permissive(Pihole): ''' confirms installer continues when SELinux is Permissive @@ -55,7 +55,7 @@ def test_selinux_permissive(Pihole): assert check_selinux.rc == 0 -@pytest.mark.parametrize("tag", [('centos'), ('fedora'), ]) +@pytest.mark.parametrize("tag", [('centos_7'), ('centos_8'), ('fedora_31'), ('fedora_32'), ]) def test_selinux_disabled(Pihole): ''' confirms installer continues when SELinux is Disabled @@ -70,7 +70,7 @@ def test_selinux_disabled(Pihole): assert check_selinux.rc == 0 -@pytest.mark.parametrize("tag", [('fedora'), ]) +@pytest.mark.parametrize("tag", [('fedora_31'), ('fedora_32'), ]) def test_epel_and_remi_not_installed_fedora(Pihole): ''' confirms installer does not attempt to install EPEL/REMI repositories @@ -88,7 +88,7 @@ def test_epel_and_remi_not_installed_fedora(Pihole): assert not remi_package.is_installed -@pytest.mark.parametrize("tag", [('centos'), ]) +@pytest.mark.parametrize("tag", [('centos_7'), ('centos_8'), ]) def test_release_supported_version_check_centos(Pihole): ''' confirms installer exits on unsupported releases of CentOS @@ -105,7 +105,7 @@ def test_release_supported_version_check_centos(Pihole): assert expected_stdout in distro_check.stdout -@pytest.mark.parametrize("tag", [('centos'), ]) +@pytest.mark.parametrize("tag", [('centos_7'), ('centos_8'), ]) def test_enable_epel_repository_centos(Pihole): ''' confirms the EPEL package repository is enabled when installed on CentOS @@ -123,8 +123,8 @@ def test_enable_epel_repository_centos(Pihole): assert epel_package.is_installed -@pytest.mark.parametrize("tag", [('centos'), ]) -def test_php_upgrade_default_optout_centos(Pihole): +@pytest.mark.parametrize("tag", [('centos_7'), ]) +def test_php_upgrade_default_optout_centos_eq_7(Pihole): ''' confirms the default behavior to opt-out of installing PHP7 from REMI ''' @@ -139,8 +139,26 @@ def test_php_upgrade_default_optout_centos(Pihole): assert not remi_package.is_installed -@pytest.mark.parametrize("tag", [('centos'), ]) -def test_php_upgrade_user_optout_centos(Pihole): +@pytest.mark.parametrize("tag", [('centos_8'), ]) +def test_php_upgrade_default_continue_centos_gte_8(Pihole): + ''' + confirms the latest version of CentOS continues / does not optout + (should trigger on CentOS7 only) + ''' + distro_check = Pihole.run(''' + source /opt/pihole/basic-install.sh + distro_check + ''') + unexpected_stdout = info_box + (' User opt-out of PHP 7 upgrade on CentOS.' + ' Deprecated PHP may be in use.') + assert unexpected_stdout not in distro_check.stdout + # ensure remi was not installed on latest CentOS + remi_package = Pihole.package('remi-release') + assert not remi_package.is_installed + + +@pytest.mark.parametrize("tag", [('centos_7'), ]) +def test_php_upgrade_user_optout_centos_eq_7(Pihole): ''' confirms installer behavior when user opt-out of installing PHP7 from REMI (php not currently installed) @@ -158,8 +176,29 @@ def test_php_upgrade_user_optout_centos(Pihole): assert not remi_package.is_installed -@pytest.mark.parametrize("tag", [('centos'), ]) -def test_php_upgrade_user_optin_centos(Pihole): +@pytest.mark.parametrize("tag", [('centos_8'), ]) +def test_php_upgrade_user_optout_skipped_centos_gte_8(Pihole): + ''' + confirms installer skips user opt-out of installing PHP7 from REMI on + latest CentOS (should trigger on CentOS7 only) + (php not currently installed) + ''' + # Whiptail dialog returns Cancel for user prompt + mock_command('whiptail', {'*': ('', '1')}, Pihole) + distro_check = Pihole.run(''' + source /opt/pihole/basic-install.sh + distro_check + ''') + unexpected_stdout = info_box + (' User opt-out of PHP 7 upgrade on CentOS.' + ' Deprecated PHP may be in use.') + assert unexpected_stdout not in distro_check.stdout + # ensure remi was not installed on latest CentOS + remi_package = Pihole.package('remi-release') + assert not remi_package.is_installed + + +@pytest.mark.parametrize("tag", [('centos_7'), ]) +def test_php_upgrade_user_optin_centos_eq_7(Pihole): ''' confirms installer behavior when user opt-in to installing PHP7 from REMI (php not currently installed) @@ -181,7 +220,31 @@ def test_php_upgrade_user_optin_centos(Pihole): assert remi_package.is_installed -@pytest.mark.parametrize("tag", [('centos'), ]) +@pytest.mark.parametrize("tag", [('centos_8'), ]) +def test_php_upgrade_user_optin_skipped_centos_gte_8(Pihole): + ''' + confirms installer skips user opt-in to installing PHP7 from REMI on + latest CentOS (should trigger on CentOS7 only) + (php not currently installed) + ''' + # Whiptail dialog returns Continue for user prompt + mock_command('whiptail', {'*': ('', '0')}, Pihole) + distro_check = Pihole.run(''' + source /opt/pihole/basic-install.sh + distro_check + ''') + assert 'opt-out' not in distro_check.stdout + unexpected_stdout = info_box + (' Enabling Remi\'s RPM repository ' + '(https://rpms.remirepo.net)') + assert unexpected_stdout not in distro_check.stdout + unexpected_stdout = tick_box + (' Remi\'s RPM repository has ' + 'been enabled for PHP7') + assert unexpected_stdout not in distro_check.stdout + remi_package = Pihole.package('remi-release') + assert not remi_package.is_installed + + +@pytest.mark.parametrize("tag", [('centos_7'), ('centos_8'), ]) def test_php_version_lt_7_detected_upgrade_default_optout_centos(Pihole): ''' confirms the default behavior to opt-out of upgrading to PHP7 from REMI @@ -204,7 +267,7 @@ def test_php_version_lt_7_detected_upgrade_default_optout_centos(Pihole): assert not remi_package.is_installed -@pytest.mark.parametrize("tag", [('centos'), ]) +@pytest.mark.parametrize("tag", [('centos_7'), ('centos_8'), ]) def test_php_version_lt_7_detected_upgrade_user_optout_centos(Pihole): ''' confirms installer behavior when user opt-out to upgrade to PHP7 via REMI @@ -229,7 +292,7 @@ def test_php_version_lt_7_detected_upgrade_user_optout_centos(Pihole): assert not remi_package.is_installed -@pytest.mark.parametrize("tag", [('centos'), ]) +@pytest.mark.parametrize("tag", [('centos_7'), ('centos_8'), ]) def test_php_version_lt_7_detected_upgrade_user_optin_centos(Pihole): ''' confirms installer behavior when user opt-in to upgrade to PHP7 via REMI diff --git a/test/ubuntu_16.Dockerfile b/test/ubuntu_16.Dockerfile new file mode 100644 index 00000000..d1107082 --- /dev/null +++ b/test/ubuntu_16.Dockerfile @@ -0,0 +1,16 @@ +FROM buildpack-deps:xenial-scm + +ENV GITDIR /etc/.pihole +ENV SCRIPTDIR /opt/pihole + +RUN mkdir -p $GITDIR $SCRIPTDIR /etc/pihole +ADD . $GITDIR +RUN cp $GITDIR/advanced/Scripts/*.sh $GITDIR/gravity.sh $GITDIR/pihole $GITDIR/automated\ install/*.sh $SCRIPTDIR/ +ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$SCRIPTDIR + +RUN true && \ + chmod +x $SCRIPTDIR/* + +ENV PH_TEST true + +#sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \ diff --git a/test/ubuntu_18.Dockerfile b/test/ubuntu_18.Dockerfile new file mode 100644 index 00000000..2f63ea89 --- /dev/null +++ b/test/ubuntu_18.Dockerfile @@ -0,0 +1,16 @@ +FROM buildpack-deps:bionic-scm + +ENV GITDIR /etc/.pihole +ENV SCRIPTDIR /opt/pihole + +RUN mkdir -p $GITDIR $SCRIPTDIR /etc/pihole +ADD . $GITDIR +RUN cp $GITDIR/advanced/Scripts/*.sh $GITDIR/gravity.sh $GITDIR/pihole $GITDIR/automated\ install/*.sh $SCRIPTDIR/ +ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$SCRIPTDIR + +RUN true && \ + chmod +x $SCRIPTDIR/* + +ENV PH_TEST true + +#sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \ diff --git a/test/ubuntu_20.Dockerfile b/test/ubuntu_20.Dockerfile new file mode 100644 index 00000000..caa6261f --- /dev/null +++ b/test/ubuntu_20.Dockerfile @@ -0,0 +1,17 @@ +FROM buildpack-deps:focal-scm + +ENV GITDIR /etc/.pihole +ENV SCRIPTDIR /opt/pihole + +RUN mkdir -p $GITDIR $SCRIPTDIR /etc/pihole +ADD . $GITDIR +RUN cp $GITDIR/advanced/Scripts/*.sh $GITDIR/gravity.sh $GITDIR/pihole $GITDIR/automated\ install/*.sh $SCRIPTDIR/ +ENV PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:$SCRIPTDIR +ENV DEBIAN_FRONTEND=noninteractive + +RUN true && \ + chmod +x $SCRIPTDIR/* + +ENV PH_TEST true + +#sed '/# Start the installer/Q' /opt/pihole/basic-install.sh > /opt/pihole/stub_basic-install.sh && \ @@ -1,10 +1,16 @@ [tox] -envlist = py36 +envlist = py37 [testenv] whitelist_externals = docker deps = -rrequirements.txt -commands = docker build -f test/debian.Dockerfile -t pytest_pihole:debian . - docker build -f test/centos.Dockerfile -t pytest_pihole:centos . - docker build -f test/fedora.Dockerfile -t pytest_pihole:fedora . +commands = docker build -f test/debian_9.Dockerfile -t pytest_pihole:debian_9 . + docker build -f test/debian_10.Dockerfile -t pytest_pihole:debian_10 . + docker build -f test/centos_7.Dockerfile -t pytest_pihole:centos_7 . + docker build -f test/centos_8.Dockerfile -t pytest_pihole:centos_8 . + docker build -f test/fedora_31.Dockerfile -t pytest_pihole:fedora_31 . + docker build -f test/fedora_32.Dockerfile -t pytest_pihole:fedora_32 . + docker build -f test/ubuntu_16.Dockerfile -t pytest_pihole:ubuntu_16 . + docker build -f test/ubuntu_18.Dockerfile -t pytest_pihole:ubuntu_18 . + docker build -f test/ubuntu_20.Dockerfile -t pytest_pihole:ubuntu_20 . pytest {posargs:-vv -n auto} -m "not build_stage" ./test/ |