################################################################### # pf-badhost 0.6 Release Notes / Changelog # Copyright 2018-2021 Jordan Geoghegan ################################################################### # Preamble The pf-badhost v0.6 release features a great number of usability, efficiency and performance improvements. Release Notes: - Improvements to URL Fetching: * Add support for aria2c HTTP/FTP fetch utility. * Add support for HTTP compression when using aria2c, curl or wget for fetching block lists. * Mirror RIR statistics exchage data on Github. This allows for fast CDN speeds + transport compression. * We now opportunistically make use of the best available URL fetch utility, with overrides enabled via long options. * aria2c, curl and wget are now fed URLs over a pipe, which eliminates issues with argument length limits and maximizes potential socket and cache reuse. * Curl now properly follows redirects. * Enforce strict timeout limits to avoid hangs or infinite reads when fetching blocklists (functionality varies amoung supported URL fetch utilities). * FreeBSD/DragonflyBSD "fetch" utility now supports custom user agents. * Explicitely disable URL globbing. - Use parallelism/concurrency where appropriate: * Blocklist URLs and RIR data sets can now be fetched in parallel thanks to aria2c. * Geoblocking data is now parsed/extracted in parallel. - New feature flags: * All non-option arguments are now interprated as paths to local blocklists. * Add '-C' flag to specify a configuration file to load. * Add '-P' flag to control maximum number of parallel processes. * Add '-U' option to specify HTTP user agent. * Add '-c' flag to check if single IP address exists in PF table. * Add '-f' flag to load blocklist from local file system. * Add '-i' flag to check if list of IP addresses exists in PF table. * Add '-m' flag to specifiy a path to a local list of ASN blocklist URLs. * Add '-s' flag to specify a path to a local list of two letter ISO-3166 country codes to geoblock. * Add '-t' flag to specifiy a single ASN blocklist URL. * Add '-v' flag to print version information. * Add '-o mirror' long option to improve geoblocking dataset retreival times. * Add '-o' long options to set awk/grep/fetch utility preference. Please see manpage for details. * Add '-o' long options: 'print-debug', 'show-stats', 'show-stats-address' and 'show-stats-global' to print debugging information and statistics. Please see manpage for details. - Performance Improvements: * Add support for HTTP compression when using aria2c, curl or wget for fetching block lists. * The ASN lookup/blocking function now uses significantly less network and CPU resources. * Changes to the geoblocking and URL fetching code has allowed for the removal of a bunch of superflous workaround code. * Many internal functions have been adjusted to avoid fork+exec, temp file and subshell overhead. * Many internal functions are now written in pure forkless shell, facilitating some rather impressive micro-benchmark performances. * We now make use of modern shell features to eliminate numerous potential footguns. * Greatly improved geoblocking function performance, especially on devices without an FPU and/or platforms that ship old versions of awk. * Geoblocking datasets can now optionally be fetched from a Github mirror, greatly improving download times. * Geoblocking datasets are now parsed concurrently in a single pass -- users blocking many countries should notice a significant performance improvement. - Configuration Changes: * The '-b' flag no longer requires an argument as it determines bogon filtering type based on specified IP family. * Use of 'set -o pipefail' is now required. This means you'll need a modern oksh, ksh93, bash, or zsh shell to run pf-badhost. * OS type decleration with the '-O' flag has been removed as OS type is now auto-detected. * Compressed lists may now only be declared in the "Custom Lists" section from a config file specified with '-C'. * Use of doas/sudo is now determined automatically, with doas being preferred. Use long options to override. * The '-l' and '-u' flag functionalities have been swapped to promote harmony between similarly used flags in unbound-adblock. * The '-o [no-]whitelist flags were renamed to '-o [no-]custom-filter' to reflect the features versatility. * Rename "-T block_exit" to "-T block-exit" to ensure consistency with established naming conventions. --- * User Configuration Area Changes: The "User Configuration Area" at the top the script has been relocated to an external file. Config file path may be specified with the '-C' flag. This change allows for users to easily switch between a command line or config file driven workflow. As such, pf-badhost may now be used as a versatile commandline utility with many potential use cases. If no config file is specified, then all options will be read from the commandline. - Correctness: * Greatly improved commandline options parsing and error checking. * Greatly improved SSH authlog parsing. * "Extended" RIR statistics are now utilized. * Use specific integer/boolean checks to avoid undefined behavior. * An error is now thrown when an improperly formatted country code is detected, rather than silently ignoring it. * Validation is now performed on custom rules specified with the '-r' and '-w' flags. * Stop referencing RFC-3330 and RFC-5156; replace with references to IANA IPv4/IPv6 Special-Purpose Address Registry. * pf-badhost is now capable of properly handling Windows line endings (carriage returns). * Add a 15 second timeout to ASN lookups to prevent potential hangs. * Make sure ZSH doesn't clear the 'set -efo pipefail' options. - Misc: * Add support for 'ugrep' - a fast, portable, multithreaded grep replacement written in modern C++. * Add support for "goawk", a POSIX compliant awk implementation written in Go.