Known Bots v6.0.3

• This update contains an important bugfix to handle malformed UTF-8 in user agent strings.
• This update will simply ignore any user agents with malformed UTF-8, avoiding errors when trying to send updates via the API. These user agents are invalid and so there is no point undertaking any further analysis - thus they are silently discarded.
• This new version also contains an additional CLI tool for importing user agents from a text file, for testing purposes.

Minor bug fix - no need to update unless you are experiencing problems sending bot updates.

• when sending bots via email, include the bot list as an attachment rather than in the body
• new CLI tool to send bots via email directly, used for debugging bot sending issues

NA

v5.0.0 is a major rewrite of the core functionality of this addon aimed at improving processing speed, bot detection sophistication and greatly enhancing our ability to identify new bots.

Note that the options have changed - so please check the options after upgrading. More information about each option is provided on the main addon page.

• major rewrite - no longer use "bot|spider|crawl" search strings and false-positive lists to identify possible bots, rely instead on search strings supplied by API to identify valid browsers and store them directly in the database rather than the SimpleCache, ready for emailing
• more complete agent reprocessing - check for valid browsers and ignored agents
• change the core userAgentMatchesRobot function to use strpos instead of preg_match, it's much faster and won't fall over with extremely high numbers of bot match strings
• allow BotFetcher to be manually configured to bypass untrusted http agent - used for testing when API source is on a .local domain. Default action remains to use the untrusted http agent to allow for proxying outbound API calls.
• change email cron to daily send
• using new v2 API from KnownBots
• replace generic bots with complex (regex) based searches
• add "Fetch new bots" button to Known Bots List in admin UI
• automatically reprocess user agents after loading new bot data
• new Cli command for reprocessing user agents, including the option to force all user agents to be reprocessed
• improvements to user agent test in admin ui to be more descriptive
• bcc additional email address to keep them private
• bugfix: don't linkify known bot list when no links supplied

• This release includes additional sanity checks to prevent bad data returned from the API from breaking the forums.
• If any of the data returned by the API is not in the exact format we expect, the entire download is discarded and no changes applied to the forum. An error message will be logged prompting further investigation.
• After upgrading to 4.0.1, you should manually force a fetch of new API data by executing the following command from your forum root:

php cmd.php known-bots:fetch -f