spamdyke

spamdyke is an open source software written in C as a drop-in filter for the qmail application, replacement for sendmail, designed to provide numerous features, including graylisting, connection-time blacklisting, DNS RHSBL/RBL checking, improved logging, and sender MX checking.

Features at a glance

spamdyke has been engineered in such a way that it provides TLS and SMTP AUTH to unpatched qmail servers. It can be used to monitor and intercept SMTP connections between a qmail server and a remote host.

The filter is capable of blocking spam while the spammer is still connected, without requiring any additional storage or processing. spamdyke features numerous anti-spam filters, as well as various features that will enhance qmail.

spamdyke includes a wide variety of anti-spam filters, organized in four main categories, such as reverse DNS filters, whitelist and blacklist filters, graylist filters and other filters that can be viewed at a glance in the next section.

Included anti-spam filters

Among the reverse DNS filters, we can mention support for requiring reverse DNS names, support for requiring reverse DNS names to resolve with IP addresses, support for requiring reverse DNS names to be real instead of dynamically assigned, as well as support for requiring a valid MX (Mail Exchanger) for sender's domain name.

Whitelist/blacklist filters include the ability to search local IP addresses and reverse DNS whitelist or blacklist files, search DNS RBLs (real-time whitelists and blacklists), search DNS RHSBLs (righthand-side whitelists and blacklists), search recipient and sender whitelist or blacklist files, as well as to search message headers for certain content.

On the other hand, the graylist filters will allow you to greylist certain recipient domains, to greylist connections from certain reverse DNS domains or IP addresses, to skip greylisting for certain domains, reverse DNS domains or IP addresses, and to implement maximum and minimum ages for greylist entries.

What is new in this release:

• Rearranged the test scripts to put them in folders by category. This just makes the directory listing a little more manageable.
• Corrected some typos in the README file. Thanks to John Mendoza for reporting those.
• Fixed a very obscure bug in spamdyke_log(): on Linux systems (possibly only 64-bit systems), vsyslog() occasionally will not print all the variable arguments. One way was found to trigger this behavior -- when the rdns-blacklist-dir filter is activated from a configuration directory.
• Fixed a bug in find_domain() that could cause segfaults when parsing certain invalid formats. Thanks to Gary Gendel for reporting this one.
• Added a backup/restore feature to the "run" script in the "tests" folder to save a copy of the most critical system and qmail files before running any scripts. This is needed because some of the scripts alter those files and, if they don't run correctly or are cancelled, the originals are lost.
• Added a "-skipcompile" flag to the "run" script in the "tests" folder to skip reconfiguring and recompiling all of the binaries when the script is run.
• Changed the "run" script in the "tests" folder to empty qmail's queue before and after the tests are run.
• Changed the "run" script in the "tests" folder to compare the current system and qmail configuration files to the latest backup after every script finishes. If they don't match, the latest backup is restored. If they still don't match, the script stops with an error.
• Changed nihdns_query() to accept an optional "preferred" type of response. If multiple types are queried, it will wait for at least one timeout period for an answer of that type to arrive instead of always accepting the first answer to arrive. It will accept a saved answer before resending the queries, however.
• NOT BACKWARDS COMPATIBLE: Changed nihdns_mx() to prefer an MX record over an A record, if both exist. Given the choice, the MX record will be checked for validity and the A record will be ignored. Thanks to Bruce Schreiber for suggesting this one.
• Fixed filter_level() and smtp_filter() to disregard whitelisting and require authentication if the "filter-level" option is set to "require-auth", as the documentation says it should. Thanks to Arne for reporting this one.
• Changed nihdns_create_packet() to strip trailing dots from names before using them in DNS queries. A trailing dot is the traditional way to tell libc's resolver not to append the local domain name and many sysadmins expect to have to use it. Since spamdyke never appends the local domain and doesn't use libc's resolver, it isn't necessary and causes lookups to fail. Thanks to Dossy Shiobara for reporting this one.
• Changed middleman() to always send a "STARTTLS" response to "EHLO" as a continuation, never as the last line (only when spamdyke is inserting "STARTTLS"). This works around a bug in the Android mail client, which only looks for "STARTTLS" as a continuation. Thanks to Jonas Pasche for writing about how to work around this bug on his blog.
• NOT BACKWARDS COMPATIBLE: Changed the meaning of "whitelisted" to only exempt the connection from spamdyke's spam filters; whitelisting no longer allows the connection to relay mail. This means spamdyke will now only set the RELAYCLIENT environment variable if the "relay-level" option is set to "allow-all". Relaying must now be controlled through tcpserver or xinetd. Many thanks to Eric Shubert for suggesting and debating this with me.
• NOT BACKWARDS COMPATIBLE: Removed the "access-file" and "rejection-text-access-denied" options because they were only needed for controlling relaying. Also removed the test scripts that exercised them and modified many other test scripts that used them.
• NOT BACKWARDS COMPATIBLE: Removed the "no-check" value from the "relay-level" option and changed the meaning of the "normal" value to use the logic previously assigned to "no-check".
• Added the option "reject-sender" to take multiple values. If the value "not-local" is given, the sender will be rejected if the domain name is not hosted locally. If the value "authentication-mismatch" is given, the sender will be rejected if the sender address does not exactly match the username given during authentication (or if the authentication username is not an email address, the sender username must match the authentication username). If the value "authentication-domain-mismatch" is given, the sender will be rejected if the domain name is not part of the username given during authentication. Thanks to Mark Frater for suggesting this one.
• Added the options "rejection-text-sender-not-local" and "rejection-text-sender-authentication-mismatch" to set the rejection text given when the "reject-sender" option's filters are triggered.
• NOT BACKWARDS COMPATIBLE: Removed the option "reject-missing-sender-mx" and folded its filter into the "reject-sender" filter's "no-mx" option.
• NOT BACKWARDS COMPATIBLE: Renamed the option "rejection-text-missing-sender-mx" to "rejection-text-sender-no-mx".
• NOT BACKWARDS COMPATIBLE: Renamed the option "reject-identical-sender-recipient" to "reject-recipient" with the value "same-as-sender". The functionality remains the same.
• NOT BACKWARDS COMPATIBLE: Renamed the option "rejection-text-identical-sender-recipient" to "rejection-text-recipient-same-as-sender".
• NOT BACKWARDS COMPATIBLE: Renamed the option "local-domains-file" to "qmail-rcpthosts-file". The naming has always been confusing, since qmail distinguishes between domains that should be accepted by qmail-smtpd during SMTP (rcpthosts) and domains that are actually hosted locally with mailboxes on the local filesystem (locals). These options have always meant the former, but now that spamdyke needs to know both lists of domains, it's time to rename them. This option is also now allowed in configuration directories.
• NOT BACKWARDS COMPATIBLE: Removed the option "local-domains-entry" because supplying domains that can be accepted during SMTP to spamdyke only (but not qmail) will cause inconsistent results during recipient validation. If a domain is to be accepted during SMTP, it should be added to the control files used by both spamdyke and qmail.
• Added CDB searching code in cdb.[ch] to read DJB's "constant database" files during recipient validation. The format of these files is claimed (by DJB) to be fast and efficient. Don't believe the hype...
• Added the option "qmail-morercpthosts-cdb" to allow CDB files to be provided that contain lists of domains for which mail should be accepted during SMTP. Does anyone actually use this qmail "feature"?
• Poured over qmail's documentation and source code to figure out exactly how it determines where to deliver a message. The documentation is frequently in error and extensive testing was required to discover the truth. The resulting procedure is encapsulated in a flowchart in the documentation folder.
• Added the "generator" program to create test scripts to check every possible path through the recipient validation flowchart, both with spamdyke in place and without (to check the flowchart is correct). A program to generate the scripts was required, since there are nearly 250K possible paths to test.
• Added the value "invalid" to the option "reject-recipient" to check if a local recipient address exists before accepting a message. This validation process uses the same logic as qmail when deciding whether/where to deliver a message, so no extra steps are needed to make this work (e.g. maintaining a list of valid addresses in a separate file). If this process determines a local address is valid, delivery is guaranteed. This option should eliminate qmail's habit of sending backscatter spam.
• Added the value "unavailable" to the option "reject-recipient" to check if a local recipient is accepting mail at the moment. Probably as a holdover from the elder days when people actually edited .qmail files by hand, qmail checks file permissions on files and folders before delivering a message. If they are set to certain values, qmail will queue the message until the permissions are fixed or bounce the message if is queued too long. In these enlightened times, such permissions are more likely to be due to an error or oversight than deliberate intent.
• Added the options "qmail-assign-cdb", "qmail-defaultdelivery-file", "qmail-envnoathost-file", "qmail-locals-file", "qmail-me-file", "qmail-percenthack-file" and "qmail-virtualdomains-file" to allow spamdyke to use different control files than qmail. It's very unlikely anyone will ever need these options (and it would be unwise to use them), but they're available just in case.
• Added the option "rejection-text-recipient-invalid" to set the rejection text when the "invalid" filter on "reject-recipient" is triggered.
• Added the option "rejection-text-recipient-unavailable" to set the rejection text when the "unavailable" filter on "reject-recipient" is triggered.
• Removed the function filter_recipient_local() and moved its logic into filter_recipient_valid().
• Removed the function filter_recipient_relay() and moved its logic into filter_recipient_valid().
• Changed the "help" option to just show a listing of available options without help text.
• Added the "more-help" option to show the full listing of options with all help text.
• Added the options "ip-relay-entry", "ip-relay-file", "rdns-relay-entry" and "rdns-relay-file" to allow relaying from specific IPs and/or rDNS names, since whitelisting no longer implies the ability to relay. If any of these options are matched, the RELAYCLIENT variable will be set before qmail is started.
• Created the "create_cdb" program to generate CDB files of arbitrary size, filled with random data, for testing spamdyke's CDB validation routines. create_cdb also has the ability to corrupt the generated CDB in seven ways; this makes for more specific testing than simply using a file of random garbage.
• Removed all uses of the TESTSD_* environment variables from the test scripts and replaced them with appropriate invocations of dnsdummy. This allows the test scripts to run without potential interference from external DNS changes and without needing a running spamdyke server to find example values.
• Fixed smtp_filter() and middleman() to clear the list of saved recipient addresses after printing the log messages. This prevents duplicate log messages when multiple email messages are delivered in the same connection. Thanks to Teodor Milkov and David Davidov for reporting this one.
• Added the "-skippatched" and "-skipunpatched" flags to the "run" scripts to skip any tests that require a patched or unpatched version of qmail, respectively.
• Fixed a minor bug in find_username() that would truncate the last character of the username when no domain is given. This hasn't been a problem since spamdyke rejects recipient addresses without domain names anyway, but one of the recipient validation test scripts found it.
• Added the option "tls-dhparams-file" option to read DH params from a file for creating ephemeral keys during SSL/TLS key negotiation. Thanks to Marc Gregel for suggesting this one.
• Changed all error messages to output the filename, function name and line number that generated them, just like the debug and excessive messages.
• Added a new log level, LOG_LEVEL_CONFIG_TEST, for config-test error messages. The level is treated much the same as LOG_LEVEL_ERROR except the filename, function name and line numbers are not printed.
• Added a new decision level, FILTER_DECISION_AUTHENTICATED for authenticated connections. The filter routines use this level to distinguish between connections that should be unfiltered due to authentication versus whitelisting.
• Added a new config option type: CONFIG_TYPE_ALIAS. Options of this type are aliases for other options. This eliminates the duplication of values and potential for oversights in the graylist/greylist options.
• Added some code to the "run" script in the "tests" directory to try to detect core dumps. Some of the tests will declare success even if spamdyke segfaults and cuts off the output prematurely.
• Removed the unused functions reset_rejection() and skip_cfws().
• Discovered spamdyke cannot read all the files it needs for recipient validation during normal operation because they are owned by different users with restrictive permissions and spamdyke does not run as root. I'm not sure how I missed that, but it completely moots more than a year of work.
• Moved all the recipient valiation code into an external program named "spamdyke-qrv". This program is meant to only perform recipient validation and nothing else, so it should be safe to run as root (at least safer than running spamdyke as root).
• Removed the options "qmail-assign-cdb", "qmail-defaultdelivery-file", "qmail-envnoathost-file", "qmail-locals-file", "qmail-me-file" and "qmail-percenthack-file" from spamdyke, since the recipient validation code is gone.
• Added the option "recipient-validation-command" for passing the path to spamdyke-qrv, which will be called when recipient validation is needed.

What is new in version 4.3.1:

• This release fixes a couple of processing bugs in the new header blacklist filter and an access violation that can lead to crashes.
• The header blacklist should is safe to enable now.

What is new in version 4.3.0:

• It adds the ability to filter messages based on the content of their headers. It fixes some small bugs, a compile error on Debian 7 and a major series of bugs that could result in buffer overflows (possibly remotely exploitable, depending on the configuration options). Please upgrade immediately!

What is new in version 4.2.0:

• Changed read_file() to return the number of usable lines read, instead of the total number of lines (including comments and whitespace).
• Fixed a huge thinko in many calls to read_file() -- when the function returns 0, the returned value is NULL. This was causing spamdyke to crash when no content was read from files by "dns-blacklist-file", "dns-whitelist-file", "rhs-blacklist-file", "rhs-whitelist-file" and "hostname-file". Thanks to David Stiller for reporting this one and providing a lot of help in tracking it down.
• Added the option "tls-cipher-list" for specifying the list of ciphers to use in SSL/TLS connections. This won't be an option many people will ever use, but in specific setups it is required. Thanks to Chris Boulton for suggesting this one and producing a patch to implement it.
• Added a new value to "tls-level": "smtp-no-passthrough" to allow spamdyke to offer TLS but prevent it from passing TLS through to qmail if the SSL library cannot be initialized for some reason.
• Fixed a bug in smtp_filter that allowed open relaying when spamdyke was configured with "local-domains-entry" instead of "local-domains-file".
• Moved code from do_spamdyke() that set stdin and stdout sockets to non-blocking into tls_read() and tls_write() instead. Setting the sockets to non-blocking through the entire run was causing some strange behavior where logging would stop after a series of large inputs.
• Refactored the address parser (yet again) to fix a bug that wasn't handling routing addresses properly. Thanks to Chris Boulton for reporting this one.
• Fixed process_config_file() to not reset a "multiple" value to default if it was deliberately cleared during configuration.
• Fixed prepare_settings() to initialize all default values before processing the command line or configuration files so a "multiple" value can be cleared during configuration.
• Fixed configure.ac to use a gcc #pragma command to treat format warnings as errors instead of relying on AC_LANG_WERROR (which doesn't always work).
• Added the options "dns-query-type-a", "dns-query-type-mx", "dns-query-type-ptr" and "dns-query-type-rbl" to limit the types of DNS queries that can be sent for different purposes. Thanks to Teodor Milkov for suggesting this one.
• Fixed a bug that caused a timeout whenever a post-RCPT filter is triggered on a non-local address. spamdyke is supposed to close the connection to qmail and wait for its exit, but instead was just waiting for its exit, leading to unnecessary timeouts. Thanks to Ulrich C. Manns for reporting this one.
• Fixed a typo in policy.php.example. Thanks to Richard Lamse for reporting this one.
• Fixed compiler warnings on Fedora 11. Thanks to Ertan Orhan for reporting this one.
• Fixed a bug in sendrecv where an uninitialized variable was causing erroneous stalls and timeouts in CentOS 5.5.

What is new in version 4.1.0:

• Several bug fixes, including a TLS/SSL issue that can cause spamdyke to hang forever. Some new features, including rejecting messages when the sender and recipient addresses are the same.

What is new in version 4.0.10:

• A bug in Plesk 9 that sets the remote IP address to a name instead of an IP address was worked around.

What is new in version 4.0.9:

• This release fixes a bug in the address parser that was preventing some sender/recipient blacklist/whitelist entries from matching.
• It also fixes a bug in the configuration testing feature that was unable to locate the spamdyke binary if it was outside the current directory.

What is new in version 4.0.8:

• A workaround was added for a design flaw in DJB's multilog that can cause corrupted log messages when the server is under load.

What is new in version 4.0.6:

• Keywords for the IP-in-rDNS filter are now correctly terminated so they can be matched.
• AUTH advertisements from a patched qmail are now completely removed when the "smtp-auth-level" option is "always" or "always-encrypted".
• The name of the matching RBL/RHSBL is now logged when multiple RBLs/RHSBLs are used instead of always logging the first one.
• Slow remote servers are now handled politely so spamdyke will not consume 100% CPU.
• Using the full logging feature with a "log-level" option set to "excessive" no longer sends log output to the remote server when spamdyke is used on Plesk servers.

Requirements:

• qmail