3k Associates
NetMail/3000
New features and enhancements in the new version of NetMail/3000:
(B.08, as of 2001/09/05)
----------------------------------------------------------
Selectively enable "blacklists" by mailbox
Spammers are constantly working to find new ways to deliver their junk
mail into your mailbox, and likewise several organizations are constantly
working on new methods to keep the unwanted "stuff" out. NetMail/3000 and
DeskLink have had advanced filtering capabilities for many years now, but
recently we have significantly improved the capability and flexibility of
these features.
As of B.07 01/08/24 versions of NetMail/3000 you can now selectively activate
the "blacklist filtering" policy you setup (in the NetMail/3000 background
job) "per mailbox". The "Advertise Mailbox" attribute (configurable in the
Netmaint program in the MAIL-USERS screen) now has two new allowable values,
for a total of four:
"Y" - Advertise (i.e. allow SMTP based queries on this mailbox name) but
do NOT utilize any site-defined blacklists.
"N" - Don't advertise the mailbox and do NOT utilize any blacklists
"C" - Advertise the mailbox and DO utilize any site-defined blacklists
"H" - Don't advertise and DO utilize any site-defined blacklists
Note that the old default values were "Y" or "N" - so by default with this
update blacklists will NOT be enabled for your users unless you change their
"advertise" flag in netmaint. This was done intentionally since with any
blacklist there is the possibility of "false positives" where legitimate mail
can be refused, and we feel the mail system defaults should refuse as little
communications as possible. Keep in mind that the flags can be changed
individually with netmaint, or en masse using a tool like Query.
----------------------------------------------------------
Fully configurable spam-filter "blacklists"
In the past few months, several of the common and popular Internet blacklist
databases have gone offline or been moved. Rather than constantly updating
the hard-coded domain names in our code, we have made all the services
configurable via CIVars in the background job. There are 5 possible services
you can enable for your system, and you can now redirect each of these 5
to any service you like. [We will provide some defaults in our job stream, but
for a list of other possibilities contact 3k/Entrix technical support, or
check out the network-abuse newsgroups, where updates and statistics on the
various services are posted from time to time.]
CIVariables are provided for both the domain name of the blacklist's DNS
servers (Internet blacklists work like DNS servers, returning positive
results for lookups of hosts that are identified as high-risk hosts) as well
as a "contact" url - which will be included in any error logs and error
messages returned to senders of refused messages. For example:
!setvar NETMAIL_ORCA_LIST ".or.orbl.org"
!setvar NETMAIL_ORCA_CONTACT "www.orbl.org"
!setvar NETMAIL_MAPS_LIST ".relays.ordb.org"
!setvar NETMAIL_MAPS_CONTACT "www.ordb.org"
!setvar NETMAIL_ORBS_LIST ".orbs.dorkslayers.com"
!setvar NETMAIL_ORBS_CONTACT "www.dorkslayers.com"
!setvar NETMAIL_IMRSS_LIST ".relays.mail-abuse.org"
!setvar NETMAIL_IMRSS_CONTACT "www.mail-abuse.org"
!setvar NETMAIL_DSSL_LIST ".dssl.imrss.org"
!setvar NETMAIL_DSSL_CONTACT "www.imrss.org"
The "names" in the variables (ORCA, MAPS, ORBS, etc) merely correspond to
the original/default names of the services. You can redefine each one to
point to any other service you choose (i.e. the ORCA list need not actually
point to anything to do with the old ORCA service).
Each "..._LIST" name is the domain name appended to DNS lookups in order
to use that service. Each value MUST begin with a period as in the example.
The "..._CONTACT" variables should contain a short URL, which will be
recorded in your log files when mail is refused, as well as returned as part
of the error message to senders whose mail is "bounced" -- hopefully helping
to point legitimate users to a resource where they can understand why people
are refusing their email - and get their servers fixed (and subsequently
removed from the blacklist).
----------------------------------------------------------
Full support for byte stream files and posix filenames
As of versions B.07 01/08/22 NetMail/3000 correctly parses and attaches
MPE bytestream files. Also, when attaching a file from the 3000, NetMail/3000
will look for an MPE filecode (first) to assist in identifying the type of
file (and how to encode/label it). If there is no filecode (i.e. "0") and
the file was expressed as a posix (HFS) name, NetMail/3000 will also check the
file extension (i.e. ./myfile.txt /SYS/PUB/Inventory.xls ) to assist in
recognizing and labelling the attachment appropriately.
----------------------------------------------------------
Specify "disposition" of files attached to messages
We have had requests from users to override NetMail/3000's default behavior
when attaching text files. By default, when we attach a text file to a mail
message, the attachment is "tagged" with a label called "inline" - which
recommends to the receiving mail system that the attachment be viewed as
part of the standard message. The alternative being having the user be
prompted to launch a "helper" application (word, notepad, etc) to "view" the
attachment.
While in most cases this makes sense, there are cases where you may want to
force the behavior of recommending that the recipient view the attachment
with a "helper". Also, "inline" attachments cannot also provide "filenames"
in the label (at least several mailers have this limitation - including
Microsoft Outlook), so if you want to provide a filename with the text file
you are attaching, the attachment needs to be "labelled" by the mailer as
an "attachment" (as opposed to an "inline" portion).
In the past the only means of doing this was by setting a JCW
(NETMAILTEXTATTACH=1) before sending the message, which would force ALL text
attachments to be labelled with a recommended disposition of "attachment".
This was an all-or-nothing prospect per message, and some users requested that
this attribute be selectable by individual attachment.
As of version B.07 01/08/23 when using NetMail/3000 in command mode (or in
batch jobs) when prompted for a filename to attach, you can now add a
";ATTACH" or ";ATT" to the filename. This tells NetMail/3000 to attach the
label "disposition: attachment" to the attachment when it goes out, as well
as supplying the filename you gave it. [Note that MPE filenames will get a
".txt" appended to their names in the mail message, to make it easier for
the receiving mail system - some of which pay more attention to filename
extensions that encoding labels - to display the message.]
----------------------------------------------------------
More miscellaneous enhancements
As of B.07 99/06/02 NetMail/3000 and DeskLink installations routing their
outbound email through another server have two additional configuration
options for routing these messages.
The "trusted gateway" routing option in the globals option of the netmaint
program now accepts two new options for routing mail without adding the
old-style/traditional "source routing" information to the address. Source
routes look like "@relayhost.domain.org:finaluser@acme.com" - with the relay
host's name being prepended to the outbound email address. While this was the
accepted (and standards defined method) of routing mail through "relay
servers" in the past, some new mail systems do not properly parse or handle
these address formats correctly - including some Microsoft Exchange servers.
So, If you do encounter problems with mail routing through a "trusted gateway"
from your NetMail/3000 system because of the remote server not "liking"
addresses in this format, you have two new options that you can select in the
netmail globals screen:
"R" - same as "Y" but does NOT source-route outbound addresses
"A" - same as "F" but does NOT source-route outbound addresses
----------------------------------------------------------
As of B.07 00/03/21 NetMail/3000 understands and utilizes the "Importance:
high" headers that Microsoft Exchange generates.
----------------------------------------------------------
As of B.07 99/12/20 a new JCW was added "THROTTLEPOP" to combat high cpu and
network utilization caused by POP clients on a local network downloading
and sending very large messages. Since a local pop client can access the 3000
at close to the full speed of the 3000's lan card, and it isn't uncommon for
POP mail clients to attach or send very large attachments, it wasn't
difficult for a pop client to completely dominate the cpu and network
interface of a 3000 by downloading multi megabyte files from the 3000 to their
PC. Since these connections are highly efficient and the NetMail/3000
background processes typically operate in the CS queue (to provide good
response time to these interactive-type processes) several users doing this
couple effectively "lock out" all other processes on the 3000. To combat this,
a "throttle" option was created in NetMail/3000, causing a short "pause" to
be invoked after every few "packets" of data once the user has exceeded 1Mb
of data being downloaded. From that point on, these pauses would allow other
processes to utilize the system, while only moderately slowing down what is a
very large download anyway for the pop client.
To enable this feature,
:SETJCW THROTTLEPOP=n
where 'n' is the delay desired (per packet) in tenths of a second. I.e. for a
1 second delay per packet, :setjcw throttlepop=10
----------------------------------------------------------------------
New features and enhancements in the new version of NetMail/3000:
(B.07, as of 00/09/01)
----------------------------------------------------------------------
INSTALLATION IMPROVEMENTS AND LOG ARCHIVING
Several improvements were made to the installation job, among which is
support for installing the THREEK account onto a volume set other than
the MPEXL_SYSTEM_VOLUME_SET. See the THREEKLD.PUB.SYS job stream for
details.
A modification was made to the background jobs so that whenever they are
launched on the 1st of the month, they'll automatically rename the error
log and SMTPINLG,SMTPOULG (and DESKINLG/DESKOULG for DeskLink users) files
for you to SMTPyymm (and DESKyymm) - based on the month/year of the previous
month. This should result in log files that are much easier to manage.
(You're still free to purge the log files if you don't want them.)
The installation process also now supports use of a store-to-disc archive
file (to restore files from).
----------------------------------------------------------------------
FOOLPROOF ANTI-SPAM-RELAY FEATURE
NetMail/3000 (and DeskLink) now supports a foolproof anti-spam feature
known as "check before send". This feature is enabled by a JCW in the
background job (RELAYPOP) which is set to a number of hours (>0). When
set, the mail server will not accept any incoming (i.e. from outside systems)
mail messages to be delivered to non-local recipients.
This ability to "relay" a message to another server was defined in the earl-
iest SMTP mail standards and was a necessity in the "early days" of the
Internet where some hosts were "better connected" than others. It is still
utilized by mail clients that use POP or IMAP capability to retrieve their
mail as they pass their "outbound" mail on to a dedicated server to take
care of final delivery for them.
Unfortunately, these days many hosts that support this capability are being
hijacked by unscrupulous "spammers", using other organizations' resources
(cpu time, disc space, and bandwidth) to deliver their trash mail. Simply
disabling all relay capability solves this problem, but leaves POP and IMAP
mail clients without a server to deliver their mail.
A compromise was designed by some Internet experts a few months ago - a new
practice called "check before send". What it does is disallows all "relaying"
UNLESS one of the following is true;
1) the recipient has a local forwarded mailbox
2) the recipient is identified in the ALIASES file with a redirected
non-local address
3) the sending host's IP address is in the "RELAY.NETMAIL" file
or
4) the sending host has logged into a pop mailbox (i.e. successfully
checked their mail) on this system within the last 'relaypop' hours
Any other attempts to relay are met with error messages to the sending
host, entries logged in the errorlog file, and an urgent mail message is
sent to the mail administrator
To add permanently allowed relay-hosts, you need merely create a file in
the NETMAIL group of the HPOFFICE or THREEK account (wherever your background
job logs on) and enter the IP address(es) you want to allow to relay through
your system. The file should be an 80 byte fixed-record size ascii file, and
records look like;
192.001.002.003 # comments go here
192.001.003.255 # comments here too
One address per line, each octet (number) must be three-digits long, and you
can use a "255" at any digit as a wildcard (255 will match any number in that
position).
Keep the file as "RELAY.NETMAIL" (unnumbered). This file will be read only
when the background job first starts up, so if you make any changes, you will
have to stop and restart the background job for them to take effect.
Note that by using the "relaypop" setting, the old "maxsmtprcpts" jcw can
be disabled, as it is now completely redundant.
----------------------------------------------------------------------
POP3 SERVER NOW SUPPORTS ENCRYPTED PASSWORDS
The POP3 Server product now supports the optional "APOP" command, which
uses an MD5 (encrypted hash code) method of passing the mailbox password
over the network. Instead of the normal "PASSWORD" method where the mailbox
name and password are passed over the network "in the clear" (i.e. not
encoded in any way), the "APOP" method passes a time-sensitive hashed
code which the server verifies (using it's own hash code). No plaintext
mailbox passwords are ever passed over "the wire", and hash codes are
time (and process) sensitive, so captured hash codes cannot be used in
"replay" attacks (i.e. trying the same hash code at another time to attempt
to gain access to the mailbox).
The hash code is the industry standard hash known as "MD5", and is supported
by most POP mail clients (select "APOP" instead of "PASSWORD" as your logon
authentication method in your POP client).
----------------------------------------------------------------------
NEW EXTENDED LOGGING FACILITY
For administrators needing more information to debug mail connectivity
problems, we have added an "extended logging" facility. This facility logs
information related to the flow (incoming and outgoing) of mail messages
and delivery attempts.
In normal SMTP (Internet Mail) delivery, a mail system may retry for many
hours (or even days) before finally being able to deliver a mail message to
the intended recipient (or returning the message as undeliverable). In the
interim, there may be several hundred (or even thousand) delivery attempts;
normally the administrator is not aware of the individual attempts, as SMTP
mail was designed to accomodate this and it is not (usually) considered
extraordinary.
There are occasions though, when it is useful to be able to follow the
complete delivery cycle (in case there are unanticipated errors or consistent
errors with a certain host). In these cases, by temporarily enabling extended
logging for the appropriate process (SMTPSERV to monitor incoming mail traffic
or SMTPOUT to monitor outbound delivery attempts) you can later print the
extlog file and see a complete session history of all delivery attempts.
(For novices, this trace does include detailed SMTP dialogues and some DNS
related messages - you will need knowledge of SMTP mail delivery standards to
interpret the trace information, but there is nothing specific to NetMail's
implementation -- all data is presented such that an administrator familiar
with SMTP should be able to read and fully understand all logged data).
If you're going to use extended logging, you need to build the log file it
will use in the NETMAIL group of the account your background job logs on into
(i.e. either THREEK or HPOFFICE). Build it as follows:
:BUILD EXTLOG.NETMAIL;REC=-132,,f,ascii;disc=100000;CIR
Be aware that since it is a circular file, while it is in use by the
background job (i.e. if you have extended logging enabled at any level) you
will not be able to print the file. You must turn the extended logging
facility off before you will be able to print the file.
Every record in the file starts with a "<" (meaning an incoming message)
or a ">" (meaning an outgoing message). This is followed by the timestamp
"YYYYMMDDHHMMSSTT" (year,month,day,hours,minutes,seconds,tenths of seconds),
then "/" followed by the PIN (process id) of the reporting process, followed
by another "/" then the name of the process. It is important to pay attention
to the PIN as for each process (SMTPSERV/SMTPOUT) there may be several
instances of that process running at any given moment.
Following the process name is the message; this is either a literal SMTP
dialogue message, or a textual ddescription of a process (like "Looking up
host" or "Mail queue delivery attempted for host=").
An example of extended logging information showing delivery of a mail
message is:
<1998043019313306/55/SMTPSERV/EHLO staffmail.digex.net
>1998043019313306/55/SMTPSERV/250-Howdy STAFFMAIL.DIGEX.NET Groovey to meet ya
Duud
250 PIPELINING
<1998043019313308/55/SMTPSERV/MAIL From:
>1998043019313605/55/SMTPSERV/250 Ok
<1998043019313605/55/SMTPSERV/RCPT To:
>1998043019313700/55/SMTPSERV/250 Ok
<1998043019313700/55/SMTPSERV/DATA
>1998043019313700/55/SMTPSERV/354 Start mail input; end with .
<1998043019314700/55/SMTPSERV/QUIT
>1998043019314700/55/SMTPSERV/221 Cool rappin wit ya, duud. Be excellent.
----------------------------------------------------------------------
ABILITY TO TURN TRACING ON/OFF WHILE JOB IS RUNNING
With B.07 95/05/04 and later, three new command files in the XEQ group
of the THREEK account allow you to control traceing/extended logging without
having to stop/restart the background job.
:TRACEON.XEQ.THREEK
Turns traceing ON in the background job
:TRACEOFF.XEQ.THREEK
Turns traceing OFF in the background job
:EXTLOG.XEQ.THREEK 0
^Replace with level of extended logging you want
The traceing facility dumps large amounts of data to spool files, recording
the actions and internal processing of messages. Usually this data must be
interpreted by 3k Technical Support Personnel as it is highly technical and
full of source code references. If directed to obtain traceing information by
3k Technical support personnel, this is how it is done. We should also note
that as trace data is printed to the process $stdlist spoolfiles, if the
SEPARATESTDLISTS jcw was turned off (set to '0') in the main job, trace data
will not be useful as it will all be intermixed in the single spool file.
Extended logging provides process and mail flow data that, while moderately
technical, may prove useful to system administrators that are familiar with
basic SMTP/POP operations and need to monitor or troubleshoot a mail
connection. Since this information is logged to a separate disc file, the
SEPARATESTDLIST jcw in the background job has no impact on it.
Extended logging is enabled per process (type) you want to monitor. The
"flag" or "parameter" you provide is a bit-mask specifying which processes
you want to monitor. The settings are:
Bit 15 = Monitor the ARPA process
Bit 14 = Monitor the SMTPOUT processes
Bit 13 = Monitor the SMTPSERVer processes
So, a value of "1" would trace ARPA, "2" only SMTPOUT, "4" only SMTPSERV,
"7" all three, "6" SMTPOUT and SMTPSERV.