Electronic Mail -
The Standards and the Hype
One of the hottest topics in the computer world today is interoperability, but it's by no means a new topic. The earliest computer users realized that as expensive hardware was not always available in the same room, ways to share resources and information was an ever growing necessity. In addition, as most can testify, not all budgets can afford single-vendor solutions. Making it work no matter who built it, and getting the most use out of it was (and still is) a way of life.
The earliest supercomputers were tremendously expensive, and demands for resources were spread among many users and spanned great distances. Early pioneers needed to develop means of sharing these resources among users and equipment as varied as the research and work that was being done. In order to do this, one of the earliest requirements was some means of electronic communication among the users and researchers.
Goals were lofty, and though the requirements were varied, a democratic process was developed to design the software "standards" needed by this growing and enthusiastic community to communicate and make the most effective use of the computing resources available. Early development efforts were sponsored by educational institutions and the U.S. department of defense. A computer network was created, the likes of which the world had never known. The early network was named the "Arpanet", and even its creators wildest dreams couldn't picture the degree to which this network would grow to affect computing across the planet. Today this network (and thousands of other linked networks) are collectively known as the "Internet", and spans every continent and almost every country in the world. The Internet now links over one million computers and is growing explosively. The network now encompasses more than just educational and government institutions. Commercial companies are now jumping on the bandwagon as the Internet becomes an increasingly valuable business link. While new technology speeds up the network backbones, the same basic protocols, designed in the 1970s and 1980s, still forge the links between the systems.
Today, every major computer system and solution vendor cites standards before their own phone numbers. The rules learned in the establishment of the Internet over a decade ago now drive computer systems and network design from the smallest shops to the largest private networks in the world. As every forward-looking company today sees; there is no longer such a thing as a private network. The purpose of a network is to make more effective use of expensive or scarce resources and to improve the flow of information throughout an organization. Information is money, and any process that improves that flow can have a direct impact on the bottom line.
Electronic mail was one of the earliest and most obvious applications for computer networks. Developed as (among other things) a means of sharing research work and standards proposals, electronic mail became an early and indispensable tool. Today, electronic mail is one of the fastest growing computer based applications in the world, and in many organizations is the primary and most important computer application in use.
Electronic mail began life as a means of exchanging simple text messages between users in varied locations and time zones. From the beginning, electronic mail had big advantages over more traditional means of communications: It provided a rapid (more so than traditional post office mail) means of sending messages; it provided an easy means of exchanging written communications (no more mis-interpreted notes or missed phrases); it provides a means of communication that doesn't interrupt the recipient and his train of thought or work in progress; it allows many, many people to exchange information over a shared communication line (often cheaper than telephone communication); and it provides a means of exchanging information where the recipient does not need to be present to receive the message (as in phone communications).
Electronic mail today still retains the early benefits, but goes far beyond those capabilities. The term "electronic mail" really doesn't do justice to the capabilities of these applications today. E-mail users take new capabilities such as transfer of formatted word processing documents, imbedded or linked graphics (including fax), transfer of executable programs and data files, return receipts for delivered mail, prioritized mail, and easy to use network mail directories for granted. Electronic mail applications are now entering a new era as multimedia applications take off and e-mail applications adapt to transfer these new objects. E-mail systems today are becoming network data transfer tools. They provide a common transfer medium upon which users can exchange data among diverse computer environments. Sounds a lot like open systems, huh?
All is not roses
Well, I would be remiss if I didn't point out that you can almost never order a few tapes, install the software on all your systems, and come up running. Network planning and implementation today is still very much an art. And did I mention that standards are not all standard?
To begin with, there are as many ways of passing data between systems as there are combinations of systems and software. Vendors have for years developed their own means of linking systems, often ignoring or circumventing the standards that did exist. HP has long been guilty of this, and is today spending alot of money and effort to try and shake the bad "rap" they've gotten for using "proprietary" protocols on their systems.
The funny thing is, that HP has come full circle on its "standards" quest. Though not commonly known, the HP3000 was one of the early pioneers in Internet "standards". Hard to believe; in fact I only found out after much research and digging through old standards documents. HP's "proprietary" NS (Networked Systems) and DS (Distributed Systems) protocols are protocols built on the famous "IP" (Internet Protocol), and the "TCP" (Transmission Control Protocol), the protocols upon which the original Internet designers based their now worldwide network. In fact, in 1980 HP software engineers working on the HP3000 series III published a document (still available in the Internet public domain archives) that outlined their early implementation of the now famous networking protocol standards that vendors around the world are now falling over themselves to support (even the venerable slow-moving IBM is just now seeing the light). For those interested, the document is IEN 167, published in 1980 and available via FTP or by automated e-mail retrieval from the Internet Network Information Center computer archives. The tools to use TCP/IP on the HP3000 have been there all along, though an unlikely product name (NetIPC) and a general lack of information on the capabilities that were hidden there have escaped the eyes of most HP users and vendors alike. You have to dig through the NetIPC manuals (if you even have them) to realize that what these routines are giving you is a direct TCP/IP link, compatible with every other TCP/IP implementation in the book. Today's industry-standard equivalent for these calls is called the "Berkeley Sockets" interface. Though not as robust (yet) as the NetIPC library, the calls have the same format and parameters as the equivalent libraries on Unix systems throughout the world.
Anyway, when it comes to electronic mail there are two levels of standards you need to know about. The first factor is the standard used among mail applications to exchange messages. There are several popular standards here, and each depends on the underlying network and low-level network software available. The big names here are "SMTP" (Simple Mail Transfer Protocol), "X.400" (named for the standards committee document number it was assigned), and "MHS" (a defacto standard common in the PC world).
SMTP is the e-mail standard designed by the Internet designers specified in a series of RFCs (Internet standards documents), and depends on TCP/IP as its low-level network transfer protocol. X.400 is the electronic mail standard first defined in 1984, later revised in a new release in 1988, and again revised in 1992, and was designed to use the OSI stack (low-level network protocol). MHS implementations are built on Novell's IPX low-level protocol. Depending on the type of network you have (or have planned), your choices for standardized electronic mail are potentially limited. Additionally, it is not uncommon in many organizations to have more than one local area network, or at least local area networks with mixed protocols. Since IPX, TCP/IP, and OSI will all coexist on the same physical media, the job remains to find a common denominator when it comes time to get everyone talking.
There are basically four ways to get multiple electronic mail systems talking: 1) You implement the same electronic mail protocol on all systems and networks, 2) You install e-mail "gateways" between every combination of e-mail systems and networks in your organization, 3) install an electronic mail "hub" that handles multiple protocols, or 4) You pick a common standard "backbone" protocol and implement packages that either use that protocol or provide a gateway to that common protocol.
Option 1 is usually not an option except in small shops or very well organized and designed networks. When this is an option, it usually insures the best portability of message objects and sure makes training a lot easier. For most of us however, this is usually not an option. With the mix of hardware and software prevalent in most shops, finding one package that runs on all combinations of platforms is the impossible dream. Even then, a package that works for the least capable users and boxes may not suit the needs of the "power users" or the engineers with the new multimedia workstations.
Option 2 is a common solution to a common problem. Networks come about, more often than not, in bits and pieces, from a dozen vendors and each portion picked by a separate user. When management finally decides to start integrating networks, the often chosen path is to start installing gateways between the various e-mail packages in use. For two or three systems this can still be a manageable solution, but rapidly gets out of hand (for three systems you need six gateways, for four systems you need 12 gateways, and on and on.). This solution, however, does offer the advantage that users get to keep the mail package that they know how to use and/or are most comfortable with. The hodge-podge of mail systems can generally support the varied needs of varied user bases.
Per option 3, several large companies make their livings providing large organizational electronic mail "hubs" - multi-protocol systems that function as gateways among many diverse e-mail systems. These are not cheap, but are sometimes more economical than other options. These are also often large (powerful) systems designed for very heavy throughput, and need to be so since any e-mail traffic passing between any other networks or protocols on the network all must pass through this single point. They can be critical resources, as a failure of an e-mail hub can cripple the entire electronic communications capability of an organization. Like the previous option, a central hub still allows users to mix and match the mail systems that best suit their needs and tastes.
In the "open systems" world, option 4 is usually the most powerful, flexible, and cost effective option for any growing organization. By adopting a common backbone (though not necessarily the same user front-ends) organizations can mix and match the user interfaces to match the varied needs of the user base, while still maintaining a common communications and data exchange patch among all users. Today, this backbone doesn't even need to be on-site; public network service providers are available to happily tie your conformant systems together, be it across the street or around the world. Service providers are available to supply your organization with links to the Internet, the large public X.400 based networks, as well as other private network options. The big question on everyone's lips nowadays though is "which standard should we adopt?" Very workable enterprise-wide electronic mail solutions can be built on proprietary protocols, but we all know the risks there. With the big three e-mail standards out there, almost every electronic mail package sold today supports one or all of these, so the choices can be made based on the criteria we like to see: price, features, and support.
In the HP3000 world I know of no current or planned MHS implementations, though there are gateways available from many mail systems to MHS based systems. Since HP3000s don't come with the IPX network protocol (unless you purchase the Netware/iX product also) MHS isn't likely to catch on in this arena, and MHS based e-mail networks are not a viable alternative for HP3000 based users. For those with MHS based networks already in place, a gateway or hub is the only option right now. A point to consider here is that the IPX protocol upon which MHS systems depend is a more appropriate protocol for local area networks (as opposed to wide area networks). While MHS is not uncommon as an in-house e-mail solution, when it comes time to talk to the rest of the world (business partners, suppliers, etc.) you're likely going to need to look to one of the more common internetwork protocols (SMTP or X.400).
We all hear a lot about X.400. X.400 has been proclaimed by the industry press as "the" electronic mail protocol of the future, and that no company in its right mind should settle for any less. I have seen that statement more than once in the trade press, and we all believe everything we read, right? The facts of the matter are that there are good reasons for the praise of X.400, and there are more questionable motives behind other praises of X.400. As an end-user, it is important to understand the motives behind what you might read in order to put it all in perspective. Let me preface all this by stating MY motives; as a vendor I chose to support the "other" e-mail standard, for a variety of reasons which I will talk about later. As a heavy e-mail user and network manager, I was faced with a choice between SMTP or X.400 based solutions for electronic mail, and the more research I did, the more "interesting" things I learned about the ongoing standards debates. Not to mention that as an HP3000 user, some choices have been forced upon us.
To help understand the rhetoric coming from all camps, it's useful to understand just how these "standards" come about and who develops them. First, the Internet standards: the Internet started out as a group of researchers and defense department backed activities tasked with developing a computer "network" where heterogeneous equipment and resources could be shared and information passed among the members. Groups of researchers and engineers formed ad-hoc committees to come up with innovative solutions to the problem. While a lot of new ground was broken, as the group of networked users grew, a primary concern among those groups forming standards was in developing solutions that were compatible with the hardware in use throughout the network and that were efficient, easy to implement, and inexpensive. To this end, much of the software that became the basis for the standards later came from software developed by these researchers or graduate students and was placed in the public domain. To this day, the Internet Engineering Task Force, the group that oversees the Internet standards is driven by users and researchers. The standards produced are made available on-line to the general public (at no charge), and often include source code and examples. Almost without exception, the standards are public domain, with no royalties or patents to contend with. And as you might expect from a group driven by actual users of the technology, cost and compatibility with existing hardware and software are high on the list of criteria for new standards.
On the other hand, the International Standards Organization (ISO) was founded and is sponsored primarily as a consortium of software and hardware vendors. While users and researchers obviously play a big part in the standards process, standards that are adopted are often based on technologies developed by member vendors or groups of vendors. Just to get copies of these standards documents cost thousands of dollars, and as they are usually copyrighted documents, are not available on-line to the general public. Even the certification process for OSI based products is expensive (and not very accurate). Certification of a single OSI based package can cost the developer up to half a million dollars. Even then, packages certified as compliant under the government OSI protocol program (GOSIP) have often proved incompatible with each other. Since the technologies are usually developed by vendors (at their own costs), when/if adopted, the developers recoup some of these costs through royalty and license fees to any vendors that choose to implement these "standards" and build solutions with them. In this light, it also makes sense as to why you see so much press about X.400 - the people with the advertising dollars are the ones that spent all the money coding to OSI network standards (it costs serious money to even step into the OSI standards ring). X.400, the ISO's standard for electronic mail transfers is built on another ISO networking standard, the famous seven layer OSI protocol stack. This was developed by the ISO members, and in its defense is a very powerful and flexible communications protocol. It is also a very complex protocol to implement, and if you have shopped around for OSI based datacomm products, you'll notice both a hefty price tag for the package, as well as lofty system requirements for its implementation. You won't likely ever see a 286 based PC running a full OSI implementation, and even a 486 will need sizable memory to handle the overhead. For those of us in the HP3000 world, we also will probably never see the OSI stack made available on a "classic" HP3000.
One big glaring hole in the Internet standards, and one that the ISO people capitalized on, is a major design flaw in IP, the Internet's underlying communications protocol. If you've looked into the two, you've probably heard that the Internet Protocol has a limited address space; much too limited for the popularity it has achieved of late. With a 32 bit machine address space and the literal explosion of networks coming online every day, the available addresses will all be used up within the next 4-6 years. Call it short-sightedness, but the original designers of the Internet never imagined the day when over one million computer systems would be linked with the protocol they invented, and the growth rate is increasing every day. The OSI stack was designed to be the eventual replacement of IP, with implementation phased in over the next several years. Unfortunately, the standards process in developing the new standards was bogged down by much corporate red-tape and took much longer to come out with a useable standard, and in the meantime everyone has been jumping on the IP bandwagon. OSI implementations over the past few years have been creeping along, while IP based network growth has exploded as even big players like IBM jump on board. With so many IP networks out there and a corresponding explosion of available tools and software based on the protocols fueling the growth, the networking giants are realizing that IP and OSI are going to co-exist for a LONG time (at least). Meanwhile, the Internet standards people are in the process of adapting the IP standard to accommodate the address space flaw, and a solution is expected in the first quarter of 1995.
Recently, standards groups have been busily working on integration strategies to ensure the coexistence of the expanding base of IP packages with the emerging OSI based standards. Task forces are back fitting some of the new network management capabilities from the ISO's standards into the corresponding Internet standards. Even X.400 to SMTP connectivity is becoming a major issue. In fact, the London-based European standards group ISODE (International Standards Organization Development Environment), not affiliated with the ISO, created software definitions enabling OSI application level protocols (like X.400 and X.500) to work over both OSI and TCP/IP. Other US based standards groups are now in the process of formally adopting similar standards. The ISO and IETF have even met recently, and are working on procedures to ensure that the standards developed by both groups begin to converge, or at least compliment each other. With products already announced adopting the ISODE standard, and many anxiously awaiting the standards that will adopt X.400 and X.500 to TCP/IP, the coexistence of X.400 and SMTP is ensured for a long time to come. Since even the US Government is reconsidering its decision to purchase ONLY GOSIP based networking solutions, both camps are scrambling to meet the "real world" demands and end the "protocol wars".
X.400 VS SMTP
SMTP was designed by the folks that brought us the Internet protocols and TCP/IP. Originally designed as simple text message exchange (SMTP stands for Simple Mail Transfer Protocol), it was created to handle the lowest common denominator among computer systems at the time - seven bit ASCII data. SMTP is often incorrectly referred to as "UNIX mail", though this name came about as many of the first SMTP implementations were done on Unix based systems. SMTP has been a standard available on most computer platforms in the world for more than a dozen years; everything from MS/DOS, Macintosh, and OS/2 to VAX systems, SUN workstations, and IBM mainframes.
With address domains maintained by the Network Information Center (NIC - a contractor designated to manage administrative functions for the Internet's U.S. functions), a limited number of globally recognized domains were created, though unless succeeded with a nation's domain code they are assumed to be U.S. based. SMTP addresses are in a format of:
username@hostname.domain.organization
Though many levels of domains are allowed, an addresses' national or organizational origins are sometimes not readily identifiable (foreign networks that may tie into the Internet are not bound to adopt the same naming conventions as U.S. members are). Since SMTP is the prevalent electronic mail standard on the Internet (a very well linked and informed community overall), much work has been done linking SMTP mailers to other e-mail systems. The Internet has public "gateways" available where users can send mail from their SMTP mailer to other networks, including Sprint's public network, MCI's MCI Mail network, Compuserve's mail system, GE's Genie system, the global PC based FIDO BBS network, the global UUCP dialup network, as well as gateways into the Internet's network news service, and many others. Here again though, these other mail systems adopt their own mailbox naming conventions so Internet mail addressed to a user on one of these foreign networks is often further complicated with more levels of machines and domains and funny characters ("%", "!", "$", etc.). (the X.400 people couldn't solve this problem either though.)
Since SMTP mail transport has traditionally been limited to seven bit ASCII data, passing of data other than USASCII text (i.e. foreign character sets, binary data or executable files, or multimedia objects) was not allowed in the original specification. SMTP has grown however, and as is common with Internet standards, enhancements to support the transport of these new "message" types was accomplished in such a way as to not "break" existing SMTP mailers. The newest enhancements to SMTP include the "MIME" standard (Multimedia Internet Mail Extensions) as well as "PEM" (Privacy Enhanced Mail). MIME is the Internet mail community's answer to the latest revision of X.400 - the 1992 release). MIME extensions allow SMTP mailers to transfer any arbitrary file type, from audio, video, fax, application data files, or arbitrary bit stream files, as well as messages in any defined character set (including foreign languages), and allow multi-part messages with contingency for messages displayed in parallel, sequence, as imbedded recursive messages, or as attached disc files. Flexibility was designed in for future to-be-determined types, as well as the privacy enhancements which include message encryption and digital signatures. All this was included in such a way that even old non-MIME SMTP implementations could still pass the messages (though they may not be able to meaningfully display the new message types without user-intervention). Some of this work was actually done by researchers that had worked with the ISO (some of which had become disillusioned with the bureaucracy involved in the ISO). As a result, facilities were also designed into SMTP to ease integration of SMTP mailers with X.400 based mail systems.
X.400 was designed by the ISO to be the next generation of electronic information transfer. Designed to accommodate all the bells and whistles that the original Internet mail designers never envisioned as well as facilities for electronically routing mail to and from traditional post office facilities as well as other public data networks. It also accommodated a worldwide naming structure to support global data transfers and guarantee globally unique addresses. In order to do this, X.400 utilizes some verrrrrry long addresses. Addresses longer than 50 characters are not uncommon. The idea however, was that users wouldn't have to type in addresses like this -- X.400 was supposed to be accompanied by another related standard, X.500, the ISO standard for a global name directory (kind of like a worldwide on-line phone book). The X.500 standard turned out to be a more ambitious project than originally imagined, and even now only pilot projects are on-line and several major issues remain to be resolved before any final version of a useable directory service is made available to the general public. One of X.500's challenges was that it was designed to operate as a global e-mail directory, supporting users in every county in the world. What the designers didn't realize though, was that many organizations didn't WANT a list of their employees (much less their e-mail addresses) made available to the general public, which would include, of course, competitors and headhunters. In the meantime, X.400 implementations are forced to use proprietary directory services and/or force users to enter long and clumsy e-mail addresses when sending electronic correspondence.
X.400 had another problem. Originally released in 1984, the standard was immediately found to have major problems and redundancies that made it difficult to implement and insecure when completed. It was revamped and finally re-released in 1988, however by this time several vendors had already released 1984 compliant X.400 mail systems. It turned out that 1984 and 1988 X.400 implementations were different enough in key areas that they did not end up very compatible with each other, so much work has been done in bridging the two X.400 implementations. Even then though, the 1988 release didn't address all the new multi-media implications that were becoming a hot issue in the industry. The ISO people went back to the conference rooms and came up with a new X.400 release four years later. Drafts of the new standard, 1992 X.400 have been released for comment though as of this writing the final version has not been released. Meanwhile, dozens of commercial (and free public domain) SMTP/MIME implementations have been released and SMTP users are excitedly exchanging video, audio, and application files around the world.
A case of SMTP
In our case, we saw a need for an electronic mail system for the HP3000 that was actually designed to interoperate with other systems. To do this we obviously had to pick a standard to adopt, and for the reasons outlined above, as well as the fact that we wanted to provide a solution for ALL HP3000 sites -- not just spectrum systems, we designed our electronic mail system to work with the Internet (SMTP) standard. The TCP/IP stack is present on any HP3000 (classic or spectrum) that has any network link product (i.e. lanlink or X.25 link; they come with NS/3000 or ARPA/3000), so SMTP seemed to be the logical solution.
Starting with a well-defined standard has significant advantages. By designing the system based on the standards we avoided the potential for critical design flaws or clumsy gateway "kludges" when it inevitably comes time to try and talk to someone else's system. Since gateways from SMTP to most other e-mail systems have already been done, by utilizing SMTP we immediately became accessible from most other mail systems in the world. This was also handy when it came time to do the initial testing of our system; by bouncing messages back from IBM mainframes, VAX systems, DEC TOPS systems, PCs and PC LAN systems, Sun workstations, and dozens of Unix mail implementations we were able to easily pinpoint initial bugs and perform extensive conformance testing.
A standards based system has many advantages for potential users as well. First and foremost, users can utilize existing technology and equipment to communicate. Whatever platforms are available can be electronically linked, and users are free to pick and choose implementations for the varied platforms based on the features and price that best match their needs. In the case of TCP/IP based solutions, there are many public domain packages available for a wide variety of platforms that can also be utilized. As an added bonus, our designing to the SMTP standard made it easy for us to accommodate several related Internet standards which can greatly benefit our users, not the least of which is a protocol called the Post Office Protocol (POP). POP is a client-server mail protocol where a server maintains users' mailboxes, receiving and holding mail for them even while their own systems are not available (or turned off). POP allows a client on a PC or other system to logically connect to the POP server and retrieve (and optionally delete) accumulated mail. POP has the significant advantage that very nice PC, Macintosh, and workstation based clients are available for free from the public domain. Thus with the POP server we provide, any number of PC users (on the network) can retrieve and manage their mail from their PCs, without ever logging onto the HP3000. These front ends have all the PC niceties you would expect, and windows and OS/2 based versions are available as well. (Even the LAN card drivers for the TCP/IP stack are available for free in the public domain.)
The SMTP protocol is designed around the way traditional business correspondence is generated. SMTP messages have two parts; a heading area and a message body. The message body is an arbitrarily large block of text (or other enclosures when enhanced with the MIME standard). The message heading area consists of a group of defined special "header" fields that identify the message, its recipients, the sender, the time sent, and other useful information. These well-defined headers are what makes the SMTP standard so portable, yet even a user unfamiliar with the standard can still deduce most of the useful information contained in the headers.
SMTP headers follow traditional business correspondence, the kind you typically find on the heading of company memos or bulletins. The important factor as far as interoperability though is in defining exactly which headers are always required and defining a non-ambiguous meaning for each header so as to be understood by implementations across all computing platforms.
The most important headers (from a user's standpoint) are the "From:" header, which as you probably guessed, defines who the message is from; the "To:" which defines who the intended recipients of the message are (along with the traditional "Cc:" (carbon copy recipients); and a "Subject:" line.
No SMTP based mailer today would be complete without the spiffy "MIME" enhancements. Though the details of MIME are transparent to end-users of the mail system, the results are the kind of electronic transfers that truly increase productivity and enhance an organization's communications capabilities. Through implementation of the MIME standard, the mail system (and thus the users of the mail system) are able to transmit true multi-media documents. Imbedded program files, data files, spreadsheets, graphics, fax images, animation, and sound can be transmitted. While an HP2392 terminal would be hard-pressed to display a color graphic image or play back an animated sequence with a sound track, even HP terminal based users can transmit and view applicable portions of a MIME message. (Viewable portions are displayed for the user, while non-applicable parts are described for the user with notations describing the type and placement of the enclosure within the document). MIME attachments can be saved as disc files either to the HP3000 or to a local PC drive if the user is using a supported terminal emulator (the saving of attachments must be done manually as a security measure). In addition, for Reflection for Windows (from WRQ) users, our mailer can automatically pick out those fancy MIME encoded attachments and download and present them for you; display graphics, play sound files, invoke spreadsheets or other PC applications automatically. MIME enclosed files that originally came from an HP3000 will be recreated exactly as they originally existed when transmitted; including program files, vfast files, and any other MPE file type. In addition, a MIME message can contain any number of imbedded files, making sophisticated automated software distribution applications possible. Since the MIME standard is platform independent and compatible with older SMTP implementations, users of other MIME compliant mail systems can send application or executable files among themselves, even routing mail through intermediate SMTP systems that are not MIME compliant. And just to be nice, we also implemented a MIME compatible SMTP gateway for HPDesk users. Now HPDesk users can also send and receive electronic mail messages with partners on the Internet, complete with fancy multimedia attachments if desired.
- For more information or just general questions, feel
free to contact us at support@3kassociates.com or
sales@3kassociates.com .
Chris Bartram
3k Associates Technical Support