Implementing A Complex Internetwork: A Case Study Roy A. Standing and Anton Dahbura, Jr.* National Library of Medicine Bethesda, MD Abstract This paper concerns the implementation of a complex internetwork in late 1989 through early 1990. The purpose is to discuss the methods and approaches employed. The context is that as technology improves and computer science advances, the same problems are re-solved over and over for slightly different environments. The perspective is that the methods and approaches will remain viable while the hardware and software changes. Introduction The problem to be solved was to develop and provide network services to a group of library personnel, contractors and consultants located throughout the National Library of Medicine (NLM) and at multiple remote sites. The solution had to integrate with the existing network, support multiple vendors' hardware, and provide maximal reliability with minimal maintenance. The workstations were Apple Macintoshes, IBM PCs and compatibles. The complex intemetwork we developed is represented in Figure 1 with the preexisting portion shaded. While all the information displayed is accurate, six networks have been reduced to stubs and only a portion of the attached equipment is represented. Definitions [2,12] Terminology AppleTalk: A set of LAN communication protocols defined by Apple. There are two types: Phase I and II. Bridge: A device linking two network segments using the same protocol into one larger segment. It transfers non-local traffic from each segment. NLM uses DEC LAN Bridges to join Ethernet segments and Shiva NetBridges to join LocalTalk segments. Broadband: A broad frequency carrier system. The frequency band is subdivided into audiovideo and data channels. NLM uses 1Mbps data channels. Ethernet: A set of LAN communication protocols defined by Xerox and standardized by the IEEE (802.3). Ethernet uses a transmission speed of 10Mbps (ten million bits per second). There are two media standards: Ethernet (Thick Ethernet) and Thinnet (Thin Ethernet). Gateway: A router linking two networks using different protocols. NLM uses Shiva EtherGate, Kinetics Fastpath 4, Cayman Gatorbox, Apple Internet Router (software), and Retix gateways. Hops: Distance on an internetwork is measured in hops, the number of routers network transmissions must l
Mr. Dahbura is now with Apple Computer, Inc.
0195-4210/91/$5.00 © 1992 AMIA, Inc.
545
travel through to reach their destination. Zero hops means originator and recipient are on the same network. Internetwork: A collection of individual networks linked
by routers/gateways. LocalTalk: AppleTalk networks implemented using shielded twisted pair wiring. LocalTalk uses a transmission speed of 230Kbps. NLM uses LocalTalk. Network (a.k.a. LAN): A group of computers and devices that intercommunicate over the same medium. Repeater: A device which extends the length of a network segment and transfers all traffic. NLM uses DESPRs (DEC Ethernet Single-Port Repeater) to join Ethernet to Thinnet. Router: A device linking two networks using the same protocol to form an internetwork. Routers maintain tables describing the internetwork, calculate the shortest path (in routers) for non-local traffic to take to its destination, and route traffic to the destination network. Routers isolate and define network boundaries. NLM uses Cisco routers to link Ethernet networks. 112 Router: A pair link two remote networks, each one constitutes one-half of the router. NLM uses Shiva TeleBridges to link remote LocalTalk networks. Zone: A zone is a named group of networks. NLM has assigned a one-to-one correspondence between physical network and zone. Environments [3,11] NLM is a two building facility located on the National Institutes of Health campus in Bethesda, Maryland. Broadband connections have been installed in every office and room of both buildings. Ethernet has been installed throughout most of the second building. Internet access is available through the Ethernet. At the time this project began, the broadband system provided data channels supporting Sytek and PC Net LANs. Our contractor's off-site office is a remote environment requiring both Ethernet and LocalTalk LANs. A consultant or home-based system is our typical single protocol remote environment. Planning Design Philosophy We began by establishing three basic tenets to guide our efforts: transparency, seamlessness and low maintenance. Transparency means the user's interface to network services does not change regardless of where or how the user is connected to the network. Hardware and software specifics may preclude identity across vendor's platforms;
ir1-
-----------
'r'':
Legend G
Apple File Server
M
El IEJ
Apple Macintosh
B
IBM PC
Rc')'u''"t'e'
.............
..........
......
Printer
is
...............
......
....
.1 ......
.....
.........
Figure 1: AppleTalk Network Layout at NLM (Simplified, the acid test is whether the user interface to all services is identical from any workstation of the same make anywhere in the internetwork. Transparency had three primary implications for us. First, the network administrator must take all responsibility for managing the network complexity. Second, all users will learn and use the same procedures, and have the same perception and understanding of the network. This minimizes training, promotes user confidence, encourages collegial support, and enables users to physically relocate without loss of network accessibility. Thirdly, ongoing network user support requirements are reduced due to the elimination of special cases. Seamlessness means the user's interface to network services is integrated into the methods normally used to control the user's workstation. If a service is equivalent, the same control method is used. New functions use natural extensions to existing control methods. Novel controls may improve on local workstation control methods, but should be provided in additional to seamless controls. The acid test of seamlessness is whether the interfaces use natural extensions of existing methods or require novel user skills, knowledge and abilities. Low Maintenande means hardware and software are self-configuring, reliable, easy to use and administer from anywhere in the internetwork. Technically networks may have to be complicated, but administratively they don't.
546
represents original internetwork) Where early operating systems provided complex control to highly trained operators, later operating systems embedded automatic control mechanisms. Similarly, while many network administration systems provide complex control to highly trained administrators, some have already taken the latter approach. They function like 'black boxes', creating unique network addresses, connecting gracefully to the network, and managing themselves. 'Black box' systems are designed for people, offices and companies wanting network benefits without training or administrative responsibilities. They minimize the staff resources required for ongoing network administration. Implementation Methodology We based our methodology on three technical decisions: gateway isolation, Ethernet as 'glue', and AppleShare. Gateway Isolation Gateways are commercially available to link AppleTalk, broadband and Ethernet and are discussed below. Since gateways only route non-local outbound traffic to another router they completely isolate the local network; the 'firewall' effect. This minimizes internetwork traffic, optimizing total internetwork performance and prevents problems in one part of the internetwork from effecting any other part. Gateways also ensure that internetwork access does not deprive the user of local network facilities. This can be illustrated using an Ethernet network joined to a LocalTalk network through a gateway plus a Macintosh work-
station with an Ethernet card and a LocalTalk connection. While the workstation can be physically connected to both networks simultaneously, the workstation can only use one set of network software drivers at a time. If the gateway is off the workstation only has access to the services of its local network, i.e., determined by the active set of network software drivers. However, if the gateway is on the workstation has full access to the services of both networks regardless of which network is local. Ethernet as 'Glue' We decided to use Ethernet as the 'glue' to connect our LANs. Ethernet's 10Mbps transmission speed ensured that internetwork communications would not be slower than local network communications for most users and activities. While geographic barriers sometimes require multiple hops, all LANs ultimately link to the Ethernet backbone as shown in Figure 1. AppleShare AppleShare was designed to make powerful network facilities simple to use and administer. It integrated transparency, seamlessness, and low maintenance. The project PCs use AppleShare PC version 2.0.1. An initial disadvantage was the need to run AppleTalk Phase II. Prior to the integration of PCs we had been a strict Phase I environment. A complete transition to AppleTalk Phase II could not be implemented due to Phase I only services running elsewhere on the internetwork. Our intermediate solution was to use the Apple Internet Router software running the Phase II upgrade utility as a background process on a file server. This allowed our intemetwork to handle both Phase I and Phase II. Installation [4,6,9] The internetwork was installed and tested network by network with only minor adjustments to the original plan. Pivotal to success was the ability of the network devices to perform different functions according to traffic needs. Naming Conventions We had to establish rules for naming zones and devices. Our zone names have two parts; one ascribes organizational ownership and the other physical location. For example, zone MMS-4th means that the network belongs to the Medlars Management Services (MMS) section and the network is physically located on the 4th floor. Similarly, device naming combines device type and zone name. Our implementation added sixteen new zones, ten types of network devices from six vendors, plus workstations, printers, and special equipment turning the preexisting NLM internetwork into a complex internetwork.
Gateways Ethernet and LocalTalk LocalTalk LAN interfaces are built-in to every Macintosh and network product from Apple Computer. As previously noted, all AppleTalk LANs at NLM are LocalTalk implementations. Where Ethernet was available we used Shiva EtherGates to connect LocalTalk to Ethernet, e.g., the MMS-4th zone in Figure 1. The EtherGate V1.50 provides full support
for AppleTalk Phase I, II and TCP/IP protocols. The EtherGate autoconfigures and uses non-volatile RAM for its programming. The EtherGate will automatically reboot and resume operation after a power failure. These features provided reliability and low maintenance. LocalTalk and Broadband When only broadband was available we used a specially modified Kinetics Fastpath 4** to connect LocalTalk to broadband, see the GM-RR zone in Figure 1. The Cactus Computer modified Fastpath actually contains a LocalTalk to Ethernet gateway which is plugged into an Ethernet to broadband modem! The Fastpath is complicated to program and must be manually configured but has been robust and reliable. Getting to broadband only answers half of the problem; a gateway from broadband to Ethernet is still required. Broadband and Ethernet The Retix gateway shares a broadband communications channel and one Retix was already in operational use. Adding an additional unit was a simple, solution completing the pathway for a LocalTalk zone gateway to Thinnet, see the GM-Mezz zone in Figure 1. The only device on this Thinnet segment is a Shiva EtherGate supporting the GM-Mezz zone. Linking Remote Networks Connecting remote sites is primarily a tradeoff between cost and communications speed. Modest usage projections led us to choose the cheapest but slowest solution, standard modem-controlled dial-up phone lines. We held the option to upgrade to a faster solution if performance requirements could not be met; see the OCCS-MR zone in Figure 1. Shiva TeleBridges with high speed modems attached were placed at NLM and the remote site. The Telebridge acts as a half bridge, linking the two site's networks with full network services. Ignoring network and modem overhead and just comparing the transmission speeds of each pathway, i.e., Ethernet 1OMbs, LocalTalk 23OKbs, and modems 19.2 Kbs, says roughly that transmissions across the modem link are almost 12 times slower than transmissions on LocalTalk and over 520 times slower than Ethernet! Thus, while the remote network connection is a vital capability, activities involving large amounts of data are scheduled. Most internetwork activities involve small transmission bursts and the speed of the internetwork link is not a limitation. We tested three modems on the remote links. The US Robotics Courier HST Dual Standard in HST mode uses asynchronous channel speeds (9600/300) and had the poorest data transfer rates because of the channel turnaround speed. The Telebit Trailblazer T1000 was very fast but unreliable with other vendor's modems. The Microcom QX3296c gave the most impressive transfer rates, reliable connection with other modem brands and became our modem of choice for linking to remote networks. Linking Remote Workstations Individual work-
**The Fastpath is now a product of Shiva Corporation. 547
stations can link to the internetwork using Shiva Dial-in software and a modem. Dial-in lines are provided using Shiva TeleBridges at the NLM site, see the GM-B 1 zone in Figure 1. The Telebridges support both network to network connections and single workstation connections. The workstations receive full network services subject to the speed penalty of the modem link. Network Services (Hardware) File servers, printers and modems are distributed throughout the zones within the internetwork for the convenience of the users. In addition, placing the equipment in the same zone as its primary users localizes network traffic and permits uninterrupted services if the zone is temporarily severed from the internetwork. File Servers are 'headless', meaning they have no keyboards or monitors. They do contain video boards to support local servicing when keyboards and monitors would be attached. The common file servers are located in the main computer room for physical space, security, and electrical power considerations. They are high performance systems connected to Thinnet communicating at Ethernet speed; see the Thinnet segment in the B1 computer room in Figure 1. The Thinnet links to Ethernet through a DESPR and hence the main Ethernet backbone through a DEC LAN bridge. The file servers are isolated zero hops from the main Ethernet backbone and centrally located to workstations throughout the internetwork. Timbuktu, further described below, permits administration from any Macintosh workstation in the internetwork. Printers and Modems for public use are located on the GM-B 1 zone; see Figure 1. The printers include a color Postscript laserprinter and a special label printer. These devices are physically accessible to all users. Network Services (Software) Each product was selected after being evaluated for sufficiency, transparency, seamlessness, and low maintenance. Electronic Mail [1,5,7] The email system had to be able to work with SMTP email, IBM Profs, ccMail, AppleLink, a special purpose bulletin board email system and allow email to be delivered to, and sent from, the email system of the user's choice. QuickMail, from CE Software, a server-based mail system, integrates realtime electronic teleconferencing, supports file enclosures and voice mail. QuickMail executes as a Desk Accessory on the Macintosh and a Terminate and Stay Resident on the PC allowing email facilities to be available regardless of the user's activity. QuickMail permits remote administration via Timbuktu. Gatormail, a Simple Mail Transfer Protocol (SMTP) bridge, connects QuickMail to Unix, NeXT and the worldwide Internet email. QM-Link connects to AppleLink during off peak hours. A Message Handling System (MHS) bridge to link to ccMail proved unreliable, but has since been resolved. An X400 bridge by Touch Com-
548
munications was later installed with bridges to ccMail and Proffs. Special arrangements have been made to develop a bridge for the bulletin board email system. FAXGATE and MailMaker, in association with a fax modem, permit files to be transmitted as facsimile documents. Any user with a QuickMail account can generate a facsimile transmission from any Macintosh workstation using any application that supports printing. QM Concierge allows users to reroute selected email, e.g., while on business travel. It also supports pseudoaccounts, e.g., System Administrator, that map to real users. The user gets a single point of contact and the internetwork manager can reassign responsibilities freely. Electronic Teleconferencing links two workstations for direct keyboard communication. Part of QuickMail, it permits online communication with other users; essential when a telephone is inconvenient or unavailable. Centralized File Storage Facilities AppleShare uses Macintoshes as file servers. The file servers appear as additional disk devices. The network interface is transparent and the user interface seamless. Multiple groups can share a file server for independent purposes with complete separation, security and privacy. File Interchange Files may be placed on a file server in a commonly accessible account. Files may be enclosed in QuickMail email messages. The most powerful method uses Timbuktu, further discussed below, to connect to the target workstation and uses several transfer methods. Remote Monitoring and Control [8,10,13] is possible between any two Macintosh workstations. The client's display is transmitted to a window in the master's display. It is not necessary for the client to have a keyboard, monitor or even a video card! Timbuktu is used to administer file servers, monitor public workstations, demonstrate problems to remote support staff, and effect repairs remotely. Timbuktu's security system manages access privileges and helps avoid misuse. The Internet Manager from Shiva Corporation is used to administer all Shiva network products. The Internet Manager can be run from any workstation on the internetwork, password protection guards against misuse. InterPoll and Peek are diagnostic utilities that monitor and report network traffic. They are used to isolate and diagnose network problems. Remote Backup and Restore for Macintosh workstations are provided across the network. NetStream software produces streaming or file oriented backups to 8mm data cartridges. Data transfer speeds up to 1OMBps are supported and the data cartridges store up to 2.2GB. Software Development Management for the Macintosh came from the Macintosh Programmers Workshop (MPW). IBM support came from Polytron Version Control System (PVCS).
Evaluation The project was completed over eight months meeting or exceeding virtually all of its goals. User acceptance has been excellent and the number of users has doubled. Reliability and performance have been outstanding with one hardware failure, a manufacturing defect found at installation. Options to tune performance have not been necessary. The internetwork operates continuously as an evolving facility, it is no longer special -- it's expected. What Worked Complex internetworks are realizable today using existing commercial products. Transparent and seamless to users, these networks can be delivered and maintained at a minimal cost in capital and support personnel. What Didn't There were numerous small failures; voice mail seemed promising but without compression the files are too large and cross the internetwork too slowly. Simultaneous monitoring of file servers, mail centers, and workstations is impractical. The data load exacts a high internetwork performance penalty and is too slow to be useful. What Needs Improvement Apple originally envisioned LocalTalk networks to be small with few zones. Zone selection works well in small networks but is cumbersome with large internetworks. Some evolutionary improvements are obvious, but some revolutionary new solutions are needed. What It Cost Developing an intemetwork involves materials, hardware, software and staff. Generally materials and hardware costs decrease over time while software and staff costs increase. The single most important cost is maintenance since that is a continuing requirement. The hallmark of our internetwork is its extremely low maintenance cost. Materials costs for thick Ethernet were $1.10/ft plus $250/node; Thinnet was $.60/ft plus $250/node; and LocalTalk was $.50/ft plus $35/node. Miscellaneous costs were under $500 dollars. Hardware The gateways cost $1200 to $4000; bridges about $350; repeaters about $200; and modems $800. Software Maintenance software was bundled with the hardware. Network user software costs were about $32/user for QuickMail and $100/user for Timbuktu. Staff costs were divided between planning, installation and maintenance. Planning consumed about .5 FrE (Full Time Equivalent). Installation consumed .3 FTE since we used concealed cable runs, special wall plates, etc. Maintenance requires about .1 FTE annually.
Summary In a perfect world you analyze your needs, design an optimal solution and implement it. In our case, a complex environment already existed and we had to adapt available solutions to it. The most important decisions were the first ones establishing our design philosophy. Looking
549
back now, the planning decisions seem clear and obvious but at the time, much of the hardware and software was just entering the marketplace. The project was an enjoyable challenge, the internetwork is reliable and robust, and the design philosophy continues to direct its evolution. Trademarks PostScript is a registered trademark of Adobe Corporation. Apple, AppleShare, AppleTalk, Interpoll, LaserWriter, Macintosh, and MPW are registered trademarks of Apple Co., Inc. LocalTalk is a trademark of Apple Computer, Inc. Gatormail is a trademark of Cayman Systems. QuickMail and QM-Link are trademarks of CE Software. Timbuktu"m is a trademark of Farallon Computing. Proffs is a trademark of IBM. QM Concierge is a trademark of Information Electronics. ccMail is a trademark of Lotus Corporation. Microcom and QX3296c are trademarks of Microcom, Inc. NetStream is a trademark of PCPC Corporation. PVCS is a trademark of Polytron, Inc. Dial-in, EtherGate, FastPath, Internet Manager, NetBridge, NetModem, NetSerial, and TeleBridge are trademarks of Shiva Corporation. FAXGATE and Mailmaker are trademarks of Solutions, Inc. Telebit and Trailblazer are trademarks of Telebit Corporation. X400 is a trademark of Touch Communications. Courier HST Dual Standard is a trademark of US Robotics, Inc.
References The articles and product reviews have been published since the end of the project. They discuss the hardware and software we used and competitor's products. 1. Deborah Branscum. Conspicuous Consumer. MacWorld March, 1991:63-83. 2. Daniel P. Dern. Multifunction Routers. SunExpert March, 1991 2(3):78-82. 3. Jim Heid. How To/Getting Started with Networks. MacWorld December, 1990:291-295. 4. David Kosiur. On the Ethernet Highway. MacWorld March, 1990:132-137. 5. op cit. E-Mail: A postal Inspection. MacWorld June, 1990:164-171. 6. op cit. EtherGate 1.45 (review). MacWorld August, 1990:196-197. 7. op cit. High-Speed Modems. MacWorld November, 1990:191-197. 8. op cit. Managing Networks. MacWorld February, 1991:152-159. 9. op cit. Going the Ethernet Route. MacWorld April, 1991:130-137. 10. Brita Meng. Timbuktu 3.1 (review). MacWorld October, 1990:237-239. 11. op cit. Networking for the Novice. MacWorld December, 1990:202-209. 12. Steven Sooby. Shiva EtherGate User Manual. Shiva Corporation. 13. Ed Tittel. Timbuktu/Remote 1.0.1 (review). MacWorld March, 1990:195.
Mr. Dahbura is now with Apple Computer, Inc.
0195-4210/91/$5.00 © 1992 AMIA, Inc.
545
travel through to reach their destination. Zero hops means originator and recipient are on the same network. Internetwork: A collection of individual networks linked
by routers/gateways. LocalTalk: AppleTalk networks implemented using shielded twisted pair wiring. LocalTalk uses a transmission speed of 230Kbps. NLM uses LocalTalk. Network (a.k.a. LAN): A group of computers and devices that intercommunicate over the same medium. Repeater: A device which extends the length of a network segment and transfers all traffic. NLM uses DESPRs (DEC Ethernet Single-Port Repeater) to join Ethernet to Thinnet. Router: A device linking two networks using the same protocol to form an internetwork. Routers maintain tables describing the internetwork, calculate the shortest path (in routers) for non-local traffic to take to its destination, and route traffic to the destination network. Routers isolate and define network boundaries. NLM uses Cisco routers to link Ethernet networks. 112 Router: A pair link two remote networks, each one constitutes one-half of the router. NLM uses Shiva TeleBridges to link remote LocalTalk networks. Zone: A zone is a named group of networks. NLM has assigned a one-to-one correspondence between physical network and zone. Environments [3,11] NLM is a two building facility located on the National Institutes of Health campus in Bethesda, Maryland. Broadband connections have been installed in every office and room of both buildings. Ethernet has been installed throughout most of the second building. Internet access is available through the Ethernet. At the time this project began, the broadband system provided data channels supporting Sytek and PC Net LANs. Our contractor's off-site office is a remote environment requiring both Ethernet and LocalTalk LANs. A consultant or home-based system is our typical single protocol remote environment. Planning Design Philosophy We began by establishing three basic tenets to guide our efforts: transparency, seamlessness and low maintenance. Transparency means the user's interface to network services does not change regardless of where or how the user is connected to the network. Hardware and software specifics may preclude identity across vendor's platforms;
ir1-
-----------
'r'':
Legend G
Apple File Server
M
El IEJ
Apple Macintosh
B
IBM PC
Rc')'u''"t'e'
.............
..........
......
Printer
is
...............
......
....
.1 ......
.....
.........
Figure 1: AppleTalk Network Layout at NLM (Simplified, the acid test is whether the user interface to all services is identical from any workstation of the same make anywhere in the internetwork. Transparency had three primary implications for us. First, the network administrator must take all responsibility for managing the network complexity. Second, all users will learn and use the same procedures, and have the same perception and understanding of the network. This minimizes training, promotes user confidence, encourages collegial support, and enables users to physically relocate without loss of network accessibility. Thirdly, ongoing network user support requirements are reduced due to the elimination of special cases. Seamlessness means the user's interface to network services is integrated into the methods normally used to control the user's workstation. If a service is equivalent, the same control method is used. New functions use natural extensions to existing control methods. Novel controls may improve on local workstation control methods, but should be provided in additional to seamless controls. The acid test of seamlessness is whether the interfaces use natural extensions of existing methods or require novel user skills, knowledge and abilities. Low Maintenande means hardware and software are self-configuring, reliable, easy to use and administer from anywhere in the internetwork. Technically networks may have to be complicated, but administratively they don't.
546
represents original internetwork) Where early operating systems provided complex control to highly trained operators, later operating systems embedded automatic control mechanisms. Similarly, while many network administration systems provide complex control to highly trained administrators, some have already taken the latter approach. They function like 'black boxes', creating unique network addresses, connecting gracefully to the network, and managing themselves. 'Black box' systems are designed for people, offices and companies wanting network benefits without training or administrative responsibilities. They minimize the staff resources required for ongoing network administration. Implementation Methodology We based our methodology on three technical decisions: gateway isolation, Ethernet as 'glue', and AppleShare. Gateway Isolation Gateways are commercially available to link AppleTalk, broadband and Ethernet and are discussed below. Since gateways only route non-local outbound traffic to another router they completely isolate the local network; the 'firewall' effect. This minimizes internetwork traffic, optimizing total internetwork performance and prevents problems in one part of the internetwork from effecting any other part. Gateways also ensure that internetwork access does not deprive the user of local network facilities. This can be illustrated using an Ethernet network joined to a LocalTalk network through a gateway plus a Macintosh work-
station with an Ethernet card and a LocalTalk connection. While the workstation can be physically connected to both networks simultaneously, the workstation can only use one set of network software drivers at a time. If the gateway is off the workstation only has access to the services of its local network, i.e., determined by the active set of network software drivers. However, if the gateway is on the workstation has full access to the services of both networks regardless of which network is local. Ethernet as 'Glue' We decided to use Ethernet as the 'glue' to connect our LANs. Ethernet's 10Mbps transmission speed ensured that internetwork communications would not be slower than local network communications for most users and activities. While geographic barriers sometimes require multiple hops, all LANs ultimately link to the Ethernet backbone as shown in Figure 1. AppleShare AppleShare was designed to make powerful network facilities simple to use and administer. It integrated transparency, seamlessness, and low maintenance. The project PCs use AppleShare PC version 2.0.1. An initial disadvantage was the need to run AppleTalk Phase II. Prior to the integration of PCs we had been a strict Phase I environment. A complete transition to AppleTalk Phase II could not be implemented due to Phase I only services running elsewhere on the internetwork. Our intermediate solution was to use the Apple Internet Router software running the Phase II upgrade utility as a background process on a file server. This allowed our intemetwork to handle both Phase I and Phase II. Installation [4,6,9] The internetwork was installed and tested network by network with only minor adjustments to the original plan. Pivotal to success was the ability of the network devices to perform different functions according to traffic needs. Naming Conventions We had to establish rules for naming zones and devices. Our zone names have two parts; one ascribes organizational ownership and the other physical location. For example, zone MMS-4th means that the network belongs to the Medlars Management Services (MMS) section and the network is physically located on the 4th floor. Similarly, device naming combines device type and zone name. Our implementation added sixteen new zones, ten types of network devices from six vendors, plus workstations, printers, and special equipment turning the preexisting NLM internetwork into a complex internetwork.
Gateways Ethernet and LocalTalk LocalTalk LAN interfaces are built-in to every Macintosh and network product from Apple Computer. As previously noted, all AppleTalk LANs at NLM are LocalTalk implementations. Where Ethernet was available we used Shiva EtherGates to connect LocalTalk to Ethernet, e.g., the MMS-4th zone in Figure 1. The EtherGate V1.50 provides full support
for AppleTalk Phase I, II and TCP/IP protocols. The EtherGate autoconfigures and uses non-volatile RAM for its programming. The EtherGate will automatically reboot and resume operation after a power failure. These features provided reliability and low maintenance. LocalTalk and Broadband When only broadband was available we used a specially modified Kinetics Fastpath 4** to connect LocalTalk to broadband, see the GM-RR zone in Figure 1. The Cactus Computer modified Fastpath actually contains a LocalTalk to Ethernet gateway which is plugged into an Ethernet to broadband modem! The Fastpath is complicated to program and must be manually configured but has been robust and reliable. Getting to broadband only answers half of the problem; a gateway from broadband to Ethernet is still required. Broadband and Ethernet The Retix gateway shares a broadband communications channel and one Retix was already in operational use. Adding an additional unit was a simple, solution completing the pathway for a LocalTalk zone gateway to Thinnet, see the GM-Mezz zone in Figure 1. The only device on this Thinnet segment is a Shiva EtherGate supporting the GM-Mezz zone. Linking Remote Networks Connecting remote sites is primarily a tradeoff between cost and communications speed. Modest usage projections led us to choose the cheapest but slowest solution, standard modem-controlled dial-up phone lines. We held the option to upgrade to a faster solution if performance requirements could not be met; see the OCCS-MR zone in Figure 1. Shiva TeleBridges with high speed modems attached were placed at NLM and the remote site. The Telebridge acts as a half bridge, linking the two site's networks with full network services. Ignoring network and modem overhead and just comparing the transmission speeds of each pathway, i.e., Ethernet 1OMbs, LocalTalk 23OKbs, and modems 19.2 Kbs, says roughly that transmissions across the modem link are almost 12 times slower than transmissions on LocalTalk and over 520 times slower than Ethernet! Thus, while the remote network connection is a vital capability, activities involving large amounts of data are scheduled. Most internetwork activities involve small transmission bursts and the speed of the internetwork link is not a limitation. We tested three modems on the remote links. The US Robotics Courier HST Dual Standard in HST mode uses asynchronous channel speeds (9600/300) and had the poorest data transfer rates because of the channel turnaround speed. The Telebit Trailblazer T1000 was very fast but unreliable with other vendor's modems. The Microcom QX3296c gave the most impressive transfer rates, reliable connection with other modem brands and became our modem of choice for linking to remote networks. Linking Remote Workstations Individual work-
**The Fastpath is now a product of Shiva Corporation. 547
stations can link to the internetwork using Shiva Dial-in software and a modem. Dial-in lines are provided using Shiva TeleBridges at the NLM site, see the GM-B 1 zone in Figure 1. The Telebridges support both network to network connections and single workstation connections. The workstations receive full network services subject to the speed penalty of the modem link. Network Services (Hardware) File servers, printers and modems are distributed throughout the zones within the internetwork for the convenience of the users. In addition, placing the equipment in the same zone as its primary users localizes network traffic and permits uninterrupted services if the zone is temporarily severed from the internetwork. File Servers are 'headless', meaning they have no keyboards or monitors. They do contain video boards to support local servicing when keyboards and monitors would be attached. The common file servers are located in the main computer room for physical space, security, and electrical power considerations. They are high performance systems connected to Thinnet communicating at Ethernet speed; see the Thinnet segment in the B1 computer room in Figure 1. The Thinnet links to Ethernet through a DESPR and hence the main Ethernet backbone through a DEC LAN bridge. The file servers are isolated zero hops from the main Ethernet backbone and centrally located to workstations throughout the internetwork. Timbuktu, further described below, permits administration from any Macintosh workstation in the internetwork. Printers and Modems for public use are located on the GM-B 1 zone; see Figure 1. The printers include a color Postscript laserprinter and a special label printer. These devices are physically accessible to all users. Network Services (Software) Each product was selected after being evaluated for sufficiency, transparency, seamlessness, and low maintenance. Electronic Mail [1,5,7] The email system had to be able to work with SMTP email, IBM Profs, ccMail, AppleLink, a special purpose bulletin board email system and allow email to be delivered to, and sent from, the email system of the user's choice. QuickMail, from CE Software, a server-based mail system, integrates realtime electronic teleconferencing, supports file enclosures and voice mail. QuickMail executes as a Desk Accessory on the Macintosh and a Terminate and Stay Resident on the PC allowing email facilities to be available regardless of the user's activity. QuickMail permits remote administration via Timbuktu. Gatormail, a Simple Mail Transfer Protocol (SMTP) bridge, connects QuickMail to Unix, NeXT and the worldwide Internet email. QM-Link connects to AppleLink during off peak hours. A Message Handling System (MHS) bridge to link to ccMail proved unreliable, but has since been resolved. An X400 bridge by Touch Com-
548
munications was later installed with bridges to ccMail and Proffs. Special arrangements have been made to develop a bridge for the bulletin board email system. FAXGATE and MailMaker, in association with a fax modem, permit files to be transmitted as facsimile documents. Any user with a QuickMail account can generate a facsimile transmission from any Macintosh workstation using any application that supports printing. QM Concierge allows users to reroute selected email, e.g., while on business travel. It also supports pseudoaccounts, e.g., System Administrator, that map to real users. The user gets a single point of contact and the internetwork manager can reassign responsibilities freely. Electronic Teleconferencing links two workstations for direct keyboard communication. Part of QuickMail, it permits online communication with other users; essential when a telephone is inconvenient or unavailable. Centralized File Storage Facilities AppleShare uses Macintoshes as file servers. The file servers appear as additional disk devices. The network interface is transparent and the user interface seamless. Multiple groups can share a file server for independent purposes with complete separation, security and privacy. File Interchange Files may be placed on a file server in a commonly accessible account. Files may be enclosed in QuickMail email messages. The most powerful method uses Timbuktu, further discussed below, to connect to the target workstation and uses several transfer methods. Remote Monitoring and Control [8,10,13] is possible between any two Macintosh workstations. The client's display is transmitted to a window in the master's display. It is not necessary for the client to have a keyboard, monitor or even a video card! Timbuktu is used to administer file servers, monitor public workstations, demonstrate problems to remote support staff, and effect repairs remotely. Timbuktu's security system manages access privileges and helps avoid misuse. The Internet Manager from Shiva Corporation is used to administer all Shiva network products. The Internet Manager can be run from any workstation on the internetwork, password protection guards against misuse. InterPoll and Peek are diagnostic utilities that monitor and report network traffic. They are used to isolate and diagnose network problems. Remote Backup and Restore for Macintosh workstations are provided across the network. NetStream software produces streaming or file oriented backups to 8mm data cartridges. Data transfer speeds up to 1OMBps are supported and the data cartridges store up to 2.2GB. Software Development Management for the Macintosh came from the Macintosh Programmers Workshop (MPW). IBM support came from Polytron Version Control System (PVCS).
Evaluation The project was completed over eight months meeting or exceeding virtually all of its goals. User acceptance has been excellent and the number of users has doubled. Reliability and performance have been outstanding with one hardware failure, a manufacturing defect found at installation. Options to tune performance have not been necessary. The internetwork operates continuously as an evolving facility, it is no longer special -- it's expected. What Worked Complex internetworks are realizable today using existing commercial products. Transparent and seamless to users, these networks can be delivered and maintained at a minimal cost in capital and support personnel. What Didn't There were numerous small failures; voice mail seemed promising but without compression the files are too large and cross the internetwork too slowly. Simultaneous monitoring of file servers, mail centers, and workstations is impractical. The data load exacts a high internetwork performance penalty and is too slow to be useful. What Needs Improvement Apple originally envisioned LocalTalk networks to be small with few zones. Zone selection works well in small networks but is cumbersome with large internetworks. Some evolutionary improvements are obvious, but some revolutionary new solutions are needed. What It Cost Developing an intemetwork involves materials, hardware, software and staff. Generally materials and hardware costs decrease over time while software and staff costs increase. The single most important cost is maintenance since that is a continuing requirement. The hallmark of our internetwork is its extremely low maintenance cost. Materials costs for thick Ethernet were $1.10/ft plus $250/node; Thinnet was $.60/ft plus $250/node; and LocalTalk was $.50/ft plus $35/node. Miscellaneous costs were under $500 dollars. Hardware The gateways cost $1200 to $4000; bridges about $350; repeaters about $200; and modems $800. Software Maintenance software was bundled with the hardware. Network user software costs were about $32/user for QuickMail and $100/user for Timbuktu. Staff costs were divided between planning, installation and maintenance. Planning consumed about .5 FrE (Full Time Equivalent). Installation consumed .3 FTE since we used concealed cable runs, special wall plates, etc. Maintenance requires about .1 FTE annually.
Summary In a perfect world you analyze your needs, design an optimal solution and implement it. In our case, a complex environment already existed and we had to adapt available solutions to it. The most important decisions were the first ones establishing our design philosophy. Looking
549
back now, the planning decisions seem clear and obvious but at the time, much of the hardware and software was just entering the marketplace. The project was an enjoyable challenge, the internetwork is reliable and robust, and the design philosophy continues to direct its evolution. Trademarks PostScript is a registered trademark of Adobe Corporation. Apple, AppleShare, AppleTalk, Interpoll, LaserWriter, Macintosh, and MPW are registered trademarks of Apple Co., Inc. LocalTalk is a trademark of Apple Computer, Inc. Gatormail is a trademark of Cayman Systems. QuickMail and QM-Link are trademarks of CE Software. Timbuktu"m is a trademark of Farallon Computing. Proffs is a trademark of IBM. QM Concierge is a trademark of Information Electronics. ccMail is a trademark of Lotus Corporation. Microcom and QX3296c are trademarks of Microcom, Inc. NetStream is a trademark of PCPC Corporation. PVCS is a trademark of Polytron, Inc. Dial-in, EtherGate, FastPath, Internet Manager, NetBridge, NetModem, NetSerial, and TeleBridge are trademarks of Shiva Corporation. FAXGATE and Mailmaker are trademarks of Solutions, Inc. Telebit and Trailblazer are trademarks of Telebit Corporation. X400 is a trademark of Touch Communications. Courier HST Dual Standard is a trademark of US Robotics, Inc.
References The articles and product reviews have been published since the end of the project. They discuss the hardware and software we used and competitor's products. 1. Deborah Branscum. Conspicuous Consumer. MacWorld March, 1991:63-83. 2. Daniel P. Dern. Multifunction Routers. SunExpert March, 1991 2(3):78-82. 3. Jim Heid. How To/Getting Started with Networks. MacWorld December, 1990:291-295. 4. David Kosiur. On the Ethernet Highway. MacWorld March, 1990:132-137. 5. op cit. E-Mail: A postal Inspection. MacWorld June, 1990:164-171. 6. op cit. EtherGate 1.45 (review). MacWorld August, 1990:196-197. 7. op cit. High-Speed Modems. MacWorld November, 1990:191-197. 8. op cit. Managing Networks. MacWorld February, 1991:152-159. 9. op cit. Going the Ethernet Route. MacWorld April, 1991:130-137. 10. Brita Meng. Timbuktu 3.1 (review). MacWorld October, 1990:237-239. 11. op cit. Networking for the Novice. MacWorld December, 1990:202-209. 12. Steven Sooby. Shiva EtherGate User Manual. Shiva Corporation. 13. Ed Tittel. Timbuktu/Remote 1.0.1 (review). MacWorld March, 1990:195.
