Newsgroups: fj.sys.sun
Path: galaxy.trc.rwcp.or.jp!coconuts.jaist!wnoc-tyo-news!wnoc-sfc-news!venus.sun.com!cs.utexas.edu!howland.reston.ans.net!ix.netcom.com!netcom.com!sflower
From: sflower@netcom.com (Kim Vu)
Subject: Fast sharing data without accessing the net
Message-ID: <sflowerD71LKz.DKx@netcom.com>
Organization: NETCOM On-line Communication Services (408 261-4700 guest)
X-Newsreader: TIN [version 1.2 PL1]
Date: Fri, 14 Apr 1995 20:43:47 GMT
Lines: 109
Sender: sflower@netcom21.netcom.com
Xref: galaxy.trc.rwcp.or.jp fj.sys.sun:5935
X-originally-archived-at: http://galaxy.rwcp.or.jp/text/cgi-bin/newsarticle2?ng=fj.sys.sun&nb=5935&hd=a
X-reformat-date: Mon, 18 Oct 2004 15:18:22 +0900
X-reformat-comment: Tabs were expanded into 4 column tabstops by the Galaxy's archiver. See http://katsu.watanabe.name/ancientfj/galaxy-format.html for more info.

A Storage Crossbar for UNIX Workstations

The power of a few workstations with access to each other and the same
data has driven mainframes into history.  As in the human brain, this
power is as much in the connections as in the comput- ing nodes.  The
network is the computer," as Sun says. 

But the thing that connects most workstations today is a rather slow wire
with many layers of software.  A typical client works- tation connects to
a server on an Ethernet with other clients and servers.  A four Megabyte
read request in the client becomes a lookup, a getattr and 512 eight
Kilobyte reads.  Each of these 514 operations travels through six layers
of network software to reach the Ethernet wire, waits among the other
clients and servers on the net for a chance to use the wire, travels
across the wire at a slow 10 Mbits/s, and finally travels up through
another six layers of network software at the server.  Slowing the request
even more, the bottom network layer breaks each eight Kilobytes into six
smaller packets that must each wait for a turn to cross the wire.  The
result of all this is that a larger file that takes less than a second to
read locally takes 20 seconds or more to read over Ethernet. 

The Storage Crossbar from Solflower bypasses the net and its overhead by
connecting workstations directly to shared storage.  Workstations can
share files, the main reason for using a net, at local disk speeds. The
Storage Crossbar connects 16 workstations directly to 56 SCSI devices
through a physical crossbar that al- lows simultaneous mounting and
sharing of file systems.  All the workstations see all the data.  A net
may connect to the worksta- tions for communication and to serve the
Storage Crossbar data out to workstations not directly connected. 
Attached worksta- tions may also communicate with each other through the
Storage Crossbar.  A 64 Megabyte battery-backed cache eliminates most
mechanical movement.  The eight SCSI buses attached to the Storage
Crossbar and the Storage Crossbar cache may be transfer- ring
simultaneously to workstations for a total bandwidth of 75 Megabytes per
second. 

Sharing UNIX Disks

Attaching a UNIX disk directly to more than one workstation, us- ing a
dual ported disk or multi-initiator SCSI, is normally a disaster. The
reason we haven't been able to share UNIX disks directly until now is that
the UNIX file system in each worksta- tion UFS, caches the state of the
disk file system, things like the number of free blocks and where they
are.  When two worksta- tions without knowledge of each other expand files
into the same area, the file system crashes. 

The Storage Crossbar permits direct sharing by replacing UFS with
Solflower's Shareable File System. SFS executes all of the UFS operations
and knows that other workstations may also be access- ing the file system. 
Working through and with the Storage Crossbar, SFS resolves all contention
problems and provides safe, transparent sharing for applications.  The
interface to SFS is identical to the UFS interface, making SFS invisible
to the ap- plications and making SFS disks readable and writable by UFS
sys- tems. 

SFS is loaded into the workstations with pkgadd and resides next to UFS as
an option.  Users simply "mount -F ufs" for private use of a file system
or "mount -F sfs" for shared use of a file system.  A file system is
automatically mounted for sharing if sfs appears in its FS Type column in
/etc/vfstab. 

Applications

When workstations can share disks directly, new applications open up. 
Most obvious is a net replacement, where data intensive ap- plications in
up to 16 workstations within a 200-foot radius, the maximum length of the
coax cable attaching the workstations to the Storage Crossbar, access
shared data at local disk speeds. 

Other applications focus on servers, where the Storage Crossbar allows all
the servers to see all the storage.  A number of servers are paired, with
each member of each pair receiving the other member's Ethernet as well as
its own. The servers listen for the partner's heart beat through the
Storage Crossbar.  When a server stops, the opposite server of the pair
enables itself to service requests from the abandoned Ethernet, in
addition to its own Ethernet.  Any of the servers can fail or be taken off
line for maintenance with no interruption to the clients.  Since each
server already has the file systems of its partner mounted, fail- over is
instantaneous. 

Comparison to Other Solutions

Other ways to speed up access to shared disks include fast nets and super
servers.  Both leave the net in place, which leaves the 12 layers of
network software in place.  The 12 layers of software explain why fast
nets typically only double performance while increasing the net speed by
ten. 

Super servers speed up the server end of the client's request with
dedicated processors, but do not help the client to work through its
network protocol layers or to get a turn faster on the Ethernet. The
throughput for super servers is high because a large number of nets can be
served simultaneously, but the per- formance for any one client on any one
net still does not ap- proach local disk performance. 

Neither fast nets nor super servers give servers access to each other's
storage, as the Storage Crossbar does to allow failover and fast backups.
 
Conclusion

The Storage Crossbar from Solflower Computer begins a new era in
workstation storage.  Instead of a server connected to storage which it
serves out over a net to clients, all the clients now access the storage
directly, without the server and without the net overhead. The sharing
capabilities of the net are now provid- ed with the speed of local disks.
For server applications, where there are too many clients to attach
directly, the Storage Crossbar finds use among the servers by providing
connectivity that allows failover and netless backup. 

