The 68030 and 68040 on the Zorro III Bus

by Michael Sinz


The Zorro III bus presents several special design issues for systems
with either a 68030 or 68040 CPU.  This article discusses those
design issues and offers solutions to the potential problems that
they present to those developing Zorro III devices, in particular,
Zorro III devices that are not exclusively memory expansion devices.


Background - 68030 and 68040 Caches

Both the 68030 and the 68040 have two caches; one for instructions
and one for data.  The 68030's caches are 256-bytes long.  The
68040's caches are considerably larger: each is 4K long.  Both CPU's
caches store memory in 16-byte blocks which are referred to as a
cache line.  The CPU only keeps one address for each cache line.
Each cache line is further broken up into long words called cache
entries.  On the 68030, each cache entry is marked as either valid or
invalid, telling the CPU which long words in the cache still contain
valid data.  On the 68040, only an entire cache line can be marked as
valid or invalid.

When the 68030 caches a memory address, it uses bits four through
seven of the address as a hash value.  This value is an index that
tells the 68030 which cache line to use for a specific address.  This
means each memory address corresponds to only one cache line.  For
example, if the 68030 tried to read a long word from 0x0FFFFF9F,
since the 68030 index extends from bit four through seven, the index
is 0x9, which corresponds to the tenth cache line.  This also means
that many memory addresses correspond to the same cache line.  For
example, the addresses 0x01FFFF9F, 0x02FFFF9F, 0x03FFFF9F, and
0x04FFFF9F all correspond to the same cache line.

The 68040 cache uses a similar indexing scheme, but the 68040 is a
little more dynamic.  The 68040 has four cache lines for each index,
so the 68040 cache can hold on to four different addresses that share
the same index.  The 68040 also has a larger index (six bits).

While the caches are active, when the CPU executes an instruction
that reads memory, it will check if that memory is already in the
cache.  If the memory is in the cache and the cache entry (or, in the
case of the 68040, the cache line) is marked as valid, a cache hit
occurs, so the CPU reads from the cache instead of main memory.  If
the memory is not in the cache, or the cache entry (or cache line) is
marked invalid, a cache miss occurs, so the CPU has to perform a main
memory fetch.

The cache lines are used in conjunction with the 68030's and 68040's
burst mode.  Normally, the 68030 fills its caches one entry at a
time.  While in burst mode, the CPU fills its caches a whole cache
line at a time. This helps to reduce the number of cache misses.

On the 68030, it is possible to turn burst mode on and off
independently of its caches.  If the 68030 cache is on and burst mode
is off, the 68030 can fill its cache a single long word at a time,
rather than the four words at a time it would do in burst mode.  The
68040 is different.  On the 68040, the only way to turn on burst mode
is to turn on the cache, so there is no way to prevent a burst access
when using the cache.  The 68040 always fills a whole cache line at a
time.

The instruction cache on both CPUs is fairly straightforward.  As the
CPU fetches instructions from memory, it copies them into the cache
for quick access later.  The only time the CPU changes the
instruction cache is when it does a memory fetch.

The data cache is different.  The data cache can change when the CPU
fetches memory and when the CPU executes an instruction that writes
to memory.  The 68030 and 68040 deal with this differently.

The 68030 data cache is a write-through cache.  This means whenever
the 68030 executes an instruction that writes to memory, the 68030
always performs a write to main memory, even if that address is in
the data cache.

When a data write causes a cache miss, the 68030 will act as if it
has no data cache and write directly to memory (except in
write-allocate mode--see the next paragraph).  When a data write
causes a cache hit, the 68030 will update the cache entry (or
entries) as well as write to memory.  Basically, on a memory write
operation, the 68030 will only update cache entries that are
currently cached.  It will not allocate a new cache entry for a cache
miss.

The 68030 data cache has a mode called write-allocate.  In this mode,
the 68030 not only updates the data in the cache, but, in the case of
a cache miss, the 68030 can also allocate a new cache entry.  While
in this mode, if a data write causes a cache miss, the CPU first
marks the corresponding cache entry as invalid (or, if in burst mode,
the CPU marks the entire cache line as invalid).  If the data write
is a long word write and it is aligned on a long word boundary, the
CPU updates that long word in the cache and marks it as valid. The
68040 data cache is not always a write through cache.  It has a mode
called copyback.  While in copyback mode, a write operation on the
68040 will not write through to memory.  The data will remain in the
data cache until the CPU flushes it out.  For more information, see
the article, ``68040 Compatibility Warning'' from the July/August
1991 Amiga Mail.


Zorro III and the 68030

When the 68030 reads data from a memory address, it will cache that
address only if that memory address is marked as cachable.  Certain
areas of memory cannot be cachable, for example, hardware registers
of a Zorro III card.  On the Zorro III bus, when the CPU attempts to
read an address that is not cachable, the device that exists in that
address space asserts the Zorro III cache inhibit line (/CINH).  The
bus controller will turn this signal into the CPU's cache inhibit
signal, which tells the CPU not to cache the address.

The problem is with the 68030's data cache in write-allocate mode
(which the Amiga OS requires).  When write-allocate mode is disabled,
the 68030 will only allocate a cache entry for a data address if the
address is cachable.  The CPU knows if the address is cachable
because the device told it using the cache inhibit line.

While in write-allocate mode, the 68030 will also allocate a cache
entry during certain write operations.  If, while in write-allocate
mode, the 68030 writes a long word to a long word aligned address,
the 68030 will write to that address and will allocate a cache entry
for that address.  This provides a loophole where the 68030 will
allocate a cache entry for a non-cachable memory address.  If the CPU
does a long word write to a Zorro III hardware register that happens
to be aligned on a long word address, the 68030 will put that address
in the cache.  If the CPU attempts to read from that address again
and that address happens to still be in the data cache, it will see
the value in the cache and will not attempt to read the hardware
register.

So far, the conditions under which this loophole can occur have been
rare.  The loophole requires that a hardware register be both
writable and readable, aligned on a long word address, and be four
bytes long. This precludes the Amiga custom chip registers as they
are not both readable and writable (in general they are not four
bytes long either).  Zorro II devices don't apply as Zorro II devices
only have a 16 bit wide data path.  The small size of the 68030's
data cache also makes it tough for a register write and read to occur
without a cache flush happening in between.

However, as Zorro III devices start to hit the market, the conditions
under which the loophole can occur will become more commonplace.  To
avoid this problem, Zorro III card designers can utilize the
following hardware trick.

The trick is to ``mirror'' all of the hardware registers.  In this
scheme, every register that is both readable and writable is
accessible at two addresses.  One address is exclusively for reading
and the other address is exclusively for writing.  Now, if the 68030
performs a write and allocates a cache entry, the 68030 caches the
writing address, but not the reading address.

Another hardware trick that might seem to be a viable solution is to
align 32-bit register ports so that they do not fall on a long word
boundary.  Using this method, the 68030 will never cache the register
address on a data write because it is not aligned properly.  The
problem with this method is that reading (or writing) a long word
from a non-long word aligned address is considerably slower than from
a long word aligned address. This can almost double the amount of bus
traffic, making the entire system slower.


The 68040 and Zorro III

The 68040 does not have the problem that the 68030 has with Zorro II
space.  The 68040 contains two registers to give data space a default
mapping without the need of a Memory Management Unit (MMU).  On an
Amiga with a 68040, Exec uses one of these registers to map the low
24-bits of the Amiga's address space (the Zorro II range,
$00000000-$00FFFFFFFF) as non-cachable and serialized1 .

The Amiga uses the second register to map the remaining memory
($01000000-$FFFFFFFF) as cachable and non-serialized.  Because of its
mapping, any RAM in this region will yield considerably higher
performance than RAM in Zorro II space.  Unfortunately, this mapping
can cause problems for a Zorro III device that is not RAM.

When the 68040 accesses a Zorro III device that is in cachable
address space, the device can still tell the CPU that an address is
not cachable by asserting the CPU's cache inhibit line.  This
overrides the default mapping the 68040 has placed on the address
space.  However, this does not stop the CPU from doing a full line
burst.  When accessing address space mapped as cachable, the 68040
will always attempt to read or write a block the size of an entire
cache line (four long words).

This presents a problem when the 68040 attempts to read from a Zorro
III device that is in cachable address space and the device asserts
the CPU's cache inhibit line.  The 68040 cannot notice that the Zorro
III device asserted the CPU's cache inhibit line until the 68040
reads the first long word of the burst cycle.  By the time the 68040
sees that the first long word is not cachable, it is already too late
to stop the burst cycle, so the 68040 finishes the burst.  When the
burst is done, the 68040 throws out the extra three long words from
the burst read.  In this case, the 68040 performs four long word
memory accesses instead of just one.

Writing to the device is even worse.  When the 68040 writes data to
an address that is not currently in the data cache, the 68040 will
first try to fill a cache line.  When the 68040 sees that the device
asserts the CPU's cache inhibit line, it will finish the read and
then write out one long word.  Essentially, to perform a single
memory write, the 68040 performs four memory reads and one memory
write.

These excessive memory accesses can significantly hinder system
performance.  Certain Zorro III designs could make the 68040 as much
as four to five times slower.

The full line bursts also cause a second potential problem for some
possible Zorro III devices.  Reading certain types of hardware
registers will trigger the hardware to perform some extra function as
well.  It is not uncommon for a hardware device to supply a new data
value for a register after the CPU reads that register. If a Zorro
III device has such a register and the device is located in cachable
address space, the device can experience problems with reads and
writes of addresses surrounding the register.  If the CPU reads a
second hardware register at an address that is in the same quad-long
word as the register (i.e. the first register's address would be in
the same cache line as the second register's address),   when the CPU
performs its full line burst, it will read the first register in
addition to the second register.  Because the CPU reads the first
register, the device will reload the first register with a new value,
losing the previous value.


The Solution

There is a solution that will fix both potential problems for Zorro
III cards on 68040-based Amigas.  The MMU in the 68040 can map
specific pages of memory as non-cachable.  The 37.10 version of the
68040.library creates MMU tables that map only Zorro III memory
devices as cachable (actually it maps all RAM except Chip RAM as
cachable).  The library marks other Zorro III devices as
non-cachable.  The new library prevents the 68040 from doing full
line bursts to non-cachable devices, so the CPU only reads or writes
one long word at a time.  As the 68040.library uses the MMU to map
all address space, invalid addresses can no longer cause bus errors
(Guru #00000002), which may help a few ill-behaved products to work
on 68040 systems.

The 37.10 68040.library is part of the V39 OS present on the current
A4000.  Developers who are working on or have released 68040-based
expansion devices should contact CATS to obtain information on
distributing the library with their product.

There is only one problem with this solution.  Not all 68040s have
MMUs.  There are three kinds of 68040 chip: the MC68040, the
MC68LC040, and the MC68EC040.  The MC68040 has both an Floating Point
Unit (FPU) and an MMU.  The MC68LC040 is a regular MC68040 without an
FPU.  The MC68EC040 is a MC68LC040 without an MMU.

As the 68040.library requires an MMU to map address space, the fix
described above will not work on systems with an MC68EC040.  Because
burst mode on the 68040 is activated along with the cache, there is
no way to prevent a 68EC040-equipped Amiga from doing full line
bursts when accessing cachable address space.  This means a 68EC040
cannot prevent the excessive reads and writes when reading
non-cachable Zorro III devices that reside in cachable address space.
A 68EC040-equipped Amiga will experience a significant decrease in
performance when accessing non-cachable Zorro III devices.  For this
reason we cannot recommend that anyone use a 68EC040 (or any future
68000 series CPU that has no MMU) as the CPU on a Zorro III bus
system.

If someone decides not to heed this warning and create a 68EC040 CPU
card for the Zorro III bus, there is nothing the 68EC040 card can do
to prevent these problems, although there is still a way for a Zorro
III card to prevent the second 68040 problem (the ``register
trigger'' problem).  A Zorro III card that needs to use ``trigger''
registers can arrange the trigger registers so that each register is
in its own quad-long word.  This way, when the 68EC040 reads one of
these registers, the read operation won't disturb other registers, as
two registers do not reside in the same quad-long word.  Note that
this fix will not prevent the first problem (the performance decrease
problem) and it does not address the possibility that future CPUs may
have an eight or sixteen long word cache line.