Cleverer cache management for better chip performance

Cleverer management of the local memory banks known as ‘caches’ could improve computer chips’ performance while reducing their energy consumption.

Computer chips keep getting faster because transistors keep getting smaller. But the chips themselves are as big as ever, so data moving around the chip, and between chips and main memory, has to travel just as far. As transistors get faster, the cost of moving data becomes, proportionally, a more severe limitation.

In a pair of recent papers, researchers at MIT and the University of Connecticut have developed a set of new caching strategies for massively multicore chips that, in simulations, significantly improved chip performance while actually reducing energy consumption.

The first paper, presented at the most recent ACM/IEEE International Symposium on Computer Architecture, reported average gains of 15% in execution time and energy savings of 25%. The second paper, which describes a complementary set of caching strategies and will be presented at the IEEE International Symposium on High Performance Computer Architecture, reports gains of 6% and 13%, respectively. The caches on multicore chips are typically arranged in a hierarchy. Each core has its own private cache, which may itself have several levels, while all the cores share the so-called last-level cache, or LLC.

Chips’ caching protocols usually adhere to the simple but surprisingly effective principle of ‘spatiotemporal locality’. Temporal locality means that if a core requests a particular piece of data, it will probably request it again. Spatial locality means that if a core requests a particular piece of data, it will probably request other data stored near it in main memory.

So every requested data item gets stored, along with those immediately adjacent to it, in the private cache. If it falls idle, it will eventually be squeezed out by more recently requested data, falling down through the hierarchy - from the private cache to the LLC to main memory - until it’s requested again.