I ran across an interesting research paper titled “Perceiver: General Perception with Iterative Attention”, by A. Jaegle et al. (2021). A Perceiver model is a general Transformer. There are two key ideas. First, the architecture of a Transformer model has to be customized for different problem scenarios — image data, text data, etc. A Perceiver can accept any type of input. Second, Transformer models don’t scale well. A Perceiver uses a clever architecture that scales to huge models.

The first two pages of the Perceiver research paper.

The paper states:

Our core idea is to introduce a small set of latent units that forms an attention bottleneck through which the inputs must pass (Fig. 1). This eliminates the quadratic scaling problem of all-to-all attention of a classical Transformer and decouples the network depth from the input’s size, allowing us to construct very deep models. By attending to the inputs iteratively, the Perceiver can channel its limited capacity to the most relevant inputs, informed by previous steps.

Part of my brain is thinking, “Wow, Perceiver models are awesome!” Another part of my brain is thinking, “Sheesh! I am still figuring out the details of Transformer models and now I have this new architecture to figure out.”

The continuous stream of new ideas is simultaneously the beauty and challenge of machine learning.

Three images from an Internet search for “perceive portrait art”. Left: By artist Sergio. Center: By artist Mira Lanzillo. Right: By artist Renate Relenvie.