PaperStack

A desktop workspace for the papers you're actually reading.

Download for macOS

Version 0.1.0·20 MB·Apple Silicon & Intel

To read 4

⋮

ImageNet Classification with Deep Convolutional Neural Networks Krizhevsky et al. 2012

Dropout: A Simple Way to Prevent Neural Networks from Overfitting Srivastava et al. 2014

Generative Adversarial Networks Goodfellow et al. 2014

Method 2

⋮

Attention Is All You Need Vaswani et al. 2017

BERT: Pre-training of Deep Bidirectional Transformers Devlin et al. 2019

Deep Residual Learning for Image Recognition He et al. 2016

Hold ⇧ to make a pile

Method 5

⋮

Positional encoding

Attention Is All You Need Vaswani et al. 2017

Self-Attention with Relative Position Representations Shaw et al. 2018

Normalization tricks 3 papers

3.2 Scaled dot-product attention

We compute attention as a weighted sum over value vectors, where the weight assigned to each value follows from a compatibility function [12] of the query with the corresponding key. Scaling the dot products by the square root of the key dimension keeps the softmax in a regime with useful gradients, an adjustment first reported for deep recurrent stacks. Without it the products grow with dimension and the distribution saturates.

In practice we found it useful to project queries, keys and values several times with different learned matrices and to run the attention function in parallel over those projections. Concatenating the results recovers a representation of the same width as a single full-dimensional head at roughly the same cost [7].

The same formulation covers the encoder–decoder case: queries come from the previous decoder layer while keys and values come from the encoder output, so every position in the decoder may attend over the whole input sequence.

Layer Normalization Ba, Kiros, Hinton 2016 arXiv preprint arXiv:1607.06450 Training state-of-the-art, deep neural networks is computationally expensive. One way of reducing the training time is to normalize the activities of the neurons.

Open

BibTeX

Citation key

3.2 Scaled dot-product attention

We compute attention as a weighted sum over value vectors, where the weight assigned to each value follows from a compatibility function of the query with the corresponding key.

Scaling the dot products by the square root of the key dimension keeps the softmax in a regime with useful gradients, an adjustment first reported for deep recurrent stacks. Without it the products grow with dimension and the distribution saturates. Highlight

The same formulation covers the encoder–decoder case: queries come from the previous decoder layer while keys and values come from the encoder output.

Attention Is All You Need Vaswani et al. 2017

Open

Show in Folder

Copy BibTeX

Copy as LaTeX

Add note…

Analyze again

Remove from project

On your clipboard

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks. \cite{vaswani2017attention}

We propose a new simple network architecture, the Transformer, based solely on attention mechanisms. \cite{vaswani2017attention}

Self-attention layers connect all positions with a constant number of sequential operations. \cite{vaswani2017attention}

macOS will say it's damaged. It isn't.

I just don't want to pay 99$ a year to Apple so the message shows for any app whose developer hasn't paid Apple's yearly fee. Opening it the first time takes three steps, and you only do them once.

Drag PaperStack into Applications, then double-click it once and let it fail.
Open System Settings → Privacy & Security and scroll down to Security.
Beside the message about PaperStack, click Open Anyway.

Or build it yourself

It's the same code but needs Node, Rust and the Xcode command line tools.

# clone, install, build straight into /Applications
git clone https://github.com/Antonio-Leitao/paperstack.git
cd paperstack
npm install
npm run install:local