Writing is part of the research

It's tempting to think of writing as the chore that comes after the science. In practice, writing is where you discover whether you understand your own result. A vague paragraph usually means a vague idea. A paper is also how your work enters the conversation: an unwritten result, or an unreadable one, might as well not exist.

In his much-loved talk, computer scientist Simon Peyton Jones argues that you should write early: draft the paper before the research is finished, because the draft tells you what you still need to find out.

The shape of a paper

A good ML paper tells one story. It rarely follows the order in which you actually did the work. It follows the order in which a reader needs to understand it:

1. 1

   The problem

   What are we trying to do, and why does anyone care?
2. 2

   The gap

   Why don't existing approaches already solve it?
3. 3

   The idea

   Our key insight, in one or two sentences.
4. 4

   The evidence

   Experiments that would convince a skeptic, with baselines, ablations and error bars.
5. 5

   The implications

   What changes if we're right? What are the limitations?

The introduction carries that whole arc in miniature and ends with an explicit list of contributions, the same list you learned to hunt for in Chamber 6. Many readers stop after the introduction, so make it complete. And design Figure 1 to communicate the key idea or result at a glance: plenty of people look at it before they read anything else.

The abstract: five moves

The abstract is the most-read part of any paper, so it deserves the most rewriting. Strong ML abstracts almost always make the same five moves, in the same order: context → problem → approach → results → implications. Put this one back together:

Writing clearly

Clear technical writing is a learnable skill. A few principles do most of the work:

• One idea per paragraph, announced in its first sentence.
• Define notation before you use it, then never change it.
• Numbers beat adjectives. "Improves accuracy by 2.3 points" says more than "dramatically improves".
• Hedge honestly. Results suggest; proofs show. Don't claim more than your evidence supports.
• Cut ruthlessly. Every sentence should earn its place.

LaTeX essentials

Almost every ML paper is typeset in LaTeX, and the major conferences (NeurIPS, ICML, ICLR and more) provide LaTeX templates. You write plain text with markup, and LaTeX handles the typography, especially the mathematics. Online editors such as Overleaf mean you don't need to install anything to get started.

Here is the skeleton of a paper, including an equation you can reference, a citation and a publication-quality table:

\documentclass{article}
\usepackage{amsmath, amssymb}   % mathematics
\usepackage{graphicx}           % figures
\usepackage{booktabs}           % professional tables
\usepackage{hyperref}           % clickable references

\title{Gradient-Aware Warmup for Small Networks}
\author{Ada Researcher}

\begin{document}
\maketitle

\begin{abstract}
Small networks trained with large learning rates often diverge early in training...
\end{abstract}

\section{Introduction}
We minimise the empirical risk
\begin{equation}
  \mathcal{L}(\theta) = \frac{1}{n}\sum_{i=1}^{n} \ell\big(f_\theta(x_i), y_i\big)
  \label{eq:risk}
\end{equation}
using gradient descent. Equation~\eqref{eq:risk} is standard~\cite{rumelhart1986learning}.

\begin{table}[t]
  \centering
  \begin{tabular}{lcc}
    \toprule
    Method        & Accuracy (\%)    & Diverged runs \\
    \midrule
    Linear warmup & $91.2 \pm 0.4$   & 3/20 \\
    Ours          & $92.0 \pm 0.3$   & 0/20 \\
    \bottomrule
  \end{tabular}
  \caption{Mean $\pm$ standard deviation over 20 seeds.}
  \label{tab:main}
\end{table}

\bibliographystyle{plain}
\bibliography{references}   % entries live in references.bib
\end{document}

Citations live in a separate BibTeX file. Most paper pages (and Google Scholar) will give you the entry:

@article{rumelhart1986learning,
  title   = {Learning representations by back-propagating errors},
  author  = {Rumelhart, David E. and Hinton, Geoffrey E. and Williams, Ronald J.},
  journal = {Nature},
  volume  = {323},
  number  = {6088},
  pages   = {533--536},
  year    = {1986}
}

The commands you'll type most often:

Now forge some equations of your own. Every one of them is an old friend from this course:

Peer review

Before a paper is published at a conference or in a journal, other researchers review it. They judge its soundness (are the claims supported?), significance (does it matter?), novelty (is it new?) and clarity (can it be understood?). Chamber 7 made you a reviewer. Being one makes you a better author, because you learn to anticipate the questions.

When the reviews come back:

• Thank reviewers, then address every point specifically, with evidence where you can.
• Change the paper, not just the rebuttal. If a reviewer misunderstood, other readers will too.
• Stay calm. Review is noisy: in the NeurIPS 2014 experiment, two independent committees reviewed the same set of papers, and roughly half of the papers accepted by one were rejected by the other. A rejection is data, not a verdict on you.

Your first research note

Time to write. In Chamber 5 you ran a real experiment: how often does a network with a given number of hidden units solve XOR? Write it up as a short research note, with a question, a hypothesis, a setup, results and conclusions. This is the same structure as a paper, in miniature. If you haven't collected your numbers yet, open the trainer in another tab.