Estimating proportions
A proportion is just a mean of 0/1 indicators — so SE and CI ideas carry over.
If $n_x$ out of $n$ items have a property, the sample proportion is $p = n_x / n$. Think of each item as $z_i \in \{0,1\}$; then $p$ is their mean.
$$p = \frac{n_x}{n}$$
$$\sigma_p = \sqrt{\frac{p(1-p)}{n}}$$
$$p \pm 1.96\,\sigma_p$$
Classifier accuracy
67 correct out of 73 images ⇒ $p \approx 0.918$. SE $\approx \sqrt{{0.918\times 0.082 / 73}} \approx 0.032$, so a 95% CI is roughly $91.8\% \pm 6.3\%$. Wide! You cannot yet claim a clear win over a 90% old system.
Extreme $p$
When $p$ is near 0 or 1, use Agresti–Coull: $\tilde{{p}}=(n_x+2)/(n+4)$, $\tilde{{n}}=n+4$.
Key takeaways
- Proportion = mean of indicators.
- SE $=\sqrt{{p(1-p)/n}}$.
- Near 0% or 100%, prefer Agresti–Coull.