95% confidence interval
Assuming that the error distribution is normal (or approximately so), we can use the fact that 95% of any normal distribution is between 1.96 standard deviations of the mean.
We can therefore write
\[ \begin{align} P\left(-1.96 \times \se(\hat {\theta}) \;\;\lt\;\; error \;\;\lt\;\; 1.96 \times \se(\hat {\theta})\right) \;\;&\approx\;\; 0.95 \\ P\left(\hat{\theta}-1.96 \times \se(\hat {\theta}) \;\;\lt\;\; \theta \;\;\lt\;\; \hat{\theta}+1.96 \times \se(\hat {\theta})\right) \;\;&\approx\;\; 0.95 \end{align} \]We call the interval
\[ \hat{\theta}-1.96 \times \se(\hat {\theta}) \quad \text{ to } \quad \hat{\theta}+1.96 \times \se(\hat {\theta}) \]a 95% confidence interval for \(\theta\) and we have 95% confidence that it will include the actual value of the parameter.
Other confidence levels
90% of values from a normal distribution are within 1.645 standard deviations of the distribution's mean, so
This leads to a 90% confidence interval,
\[ \hat{\theta}-1.645 \times \se(\hat {\theta}) \quad \text{ to } \quad \hat{\theta}+1.645 \times \se(\hat {\theta}) \]We say that we are 90% confident that an interval that is calculated in this way will include the true parameter value, \(\theta\).
Binomial example