Technical Notes for The Payroll Employment Game

This report briefly describes the source of technical information about the BLS Establishment Survey used in constructing The Payroll Employment Game. The links are current as of August 27, 2006.

The BLS home page and employment statistics page point to a great deal of information about the monthly job report, The Employment Situation.

There is an excellent overview Employment Situation Explanatory Note.

For additional technical information about the design, conduct, and analysis of the Establishment Survey from which the job report is derived, see Technical Note 1 and Technical Note 2.

These Technical Note Tables are of particular interest:

The key points the Employment Game is designed to make are the nature of the information contained in the job report and the risks inherent in acting on a mistaken belief that the BLS job growth estimate for the current month represents an absolutely exact and true measure of the economy.

We do not intend to be critical of BLS in general nor of the Establishment Survey in particular. We believe the Establishment Survey is an excellent survey conducted, analyzed, and reported by dedicated and skillful scientists. Moreover, we believe the Establishment Survey provides an estimate of the change in seasonally adjusted jobs from one month to the next which is as accurate as it is possible to be in measuring anything at all involving actual humans in that time frame. Nor does the BLS misrepresent their work. They go to great lengths to explain what they do, why, and how and to explain how to interpret the results. As the old guy said: 'The fault, dear Brutus, is not in our stars, but in ourselves...'

Here is a brief summary of the facts as we understand them. The BLS Establishment Survey is used to estimate the number of jobs in the economy every month. From that estimate, a seasonally adjusted number of jobs is calculated. The seasonally-adjusted monthly change in jobs is calculated by subtracting the estimate for the current month from the previous month. This estimate is revised in subsequent months as more complete data are available.

There are usually two adjustments to monthly survey estimates of number of jobs and month-to-month change in number of jobs as more survey data are reported and analyzed. One may expect these successive monthly adjustments to be increasingly more accurate. However, the initial estimate is so good that the net revisions made within two months of the original survey estimate are zero over a period of many years. On balance, one may argue that the initial monthly estimate is as good an estimate as it is possible to have. For more information on these near-term monthly revisions, from Technical Notes Tables , select: 'Table 2-D. Errors of preliminary employment estimates'

So, that is the estimate of monthly change in jobs. The question is, how good is the estimate? How close is the estimate likely to be to the true (seasonally-adjusted) difference in the actual number of the jobs at some point this month from the actual number of jobs at the same point in the previous month? This is where the BLS sampling statisticians come in, and what they provide is absolutely crucial to understanding what the estimate really means.

BLS estimates that the standard error in the month-to-month change in number of nonfarm jobs is about 60,000 jobs. This means that about 2/3 of the time, the BLS estimate in job change will be within +/- 60,000 jobs of the true change in the number of jobs in the economy (as defined by BLS). See overview of revisions of monthly estimates at Technical Notes 2. For detailed information, from Technical Notes Tables , select: 'Table 2-F. Standard Error for Change in Levels, First Closing Detail'

It is customary to pick a meaningful value to use as the standard for statistical significance. BLS has chosen the 90% confidence level as their measure of statistical significance for this study. This means that they regard an estimated increase of more than 99,000 jobs as a statistically significant increase in jobs and regard an estimated decrease of more than 99,000 jobs as a statistically significant decrease. If the estimated change in jobs is less than 99,000, then the change is not statistically significant; BLS is not confident that there was an increase or a decrease and neither should you. BLS has a very readable explanation of use and interpretation of the 90% confidence interval. From Explanatory Note find the section 'Reliability of the estimates'.

For an example on calculating a 90% confidence interval, in Technical Notes 2. find the section 'Illustration of the use of table 2-f' near the end of the page.

BLS is careful to note that every survey including the Establishment Survey contains both sampling error and non-sampling error. It is only the sampling error, the natural variation from one random set of observations to another possible random set of observations, which is estimated by the standard error. Non-sampling error includes everything else: failure to respond or be measured, differences in how the questions are interpreted from person to person, error in recording the intended answer, and so on. Non-sampling error specific to the Establishment Survey also comes from the creation, change, and dissollution of businesses at any point in time.

BLS also makes Benchmark revisions to the job estimates made with the Establishment Survey. In one month per year, BLS counts the number of jobs based on actual unemployment insurance tax forms in each state. BLS scientists then compare the actual count of jobs from the Benchmark to the number of jobs estimated by with the Establishment Survey. Then BLS revises the survey estimates for several following months based on Benchmark information. It is very likely that all systematic knowledge BLS gains from these adjustments finds its way into the design, conduct, and analysis of future employment surveys. BLS cautiously notes that although the Benchmark seems like an exact count, it is also subject to some non-sampling error.

What can we learn from the Benchmark revisions? The 12 monthly 2005 adjustments (differences between previous estimates and the BM2005 benchmark report) in total nonfarm jobs (seasonally adjusted) range from a decrease of 64,000 jobs to an increase of 49,000 jobs with a mean adjustment of a decrease of 3,000 jobs. Another way to look at the magnitude of the Benchmark revisions: 'Over the past decade, the benchmark revision for total nonfarm employment has averaged 0.2 percent, ranging from less than 0.05 percent to 0.4 percent.'(*) We can see confirmation of two things: First, the Benchmark revisions confirm the quality of the estimates made in the monthly estimates of number of jobs. Second, the the range of differences between survey estimates and revised estimates provides additional confirmation to the confidence interval BLS provides as guidance in interpreting and using the monthly BLS job report. (*) See Technical Note 1 at "Benchmarks" for a good overview. Also see more See extensive and detailed information on the benchmark revision process.

The Employment Game deals only with the BLS estimate of month-to-month change in number of nonfarm jobs and the sampling error. It does not include the effect of non-sampling error or the subsequent Benchmark revision.