An Audit Report on

Performance Measures at the
Commission on Jail Standards

John Keel, CPA
State Auditor

May 25, 2006
Members of the Legislative Audit Committee:
Five (83 percent) of the six performance measures audited for fiscal
year 2005 and the first quarter of fiscal year 2006 at the Commission
on Jail Standards (Commission) were unreliable. Specifically:

Background
Entities report results for their key
measures to the Legislative Budget
Board’s budget and evaluation
system, which is called the
Automated Budget and Evaluation
System of Texas, or ABEST.

ƒ

Factors prevented the certification of three output measures:
“Number of On-site Planning and Construction Consultations
with Jail Representatives,” “Number of On-site Operation and
Management Consultations with Jail Representatives,” and
“Number of Reports Audited.” The Commission should retain
sufficient documentation to support its reported numbers for these three measures.

ƒ

Two outcome measures--“Number of Jails Achieving Compliance” and “Percent of Jails with
Management-related Deficiencies”--were inaccurate. To improve accuracy, the Commission should
review data used in the calculation of these measures; review data entered into the Automated Budget
and Evaluation System of Texas (ABEST) for these measures; and document policies and procedures for
the data collection, entry, and calculation of these measures.

ƒ

Auditors certified with qualification the output measure “Number of Annual Inspections Conducted.”
However, the Commission should improve controls over data collection, calculation, and reporting to
ensure the continued reliability of this measure.

The State Auditor’s Office assessed the accuracy of reported performance measures based on criteria in the
Guide to Performance Measure Management: 2000 Edition (State Auditor’s Office Report No. 00-318,
December 1999).
The attachment to this letter contains additional details on the performance measures tested, certification
results, recommendations, and management’s responses.

SAO Report No. 06-040
Robert E. Johnson Building
1501 North C ongre ss Avenue
Austin, Te xas 78701

P.O. Box 12067
Austin, Te xas 78711-2067

Phone: (512) 936-9500
Fax : (512) 936-9400
Interne t: www.sao.state.tx .us

Members of the Legislative Audit Committee
May 25, 2006
Page 2
The Commission generally agrees with our recommendations, and we appreciate its cooperation during this
audit. If you have any questions, please contact Verma Elliott, Audit Manager, or me at (512) 936-9500.
Sincerely,

John Keel, CPA
State Auditor
Attachment
cc:

Members of the Commission on Jail Standards
Mr. Terry Julian, Executive Director, Commission on Jail Standards

This document is not copyrighted. Readers may make additional copies of this report as needed. In
addition, most State Auditor’s Office reports may be downloaded from our Web site: www.sao.state.tx.us.
In compliance with the Americans with Disabilities Act, this document may also be requested in alternative
formats. To do so, contact our report request line at (512) 936-9880 (Voice), (512) 936-9400 (FAX), 1-800RELAY-TX (TDD), or visit the Robert E. Johnson Building, 1501 North Congress Avenue, Suite 4.224, Austin,
Texas 78701.
The State Auditor’s Office is an equal opportunity employer and does not discriminate on the basis of race,
color, religion, sex, national origin, age, or disability in employment or in the provision of services,
programs, or activities.
To report waste, fraud, or abuse in state government call the SAO Hotline: 1-800-TX-AUDIT.

Attachment
Performance Measure Certification Results
Auditors tested the accuracy of the Commission on Jail Standards’
(Commission) key outcome performance measures for fiscal year 2005 and its
key output measures for fiscal year 2005 and the first quarter of fiscal year
2006.1 Table 1 summarizes the certification results from audit testing.
Table 1

Commission on Jail Standards (Agency No. 409)
Fiscal Year

Results Reported
in ABEST

A, Outcome, Number of Jails Achieving
Compliance

2005

211

Inaccurate

A, Outcome, Percent of Jails with
Management-related Deficiencies

2005

15%

Inaccurate

2005

252

2006 (1st Quarter)

69

2005

73

2006 (1st Quarter)

4

2005

171

2006 (1st Quarter)

41

2005

6,248

2006 (1st Quarter)

1,593

Related Objective or Strategy,
Classification, and Description of Measure

A.1.1, Output, Number of Annual Inspections
Conducted
A.2.1, Output, Number of On-site Planning
and Construction Consultations with Jail
Representatives
A.2.2, Output, Number of On-site Operation
and Management Consultations with Jail
Representatives
A.3.1, Output, Number of Reports Audited
a

a

Certification Results

Certified with Qualification

Factors Prevent Certification

Factors Prevent Certification

Factors Prevent Certification

In fiscal year 2006, the name of this measure was changed to “Number of Reports Analyzed.”

A measure is Certified if reported performance is within +/-5 percent of actual performance and if controls appear adequate to
ensure accuracy for collecting and reporting performance data.
A measure is Certified With Qualification if reported performance is within +/-5 percent of actual performance but controls
over data collection and reporting are not adequate to ensure continued accuracy.
A measure is Inaccurate when reported performance is not within +/-5 percent of actual performance or there are more than
two errors in the sample tested.
Factors Prevent Certification when actual performance cannot be determined because of insufficient documentation and
inadequate controls or when there is deviation from the measure definition and the auditor cannot determine the correct result.

1

Outcome measures are reported only annually; output measures are reported quarterly.
Attachment
An Audit Report on Performance Measures at the
Commission on Jail Standards
SAO Report No. 06-040
May 2006
Page 1

The Commission Should Improve its Reviews of and Policies and
Procedures for Reporting Performance Measures

For all performance measures tested, the Commission does not have sufficient
controls to ensure its reported performance measures are accurate.
Specifically:
ƒ

The Commission does not perform supervisory review of performance
measure calculations.

ƒ

The Commission does not review data that has been entered into the
Automated Budget and Evaluation System of Texas (ABEST) before it is
released into ABEST.

ƒ

The Commission does not have written policies and procedures
documenting data collection, entry, calculation, and review of
performance measures.

Lack of supervisory review and policies and procedures impairs the accuracy
of reported performance measures.
Recommendations

The Commission should:
ƒ

Implement a supervisory review process to ensure that the data entry,
calculation, and reporting to ABEST of performance measure results are
accurate.

ƒ

Develop and implement written policies and procedures for data entry,
calculation, and reporting of performance measures.

Management’s Responses

Commission staff have developed a documented review and verification of
performance measure calculations and ABEST data entry (prior to release),
and are in the process of developing written policies and procedures
regarding these activities. These policies and procedures will include clear
and concise instructions on exactly how each measure is calculated, including
formulas, documented data sources, and location of all data sources.
Documented review processes have been implemented and will be included in
the policy and procedures revisions. Definitions have been clarified in the
ABEST descriptions of relevant performance measures; staff is awaiting
responses/approval by the LBB and the Governor’s Office on Budget and
Attachment
An Audit Report on Performance Measures at the
Commission on Jail Standards
SAO Report No. 06-040
May 2006
Page 2

Policy. Retention periods of less than three (3) years for any documentation
employed in performance measure calculations has been extended to three (3)
years.

Key Measures

Number of On-site Planning and Construction Consultations with
Jail Representatives
Number of On-site Operation and Management Consultations with
Jail Representatives
Factors prevented the certification of these measures because
the Commission did not maintain adequate documentation to
support the results it reported in ABEST. The Commission
has little documentation to support the number of on-site
consultations reported in ABEST, and documentation that
does exist may not indicate which type of consultation
(planning and construction or operation and management)
took place or what was discussed during the consultation. Therefore, the
number of consultations reported in ABEST could not be re-created.

Results: Factors Prevent
Certification
Actual performance cannot be
determined because of insufficient
documentation and inadequate
controls or when there is deviation
from the measure definition and
the auditor cannot determine the
correct result.

Recommendations

The Commission should:
ƒ

Develop and maintain all source documentation that supports the
performance measure results, including a synopsis of each on-site
consultation.

ƒ

Develop written policies and procedures for performing on-site planning
and construction and operation and management consultations with jail
representatives.

Management’s Responses

New documentation has been developed as additional data support for the
tracking of on-site consultations to ensure accurate reporting of these
planning and construction and management consultations. Policies and
procedures are being revised to incorporate the new documentation into the
clear and concise instructions for the process of documenting and reporting
these measures. Definitions have also been clarified.

Attachment
An Audit Report on Performance Measures at the
Commission on Jail Standards
SAO Report No. 06-040
May 2006
Page 3

Number of Reports Audited
Results: Factors Prevent
Certification
Actual performance cannot be
determined because of insufficient
documentation and inadequate
controls or when there is deviation
from the measure definition and
the auditor cannot determine the
correct result.

Factors prevented the certification of this measure because the
Commission did not maintain adequate documentation to support
the number of reports received, analyzed, or revised. Therefore,
the number of reports audited as reported in ABEST could not be
re-created. Specifically:
ƒ

The Commission does not consistently date-stamp original
reports upon receipt.

ƒ

The Commission overwrites electronic checklists indicating reports
received, and it does not maintain hard copies.

ƒ

The Commission includes reports that have been revised in this measure.
It disposes of a handwritten checklist indicating monthly revised reports
after one year.

Recommendation

The Commission should date-stamp all reports received and maintain source
documentation to support the performance measure results.
Management’s Responses

Staff has been instructed to ensure consistent procedures in date-stamping
original reports as they are received. Electronic checklists are being stored,
rather than over-written, and hard copies of these checklists will be produced
and retained. The hard copies of the checklists, as well as both original and
revised reports, will be retained for a minimum of three (3) years. Policies
and procedures are being revised to reflect these changes.

Number of Jails Achieving Compliance
This measure was inaccurate because testing found that the number of jails
achieving compliance was incorrect in 5.2 percent of the items
Results: Inaccurate
tested. For example, the Commission included two counties as
Reported performance is
not within +/-5 percent
compliant that had not received an inspection within the fiscal
of actual performance or
year, and it excluded two counties that had received compliant
there are more than two
errors in the sample
inspections. Jails are certified when they receive a compliant
tested.
rating on their most recent inspection.
The measure definition also does not contain enough information to be clearly
understood and easily recalculated. The measure definition is not specific
Attachment
An Audit Report on Performance Measures at the
Commission on Jail Standards
SAO Report No. 06-040
May 2006
Page 4

regarding what constitutes a recent inspection or which inspections are to be
counted.
Recommendations

The Commission should:
ƒ

Work with the Legislative Budget Board to improve the performance
measure calculation and methodology.

ƒ

Ensure that data is uniformly and consistently calculated and reported
within that definition and methodology.

Management’s Responses

Revisions of the definition, data source and methodology have been submitted
to the LBB and Governor’s Office for approval, to clarify and expand the
information, in order to ensure understanding and proper calculation of this
measure. These clarifications are more specific in terms of which inspections,
and therefore which compliant jails, are to be counted. These steps will ensure
that performance measure data is uniformly and consistently calculated and
reported, and that the calculations and numbers reported are consistent with
the definitions and methodology.

Percentage of Jails with Management-related Deficiencies
This measure was inaccurate because auditors identified an 11.67 percent
variance between the recalculated performance measure and the
Results: Inaccurate
measure the Commission reported in ABEST. This measure is
Reported performance is
not within +/-5 percent
calculated by dividing the number of non-compliant jails with
of actual performance or
management deficiencies by the total number of operational jails.
there are more than two
errors in the sample
tested.

In addition, audit testing found that the number of jails with
management-related deficiencies was incorrect for 5.2 percent of the items
tested. For example, the Commission included a jail that was actually
compliant, and it included three counties whose deficiencies were not clearly
management-related. The Commission uses a checklist to determine what
types of deficiencies are management-related, but that checklist is subject to
interpretation and is not always adhered to.
The measure definition also does not contain enough information to be clearly
understood and easily recalculated. There are discrepancies within the
definition and many of the terms in the definition are not defined:
Attachment
An Audit Report on Performance Measures at the
Commission on Jail Standards
SAO Report No. 06-040
May 2006
Page 5

ƒ

There were inconsistencies among the measure name, definition, and
methodology. The measure definition includes “operational standards”
and “management deficiencies,” while the name and methodology include
only “management deficiencies.”

ƒ

It is not clear which jails should be included as “operational jails.”

Recommendation

The Commission should work with the Legislative Budget Board to improve
the performance measure calculation, methodology, and definition; ensure that
data is uniformly and consistently calculated and reported within that
definition and methodology; and document the types of deficiencies that are to
be included in the measure.
Management’s Responses

Revisions that clarify the definition, data source, and methodology of this
measure have been submitted to the LBB and Governor’s Office for approval.
These revisions reflect new procedures to ensure uniform and consistent
calculation and reporting of this information. Policies and procedures are
also being revised to incorporate the new definitions and will include specific
steps in recording, calculating and reporting of jails found to be noncompliant
due to deficiencies that may be remedied through adjustment(s) to internal jail
procedures by jail management staff.

Number of Annual Inspections Conducted
The Commission’s reported result for this measure was accurate. However,
this measure was certified with qualification because the
Results: Certified With
Qualification
Commission does not review calculations before they are
Reported performance is within +/- 5
entered and released into ABEST and does not have written
percent of actual performance, but
policies and procedures for entering data, calculating results,
controls over data collection and
reporting are not adequate to ensure
and reporting performance measure results into ABEST. To
continued accuracy.
improve accuracy, the Commission needs to implement the
recommendations on page 2.

Attachment
An Audit Report on Performance Measures at the
Commission on Jail Standards
SAO Report No. 06-040
May 2006
Page 6

Management’s Responses

Commission staff has implemented a documented review process for the
calculation and reporting of this measure. Revisions to the definition, data
limitations, data source, and methodology have been submitted to the LBB
and the Governor’s Office for approval. Policies and procedures revisions are
being developed consistent with the foregoing in order to ensure accurate,
uniform and consistent calculation and reporting of this data.

Specific Information Technology Controls Should Be Improved

General controls over the Commission’s network and databases appear
adequate to ensure that the data supporting the Commission’s reported
performance measures is accurate and reliable. Auditors did not test
application controls because the Commission uses only Microsoft products
(such as Access and Excel) and does not use other automated applications.
Auditors also identified weaknesses related to physical security and business
continuity. These weaknesses put the Commission at risk of network failure
or loss of data. Specifically:
ƒ

The Commission’s full network back-up tapes, which contain sensitive
accounting and human resources information, are stored at an unsecured
location.

ƒ

Only one individual has administrator rights to access the network. Title
1, Texas Administrative Code, Section 202.20, requires the continuity of
information resources in the event of a business disruption.

ƒ

The server is not located in a temperature-controlled room. In addition,
physical access to the server room is not adequately monitored.

ƒ

The Commission has an inspection database, but it does not use this
database to calculate performance measure results.

ƒ

The Commission does not test its disaster recovery plan at least annually
as required by the Title 1, Texas Administrative Code, Section 202.24 (5).

Attachment
An Audit Report on Performance Measures at the
Commission on Jail Standards
SAO Report No. 06-040
May 2006
Page 7

Recommendations

The Commission should:
ƒ

Store its full back-up tapes in a secure location.

ƒ

Give administrator access to an additional employee who can serve as a
backup for information technology support.

ƒ

Identify options to control the heat in the server room and control access
to that room.

ƒ

Expand the functionality of the inspection database so that it can be used
to calculate performance measure results.

ƒ

Test its disaster recovery plan at least annually to ensure it is current.

Management’s Responses

Weaknesses related to physical security and business continuity, identified by
the State Auditors, are being addressed by Commission staff in order to
reduce, if not eliminate, the risk of network failure or loss of data.
The full back-up tapes are being stored in a secure location.
Passwords for administrator access are kept in a locked cabinet in the Agency
suite. An employee selected by the Executive Director or the Deputy Director
will be provided with these codes should backup access by someone other
than the current administrator be required.
The temperature in the server room has been a concern for quite some time,
and the staff has investigated options for a system to control the heat. Due to
the Agency’s limited appropriation, however, such an expenditure is not a
viable option. Whenever current budget constraints are no longer an issue,
the staff will arrange to have a temperature control system installed.
The server room is kept locked, and the IT staff member has one key; another
key is kept in a locked key-box, which may be accessed by the Deputy Director
and two additional staff members.
Expanding the data and formulas available for the inspection database has
been a topic of discussion among Agency staff; the development of the
database into a useful tool for calculating performance measure results, as
well as for gathering and utilizing other information regarding the jails under
Commission authority, are but one of the long-term goals identified by the
Attachment
An Audit Report on Performance Measures at the
Commission on Jail Standards
SAO Report No. 06-040
May 2006
Page 8

staff. The Agency is currently, however, unable to budget the cost of bringing
in temporary or even permanent staff in order to develop such a program.
Again, when the current budget restrictions are eased, this will be an item that
will certainly be revisited.
Commission staff is developing policies and procedures, to include annual
review of the disaster recovery plan to ensure these plans are currently viable.
Staff is also working to coordinate with DIR to ensure a current disaster
recovery plan for information technology.

Attachment
An Audit Report on Performance Measures at the
Commission on Jail Standards
SAO Report No. 06-040
May 2006
Page 9

Objectives, Scope, and Methodology
Objectives
The audit objectives were to determine whether the Commission (1) is
accurately reporting its performance measures to ABEST and (2) has adequate
control systems in place over the collection and reporting of its performance
measures.
Scope
The audit scope covered key performance measure results reported by the
Commission for fiscal year 2005 and the first quarter of fiscal year 2006.
Auditors also reviewed controls over the submission of data used in reporting
performance measures and traced performance measure information to the
original source documents when possible.
Methodology
The audit methodology included selecting measures to audit, auditing results
for accuracy and adherence to the measure definitions, evaluating controls
over the performance measure certification process and related information
systems, and testing samples of source documentation when possible.
Project Information
Auditors conducted fieldwork from March through April 2006. This audit
was conducted in compliance with generally accepted government auditing
standards.
The following staff of the State Auditor’s Office performed the audit:
ƒ

Jennifer Wiederhold (Project Manager)

ƒ

Lauren L. Godfrey

ƒ

Jennifer Lehman, MBA

ƒ

Kelly Vogler

ƒ

Mary Ann Wise, CPA

ƒ

J. Scott Killingsworth, CIA, CGFM (Quality Control Reviewer)

ƒ

Charles P. Dunlap, Jr., CPA (Quality Control Reviewer)

ƒ

Verma Elliott, MBA, CGAP (Audit Manager)
Attachment
An Audit Report on Performance Measures at the
Commission on Jail Standards
SAO Report No. 06-040
May 2006
Page 10