A new congressionally mandated report from the National Research Council has found that changes EPA has proposed and implemented into its Integrated Risk Information System (IRIS) process are "substantial improvements"  over its old system.

IRIS is used to assess the hazards posed by environmental contaminants but has been rather sporadically followed in the past. A court had to order the EPA to stop blaming fracking for problems without doing a study and the agency has had to defend itself against charges of 'secret' science in determining greenhouse gas policy.

While acknowledging the progress made to date, the report offers further guidance and recommendations to improve the overall scientific and technical validity of the program.

In 2011, a Research Council committee reviewed EPA's IRIS assessment for formaldehyde and found deficiencies both in the particular assessment as well as more broadly in EPA's general assessment methods. EPA was directed by Congress to implement the report's general recommendations on the IRIS process, and the Research Council was then tasked with assessing the changes made and recommending additional modifications.

In response to the recommendations in the formaldehyde report, EPA developed a new document structure, added a standard preamble to all assessments that describes the IRIS process, drafted a handbook that provides a more detailed description of this process and its underlying principles, formed chemical assessment support teams to oversee the process and ensure consistency, and increased opportunities for stakeholder input.

The improved documentation better organizes and streamlines IRIS assessments, and the preamble is useful, but the new report found that it doesn't fulfill the need for a description in each assessment that indicates how the general principles are applied. The report recommends that the handbook be peer-reviewed, that IRIS assessments clearly identify the members of all teams involved, and that outside experts be engaged when needed.

EPA's progress indicates that the agency is incorporating principles of systematic review, a method for synthesizing scientific evidence that focuses on a specific question and uses predefined methods to identify, select, assess, and summarize the findings of the full body of literature relevant to the question. The committee agreed that using this approach would strengthen the IRIS process, and used published systematic-review standards as a reference point to evaluate the changes that EPA has made.

A major challenge for EPA in problem formulation is determining which adverse health outcomes should be evaluated in a specific IRIS assessment, the report says. The committee suggested a process that includes conducting a broad literature search, creating a table that organizes the lines of evidence and possible health outcomes, and then examining the table to decide which outcomes warrant a review. Once a systematic-review question is specified, a protocol should be developed that makes the methods of the review transparent.

The protocol should include descriptions of the literature search strategy for each question and explicitly state the criteria for including or excluding studies, and should be reviewed by an information specialist. Such a standardized search strategy is essential for evidence identification, the report says. EPA has also implemented a standardized approach to evaluating evidence, and while it correctly identifies attributes that can be used to judge study quality, it does not describe how it plans to assess the risk of bias in the identified studies. The committee did not recommend any specific approach to assessing bias, but said that the approach chosen by EPA and its results should be fully described and reported in the IRIS assessment.

After systematic review is completed, an IRIS assessment must combine all the individual lines of evidence to come to a judgment about whether a chemical is hazardous to human health, a process the committee referred to as "evidence integration." EPA currently relies on a guided expert judgment process for evidence integration. EPA should either make this process more transparent if it chooses to continue using this approach or adopt a structured process for evidence integration. EPA should also develop templates for narrative justifications of the evidence integration process and its conclusions, and work to ensure that its guidelines for integration are uniform for cancer and noncancer outcomes, the report says.

In addition to hazard identification, IRIS assessments derive toxicity values for given substances when data allow. The committee was encouraged by the improvements that EPA has made in this area, particularly the shift away from choosing one study as the "best" study for deriving a toxicity value and toward deriving and graphically presenting multiple candidate values. EPA, however, should develop formal methods for combining results of multiple studies and selecting the final IRIS values with an emphasis on achieving a transparent and replicable process.

To ensure that the IRIS program provides the best assessments possible, the committee recommended that EPA develop a plan for strategically updating its methodology, systematically addressing any identified inefficiencies, and continually evaluating whether the IRIS teams have the appropriate expertise and training.