Reproductive Toxicology 90 (2019) 102–108

Contents lists available at ScienceDirect

Reproductive Toxicology
journal homepage: www.elsevier.com/locate/reprotox

Python BMDS: A Python interface library and web application for the T
canonical EPA dose-response modeling software
⁎
Ly Ly Phama,1, Sean Watforda,2, Katie Paul Friedmana, Jessica Wignallb, Andrew J. Shapiroc,
a
National Center for Computational Toxicology, Office of Research and Development, United States Environmental Protection Agency, Research Triangle Park, NC USA
b
ICF, Burlington, Vermont, USA
c
National Toxicology Program at NIEHS, Research Triangle Park, NC, USA

A R T I C LE I N FO A B S T R A C T

Keywords: Several primary sources of publicly available, quantitative dose-response data from traditional toxicology study
Benchmark dose modeling designs relevant to predictive toxicology applications are now available, including the redeveloped U.S.
Software Environmental Protection Agency’s Toxicity Reference Database (ToxRefDB v2.0), the Health Assessment
Dose-Response Workspace Collaborative (HAWC), and the National Toxicology Program’s Chemical Program’s Chemical Effects
Systematic review
in Biological Systems (CEBS). These resources provide effect level information but modeling these data to a curve
In vivo toxicology
may be more informative for predictive toxicology applications. Benchmark Dose Software (BMDS) has been
recognized broadly and used for regulatory applications at multiple agencies. However, the current BMDS
software was not amenable to modeling large datasets. Herein we describe development and use of a Python
package that implements a wrapper around BMDS, a software that requires manual input in the dose-response
modeling process (i.e., best-fitting model-selection, reporting, and dose-dropping). In the Python BMDS, users
can select the BMDS version, customize model recommendation logic, and export summaries of the resultant
BMDS output. Further, using the Python interface, a web-based application programming interface (API) has
been developed for easy integration into other software systems, pipelines, or databases. Software utility was
demonstrated via modeling nearly 28,000 datasets in ToxRefDB v2.0, re-creation of an existing, published large-
scale analysis, and demonstration of usage in software such as CEBS and HAWC. Python BMDS enables rapid-
batch processing of dose-response datasets using a modeling software with broad acceptance in the toxicology
community, thereby providing an important tool for leveraging the publicly available quantitative toxicology
data in a reproducible manner.

Availability repeat dose animal toxicity study is selected for each study to inform
these assessments. The selected doses are called points-of-departure
Implemented in Python (available on the Python packaging index, (PODs). Traditionally, PODs have been dose levels corresponding to the
https://pypi.org/project/bmds/). Source code is open-source and MIT no or lowest observable adverse effect levels (NOAELs, LOAELs) from
licensed: Shapiro 2018a [1], Shapiro 2018b [2]; this code is available traditional animal studies. However, this traditional approach is limited
for anyone to use. A demo web application where one can model data is because of dependencies on the dose selection, dose spacing, and
available at bmds-python.com; this URL will be available for at least 2 sample size used in the original study. It is widely accepted that fitting a
years post-publication of this article. model to the dose-response data for observed effects can address some
of the limitations of the traditional approach to POD selection [3–5].
1. Introduction Indeed, many modeling methods exist, including Benchmark Dose
Software (BMDS) [6], PROAST [7], and the ToxCast Data Analysis
Dose-response analyses of hazard information are used to inform (tcpl) and curve-fitting procedure [8]; however, one software program
selection of a toxicologically-relevant dose for use in chemical risk as- has predominately been used in toxicology: BMDS. It is important to
sessments. Typically, a dose corresponding to observed effects from a implement a high-throughput functionality for BMDS because the

⁎
Corresponding Author. Current affiliation: Infinia ML, Durham, NC USA.
E-mail address: [email protected] (A.J. Shapiro).
1
ORISE Postdoctoral Research Participant.
2
ORAU, contractor to U.S. Environmental Protection Agency through the National Student Services Contract.

https://doi.org/10.1016/j.reprotox.2019.07.013
Received 27 February 2019; Received in revised form 11 July 2019; Accepted 12 July 2019
Available online 12 August 2019
0890-6238/ © 2019 Elsevier Inc. All rights reserved.
L.L. Pham, et al. Reproductive Toxicology 90 (2019) 102–108

model fitting procedure included in BMDS is familiar to toxicologists, programming language interfaces well with other programming lan-
risk assessors, and other stakeholders interested in defining a tox- guages such as C, Fortran, R, and Java; therefore; is an ideal language
icologically-relevant dose based on a benchmark response (BMR), i.e., a for a scientific software interface, as these languages are frequently
user-defined value signifying a toxicologically significant change based used in scientific applications.
on baseline recommendations or specific knowledge of the quantitative
relationship between the specific effect and an adverse outcome. In 2.2. Benchmark dose modeling features
many cases, the BMR has been based on the type of effect observed
(e.g., cancer versus non-cancer endpoints). Suggested defaults for BMR The interface library, i.e., the Python package which acts as an entry
selection are available in EPA BMD guidance [6]. point to the underlying BMDS models, was engineered with three key
Adaptation of BMDS for automated analysis of large amounts of design considerations. First, it was designed to be adaptable to analysis-
legacy data follows in the steps of previous work, briefly overviewed specific modeling decisions, e.g. model recommendation and/or high
here. BMDS has been adapted for analysis of transcriptomic information dose-dropping, and flexible so that users of the software could imple-
with BMDExpress [9,10], and now BMDExpress2.0 [11–13] which ment modeling per the needs of their application. However, common
provides a command line interface to fit continuous, transcriptomic use case patterns were also implemented, such as automated model
concentration response data in parallel. The BMDS Wizard [14] was recommendation, as previously implemented in the BMDS Wizard
designed to capture all required input and outputs for a BMDS session, software [14]. Second, performance (i.e., speed) was also important
defined as application of multiple models to a single dataset, in an Excel when designing the interface library; by default modeling is highly
file. These software libraries or user-interfaces for BMDS are heavily concurrent with multiprocessor support. Finally, it was important to
customized for the specific use-cases for which they were designed; for ensure that the EPA BMDS modeling library, instead of a surrogate, is
instance, the BMDS graphical user interface and the BMDS Wizard are used for all complex numerical evaluations of the dose-response data.
both useful in cases with limited numbers of datasets to be modeled, This includes using the compiled code provided by EPA with no mod-
and BMDExpress constrains the input data to continuous data, where ifications. Since the BMD software has previously been applied in
there are thousands of response (such as probes on a microarray) re- myriad research and regulatory applications, we did not want to change
lated to a single dose profile. any features which have already gone through extensive peer-review. A
The work presented herein addresses a new use case: the need for comparison of four applications of BMDS are shown in Table 1.
BMD modeling of continuous and dichotomous data, integrated with The EPA BMDS package and associated EPA BMD tools implement
other software and database systems used in computational toxicology. dose-response modeling for many types of toxicological data, including
A new web application and software interface for BMDS was built using continuous, dichotomous, nested-dichotomous, categorical (via
Python, which can be used for efficient modeling of large volumes of Categorical Regression Analysis, or CatReg [21]), repeated response,
data. Unlike the BMDS Wizard where only one dataset can be modeled etc. The Python BMDS interface library was implemented to include
per Excel file, or BMD Express, which is designed specifically for con- three standard widely-used data types, continuous, dichotomous, and
tinuous data from toxicogenomic studies, the Python BMDS interface cancer-dichotomous; however, the Python BMDS could be extended in
library and web application can enable automation of the full suite of the future to include new releases of BMDS (version 3.0, www.epa.gov/
models available from the canonical EPA BMDS. This can facilitate the bmds), other data types, or even non-BMDS approaches, such as
use of BMDS on large numbers of datasets, such as those now available Bayesian benchmark dose estimation [22]. The Python BMDS workflow
in the US EPA’s Toxicity Reference Database (ToxRefDB v2.0) [15], the is depicted in Fig. 1, starting with the (A) data input, working through
National Toxicology Program’s Chemical Effects in Biological Systems the (B) Python BMDS processing, to (C) outputs from the software.
(CEBS) [16], or the Health Assessment Workspace Collaborative A primary feature of using Python BMDS is the ability to run data
(HAWC), an open-source content management system for human health modeling sessions concurrently. As in all versions of EPA BMDS, a
assessments [17,18]. The application described herein enables easier single dataset can be added and modeled with multiple dose-response
software integration of BMD modeling, which may promote use of BMD modeling curves in a session. However, there are limitations to the
modeling in toxicology research more broadly, as opposed to the batch processing capabilities of BMDS (i.e., running multiple sessions).
NOAEL/LOAEL methods which are still most frequently used. In contrast, the Python BMDS interface library, allows for dozens to
thousands of sessions to be created and executed in parallel, thus dra-
2. Implementation and features matically increasing the possible use-cases for the software. Higher-
throughput analyses that could take advantage of this include modeling
2.1. Programming languages of datasets from large legacy databases, systematic reviews, or possibly
even high-throughput screening modeling applications.
Concurrent with each release of BMDS, EPA releases both source An additional strength of the Python BMDS interface is its ease in
code, developer documentation, and compiled versions of the software integrating benchmark dose modeling into other applications by pro-
(www.epa.gov/bmds; [6]). As discussed in Section 5.7, Future Work, viding a simple API (Fig. 1B). For continuous datasets, the BMDS soft-
the source code for the very recently released BMDS 3.0 was not ware includes two different variance models, i.e., constant or modeled
available at the time of this publication. For previous versions of BMDS variance [14] when using BMDS Wizard, models may need to be exe-
(2.7 and below), the published source-code includes two components: a cuted twice in order to use the correct variance model. The Python
user-interface for BMD modeling which is written in the C# program- BMDS interface library implements the same test used in BMDS, and the
ming language, and the individual dose-response models, where each appropriate variance model can be automatically selected, reducing
executable represents a single model or a model family (e.g., Hill, model outputs and runtime by half for each dataset. Evaluating mul-
multistage, exponential, logistic), written in C. Extensive developer tiple BMRs is also simple; sessions can be cloned and evaluated with
documentation is provided for input files to the dose-response models. different BMRs. Model recommendation logic largely follows the logic
Due to its common use, performance characteristics with regards to used in the BMDS Wizard as described by Wignall and colleagues
interoperability with C, and extensive existing software libraries in data (2014) [14] (Fig. 1), with some minor updates noted in Section 5.6
science and application development, we selected the Python software (details on default recommendation logic are in the online doc-
language for development of the BMDS interface library. Python is one umentation) per more recent BMDS guidance. The logic used for model
of the most widely used programming languages and has seen sub- recommendation in Python BMDS is fully customizable, i.e., new rules
stantial growth in the last few years in the data-science domain, as well could be created, or existing rules removed. The software could also be
as general scripting and web-applications [19,20]. Further, the Python extended to use Bayesian model-averaging techniques such as those

103
L.L. Pham, et al. Reproductive Toxicology 90 (2019) 102–108

implemented in BMDS 3.0. Finally, the software includes an option to

Python BMDS interface and web application
drop the highest dose-group when a model cannot be recommended, as

application (using native Windows BMDS

Single/multiple dose-response dataset(s)

Automated; customizable decision logic

long as there are a sufficient number of dose-groups for modeling. This

Windows with library; any with web

Word, Excel, JSON, Text, PNG, SVG
is a common approach to dose-response modeling where higher dose
Web-based REST API or Python

groups may behave non-monotonically due to mortality or acute toxi-

city rather than the endpoint of interest. By default, this option is dis-

Continuous, dichotomous
abled; however, when applied, the BMDS output preserves the original
dataset and notes the number of doses dropped to enable a model with a
Manual or Scripted
programmatic API

recommendation, rather than forcing a user to track the re-analysis

manually.

Automatic

binaries)
Python

2.3. Outputs
High

Reports can be generated in standard human readable formats

Automated; predefined decision

Windows, Mac Linux, (BMDS

(Word, Excel), or machine-readable formats (JSON, Python pickle se-
rialization, etc.) (Fig. 1C). Dose-response plots can be generated and
Toxicogenomic dataset

TSV, JSON, PNG, SVG

customized using the popular matplotlib [23] Python plotting library

GUI or command line

models recompiled)
Manual or Scripted

and exported in standard formats (PNG, SVG, PDF, TIFF, etc.). The
BMDExpress2.0

software was written in an object-oriented design pattern, e.g., classes

Continuous

for common idioms in dose-response modeling include descriptors such

Scripted

as Session, SessionBatch, Dataset, and Model, which can be composed

High

logic
Java

by software developers or data analysts to build pipelines or integrate

into other software.
Continuous, dichotomous

Automated; customizable

2.4. Versioning, compatibility, and web application

Single dose-response

Manual or Scripted

The Python BMDS interface library uses the most recent version of
BMDS Wizard

decision logic
Excel + VBA

Excel, Word

EPA BMDS in the version 2 series by default (2.7), however prior ver-
Windows

sions of EPA BMDS are also available in the Python BMDS interface
Manual
dataset
Excel

library. Generally, BMD analyses and assessments are done with a

Low

single version of EPA BMDS; the interface library was designed to allow
forwards and backwards compatibility with BMDS as new versions are
Automated; customizable decision logic or

created, so that prior analyses can be reproduced. The Python BMDS

Single/multiple dose-response dataset(s)

interface library currently includes multiple versions of BMDS, in-

Continuous, Dichotomous, Nested-

cluding versions 2.3.1, 2.4, 2.6, and 2.7. The software will be updated
Excel, Word, native excel plots
dichotomous, Categorical, etc.

to use the newly released version 3 series of BMDS [24] when developer
Excel + programmatic API

documentation becomes available, as described below in Section 5.7.

The US EPA releases both the source-code for dose-response models
U.S. EPA BMDS 3.0

used in BMDS, as well as compiled-executables of these models for

C + Excel + VBA

model-averaging

Microsoft Windows operating system. However, for BMDS 2.7 and

below there can be minor differences in versions of the model execu-
Automatic
Moderate

Windows

tables based on various settings during compiling, which are operating-

Manual

system specific (e.g., BMDS models compiled on Microsoft Windows

may give slightly different answers that BMDS on Apple MacOS, or
Linux web-servers). Therefore, while it is possible to compile and use
the EPA models distributed prior to BMDS 3.0 on different operating
Continuous, Dichotomous, Nested-
Standalone GUI + command line

No automated recommendation

systems, in order to make results fully reproducible with previous

dichotomous, Categorical, etc.
Single dose-response dataset

versions, it is important to rely exclusively on using the pre-compiled

versions of the software which are distributed by EPA. Thus, for BMDS
2.7 and below, reproducibility is only ensured by the use of pre-com-
U.S. EPA BMDS 2.7
Comparison of software libraries using EPA BMDS.

piled models executed in a Windows operating system environment.

Text files, EMF

Restricting use of the Python BMDS interface library on a Windows

C +. NET

Windows

operating system may pose problems in developing pipelines or in-

Manual

Manual

tegrating into other software systems, as Linux environments are fre-

Low

quently used for web-applications and high-performance computing.

Therefore, in addition to developing a Python BMDS interface library, a
BMDS web application was also developed that allows for the Python
Continuous data variance model

Supported Operating Systems

BMDS interface library to be used on any computer that can access the
web. Users provide a dataset or list of datasets and BMDS modeling
Model recommendation

options. The web application converts the request to “jobs,” which are
placed on a queue and are modeled as computing resources are avail-
Output formats

able. BMDS modeling results can be downloaded and are deleted after a
Data structure
User interface

selection
Language(s)

Throughput

period of time (defaults to one week). While users can interact directly
Data input

Data types
Feature

with the web application using a web browser, the web application was
Table 1

primarily designed for interfacing with other software platforms or

pipelines where dose-response modeling was needed. The web

104
L.L. Pham, et al. Reproductive Toxicology 90 (2019) 102–108

Fig. 1. Python BMDS workflow. The workflow, including necessary inputs (A), the logic used in the Python BMDS for fitting (B), and the stored output (C) are
illustrated.

application was implemented using the widely-used Python Django web 22 h of time. Note also that all pre- and post-processing using the Py-
framework [25], and is deployed on a Windows server so that the thon BMDS interface is captured in code, whereas the prior study re-
compiled executable files provided by EPA are used, ensuring results quired substantial manual editing and aggregation of tables, which
are fully reproducible in any environment. A Representational State limits reproducibility. Detailed results for the analysis, including input
Transfer (REST) application programming interface (API) was im- and output files, as well as the source code for the analysis, are avail-
plemented for submitting/receiving data from the web application, a de able in Supplemental Material 1.
facto standard in transferring data/requests on the internet [26]. A Overall, high similarity was found in results between the prior
demonstration website, at bmds-python.com, is available as described analysis and the reanalysis using this software library (Fig. 2). In terms
above. of the number and type of datasets fit, 87.7% of the 896 datasets re-
analyzed demonstrated a concordant result (either recommended or did
2.5. Large scale batch processing using the Python BMDS interface not recommend a best-fitting model), while in 12.3% of the cases a
different result was achieved (83.2% matching continuous, 86.9% di-
Wignall et al. (2014) [14] first demonstrated the ability to use chotomous, and 90.2% cancer dichotomous). In many cases the Python
BMDS with a set of standardized BMD settings on a database of tox- BMDS interface recommended a model when the BMDS wizard analysis
icological data. The original analysis was conducted using the BMDS was unable to (24, 37, and 38 datasets, for continuous, dichotomous,
Wizard, an Excel-based wrapper of the EPA BMDS modeling software and cancer dichotomous, respectively). The opposite case was also
version 2.3.1, where each dataset along with model settings and results observed (3, 6, and 2 datasets, respectively).
are saved in a single Excel file. The modeling was conducted in a semi- For cases where both Python BMDS and BMDS Wizard calculated a
automated fashion; each Excel file was automatically created but BMD value, there is a high degree of concordance between re-
manually executed. Doses may have been dropped as needed and saved commended BMD values; a linear regression of the Python BMDS and
in separate Excel files, a manual process, and results were aggregated BMDS Wizard BMD values produced an R2 0.941 (Fig. 2B). Over 75% of
manually for subsequent analysis. datasets with BMD values in both cases were within 0.01 relative dose-
We referenced the analysis from Wignall et al. (2014) [14] to test units equivalence; in absolute terms 84% of datasets were within 1
concordance of the Python BMDS interface library with a previous dose-unit. Discordances between analyses could be a byproduct of
analysis. After preparing data, modeling setup and execution was BMDS version (2.3.1 vs 2.7), workflow implementation and manual
conducted in 18.5 min, or approximately 1.0 s per dataset (using a 2012 processing in the prior analysis, additional model-options in Python
laptop with 4-core hyperthreaded Intel i7-3610 processors and 8GB BMDS, or mapping old summary files to the new analysis. There are
RAM). In total, over 7000 individual dose-response models were exe- also slight differences in workflow implementation; for example, pre-
cuted, a throughput of approximately 390 models per minute. The total viously in BMDS Wizard, only a single multistage model was executed
time taken for model executions was not captured in Wignall et al. 2014 with a polynomial degree equal to the number of dose groups (n) minus
[14], but assuming 1.5 min per dataset for execution and human re- 1; in the Python BMDS interface, multiple multistage models are exe-
view, selection of model recommendation, manual dose-dropping, and cuted from order 1 to n-1 by default, and therefore additional models
data transfer into summary spreadsheets, it would take approximately are available for possible model-fit. As previously stated, all

105
L.L. Pham, et al. Reproductive Toxicology 90 (2019) 102–108

Fig. 2. Comparison of BMD estimates from BMDS Wizard and Python BMDS Interface. Re-analysis using the Python BMDS library was compared for 896
datasets from Wignall et. al. 2014 [14]. (A) 87.7% of the reanalyzed datasets demonstrated a concordant result, with either both libraries fitting (dark blue) or failing
(light blue). In some cases (for 24, 37, and 38 datasets, for continuous, dichotomous, and cancer dichotomous, respectively), the Python BMDS interface re-
commended a model when the BMDS wizard analysis did not (dark orange). Conversely, for 3, 6, and 2 datasets, Python BMDS failed and the BMDS Wizard fit (light
orange). (B) When BMD estimates were recommended in both analyses, a linear regression demonstrates high concordance, with an R-squared of 0.941 (For
interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article).

recommendation logic can be modified, but the more stringent logic can 2.6. Application integrations using the Python BMDS web application
result in differences in model recommendations. A fully automated
workflow as implemented with the Python BMDS interface improves on Here, we discuss additional examples early adoption of the publicly
limitations with manual processing, including facilitating and main- available Python BMDS code and/or web application by other data
taining ease of reproducibility, standardizing the approach, and lim- aggregation projects. The Python BMDS web application has been
iting user-error in configuration or manipulation of input and output adopted for use in the NTP CEBS database [16]. BMDS was used to
data. It should be noted that even repeating an older analysis conducted model data from the National Toxicology Program’s chronic carcino-
in a different platform (BMDS Wizard) demonstrates the utility of the genicity studies stored in CEBS, with BMD and BMDL values calculated
Python BMDS interface; it allows flexibility in modeling, model re- for significant histopathological findings using a default BMR. In the
commendation logic, and analysis of results. future, CEBS is planning to calculate BMD values for all treatment-re-
The application of Python BMDS in ToxRefDB v2.0 is currently the lated findings in NTP public data (from toxicology and carcinogenicity
largest automated implementation of BMDS (1170 cancer datasets; studies, five-day dose-response transcriptomics studies, genetic tox-
17,318 non-cancer dichotomous datasets; and 9268 continuous data- icology assessments, etc.). Currently, internal NTP users can access data
sets), thus serving as a proof-of-concept for large batch-processing of in CEBS and run the Python BMDS service for these data; however, it is
dose-response data using BMDS. Using Python BMDS to analyze data in anticipated that these features in CEBS will become available to the
ToxRefDB v2.0, models were built for effect data from five different public in the future. Another project utilizing Python BMDS is the
types of study designs, including chronic, subchronic, subacute, multi- HAWC project (https://hawcproject.org/), a free, open-source content-
generation reproduction, and developmental study designs, and over management designed for systematic reviews of human health assess-
300 categories of observed effects, e.g. the tissue or effect of interest ments [17]. The software has been used for a number of reports by
[15]. Based on BMDS guidance [27], default BMRs were selected for state, federal, and international agencies, including a National
each dose response-type, i.e. cancer (5% and 10%), non-cancer di- Academy of Sciences report [28]. Users can create an assessment where
chotomous (5% and 10%), continuous body/organ weight (10% re- bioassay, epidemiological, and in vitro data can be extracted and in-
lative deviation), and continuous non-body/organ weight (1 standard tegrated in summary visualizations. Bioassay data can be modeled using
deviation), though these could be expanded in the future. Approxi- BMDS in HAWC, which is implemented using the Python BMDS library.
mately one-third of the available quantitative dose-response datasets in BMD analyses can be created by the user with full control of endpoint
ToxRefDB v2.0 passed quality control and minimum dataset require- selection, BMR(s), and model settings. An example analysis is shown
ments imposed (e.g., N, three dose groups and a control, and an esti- here: https://hawcproject.org/ani/endpoint/33/.
mate of variance such as standard deviation) for batch processing by
Python BMDS. Of the 27,756 total datasets, 73–86.7% had at least one
2.7. Future work
recommended model that yielded a BMD and BMDL value for the
chemical-effect pair, depending if 5 or 10% BMR is used, respectively.
The Python BMDS interface library is dependent on the official EPA
However, specific use cases may necessitate the need for expert jud-
BMDS software releases. Thus, when new versions of the EPA software
gement and evaluation before use. The datasets and models are avail-
are released, Python BMDS will also be updated. A new version of
able in the ToxRefDB v2.0 release ([15]; GitHub link https://github.
BMDS (version 3.0) was released in October of 2018. This release was a
com/USEPA/CompTox-ToxRefDB).
major update to the software, including completely rewriting the ex-
isting API for how models are executed; instead of executing models via
a command-line interface, they are executed via function calls using

106
L.L. Pham, et al. Reproductive Toxicology 90 (2019) 102–108

shared-object libraries. At the time of writing, the software was released authors and do not necessarily represent the views or policies of the US
to the public, but source code and developer documentation were not EPA.
yet available. However, when these become available, the Python
BMDS interface library will be updated to include new BMDS versions. Declaration of Competing Interest
The authors believe that in addition to improvements in modeling ap-
proaches (e.g., Bayesian model averaging for dichotomous datasets, Andrew Shapiro has changed positions and is currently working at
etc.), there will likely be faster performance of modeling. Further, Infinia ML; however, work conducted at the current position is in-
should shared-object libraries be released in operating systems other dependent and unrelated to work in this publication. All other authors
than Windows (i.e., 64-bit Linux, Mac), these could also be used to declare they have no actual or potential competing financial interests.
allow for platform agnostic model execution, thus relaxing the
Windows-only execution restriction of the current implementation. As Acknowledgements
guidance and user experience will BMDS version 3.0, these can shape
the evolution of Python BMDS implementation. The authors would like to thank Jeff Gift, Reeder Sams, R. Woodrow
Setzer, and Imran Shah (U.S. EPA) and Scott Auerbach and Jennifer
3. Conclusions Fostel (U.S. National Toxicology Program) for their insightful com-
ments on previous versions of this manuscript.
From a regulatory and scientific perspective, EPA BMDS has pro-
vided dramatic advances in the ability to build a unified approach to Appendix A. Supplementary data
dose-response modeling, which is a foundational principal in human
health risk assessment and toxicology research applications. EPA has Supplementary material related to this article can be found, in the
publicly provided source code and extensive documentation on how to online version, at doi:https://doi.org/10.1016/j.reprotox.2019.07.013.
use the software both at a user-interface level, as well as for software
developers. However, from a software integration perspective, BMDS References
was difficult to integrate into other software systems due to operating
system dependencies and input/output formatting constraints. Herein [1] A. Shapiro, BMDS python source code, Zenodo (2018) 2008, https://doi.org/10.
we present a new Python interface for BMDS and a web-application for 5281/zenodo.1419333.
[2] A. Shapiro, BMDS python server source code, Zenodo (2018) 2018, https://doi.org/
executing BMD modeling. We have demonstrated the utility of this 10.5281/zenodo.1419828.
system in implementing the gold-standard BMDS modeling methods on [3] W. Slob, Benchmark dose and the three Rs. Part I. Getting more information from
medium to large dose-response datasets in an automated fashion. The the same number of animals, Crit. Rev. Toxicol. 44 (7) (2014) 557–567.
[4] W. Slob, Benchmark dose and the three Rs. Part II. Consequences for study design
robustness and flexibility of the Python BMDS interface to be in- and animal use, Crit. Rev. Toxicol. 44 (7) (2014) 568–580.
corporated into automated pipeline have been demonstrated in mul- [5] W. Slob, R.W. Setzer, Shape and steepness of toxicological dose-response relation-
tiple use cases, such as during systematic review where data are ex- ships of continuous endpoints, Crit. Rev. Toxicol. 44 (3) (2014) 270–297.
[6] J.A. Davis, J.S. Gift, Q.J. Zhao, Introduction to benchmark dose methods and U.S.
tracted from literature or using large datasets in existing databases. The EPA’s benchmark dose software (BMDS) version 2.1.1, Toxicol. Appl. Pharmacol.
Python BMDS interface was designed in such a way that the software 254 (2) (2011) 181–191.
can be used in the future for further integrations with other data sys- [7] A. Hardy, D. Benford, T. Halldorsson, M.J. Jeger, K.H. Knutsen, S. More,
A. Mortensen, H. Naegeli, H. Noteborn, C. Ockleford, A. Ricci, G. Rychen, V. Silano,
tems or data pipelines, using modern software interfaces and develop-
R. Solecki, D. Turck, M. Aerts, L. Bodin, A. Davis, L. Edler, U. Gundert‐Remy,
ment techniques. Finally, it is open-source and publicly available with a S. Sand, W. Slob, B. Bottex, J.C. Abrahantes, D.C. Marques, G. Kass, J.R. Schlatter,
permissive software license, which enables re-use and future colla- Update: use of the benchmark dose approach in risk assessment, EFSA J. 15 (1)
borations. Python BMDS provides a tool that advances reproducibility (2017) e04658.
[8] D.L. Filer, P. Kothiya, R.W. Setzer, R.S. Judson, M.T. Martin, Tcpl: the ToxCast
in modeling of large toxicology databases, substantially extending the pipeline for high-throughput screening data, Bioinformatics 33 (4) (2017) 618–620.
functionality of BMDS for predictive toxicology applications. [9] L. Yang, B.C. Allen, R.S. Thomas, BMDExpress: a software tool for the benchmark
dose analyses of genomic data, BMC Genomics 8 (1) (2007) 387.
[10] S. Auerbach, The NTP Proposed Approach to Genomic Dose-Response Modeling,
Author’s contributions Scott Auerbach, Ph.D. NIH/NIEHS/DNTP, 2017, https://ntp.niehs.nih.gov/about/
org/sep/ntpexpertpanel/meetings/2017/videos/index.html.
LLP, KPF, and SW wrote the manuscript, and used the software in [11] Sciome, BMDExpress 2 software, National Toxicology Program, (2018) https://
www.sciome.com/bmdexpress/.
the ToxRefDB v2.0. JW provided data and analysis for comparison to [12] S. Auerbach, BMDExpress-2 Wiki, (2018) https://github.com/auerbachs/
manual BMDS Wizard modeling and provided feedback on the re-ana- BMDExpress-2/wiki/.
lysis. AJS designed and implemented the Python BMDS interface and [13] J.R. Phillips, D.L. Svoboda, A. Tandon, S. Patel, A. Sedykh, D. Mav, B. Kuo,
C.L. Yauk, L. Yang, R.S. Thomas, J.S. Gift, J.A. Davis, L. Olysyzk, B.A. Merrick,
web application, portions of the manuscript, and integrated the BMDS
R.S. Paules, F. Parham, T. Saddler, R.R. Shah, S.S. Auerbach, BMDExpress 2: en-
interface in HAWC. All authors read and reviewed the final manuscript. hanced transcriptomic dose-response analysis workflow, Bioinformatics (2018)
bty878-bty878.
[14] J.A. Wignall, A.J. Shapiro, F.A. Wright, T.J. Woodruff, W.A. Chiu, K.Z. Guyton,
Funding
I. Rusyn, Standardizing benchmark dose calculations to improve science-based
decisions in human health assessments, Environ. Health Perspect. 122 (5) (2014)
L. L. P. was supported by appointment to the Research Participation 499–505.
Program of the U.S. Environmental Protection Agency, Office of [15] S. Watford, L.L. Pham, J.A. Wignall, R. Shin, M.T. Martin, K. Paul Friedman,
ToxRefDB version 2.0: Improved utility for predictive and retrospective toxicology
Research and Development, administered by the Oak Ridge Institute for analyses, Reproductive Toxicology (submitted).
Science and Education through an interagency agreement between the [16] I.A. Lea, H. Gong, A. Paleja, A. Rashid, J. Fostel, CEBS: a comprehensive annotated
U.S. Department of Energy and the U.S. EPA. database of toxicological data, Nucleic Acids Res. 45 (D1) (2017) D964–d971.
[17] HAWC, HAWC: Health Assessment Workplace Collaborative, (2013) (Accessed 23
August 2018), https://hawcproject.org/.
Disclaimer [18] A.J. Shapiro, S. Antoni, K.Z. Guyton, R.M. Lunn, D. Loomis, I. Rusyn, G.D. Jahnke,
P.J. Schwingl, S.S. Mehta, J. Addington, N. Guha, Software tools to facilitate sys-
tematic review used for Cancer hazard identification, Environ. Health Perspect. 126
The United States Environmental Protection Agency (U.S. EPA) (10) (2018) 104501.
through its Office of Research and Development has subjected this ar- [19] D. Robinson, The Incredible Growth of Python, (2017) https://stackoverflow.blog/
ticle to Agency administrative review and approved it for publication. 2017/09/06/incredible-growth-python/.
[20] S. Cass, The 2017 Top Programming Languages Python jumps to No. 1, and Swift
Mention of trade names or commercial products does not constitute Enters the Top Ten, 2017, (2018) https://spectrum.ieee.org/computing/software/
endorsement for use. The views expressed in this article are those of the

107
L.L. Pham, et al. Reproductive Toxicology 90 (2019) 102–108

the-2017-top-programming-languages. djangoproject.com/.
[21] U. EPA, Categorical Regression (CatReg) Software Version 3.1.0.7, (2017) https:// [26] R.T. Fielding, R.N. Taylor, Principled design of the modern Web architecture, ACM
www.epa.gov/bmds/catreg. Trans. Internet Technol. 2 (2) (2002) 115–150.
[22] K. Shao, A.J. Shapiro, A web-based system for bayesian benchmark dose estimation, [27] Benchmark Dose Technical Guidance, (2012), pp. 1–99 https://www.epa.gov/risk/
Environ. Health Perspect. 126 (1) (2018) 017002. benchmark-dose-technical-guidance.
[23] J.D. Hunter, Matplotlib: a 2D graphics environment, Comput. Sci. Eng. 9 (3) (2007) [28] E. National Academies of Sciences, and Medicine, Application of Systematic Review
90–95. Methods in an Overall Strategy for Evaluating Low-dose Toxicity From Endocrine
[24] USEPA, Benchmark Dose Software (BMDS) - Version 3.0, (2018). Active Chemicals, National Academies Press (US), Washington, D.C, 2017.
[25] jangoSoftwareFoundation, Django, (2018) [Computer Software] https://www.

108