Unlocking the potential of simulation studies in network

psychometrics: A tutorial

Ria H. A. Hoekstra1 , Sacha Epskamp2 , Adam T. K. Finnemann1,3 , & Denny Borsboom1

1 Department of Psychology, University of Amsterdam, Amsterdam, The Netherlands
2 Department of Psychology, National University of Singapore, Singapore, Singapore
3 Center for Urban Mental Health, University of Amsterdam, Amsterdam, The Netherlands

PREPRINT WITHOUT PEER-REVIEW

Corresponding author: Ria Hoekstra, University of Amsterdam, Department of Psychology. Nieuwe

Achtergracht 129B, 1018 WS Amsterdam, The Netherlands. E-Mail should be sent to: [email protected].
 Unlocking the potential of simulation studies in network psychometrics: A
tutorial

Abstract
Simulation studies are a powerful tool within the field of network psychometrics, providing
insight into sample size determination and best practices for achieving accurate results.
Regrettably, the advantages simulation studies provide have thus far been predominately
harnessed by psychological researchers with a statistical or methodological background.
While user-friendly methods for simulating psychological networks have been developed,
there remains a lack of comprehensive guidelines for their effective utilization. In response,
this tutorial paper serves as a guide on how to set up simulation studies for three widely
used network modeling techniques: the Ising model, Gaussian Graphical Model (GGM),
and the Graphical Vector Autoregressive (GVAR) model. We explain the rationale behind
simulation studies and present a step-by-step roadmap to initiate one’s own simulation
studies. This roadmap covers all essential aspects using a practical example to guide the
reader. In addition, we provide an overview of open-source software packages available for
simulating network models in the statistical programming environment R, and demonstrate
their practical application. In doing so, we aim to enable more researchers to explore and
resolve statistical questions within the context of network psychometrics.

Keywords: psychological networks, simulation study, tutorial, Ising model, GGM, GVAR

1 In the last decade, network psychometrics has gained substantial attention in psychological

2 sciences (Borsboom et al., 2021; Isvoranu et al., 2022; Robinaugh et al., 2020). In this field

3 of research, psychological constructs are conceptualized as a complex interplay of components.

4 Psychological networks consist of nodes representing observed variables, connected by edges

5 representing the statistical relationships between these variables. The type of statistical rela-

6 tionship depends on the estimation technique used (e.g., partial correlations when estimating a

7 Gaussian Graphical Model (GGM), regression coefficients when estimating a temporal network

8 structure using Graphical Vector Autoregressive (GVAR) models).

9 While how to estimation network models has been described in numerous tutorial papers

10 (e.g., Costantini et al., 2019; Dalege et al., 2017; Epskamp, Borsboom, & Fried, 2018; Huth

11 et al., 2023), descriptions of how to simulate network models is fragmented and a clear overview

12 is missing (Dalege et al., 2017; Epskamp, Borsboom, & Fried, 2018; Finnemann et al., 2021). As

13 a result, the advantages simulation studies provide have thus far been predominately harnessed

14 by psychological researchers with a statistical or methodological background (e.g., Epskamp &

15 Fried, 2018; Isvoranu & Epskamp, 2023; Mansueto et al., 2023). Especially within the field of

1
16 network analysis, more researchers could benefit from the skills to perform simulation studies

17 as, for example, a way to determine the sample size or time points that are necessary to pick

18 up the desired effects is missing. Simulation tools can help researchers determine how much

19 data is required in order to estimate an accurate network model. In addition, several papers

20 have questioned the stability of estimated network models across repeated samples (Forbes

21 et al., 2017, 2019, 2021; Fried et al., 2018). Simulation studies can be used to determine which

22 factors contribute to the replication of the network structure and to determine the expected

23 reproducibility of a given network structure (Borsboom et al., 2021; Epskamp, Borsboom, &

24 Fried, 2018).

25 The aim of this tutorial is to broaden the accessibility of simulation tools for researchers.

26 We begin by outlining the rationale behind simulation studies in network psychometrics. Using

27 a practical example, we guide readers through specifying a network model, simulating data,

28 estimating a network, and evaluating the simulation results for the three most often used

29 modeling frameworks: Ising model, GGM, and GVAR model. We include an overview of the

30 open-source software to set up simulation studies in R, and common used evaluation metrics in

31 network psychometrics. R code are made available for the reader to replicate our simulations

32 and to perform their own simulations.

33

34 Setting up a simulation study

35 Simulation studies can be thought of as computer-based experiments. One key strength of

36 simulation studies is the ability to understand the behavior of statistical methods under various

37 conditions because the“true” underlying model is known for the data generating process (Morris

38 et al., 2019). Therewith, simulation studies are a powerful tool to answer methodological as well

39 as theoretical questions and provide a framework to answer specific questions relevant to one’s

40 research.

41 Similar to empirical studies, simulation studies center around a research question. The

42 research question in a simulation study typically pertains to statistics and involves questions

43 such as understanding the statistical power of a particular method, how a method performs

44 under ideal and non-ideal conditions, or the comparative strengths and limitations of competing

45 methodologies. While some statistical questions may be resolved through mathematical analysis,

46 simulation studies are often better suited for answering more complex questions (Hallgren, 2013).

2
47 With respect to network psychometrics, simulation studies can help us answer questions

48 about the performance of particular estimation techniques under various conditions, i.e., how

49 well does the technique recover the true underlying network structure, (e.g., Epskamp, Waldorp,

50 et al., 2018; Epskamp et al., 2017; Haslbeck, 2022; Van Borkulo et al., 2014, 2022; Williams

51 et al., 2019), which network estimation technique works best under which conditions (Isvoranu

52 & Epskamp, 2023), and if idiographic network model estimation is feasible in clinical practice

53 (Mansueto et al., 2023).

54 In general, simulation studies we follow a consistent process to provide an answer to our

55 research question. Given the research question, we identify some evaluation metrics (e.g., the

56 sensitivity or specificity of an estimation technique), the conditions to be varied (e.g., sample

57 size, network structure, number of time series), and specify a data generating model (e.g., a

58 GGM or GVAR model) . This data generating model serves as the true underlying model. With

59 the true model in hand we simulate data, and perform statistical analyses on this simulated

60 data, such as estimating a network model. Since we know the true model for the simulated data,

61 we can compare the estimated parameters to the true parameters and compute our evaluation

62 metrics. This process is repeated multiple times to produce distributions for the evaluation

63 metrics.

64 Thus, approximately a simulation study can be broken down into eight consecutive steps:

65 1. Define the aim or research question

66 2. Define evaluation metric and conditions to be varied based on the aim or research

67 question

68 3. Specify the true model (e.g., data generating mechanism) based on the aim or

69 research question

70 4. Generate data based on the true model

71 5. Perform statistical analysis on the simulated data and retain evaluation metrics

72 6. Repeat steps 3-4 numerous of times to generate a distribution for the evaluation

73 metrics

74 7. Analyze the distribution of the evaluation metrics

75 8. Evaluate the simulation results in light of the research question

3
 76 In this tutorial, we will provide the tools for readers to walk through all steps involved in

77 conducting a simulation study. As a running example we set up a small simulation study to

78 answer the question: “To what extend can we accurately estimate a GGM for reasonable sample

79 sizes?”, see Tutorial box 1. In order to answer this research question, we varied the sample size

80 condition (i.e., n ∈ 100, 250, 500), and obtained the sensitivity, specificity, and correlation as our

81 evaluation metrics. See Appendix A for a description of the most commonly used evaluation

82 metrics in network psychometrics, and table 1 for all evaluation metrics implemented in the

83 evalNet() function.

84 In Tutorial box 1, we use the function parSim() from the parSim package (Epskamp, 2023c)

85 as an easy way to provide the skeleton for our simulation study. With the specification of just a

86 few arguments this function circumvents the process of combining multiple for loops in order to

87 loop over all conditions in our simulation study. parSim() is a parallel simulator that takes a set

88 of conditions and an R expression as arguments and returns a data frame with the simulation

89 results. We can specify any number of conditions ourselves. Here we specified our sample

90 size condition (i.e., sampleSize = c(100, 250, 500)). Within the R expression we can write

91 whatever code we would like to use within our simulation study using the specified conditions

92 as objects. This way, the R expression is performed on each of the specified conditions. With

93 the reps argument we control the number of repetitions for each of the conditions. In Tutorial

94 box 1 we set the reps argument to 10 by means of a simplification, but ideally this number is

95 much higher such as 100, or 500 repetitions per condition. Within the R expression we load the

96 required R packages, specify the true data generating network, simulate data based upon this

97 network, estimate a network based on the simulated data to ultimately compare the estimated

98 to the true network using our evaluation metrics. The code in Tutorial box 1 can be used as a

99 starting point and adapted as one likes using the techniques described in the subsequent sections

100 of this tutorial.

101 The remainder of this tutorial paper is structured in accordance with the specification of

102 the R expression within the parSim() function in Tutorial box 1. First, we describe how to

103 specify the true data generating network. Second, how to simulate data based upon this true

104 data generating network. And third, how to estimate a network model based on the simulated

105 data. We do so for each of the three most commonly used network models: the Ising model,

106 the GGM, and the GVAR model. Final, we exemplify how to evaluate the simulation results.

107 Annotated R code to follow along with the tutorial can be found on OSF: https://osf.io/ygjn7/.

4
108
Table 1

Evaluation metrics

Evaluation metric Short description

Network accuracy
Sensitivity proportion of true edges
Signed sensitivity proportion of true same sign edges
Top 50% sensitivity proportion of top 50% strongest true edges
Top 25% sensitivity proportion of top 25% strongest true edges
Top 10% sensitivity proportion of top 10% strongest true edges
Specificity proportion of true absent edges
Precision proportion of true present edges
Top 50% precision proportion of top 50% strongest present edges
Top 25% precision proportion of top 25% strongest present edges
Top 10% precision proportion of top 10% strongest present edges
Jaccard index proportion of shared edges relative to the total number of edges
Edge accuracy
Edge weight correlation correlation between edge weights
Absolute edge weight correlation correlation between absolute edge weights
Nonzero edge weight correlation correlation between non-zero edge weights
Bias for all edges average absolute deviation between all edge weights
Bias for included edges average absolute deviation between estimated edge weights
Centrality accuracy
Pearson correlation centrality measures pearson correlation between centrality measures
Kendall correlation centrality measures rank order correlation between centrality measures
Top 5 strongest centrality indices comparison between top 5 centrality indices
Top 3 strongest centrality indices comparison between top 3 centrality indices
Top 1 strongest centrality indices comparison between top 1 centrality indices

This list of evaluation metrics is based on metrics suggested by Borsboom et al. (2017), Forbes et al. (2017), and

109
Isvoranu and Epskamp (2023)

5
 # Set random number generator:
set.seed(5)

# Load libraries:
library("parSim")
library("devtools")

# Run the simulation:

results <- parSim(
# Specify any number of conditions:
sampleSize = c(100, 250, 500),

# Number of repetitions:
reps = 10, # preferably more repetitions

# Number of cores:
nCores = 1,

# R expression:
expression = {
# Load R packages
library("bootnet")
source("https://raw.githubusercontent.com/RiaHoekstra/simulation_functions/
main/evalNet.R")
source("https://raw.githubusercontent.com/RiaHoekstra/simulation_functions/
main/GGMsim.R")

# Specify the true data generating network

trueGGM <- genGGM(Nvar = 6)

# Simulate data based on the network structure

simData <- GGMsim(n = sampleSize, PCC = trueGGM)

# Estimate network
estGGM <- estimateNetwork(simData, default = "EBICglasso")

# Obtain evaluation metrics

metrics <- evalNet(trueGGM, estGGM$graph, metric = c("sensitivity",
"specificity",
"correlation"))

# Return results
data.frame(metrics)
}
)

# Inspect results:
View(results)

Tutorial box 1

Exemplifying the use of the parSim() function from the parSim package. The parSim() function is a
parallel simulator that takes a set of conditions and an R expression as arguments and returns a data
frame with the simulation results. Within the R expression we can write whatever code we would like to
use within our simulation study using the specified conditions as objects. This way, the R expression is
performed on each of the specified conditions.

6
110 Specifying a network model

111 In the previous section we described the basic’s of setting up a simulation study. As we know

112 now, a simulation study starts with specifying a research question or aim that leads to the

113 specification of the true, data generating model. But how to specify such a true data generating

114 model within the network modeling framework?

115 To specify the true data generating network, two routes can be taken: (i) one can use an

116 estimated network structure based on empirical data, or (ii) one can specify a synthetic network

117 structure with easy to use functions in R.

118 Empirical network

119 Taking a (previously) estimated network model based on empirical data as the true data

120 generating network has several benefits. One clear benefit is that an estimated network can be

121 chosen that represents the scenario of interest as closely as possible, such that the variables in

122 the empirical network can be a direct representation of the variables of interest and the sample

123 is a representative sample of the sample of interest.

124 However, limitations need to be taken into account when using an empirically estimated

125 network model as the true data generating network. As the network is based on empirical

126 estimation, the resulting network is as reliable as the data sampling procedure. For example,

127 if the empirical network is based on a relatively large sample, the network is less likely to

128 contain false positive edges, while if the empirical network is based on a relatively small sample,

129 chances increase that the estimated network contains false-positive edges and thus forfeits

130 its representative appeal. Therefore, it is recommend when one wishes to use an empirical

131 estimated network as the true underlying data generating model, one chooses a network that

132 is (i) representative of the situation of interest and (ii) a network that is estimated from a

133 sufficiently large sample.

134 Regarding the latter, an interesting trade-off should be noted. In general, if the true network

135 structure is a dense network that contains many and weak edges, then the higher the sample

136 size, the more edges can be detected when estimating the network. As a consequence, if we

137 would estimate such a densely connected network based on a n = 500 sample compared to a

138 n = 1000 sample, the latter data set would allow us to pick up more effects in the estimated

139 network. However, this leads to a network structure that should be recoverable for data sets

7
140 with a n > 1000 sample, whereas simulating under the network structure we obtained from our

141 n = 500 sample, leads to a network structure that should be recoverable for data sets with a

142 n > 500 sample. In a sense this is problematic, as the higher sample size we use to determine

143 the true underlying network structure, the higher sample size is required in simulation studies

144 to recover that structure.

145 Synthetic network

146 Giving some of the difficulties with taking an empirical network model as the true data generating

147 network, it might be preferred to specify the true data generating network oneself. We shall

148 refer to the later as a synthetic network. Specifying a synthetic network as the data generating

149 network has some advantages, as it becomes easier to specify a structure with reasonable edge

150 strengths (e.g., no edges with values ambiguously close to zero), and you know works well in

151 simulation studies (e.g., results in a positive semi-definite covariance matrix). Constructing a

152 synthetic network can be done based on prior knowledge, theoretical reasoning, or a specific

153 hypothesis. An additional benefit of specifying a synthetic network structure as the true data

154 generating network, is that it leaves more room to tweak the network. For example the number

155 of variables in the network can easily be altered1 , the number of edges, and the strength of the

156 edge weights.

157 As with choosing an empirical network as the true data generating network model, so does

158 specifying a synthetic network model presents us with challenges. We discuss these challenges for

159 the Ising model, GGM, and GVAR model in the following sections. For each of these statistical

160 models, a short description of the model will be given. In addition, we present an overview of

161 open-source software packages and exemplify their accompanying functions to specify a synthetic

162 network for each of these models. For an overview of software packages to specify a synthetic

163 network structure in R and their relevant functions, see Table 2.

1
Note here, when altering the number of nodes in the network it is important to keep in mind that this has an
effect on the edge weights. This means one cannot directly compare a network with 5 nodes to a network with 10
nodes and say the changes found in estimation are a direct result of the change in the number of nodes. Changes
the number of nodes in the network also always have an influence on the edge weights, where the more nodes
included the overall effect between edges becomes smaller.

8
164
Table 2

Software packages in R relevant to simulations within network psychometrics. Functions are related to
their packages using the ‘::’ notation from R.

Model package::function Description

set.seed() random number generator

ggplot2::ggplot() visualization
parSim::parSim() easy simulations using multiple cores
bootnet::replicationSimulator() easy simulations to determine reproducibility
bootnet::netSimulator() easy simulations to determine sample size
evalNet() computes evaluation metrics for networks
bootnet::estimateNetwork() estimates network structure
Ising model

genIsing() generates an Ising model

IsingSampler::IsingSampler() generates data from an Ising model
IsingFit::LinTransform() transforms between {0, 1} and {−1, 1} encoding
IsingFit::IsingFit() estimates Ising model using eLASSO
GGM

bootnet::genGGM() generates a GGM

ggmSim() generates data from a GGM
GVAR

genGVAR() generates a GVAR model

graphicalVAR::graphicalVARsim() generates data from a GVAR model
graphicalVAR::graphicalVAR() estimates GVAR model using EBICglasso

To access a function, first install the package with install.package("example") and then load the package with
library("example"). The functions evalNet(), genIsing(), ggmSim(), and genVar() can be downloaded from

165
github via: https:// github.com/ RiaHoekstra/ simulation functions

166 Ising model

167 The Ising model is a probability model in which variables can have either one of two states. In

168 psychology, the Ising model is frequently used in research on psychopathology to model the

169 absence or presence of symptoms, and in attitude research to study pro versus contra attitudes

170 (Dalege et al., 2016; Van Borkulo et al., 2014). The Ising model can be used as a theoretical

171 model to represent and study large systems of interacting elements, and as a statistical model to

172 estimate network structures from binary data. Finnemann et al. (2021) discusses both uses of

9
173 the Ising model in more detail. In this paper, we only focus on its statistical use.

174 Formally, the Ising model can be denoted as a probability distribution on a random vector y

175 in which each element can take one of two states:

 
1 X X
P (yy ) = exp  τi yi + ωij yi yj  (1)
Z
i <i,j>

176 in which τi indicates a threshold parameter (intercept) that is related to the predisposition of

177 a person to respond with one of the two answer categories (either {0, 1} or {−1, 1} depending

178 on the encoding of the data, see Haslbeck et al. (2021)), ωij is the network edge parameter

179 between node i and node j, and Z is a normalizing constant2 . Typically, the Ising model also

180 includes a parameter denoting the inverse temperature β. However, from a statistical perspective

181 β typically is not identified, and to this end in the expression above we identified the model by

182 specifying β = 1.

183 Three aspects are important when specifying a synthetic Ising model: (i) the encoding

184 used (either {0, 1} or {−1, 1}), (ii) the threshold parameters values (τ ), and (iii) the network

185 edge parameters (ω). We need to specify these three aspect in such a way that the synthetic

186 Ising model leads to generated data with enough entropy. Entropy denotes the average level of

187 information in the data, usually stored in “bits.” In essence, entropy answers the question:“how

188 many yes-no questions are on average needed to describe the response pattern of a person in the

189 data set?”. Suppose we model a 3-node network, meaning we have 3 dichotomous responses

190 per person. If there is no overlap between the three responses, we would need full information

191 on each of the three responses to describe the outcome. This means that in this case entropy

192 = 3. However, suppose we have strong information on the responses: everyone in the sample

193 always responds with the same answer to each of the questions. For example: the questions are

194 “are you born on Mars?”, “are you over 250 years old?”, and “can you jump higher than 100

195 meters?”. We would hope that all subjects in our sample would always respond with “no” to

196 each of these questions. In this case, entropy = 0, as we don’t need to ask any yes-no question,

197 we already know the answer to each of the responses without information needed from a subject.

198 Ideally, we would like entropy to be somewhere between these extremes, meaning that we want

199 specify the synthetic Ising model in such a way that the data generated from this network is
2
The normalizing constant Z, also termed the partition function, is important in parameter estimation, but not in
data generation as it is not a function of the data and therefore can be ignored when specifying a synthetic Ising
model.

10
200 not completely random, while at the same time still having suitable and realistic variation per

201 variable.

202 To investigate what combination of endocing (either {0, 1} or {−1, 1}), threshold parameters

203 (τ ∈ {−1, 0, 1}), and network edge parameters (ω ∈ {0, 0.25, 0.5, 1, 2, 3, 4}), would ensure data

204 containing enough entropy, we performed a small simulation study. A description of the

205 simulation study and the simulation code is available in Appendix B. Results show that the

206 entropy goes down as a function of the edge weight, meaning that the stronger connected the

207 network, the fewer response patters are generated. Entropy is at a maximum when all threshold

208 parameters (τ ) and edge weight parameters (ω) are set to zero, but this is not an interesting

209 network to generate data under as this would just imply completely random (coin-flip) data.

210 Entropy reduces to 0 in most settings, except in symmetric encoded data with all threshold

211 parameters set to zero (τ = 0). In general, when generating symmetric encoded data with all

212 threshold parameters set to 0 edge weight parameters around 0.25 and 0.5 tend to work well.

213 Therefore we recommend when specifying a synthetic Ising model, to use symmetric encoding,

214 set threshold parameters to 0 and sample edge weights around 0.25 and 0.5.

215 Obliging to the above specified conditions, we can specify a synthetic Ising model in R with

216 the genIsing() function. This functions first generates a skeleton for the network structure, i.e.,

217 a structure that indicates whether there is an edge present between two nodes. Next, it samples

218 parameter values for the edge weights from a random normal distribution with a specified mean

219 and standard deviation. The genIsing() functions has 6 main arguments: nNode determines

220 the number of nodes in the network, propPositive determines the proportion of positive edges,

221 rewire determines the probability of re-wiring random edge weights, inclusion determines the

222 probability of edge inclusion, mean determines the mean edge weight value, sd determines the

223 standard deviation for the edge weight values.

224 As input the genIsing() function simply requires the specification of the number of nodes

225 in nNode. In that case a chain graph in which each node is connected to its consecutive node

226 in the form of a chain is generated, i.e., if we consider an undirected network with 3 nodes

227 there is an edge between the nodes 1–2, 2–3, and 3–1, with a mean edge weight value of 0.25

228 and a standard deviation of 0.1. To move away from this chain-graph structure the rewire

229 argument can be specified with a certain probability. This means randomly selected edges in

230 the chain-graph will be rewired. Another way to move away from a chain graph is to specify

231 the inclusion argument. The inclusion argument can be used to generate a network with

11
232 a certain probability of edge inclusion. In Tutorial Box 2 we exemplify generating a synthetic

233 Ising model using the genIsing() function.

To specify an Ising model in R we can use the genIsing() function. This function
simply requires the specification of the number of nodes:

# Load libraries:
library("devtools")
source("https://raw.githubusercontent.com/RiaHoekstra/simulation_functions/
main/genIsing.R")
library("qgraph")

# Generate an Ising model with 6 nodes:

trueIsing <- genIsing(nNode = 6)

# Determine thresholds for each node:

thresholds = 0
thresholds <- rep(thresholds, 6)

# Plot Ising model:

qgraph(trueIsing)

This results in the generating of a chain graph. To move away from this chain structure,
a rewiring probability can be specified to randomly rewire a proportion of the edges.
Another option is to determine the proportion of edges with a specified probability:

# Generate an Ising model with 6 nodes and rewiring probability of 50%

trueIsing <- genIsing(6, rewire = 0.5)

# Plot Ising model:

qgraph(trueIsing)

# Generate an Ising model with 6 nodes and inclusion probability of 50%

trueIsing <- genIsing(6, inclusion = 0.5)

# Determine thresholds for each node:

thresholds = 0
thresholds <- rep(thresholds, 6)

# Plot Ising model:

qgraph(trueIsing)

Tutorial box 2

Exemplifying the use of the genIsing() function to generate an Ising model. The genIsing() function
is a developmental function that can be downloaded from github:
https:// github.com/ RiaHoekstra/ simulation functions/ blob/ main/ genIsing.R.

12
234 GGM

235 The GGM is a model in which the relations between variables are modeled through partial

236 correlation coefficients. In psychology, this model is often used when we have measured every

237 subject only once and we are interested in the relations between continuous normally distributed

238 variables (e.g., Beard et al., 2016; Isvoranu et al., 2017; Vervaet et al., 2021). When we have

239 continuous normally distributed data, we assume that all subjects share the same distribution,

240 that is:

y ∼ N (µ
µ, Σ ) (2)

241 The covariance matrix Σ is our matrix of interest, as the inverse of this matrix, K , the precision

242 matrix, is also known as the GGM (Epskamp, Waldorp, et al., 2018):

K = Σ −1 (3)

243 The elements in this precision matrix can be standardized to obtain the partial correlation

244 coefficients as follows:

kij
ωij = − p (4)
kii kjj

245 where i ̸= j. Thus a zero entry in the precision matrix K implies conditional independence

246 between the connected variables, while a non-zero entry indicates a conditional dependence.

247 This partial correlation is then plotted as a network

248 The covariance matrix Σ , precision matrix K , and partial correlation matrix ω need to be

249 positive semi-definite. As these matrices have a one-to-one relation, if one matrix is positive

250 semi-definite, all of them are. To determine if the precision matrix K is positive semi-definite we

251 can check for positive definiteness, which is a stronger condition than positive semi-definiteness.

252 A matrix is positive definite if all of its eigenvalues are positive, and if a matrix is positive

253 definite, it is also positive semi-definite. We can use the eigen() function in R to compute the

254 eigenvalues of the precision matrix. If all eigenvalues are strictly positive, then the matrix is

255 positive definite and, by extension, positive semi-definite. Alternatively, we can verify if the

256 inverse of the precision matrix K leads to a valid covariance matrix Σ . A covariance matrix

13
257 is valid when all of its eigenvalues are non-negative. Again, we can use the eigen() function

258 in R to compute the eigenvalues of the covariance matrix Σ , to see if all of its eigenvalues are

259 non-negative. If at least one eigenvalue is negative, it implies that the matrix is not positive

260 semi-definite. See Tutorial box 3 for an example how to check if the covariance matrix as

261 obtained from the genGGM() function is positive semi-definite.

262 An easy way to specify a GGM in R obliging to the above mentioned condition can be

263 done using the genGGM() function from the bootnet package (Epskamp & Fried, 2023). This

264 function generates a GGM as described in Yin and Li (2011) using the Watts and Strogatz

265 (1998) algorithm for generating a graph structure. This creates the skeleton for our network.

266 Generating a GGM with this function can be as easy as specifying the number of nodes one

267 wants to include in the network. This results in the generation of a chain or ring graph. In

268 a chain graph each node is connected to its consecutive node in the form of a chain, i.e., if

269 we consider an undirected network with 3 nodes there is an edge between the nodes 1–2, 2–3,

270 and 3–1. To move away from this chain structure, a rewiring probability p can be specified to

271 randomly rewire a proportion of the edges. Depending on the proportion of edges rewired, we

272 obtain a random graph. Between the chain graph and random graph, we find a small-world

273 network. To ensure small-world properties of the network, we can specify the graph argument

274 as "smallworld" to create a small world network. Clusters in the network can be created by

275 specifying "clusters" in the graph argument. The probability to estimate a positive edge can

276 be specified through the propPositive argument, and the range of the parameter values can be

277 specified through the argument parRange. For exemplary use of the genGGM function to specify

278 a synthetic network structure, see Tutorial box 3.

279 GVAR

280 The GVAR model is a graphical depiction of the VAR model and is used when we want to model

281 time series data of one individual (e.g., Beck & Jackson, 2020; Epskamp, van Borkulo, et al.,

282 2018). The lag-1 GVAR model can be depicted as:

y t = c + B y t−1 + ε t (5)

ε t ∼ N (0, Σ ) (6)

14
 To specify a GGM in R we can use the genGGM() function from the bootnet package.
This function simply requires the specification of the number of nodes:

# Load libraries:
library("bootnet")
library("qgraph")

# Generate a GGM with 6 nodes:

trueGGM <- genGGM(Nvar = 6)

# Plot GGM:
qgraph(trueGGM)

This results in the generating of a chain graph. To move away from this chain structure,
a rewiring probability p can be specified to randomly rewire a proportion of the edges.
The probability to estimate a positive edge can be specified through the propPositive
argument:

# Generate a GGM with 6 nodes:

trueGGM <- genGGM(Nvar = 6,
p = 0.5,
propPositive = 0.8)

# Plot GGM:
qgraph(trueGGM)

To check if the covariance matrix is a positive semi definite matrix, we can check if all
eigenvalues are non-negative:

# Check if covariance matrix is positive semi definite:

all(eigen(diag(ncol(trueGGM)) - trueGGM)$values >= 0)

Tutorial box 3

Exemplifying the use of the genGGM() function from the bootnet package in R to specify a GGM.

15
283 where y t represents a vector of continuous responses by a given person on time-point t and

284 c is the intercept. The B matrix contains all lag-1 regression coefficients βij of variable j on

285 variable i encoding temporal effects, and its transpose is used to display a personalized directed

286 weighted network structure referred to as the temporal network. Moreover, ε t represents a

287 vector of normally distributed innovations with variance–covariance structure Σ . This innovation

288 variance-covariance matrix Σ matrix can further be modeled as a GGM (Epskamp, Waldorp,

289 et al., 2018). The inverse of the residual matrix K = Σ −1 is used to create a contemporaneous

290 network structure per individual. While the temporal network gives insight in how deviations

291 from the person-wise mean in one variable at a certain measurement occasion predict deviations

292 from the person-wise mean in the next measurement occasion, the contemporaneous network

293 gives insight how variables within the same measurement occasion co-occur after controlling for

294 temporal effects (Burger et al., 2022).

295 Thus, when specifying a graphical VAR model in R we need to specify two true data generating

296 networks: the contemporaneous network, and the temporal network. As the contemporaneous

297 network can be modelled as a GGM, we can simply use the genGGM() function in R as discussed

298 in the previous section to generate the contemporaneous network structure. We can specify a

299 temporal network by first creating a skeleton for the network structure. This means a structure

300 that indicates whether there is an edge present between two nodes and in which direction. Next,

301 we can sample the parameters, i.e., edge weights, from a random normal distribution with a

302 certain mean and standard deviation. This procedure has been implemented in the genGVAR()

303 function.

304 genGVAR() requires the specification of the number of nodes, and the mean (mean and standard

305 deviation sd for the sample distribution determining the edge weights of the temporal network.

306 As genGVAR() functions as a wrapper around genGGM() to generate the contemporaneous

307 network, all arguments for genGGM() can be directly specified in the genGVAR() function to

308 generate the contemporaneous network. In addition, the proportion of positive edge weights

309 in both the contemporaneous and temporal network can be specified using the arguments

310 propPositiveCon and propPositiveTemp respectively. As the edge weights for the temporal

311 network are sampled from a random normal distribution, a mean and standard deviation need

312 to be specified to determine the shape of this distribution, see Tutorial box 4 for the use of

313 genVar(). As output, genVar() returns the partial correlation network PCC that can be used to

314 plot the contemporaneous network and the directed partial correlation network PDC, that can be

16
315 used to plot the temporal network in addition to the B and K matrix.

To specify a synthetic GVAR model in R we can use the genGVAR() function. This
function requires the number of nodes as input and a mean and standard deviation for
the temporal edge weight parameters:

# Load in libraries:
library("devtools")
source("https://raw.githubusercontent.com/RiaHoekstra/simulation_functions/
main/genGVAR.R")
library("qgraph")

# Generate a GVAR with 6 nodes:

trueGVAR <- genGVAR(nNode = 6,
mean = 0.3,
sd = 0.1)

# Plot contemporaneous network:

qgraph(trueGVAR$PCC)

# Plot temporal network:

qgraph(trueGVAR$PDC)

In addition, a probability for positive edge weights can be specified for the temporal
and contemporaneous network:

# Generate a GVAR with 6 nodes:

trueGVAR <- genGVAR(nNode = 6,
propPositiveCon = 0.8,
propPositiveTemp = 0.5,
mean = 0.3,
sd = 0.1)

# Plot contemporaneous network:

qgraph(trueGVAR$PCC)

# Plot temporal network:

qgraph(trueGVAR$PDC)

Tutorial box 4

Exemplifying the use of the genGVAR() function to specify a GVAR model. The genGVAR() function is a
developmental function that can be downloaded from github:
https:// github.com/ RiaHoekstra/ simulation functions/ blob/ main/ genGVAR.R.

316 Simulating data based on a network model

317 The open-source software packages to specify psychological network structures discussed in

318 the previous section, often implement methods to generate data from these structures, for an

17
319 overview see Table 2. In this section we examplify the use of the IsingSampler(), GGMsim(),

320 GVARsim(), to simulate data from an Ising model, a GGM, and a GVAR model respectively.

321 Ising model

322 Simulating data from an Ising model can easily be done using the IsingSampler() function

323 from the IsingSampler package (Epskamp, 2023a). The IsingSampler() function employs three

324 methods to generate data from the Ising model. Data can be generated using a Metropolis-

325 Hastings algorithm (Chib & Greenberg, 1995), using the Metropolis-Hastings algorithm together

326 with Coupling from the Past (Murray, 2007), and simply computing the probability for every

327 possible state and sample directly from the distribution of states using these probabilities, see

328 Epskamp (2023a). The latter method quickly becomes intractable (roughly above 10 nodes).

329 In Tutorial Box 5, we exemplify the use of the IsingSampler() function using the default

330 Metropolis-Hastings algorithm to sample data from our specified synthetic Ising model in

331 Tutorial box 2. IsingSampler() has four main arguments: n specifies the number of generated

332 observations, graph specifies the underlying true network structure, thresholds controls the

333 node specific external fields, and responses controls the encoding. See Tutorial box 5, how to

334 generate data for 500 responses within the {−1, 1} domain.

We can generate data from an Ising model in R by using the Isingsampler() function
from the IsingSampler package.

# Load libraries:
library("IsingSampler")

# Generate data:
dataIsing <- IsingSampler(n = 500,
graph = trueIsing,
thresholds = thresholds,
responses = c(-1L, 1L))

Tutorial box 5

Exemplifying the use of the IsingSampler() function from the Isingsampler package to generate binary
data from an Ising model.

335 GGM

336 To simulate data based on a GGM we can use the GGMsim() function. GGMsim() provides an easy

337 way to simulate data from a multivariate normal distribution. As input, it requires the number

18
338 of individuals for one wishes to generate data in the n argument, and the partial correlation

339 matrix to generate random numbers from a multivariate normal distribution with a mean of

340 zero and covariance matrix Σ in the PCC argument. Optionally, a proportion of missing can be

341 specified using the missing argument to generate data with a certain percentage of missing, see

342 Tutorial Box 63 .

To generate multivariate normal data for n = 100 based on a GGM we can use the
GGMsim() function in R.

# Load libraries:
library("devtools")
source("https://raw.githubusercontent.com/RiaHoekstra/simulation_functions/
main/GGMsim.R")

# Generate data:
dataGGM <- GGMsim(n = 500,
PCC = trueGGM)

The function comes with he option to add missing values to the data by specifying a
proportion of missing values for the argument missing.
# Generate data with 30% missing values:
dataGGM <- GGMsim(n = 500,
PCC = trueGGM,
missing = 0.3)

Tutorial box 6

Exemplifying the use of the GGMsim() function from the xxx package in R to generate multivariate
normal data from a GGM.

343 GVAR

344 To generate data based on the GVAR model, we can use the graphicalVARsim() function from

345 the graphicalVAR package (Epskamp, 2023b). This function provides an easy way to simulate

346 stationary multivariate time series data based on a GVAR model using the rmvnorm() function4 .

347 In essence, graphicalVARsim() solves equation 5 for every time point using a specified B and K
3
GGMsim() in essence is a wrapper around the rmvnorm() function from the mvtnorm package, that generates
random values from a multivariate normal distribution.
4
Stationary time series is the core assumption of the GVAR model and refers to a time series in which no changes
over time are indicated by its defining characteristics, such as no change in means, variances, and network
parameters (Burger et al., 2022). Violations of this assumption arise if there are trends in the time series data,
such as linear or seasonal trends. It is important to note that deviations from the stationarity assumption are
not implausible in psychological time series. For example, we might observe shifts in the means of symptoms
after certain life events or following treatment. However, as stationarity is one of the model assumptions, we
simulate stationary time series data from a GVAR model.

19
348 matrix. Therefore we need to provide the B and K matrix as input in the graphicalVARsim()

349 using the beta and kappa arguments. Either, we have obtained these matrices by estimating a

350 GVAR model from data, or, we obtained these matrices by generating a GVAR model using the

351 genVar() function. In addition, we want to specify the number of time series data we want to

352 generate, using the nTime argument. Optionally, a vector µ , containing person means can be

353 specified in the mean argument, see tutorial box 7 for the use of graphicalVARsim().

To generate stationary time series data based on the GVAR model we can use the
graphicalVARsim() function from the graphicalVAR package in R. The function
requires the specification of the number of time series, the B and the K matrix:

# Load in libraries:
library("graphicalVAR")

# Generate time series data from GVAR model:

dataGVAR <- graphicalVARsim(nTime = 200,
beta = trueGVAR$beta,
kappa = trueGVAR$kappa)

Tutorial box 7

Exemplifying the use of the graphicalVARsim() function from the graphicalVAR package in R to
simulate multivariate normal data from a GVAR.

354 Estimating a network model

355 After defining the data generating network and simulating data from this network, depending

356 on the defined research question or aim of the simulation study, we perform some form of

357 statistical analysis on the data. Often this involves the estimation of a network model based on

358 the simulated data. What estimation technique to use however is entirely up to the researcher

359 and dependend upon the research question. This part of the simulation study leaves a lot of

360 room for the researcher as, for example, this is also the part where newly developed methods

361 can be applied to the data to ultimately determine their performance.

362 To estimate network models based on data many estimation techniques exist (e.g., Epskamp,

363 Borsboom, & Fried, 2018; Huth & Keetelaar, 2023; Isvoranu & Epskamp, 2023; Isvoranu et al.,

364 2022; Van Borkulo et al., 2014; Williams et al., 2019), and it goes beyond of the scope if this

365 paper to give an overview of all network estimation techniques to estimate an Ising model, a

366 GGM, and a GVAR within network psychometrics. Numerous tutorial papers describe these

20
367 estimation procedures in detail, (e.g., Costantini et al., 2019; Dalege et al., 2017; Epskamp,

368 Borsboom, & Fried, 2018; Finnemann et al., 2021; Huth & Keetelaar, 2023; Isvoranu et al.,

369 2022), and we refer the interested reader to those. In this section we briefly touch upon the

370 estimation of network models as implemented in the estimateNetwork() function from the

371 bootnet package as this function allows for the estimation of all three models discussed in this

372 paper.

373 Ising model

374 Here, we exemplify how to estimate an Ising model using the estimateNetwork() function

375 from the bootnet package (Epskamp & Fried, 2023). This function takes the data as input,

376 and the default argument needs to be specified as "IsingFit" in order to estimate an Ising

377 model5 . Most network estimation techniques fit the Ising model in a binary domain (i.e.,

378 {0, 1}), however, we simulated symmetric encoded data (i.e., {−1, 1}).6 Therefore we need to

379 transform the symmetric data to binary data before we can estimate the network model using

380 the estimateNetwork() function. We can do so by encoding our −1 variables as 0, see Tutorial

381 Box 8.

382 In simulation studies we often like to say something about how well a certain technique

383 managed to estimate the network parameters. As we now have a true network structure based

384 on the {−1, 1} domain, and an estimated network based on the {0, 1} domain, we cannot

385 simply compare the obtained network parameters from our estimated network to our true data

386 generating network. We first need to transform our estimated network parameters back to the

387 {−1, 1} domain. This can be done using the LinTransform() function from the IsingSampler

388 package (Epskamp, 2023a). The LinTransform() function requires the estimated network graph

389 and threshold values as input, the specification from which domain the network is estimated

390 (can be specified in from the argument), and to which domain we would like to transform the

391 network parameters (can be specified in the to argument), see Tutorial Box 8. This way we can

392 compare the estimated network parameters to the true network parameters.
5
The estimateNetwork() function in essence a wrapper around the IsingFit() function from the Isingfit package
(Epskamp & Fried, 2023; van Borkulo & Epskamp, 2023).
6
There are some packages that allow for the estimation of Ising models within the {−1, 1} domain such as
psychonetrics (Epskamp, 2023d).

21
 We can estimate an Ising model using the estimateNetwork() function from the
bootnet package. As many statistical models to estimate the Ising model require the
data to be binary, we first transform the data from {−1, 1} encoding to {0, 1} encoding:

# Load libraries:
library("bootnet")

# Transform encoded data to binary data:

dataIsing[dataIsing == -1] <- 0

# Estimate Ising model from simulated data in binary domain:

estIsing <- estimateNetwork(data = dataIsing,
default = "IsingFit")

Next we can transform the estimated network parameters to the {−1, 1} domain in
order to compare the estimated network parameters to the true network parameters:

# Load libraries:
library("IsingSampler")

# Transform estimated parameters from in binary domain to symmetric domain:

estIsing <- LinTransform(graph = estIsing$graph,
thresholds = estIsing$results$thresholds,
from = c(0L, 1L),
to = c(-1L, 1L))

Next we can plot the estimated Ising model:

# Load libraries:
library("qgraph")

# Plot estimated network:

qgraph(estIsing$graph)

Tutorial box 8

Exemplifying the use of the estimateNetwork() function from the bootnet package to estimate an Ising
model, and the LinTransform() function from the IsingSampler package to transform the network
parameters from the {0, 1} to {−1, 1} domain.

22
393 GGM

394 To estimate a GGM from continuous normal data several estimation techniques exist. We refer

395 the interested reader to Isvoranu and Epskamp (2023) for an in dept discussion of the most

396 common estimation techniques and recommendations for when to use which. Many of the

397 model estimation procedures for the GGM have been implemented in the estimateNetwork()

398 function from the bootnet package. Regularization, and especially LASSO regularization, has

399 been applied often within the psychological network estimation literature and therefore we shall

400 illustrate how to estimate a GGM using this method. This method has been implemented in

401 the estimateNetwork() function from the bootnet package (Epskamp & Fried, 2023) and can

402 be used by setting the default argument to "EBICglasso". The function simply requires the

403 data as input, as well as the specification of which estimation technique to use by setting the

404 default argument, see Tutorial Box 9.

To estimate a GGM we can use the estimateNetwork function from the bootnet
package in R. This function requires the data and the type of model selection procedure
as input:

# Load libraries:
library("bootnet")
library("qgraph")

# Estimate GGM from simulated data:

estGGM <- estimateNetwork(dataGGM,
default = "EBICglasso")

# Plot estimated GGM:

qgraph(estGGM$graph)

Tutorial box 9

Exemplifying the use of the estimateNetwork() function from the bootnet package to estimate a GGM.

405 GVAR

406 The most commonly used technique to estimate a GVAR model by finding a sparse solution

407 for both B and K is by estimating a regularized temporal and contemporaneous network using

408 LASSO regularization. This algorithm requires two tuning parameters, one for the temporal

409 coefficients and one for the contemporaneous coefficients, that are selected via Extended Bayesian

410 Information Criterion (EBIC) (Burger et al., 2022; Epskamp, Waldorp, et al., 2018). This proce-

23
411 dure of estimating individual network models has been implemented in the estimateNetwork()

412 function from the bootnet pacakge.7 The estimateNetwork() function simply requires the gener-

413 ated time series data as input, and the specification of the default argument to "graphicalVAR",

414 see Tutorial Box 10.

To estimate a GVAR model we can use the estimateNetwork() function from the
bootnet package. The functions requires the timeseries data of one individual as input
as well as the specification of which estimation technique to use.

Estimating a GVAR model using the estimateNetwork() function can be done as

follows:

# Load libraries:
library("bootnet")
library("qgraph")

# Estimate GVAR model from simulated data:

estGVAR <- estimateNetwork(data = dataGVAR,
default = "graphicalVAR")

We can plot the estimated GVAR as follows:

# Plot results:
layout(t(1:2))

# Plot temporal network:

qgraph(estGVAR$graph$temporal)

# Plot contemporaneous network:

qgraph(estGVAR$graph$contemporaneous)

Tutorial box 10

Exemplifying the use of the graphicalVAR() function from the graphicalVAR package and gvar()
function from the psychonetrics package in R to estimate a GVAR.

415 Evaluating results

416 In the previous sections, we learned how to determine the data generating network model, how

417 to simulate data based upon this true data generating network, and how to estimate a network

418 based on the simulated data for each of the three most popular modeling frameworks (i.e., Ising

419 model, GGM, GVAR). Let us return to our original example from Tutorial Box 1 to exemplify
7
In essence this function makes use of the graphicalVAR() function from the graphicalVAR package (Epskamp,
2023b)

24
420 the evaluation of the simulation results.

421 After running our simulation study, we want to analyze the results. How, and which results to

422 display is highly dependent upon the research question. In general, we would like to summarize

423 our results in the form of a table or plot showing the distribution of evaluation metrics obtained

424 for our various conditions. We can use the packages dplyr and tidyr for easy data manipulation,

425 and ggplot2 for plotting the results (Wickham, 2016) (Wickham & Grolemund, 2023; Wickham

426 et al., 2023; Wickham, 2016; Wickham et al., 2019).

427 Remember that we are interested in determining “To what extend can we accurately estimate

428 a GGM for reasonable sample sizes?”. And in order to answer this research question, we varied

429 the sample size condition (i.e., n ∈ 50, 100, 150), and obtained the sensitivity, specificity, and

430 correlation as our evaluation metrics. In order to plot our results with the ggplot() function,

431 some data modifications need to be made. First we want to format our results in such a way

432 that they are easy to use within ggplot(). Meaning, we need to go from a wide data format to

433 a long data format. We can use the gather() function from the tidyr package to do so, see

434 Tutorial box 11. In addition, we want to group the metrics according to their condition. With

435 the mutate() function from the dplyr package we can add an additional column to the results

436 to change the name of the metric variable, see Tutorial box 11.

437 After cleaning up the data and creating the right format, we can use ggplot() to display

438 the simulated results and start interpreting the results, see Tutorial box 11. As our sample size

439 is the independent variable we plot these conditions on the x-axis. The values of our outcome

440 metrics (i.e., correlation, sensitivity, and specificity) are our dependent variables, and are plotted

441 on the y-axis. See Figure 1 for the simulation results.

442 Our results indicate that the median correlation between the true and estimated network

443 increases up as a function of sample size. So does the median sensitivity, meaning we find

444 increasingly more true effects with increasing sample size. Median specificity goes somewhat

445 down as a function of sample size. This is not surprising since sensitivity and specificity are in a

446 trade-off, where increasing one, leads to decreasing another (Isvoranu et al., 2022)8 .

447 In Appendix C one can find a simulation set up like the one from Tutorial Box1 in combination

448 with how to analyze the results as exemplified in Tutorial box 11 for the Ising model. In Appendix

449 D one can find a simulation set up like the one from Tutorial Box1 in combination with how to
8
These results are merely as a means of example. In reality we would like to repeat the simulation study with a
lot more runs then the 10 runs we used in our example.

25
 1.0

0.8

correlation
sensitivity
specificity
0.6

0.4

100 250 500

sample size

Figure 1

Results from GGM simulation tutorial, see Tutorial Box 11

450 analyze the results as exemplified in Tutorial box 11 for the GVAR model.

# Load libraries:
library("dplyr")
library("tidyr")
library("ggplot2")

# Inspect results:
View(results) # These are the results obtained from Tutorial box 1

# Put data frame in long format for easy use with ggplot:
long <- gather(results, metric, value,
sensitivity, specificity, correlation)

# Make factors with labels:

long$nSample <- factor(long$nSample,
levels = c(100, 250, 500),
labels = c("100", "250", "500"))

# Let’s plot:
ggplot(long, aes(x = factor(nSample), y = value, fill = metric)) +
geom_boxplot() +
theme_bw() +
scale_fill_manual(values = c("#4C71BB","#CC79A7","#A22016")) +
xlab("sample size") +
ylab("") +
guides(fill = guide_legend(title = ""))

Tutorial box 11

Tutorial on how to analyze simulation results as obtained from Tutorial box 1.

26
451 Concluding notes

452 In this paper, we showed how to use the large exiting body of open-source software to set up a

453 simulation study within network psychometrics. Our aim was to make simulation tools available

454 to a broader audience of researches by providing a step-by-step guide in setting up a simulation

455 study for the three most often used models: the Ising model, GGM, and the GVAR model.

456 As a running example, we highlighted how these simulations tools can be used to determine

457 the estimation accuracy of model parameters for various sample sizes. The general idea we

458 highlighted with this example can easily be extended to the modeling frameworks discussed in

459 this paper (see Appendix C and Appendix D), to different estimation techniques, and to asses

460 the expected replicability for a given network structure, by, for example, adding an additional

461 independent sample to the simulation procedure.

462 Being able to performing simulation studies has many advantages as for example methods for

463 sample size determination are limited within the network modeling framework. Some simulation

464 based tools have been proposed in the context of cross-sectional network estimation to determine

465 the expected accuracy of a given network structure. For example, netSimulator() from the

466 bootnet package provides a framework for simple simulation studies regarding the performance

467 of network estimations techniques as a function of the sample size (Epskamp & Fried, 2023).

468 In addition, recently a new R-package has been introduced to compute sample sizes: powerly

469 (Constantin et al., 2023). Powerly makes use of an automated Monte Carlo algorithm to find

470 an optimal sample size given a network model structure, a performance measure of interest

471 (e.g., a sensitivity of 0.7) and a corresponding target value (e.g., reaching a sensitivity of 0.7

472 with a probability of 0.8). One of the benefits of powerly its hat the implemented stratified

473 bootstrapping procedure allows to quantify uncertainty around the recommendations provided

474 by the package. While the methods exploited in both packages can be extended to other types

475 of network models, currently the packages only support the determination of sample sizes for

476 cross-sectional network estimation techniques such as GGM.

477 Thus, currently it not possible to apply these simulations based tools to asses the number of

478 time-points needed for network estimated from intensive longitudinal data using GVAR estimation

479 as implemented in the GVAR package or multilevel network estimation as implemented in the

480 mlVAR package (Epskamp, 2023b; Epskamp et al., 2023). In addition, it is not possible to asses

481 the required sample size for more recently developed cross-sectional estimation techniques such

27
482 as (pseudo) ML estimation as implemented in the psychonerics package (Epskamp, 2020, 2023d)

483 or Bayesian estimation of network models as implemented in the easyBGM package (Huth &

484 Keetelaar, 2023; Huth et al., 2023). The simulation tools provided in this tutorial will enable

485 researchers to do so.

486 Assessing the expected replicability is another benefit of performing simulation studies.

487 Given sampling variability we would not expect networks to reproduce completely, and to what

488 extend we do expect networks to replicate depends on several factors such as the network

489 structure. Therefore, it is important to determine the replicability and generalizability of

490 estimated psychological network models across samples. To that extend, tools have been

491 developed to asses the expected replication in the context of cross-sectional models such as

492 the replicationSimulator() function from the bootnet package (Epskamp & Fried, 2023).

493 The replicationSimulator() function can be used to asses how well a network based on a

494 second independent sample would replicate based on the first independent sample. As for the

495 netSimulator(), and powerly() the replicationSimulator() is only applicable for cross-

496 sectional network estimation techniques.

497 Armed with the easy-to-use simulation tools described in this paper at hand, more researchers

498 can easily extend the ideas behind functions as netSimulator() and replicationSimulator()

499 to any type of network estimation technique to wish to employ.

500 Acknowledgments

501 This manuscript was supported by an NWO Research Talent Grant (no. 406-18-532) awarded to

502 R. H. A. Hoekstra.

503 Declaration of interest statement

504 The authors have declared that there are no conflicts of interest to the subject of this study.

28
505 References

506 Beard, C., Millner, A. J., Forgeard, M. J., Fried, E. I., Hsu, K. J., Treadway, M. T., Leonard,

507 C. V., Kertz, S., & Björgvinsson, T. (2016). Network analysis of depression and anxiety

508 symptom relationships in a psychiatric sample. Psychological Medicine, 46 (16), 3359–

509 3369.

510 Beck, E. D., & Jackson, J. J. (2020). Consistency and change in idiographic personality: A

511 longitudinal esm network study. Journal of Personality and Social Psychology, 118 (5),

512 1080–1100.

513 Borsboom, D., Cramer, A. O., Schmittmann, V. D., Epskamp, S., & Waldorp, L. J. (2011). The

514 small world of psychopathology. PloS one, 6 (11), e27407.

515 Borsboom, D., Deserno, M. K., Rhemtulla, M., Epskamp, S., Fried, E. I., McNally, R. J.,

516 Robinaugh, D. J., Perugini, M., Dalege, J., Costantini, G., et al. (2021). Network analysis

517 of multivariate data in psychological science. Nature Reviews Methods Primers, 1 (1),

518 1–18.

519 Borsboom, D., Fried, E. I., Epskamp, S., Waldorp, L. J., van Borkulo, C. D., van der Maas,

520 H. L., & Cramer, A. O. (2017). False alarm? a comprehensive reanalysis of “evidence

521 that psychopathology symptom networks have limited replicability” by forbes, wright,

522 markon, and krueger (2017). Journal of Abnormal Psychology, 126 (7), 989–999.

523 Bringmann, L. F., Elmer, T., Epskamp, S., Krause, R. W., Schoch, D., Wichers, M., Wigman,

524 J. T., & Snippe, E. (2019). What do centrality measures measure in psychological

525 networks? Journal of Abnormal Psychology, 128 (8), 892–903.

526 Burger, J., Hoekstra, R. H. A., Mansueto, A. C., & Epskamp, S. (2022). Network estimation

527 from time series and panel data. In A.-M. Isvoranu, S. Epskamp, L. J. Waldorp, &

528 D. Borsboom (Eds.), Network psychometrics with R: A guide for behavioral and social

529 scientists (pp. 169–192). Routledge.

530 Chib, S., & Greenberg, E. (1995). Understanding the metropolis-hastings algorithm. The Ameri-

531 can Statistician, 49 (4), 327–335.

532 Constantin, M., Schuurman, N. K., & Vermunt, J. (2023). A general monte carlo method for

533 sample size analysis in the context of network models. Psychological Methods, Advanced

534 online publication. https://doi.org/https://dx.doi.org/10.1037/met0000555

29
535 Costantini, G., Richetin, J., Preti, E., Casini, E., Epskamp, S., & Perugini, M. (2019). Stability

536 and variability of personality networks. A tutorial on recent developments in network

537 psychometrics. Personality and Individual Differences, 136, 68–78.

538 Dalege, J., Borsboom, D., Van Harreveld, F., Van den Berg, H., Conner, M., & Van der Maas,

539 H. L. (2016). Toward a formalized account of attitudes: The causal attitude network

540 (can) model. Psychological Review, 123 (1), 2–22.

541 Dalege, J., Borsboom, D., van Harreveld, F., & van der Maas, H. L. (2017). Network analysis on

542 attitudes: A brief tutorial. Social Psychological and Personality Science, 8 (5), 528–537.

543 Epskamp, S. (2023a). IsingSampler [R package version 10.2.3]. https://cran.r- project.org/

544 package=IsingSampler

545 Epskamp, S. (2020). Psychometric network models from time-series and panel data. Psychome-

546 trika, 85 (1), 206–231.

547 Epskamp, S. (2023b). graphicalVAR: Graphical VAR for experience sampling data [R package

548 version 0.3.3]. https://cran.r-project.org/package=graphicalVAR

549 Epskamp, S. (2023c). parSim: Parallel simulation studies [R package version 0.1.5]. https :

550 //cran.r-project.org/package=parSim

551 Epskamp, S. (2023d). psychonetrics [R package version 0.11.5]. https://cran.r- project.org/

552 package=psychonetrics

553 Epskamp, S., Borsboom, D., & Fried, E. I. (2018). Estimating psychological networks and their

554 accuracy: A tutorial paper. Behavior Research Methods, 50 (1), 195–212.

555 Epskamp, S., Deserno, M. K., & Bringmann, L. F. (2023). mlVAR: Multi-Level Vector Autore-

556 gression [R package version 0.5.1]. https://cran.r-project.org/package=mlVAR

557 Epskamp, S., & Fried, E. I. (2018). A tutorial on regularized partial correlation networks.

558 Psychological Methods, 23 (4), 617–212.

559 Epskamp, S., & Fried, E. I. (2023). bootnet: Bootstrap methods for various network estimation

560 routines [R package version 1.5.6]. https://cran.r-project.org/package=bootnet

561 Epskamp, S., Rhemtulla, M., & Borsboom, D. (2017). Generalized network psychometrics:

562 Combining network and latent variable models. Psychometrika, 82, 904–927.

563 Epskamp, S., van Borkulo, C. D., van der Veen, D. C., Servaas, M. N., Isvoranu, A.-M., Riese,

564 H., & Cramer, A. O. (2018). Personalized network modeling in psychopathology: The

565 importance of contemporaneous and temporal connections. Clinical Psychological Science,

566 6 (3), 416–427.

30
567 Epskamp, S., Waldorp, L. J., Mõttus, R., & Borsboom, D. (2018). The Gaussian graphical model

568 in cross-sectional and time-series data. Multivariate Behavioral Research, 53 (4), 453–480.

569 Finnemann, A., Borsboom, D., Epskamp, S., & van der Maas, H. L. (2021). The theoretical and

570 statistical ising model: A practical guide in R. Psych, 3 (04), 593–617.

571 Forbes, M. K., Wright, A. G., Markon, K. E., & Krueger, R. F. (2017). Evidence that psychopathol-

572 ogy symptom networks have limited replicability. Journal of Abnormal Psychology, 126 (7),

573 969–988.

574 Forbes, M. K., Wright, A. G., Markon, K. E., & Krueger, R. F. (2019). The network approach

575 to psychopathology: Promise versus reality. World Psychiatry, 18 (3), 272–273.

576 Forbes, M. K., Wright, A. G., Markon, K. E., & Krueger, R. F. (2021). Quantifying the reliability

577 and replicability of psychopathology network characteristics. Multivariate Behavioral

578 Research, 56 (2), 224–242.

579 Fried, E. I., Eidhof, M. B., Palic, S., Costantini, G., Huisman-van Dijk, H. M., Bockting,

580 C. L., Engelhard, I., Armour, C., Nielsen, A. B., & Karstoft, K.-I. (2018). Replicability

581 and generalizability of posttraumatic stress disorder (ptsd) networks: A cross-cultural

582 multisite study of ptsd symptoms in four trauma patient samples. Clinical Psychological

583 Science, 6 (3), 335–351.

584 Hallgren, K. A. (2013). Conducting simulation studies in the R programming environment.

585 Tutorials in Quantitative Methods for Psychology, 9 (2), 43–60.

586 Haslbeck, J. (2023). Mgm [R package version 1.2-14]. https://cran.r-project.org/package=IsingFit

587 Haslbeck, J. M. (2022). Estimating group differences in network models using moderation

588 analysis. Behavior Research Methods, 1–19.

589 Haslbeck, J. M., Epskamp, S., Marsman, M., & Waldorp, L. J. (2021). Interpreting the ising

590 model: The input matters. Multivariate Behavioral Research, 56 (2), 303–313.

591 Hoekstra, R. H. A., Epskamp, S., & Borsboom, D. (2023). Heterogeneity in individual network

592 analysis: Reality or illusion? Multivariate Behavioural Research, 58 (4), 762–786.

593 Huth, K., & Keetelaar, S. (2023). easybgm [R package version 0.1.1]. https://cran.r-project.org/

594 package=easybgm

595 Huth, K. B., de Ron, J., Goudriaan, A. E., Luigjes, J., Mohammadi, R., van Holst, R. J.,

596 Wagenmakers, E.-J., & Marsman, M. (2023). Bayesian analysis of cross-sectional networks:

597 A tutorial in R and JASP. Advances in Methods and Practices in Psychological Science,

598 6 (4), 1–18.

31
599 Isvoranu, A.-M., & Epskamp, S. (2023). Which estimation method to choose in network psy-

600 chometrics? Deriving guidelines for applied researchers. Psychological Methods, 28 (4),

601 925–946.

602 Isvoranu, A.-M., Epskamp, S., Waldorp, L., & Borsboom, D. (2022). Network psychometrics

603 with R: A guide for behavioral and social scientists. Routledge.

604 Isvoranu, A.-M., van Borkulo, C. D., Boyette, L.-L., Wigman, J. T., Vinkers, C. H., Borsboom,

605 D., & Investigators, G. (2017). A network approach to psychosis: Pathways between

606 childhood trauma and psychotic symptoms. Schizophrenia Bulletin, 43 (1), 187–196.

607 Mansueto, A. C., Wiers, R. W., van Weert, J., Schouten, B. C., & Epskamp, S. (2023). In-

608 vestigating the feasibility of idiographic network models. Psychological Methods, 28 (5),

609 1052–1068.

610 Morris, T. P., White, I. R., & Crowther, M. J. (2019). Using simulation studies to evaluate

611 statistical methods. Statistics in Medicine, 38 (11), 2074–2102.

612 Murray, I. (2007). Advances in markov chain monte carlo methods. University of London,

613 University College London (United Kingdom).

614 Robinaugh, D. J., Hoekstra, R. H., Toner, E. R., & Borsboom, D. (2020). The network approach

615 to psychopathology: A review of the literature 2008–2018 and an agenda for future

616 research. Psychological Medicine, 50 (3), 353–366.

617 Robinaugh, D. J., Millner, A. J., & McNally, R. J. (2016). Identifying highly influential nodes in

618 the complicated grief network. Journal of Abnormal Psychology, 125 (6), 747–757.

619 Van Borkulo, C. D., Borsboom, D., Epskamp, S., Blanken, T. F., Boschloo, L., Schoevers, R. A.,

620 & Waldorp, L. J. (2014). A new method for constructing networks from binary data.

621 Scientific Reports, 4 (1), 5918.

622 Van Borkulo, C. D., van Bork, R., Boschloo, L., Kossakowski, J. J., Tio, P., Schoevers, R. A.,

623 Borsboom, D., & Waldorp, L. J. (2022). Comparing network structures on three aspects:

624 A permutation test. Psychological Methods, Advanced online publication. https://doi.

625 org/10.1037/met0000476.

626 van Borkulo, C., & Epskamp, S. (2023). IsingFit [R package version 0.4]. https : / / cran . r -

627 project.org/package=IsingFit

628 Vervaet, M., Puttevils, L., Hoekstra, R. H., Fried, E., & Vanderhasselt, M.-A. (2021). Transdi-

629 agnostic vulnerability factors in eating disorders: A network analysis. European Eating

630 Disorders Review, 29 (1), 86–100.

32
631 Watts, D. J., & Strogatz, S. H. (1998). Collective dynamics of ‘small-world’networks. Nature,

632 393 (6684), 440–442.

633 Wickham, H., François, R., Henry, L., & Müller, K. (2023). dplyr: A grammar of data manipulation

634 [R package version 1.1.4]. https://CRAN.R-project.org/package=dplyr

635 Wickham, H., & Grolemund, G. (2023). R for data science: Import, tidy, transform, visualize,

636 and model data [R package version 1.3.0]. https://CRAN.R-project.org/package=tidyr

637 Wickham, H. (2016). ggplot2: Elegant graphics for data analysis. Springer-Verlag New York.

638 https://ggplot2.tidyverse.org

639 Wickham, H., Averick, M., Bryan, J., Chang, W., McGowan, L. D., François, R., Grolemund, G.,

640 Hayes, A., Henry, L., Hester, J., Kuhn, M., Pedersen, T. L., Miller, E., Bache, S. M.,

641 Müller, K., Ooms, J., Robinson, D., Seidel, D. P., Spinu, V., . . . Yutani, H. (2019).

642 Welcome to the tidyverse. Journal of Open Source Software, 4 (43), 1686.

643 Williams, D. R., Rhemtulla, M., Wysocki, A. C., & Rast, P. (2019). On nonregularized estimation

644 of psychological networks. Multivariate Behavioral Research, 54 (5), 719–750.

645 Yin, J., & Li, H. (2011). A sparse conditional gaussian graphical model for analysis of genetical

646 genomics data. The Annals of Applied Statistics, 5 (4), 2630–2650.

33
647 Appendix A

648 Evaluation metrics

649 Several metrics can be used to evaluate the performance of the estimated network. These metrics

650 evaluate different aspects of the estimation procedure, such as the ability to retrieve edges, the

651 accuracy of the edge weights, the accuracy of the centrality indices. Here we will discuss the

652 most commonly used metrics in detail.

653 Network accuracy

654 Sensitivity. Sensitivity is a metric that is used to determine the ability to identify true

655 edges. That means sensitivity can be used to determine if the estimation technique is capable

656 of picking up effects that we know are there. The sensitivity is expressed as the proportion of

657 edges in the true network that is also included in the estimated network:

# of true present edges in the estimated network

sensitivity =
# of total present edges in the true network

658 Here, ’true present edges’ denotes edges that are shared between both the estimated network

659 and the true network.

660 Various variants of sensitivity can be computed, including the ’signed sensitivity.’ In signed

661 sensitivity, the numerator considers only edges estimated with the same sign as in the true

662 network. Put differently, if an edge in the true network is positive but is estimated as a negative

663 edge in the estimated network, this edge no longer contributes to the count of true present edges

664 in the estimated network.

665 Additional sensitivity variants encompass the ‘top 50%’, ‘top 25%’, and ‘top 10%’ sensitivity.

666 In these instances, both the numerator and denominator restrict their calculation to the top

667 percentage of edges. For example, the ‘top 50%’ sensitivity, denotes the proportion of the 50%

668 strongest edges in the true network that are also included in the estimated network.

669 High sensitivity signifies most existing edges were successfully detected, indicating the

670 evaluated method excels in capturing effects that are truly present. In general, sensitivity

671 increases when power increases (Isvoranu & Epskamp, 2023). Power can be increased by

672 increasing the number of participants in cross-sectional analysis, increasing both the number of

673 participants and time series in longitudinal multilevel analysis, or increasing the number of time

674 series in idiographic longitudinal analysis. Conversely, a low sensitivity implies most existing

34
675 edges were not detected, potentially leading to sub-optimal replication results and inflated visual

676 heterogeneity across estimated networks (Hoekstra et al., 2023; Mansueto et al., 2023).

677 Specificity. Specificity is a metric that can be used to determine the capacity to avoid the

678 inclusion of false edges, i.e., the capability to correctly identify absent edges.

679 Specificity is quantified as the ratio of edges that are genuinely missing in the true network

680 and accurately excluded from the estimated network:

681 of missing edges in the true network that were correctly not included in the estimated

682 network:
# of true absent edges in the estimated network
specificity =
# of total absent edges in the true network

683 Here ‘true absent edges’ refers to edges that are absent in both the true and the estimated

684 network.

685 In principle, specificity should be high. When trends in specificity are encountered as a

686 function of sample size, this can be an indication of problems with the estimation method

687 (Isvoranu & Epskamp, 2023).

688 Precision. Precision is a metric to determine the accuracy of the estimation method. It

689 measures the proportion of edges present in the estimated network that are also present in the

690 true network:

# of true present edges in the estimated network

precision =
# of edges in the estimated network

691 Here ‘true present edges’ entails edges that are shared between both the estimated network and

692 the true network.

693 Similar to the sensitivity metric, precision allows for the calculation of various variants such

694 as the ‘top 50%’, ‘top 25%’, and ‘top 10%’ precision, which involve considering only the top %

695 prominent edges when computing precision.

696 It is important to note specificity and precision are closely related: like specificity, precision

697 tends to be low when numerous edges are included in the estimated network that are absent

698 from the true network, resulting in a high number of false positive edges.

699 Jaccard index. The Jaccard index is the proportion of shared edges relative to the total

700 number of edges:

|A ∩ B| |A ∩ B|
J(A, B) = =
|A ∪ B| |A| + |B| − |A ∩ B|

35
701 Where A and B are matrices containing 0 when there is no edge present and 1 if there is an

702 edge present. When we are mostly interested in the accuracy of the non-zero elements rather

703 than the overall accuracy, the Jaccard index allows us to focus specifically on the overlapping

704 edges between the two network structures.

705 Edge accuracy

706 Sensitivity, specificity, precision, and the Jaccard index are metrics to investigate the structure

707 of networks by assessing whether edges are present or absent. These metrics, however, do not

708 tell us something about the accuracy of the estimated edge weight. Nevertheless, we can think

709 of many scenarios in which it is important to determine accuracy of the estimated edge weights.

710 For one, edge weights play a pivotal role in determining the visual appearance of the network

711 structure, and two, various other metrics that are often of interest such as network density and

712 centrality metrics are contingent upon the accuracy of the estimated edge weights.

713 Correlation. To investigate the accuracy of edge weights correlations are frequently

714 employed as a metric. The Pearson correlation between true and estimated edge-weights

715 evaluates their similarity, thereby providing an indication of the precision of the estimated edge

716 weights. In addition the correlation between aboslute edge weights and non-zero edge weights

717 can be computed as a means to compare edges weights between the estimated network structure

718 and true network structure.

719 In general, a higher correlation coefficient implies a more accurate estimation of the edge

720 weight. However, it is important to consider that the strength of the correlation coefficient is

721 also determined by variance. In cases, where the true and estimated network structures are

722 perfectly identical, exhibiting little to no variance, the correlation coefficient is close to zero.

723 This means the strength of the correlation and the accuracy of the estimated edge-weights do not

724 share a one-to-one correspondence. Consequently, it is important to inspect additional metrics

725 in addition to the Pearson correlation.

726 Bias. In addition to correlating edge weights to determine the accuracy of their estimation

727 we can also investigate the bias in edge weights. For example we can compute the average

728 absolute deviation (bias) between the true edge weight and the estimated edge weight. We can

729 compute this bias for either all edges in both networks, or for only for the true edges that were

730 included in the estimated network.

36
731 Centrality accuracy

732 To give an indication of important nodes in the network, centrality indices such as strength,

733 closeness, betweenness, or expected influence are computed to provide insights into the prominence

734 of nodes within a network (e.g., Borsboom et al., 2011; Bringmann et al., 2019; Epskamp &

735 Fried, 2018; Robinaugh et al., 2016).

736 Correlations. To assess the accuracy of these centrality indices, Pearson correlations

737 as well as Kendall correlations can be computed between the true centrality indices and the

738 centrality indices as computed based on the estimated network structure (Isvoranu & Epskamp,

739 2023).

740 Top-ranked. Often, a primary focus often centers on nodes with the highest centrality

741 indices (Bringmann et al., 2019). If interested in the ability to correctly detect the nodes with

742 the highest centrality indices, an examination of the top-ranked nodes is optional. This entails

743 an investigation into the frequency with which the same nodes occupy the ‘top 1’, ‘top 3’, or

744 ‘top 5’ position of most central nodes.

745 Replicability

746 The evaluation metrics discussed here can be extended to assess the replicability of network

747 estimation procedures. In this context, the focus shifts from comparing the true network to an

748 estimated network, to a comparison between two estimated networks. Thus, we are no longer

749 interested in comparing the true data generating network to an estimated network, instead, we

750 would want to compare two estimated networks with each other. Metrics such as the correlation

751 between edge-weights, or the correlation between centrality indices are intuitive indicators to

752 determine the replication of a network structures.

753 Furthermore, networks can be compared on network characteristics such as connectivity

754 (i.e., percentage of actualized edges) and network density. The quality of the replication can be

755 evaluated using correlation metrics or metrics that gauge the dissimilarity between two matrices,

756 such as the Jaccard index which measures the proportion of shared edges relative to the total

757 number of edges.

758 Changes in the estimated edges can be quantified by calculating the percentage of change in

759 edge weights, the proportion of replicated edges (i.e., edges that are included in both network

760 structures), the proportion of non-replicated edges, and the proportion of edges unique to the

37
761 replication set. Discrepancies in node centrality can be evaluated by correlating centrality indices,

762 rank order correspondence, and assessing the consistency of the top 1, 3, and 5 most central

763 nodes across the replicated networks.

38
764 Appendix B

765 Determining parameter values for Ising model

766 Generating a synthetic Ising model can be complicated, as many choices can be made such as

767 determining the values for the parameters in the model. This is not an arbitrary choice, as

768 we discussed in the main paper that the encoding used (i.e., either {0, 1} or {−1, 1}), and the

769 parameter values (i.e., thresholds denoted as τ and edge weights denoted as ω), need to be

770 specified in such a way that the constructed Ising model leads to generated data with enough

771 entropy. To determine what combination of encoding, threshold parameters, and edge weights,

772 leads to a structure that generates data with enough entropy, is the very reason to conduct a

773 simulation study. In this appendix, we briefly explain why symmetric endocing is preferred when

774 generating an Ising model. How this choice of encoding works best when specifying threshold

775 parameters as 0, and determine what value of edge weights works well by means of a simulation

776 study.

777 Choosing the encoding and thresholds

778 Typically, the dichotomous responses for the Ising model are either encoded using 0 and 1, which

779 we term binary encoding, or −1 and 1, which we term symmetric encoding. Binary encoding is

780 often the default in psychological research and psychometrics (Haslbeck et al., 2021). This also

781 means that some of the software packages that estimate Ising models from psychological data

782 assume binary encoded data (e.g., IsingFit requires binary data, and mgm prefers binary data

783 for parameterization purposes (Haslbeck, 2023; van Borkulo & Epskamp, 2023)). Symmetric

784 encoding, on the other hand, is the default used in statistical physics when simulating data

785 under the Ising model. Only a few R packages for psychological network estimation allow for

786 symmetric encoded data (e.g., the psychonetrics package allows for both binary and symmetric

787 encoded data (Epskamp, 2020)).

788 Important to note is that the two encodings are equivalent. This means that if we generate

789 data using symmetric encoding, and subsequently transform the data to binary encoding (i.e.,

790 making all the −1 values 0) to fit an Ising model, we obtain an equivalent model. The obtained

791 model parameters can easily be transformed into one-another. For example, we can transform

792 Ising parameters based on the binary encoding {0, 1} to parameters based on the symmetric

39
793 encoding {−1, 1} as follows:

(−1,1) (0,1)
ωij = 0.25ωij
(−1,1) (0,1) (0,1)
X
τi = 0.5τi − 0.25 ωij
j

794 Here, the superscript denotes the encoding. Likewise, we can transform parameters based on

795 the symmetric encoding to parameters based on the binary encoding as follows:

(0,1)
ωij = 4ω (−1,1)
(0,1) (−1,1)
X (−1,1)
τi = 2τi −2 ωij
j

796 These transformations to transform Ising parameters from one encoding to another are imple-

797 mented in the LinTransform() function in the IsingSampler package (Epskamp, 2023a). These

798 transformations can be very useful in data generation, as it, for example, allows one to generate

799 symmetric encoded data (which typically is easier as explained below), transform the data to

800 binary encoded data for a network estimation technique (e.g., IsingFit), fit a network in the

801 binary domain, and then transform those results back to the symmetric domain to assess the

802 performance of the estimated network parameters.

803 If the threshold parameters (τ ) are chosen too low or too high, all responses will simply be

804 identical (e.g., not endorsed or endorsed), and no information is present for statistical modeling.

805 A very useful property of the symmetric encoding of the Ising model is the following:

 
(−1,1)
E yi |τi = 0 = 0.

806 That is, if the threshold parameter is 0, a symmetric encoded variable has an expected value

807 of 0, meaning that it has a 50% probability on −1 and a 50% probability on 1. This property

808 ensures we obtain some variation in the variables we simulate. Therefore, it is preferred to use

809 symmetric encoding {−1, 1} and set the threshold parameters of all variables to 0 for the data

810 generating network structure.

811 Binary encoded data does not have this same property. Instead, setting all thresholds to 0

40
812 for binary encoded data could instead lead to almost all responses being 1 (depending on the

813 strength of the network parameters in ω). We can derive from the transformations above that
(0,1)
814 when we have binary encoded data, we require τi to be set to negative half the node strength

815 (sum of the connected edge weights) to achieve the same result. For example, if node 1 has three

816 connected edges each with a edge strength of 0.20 (e.g., ω12 = ω13 = ω14 = 0.20, then the node

817 strength of node 1 is 0.20 + 0.20 + 0.20 = 0.6 and we would need to set τ1 = −0.3 to obtain a

818 50% probability of (not) endorsing the variable.

819 With the encoding and thresholds chosen, all that is left is to generate the network structure.

820 To investigate what parameter values work well, we performed a small simulation study.

821 Simulation set up

822 We generated synthetics Ising models, varying the following conditions:

823 Number of nodes: 5 or 10

824 Network structure: random (25% or 50% of potential edges were included at random), chain

825 (e.g., node 1 is connected to node 2, node 2 is connected to node 3, etc.), Curie-Weiss (a

826 fully connected network).

827 Edge weight: Different mean edge weights were simulated (the weights were chosen at random

828 using a normal distribution with the set mean edge weight and a standard deviation of 0.1)

829 Encoding: Binary or symmetric

830 Thresholds: -1, 0, 1

831 As an evaluation metric we inspected the entropy of the specified Ising model using the

832 IsingEntrophy() function from the IsingSampler package (Epskamp, 2023a). The simulation

833 code used to conduct this study can be found below.

834 Results

835 We have visualised the results in Figure B.1. Results indicate that the entropy goes down as a

836 function of the edge weight, meaning that the stronger connected the network the fewer response

837 patters are generated. Entropy is at a maximum when all τ and ω are set to zero, but this is not

838 an interesting structure to generate data under as this would simply imply completely random

41
839 (coin-flip) data. Entropy reduces to 0 in most settings, except in symmetric encoded data with

840 all τ parameters set to 0 (our above described preferred simulation setting), in which case, due

841 to symmetry, two response patterns eventually emerge (all variables −1 or all variables 1). In

842 general, when generating symmetric encoded data with all threshold parameters set to 0 edge

843 weights around 0.25 and 0.5 tend to work well.

encoding (−1, 1)
thresholds:−1 thresholds:0 thresholds:1
1.0
0.8

# nodes: 5
0.6
entropy / # of nodes

0.4
0.2
0.0
1.0
0.8

# nodes: 10
0.6
0.4
0.2
0.0
0 0.25 0.5 1 2 4 0 0.25 0.5 1 2 4 0 0.25 0.5 1 2 4
edge weight

encoding (0, 1)
thresholds:−1 thresholds:0 thresholds:1
1.0
0.8 # nodes: 5
0.6
entropy / # of nodes

0.4
0.2
0.0
1.0
0.8
# nodes: 10

0.6
0.4
0.2
0.0
0 0.25 0.5 1 2 4 0 0.25 0.5 1 2 4 0 0.25 0.5 1 2 4
edge weight

model chain curie−weiss random_p0.25 random_p0.5

Figure B.1

Entropies (in scaled bits) based on various simulated Ising network parameters.

844 Simulation code

845 # Load in libraries:

42
846 library("parSim")
847 library("dplyr")
848 library("ggplot2")
849 library("patchwork")
850

851 # Simulation:
852 results <- parSim(
853 nNode = c(5,10),
854 weight = c(0, 0.25, 0.5, 1, 2, 4),
855 sdWeight = 0.1,
856 model = c("chain","random_p0.25", "random_p0.5", "curie-weiss"),
857 thresholds = c(-1, 0, 1),
858 encoding = c(0,-1),
859 reps = 100,
860 nCores = 12,
861 expression = {
862 # Libraries:
863 library("IsingSampler")
864 library("bootnet")
865

866 # Determine thresholds for each node:

867 thresholds <- rep(thresholds, 5)
868

869 # Create network based on type of model:

870 if (model == "chain"){
871 graph = ifelse(genGGM(nNode) != 0, 1, 0)
872 } else if (model %in% c("random_p0.25","random_p0.5")){
873 # Determine probability of edge inclusion:
874 p <- ifelse(model == "random_p0.25", 0.25, 0.5)
875 edges <- sample(c(0, 1), nNode * (nNode-1)/2, TRUE, prob = c(1-p, p))
876

877 # Determine graph structure:

878 graph <- matrix(0, nNode, nNode)
879 graph[lower.tri(graph)] <- edges
880 graph <- graph + t(graph)
881

43
882 } else if (model == "curie-weiss"){
883 graph <- matrix(1, nNode, nNode)
884 diag(graph) <- 0
885 }
886

887 # Add weights:

888 graph[lower.tri(graph)] <- graph[lower.tri(graph)] * rnorm(sum(lower.tri(graph)),
889 weight,
890 sdWeight)
891 graph[upper.tri(graph)] <- t(graph)[upper.tri(graph)]
892 diag(graph) <- 0
893

894 # Compute entropy:

895 list(
896 entropy = IsingEntrophy(graph, thresholds, responses = c(encoding,1))
897 )
898 })
899

900 ## Plot results

901

902 # Scale entropy:

903 results$entropy_scaled <- results$entropy / results$nNode
904

905 # Make factors with labels:

906 results$nNode <- factor(results$nNode,
907 levels = sort(unique(results$nNode)),
908 labels = paste0("# nodes: ",sort(unique(results$nNode))))
909

910 results$thresholds <- factor(results$thresholds,

911 levels = sort(unique(results$thresholds)),
912 labels = paste0("thresholds:",sort(unique(results$thresholds))))
913

914

915 results$encoding <- factor(results$encoding,

916 levels = sort(unique(results$encoding)),
917 labels = paste0("encoding (", sort(unique(results$encoding)), ", 1)"))

44
918

919 results$weight_factor <- as.factor(results$weight)

920

921 # Summarize results:

922 results_sum <- results %>%
923 group_by(nNode, thresholds, encoding, model, weight_factor) %>%
924 summarize(mean = mean(entropy_scaled, na.rm=TRUE),
925 se = sd(entropy_scaled, na.rm=TRUE)/sqrt(n()))
926

927 # Create seperate plot for each type of encoding:

928 graphs <- list()
929 for (enc in sort(unique(results$encoding))){
930 sub <- results_sum %>% filter(encoding == enc)
931

932 graphs <- c(graphs, list(

933 ggplot(sub, aes(x = as.numeric(weight_factor), y = mean, colour = model)) +
934 geom_line() +
935 geom_point() +
936 facet_grid(nNode ~ thresholds) +
937 #theme(legend.position="bottom") +
938 theme_bw() +
939 ggtitle(enc) +
940 scale_x_continuous(breaks = seq_along(levels(sub$weight_factor)),
941 labels = levels(sub$weight_factor)) +
942 scale_y_continuous(breaks = seq(0, 1, by = 0.2),
943 minor_breaks = seq(0, 1, by = 0.1),
944 limits = c(0, 1)) +
945 scale_color_manual(values = c("#943d7a", "#4C71BB","#CC79A7","#A22016")) +
946 xlab("edge weight") +
947 ylab("entropy / # of nodes")
948 ))
949 }
950

951 # Combine the two ggplots, vertically align*ed and sharing the same legend:
952 graphs[[1]] + graphs[[2]] + plot_layout(nrow = 2, guides = ’collect’)

45
953 Appendix C

954 Tutorial: Setting up a simulation study within the Ising modelling framework

955 In this tutorial, we exemplify how a simulation study can be set up within an Ising model

956 framework to answer the question: “To what extend can we accurately estimate an Ising

957 model for reasonable sample sizes?”. The tutorial code can be found below, and on OSF:

958 https://osf.io/ygjn7/. In order to answer this research question, we varied the sample size

959 condition (i.e., n ∈ 100, 250, 500), and obtained the sensitivity, specificity, and correlation as our

960 evaluation metrics. See Appendix A for a description of the most commonly used evaluation

961 metrics in network psychometrics, and Table 1 for all evaluation metrics implemented in the

962 evalNet() function.

963 We use the parSim() function from the parSim package (Epskamp, 2023c) to provide the

964 skeleton for our simulation study. We specified our sample size condition (i.e., sampleSize =

965 c(100, 250, 500)), and within the R expression we load the required R packages, specify the

966 true data generating network, simulate data based upon this network, and estimate a network

967 based on the simulated data to ultimately compare the estimated to the true network using

968 our evaluation metrics. In the tutorial code we set the reps argument to 100 by means of a

969 simplification, but ideally this number of repetitions would higher.

970 After we ran our simulation, we can analyze the results. We use the tidyverse packages

971 dplyr and tidyr for easy data manipulation, and ggplot2 for plotting the results (Wickham &

972 Grolemund, 2023; Wickham et al., 2023; Wickham, 2016; Wickham et al., 2019). We display our

973 results in a boxplot with our sample size condition on the x-axis the evaluation metrics (i.e.,

974 correlation, sensitivity, and specificity) are plotted on the y-axis. In order to plot our results

975 with the ggplot() function, some data modifications need to be made. First we want to format

976 our results in such a way that they are easy to use within ggplot(). This means, we need to go

977 from a wide data format to a long data format. We can use the gather() function from the

978 tidyr package to do so. After cleaning up the data and creating the right format, we can use

979 ggplot() to display the simulated results and start interpreting the results, see the tutorial

980 code below for the simulation study R-code and Figure C.1 for the simulation results.

981 Our results indicate that the correlation between the estimated network model and the

982 true data generating network model goes up as the sample size increases. In addition, the

983 sensitivity increases, meaning more true effects are estimated with increasing sample sizes. We

46
984 see for n = 500, the specificity is decreasing. As was noted in the main paper, sensitivity and

985 specificity are in a trade-off with one another. As sensitivity increases, specificity tends to go

986 down (Isvoranu et al., 2022).9

1.00

0.75

correlation
0.50
sensitivity
specificity

0.25

0.00

50 100 150
sample size

Figure C.1

Results from Ising model simulation tutorial

987 Ising tutorial code

988 # Set random number generator:

989 set.seed(5)
990

991 # Load libraries:

992 library("parSim")
993 library("devtools")
994 library("dplyr")
995 library("tidyr")
996 library("ggplot2")
997

998 # Run the simulation:

999 results <- parSim(
1000 # Specify number of conditions:
1001 sampleSize = c(100, 250, 500),
1002

9
These results are merely as a means of example. In reality we would like to repeat the simulation study with
more runs then the 100 runs we used in our example.

47
1003 # Number of repetitions:
1004 reps = 100,
1005

1006 # Number of cores:

1007 nCores = 1,
1008

1009 # R expression:
1010 expression = {
1011 # Load R packages
1012 source("https://raw.githubusercontent.com/RiaHoekstra/simulation_functions/main/evalNet.R")
1013 source("https://raw.githubusercontent.com/RiaHoekstra/simulation_functions/main/genIsing.R")
1014 library("bootnet")
1015 library("IsingSampler")
1016

1017 # Generate an Ising model with 6 nodes:

1018 trueIsing <- genIsing(nNode = 6)
1019

1020 # Determine thresholds for each node:

1021 thresholds = 0
1022 thresholds <- rep(thresholds, 6)
1023

1024 # Generate data:

1025 dataIsing <- IsingSampler(n = sampleSize,
1026 graph = trueIsing,
1027 thresholds = thresholds,
1028 responses = c(-1L, 1L))
1029

1030 # Transform encoded data to binary data:

1031 dataIsing[dataIsing == -1] <- 0
1032

1033 # Estimate network

1034 estIsing <- estimateNetwork(dataIsing, default = "IsingFit")
1035

1036 # Transform estimated parameters from in binary domain to symmetric domain:

1037 estIsing <- LinTransform(graph = estIsing$graph,
1038 thresholds = estIsing$results$thresholds,

48
1039 from = c(0L, 1L),
1040 to = c(-1L, 1L))
1041

1042 # Obtain evaluation metric of interest

1043 metrics <- evalNet(trueIsing, estIsing$graph, metric = c("sensitivity",
1044 "specificity",
1045 "correlation"))
1046

1047 # Return results

1048 data.frame(metrics)
1049 }
1050 )
1051

1052 # Inspect results:

1053 View(results)
1054

1055 # Put data frame in long format for easy use with ggplot:
1056 long <- gather(results, metric, value,
1057 sensitivity, specificity, correlation)
1058

1059 # Make factors with labels:

1060 long$sampleSize <- factor(long$sampleSize,
1061 levels = c(100, 250, 500),
1062 labels = c("100", "250", "500"))
1063

1064 # Inspect median values for our evaluation metrics per sample size:
1065 long %>% group_by(metric, sampleSize) %>%
1066 summarize(median(value))
1067

1068 # Let’s plot:

1069 ggplot(long, aes(x = factor(sampleSize), y = value, fill = metric)) +
1070 geom_boxplot() +
1071 theme_bw() +
1072 scale_fill_manual(values = c("#4C71BB","#D291BC","#A22016")) +
1073 xlab("sample size") +
1074 ylab("") +

49
1075 guides(fill = guide_legend(title = ""))

50
1076 Appendix D

1077 Tutorial: Setting up a simulation study within a GVAR modelling framework

1078 In this tutorial, we exemplify how a simulation study can be set up within a GVAR model

1079 framework to answer the question: “To what extend can we accurately estimate a GVAR

1080 model for reasonable time series?”. The tutorial code can be found below, and on OSF:

1081 https://osf.io/ygjn7/. In order to answer this research question, we varied the number of time

1082 series (i.e., t ∈ 100, 150, 200), and obtained the sensitivity, specificity, and correlation as our

1083 evaluation metrics for both the contemporaneous network, as well as the temporal network.

1084 See Appendix A for a description of the most commonly used evaluation metrics in network

1085 psychometrics, and Table 1 for all evaluation metrics implemented in the evalNet() function.

1086 We use the parSim() function from the parSim package (Epskamp, 2023c) to provide the

1087 skeleton for our simulation study. We specified our sample size condition (i.e., nTime = c(100,

1088 150, 200)), and within the R expression we load the required R packages, specify the true data

1089 generating network, simulate data based upon this network, and estimate a network based on

1090 the simulated data to ultimately compare the estimated to the true network using our evaluation

1091 metrics. In the tutorial code we set the reps argument to 100 by means of a simplification, but

1092 ideally this number would be higher.

1093 After we ran our simulation, we can analyze the results. We use the tidyverse packages

1094 dplyr and tidyr for easy data manipulation, and ggplot2 for plotting the results (Wickham &

1095 Grolemund, 2023; Wickham et al., 2023; Wickham, 2016; Wickham et al., 2019). We display our

1096 results in a boxplot with our sample size condition on the x-axis the evaluation metrics (i.e.,

1097 correlation, sensitivity, and specificity) are plotted on the y-axis. In order to plot our results

1098 with the ggplot() function, some data modifications need to be made. First we want to format

1099 our results in such a way that they are easy to use within ggplot(). This means, we need to go

1100 from a wide data format to a long data format. We can use the gather() function from the

1101 tidyr package to do so. After cleaning up the data and creating the right format, we can use

1102 ggplot() to display the simulated results and start interpreting the results, see the tutorial

1103 code below for the simulation study R-code and Figure C.1 for the simulation results.

1104 Results indicate that the correlation between the estimated network model and the true

1105 data generating network model goes up as the sample size increases. For the contemporaneous

1106 network model, both sensitivity and specificity increase. The estimated network structure is

51
1107 accurately estimated from t = 250 onward. For the temporal network, we see the same result as

1108 with the other exemplary simulation studies: specificity is decreasing as the sensitivity increases.

1109 In addition, sensitivity is low when t = 100, and in general is the correlation, sensitivity lower

1110 for the temporal network as for the contemporaneous network. This is a logical consequence

1111 of the number of parameters that need to be estimated for each of these network models. As

1112 the temporal network model is a directed network model, many more parameters need to be

1113 estimated, and that often requires more power, i.e., bigger sample sizes10 .

contemporaneous temporal

1.00

0.75

correlation
0.50
sensitivity
specificity

0.25

0.00

100 250 500 100 250 500

number of time points

Figure D.1

Results from GVAR simulation tutorial, see Tutorial Box ??

1114 GVAR tutorial code

1115 # set random number generator

1116 set.seed(5)
1117

1118 # Load in libraries:

1119 library("parSim")
1120 library("devtools")
1121

1122 # Simulation:
1123 results <- parSim(
1124 # Specify number of conditions:
10
These results are merely as a means of example. In reality we would like to repeat the simulation study with
more runs then the 100 runs we used in our example.

52
1125 nTime = c(100, 250, 500),
1126

1127 # Number of repetitions:

1128 reps = 100,
1129

1130 # Number of cores:

1131 nCores = 7,
1132

1133 # R expression:
1134 expression = {
1135

1136 # Load libraries:

1137 library("bootnet")
1138 library("graphicalVAR")
1139 source("https://raw.githubusercontent.com/RiaHoekstra/simulation_functions/main/genGVAR.R")
1140 source("https://raw.githubusercontent.com/RiaHoekstra/simulation_functions/main/evalNet.R")
1141

1142 ## Generate true GVAR model

1143 trueGVAR <- genGVAR(6,
1144 propPositiveCon = 0.7,
1145 propPositiveTemp = 0.7)
1146

1147 ## Generate time-series data from true GVAR model

1148 data <- as.data.frame(graphicalVARsim(nTime,
1149 trueGVAR$beta,
1150 trueGVAR$kappa))
1151

1152 ## Estimate GVAR model from simulated data with graphicalVAR

1153 estGVAR <- estimateNetwork(data, default = "graphicalVAR")
1154

1155 ## Evaluation metrics

1156

1157 # Temporal network

1158 metricB <- evalNet(trueGVAR$beta, estGVAR$results$beta[,-1],
1159 directed = TRUE,
1160 metric = c("sensitivity",

53
1161 "specificity",
1162 "correlation"))
1163

1164 # Contemporaneous network

1165 metricK <- evalNet(trueGVAR$kappa, estGVAR$results$kappa,
1166 directed = TRUE,
1167 metric = c("sensitivity",
1168 "specificity",
1169 "correlation"))
1170

1171

1172 # Specify output as data frame:

1173 res <- data.frame(metricB = metricB,
1174 metricK = metricK)
1175

1176 # Return output:

1177 return(res)
1178

1179 })
1180

1181 # Analyze the results

1182

1183 # Inspect results

1184 View(results)
1185

1186 # Put data frame in long format for easy use with ggplot:
1187 long <- gather(results, metric, value,
1188 metricB.sensitivity, metricB.specificity, metricB.correlation,
1189 metricK.sensitivity, metricK.specificity, metricK.correlation)
1190

1191 # Make factors with labels:

1192 long$nTime <- factor(long$nTime,
1193 levels = c(100, 250, 500),
1194 labels = c("100", "250", "500"))
1195

1196 # Add network type for plot:

54
1197 long <- long %>% mutate(network = case_when(startsWith(metric, "metricB") ~ "temporal",
1198 startsWith(metric, "metricK") ~ "contemporaneous"))
1199

1200 # Change metric for plot:

1201 long <- long %>% mutate(metric = case_when(endsWith(metric, "correlation") ~ "correlation",
1202 endsWith(metric, "sensitivity") ~ "sensitivity",
1203 endsWith(metric, "specificity") ~ "specificity"))
1204

1205 # Let’s plot:

1206 ggplot(long, aes(x = factor(nTime), y = value, fill = metric)) +
1207 facet_grid(~ network) +
1208 geom_boxplot() +
1209 theme_bw() +
1210 scale_fill_manual(values = c("#4C71BB","#D291BC","#A22016")) +
1211 xlab("number of time points") +
1212 ylab("") +
1213 guides(fill = guide_legend(title = ""))

55