Optimizing Schnapsen Strategy: Feature

Selection in Machine Learning Models

Artificial Intelligence 2024

Vrije Universiteit Amsterdam

Abstract. This paper aims to find the most effective features for a ma-
chine learning model. This is done by training machine learning models
with a training dataset of games between two rdeep bots and using per-
mutation importance to determine the importance of each feature in the
game Schnapsen. The weights of every feature were measured using the
ELI5 package. The features are ranked in a table where the features
are named. This demonstrates which characteristic is essential for each
machine learning model. The weights given by the supervised and unsu-
pervised models were substantially different. The results show that the
features that were expected to be important such as talon size and trump
suit didn’t always have a high weight and that the information about the
opponent’s cards was more valuable than the information about the bot’s
cards.

Keywords: Schnapsen · Machine Learning · Feature Selection.

1 Introduction

This research paper investigates the relationship between model and feature se-
lection in machine learning. To answer the research question: ”How do various
machine learning models influence feature selection for Schnapsen, and which
features effectively enhance bot performance?” We tested various machine learn-
ing algorithms, including Neural Networks, Linear Logistic Regression, Decision
Tree Regression, Decision Tree Classifier, Random Forest Classification, Random
Forest Regression, and K-means Clustering, in the game of Schnapsen. The re-
sults of this study will provide valuable insight into the role that feature selection
plays in developing effective machine-learning models for the game Schnapsen.

To analyze the impact of feature selection on the performance of the machine

learning models, an experiment was conducted using the Permutation Impor-
tance method. This method is part of the scikit-learn eli5 library, which mea-
sures the score/performance changes after permuting (removing) a single feature
[1]. Using this data, its’s possible to identify the most effective features for each
model, gaining a better understanding of the relationship between model selec-
tion and feature selection in machine learning.
Artificial Intelligence 2024 2

2 Background Information
2.1 Schnapsen
Schnapsen is a turn-based card game played with two people. The game is played
with 20 cards, 5 of each suit: the aces, kings, queens, jacks and tens. The aces
are worth 11 points, the tens are worth 10 points, the kings are worth 4 points,
the queens are worth 3 points, and the jacks are worth 2 points. To win, one of
the players needs to get seven-game points. Game points can be won by earning
66 points. The number of game points won depends on how the opponent did: if
the opponent scored 33 points or higher, the winner gets one game point. If the
opponent scores below 33 points but has scored at least one trick, the winner
receives two game points; if the opponent has not won a trick, the winner gets
three game points. Getting 66 points can be done in two ways: winning tricks
against your opponent and declaring marriages. You must play a card worth
more than your opponent’s to win a trick. The player that starts the trick is
called the leader; before the first trick, the players draw a card. The player with
the best card is the dealer. The other player will be the leader of the first trick.
After each trick, the other player becomes the leader of the new trick. Each
player has five cards. The remaining cards are turned upside down and put in
the talon(stock).[2]

The dealer deals the cards by giving the other player three cards, taking three
cards, then one of the cards is turned right side up, after which the last two
cards are dealt to both hands, and the remaining cards are put upside down on
top of the card that is right side up, that pile of cards is the talon. The card that
is placed right side up is the trump card, and the suit of that card is the trump
suit. At the start of the trick, the leader can (if possible) do a trump exchange,
where the jack of the trump suit can be exchanged with the trump card. The
leader can also declare a marriage at the start of the trick. A marriage can be
declared if the leader has a king and a queen in the same suit. A marriage is
worth 20 points, and a trump marriage (a marriage with a king and queen of
the trump suit) is worth 40 points. The game has two phases: before, the talon
is empty, and after, the talon is empty. In the second phase, the rules change:
the second player has to play a card of the same suit as the card played by the
leader and beat that card if possible. If the second player does not have a card
of the same suit, a card of the trump suit must be played, and if that is not
possible, any card may be played.[2]

3 ML Models
3.1 Neural Network Machine Learning Model
MLPClassifier is a classifier in the Scikit-Learn library and a type of neural net-
work. It is a supervised machine learning model that processes data through
layers of interconnected nodes (neurons). Each neuron receives input from the
Artificial Intelligence 2024 3

previous layer and applies a weighted sum to the inputs. The final layer outputs
represent the predicted class probabilities, and the class with the highest proba-
bility is assigned as the prediction. In the case of Schnapsen, it would select the
class with the highest probability of winning the game with a probability closest
to 1. [4]

3.2 Logistic Regression model

Logistic regression model (also known as logic model) is a supervised machine

learning classification algorithm that is used to predict the probability of certain
classes based on some dependent variables. The output is always between 0 and
1 which is suitable for a binary classification class for example if a player wins a
trick or loses a trick.[3]

MLPClassifier includes some adjustable parameters for optimizing the model.

In the usage of MLPClassifier, hidden-layer-size and learning-rate were adjusted.
Hidden-layer-sizes represents the number of neurons in a hidden layer, and
learning-rate-init controls the step size in updating the weights. These parame-
ters were set to 30 and 0.0001, respectively, while the rest were kept at default.[4]

3.3 Decision Tree Regressor model

Decision Tree Regressor model is a supervised machine learning model It has

a hierarchical/tree structure, which consists of a root node, branches, internal
nodes and leaf nodes. Decision tree regression observes features of an object and
trains a model in the structure of a tree to predict data in the future to produce
a continuous output.[5]

3.4 Decision Tree Classifier model

Decision Trees are a supervised learning method used for classification and re-
gression. The goal is to create a model that predicts the value of a target variable
by learning simple decision rules inferred from the data features. Decision Tree
Classifier is a class capable of performing multi-class classification on a dataset.[5]

3.5 Random Forest Classifier model

Random forest classifier is a supervised machine learning algorithm made of de-

cision trees on various sub-samples of the dataset and uses averaging to improve
the predictive accuracy and over-fitting. It uses bagging and features randomness
when building individual trees to create an uncorrelated forest of trees whose
prediction is more accurate than any individual tree. A random forest classifier
is used to predict a set of specified labels.[6]
Artificial Intelligence 2024 4

3.6 Random Forest Regressor model

Random Forest Regression is a supervised machine learning model, which uses

ensemble learning methods. The ensemble learning method combines predictions
from multiple decision trees to get a prediction that is more accurate. A random
forest regressor is used to predict real valued outputs which vary and don’t
require outputs predicted to be in a fixed set.[7]

3.7 K-Means Clustering model

K-Means clustering is an unsupervised machine learning model which is based

on centroid-based clustering that calculates distances between each data point
based on their similarity and centroid to assign it to a cluster. Each centroid is
a collection of feature values which define the resulting groups[8]

4 Research Question

In the game of Schnapsen there are a large number of features the machine
learning bots utilize. Some are more important and some are less. The goal of
this research paper is to find out what are the most effective features for a given
machine learning model and how the model choice affects the feature selection.
The features will be judged based on the value each machine learning model
assigned to them after their respective training.

4.1 Hypothesis

The hypothesis of this research is that there will be specific features that have a
higher value and importance compared to the other features. It is also expected
that the top features for the classification and regression models will differ due
to their distinct approaches in machine learning. The expected features that
are predicted to be generally weighed higher than others among most of the
machine learning models are high cards in opponent’s and bot’s hand, talon size
and trump suit.

5 Experiment

The experiment utilized a training dataset comprised of 10,000 games between

two instances of the rdeep bot. Rdeep was chosen due to its strong performance
against all other Schnapsen bots, making it an ideal choice for training ma-
chine learning models due to its strategic play and performance in the game of
Schnapsen.
Artificial Intelligence 2024 5

Weights Feature Name

p1 points The points earned by the bot
p1 pending points The points pending for the bot as a result of a marriage
opponents points The points earned by the opponent
opponent pending points The points pending for the opponent as a result of a marriage
trump suit one hot The suit of the trump card, one-hot-encoded
The current phase of the game
game phase encoded [0, 1] if phase 1
[1, 0] if phase 2
talon size Size of the talon
The leader of the trick
i am leader [0, 1] if bot is leader
[1, 0] if opponent is leader
hand cards Cards in bot’s hand
trump card The trump card, bottom of the talon
won cards Cards the bot has won
opponent won cards Cards the opponent has won
opponent known cards Cards in the opponent’s hand

Using the training dataset of games between two rdeep bots, the machine learn-
ing models were trained. The models were then fitted to the data and targets
using the permutation importance method. Using permutation importance the
relative importance of each feature was determined and ranked accordingly. Fea-
tures with high weights can be considered as important predictors in the model.1

Furthermore, an ANOVA test was conducted using the combined weight values
of each individual model excluding K-means clustering due to its significantly
larger weight values compared to the other models. This test shows whether
there are statistically significant differences in the mean weight values of the
machine learning models. The test was preformed using scipy.stats.foneway from
the SciPy module.[9] 2

6 Results
The weights of every feature were measured using the eli5 package by refitting the
machine learning model with our dataset. One-hot encoding helped to categorize
all features that the machine learning models have used. The First column refers
to weights assigned by the models, and the second is the feature to which the
weight belongs.

First, high cards in opponents’ hands, followed by players’ cards, were expected
to be among the top features of each bot since they often give the highest amount
of points outside marriage. Furthermore the suit of cards in different features
1
Relevant code can be found in section 8.1 of appendix
2
Code can be found in section 8.2 of appendix
Artificial Intelligence 2024 6

don’t seem to affect the weight of the features in any way since they are evenly
distributed.

Second, contrary to the expected outcome, half of the machine learning models
did not give high weights to the trump suits or talon size, which are critical
features in knowing what card is most likely to win or help predict the outcome
of the game. Additionally, certain machine learning models gave high weights to
opponent’s and player’s points which were not predicted to be among the top
expected features.

Third, features concerned about opponents’ card states are almost always higher
weighted than the players. For example, an Ace held by an opponent always
weighs more than an Ace that is unknown or held by the player. The same
occurrence follows every card and even the points of each player.

Lastly, the only unsupervised model K-means Clustering had weights in the
hundreds of thousands, while other bots gave weights that were always less than
one. Likely caused by the eli5 package not being made for an unsupervised model,
which might overstate the importance of some features. Furthermore K-means
Clustering machine learning model preferred most of the game features which
are not containing information about cards such as card in opponent’s hand.
Next this bot weighed higher all opponent’s cards, of same points, than it’s own
cards.

Table 1. Logistic Regression Permutation Importance Table

Weights Feature Name

Weights Feature Name
0.0929 ± 0.0015 talon size
0.0827 ± 0.0009 points of a player
0.0620 ± 0.0015 opponent points
0.0581 ± 0.0008 player is leader
0.0527 ± 0.0011 opponent is leader
0.0367 ± 0.0013 game phase 2
0.0329 ± 0.0014 game phase 1
0.0160 ± 0.0011 Ten Hearts opponents hand
0.0060 ± 0.0014 Ace Spades on opponents hand
0.0057 ± 0.0007 Ace Hearts on opponents hand
0.0054 ± 0.0006 Ace Clubs on opponents hand
0.0047 ± 0.0010 Ten Spades players hand
0.0046 ± 0.0006 Ace Clubs on players hand
0.0045 ± 0.0004 Ace Spades on players hand
0.0044 ± 0.0004 trump suit diamonds
0.0044 ± 0.0004 Ace Diamonds on opponents hand
Artificial Intelligence 2024 7

Table 2. Neural Network Permutation Importance Table

Weights Feature Name

0.1114 ± 0.0019 opponent points
0.0896 ± 0.0017 points of a player
0.0291 ± 0.0012 trump suit diamonds
0.0265 ± 0.0006 talon size
0.0246 ± 0.0007 trump suit hearts
0.0226 ± 0.0010 trump suit clubs
0.0216 ± 0.0008 trump suit spades
0.0121 ± 0.0007 player is leader
0.0109 ± 0.0006 opponent is leader
0.0080 ± 0.0004 Ace Hearts on opponents hand
0.0075 ± 0.0004 Ace Spades on opponents hand
0.0074 ± 0.0003 Ace Diamonds on opponents hand
0.0070 ± 0.0009 Ten Hearts opponents hand
0.0063 ± 0.0005 Ten Diamonds Unknown
0.0060 ± 0.0005 Ten Spades Unknown
0.0056 ± 0.0003 Jack Diamonds on opponents hand

Table 3. Decision Tree Classification Permutation Importance Table

Weights Feature Name

0.3124 ± 0.0028 opponent points
0.3124 ± 0.0012 points of a player
0.2149 ± 0.0013 player is leader
0.0887 ± 0.0010 Ace Spades on opponents hand
0.0649 ± 0.0005 Ace Clubs on opponents hand
0.0638 ± 0.0006 Ace Diamonds on opponents hand
0.0564 ± 0.0007 opponent is leader
0.0482 ± 0.0007 trump suit clubs
0.0470 ± 0.0005 Ace Hearts on opponents hand
0.0461 ± 0.0008 trump suit spades
0.0422 ± 0.0010 Ten Hearts on opponents hand
0.0391 ± 0.0005 trump suit hearts
0.0338 ± 0.0002 trump suit diamonds
0.0333 ± 0.0004 Ace Clubs on players hand
0.0313 ± 0.0005 trump suit hearts3
0.0311 ± 0.0007 game phase 2
Artificial Intelligence 2024 8

Table 4. Decision Tree Regression Permutation Importance Table

Weights Feature Name

1.2613 ± 0.0054 points of a player
1.2568 ± 0.0099 opponent points
0.8404 ± 0.0061 opponent is leader
0.3543 ± 0.0036 Ace Spades on opponents hand
0.2623 ± 0.0016 Ace Clubs on opponents hand
0.2597 ± 0.0045 Ace Diamonds on opponents hand
0.2280 ± 0.0023 Ten Hearts on opponents hand
0.1934 ± 0.0022 trump suit clubs
0.1850 ± 0.0026 Ace Hearts on opponents hand
0.1829 ± 0.0016 trump suit spades
0.1699 ± 0.0026 game phase 2
0.1547 ± 0.0016 trump suit hearts
0.1351 ± 0.0013 trump suit diamonds
0.1329 ± 0.0016 Ace Clubs on players hand
0.1255 ± 0.0012 Ten Clubs Unknown
0.1229 ± 0.0034 Game phase 1

Table 5. Random Forest Classification Permutation Importance Table

Weights Feature Name

0.0494 ± 0.0016 opponent points
0.0414 ± 0.0007 points of a player
0.0024 ± 0.0000 talon size
0.0024 ± 0.0001 Ace Spades on opponents hand
0.0022 ± 0.0002 Ace Clubs on opponents hand
0.0018 ± 0.0001 Ace Diamonds on opponents hand
0.0018 ± 0.0001 Ten Hearts on opponents hand
0.0018 ± 0.0002 game phase 2
0.0015 ± 0.0001 Ace Hearts on opponents hand
0.0013 ± 0.0001 Ten Spades on opponents hand
0.0013 ± 0.0001 Jack Hearts opponents hand
0.0012 ± 0.0001 Ten Diamonds opponents hand
0.0011 ± 0.0001 player is leader
0.0010 ± 0.0001 Ten Diamonds on opponent’s hand
0.0008 ± 0.0001 opponent is leader
0.0008 ± 0.0001 Ace Clubs on players hand
Artificial Intelligence 2024 9

Table 6. Random Forest Regression Permutation Importance

Weights Feature Name

0.5990 ± 0.0041 points of a player
0.5803 ± 0.0037 opponent points
0.1440 ± 0.0008 opponent is leader
0.1284 ± 0.0010 player is leader
0.1078 ± 0.0008 Ace Spades on opponents hand
0.0696 ± 0.0007 Ace Diamonds on opponents hand
0.0688 ± 0.0005 Ace Clubs on opponents hand
0.0641 ± 0.0006 Ace Hearts on opponents hand
0.0600 ± 0.0007 Ten Clubs Unknown
0.0579 ± 0.0006 trump suit clubs
0.0477 ± 0.0005 trump suit spades
0.0476 ± 0.0007 Ten Hearts on opponents hand
0.0427 ± 0.0005 trump suit hearts
0.0390 ± 0.0003 Ten Spades on opponents hand
0.0370 ± 0.0005 trump suit diamonds
0.0360 ± 0.0004 trump suit spades

Table 7. K-means Clustering Permutation Importance Table

Weights Feature Name

420308.1599 ± 1971.5218 talon size
168719.7697 ± 307.5919 game phase 1
168200.0994 ± 604.2461 game phase 2
144853.2147 ± 631.6338 opponent is leader
144802.2410 ± 638.7344 player is leader
115692.1266 ± 869.6890 player pending points
60863.9908 ± 222.7441 trump suit diamonds
60240.3905 ± 408.2612 trump suit hearts
59304.7034 ± 144.1569 trump suit clubs
58223.5526 ± 278.2673 trump suit spades
49507.8658 ± 279.8174 Ten Hearts on opponents hand
49100.0612 ± 152.6299 Ten Spades on opponents hand
49046.1333 ± 154.8688 Jack Spades opponents hand
49019.6117 ± 279.4339 Jack Clubs opponents hand
48860.8884 ± 131.4751 Jack Hearths opponents hand
48797.1800 ± 137.5981 Jack Diamonds opponents hand
Artificial Intelligence 2024 10

7 Findings

The experiment was designed to compare features essential to different machine

learning models. Most of the basic features (trump suit, scores, leader game
phase or talon size) were represented in most machine learning models, which
was expected since they show the game state for the AI. Furthermore, every
model gave high weight to the feature, indicating if the opponent had a card
worth a high number of points(Ace and 10). It is most likely because those cards
can be won in a trick and secure an advantage over the opponent.

The unsupervised model K-Means Clustering gave the highest points to al-
most all the basic features, unlike other supervised learning models, which often
preferred high-value cards over trump suits, for example. Features that show op-
ponents’ cards, scores or if its leader always had a higher value than the player’s
features of the same type. This indicates that knowledge of opponents’ features
gives an advantage in the game.

This ANOVA test on the permutation importance values of all models excluding
K-means clustering resulted in a F-statistic score of 8.95 and P-value of 6.28e-
07. The P-value is significantly below the significance level of 0.05. This implies
that there are significant variations in the feature importance values across the
different machine learning models as it rejects the null-hypothesis that there are
no significant differences in the mean importance values of features among the
different machine learning models. This suggests that certain features have a
more significant role in predicting game outcomes depending on the model. For
example, a model might have a higher importance of features related to game
phase or player status. While another model might prioritize features related to
trump suits. The statistical testing displays the importance of selecting the most
appropriate machine learning model and features for a Schnapsen playing bot
and in other applications of machine learning bots. 3

8 Conclusion

In conclusion, the results of our study demonstrate the importance of considering

feature selection when developing machine-learning models. By using the Per-
mutation Importance method to determine the most effective features for each of
the tested algorithms; Linear Logistic Regression, Neural Networks, K Nearest
Neighbor, Decision Tree Classifier, Decision Tree Regression, Random Forest Re-
gression, Random Forest Classification, and K-means Clustering. These findings
show the relationship between model selection and feature selection, and em-
phasize the importance of considering both when developing machine-learning
models.
3
Section 8.2 of Appendix
Artificial Intelligence 2024 11

The Permutation Importance testing was carried out on the same dataset used
for training the models. As a result, these values may not accurately reflect the
importance of features in different scenarios within the game. Nevertheless, it is
still able to provide a good representation of the preferred features for each type
of model and can help be a guide to feature selection and model selection.

In the exploration of relations between machine learning models and feature

selection in Schnapsen. Results show that cards in opponent’s hand were valued
the most by almost all of the machine learning models. Contrary to expectations
however, the classification and regression models have similar weight distribution
among same features. Furthermore, supervised machine learning models were
also very similar in terms of importance of features while unsupervised and neural
network models also had very similar results of feature importance. Despite
this the ANOVA test shows that there is a significant difference in the feature
permutation importance values between the supervised learning models.

This research on machine learning models feature selection which might not
correlate with how their win ratio change if they played games without these
features. For example, their features that weighed the most might not affect their
win ratios as much as different lower weight features. Further research on how
successful are machine learning bots without certain features need to be done to
holistically solve the question regarding effective feature and model selection.
Artificial Intelligence 2024 12

References
1. Permutation Importance — ELI5 0.11.0 documentation. (n.d.).
https://eli5.readthedocs.io/en/latest/blackbox/permutation-importance.html
2. McLeod, J. (2022, December 1). Schnapsen - card game rules.
https://www.pagat.com/marriage/schnaps.html
3. sklearn.linear-model.LogisticRegression. (n.d.). Scikit-learn. https://scikit-
learn.org/stable/modules/generated/sklearn.linear-model.LogisticRegression.html
4. sklearn.neural-network.MLPClassifier. (n.d.). Scikit-learn. https://scikit-
learn.org/stable/modules/generated/sklearn.neural-network.MLPClassifier.html
5. 1.10. Decision Trees. (n.d.). Scikit-learn. https://scikit-
learn.org/stable/modules/tree.html
6. sklearn.ensemble.RandomForestClassifier. (n.d.). Scikit-learn. https://scikit-
learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html
7. sklearn.ensemble.RandomForestRegressor. (n.d.). Scikit-learn. https://scikit-
learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html
8. sklearn.cluster.KMeans. (n.d.). Scikit-learn. https://scikit-
learn.org/stable/modules/generated/sklearn.cluster.KMeans.html
9. scipy.stats.f oneway — SciPy v1.12.0 Manual. (n.d.). Numpy
and Scipy Documentation — Numpy and Scipy documentation.
https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.f oneway.html
Artificial Intelligence 2024 13

Appendix
8.1 ml-bot.py

Machine l e a r n i n g bot with ELI5 Permutation Importance

from s c h n a p s e n . game import Bot , P l a y e r P e r s p e c t i v e ,
SchnapsenDeckGenerator , Move , Trick , GamePhase
from t y p i n g import Optional , c a s t , L i t e r a l
from s c h n a p s e n . deck import S u i t , Rank
# Modules / Imports needed t o implement t h e machine
l e a r n i n g models
from s k l e a r n . n e u r a l n e t w o r k import M L P C l a s s i f i e r
from s k l e a r n . l i n e a r m o d e l import L o g i s t i c R e g r e s s i o n
from s k l e a r n . t r e e import D e c i s i o n T r e e C l a s s i f i e r ,
DecisionTreeRegressor
from s k l e a r n . ensemble import R a n d o m F o r e s t C l a s s i f i e r ,
RandomForestRegressor
from s k l e a r n . mixture import GaussianMixture
import j o b l i b
import time
import p a t h l i b
# Module f o r g e t t i n g p e r m u t a t i o n i m p o r t a n c e o f machine
l e a r n i n g models
import e l i 5
from e l i 5 import PermutationImportance

c l a s s MLPlayingBot ( Bot ) :
”””
This c l a s s l o a d s a t r a i n e d ML model and u s e s i t t o
play
”””

def i n i t ( s e l f , m o d e l l o c a t i o n : p a t h l i b . Path , name

: O p t i o n a l [ s t r ] = None ) −> None :
”””
C r e a t e a new MLPlayingBot which u s e s t h e model
stored in the m o f e l l o c a t i o n .

: param m o d e l l o c a t i o n : The f i l e c o n t a i n i n g t h e
model .
”””
super () . i n i t ( name )
model location = model location
a s s e r t m o d e l l o c a t i o n . e x i s t s ( ) , f ” Model c o u l d not
be found a t : { m o d e l l o c a t i o n }”
# l o a d model
Artificial Intelligence 2024 14

self . model = j o b l i b . load ( model location )

d e f get move ( s e l f , p e r s p e c t i v e : P l a y e r P e r s p e c t i v e ,
l e a d e r m o v e : O p t i o n a l [ Move ] ) −> Move :
# get the s a t e f e a t u r e r e p r e s e n t a t i o n
state representation = get state feature vector (
perspective )
# g e t t h e l e a d e r ’ s move r e p r e s e n t a t i o n , even i f
i t i s None
leader move representation =
get move feature vector ( leader move )
# g e t a l l my v a l i d moves
my valid moves = p e r s p e c t i v e . v a l i d m o v e s ( )
# g e t t h e f e a t u r e r e p r e s e n t a t i o n s f o r a l l my
v a l i d moves
my move representations : l i s t [ l i s t [ int ] ] = [ ]
f o r my move i n my valid moves :
m y m o v e r e p r e s e n t a t i o n s . append (
g e t m o v e f e a t u r e v e c t o r ( my move ) )

# c r e a t e a l l model i n p u t s , f o r a l l bot ’ s v a l i d
moves
action state representations : l i s t [ l i s t [ int ] ] =
[]

i f perspective . am i leader () :
follower move representation =
g e t m o v e f e a t u r e v e c t o r ( None )
f o r my move representation in
my move representations :
a c t i o n s t a t e r e p r e s e n t a t i o n s . append (
state representation +
my move representation +
follower move representation )
else :
f o r my move representation in
my move representations :
a c t i o n s t a t e r e p r e s e n t a t i o n s . append (
state representation +
leader move representation +
my move representation )

model output = s e l f . m o d e l . p r e d i c t p r o b a (
action state representations )
Artificial Intelligence 2024 15

w i n n i n g p r o b a b i l i t i e s o f m o v e s = [ outcome prob [ 1 ]
f o r outcome prob i n model output ]
h i g h e s t v a l u e : f l o a t = −1
best move : Move
f o r index , v a l u e i n enumerate (
winning probabilities of moves ) :
i f value > highest value :
highest value = value
best move = my valid moves [ i n d e x ]
a s s e r t best move i s not None
r e t u r n best move

c l a s s MLDataBot ( Bot ) :
”””
This c l a s s i s d e f i n e d t o a l l o w t h e c r e a t i o n o f a
t r a i n i n g s c h na p s e n bot d a t a s e t , t h a t a l l o w s us t o
t r a i n a Machine L e a r n i n g (ML) Bot
P r a c t i c a l l y , i t h e l p s us r e c o r d how t h e game p l a y s
out a c c o r d i n g t o a p r o v i d e d Bot b e h a v i o u r ; b u i l d
what i s c a l l e d a ” r e p l a y memory”
In more d e t a i l , we c r e a t e one t r a i n i n g sample f o r
each d e c i s i o n t h e bot makes w i t h i n a game , where a
d e c i s i o n i s an a c t i o n s e l e c t i o n f o r a s p e c i f i c
game s t a t e .
Then we r e l a t e each d e c i s i o n with t h e outcome o f t h e
game , i . e . whether t h i s bot won o r not .
This way we can then t r a i n a bot a c c o r d i n g t o t h e
assumption t h a t :
” d e c i s i o n s i n e a r l i e r games t h a t ended up i n
v i c t o r i e s s h o u l d be p r e f e r r e d o v e r d e c i s i o n s
t h a t l e a d t o l o s t games ”
This c l a s s o n l y r e c o r d s t h e d e c i s i o n s and game
outcomes o f t h e p r o v i d e d bot , a c c o r d i n g t o i t s own
p e r s p e c t i v e − i n c o m p l e t e game s t a t e knowledge .
”””

def i n i t ( s e l f , bot : Bot , r e p l a y m e m o r y l o c a t i o n :

p a t h l i b . Path ) −> None :
”””
: param bot : t h e p r o v i d e d bot t h a t w i l l a c t u a l l y
p l a y t h e game and make d e c i s i o n s
: param r e p l a y m e m o r y l o c a t i o n : t h e f i l e n a m e under
which t h e r e p l a y memory r e c o r d s w i l l be
”””
Artificial Intelligence 2024 16

s e l f . bot : Bot = bot

s e l f . r e p l a y m e m o r y f i l e p a t h : p a t h l i b . Path =
replay memory location

d e f get move ( s e l f , p e r s p e c t i v e : P l a y e r P e r s p e c t i v e ,
l e a d e r m o v e : O p t i o n a l [ Move ] ) −> Move :
”””
This f u n c t i o n s i m p l y c a l l s t h e get move o f
t h e p r o v i d e d bot
”””
r e t u r n s e l f . bot . get move ( p e r s p e c t i v e=p e r s p e c t i v e ,
l e a d e r m o v e=l e a d e r m o v e )

d e f n o t i f y g a m e e n d ( s e l f , won : bool , p e r s p e c t i v e :
P l a y e r P e r s p e c t i v e ) −> None :
”””
When t h e game ends , t h i s f u n c t i o n r e t r i e v e s t h e
game h i s t o r y and more s p e c i f i c a l l y a l l t h e
r e p l a y memories t h a t can
be d e r i v e d from i t , and s t o r e s them i n t h e form
o f s t a t e −a c t i o n s v e c t o r r e p r e s e n t a t i o n s and
t h e c o r r e s p o n d i n g outcome o f t h e game

: param won : Did t h i s bot win t h e game?

: param s t a t e : The f i n a l s t a t e o f t h e game .
”””
# we r e t r i e v e t h e game h i s t o r y w h i l e a c t u a l l y
d i s c a r d i n g the l a s t u s e l e s s h i s t o r y record (
which i s a f t e r t h e game has ended ) ,
# we know none o f t h e T r i c k s can be None b e c a u s e
that i s only f o r the l a s t record
game history : l i s t [ tuple [ PlayerPerspective , Trick
] ] = cast ( l i s t [ tuple [ PlayerPerspective , Trick
] ] , perspective . get game history () [: −1])
# we a l s o s a v e t h e t r a i n i n g l a b e l ”won o r l o s t ”
w o n l a b e l = won

# we i t e r a t e o v e r a l l t h e rounds o f t h e game
for round player perspective , round trick in
game history :

i f round trick . is trump exchange () :

l e a d e r m o v e = r o u n d t r i c k . exchange
f o l l o w e r m o v e = None
Artificial Intelligence 2024 17

else :
leader move = round trick . leader move
follower move = round trick . follower move

# we do not want t h i s r e p r e s e n t a t i o n t o
i n c l u d e a c t i o n s t h a t f o l l o w e d . So i f t h i s
a g e n t was t h e l e a d e r , we i g n o r e t h e
f o l l o w e r s move
i f round player perspective . am i leader () :
f o l l o w e r m o v e = None

state actions representation =

create state and actions vector representation
(
p e r s p e c t i v e=r o u n d p l a y e r p e r s p e c t i v e ,
l e a d e r m o v e=l e a d e r m o v e , f o l l o w e r m o v e
=f o l l o w e r m o v e )

# append r e p l a y memory t o f i l e
with open ( f i l e = s e l f . r e p l a y m e m o r y f i l e p a t h ,
mode=”a ” ) a s r e p l a y m e m o r y f i l e :
# replay memory line : l i s t [ tuple [ l i s t ,
number ] ] = [
state actions representation ,
won label ]
# w r i t i n g t o r e p l a y memory f i l e i n t h e
form ” [ f e a t u r e l i s t ] | | i n t ( w o n l a b e l )
]
r e p l a y m e m o r y f i l e . w r i t e ( f ”{ s t r (
s t a t e a c t i o n s r e p r e s e n t a t i o n ) [1: −1]}
| | { i n t ( w o n l a b e l ) }\n ” )

d e f train ML model ( r e p l a y m e m o r y l o c a t i o n : O p t i o n a l [
p a t h l i b . Path ] ,
m o d e l l o c a t i o n : O p t i o n a l [ p a t h l i b . Path
],
m o d e l c l a s s : L i t e r a l [ ”NN” , ”LR” , ”DTC
” , ”DTR” , ”RFC” , ”RFR” , ”KMC” ] = ”
LR”
) −> None :
”””
Train t h e ML model f o r t h e MLPlayingBot based on
r e p l a y memory s t o r e d byt t h e MLDataBot .
Artificial Intelligence 2024 18

This i m p l e m e n t a t i o n has t h e o p t i o n t o t r a i n a n e u r a l
network model , a model based on l i n e a r r e g r e s s i o n ,
a d e c i s i o n t r e e c l a s s i f i e r model , a d e c i s i o n t r e e
r e g r e s s o r model , a random f o r e s t c l a s s i f i e r model
, a random f o r e s t r e g r e s s o r model o r a K−means
c l u s t e r i n g model .
The model c l a s s e s used i n t h i s i m p l e m n t a t i o n a r e not
n e c e s a r i l y optimal .

: param r e p l a y m e m o r y l o c a t i o n : L o c a t i o n o f t h e games
s t o r e d by MLDataBot , d e f a u l t p a t h l i b . Path ( ’
ML replay memories ’ ) / ’ t e s t r e p l a y m e m o r y ’
: param m o d e l l o c a t i o n : L o c a t i o n where t h e model w i l l
be s t o r e d , d e f a u l t p a t h l i b . Path ( ” ML models ” ) / ’
test model ’
: param m o d e l c l a s s : The machine l e a r n i n g model c l a s s
t o be used , e i t h e r ’NN’ f o r a n e u r a l network , ’LR’
f o r a l i n e a r r e g r e s s i o n , ’DTC’ f o r a d e c i s i o n
t r e e c l a s s i f i e r , ’DTR’ , f o r a d e c i s i o n t r e e
r e g r e s s o r , ’RFC’ f o r a random f o r e s t c l a s s i f i e r , ’
RFR’ f o r a random f o r e s t r e g r e s s o r and ’KMC’ f o r K
−means c l u s t e r i n g .
: param o v e r w r i t e : Whether t o o v e r w r i t e a p o s s i b l y
e x i s t i n g model .
”””
i f r e p l a y m e m o r y l o c a t i o n i s None :
r e p l a y m e m o r y l o c a t i o n = p a t h l i b . Path ( ’
ML replay memories ’ ) / ’ t e s t r e p l a y m e m o r y ’
i f m o d e l l o c a t i o n i s None :
m o d e l l o c a t i o n = p a t h l i b . Path ( ” ML models ” ) / ’
test model ’
a s s e r t m o d e l c l a s s == ’NN’ o r m o d e l c l a s s == ’LR’ o r
m o d e l c l a s s == ’DTC’ o r m o d e l c l a s s == ’DTR’ o r
m o d e l c l a s s == ’RFC’ o r m o d e l c l a s s == ’RFR’ o r
m o d e l c l a s s == ’KMC’ , ”Unknown model c l a s s ”

# check t h a t t h e r e p l a y memory d a t a s e t i s found a t

the s p e c i f i e d l o c a t i o n
i f not r e p l a y m e m o r y l o c a t i o n . e x i s t s ( ) :
r a i s e V a l u e E r r o r ( f ” D a t a s e t was not found a t : {
replay memory location } !” )

# Check i f model e x i s t s a l r e a d y
i f model location . exists () :
r a i s e ValueError (
Artificial Intelligence 2024 19

f ” Model a t { m o d e l l o c a t i o n } e x i s t s a l r e a d y
and o v e r w r i t e i s s e t t o F a l s e . \nNo new
model w i l l be t r a i n e d , p r o c e s s t e r m i n a t e s
”)

# check i f d i r e c t o r y e x i s t s , and i f not , then c r e a t e

it
m o d e l l o c a t i o n . p a r e n t . mkdir ( p a r e n t s=True , e x i s t o k=
True )

data : l i s t [ l i s t [ i n t ] ] = [ ]
targets : l i s t [ int ] = [ ]
with open ( f i l e =r e p l a y m e m o r y l o c a t i o n , mode=”r ” ) a s
replay memory file :
for l i n e in replay memory file :
feature string , won label str = line . split
(”||”)
feature list strings : l i s t [ str ] =
feature string . s p l i t (” ,”)
f e a t u r e l i s t = [ int ( feature ) for feature in
feature list strings ]
won label = int ( won label str )
data . append ( f e a t u r e l i s t )
t a r g e t s . append ( w o n l a b e l )

p r i n t (” Dataset S t a t i s t i c s : ” )
s a m p l e s o f w i n s = sum ( t a r g e t s )
samples of losses = len ( targets ) − samples of wins
p r i n t ( ” Samples o f wins : ” , s a m p l e s o f w i n s )
p r i n t ( ” Samples o f l o s s e s : ” , s a m p l e s o f l o s s e s )

# What type o f model w i l l be used depends on t h e

v a l u e o f t h e parameter u s e n e u r a l n e t w o r k
i f m o d e l c l a s s == ’NN’ :
#############################################
# Neural Network model p a r a m e t e r s :
# l e a r n more about t h e model o r how t o u s e b e t t e r
u s e i t by c h e c k i n g out i t s documentation
# h t t p s : / / s c i k i t −l e a r n . o r g / s t a b l e / modules /
generated / sklearn . neural network . MLPClassifier
. html#s k l e a r n . n e u r a l n e t w o r k . M L P C l a s s i f i e r
# Play around with t h e model p a r a m e t e r s below
p r i n t ( ” T r a i n i n g a Complex ( Neural Network ) model
.”)
Artificial Intelligence 2024 20

# F e e l f r e e t o e x p e r i m e n t with d i f f e r e n t number
o f n e u r a l l a y e r s o r d i f f e r n t type o f n e u r o n s
per l a y e r
# Tips : more n e u r o n s o r more l a y e r s o f n e u r o n s
c r e a t e a more c o m p l i c a t e d model t h a t t a k e s
more time t o t r a i n and
# n e e d s a b i g g e r d a t a s e t , but i f you f i n d t h e
c o r r e c t c o m b i n a t i o n o f n e u r o n s and n e u r a l
l a y e r s and p r o v i d e a b i g enough t r a i n i n g
d a t a s e t can l e a d t o b e t t e r p e r f o r m a n c e

# one l a y e r o f 30 n e u r o n s
hidden layer sizes = (30)
# two l a y e r s o f 30 and 5 n e u r o n s r e s p e c t i v e l y
# h i d d e n l a y e r s i z e s = (30 , 5)

# The l e a r n i n g r a t e d e t e r m i n e s how f a s t we move

towards t h e o p t i m a l s o l u t i o n .
# A low l e a r n i n g r a t e w i l l c o n v e r g e s l o w l y , but a
l a r g e one might o v e r s h o o t .
l e a r n i n g r a t e = 0.0001

# The r e g u l a r i z a t i o n term aims t o p r e v e n t over−

f i t t i n g , and we can tweak i t s s t r e n g t h h e r e .
r e g u l a r i z a t i o n s t r e n g t h = 0.0001

# Train a n e u r a l network
l e a r n e r = M L P C l a s s i f i e r ( h i d d e n l a y e r s i z e s=
h i d d e n l a y e r s i z e s , l e a r n i n g r a t e i n i t=
learning rate ,
a l p h a=
regularization strength
, v e r b o s e=True ,
e a r l y s t o p p i n g=True ,
n i t e r n o c h a n g e =6,
a c t i v a t i o n =’ tanh ’ )
e l i f m o d e l c l a s s == ’LR ’ :
# Train a s i m p l e r L i n e a r L o g i s t i c R e g r e s s i o n
model
# l e a r n more about t h e model o r how t o u s e b e t t e r
u s e i t by c h e c k i n g out i t s documentation
# h t t p s : / / s c i k i t −l e a r n . o r g / s t a b l e / modules /
generated / sklearn . linear model .
L o g i s t i c R e g r e s s i o n . html#s k l e a r n . l i n e a r m o d e l .
LogisticRegression
Artificial Intelligence 2024 21

p r i n t ( ” T r a i n i n g a Simple ( L i n e a r L o g i s t i c
R e g r e s s i o n model ) ” )

# U s u a l l y t h e r e i s no r e a s o n t o change t h e
h y p e r p a r a m e t e r s o f such a s i m p l e model but
f i l l f r e e to experiment :
l e a r n e r = L o g i s t i c R e g r e s s i o n ( m a x i t e r =1000)

e l i f m o d e l c l a s s == ’DTC’ :
# Train a d e c i s i o n t r e e c l a s s i f i e r model
( ” T r a i n i n g a D e c i s i o n Tree C l a s s i f i e r model . ” )
learner = DecisionTreeClassifier ()

e l i f m o d e l c l a s s == ’DTR’ :
# Train a d e c i s i o n t r e e r e g r e s s o r model
p r i n t ( ” T r a i n i n g a D e c i s i o n Tree R e g r e s s o r model
.”)
learner = DecisionTreeRegressor ()

e l i f m o d e l c l a s s == ”RFC” :
# Train a random f o r e s t c l a s s i f i e r model
p r i n t ( ” T r a i n i n g a Random F o r e s t C l a s s i f i e r model
.”)
learner = RandomForestClassifier ()

e l i f m o d e l c l a s s == ’RFR’ :
# Train a random f o r e s t r e g r e s s o r model
( ” T r a i n i n g a Random F o r e s t R e g r e s s o r model . ” )
l e a r n e r = RandomForestRegressor ( )

e l i f m o d e l c l a s s == ’KMC’ :
# Train a K−means c l u s t e r i n g model
p r i n t ( ” T r a i n i n g a K−Means C l u s t e r i n g model . ” )
l e a r n e r = GaussianMixture ( n components = 2 ,
i n i t p a r a m s = ”kmeans ” )

else :
r a i s e A s s e r t i o n E r r o r ( ” Unknown model c l a s s ” )

s t a r t = time . time ( )
p r i n t ( ” S t a r t i n g t r a i n i n g phase . . . ” )

model = l e a r n e r . f i t ( data , t a r g e t s )

# Save t h e model i n a f i l e
Artificial Intelligence 2024 22

# C r e a t e html f i l e with e l i 5 p e r m u t a t i o n i m p o r t a n c e
v a l u e s / w e i g h t s when t r a i n e d
perm = PermutationImportance ( model ) . f i t ( data , t a r g e t s
)
html = e l i 5 . f o r m a t a s h t m l ( e l i 5 . e x p l a i n w e i g h t s ( perm ,
top = 3 0 0 ) )
with open ( ’ s i m p l e . html ’ , ’ w’ ) a s f :
f . w r i t e ( html )

j o b l i b . dump( model , m o d e l l o c a t i o n )
end = time . time ( )
p r i n t ( ’ The model was t r a i n e d i n ’ , ( end − s t a r t ) /
6 0 , ’ minutes . ’ )

def create state and actions vector representation (

p e r s p e c t i v e : PlayerPerspective , leader move : Optional [
Move ] , f o l l o w e r m o v e : O p t i o n a l [ Move ] ) −> l i s t [ i n t ] :
”””
This f u n c t i o n t a k e s a s i n p u t a P l a y e r P e r s p e c t i v e
v a r i a b l e , and t h e two moves o f l e a d e r and f o l l o w e r
,
and r e t u r n s a l i s t o f c o m p l e t e f e a t u r e r e p r e s e n t a t i o n
that contains a l l information
”””
player game state representation =
get state feature vector ( perspective )
leader move representation = get move feature vector (
leader move )
follower move representation =
get move feature vector ( follower move )

return player game state representation +

leader move representation +
follower move representation

d e f g e t o n e h o t e n c o d i n g o f c a r d s u i t ( c a r d s u i t : S u i t ) −>
l i s t [ int ] :
”””
T r a n s l a t i n g t h e s u i t o f a c a r d i n t o one hot v e c t o r
encoding of s i z e 4 .
”””
card suit one hot : l i s t [ int ]
i f c a r d s u i t == S u i t .HEARTS:
Artificial Intelligence 2024 23

card suit one hot = [0 , 0 , 0 , 1]

e l i f c a r d s u i t == S u i t . CLUBS :
card suit one hot = [0 , 0 , 1 , 0]
e l i f c a r d s u i t == S u i t . SPADES :
card suit one hot = [0 , 1 , 0 , 0]
e l i f c a r d s u i t == S u i t .DIAMONDS:
card suit one hot = [1 , 0 , 0 , 0]
else :
r a i s e V a l u e E r r o r ( ” S u i t o f c a r d was not found ! ” )

return card suit one hot

d e f g e t o n e h o t e n c o d i n g o f c a r d r a n k ( c a r d r a n k : Rank ) −>
l i s t [ int ] :
”””
T r a n s l a t i n g t h e rank o f a c a r d i n t o one hot v e c t o r
encoding of s i z e 13.
”””
card rank one hot : l i s t [ int ]
i f c a r d r a n k == Rank .ACE:
card rank one hot = [0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 ,
0 , 0 , 0 , 1]
e l i f c a r d r a n k == Rank .TWO:
card rank one hot = [0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 ,
0 , 0 , 1 , 0]
e l i f c a r d r a n k == Rank .THREE:
card rank one hot = [0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 ,
0 , 1 , 0 , 0]
e l i f c a r d r a n k == Rank .FOUR:
card rank one hot = [0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 ,
1 , 0 , 0 , 0]
e l i f c a r d r a n k == Rank . FIVE :
card rank one hot = [0 , 0 , 0 , 0 , 0 , 0 , 0 , 0 , 1 ,
0 , 0 , 0 , 0]
e l i f c a r d r a n k == Rank . SIX :
card rank one hot = [0 , 0 , 0 , 0 , 0 , 0 , 0 , 1 , 0 ,
0 , 0 , 0 , 0]
e l i f c a r d r a n k == Rank .SEVEN:
card rank one hot = [0 , 0 , 0 , 0 , 0 , 0 , 1 , 0 , 0 ,
0 , 0 , 0 , 0]
e l i f c a r d r a n k == Rank . EIGHT :
card rank one hot = [0 , 0 , 0 , 0 , 0 , 1 , 0 , 0 , 0 ,
0 , 0 , 0 , 0]
e l i f c a r d r a n k == Rank . NINE :
Artificial Intelligence 2024 24

card rank one hot = [0 , 0 , 0 , 0 , 1 , 0, 0, 0, 0,

0 , 0 , 0 , 0]
e l i f c a r d r a n k == Rank .TEN:
card rank one hot = [0 , 0 , 0 , 1 , 0 , 0, 0, 0, 0,
0 , 0 , 0 , 0]
e l i f c a r d r a n k == Rank .JACK:
card rank one hot = [0 , 0 , 1 , 0 , 0 , 0, 0, 0, 0,
0 , 0 , 0 , 0]
e l i f c a r d r a n k == Rank .QUEEN:
card rank one hot = [0 , 1 , 0 , 0 , 0 , 0, 0, 0, 0,
0 , 0 , 0 , 0]
e l i f c a r d r a n k == Rank . KING :
card rank one hot = [1 , 0 , 0 , 0 , 0 , 0, 0, 0, 0,
0 , 0 , 0 , 0]
else :
r a i s e A s s e r t i o n E r r o r ( ” Provided c a r d Rank d o e s not
exist !”)
return card rank one hot

d e f g e t m o v e f e a t u r e v e c t o r ( move : O p t i o n a l [ Move ] ) −> l i s t

[ int ] :
”””
In c a s e t h e r e i s n ’ t any move p r o v i d e d move t o
encode , we s t i l l need t o c r e a t e a ” padding”−”
m e a n i n g l e s s ” v e c t o r o f t h e same s i z e ,
f i l l e d with 0 s , s i n c e t h e ML models need t o
r e c e i v e i n p u t o f t h e same d i m e n s i o n a l i t y
always .
Otherwise , we c r e a t e a l l t h e i n f o r m a t i o n o f t h e
move i ) move type , i i ) p l a y e d c a r d rank and
i i i ) played card s u i t
t r a n s l a t e t h i s i n f o r m a t i o n i n t o one−hot v e c t o r s
r e s p e c t i v e l y , and c o n c a t e n a t e t h e s e v e c t o r s
i n t o one move f e a t u r e r e p r e s e n t a t i o n v e c t o r
”””

i f move i s None :
move type one hot encoding numpy array = [ 0 , 0 ,
0]
card rank one hot encoding numpy array = [0 , 0 ,
0 , 0]
card suit one hot encoding numpy array = [0 , 0 ,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
Artificial Intelligence 2024 25

else :
move type one hot encoding : l i s t [ int ]
# i n c a s e t h e move i s a m a r r i a g e move
i f move . i s m a r r i a g e ( ) :
move type one hot encoding = [0 , 0 , 1]
c a r d = move . q u e e n c a r d
# i n c a s e t h e move i s a trump exchange move
e l i f move . i s t r u m p e x c h a n g e ( ) :
move type one hot encoding = [0 , 1 , 0]
c a r d = move . j a c k
# i n c a s e i t i s a r e g u l a r move
else :
move type one hot encoding = [1 , 0 , 0]
c a r d = move . c a r d
move type one hot encoding numpy array =
move type one hot encoding
card rank one hot encoding numpy array =
g e t o n e h o t e n c o d i n g o f c a r d r a n k ( c a r d . rank )
card suit one hot encoding numpy array =
g e t o n e h o t e n c o d i n g o f c a r d s u i t ( card . s u i t )

return move type one hot encoding numpy array +

card rank one hot encoding numpy array +
card suit one hot encoding numpy array

def g e t s t a t e f e a t u r e v e c t o r ( perspective :
P l a y e r P e r s p e c t i v e ) −> l i s t [ i n t ] :
”””
This f u n c t i o n g a t h e r s a l l s u b j e c t i v e i n f o r m a t i o n
t h a t t h i s bot has a c c e s s to , t h a t can be used
t o d e c i d e i t s next move , i n c l u d i n g :
− points of this player ( int )
− p o i n t s o f t h e opponent ( i n t )
− pending p o i n t s o f t h i s p l a y e r ( i n t )
− pending p o i n t s o f opponent ( i n t )
− t h e trump s u i t (1− hot e n c o d i n g )
− phase o f game (1−hoy e n c o d i n g )
− talon s i ze ( int )
− i f t h i s p l a y e r i s l e a d e r (1− hot e n c o d i n g )
− What i s t h e s t a t u s o f each c a r d o f t h e deck (
where i t i s , o r i f i t s l o c a t i o n i s unknown )

Important : This f u n c t i o n s h o u l d not i n c l u d e t h e

move o f t h i s a g e n t .
Artificial Intelligence 2024 26

I t s h o u l d o n l y i n c l u d e any e a r l i e r a c t i o n s o f
other agents ( so the a c t i o n of the other agent
in case that i s the l e a d e r )
”””
# a l i s t of a l l the f e a t u r e s that c o n s i s t the s t a t e
f e a t u r e s e t , o f type np . ndarray
s t a t e f e a t u r e l i s t : l i s t [ int ] = [ ]

player score = perspective . get my score ()

# − points of this player ( int )
player points = player score . direct points
# − pending p o i n t s o f t h i s p l a y e r ( i n t )
player pending points = player score . pending points

# add t h e f e a t u r e s t o t h e f e a t u r e s e t
s t a t e f e a t u r e l i s t += [ p l a y e r p o i n t s ]
s t a t e f e a t u r e l i s t += [ p l a y e r p e n d i n g p o i n t s ]

opponents score = perspective . get opponent score ()

# − p o i n t s o f t h e opponent ( i n t )
opponents points = opponents score . d ir e c t p o in t s
# − pending p o i n t s o f opponent ( i n t )
opponents pending points = opponents score .
pending points

# add t h e f e a t u r e s t o t h e f e a t u r e s e t
s t a t e f e a t u r e l i s t += [ o p p o n e n t s p o i n t s ]
s t a t e f e a t u r e l i s t += [ o p p o n e n t s p e n d i n g p o i n t s ]

# − t h e trump s u i t (1− hot e n c o d i n g )

trump suit = perspective . get trump suit ()
trump suit one hot =
ge t on e ho t en co din g of ca rd su it ( trump suit )
# add t h i s f e a t u r e s t o t h e f e a t u r e s e t
s t a t e f e a t u r e l i s t += t r u m p s u i t o n e h o t

# − phase o f game (1− hot e n c o d i n g )

game phase encoded = [ 1 , 0 ] i f p e r s p e c t i v e . g e t p h a s e
( ) == GamePhase .TWO e l s e [ 0 , 1 ]
# add t h i s f e a t u r e s t o t h e f e a t u r e s e t
s t a t e f e a t u r e l i s t += game phase encoded

# − talon s i ze ( int )
talon size = perspective . get talon size ()
# add t h i s f e a t u r e s t o t h e f e a t u r e s e t
Artificial Intelligence 2024 27

s t a t e f e a t u r e l i s t += [ t a l o n s i z e ]

# − i f t h i s p l a y e r i s l e a d e r (1− hot e n c o d i n g )
i am leader = [0 , 1] i f perspective . am i leader ()
else [1 , 0]
# add t h i s f e a t u r e s t o t h e f e a t u r e s e t
s t a t e f e a t u r e l i s t += i a m l e a d e r

# g a t h e r a l l known deck i n f o r m a t i o n
hand cards = p e r s p e c t i v e . get hand ( ) . cards
trump card = p e r s p e c t i v e . g e t t r u m p c a r d ( )
won cards = p e r s p e c t i v e . g e t w o n c a r d s ( ) . g e t c a r d s ( )
opponent won cards = p e r s p e c t i v e .
get opponent won cards () . get cards ()
opponent known cards = p e r s p e c t i v e .
get known cards of opponent hand () . get cards ()
# each c a r d can e i t h e r be i ) on p l a y e r ’ s hand , i i ) on
p l a y e r ’ s won c a r d s , i i i ) on opponent ’ s hand , i v )
on opponent ’ s won c a r d s
# v ) be t h e trump c a r d o r v i ) i n an unknown p o s i t i o n
−> e i t h e r on t h e t a l o n o r on t h e opponent ’ s hand
# There a r e a l l d i f f e r e n t c a s e s r e g a r d i n g card ’ s
knowledge , and we r e p r e s e n t t h e s e 6 c a s e s u s i n g
one hot e n c o d i n g v e c t o r s a s s e e n b e l l o w .

deck knowledge in consecutive one hot encodings : l i s t

[ int ] = [ ]

f o r c a r d i n SchnapsenDeckGenerator ( ) . g e t i n i t i a l deck
() :
card knowledge in one hot encoding : l i s t [ int ]
# i ) on p l a y e r ’ s hand
i f card in hand cards :
card knowledge in one hot encoding = [0 , 0,
0 , 0 , 0 , 1]
# i i ) on p l a y e r ’ s won c a r d s
e l i f c a r d i n won cards :
card knowledge in one hot encoding = [0 , 0,
0 , 0 , 1 , 0]
# i i i ) on opponent ’ s hand
e l i f c a r d i n opponent known cards :
card knowledge in one hot encoding = [0 , 0,
0 , 1 , 0 , 0]
# i v ) on opponent ’ s won c a r d s
e l i f card in opponent won cards :
Artificial Intelligence 2024 28

card knowledge in one hot encoding = [0 , 0 ,

1 , 0 , 0 , 0]
# v ) be t h e trump c a r d
e l i f c a r d == trump card :
card knowledge in one hot encoding = [0 , 1 ,
0 , 0 , 0 , 0]
# v i ) i n an unknown p o s i t i o n a s i t i s i n v i s i b l e
t o t h i s p l a y e r . Thus , i t i s e i t h e r on t h e
t a l o n o r on t h e opponent ’ s hand
else :
card knowledge in one hot encoding = [1 , 0 ,
0 , 0 , 0 , 0]
# This l i s t e v e n t u a l l y d e v e l o p s t o one l o n g 1−
d i m e n s i o n a l numpy a r r a y o f shape ( 1 2 0 , )
deck knowledge in consecutive one hot encodings
+= c a r d k n o w l e d g e i n o n e h o t e n c o d i n g
# d e c k k n o w l e d g e f l a t t e n e d : np . ndarray = np .
concatenate ( tuple (
d e c k k n o w l e d g e i n o n e h o t e n c o d i n g ) , a x i s =0)

# add t h i s f e a t u r e s t o t h e f e a t u r e s e t
s t a t e f e a t u r e l i s t +=
deck knowledge in consecutive one hot encodings

return s t a t e f e a t u r e l i s t

8.2 Weight-ANOVA-test.py

import pandas a s pd
from s c i p y . s t a t s import f oneway

# C r e a t e a DataFrame with combined v a l u e s e x c l u d i n g K−

means C l u s t e r i n g
data = {
’ Model ’ :
[ ’ L o g i s t i c R e g r e s s i o n ’ ] ∗ 1 6 + [ ’ Neural Network ’ ] ∗ 1 6 +
[ ’ D e c i s i o n Tree C l a s s i f i c a t i o n ’ ] ∗ 1 6 + [ ’ D e c i s i o n
Tree R e g r e s s i o n ’ ] ∗ 1 6 + [ ’ Random F o r e s t
C l a s s i f i c a t i o n ’ ] ∗ 1 6 + [ ’ Random F o r e s t R e g r e s s i o n
’]∗16 ,
’ Feature ’ :
[ ’ t a l o n s i z e ’ , ’ p o i n t s o f a p l a y e r ’ , ’ opponent p o i n t s
’ , ’ p l a y e r i s l e a d e r ’ , ’ opponent i s l e a d e r ’ , ’ game
phase 2 ’ , ’ game phase 1 ’ , ’ Ten H e a r t s opponents
hand ’ , ’ Ace Spades on opponents hand ’ , ’ Ace H e a r t s
Artificial Intelligence 2024 29

on opponents hand ’ , ’ Ace Clubs on opponents hand

’ , ’ Ten Spades p l a y e r s hand ’ , ’ Ace Clubs on
p l a y e r s hand ’ , ’ Ace Spades on p l a y e r s hand ’ , ’
trump s u i t diamonds ’ , ’ Ace Diamonds on opponents
hand ’ ] ∗ 6 ,
’ Importance ’ :
[0.0929 , 0.0827 , 0.0620 , 0.0581 , 0.0527 , 0.0367 ,
0.0329 , 0.0160 , 0.0060 , 0.0057 , 0.0054 , 0.0047 ,
0.0046 , 0.0045 , 0.0044 , 0.0044 ,0.1114 , 0.0896 ,
0.0291 , 0.0265 , 0.0246 , 0.0226 , 0.0216 , 0.0121 ,
0.0109 , 0.0080 , 0.0075 , 0.0074 , 0.0070 , 0.0063 ,
0.0060 , 0.0056 , 0.3124 , 0.3124 , 0.2149 , 0.0887 ,
0.0649 , 0.0638 , 0.0564 , 0.0482 , 0.0470 , 0.0461 ,
0.0422 , 0.0391 , 0.0338 , 0.0333 , 0.0313 ,
0.0311 ,1.2613 , 1.2568 , 0.8404 , 0.3543 , 0.2623 ,
0.2597 , 0.2280 , 0.1934 , 0.1850 , 0.1829 , 0.1699 ,
0.1547 , 0.1351 , 0.1329 , 0.1255 , 0.1229 ,0.0494 ,
0.0414 , 0.0024 , 0.0024 , 0.0022 , 0.0018 , 0.0018 ,
0.0018 , 0.0015 , 0.0013 , 0.0013 , 0.0012 , 0.0011 ,
0.0010 , 0.0008 , 0.0008 , 0.5990 , 0.5803 , 0.1440 ,
0.1284 , 0.1078 , 0.0696 , 0.0688 , 0.0641 , 0.0600 ,
0.0579 , 0.0477 , 0.0476 , 0.0427 , 0.0390 , 0.0370 ,
0.0360 ,]
}

d f = pd . DataFrame ( data )

# ANOVA
f s t a t i s t i c , p v a l u e = f oneway ( ∗ [ group [ ” Importance ” ] f o r
name , group i n d f . groupby ( ” Model ” ) ] )

p r i n t ( ”F− s t a t i s t i c : ” , f s t a t i s t i c )
p r i n t ( ”P−v a l u e : ” , p v a l u e )
Artificial Intelligence 2024 30

8.3 Full Original Table of Permutation Importance Logistic

Regression Top weight, Neural Network bottom weight

LR model 1.747 minutes

NN 5.456 min
Weight Feature
0.0929 ± 0.0015 x10
0.1114 ± 0.0019 x2
0.0827 ± 0.0009 x0
0.0896 ± 0.0017 x0
0.0620 ± 0.0015 x2
0.0291 ± 0.0012 x7
0.0581 ± 0.0008 x11
0.0265 ± 0.0006 x10
0.0527 ± 0.0011 x12
0.0246 ± 0.0007 x4
0.0367 ± 0.0013 x9
0.0226 ± 0.0010 x5
0.0329 ± 0.0014 x8
0.0216 ± 0.0008 x6
0.0160 ± 0.0011 x153
0.0121 ± 0.0007 x11
0.0060 ± 0.0014 x102
0.0109 ± 0.0006 x12
0.0057 ± 0.0007 x42
0.0080 ± 0.0004 x42
0.0054 ± 0.0006 x72
0.0075 ± 0.0004 x102
0.0047 ± 0.0010 x100
0.0074 ± 0.0003 x132
0.0046 ± 0.0006 x73
0.0070 ± 0.0009 x153
0.0045 ± 0.0004 x103
0.0063 ± 0.0005 x130
0.0044 ± 0.0004 x13
0.0060 ± 0.0005 x108
0.0044 ± 0.0004 x132
0.0056 ± 0.0003 x78
0.0044 ± 0.0007 x43
0.0055 ± 0.0004 x72
0.0043 ± 0.0005 x40
0.0053 ± 0.0004 x96
0.0040 ± 0.0006 x121
0.0052 ± 0.0006 x36
Artificial Intelligence 2024 31

0.0040 ± 0.0009 x91

0.0052 ± 0.0003 x70
0.0038 ± 0.0006 x61
0.0051 ± 0.0003 x100
0.0038 ± 0.0006 x109
0.0048 ± 0.0003 x18
0.0038 ± 0.0005 x70
0.0046 ± 0.0004 x8
0.0037 ± 0.0006 x31
0.0045 ± 0.0004 x48
0.0037 ± 0.0005 x130
0.0038 ± 0.0002 x40
0.0036 ± 0.0006 x55
0.0038 ± 0.0004 x66
0.0033 ± 0.0006 x19
0.0037 ± 0.0002 x24
0.0033 ± 0.0003 x85
0.0037 ± 0.0005 x84
0.0032 ± 0.0002 x127
0.0036 ± 0.0007 x126
0.0032 ± 0.0008 x37
0.0034 ± 0.0004 x94
0.0031 ± 0.0007 x97
0.0033 ± 0.0004 x114
0.0030 ± 0.0009 x67
0.0032 ± 0.0006 x124
0.0029 ± 0.0005 x79
0.0030 ± 0.0005 x90
0.0028 ± 0.0008 x49
0.0029 ± 0.0002 x16
0.0027 ± 0.0008 x115
0.0029 ± 0.0004 x64
0.0024 ± 0.0009 x25
0.0028 ± 0.0008 x60
0.0016 ± 0.0003 x124
0.0028 ± 0.0004 x30
0.0015 ± 0.0006 x48
0.0027 ± 0.0005 x76
0.0013 ± 0.0002 x34
0.0027 ± 0.0009 x120
0.0013 ± 0.0003 x126
0.0025 ± 0.0002 x34
0.0012 ± 0.0003 x94
0.0025 ± 0.0006 x97
Artificial Intelligence 2024 32

0.0010 ± 0.0006 x53

0.0025 ± 0.0003 x73
0.0010 ± 0.0004 x35
0.0025 ± 0.0006 x54
0.0009 ± 0.0003 x169
0.0025 ± 0.0006 x46
0.0009 ± 0.0005 x18
0.0024 ± 0.0004 x118
0.0009 ± 0.0003 x108
0.0023 ± 0.0002 x112
0.0009 ± 0.0003 x171
0.0022 ± 0.0003 x88
0.0009 ± 0.0002 x133
0.0022 ± 0.0004 x109
0.0008 ± 0.0006 x131
0.0021 ± 0.0007 x52
0.0008 ± 0.0004 x78
0.0020 ± 0.0007 x131
0.0008 ± 0.0002 x1
0.0019 ± 0.0003 x58
0.0008 ± 0.0003 x36
0.0019 ± 0.0003 x13
0.0008 ± 0.0004 x89
0.0019 ± 0.0002 x106
0.0007 ± 0.0003 x134
0.0019 ± 0.0006 x22
0.0007 ± 0.0005 x119
0.0019 ± 0.0005 x9
0.0006 ± 0.0001 x64
0.0018 ± 0.0002 x28
0.0006 ± 0.0003 x170
0.0018 ± 0.0003 x49
0.0006 ± 0.0002 x172
0.0017 ± 0.0004 x171
0.0006 ± 0.0003 x15
0.0017 ± 0.0005 x121
0.0006 ± 0.0006 x96
0.0016 ± 0.0007 x127
0.0005 ± 0.0004 x77
0.0015 ± 0.0004 x95
0.0005 ± 0.0002 x111
0.0015 ± 0.0004 x103
0.0005 ± 0.0005 x65
0.0014 ± 0.0004 x65
Artificial Intelligence 2024 33

0.0005 ± 0.0004 x95

0.0014 ± 0.0002 x43
0.0005 ± 0.0003 x157
0.0014 ± 0.0003 x19
0.0005 ± 0.0003 x29
0.0014 ± 0.0004 x31
0.0005 ± 0.0004 x45
0.0013 ± 0.0003 x3
0.0004 ± 0.0003 x135
0.0013 ± 0.0002 x129
0.0004 ± 0.0002 x159
0.0013 ± 0.0004 x107
0.0004 ± 0.0003 x83
0.0012 ± 0.0003 x33
0.0004 ± 0.0003 x105
0.0012 ± 0.0002 x91
0.0004 ± 0.0004 x66
0.0011 ± 0.0005 x85
0.0004 ± 0.0004 x158
0.0011 ± 0.0003 x169
0.0004 ± 0.0003 x57
0.0011 ± 0.0002 x170
0.0004 ± 0.0003 x23
0.0011 ± 0.0005 x115
0.0003 ± 0.0002 x3
0.0011 ± 0.0002 x82
0.0003 ± 0.0001 x84
0.0011 ± 0.0004 x105
0.0003 ± 0.0003 x148
0.0010 ± 0.0002 x56
0.0003 ± 0.0004 x71
0.0010 ± 0.0003 x79
0.0002 ± 0.0004 x156
0.0010 ± 0.0002 x26
0.0002 ± 0.0003 x107
0.0009 ± 0.0005 x113
0.0002 ± 0.0003 x69
0.0009 ± 0.0004 x47
0.0002 ± 0.0001 x63
0.0009 ± 0.0002 x1
0.0002 ± 0.0003 x168
0.0009 ± 0.0005 x59
0.0001 ± 0.0003 x123
0.0009 ± 0.0004 x101
Artificial Intelligence 2024 34

0.0001 ± 0.0001 x62

0.0008 ± 0.0002 x135
0.0001 ± 0.0001 x14
0.0008 ± 0.0002 x37
0.0001 ± 0.0003 x101
0.0008 ± 0.0004 x51
0.0001 ± 0.0001 x32
0.0008 ± 0.0005 x67
0.0001 ± 0.0001 x26
0.0008 ± 0.0004 x119
0.0001 ± 0.0002 x17
0.0008 ± 0.0004 x45
0.0001 ± 0.0001 x7
0.0008 ± 0.0003 x25
0.0001 ± 0.0005 x59
0.0008 ± 0.0003 x69
0.0001 ± 0.0002 x150
0.0007 ± 0.0004 x168
0.0001 ± 0.0002 x27
0.0007 ± 0.0003 x41
0.0001 ± 0.0003 x39
0.0007 ± 0.0005 x150
0.0001 ± 0.0002 x113
0.0007 ± 0.0002 x159
0.0001 ± 0.0003 x93
0.0007 ± 0.0003 x71
0.0001 ± 0.0001 x104
0.0007 ± 0.0002 x68
0.0001 ± 0.0002 x125
0.0007 ± 0.0004 x39
0.0001 ± 0.0001 x52
0.0007 ± 0.0003 x29
0.0001 ± 0.0001 x20
0.0006 ± 0.0002 x172
0.0001 ± 0.0001 x136
0.0006 ± 0.0003 x50
0.0001 ± 0.0001 x106
0.0006 ± 0.0004 x81
0.0001 ± 0.0001 x122
0.0006 ± 0.0001 x93
0.0000 ± 0.0000 x92
0.0006 ± 0.0004 x44
0.0000 ± 0.0002 x86
0.0006 ± 0.0003 x110
Artificial Intelligence 2024 35

0.0000 ± 0.0001 x58

0.0006 ± 0.0003 x111
0.0000 ± 0.0001 x22
0.0006 ± 0.0002 x17
0.0000 ± 0.0002 x139
0.0006 ± 0.0002 x53
0.0000 ± 0.0002 x50
0.0005 ± 0.0005 x152
0.0000 ± 0.0003 x114
0.0005 ± 0.0003 x89
0.0000 ± 0.0001 x138
0.0005 ± 0.0002 x87
0.0000 ± 0.0002 x118
0.0005 ± 0.0003 x77
0.0000 ± 0.0003 x110
0.0005 ± 0.0004 x21
0.0000 ± 0.0001 x98
0.0005 ± 0.0001 x158
0.0000 ± 0.0001 x30
0.0005 ± 0.0003 x23
0.0000 ± 0.0001 x54
0.0005 ± 0.0003 x99
0.0000 ± 0.0000 x82
0.0005 ± 0.0001 x134
0.0000 ± 0.0003 x81
0.0005 ± 0.0003 x104
0.0000 ± 0.0003 x75
0.0005 ± 0.0002 x38
0.0000 ± 0.0001 x46
0.0004 ± 0.0003 x75
0.0000 ± 0.0000 x33
0.0004 ± 0.0003 x149
0.0000 ± 0.0003 x47
0.0004 ± 0.0003 x123
0.0000 ± 0.0000 x128
0.0004 ± 0.0002 x35
0 ± 0.0000 x142
0.0004 ± 0.0006 x55
0 ± 0.0000 x163
0.0004 ± 0.0002 x117
0 ± 0.0000 x160
0.0004 ± 0.0002 x27
0 ± 0.0000 x141
0.0004 ± 0.0002 x61
Artificial Intelligence 2024 36

0 ± 0.0000 x140
0.0004 ± 0.0001 x15
0 ± 0.0000 x161
0.0004 ± 0.0002 x133
0 ± 0.0000 x143
0.0004 ± 0.0003 x139
0 ± 0.0000 x162
0.0003 ± 0.0003 x63
0 ± 0.0000 x155
0.0003 ± 0.0001 x151
0 ± 0.0000 x154
0.0003 ± 0.0002 x125
0 ± 0.0000 x165
0.0003 ± 0.0002 x32
0 ± 0.0000 x164
0.0003 ± 0.0003 x138
0 ± 0.0000 x166
0.0003 ± 0.0002 x83
0 ± 0.0000 x68
0.0003 ± 0.0003 x57
0 ± 0.0000 x145
0.0002 ± 0.0001 x20
0 ± 0.0000 x167
0.0002 ± 0.0002 x86
0 ± 0.0000 x147
0.0002 ± 0.0002 x116
0 ± 0.0000 x146
0.0002 ± 0.0002 x128
0 ± 0.0000 x144
0.0002 ± 0.0002 x148
-0.0000 ± x149
0.0001
0.0002 ± 0.0003 x136
-0.0000 ± x76
0.0001
0.0002 ± 0.0001 x80
-0.0000 ± x74
0.0002
0.0002 ± 0.0002 x74
-0.0000 ± x80
0.0002
0.0002 ± 0.0003 x137
-0.0000 ± x90
0.0001
Artificial Intelligence 2024 37

0.0002 ± 0.0002 x98

-0.0000 ± x41
0.0004
0.0002 ± 0.0001 x122
-0.0000 ± x120
0.0001
0.0002 ± 0.0001 x92
-0.0000 ± x152
0.0001
0.0002 ± 0.0002 x14
-0.0000 ± x112
0.0001
0.0002 ± 0.0004 x157
-0.0000 ± x87
0.0002
0.0001 ± 0.0003 x156
-0.0000 ± x21
0.0002
0.0001 ± 0.0003 x62
-0.0000 ± x5
0.0002
0 ± 0.0000 x147
-0.0000 ± x56
0.0001
0 ± 0.0000 x146
-0.0000 ± x38
0.0001
0 ± 0.0000 x167
-0.0000 ± x116
0.0001
0 ± 0.0000 x166
-0.0000 ± x16
0.0001
0 ± 0.0000 x165
-0.0000 ± x44
0.0001
0 ± 0.0000 x164
-0.0000 ± x137
0.0000
0 ± 0.0000 x163
-0.0001 ± x60
0.0002
0 ± 0.0000 x162
-0.0001 ± x117
0.0001
Artificial Intelligence 2024 38

0 ± 0.0000 x161
-0.0001 ± x88
0.0001
0 ± 0.0000 x160
-0.0001 ± x28
0.0003
0 ± 0.0000 x143
-0.0001 ± x51
0.0002
0 ± 0.0000 x144
-0.0001 ± x4
0.0002
0 ± 0.0000 x140
-0.0001 ± x6
0.0001
0 ± 0.0000 x155
-0.0001 ± x151
0.0002
0 ± 0.0000 x154
-0.0001 ± x129
0.0002
0 ± 0.0000 x145
-0.0001 ± x24
0.0003
0 ± 0.0000 x142
-0.0001 ± x99
0.0004
0 ± 0.0000 x141
Table 8:

8.4 Full Original Table of Permutation Importance Logistic

Decision Tree Classifier Top weight, Decision Tree Regressor
bottom weight

DTC 1.510 min

DTR 1.311 min
Weight Feature
0.3124 ± 0.0028 x2
1.2613 ± 0.0054 x0
0.3124 ± 0.0012 x0
1.2568 ± 0.0099 x2
0.2149 ± 0.0013 x11
0.8404 ± 0.0061 x12
Artificial Intelligence 2024 39

0.0887 ± 0.0010 x102

0.3543 ± 0.0036 x102
0.0649 ± 0.0005 x72
0.2623 ± 0.0016 x72
0.0638 ± 0.0006 x132
0.2597 ± 0.0045 x132
0.0564 ± 0.0007 x37
0.2280 ± 0.0023 x37
0.0482 ± 0.0007 x5
0.1934 ± 0.0022 x5
0.0470 ± 0.0005 x42
0.1850 ± 0.0026 x42
0.0461 ± 0.0008 x6
0.1829 ± 0.0016 x6
0.0422 ± 0.0010 x67
0.1699 ± 0.0026 x67
0.0391 ± 0.0005 x4
0.1547 ± 0.0016 x4
0.0338 ± 0.0002 x7
0.1351 ± 0.0013 x7
0.0333 ± 0.0004 x73
0.1329 ± 0.0016 x73
0.0313 ± 0.0005 x43
0.1255 ± 0.0012 x148
0.0311 ± 0.0007 x148
0.1229 ± 0.0034 x13
0.0308 ± 0.0006 x13
0.1213 ± 0.0012 x43
0.0253 ± 0.0005 x48
0.1023 ± 0.0012 x48
0.0243 ± 0.0005 x139
0.0974 ± 0.0008 x139
0.0239 ± 0.0005 x66
0.0915 ± 0.0010 x66
0.0230 ± 0.0003 x126
0.0912 ± 0.0024 x126
0.0228 ± 0.0003 x97
0.0907 ± 0.0017 x97
0.0223 ± 0.0004 x127
0.0865 ± 0.0017 x121
0.0215 ± 0.0003 x121
0.0860 ± 0.0022 x18
0.0211 ± 0.0004 x18
0.0843 ± 0.0011 x127
Artificial Intelligence 2024 40

0.0208 ± 0.0003 x108

0.0834 ± 0.0027 x96
0.0204 ± 0.0001 x49
0.0829 ± 0.0009 x108
0.0203 ± 0.0002 x96
0.0818 ± 0.0016 x78
0.0200 ± 0.0004 x103
0.0814 ± 0.0009 x49
0.0200 ± 0.0003 x78
0.0796 ± 0.0004 x103
0.0192 ± 0.0005 x36
0.0770 ± 0.0013 x60
0.0188 ± 0.0004 x54
0.0764 ± 0.0011 x24
0.0186 ± 0.0004 x60
0.0763 ± 0.0014 x36
0.0183 ± 0.0005 x31
0.0744 ± 0.0021 x54
0.0183 ± 0.0003 x24
0.0716 ± 0.0013 x31
0.0173 ± 0.0003 x84
0.0709 ± 0.0007 x84
0.0170 ± 0.0004 x120
0.0692 ± 0.0017 x115
0.0170 ± 0.0005 x115
0.0674 ± 0.0027 x30
0.0168 ± 0.0003 x90
0.0662 ± 0.0020 x61
0.0168 ± 0.0003 x19
0.0660 ± 0.0019 x120
0.0167 ± 0.0002 x10
0.0659 ± 0.0024 x90
0.0167 ± 0.0003 x151
0.0657 ± 0.0009 x150
0.0167 ± 0.0005 x30
0.0653 ± 0.0007 x19
0.0163 ± 0.0005 x109
0.0640 ± 0.0014 x114
0.0162 ± 0.0005 x91
0.0639 ± 0.0018 x151
0.0158 ± 0.0004 x150
0.0636 ± 0.0019 x109
0.0158 ± 0.0005 x114
0.0630 ± 0.0007 x100
Artificial Intelligence 2024 41

0.0156 ± 0.0004 x100

0.0615 ± 0.0009 x91
0.0156 ± 0.0003 x55
0.0608 ± 0.0010 x55
0.0155 ± 0.0004 x61
0.0603 ± 0.0017 x138
0.0152 ± 0.0002 x138
0.0601 ± 0.0009 x10
0.0149 ± 0.0002 x25
0.0585 ± 0.0012 x25
0.0141 ± 0.0005 x149
0.0580 ± 0.0019 x149
0.0136 ± 0.0003 x152
0.0551 ± 0.0006 x40
0.0135 ± 0.0004 x40
0.0545 ± 0.0020 x136
0.0131 ± 0.0005 x136
0.0543 ± 0.0012 x152
0.0118 ± 0.0001 x168
0.0492 ± 0.0011 x85
0.0118 ± 0.0004 x85
0.0475 ± 0.0008 x168
0.0115 ± 0.0002 x130
0.0453 ± 0.0005 x137
0.0112 ± 0.0005 x79
0.0453 ± 0.0022 x130
0.0112 ± 0.0004 x137
0.0433 ± 0.0008 x79
0.0102 ± 0.0002 x70
0.0418 ± 0.0007 x70
0.0097 ± 0.0002 x133
0.0395 ± 0.0019 x133
0.0095 ± 0.0003 x21
0.0387 ± 0.0011 x21
0.0087 ± 0.0002 x171
0.0355 ± 0.0007 x171
0.0087 ± 0.0002 x51
0.0351 ± 0.0014 x51
0.0086 ± 0.0002 x159
0.0351 ± 0.0008 x159
0.0083 ± 0.0002 x53
0.0334 ± 0.0013 x53
0.0082 ± 0.0003 x170
0.0326 ± 0.0007 x170
Artificial Intelligence 2024 42

0.0081 ± 0.0002 x46

0.0322 ± 0.0023 x105
0.0081 ± 0.0002 x111
0.0321 ± 0.0007 x46
0.0080 ± 0.0003 x105
0.0311 ± 0.0008 x111
0.0077 ± 0.0002 x17
0.0311 ± 0.0016 x93
0.0075 ± 0.0003 x81
0.0307 ± 0.0010 x17
0.0075 ± 0.0001 x169
0.0300 ± 0.0006 x83
0.0074 ± 0.0004 x93
0.0298 ± 0.0004 x65
0.0074 ± 0.0001 x75
0.0295 ± 0.0007 x117
0.0074 ± 0.0001 x41
0.0294 ± 0.0005 x99
0.0073 ± 0.0002 x83
0.0293 ± 0.0011 x45
0.0073 ± 0.0002 x99
0.0292 ± 0.0006 x81
0.0072 ± 0.0003 x35
0.0290 ± 0.0004 x41
0.0072 ± 0.0001 x101
0.0285 ± 0.0009 x101
0.0072 ± 0.0002 x23
0.0285 ± 0.0010 x169
0.0072 ± 0.0003 x125
0.0284 ± 0.0008 x35
0.0072 ± 0.0005 x117
0.0283 ± 0.0008 x23
0.0072 ± 0.0002 x29
0.0279 ± 0.0002 x75
0.0072 ± 0.0002 x65
0.0277 ± 0.0011 x125
0.0070 ± 0.0003 x95
0.0276 ± 0.0012 x77
0.0068 ± 0.0003 x77
0.0276 ± 0.0009 x89
0.0067 ± 0.0003 x45
0.0275 ± 0.0015 x29
0.0067 ± 0.0002 x89
0.0272 ± 0.0010 x95
Artificial Intelligence 2024 43

0.0065 ± 0.0003 x107

0.0267 ± 0.0011 x107
0.0065 ± 0.0002 x123
0.0257 ± 0.0015 x123
0.0064 ± 0.0002 x16
0.0252 ± 0.0005 x27
0.0064 ± 0.0002 x158
0.0252 ± 0.0007 x124
0.0063 ± 0.0001 x76
0.0249 ± 0.0009 x47
0.0063 ± 0.0002 x87
0.0248 ± 0.0006 x76
0.0063 ± 0.0003 x27
0.0248 ± 0.0008 x16
0.0061 ± 0.0003 x47
0.0247 ± 0.0005 x172
0.0061 ± 0.0003 x172
0.0246 ± 0.0011 x158
0.0061 ± 0.0002 x124
0.0244 ± 0.0009 x94
0.0060 ± 0.0002 x94
0.0244 ± 0.0007 x87
0.0060 ± 0.0001 x33
0.0238 ± 0.0009 x113
0.0059 ± 0.0002 x57
0.0235 ± 0.0009 x119
0.0058 ± 0.0001 x113
0.0233 ± 0.0010 x63
0.0058 ± 0.0003 x119
0.0231 ± 0.0006 x34
0.0058 ± 0.0001 x63
0.0228 ± 0.0010 x71
0.0057 ± 0.0002 x71
0.0224 ± 0.0006 x33
0.0057 ± 0.0001 x34
0.0222 ± 0.0007 x57
0.0056 ± 0.0001 x64
0.0220 ± 0.0009 x64
0.0054 ± 0.0002 x129
0.0219 ± 0.0014 x129
0.0048 ± 0.0002 x69
0.0198 ± 0.0007 x69
0.0048 ± 0.0002 x58
0.0196 ± 0.0010 x15
Artificial Intelligence 2024 44

0.0047 ± 0.0002 x39

0.0188 ± 0.0009 x106
0.0047 ± 0.0001 x15
0.0188 ± 0.0007 x39
0.0047 ± 0.0001 x106
0.0184 ± 0.0010 x131
0.0045 ± 0.0002 x59
0.0182 ± 0.0005 x157
0.0045 ± 0.0001 x88
0.0182 ± 0.0007 x58
0.0043 ± 0.0002 x22
0.0181 ± 0.0005 x88
0.0043 ± 0.0002 x112
0.0172 ± 0.0008 x112
0.0043 ± 0.0002 x131
0.0172 ± 0.0006 x20
0.0043 ± 0.0001 x20
0.0170 ± 0.0006 x59
0.0043 ± 0.0001 x157
0.0169 ± 0.0010 x82
0.0042 ± 0.0002 x52
0.0167 ± 0.0004 x22
0.0041 ± 0.0001 x82
0.0166 ± 0.0010 x156
0.0040 ± 0.0002 x156
0.0164 ± 0.0006 x52
0.0038 ± 0.0001 x104
0.0147 ± 0.0006 x104
0.0036 ± 0.0001 x153
0.0146 ± 0.0003 x98
0.0035 ± 0.0002 x118
0.0140 ± 0.0005 x118
0.0035 ± 0.0001 x98
0.0138 ± 0.0003 x38
0.0035 ± 0.0001 x38
0.0138 ± 0.0005 x28
0.0035 ± 0.0002 x28
0.0128 ± 0.0006 x80
0.0032 ± 0.0001 x50
0.0127 ± 0.0007 x50
0.0032 ± 0.0001 x80
0.0126 ± 0.0002 x110
0.0031 ± 0.0001 x110
0.0113 ± 0.0001 x14
Artificial Intelligence 2024 45

0.0028 ± 0.0001 x14

0.0110 ± 0.0004 x86
0.0028 ± 0.0001 x74
0.0108 ± 0.0008 x32
0.0028 ± 0.0002 x86
0.0104 ± 0.0003 x74
0.0027 ± 0.0001 x32
0.0101 ± 0.0005 x62
0.0025 ± 0.0001 x62
0.0100 ± 0.0004 x68
0.0025 ± 0.0002 x68
0.0097 ± 0.0001 x44
0.0024 ± 0.0001 x128
0.0096 ± 0.0003 x128
0.0024 ± 0.0001 x92
0.0096 ± 0.0005 x92
0.0023 ± 0.0001 x56
0.0093 ± 0.0006 x56
0.0023 ± 0.0001 x44
0.0087 ± 0.0007 x153
0.0021 ± 0.0001 x26
0.0082 ± 0.0002 x26
0.0020 ± 0.0001 x116
0.0079 ± 0.0005 x116
0.0018 ± 0.0001 x122
0.0073 ± 0.0006 x122
0.0014 ± 0.0001 x135
0.0072 ± 0.0005 x9
0.0012 ± 0.0000 x3
0.0056 ± 0.0002 x135
0.0011 ± 0.0001 x1
0.0050 ± 0.0005 x3
0.0007 ± 0.0001 x8
0.0043 ± 0.0003 x1
0.0004 ± 0.0000 x134
0.0019 ± 0.0002 x134
0.0003 ± 0.0000 x9
0.0010 ± 0.0002 x8
0 ± 0.0000 x147
0 ± 0.0000 x166
0 ± 0.0000 x162
0 ± 0.0000 x165
0 ± 0.0000 x167
0 ± 0.0000 x164
Artificial Intelligence 2024 46

0 ± 0.0000 x166
0 ± 0.0000 x11
0 ± 0.0000 x165
0 ± 0.0000 x163
0 ± 0.0000 x164
0 ± 0.0000 x167
0 ± 0.0000 x163
0 ± 0.0000 x162
0 ± 0.0000 x161
0 ± 0.0000 x141
0 ± 0.0000 x146
0 ± 0.0000 x161
0 ± 0.0000 x12
0 ± 0.0000 x160
0 ± 0.0000 x160
0 ± 0.0000 x155
0 ± 0.0000 x141
0 ± 0.0000 x154
0 ± 0.0000 x142
0 ± 0.0000 x147
0 ± 0.0000 x143
0 ± 0.0000 x146
0 ± 0.0000 x155
0 ± 0.0000 x145
0 ± 0.0000 x154
0 ± 0.0000 x144
0 ± 0.0000 x144
0 ± 0.0000 x143
0 ± 0.0000 x145
0 ± 0.0000 x142
0 ± 0.0000 x140
0 ± 0.0000 x140

RFC 70.628 min

RFR 40.036 min
Weight Feature
Weight Feature
0.0494 ± 0.0016 x2
0.5990 ± 0.0041 x0
Artificial Intelligence 2024 47

0.0414 ± 0.0007 x0
0.5803 ± 0.0037 x2
0.0024 ± 0.0000 x10
0.1440 ± 0.0008 x12
0.0024 ± 0.0001 x102
0.1284 ± 0.0010 x11
0.0022 ± 0.0002 x72
0.1078 ± 0.0008 x102
0.0018 ± 0.0001 x132
0.0696 ± 0.0007 x132
0.0018 ± 0.0001 x37
0.0688 ± 0.0005 x72
0.0018 ± 0.0002 x99
0.0641 ± 0.0006 x42
0.0015 ± 0.0001 x42
0.0600 ± 0.0007 x148
0.0013 ± 0.0001 x97
0.0579 ± 0.0006 x5
0.0013 ± 0.0001 x39
0.0477 ± 0.0005 x6
0.0012 ± 0.0001 x123
0.0476 ± 0.0007 x37
0.0011 ± 0.0001 x11
0.0427 ± 0.0005 x4
0.0010 ± 0.0001 x127
0.0390 ± 0.0003 x97
0.0008 ± 0.0001 x125
0.0370 ± 0.0005 x7
0.0008 ± 0.0001 x73
0.0360 ± 0.0004 x67
0.0008 ± 0.0001 x101
0.0325 ± 0.0001 x48
0.0007 ± 0.0001 x67
0.0319 ± 0.0004 x139
0.0007 ± 0.0001 x41
0.0301 ± 0.0001 x78
0.0007 ± 0.0000 x93
0.0296 ± 0.0002 x108
0.0005 ± 0.0001 x69
0.0289 ± 0.0003 x43
0.0005 ± 0.0001 x153
0.0287 ± 0.0003 x18
0.0005 ± 0.0000 x18
0.0271 ± 0.0001 x73
Artificial Intelligence 2024 48

0.0004 ± 0.0001 x71

0.0264 ± 0.0002 x127
0.0004 ± 0.0001 x148
0.0239 ± 0.0001 x13
0.0004 ± 0.0000 x108
0.0236 ± 0.0001 x10
0.0004 ± 0.0000 x100
0.0229 ± 0.0003 x126
0.0004 ± 0.0000 x95
0.0218 ± 0.0001 x138
0.0004 ± 0.0000 x40
0.0218 ± 0.0002 x103
0.0004 ± 0.0000 x78
0.0213 ± 0.0004 x100
0.0004 ± 0.0000 x70
0.0211 ± 0.0002 x96
0.0003 ± 0.0001 x6
0.0201 ± 0.0002 x66
0.0003 ± 0.0000 x130
0.0188 ± 0.0001 x36
0.0003 ± 0.0000 x48
0.0186 ± 0.0002 x150
0.0003 ± 0.0001 x13
0.0185 ± 0.0002 x114
0.0003 ± 0.0001 x138
0.0178 ± 0.0003 x84
0.0003 ± 0.0001 x12
0.0177 ± 0.0003 x40
0.0003 ± 0.0001 x43
0.0174 ± 0.0002 x54
0.0003 ± 0.0000 x5
0.0173 ± 0.0002 x149
0.0002 ± 0.0000 x103
0.0173 ± 0.0001 x121
0.0002 ± 0.0001 x121
0.0171 ± 0.0001 x24
0.0002 ± 0.0000 x63
0.0169 ± 0.0001 x19
0.0001 ± 0.0000 x65
0.0168 ± 0.0001 x49
0.0001 ± 0.0000 x4
0.0168 ± 0.0001 x31
0.0001 ± 0.0000 x61
0.0165 ± 0.0003 x61
Artificial Intelligence 2024 49

0.0001 ± 0.0000 x35

0.0165 ± 0.0002 x55
0.0001 ± 0.0000 x129
0.0165 ± 0.0001 x60
0.0001 ± 0.0000 x158
0.0161 ± 0.0002 x120
0.0001 ± 0.0000 x126
0.0159 ± 0.0001 x109
0.0001 ± 0.0000 x7
0.0158 ± 0.0001 x91
0.0001 ± 0.0000 x91
0.0156 ± 0.0002 x151
0.0001 ± 0.0000 x31
0.0156 ± 0.0002 x152
0.0001 ± 0.0000 x131
0.0155 ± 0.0002 x30
0.0001 ± 0.0000 x171
0.0155 ± 0.0001 x130
0.0001 ± 0.0000 x133
0.0155 ± 0.0001 x115
0.0001 ± 0.0000 x96
0.0152 ± 0.0001 x25
0.0001 ± 0.0000 x120
0.0151 ± 0.0001 x90
0.0001 ± 0.0000 x33
0.0140 ± 0.0001 x79
0.0001 ± 0.0000 x79
0.0139 ± 0.0001 x85
0.0001 ± 0.0000 x60
0.0128 ± 0.0001 x70
0.0001 ± 0.0000 x66
0.0126 ± 0.0002 x136
0.0001 ± 0.0000 x84
0.0125 ± 0.0003 x133
0.0001 ± 0.0000 x128
0.0124 ± 0.0001 x168
0.0001 ± 0.0000 x109
0.0124 ± 0.0001 x137
0.0001 ± 0.0000 x36
0.0095 ± 0.0002 x46
0.0001 ± 0.0000 x19
0.0094 ± 0.0001 x171
0.0001 ± 0.0000 x170
0.0089 ± 0.0001 x158
Artificial Intelligence 2024 50

0.0001 ± 0.0000 x85

0.0087 ± 0.0001 x23
0.0001 ± 0.0000 x30
0.0084 ± 0.0001 x170
0.0001 ± 0.0000 x98
0.0082 ± 0.0001 x169
0.0001 ± 0.0000 x151
0.0082 ± 0.0001 x53
0.0001 ± 0.0000 x24
0.0080 ± 0.0001 x159
0.0001 ± 0.0000 x150
0.0079 ± 0.0001 x51
0.0001 ± 0.0000 x80
0.0076 ± 0.0001 x21
0.0000 ± 0.0000 x38
0.0076 ± 0.0001 x125
0.0000 ± 0.0000 x27
0.0075 ± 0.0001 x172
0.0000 ± 0.0000 x94
0.0074 ± 0.0001 x83
0.0000 ± 0.0000 x23
0.0074 ± 0.0002 x106
0.0000 ± 0.0000 x44
0.0074 ± 0.0001 x119
0.0000 ± 0.0000 x50
0.0072 ± 0.0001 x94
0.0000 ± 0.0000 x172
0.0072 ± 0.0001 x101
0.0000 ± 0.0000 x26
0.0072 ± 0.0000 x113
0.0000 ± 0.0000 x110
0.0072 ± 0.0001 x124
0.0000 ± 0.0000 x137
0.0072 ± 0.0000 x29
0.0000 ± 0.0000 x139
0.0072 ± 0.0001 x89
0.0000 ± 0.0000 x169
0.0071 ± 0.0001 x117
0.0000 ± 0.0000 x114
0.0071 ± 0.0001 x111
0.0000 ± 0.0000 x74
0.0071 ± 0.0001 x47
0.0000 ± 0.0000 x20
0.0071 ± 0.0000 x105
Artificial Intelligence 2024 51

0.0000 ± 0.0000 x68

0.0071 ± 0.0001 x77
0.0000 ± 0.0000 x149
0.0070 ± 0.0000 x17
0.0000 ± 0.0000 x104
0.0070 ± 0.0001 x87
0.0000 ± 0.0000 x136
0.0070 ± 0.0000 x81
0.0000 ± 0.0000 x115
0.0069 ± 0.0001 x123
0.0000 ± 0.0000 x168
0.0069 ± 0.0001 x95
0.0000 ± 0.0000 x54
0.0068 ± 0.0001 x27
0.0000 ± 0.0000 x55
0.0068 ± 0.0000 x76
0.0000 ± 0.0000 x46
0.0068 ± 0.0001 x39
0.0000 ± 0.0000 x86
0.0068 ± 0.0001 x59
0.0000 ± 0.0000 x157
0.0067 ± 0.0001 x107
0.0000 ± 0.0000 x32
0.0066 ± 0.0000 x45
0.0000 ± 0.0000 x15
0.0066 ± 0.0000 x75
0.0000 ± 0.0000 x49
0.0066 ± 0.0001 x41
0.0000 ± 0.0000 x62
0.0066 ± 0.0001 x65
0.0000 ± 0.0000 x113
0.0066 ± 0.0001 x15
0.0000 ± 0.0000 x152
0.0066 ± 0.0001 x57
0.0000 ± 0.0000 x47
0.0064 ± 0.0000 x35
0.0000 ± 0.0000 x64
0.0064 ± 0.0001 x63
0.0000 ± 0.0000 x76
0.0062 ± 0.0001 x16
0.0000 ± 0.0000 x90
0.0061 ± 0.0000 x93
0.0000 ± 0.0000 x111
0.0060 ± 0.0001 x33
Artificial Intelligence 2024 52

0.0000 ± 0.0000 x16

0.0058 ± 0.0001 x157
0.0000 ± 0.0000 x53
0.0058 ± 0.0001 x34
0.0000 ± 0.0000 x45
0.0057 ± 0.0000 x131
0.0000 ± 0.0000 x83
0.0055 ± 0.0000 x99
0.0000 ± 0.0000 x117
0.0055 ± 0.0001 x71
0.0000 ± 0.0000 x119
0.0054 ± 0.0000 x129
0.0000 ± 0.0000 x92
0.0053 ± 0.0000 x20
0.0000 ± 0.0000 x56
0.0051 ± 0.0001 x69
0.0000 ± 0.0000 x122
0.0051 ± 0.0001 x64
0.0000 ± 0.0000 x57
0.0048 ± 0.0001 x104
0.0000 ± 0.0000 x107
0.0048 ± 0.0001 x82
0.0000 ± 0.0000 x25
0.0046 ± 0.0001 x88
0.0000 ± 0.0000 x77
0.0045 ± 0.0000 x156
0.0000 ± 0.0000 x29
0.0045 ± 0.0001 x58
0.0000 ± 0.0000 x59
0.0045 ± 0.0001 x52
0.0000 ± 0.0000 x135
0.0045 ± 0.0001 x110
0.0000 ± 0.0000 x14
0.0044 ± 0.0001 x22
0.0000 ± 0.0000 x81
0.0044 ± 0.0000 x28
0.0000 ± 0.0000 x89
0.0043 ± 0.0001 x112
0.0000 ± 0.0000 x17
0.0041 ± 0.0001 x118
0.0000 ± 0.0000 x3
0.0040 ± 0.0001 x86
0.0000 ± 0.0000 x22
0.0040 ± 0.0000 x38
Artificial Intelligence 2024 53

0.0000 ± 0.0000 x112

0.0039 ± 0.0000 x44
0.0000 ± 0.0000 x34
0.0039 ± 0.0000 x80
0.0000 ± 0.0000 x28
0.0038 ± 0.0000 x98
0.0000 ± 0.0000 x75
0.0036 ± 0.0000 x14
0.0000 ± 0.0000 x87
0.0035 ± 0.0000 x26
0.0000 ± 0.0000 x82
0.0034 ± 0.0000 x50
0.0000 ± 0.0000 x51
0.0034 ± 0.0000 x32
0.0000 ± 0.0000 x156
0.0032 ± 0.0000 x74
0.0000 ± 0.0000 x8
0.0032 ± 0.0000 x56
0.0000 ± 0.0000 x21
0.0031 ± 0.0000 x122
0.0000 ± 0.0000 x116
0.0030 ± 0.0000 x68
0.0000 ± 0.0000 x106
0.0029 ± 0.0000 x128
0.0000 ± 0.0000 x58
0.0029 ± 0.0001 x62
0.0000 ± 0.0000 x105
0.0028 ± 0.0001 x116
0 ± 0.0000 x155
0.0028 ± 0.0000 x92
0 ± 0.0000 x162
0.0018 ± 0.0001 x9
0 ± 0.0000 x1
0.0016 ± 0.0000 x135
0 ± 0.0000 x140
0.0014 ± 0.0000 x1
0 ± 0.0000 x141
0.0014 ± 0.0000 x8
0 ± 0.0000 x88
0.0014 ± 0.0000 x3
0 ± 0.0000 x167
0.0012 ± 0.0000 x153
0 ± 0.0000 x166
0.0005 ± 0.0000 x134
Artificial Intelligence 2024 54

0 ± 0.0000 x165
0 ± 0.0000 x162
0 ± 0.0000 x142
0 ± 0.0000 x167
0 ± 0.0000 x164
0 ± 0.0000 x166
0 ± 0.0000 x163
0 ± 0.0000 x165
0 ± 0.0000 x161
0 ± 0.0000 x164
0.0000 ± 0.0000 x134
0 ± 0.0000 x163
0 ± 0.0000 x143
0 ± 0.0000 x145
0 ± 0.0000 x144
0 ± 0.0000 x161
0 ± 0.0000 x145
0 ± 0.0000 x160
0 ± 0.0000 x160
0 ± 0.0000 x146
0.0000 ± 0.0000 x159
0 ± 0.0000 x142
0 ± 0.0000 x146
0 ± 0.0000 x143
0 ± 0.0000 x147
0 ± 0.0000 x155
0 ± 0.0000 x154
0 ± 0.0000 x140
0 ± 0.0000 x124
0 ± 0.0000 x154
0 ± 0.0000 x118
0 ± 0.0000 x144
-0.0000 ± x52
0.0000
0 ± 0.0000 x147
-0.0000 ± x9
0.0000
0 ± 0.0000 x141

8.5 Full Original Table of Permutation Importance K-means

Clustering

KMC 10.452 min

Artificial Intelligence 2024 55

Weight Feature
420308.1599 ± x10
1971.5218
168719.7697 ± x8
307.5919
168200.0994 ± x9
604.2461
144853.2147 ± x12
631.6338
144802.2410 ± x11
638.7344
115692.1266 ± x153
869.6890
60863.9908 ± x7
222.7441
60240.3905 ± x4
408.2612
59304.7034 ± x5
144.1569
58223.5526 ± x6
278.2673
49507.8658 ± x67
279.8174
49100.0612 ± x97
152.6299
49046.1333 ± x91
154.8688
49019.6117 ± x121
279.4339
48860.8884 ± x31
131.4751
48797.1800 ± x61
137.5981
48739.0284 ± x49
199.9117
48729.2202 ± x79
267.3027
48592.6339 ± x109
147.5623
48456.5323 ± x19
249.8473
47049.4670 ± x152
136.4209
Artificial Intelligence 2024 56

46962.3773 ± x150
216.2798
46944.3395 ± x151
160.3331
46666.9040 ± x149
76.6054
46363.3166 ± x43
228.4948
46125.1898 ± x103
161.8891
46066.5738 ± x73
105.2912
46053.0753 ± x13
247.2568
43072.4011 ± x127
141.9424
42998.5247 ± x37
194.3583
42838.2103 ± x85
340.5838
42793.7027 ± x115
214.4736
42701.4972 ± x55
121.2418
42533.6230 ± x25
373.9765
41771.6522 ± x72
121.8775
41352.0775 ± x126
161.4020
41339.3938 ± x66
277.6899
41298.6509 ± x102
147.1281
41269.9624 ± x36
219.1722
41247.8974 ± x96
240.3171
39896.1059 ± x24
173.7288
39841.9951 ± x84
125.2870
39766.0380 ± x114
198.2308
Artificial Intelligence 2024 57

39756.2993 ± x54
168.4137
38396.6087 ± x42
240.8642
38260.9820 ± x132
106.7866
38103.3501 ± x48
184.8240
37909.7689 ± x18
99.6934
37793.9234 ± x78
166.0346
37760.5721 ± x90
196.4883
37734.3883 ± x108
136.7099
37708.8216 ± x120
142.9657
37652.6453 ± x30
116.6496
37646.1722 ± x60
58.1251
33065.8235 ± x137
251.7146
32377.3105 ± x138
133.7776
32143.6539 ± x51
163.3497
32124.2603 ± x53
104.1482
32110.9218 ± x113
203.7616
32104.3048 ± x148
92.3177
32071.1699 ± x83
147.5913
32017.0288 ± x111
142.9326
32014.9845 ± x81
123.9986
31872.9000 ± x139
180.0975
31813.1713 ± x21
142.1148
Artificial Intelligence 2024 58

31773.8008 ± x23
247.7797
30819.2551 ± x65
186.5750
30803.7064 ± x63
168.8378
30668.7501 ± x95
80.2514
30646.8145 ± x93
209.6231
30548.7257 ± x33
133.6079
30538.2256 ± x136
100.4262
30529.3420 ± x35
29.5517
30404.0409 ± x99
181.6270
30393.6184 ± x125
161.8075
30391.7563 ± x101
105.2591
30365.2264 ± x123
82.7435
29938.3315 ± x69
141.0582
29856.6414 ± x71
102.6385
29414.8589 ± x27
98.5629
29400.6603 ± x29
68.2986
29104.2161 ± x57
85.6045
29099.3971 ± x119
152.4607
29097.3653 ± x117
129.6012
29078.8096 ± x59
141.4013
28942.2383 ± x89
74.9019
28919.8960 ± x87
131.8561
Artificial Intelligence 2024 59

28530.5207 ± x75
94.5690
28498.5219 ± x39
130.0983
28448.3075 ± x129
45.5075
28422.5563 ± x77
144.8911
28414.2082 ± x41
81.5877
28405.2021 ± x131
91.2084
28396.0382 ± x15
170.3150
28370.0028 ± x107
82.6514
28361.2497 ± x105
136.1323
28328.2373 ± x17
84.9356
28260.0626 ± x45
91.7907
28227.0409 ± x47
106.8331
26572.4406 ± x70
145.6861
26237.9685 ± x124
266.0094
26201.0758 ± x34
206.6818
26099.6413 ± x100
192.7798
25819.1745 ± x64
285.0211
25701.9789 ± x94
151.6474
23358.7390 ± x170
34.2192
23340.7800 ± x169
125.2808
23224.7835 ± x22
137.0599
23213.5238 ± x171
73.9985
Artificial Intelligence 2024 60

23211.9618 ± x172
92.8742
23084.9442 ± x82
149.1148
22950.3683 ± x46
290.1383
22858.7767 ± x106
125.9910
22832.9635 ± x76
137.1438
22820.0627 ± x16
196.1715
22734.7478 ± x112
184.2227
22554.8841 ± x52
133.6600
16572.5238 ± x168
60.0438
16450.4633 ± x159
73.3357
16431.2873 ± x133
162.3274
15634.3760 ± x156
43.2353
15307.5631 ± x158
55.6226
15028.8766 ± x157
56.7018
14989.7410 ± x20
217.2009
14728.3909 ± x104
75.8380
14282.3171 ± x14
44.1813
13981.8141 ± x74
53.2403
13879.7776 ± x44
65.5766
13458.8925 ± x110
92.5679
13223.6312 ± x130
54.4273
13193.0326 ± x32
97.6119
Artificial Intelligence 2024 61

13157.5630 ± x40
28.5600
13075.7362 ± x88
32.6480
13021.7112 ± x122
57.1614
12907.6308 ± x50
79.3466
12819.2426 ± x58
51.6681
12815.5325 ± x80
159.2604
12729.1920 ± x118
38.6092
12685.7919 ± x98
94.9549
12656.5577 ± x28
59.4703
12654.2194 ± x56
130.9794
12557.6649 ± x92
97.8118
12520.4220 ± x38
102.7318
12443.3832 ± x86
115.8954
12356.1254 ± x128
163.7202
12080.9775 ± x62
79.0429
12068.3206 ± x116
115.1230
12021.4671 ± x26
149.0293
11754.6019 ± x68
148.3217
10002.0522 ± x135
75.9444
8588.1981 ± x134
67.3481
55.3999 ± x2
0.2181
35.2083 ± x0
0.2392
Artificial Intelligence 2024 62

0.3962 ± 0.0168 x3
0.1059 ± 0.0108 x1
0 ± 0.0000 x142
0 ± 0.0000 x143
0 ± 0.0000 x140
0 ± 0.0000 x167
0 ± 0.0000 x166
0 ± 0.0000 x165
0 ± 0.0000 x164
0 ± 0.0000 x163
0 ± 0.0000 x162
0 ± 0.0000 x161
0 ± 0.0000 x160
0 ± 0.0000 x155
0 ± 0.0000 x154
0 ± 0.0000 x141
0 ± 0.0000 x146
0 ± 0.0000 x145
0 ± 0.0000 x144
0 ± 0.0000 x147