MIPRO 2022, May 23 - 27, 2022, Opatija, Croatia

Automatic, configurable, and partial assessment

of student SQL queries with subqueries

Mario Fabijanić*, Goran Đambić* and Jan Sasunić*

*
Algebra University College/Software Engineering, Zagreb, Croatia
[email protected], [email protected], [email protected]

Abstract – Universities provide database-related courses, only 14 out of 42 teachers finished the marking process in
where students demonstrate their acquired knowledge by a time slot given by the university. Furthermore, during
writing SQL code as a solution to presented problems. student insights into the exam and grades they got, initial
Manual assessment of students’ solutions lasts for days and grades changed in 28 out of 42 cases, which emphasizes the
brings up additional problems: inconsistency in grading, problem of nonconsistency during the marking process.
students waiting long for their grades, and they get
clarification feedback with details only on demand. Also, not The feedback that students get after their solution
a negligible number of teachers’ clarification results in delivery is another important component of the learning
correction of initially obtained points. Many authors process. In [4], authors concluded that automatic feedback
research the possibility to create computer systems for fully increases student performance, helps teachers by
automated assessments, which would minimize or completely decreasing their effort in the marking process, and
annul listed challenges. The authors propose a novel system compares with a correct solution is one of the main methods
for automatic, configurable, and partial assessment of for generating automatic feedback.
student SQL queries. Initially, it supported only basic
SELECT statements but was upgraded later to support also At the Algebra University College, there are two
complex joins and groupings. The comparison of new system courses regarding relational databases [5]: Introduction to
output and teachers’ manual assessments concluded that the Databases, compulsory for all first-year students, and
novel, fully automated assessment system is very promising. Database Development, compulsory only for Software
Now, the authors upgraded the assessment model to support Engineering students. Each year number of students
subqueries, which are widely used, and thus, an integral part increases and workload regarding the evaluation of their
of database courses and assessments. This paper compares knowledge is becoming more challenging, because we do
the output of the proposed system with the teachers’ not want to rely only on tests with multi choice questions
manually provided grades, focused on subqueries. The or so, although they can also be in the domain of automatic
results of this more comprehensive solution remain assessment tools, and can be similarly valuable if written
promising, even with the new features added. well [6]. Instead, the authors of this paper do not give up on
an idea to put our students into a realistic situation that they
Keywords – SQL; assessment; fully-automated; partial
marks; configurable assessment process; subqueries
need to solve an issue with writing SQL code, test it and
debug it. The main reason for this attitude is that way
students also create their own experiences like those needed
I. INTRODUCTION in their real jobs after graduating, and our experiences tell
us that they need minimal adjustment when start working
The idea of having an automated program to test the
on real-life problems later.
code is old more than 60 years [1], and these years the same
idea is still the subject of research since IT universities and Since each exam consists of around 25 assignments,
colleges enroll more and more students, which creates and last year we had around 240 students enrolled, teachers
additional pressure on teachers or teacher assistants. There need to evaluate around 6.000 student solutions. An
are several challenges: the process of marking should average time needed to evaluate one student solution is 1
remain quick enough, it should provide consistency during minute, so we need approximately 100 hours for evaluation
the evaluation of student solutions, and it should provide during the year. This number is not precise because there
precise feedback, that way improving the complete are students that dropped out during the year, but also those
educational process and student experience. Teachers and that went on exam terms more than once if they fail
teacher assistants have a direct impact on those challenges, previous time, but it is significant and slightly bigger than
therefore the human factors may compromise the process in mentioned research [3]. This year we have around 300
[2]. students enrolled, so the workload on teachers would be
even higher.
The research that focused on teachers’ marking
experiences within university teachers [3] points out that Some authors proposed their semi-automated or fully-
assessing SQL assignments represents a certain cognitive automated solutions [7] [8] [9] [10] [11] [12] [11] [12] [13]
load that needs to be maintained during days of assessing [14] [15] [16], but they implement only part of the
the assignments, which may result in delays and requirements for the system. Therefore, we set the
inconsistency. For example, teachers in that research spent
22 hours assessing 1533 SQL student assignments, and

542
Authorized licensed use limited to: Hongik Univ. Downloaded on December 23,2022 at 09:13:02 UTC from IEEE Xplore. Restrictions apply.
following goals for the comprehensive automatic In [19], the authors of this paper also use query text
assessment system: analysis to discover Levenshtein’s distance [20] [21]
between student query solution and correct query solution.
x Import of student solutions If Levenshtein’s distance is equal to or less than 1, the
x Automatic assessment of student solutions student query solution is considered as correct, which
covers situations of mistyping in the student solution.
x Recognizing syntax errors in student solution
2. Student solutions created concerning a set of
x Recognizing prohibited elements in student solution guidelines.
x Recognizing differences between student solution and Since SQL as a language supports multiple ways to
correct solution create a query, authors created a list of guidelines to apply
while creating a query. For instance, if aliases of tables exist
x Partial points for student solutions according to for one table in a query, they must be used to all other tables
teachers’ earlier defined criteria too. In WHERE clause, column name must always be the
x Report with points achieved and the list of errors. first operand (e.g., Quantity > 10 is correct, 10 < Quantity
is not). Using positions in the ORDER BY clause is not
Regarding those, two years ago authors of this paper possible, etc.
proposed a novel system for automatic, configurable, and
partial assessment of student SQL queries, limited only to 3. Equivalent joins are recognized.
simple SELECT statements, with no possibility of marking When joining data from multiple tables, student query
solutions which include JOINs, GROUP BY, HAVING solution and correct solution may differ in used join types,
clauses, aggregate functions and subqueries [5]. The special but student solution can still be correct, e.g., table1 left join
focus was on partial marking of student solutions, which table2 is equal to table2 right join table1. The system must
also makes it possible to give students precise feedback by be able to recognize all possible solutions and to achieve
the system, improving their education process. Some that, the proposed system converts join into canonical form.
authors recognized machine learning and AI as Canonicalization uses the following characteristics of joins:
technologies that have an impact in several categories of the
education process, grading students included [17], and a. A join of two tables A and B is a cross product [22]
especially for giving feedback to students [18]. However, of all their rows, applied with the predicate p:
the system proposed in this paper does not implement those
 (  )
technologies yet.
Subsequently, authors supplemented the system with b. Joins are commutative:
the possibility to mark student solutions that contain even  (  ) =  (  )
complex joins and groupings [19]. System gives partial
points based on teachers’ input. To make it possible, c. Joins are associative:
teachers need to provide the correct solution for a presented  ( (  )  ) =  (    )
problem and 17 marking rules for giving partial points.
The system extracts FROM predicates, and for each
The current version of the proposed system covers the
predicate, source and target tables extract columns and
majority of SELECT question types at the Introduction to
joins. It converts right outer join into left outer join, source
Databases course. To be able to cover all of it, a proposed
table must be alphabetically lower than the target table,
system needs to also cover subqueries, since they are also
alphabetically sort columns, and replaces aliases if they
one of the learning objectives in that course. This paper
exist with full names. All of those make it possible to
proposes a fully automated assessment system based on
compare different ways of selecting the same data from
[19], with added support for assessment SQL containing
joined tables.
subqueries.
4. Marking continues even in the case when some of
II. METHODOLOGY the clauses are wrong.
An earlier version of the automated assessment system The system awards partial, isolated clauses of a
fulfilled the following requirements [19]: SELECT statement, i.e., when comparing the result set of
student query solution with the result set of the correct
1. Queries executes against actual database and query to check the WERE clause, the system compares
assessment system processes the result set with the primary key values of the first table in FROM clause with
ability to parse and analyze query text. values from the correct query. If there are differences, the
Many authors proposed tools [7] [8] [9] [10] [11] [12] student cannot get maximum points, and further query text
[11] [12] [13] [14] [15] [16] where correct query solution analyses will happen.
and student query solution are executed at the actual 5. Students get partial grades according to the correct
database and the returned result sets are compared. Some solution and predefined set of rules created by the
of those tools give maximum or zero scores, depending on teacher.
the conclusion are the result sets equal or not. Some of those
tools give even partial points, based on result sets similarity The proposed automatic assessment system relies on
expressed in percentages. teachers’ inputs before the marking process starts. The
teacher must provide question text, e.g. “Using subquery,

543
Authorized licensed use limited to: Hongik Univ. Downloaded on December 23,2022 at 09:13:02 UTC from IEEE Xplore. Restrictions apply.
select all credit card types and credit card numbers that are "from_max_joins_allowed"
not used for any payments yet. In a result set, show records "from_use_specific_join"
sorted by credit card number descending. Using joins "where_wrong_rows"
between tables are forbidden.”, but also a correct solution "where_partial_correct_column"
example and a set of rules which assessment system will "where_use_specific_operator"
"groupby_not_correct"
apply during the marking process, both within JSON
"groupby_partial_correct_column"
configuration file: "having_wrong_number_of_rows"
A. Examples of correct solutions: "having_missing_aggregate_function"
"select_missing_regular_column"
a. Subquery in WHERE clause: "select_missing_function_column"
"select_missing_aggregate_column"
SELECT cc.CardType, cc.CardNumber
"orderby_each_missing_column"
FROM Sales.CreditCard AS cc
"orderby_each_wrong_ascdesc"
WHERE cc.CreditCardID NOT IN "orderby_each_wrong_column_order"
(
SELECT oh.CreditCardID
FROM Sales.SalesOrderHeader AS oh For each of those, JSON configuration file contains
WHERE oh.CreditCardID=cc.CreditCardID status, deduct and hard_stop entries, but in addition,
) some of them contain specific settings like:
ORDER BY cc.CardNumber DESC
x num - number of maximum joins allowed in student
b. Subquery in SELECT clause: solution or number of subqueries needed as part of
SELECT clause or WHERE clause in student solution
SELECT oh.AccountNumber, oh.DueDate,
( x specific_join - in case that student solution needs
SELECT SUM(od.LineTotal) to implement a specific type of inner, outer, full, or
FROM Sales.SalesOrderDetail AS od cross join in FROM clause
WHERE od.SalesOrderID=oh.SalesOrderID x specific_operator - if the specific operator needs
) AS TotalPrice to be used in WHERE clause of student solution.
FROM Sales.SalesOrderHeader AS oh
Additional rules added as support for assessing
solutions with subqueries are as follows:
B. General rules for each clause (FROM, WHERE, "select_subqueries_required": {
GROUP BY, HAVING, SELECT, ORDER BY) "status": "on",
that will be used by the system during the "deduct": 100,
assessment process, e.g. "hard_stop": 0,
{ "max_points": 2.0, "num": 1 },
"clause_points_from": 100, "where_subqueries_required": {
"clause_points_where": 50, "status": "on",
"clause_points_groupby": 0, "deduct": 100,
"clause_points_having": 0, "hard_stop": 0,
"clause_points_select": 30, "num": 1 }
"clause_points_orderby": 20,
"parses_and_compiles": { Those enable checking if a subquery is used in SELECT
"status": "on",
or WHERE clause of student solutions at all, how many
"deduct": 100,
"hard_stop": 1 },
subqueries must be implemented in a student solution, what
"wrong_results": { percentage of the score will be deducted in case of broken
"status": "on", rule, and if the assessment process will continue in case of
"deduct": 50, broken rule or not.
"hard_stop": 0 } As the first step during the assessment process, the
assessment system tries to compile and execute student
In this part of the configuration JSON file, the solutions against the actual database. In this process called
deduction is defined as percentages of maximum points “dynamic protocol”, the assessment system implements the
possible, and status defines will a particular rule be first rule from the configuration JSON file:
checked during assessment or not. hard_stop setting parses_and_compiles.
defines if the assessment process will continue in case of As a second step, the assessment system checks defined
breaking a particular rule or not. rules from the JSON configuration file and deducts
C. Percentage of score to deduct for each broken rule, percentages of broken rules. To be able to do this step called
which will deduct from maximum possible points “static protocol”, before checking defined rules, student
of a particular solution. Earlier implemented a set solutions are tokenized and converted into the canonical
of rules are: form [5]. Fig. 1 shows part of the correct solution with
subquery in WHERE clause tokenized into the dictionary.
"from_each_missing_table" It contains key-value pairs for each clause of the query:
"from_each_wrong_join" SELECT clause, FROM clause, WHERE clause, and

544
Authorized licensed use limited to: Hongik Univ. Downloaded on December 23,2022 at 09:13:02 UTC from IEEE Xplore. Restrictions apply.
ORDER BY clause. In this example of the correct solution, A recursive function that counts several subqueries in
there is only one condition in WHERE clause: the solution works a little bit differently than in the first
cc.CreditCardID NOT IN the list of CreditCardID values example: it searches the “value” dictionary for the “select”
that exist in the SalesOrderHeader table within the key. It works recursively too, and it returns the number of
subquery. A subquery is a dictionary too, the same way as found subqueries in the SELECT clause. In both examples,
the main SELECT statement. a number of counted subqueries compares with JSON file
configuration setting and if a student used less than needed
To be able to find subqueries in the dictionary of the number of subqueries, points deduct.
main SELECT statement and its’ WHERE clause, a special
function extracts the content of the WHERE clause and Another part of the marking process is a comparison of
searches for its key-values pairs. If it contains the SELECT result sets: correct solution result set and student solution
clause too, it is a subquery. The function also works result set [5]. In case of mismatches, points based on broken
recursively; it checks the WHERE clause of a subquery and where_wrong_rows rule deduct. If correct columns
searches the new SELECT clause in the WHERE clause of and/or operators are in WHERE clause, following rules
the already existing subquery. In the end, it returns a with the negative values awards partial points:
number of found subqueries. where_partial_correct_column and
where_use_specific_operator.
Fig. 2 shows part of the correct solution with subquery
in the SELECT clause tokenized into a list where each 6. Students get feedback with the final points and the
member of a list represents columns that will be in a result list of errors. Fig. 3 shows an example of feedback
set. Each list member is a dictionary again. for one student solution.

Figure 1. Correct solution with subquery in WHERE clause tokenized into Python dictionary

Figure 2. Correct solution with subquery in SELECT clause tokenized into Python dictionary

545
Authorized licensed use limited to: Hongik Univ. Downloaded on December 23,2022 at 09:13:02 UTC from IEEE Xplore. Restrictions apply.
 Figure 3. Example of final feedback for student solution

III. RESULTS TABLE I. OBSERVED FREQUENCIES OF GRADES

To validate the efficacy of the proposed system for Grades 0 0.1 0.25 0.5 1 Total
automatic assessment of SQL statements, we tested it using Proportion of grades
three exam assignments from real exam terms held at 53 0 1 8 88 150
assigned by teachers
Algebra University College last year. For the first Proportion of grades
55 1 0 10 84 150
assignment subquery was mandatory in the SELECT clause assigned by system
of the main query, for the second assignment subquery was Total 108 1 1 18 172 300
mandatory in the WHERE clause of the main query, and for
the third assignment, subquery was again mandatory in the
WHERE clause of the main query, but the subquery itself TABLE II. EXPECTED FREQUENCIES OF GRADES
had to contain two more subqueries. For each of those three
Grades 0 0.1 0.25 0.5 1 Total
assignments, we tested 50 student solutions, 150 in total.
The proportion of
Analyzed assignment solutions are restricted to use grades assigned by 54 0.5 0.5 9 86 150
subqueries in SELECT and WHERE clauses. teachers
The proportion of
Fig. 4 shows the average percentage of points given for
grades assigned by 54 0.5 0.5 9 86 150
each assignment, from teachers and the proposed system. the system
In assignments 1 and 3, the difference is less or equal to Total 108 1 1 18 172 300
1%, and the only assignment where there is a significant
difference (7.5%) is assignment 2. While analyzing the
cause of it, it was found that in assignments like this, The number of degrees of freedom is:
teachers practice giving students full points even in cases
when they use sign * in SELECT clause rather than (Number of rows - 1) * (Number of columns - 1) =
columns asked by assignment. This indicates that the = (2 - 1) * (5 -1) = 1 * 4 = 4 (1)
proposed system is still suitable for its purpose, even after
implementing new features. The formula to calculate χ2 value is:
For further confirmation, we conducted the Chi-square ( )
test (χ2 test) as a statistical hypothesis testing, using a χ2 = ∑ = 2.352 (2)

sample of 150 grades. A null hypothesis, in this case, is that
after implementing new features to the system, there is still With a significance level of 5% and 4 degrees of
no significant difference between the points given by the freedom (1), a critical value of χ2 is 9.488. Our calculated
proposed system and those given by the teachers. Observed χ2 value is 2.352 (2), which means there is no statistical
frequencies of grades given by the teacher and proposed significance between grades given by the proposed system
system are at Table I and expected frequencies of grades and grades given by teachers and we cannot reject our null
are at Table II. hypothesis.

IV. CONCLUSION
This paper proposed an upgraded system for automatic,
configurable, and partial assessment of student SQL
queries, with added ability to assess SELECT statements
with subqueries in SELECT and WHERE clauses. At the
same time, it kept all the advantages for students and
teachers brought by earlier versions: minimized waiting
time for students to see earned points from an exam,
supplemented with precise feedback, and for teachers, the
possibility to configure a system with exact marking rules,
improving their possible inconsistency during manual, long
time marking process.
To validate the proposed system, we used 150 student
solutions from a last year’s exam terms at Algebra
University College and compared their scores evaluated
manually by teachers with the scores given by the proposed
Figure 4. Comparison percentage of points given by teachers system. Statistical hypothesis testing using the Chi-square
and proposed system
test resulted in no statistical significance between grades

546
Authorized licensed use limited to: Hongik Univ. Downloaded on December 23,2022 at 09:13:02 UTC from IEEE Xplore. Restrictions apply.
calculated with the proposed system and grades defined by Int. Conf. Proceeding Ser., pp. 161–168, 2013, doi:
teachers manually. Since the results of using the system are 10.1145/2526968.2526986.
still promising, authors will strive to make it more [12] P. Brusilovsky, M. V. Yudelson, S. Sosnovsky, V. Zadorozhny,
comprehensive, i.e., develop support for using subqueries
D. H. Lee, and X. Zhou, “An open integrated exploratorium for
in FROM clause of the main query, or develop support for
evaluation of complex tasks that combines more SQL database courses,” Proc. Conf. Integr. Technol. into Comput.
statements together. Sci. Educ. ITiCSE, vol. 1, no. 412, pp. 22–26, 2008, doi:
10.1145/1384271.1384280.

REFERENCES [13] M. De Raadt, S. Dekeyser, and T. Y. Lee, “Do students

SQLify? Improving learning outcomes with peer review and
[1] J. Hollingsworth, “Automatic graders for programming
enhanced computer assisted assessment of querying skills,”
classes,” Commun. ACM, vol. 3, no. 10, pp. 528–529, Oct.
ACM Int. Conf. Proceeding Ser., vol. 276, pp. 101–108, 2006,
1960, doi: 10.1145/367415.367422.
doi: 10.1145/1315803.1315821.
[2] D. Fonte, D. Da Cruz, A. L. Gançarski, and P. R. Henriques,
[14] A. Cumming and G. Russell, “Automatic Checking of SQL:
“A flexible dynamic system for automatic grading of
Computerised Grading,” Int. J. Learn. Annu. Rev., vol. 12, no.
programming exercises,” OpenAccess Ser. Informatics, vol. 29,
3, pp. 127–134, 2006, doi: 10.18848/1447-
pp. 129–144, 2013, doi: 10.4230/OASIcs.SLATE.2013.129.
9494/cgp/v12i03/46714.
[3] J. Tharmaseelan, K. Manathunga, S. Reyal, D. Kasthurirathna,
[15] S. Sadiq, M. Orlowska, W. Sadiq, and J. Lin, “SQLator - An
and T. Thurairasa, “Revisit of automated marking techniques
online SQL learning workbench,” SIGCSE Bull. (Association
for programming assignments,” in IEEE Global Engineering
Comput. Mach. Spec. Interes. Gr. Comput. Sci. Educ., vol. 36,
Education Conference, EDUCON, 2021, vol. 2021-April, doi:
no. 3, pp. 223–227, 2004, doi: 10.1145/1026487.1008055.
10.1109/EDUCON46332.2021.9453889.
[16] J. C. Prior, “Online Assessment of SQL Query Formulation
[4] A. P. Cavalcanti et al., “Automatic feedback in online learning
Skills,” Ace ’03, vol. 20, pp. 247–256, 2003, [Online].
environments: A systematic literature review,” Comput. Educ.
Available:
Artif. Intell., vol. 2, p. 100027, 2021, doi:
http://delivery.acm.org/10.1145/860000/858433/p247-
10.1016/j.caeai.2021.100027.
prior.pdf?ip=201.5.254.119&id=858433&acc=PUBLIC&key
[5] M. Fabijanic, G. Dambic, and B. Fulanovic, “A novel system =4D4702B0C3E38B35.4D4702B0C3E38B35.6DB1341FA92
for automatic, configurable and partial assessment of student 4E6B8.4D4702B0C3E38B35&CFID=525851233&CFTOKE
SQL queries,” 2020 43rd Int. Conv. Information, Commun. N=96683423&__acm__=1436149768_230e7f22b7329beb08e
Electron. Technol. MIPRO 2020 - Proc., pp. 832–837, 2020, 8473b5e50b453.
doi: 10.23919/MIPRO48935.2020.9245264.
[17] D. Kučak, V. Juričić, and G. Đambić, “Machine learning in
[6] A. Chaudhuri, “Occasional appraisals,” J. Postcolonial Writ.,
education - A survey of current research trends,” Ann. DAAAM
vol. 42, no. 2, pp. 229–237, 2006, doi:
Proc. Int. DAAAM Symp., vol. 29, no. 1, pp. 0406–0410, 2018,
10.1080/17449850600973532.
doi: 10.2507/29th.daaam.proceedings.059.
[7] I. Mekterovic, L. Brkic, B. Milasinovic, and M. Baranovic, [18] V. González-Calatayud, P. Prendes-Espinosa, and R. Roig-
“Building a comprehensive automated programming Vila, “Artificial intelligence for student assessment: A
assessment system,” IEEE Access, vol. 8, pp. 81154–81172, systematic review,” Appl. Sci., vol. 11, no. 12, 2021, doi:
2020, doi: 10.1109/ACCESS.2020.2990980. 10.3390/app11125467.
[8] A. Kleerekoper and A. Schofield, “SQL tester: An online SQL [19] G. Dambic, M. Fabijanic, and A. L. Coskovic, “Automatic,
assessment tool and its impact,” Annu. Conf. Innov. Technol. Configurable and Partial Assessment of Student SQL Queries
Comput. Sci. Educ. ITiCSE, pp. 87–92, 2018, doi: with Joins and Groupings,” 2021 44th Int. Conv. Information,
10.1145/3197091.3197124. Commun. Electron. Technol. MIPRO 2021 - Proc., pp. 837–
[9] P. J. Wagner, “The sql file evaluation (sqlfe) tool: A flexible 842, 2021, doi: 10.23919/MIPRO52101.2021.9596680.
and extendible system for evaluation of sql queries,” Annu.
[20] L. Yujian and L. Bo, “A normalized Levenshtein distance
Conf. Innov. Technol. Comput. Sci. Educ. ITiCSE, p. 1334,
metric,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no.
2020, doi: 10.1145/3328778.3372599.
6, pp. 1091–1095, 2007, doi: 10.1109/TPAMI.2007.1078.
[10] A. Trongratsameethong, P. Vichianroj, and C. Mai, “ASQLAG [21] J. Wang, Y. Zhao, Z. Tang, and Z. Xing, “Combining dynamic
- Automated SQL Assignment Grading System for Multiple and static analysis for automated grading SQL statements,” J.
DBMSs,” vol. 1, no. 1, pp. 41–59, 2018, doi: Netw. Intell., vol. 5, no. 4, pp. 179–190, 2020.
10.14456/jti.2018.4.
[22] Justin Jaffray, “An Introduction to Join Ordering,” [Online].
[11] C. Kleiner, C. Tebbe, and F. Heine, “Automated grading and Available: https://www.cockroachlabs.com/blog/join-
tutoring of SQL statements to improve student learning,” ACM ordering-pt1/.

547
Authorized licensed use limited to: Hongik Univ. Downloaded on December 23,2022 at 09:13:02 UTC from IEEE Xplore. Restrictions apply.