1

E-Conf: A Conference Management System

João Luı́s, Instituto Superior Técnico, Lisboa, Portugal

Abstract—Conferences are an essential channel for researchers may be anonymous, hiding the author information for the
to exchange information. Nowadays, there are hundreds of reviewers.
thousands of conferences every year, each with its own character- After the reviewing phase, the conference team makes the
istics. However, organizing one is a difficult and time-consuming
process, involving hundreds of people. There are currently many final decision on which submissions to accept. The authors
Conference Management Systems to help with this process, some of the accepted submission must then submit the full paper,
of them analysed in this document, but they can be too complex if only the abstracts were submitted, or resubmit the paper
or lack some needed features. making small corrections the reviewers suggested.
This work introduces E-Conf, a web based Conference Man- Around this time, registration for the conference is open,
agement System designed and developed to help an organizer
planning a conference, automating a great part of the orga- with options for early bird and standard rates, workshops,
nizing process. The features of E-Conf are divided in four meals, etc.
main categories - main app, submissions, reviews and program Afterwards, it is time to create the conference timetable, cre-
management - which became the system modules. ating sessions and assigning events to sessions. The timetable
E-Conf also includes two algorithms for automatic assignment: is then published on the conference website.
AREA, to assign submissions to reviewers, and Chronos, to assign
submissions to sessions. When evaluated, both algorithms proof
Finally, the conference proceedings are prepared and printed
themselves capable, with AREA achieving pretty good results in and/or digitally published.
the used metrics. Table I, from Ex Ordo free conference planning eBook [1],
Index Terms—Conference, Conference management, Paper
illustrates the process, showing the major milestones.
management, Abstract management.
Month •
1 • Start planning
I. I NTRODUCTION
2 • Planning complete
ONFERENCES are an important channel for the ex-
C change of information between researchers. As such,
it should not be a surprise that nowadays conferences are
3 •
•
Launch conference website
Issue call for papers
4 • Invite reviewers
numerous.
6 • Close call for papers
There are many types of conferences - small and big, themed
7 • Open early bird registration
and general, with many types of events - and they all need to
• Begin peer review
be planned and organized.
8 • Peer review ends
Planning is the key to a conference success, so early
• Accept/reject abstracts
planning is essential for everything to run smoothly. To begin,
the conference team must be formed and the conference 9 • Receive copyright, final versions
parameters agreed - title, themes, topics, committees, dates, 10 • Close early bird registration
venue, sponsors, etc. • Create conference timetable
With this initial planning complete, the first call for papers 11 • Print hardcopy book / Publish digitally
is issued (usually on the conference website, which is launched 12 • Close registration
around this time), preparing the submission phase. Some orga- • Conference day
nizations opt for a two-phase submission, with a preliminary 13 • Post-conference meeting
submission first, to estimate the amount of submissions and • Final report / Handover to next chair
the number of reviewers needed. TABLE I
C ONFERENCE ORGANIZATION TIMELINE [1, F REE CONFERENCE
During the submission phase, researchers submit their work, PLANNING E B OOK ]
either the full paper or only the abstract - according to what
the organization decided. By this time, submissions can be
accepted to all the types of sessions the conference will held
(talks, posters, workshops, panels...) or only to some of them, This whole process can involve hundreds of people, thou-
the rest being accepted later. sands of e-mails and weeks of an organizer’s time. However,
When the submission phase closes, the submitted work is most of the tasks can - and, therefore, should - be automated,
evaluated. For this phase, the organization invites a number saving hours of administration time.
of reviewers, according to the number of submissions, so that As a result, there are currently many Conference Manage-
each submission is reviewed by (usually) at least three review- ment Systems (CMSs) in the market, with varying sets of
ers. The submissions are assigned to reviewers, who evaluate features and different business models, which will be presented
them with the criteria the organization decided. Reviewing in section II. However, its difficult to find an ideal system
 2

among them: some of them are too complex, others lack OpenConf: biggest lack is on program features, and as
some important features, and the majority they do not offer a EasyChair has no registration features. OpenConf has three
pleasant user experience. editions, Community (free), Plus and Professional, and is
provided as a product, i.e., the user downloads and installs
A. Objectives it on his server. OpenConf also provides a hosting service,
The main goal of this project is to design and develop a serving the Professional edition. [3]
software that can automate as many of the required tasks as Ex Ordo: is provided as a service and has no free version.
possible. It has one of the most modern designs, along with OSEM.
To do this, several existing CMSs will be studied and their Differently from the previous systems, Ex Ordo sells a basic
features analysed, along with other related work. Based on (Essential) package, which can be expanded with add-ons. [1]
that analysis, the system’s set of features will be decided and EDAS: is also provided as a service and most of the
the architecture of the new system will be designed, trying to features are included in the basic fee, the notable exception
answer to the existing problems. With the system architecture being the setup of the event public page. EDAS can also
defined, the system will be implemented. Finally, the created provide some additional material, like CD-ROMs and USB,
system will be tested. for extra fees. [4]
B. Document structure ConfTool: has two different versions: VSIS and Pro.
VSIS was designed for smaller events, has only the basic
This thesis is organized as follows: Section I describes the functions and is offered for local installation only, free on
process of organizing a conference, exposing the background request. Pro is provided as a service and has substantially
needed for the rest of the document. Section II lists and more features, being suitable for bigger and more complex
explains the features present in most CMSs, along with a com- conferences. [5]
parison between them. Further related work is also discussed
OSEM: is an event management system for free soft-
in this chapter. Section III describes the requirements for this
ware conferences. It is an open-source system and a single
system, obtained from the analysis in the previous chapter, and
installation can manage multiple conferences. Its main lack is
the architecture resulting from these requirements. Section IV
the reviewing process, as OSEM does not have any type of
lists the features implemented on this thesis and details their
reviewer assignment. Along with Ex Ordo, it has one of the
implementation. The tests of the project and its results are
most modern designs. [6]
presented on Section V. Section VI contains the conclusions
of this thesis, along with further work that can be done on ePapers: provides two services: submission & review,
E-Conf. which handles the whole process, and final collection, which
handles only the collection of final manuscripts. According
II. R ELATED WORK to its website, its main difference is that ePapers is a ”full-
For the study required for this work, several commercial service” system instead of a ”self-service” system, meaning
and open-source CMSs were analysed, both for features and that ePapers team prepares the system for the specifics needs
business models. Unfortunately, for the vast majority of the of its users. ePapers is the less comprehensive system of the
analysed systems there is not a demo or trial publicly available study. [7]
for the common user to get in touch with the system, so the
study had to rely on the feature list available on the systems’
websites. Additionally, their architectures are also unavailable, B. Other work
so they can not be analysed.
In general the studied systems have the same overall set of There are some articles about CMS solutions that are
features, helping on most of the tasks described on section also important for this study, proposing new systems or new
I - set up, submission, reviewing, program management and algorithms for some specific tasks.
registration - with the main difference being the level of COMFy: is a conference management framework, instead
automation done by the system. of a system.
COMFy is divided in layers. The bottom layer is a relational
A. Existing CMSs database. The second layer is composed of three parts: the
The studied systems are presented below, with a full feature repositories, which abstract the database to the upper layers;
comparison available on table II. Some of the systems have the state machine, which is described as the core of the
different versions (or editions), described below and referenced conference and manages the phase of each submission; and
in the comparative, each providing more features than the additional modules. The third layer its COMFy, the business
previous. logic of the CMS, which handles the requests. This layer
EasyChair: is currently probably the most commonly is designed as an Model-View-Controller (MVC) pattern and
used CMS and according to their website, hosted more than exposes a well defined RESTful API - the fourth layer - which
40000 conferences. It is the system with the most complete can be used to build applications - the top layer.
set of features of the study, although it lacks any registration To detect conflicts of interest, COMFy uses, along other
functionality. EasyChair is provided as a service with three dif- information, bibliographic data, such as co-authorships, which
ferent kinds of licenses: Free, Professional and Executive. [2] COMFy obtains from the DBLP API. [8]
 3

TABLE II
F EATURE COMPARISON

EasyChair OpenConf Ex Ordo EDAS ConfTool OSEM ePapers

Hosting and support
Hosting on own server C V only X
Hosting on a high-performance server F P X X P X
Data backup and replication F
Encrypted access only F P
Helpdesk and technical support E + X X
Miscellaneous
Import configuration from other configuration P
Custom branding P P
Authors and submissions
Anonymous submissions (double-blind review) F X X V
Custom submission types F X X
Author notification F C X X V X X
Instruction for authors P X
Custom fields in submission form P X P
Multiple file uploads P C P
Audio, video and ZIP upload P P X V
Terms & conditions for authors E
Program committee and reviewing
Event log F
Watchlist F
Online discussion of submissions F + X P X
Flexible access models for reviewers F
Leading reviewer model F
Subreviewers F X
Conflict of interest F C P
Author response phase / Rebuttal F + X P
Program committee invitations F X X P
Submission topics F C X V
Customized decisions (acceptance types) F + V
Customizable evaluation criteria F X X P X
Statistics F
Custom review forms P X X
Multi-track support P X X P (Extra) X
AAAI model E
Paper bidding and review assignment
Paper bidding F + P
Automatic submission assignment to reviewers F C X X V
Manual submission assignment to reviewers F C X X V X
Constraints on paper assignment F
Data export P C X P X
Email management
Email to authors and program committee F C X X
Email log F
Proceedings
Proceedings creation F + X
Springer LNCS proceedings F
Program and page generation
Program generation and publishing P P X X P
Page editing P P
Talks and sessions editing P P X X X
Assignments of talks to sessions P P X V X X
Scheduling constraints P X
Session chair handling P P X
Room handling P X
Program manager handling P
Automatic schedule analysis P
Scheduling suggestions P
Solutions P
List of events P
Email to managers, authors and session chairs P X X X
Multi-conference program generator E
Program page templates E P
Program download E P X
Registration
Registration X X V X
Set custom prices X X V
Extras X
Payment handling X X V
EasyChair F - Free, P - Pro, E - Enterprise;
OpenConf C - Community, + - Plus, P - Pro;
ConfTool V - VSIS, P - Pro.
 4

GRAPE: is a review assignment component for CMSs. Organizers shall then be able to assign the submissions to
GRAPE takes advantage from both the papers content (topics) reviewers or order the system to do it, and then be able to
and the reviewers preferences (biddings) and defines two edit the assignments. The system shall also detect conflicts of
measures to guide the system: reviewer’s gratification and interest, both for manual and automatic assignment.
article’s coverage, and tries to maximize both of them for all When the revision period starts, reviewers shall be able to
reviewers and all papers. GRAPE also accounts for conflicts review and mark the submissions assigned to them or delegate
of interest, both explicit by the organization and deducted by them to a subreviewer.
itself. [9] 4) Pre-conference: After the revision phase, an organizer
shall be able to accept or refuse submissions, as well as define
III. A RCHITECTURE how many submissions will be accepted. Both this value and
the submission score shall be shown to organizers, although
As stated in the introduction, organizing a conference is a
they shall not impose anything.
process that usually follows a sequence of well-defined and
With the accepted submissions chosen, authors shall be able
well-delimited steps, a linear workflow described on figure 1.
to submit the final work and pick which of the authors will
As such, E-Conf should be a state-based system - mainly the
be the presenter.
non-administration end - with some tasks confined to their
respective phase. Program management: Also when the accepted submis-
The system architecture will, as a result, be designed around sions are chosen, organizers shall be able to define the venues
this workflow, as well as the features of the systems presented and rooms (including their capacity) where the events will
in II. take place and create sessions. Organizers shall then be able
to assign submissions to sessions or order the system to do it,
and then be able to adjust the assignments. The program shall
A. Functional requirements then be made available through the conference page.
The functional requirements for E-Conf are based on the
feature set offered by existing CMSs, along with experience
from conference chairs. They will be presented by phase, B. Non-functional requirements
according to figure 1. Given the functional requirements that were analysed in
1) Set up: When first preparing E-Conf, an organizer shall the previous subsection, E-Conf has specific requirements that
be able to set the conference details - name, brand, dates and justify the architecture of the solution.
topics. An organizer shall also be able to configure the types
1) Configurability: As stated in the introduction, confer-
of events the conference will hold (e.g., papers, workshops,
ences are numerous and varied. As such, E-Conf must be
panels...).
configurable and adapt to the needs of a specific conference
2) Submission: To set up this phase, an organizer shall
and all its phases.
be able to define the start and end dates of the submission
phase. An organizer shall be able to configure the submission 2) Modularity: A conference flow is clearly divided in
form, both by adding text instructions for authors and by phases, each with its own set of features. These features
adding/removing form fields, including deciding if the full should be separated in modules, each module having its own
paper should be submitted on this phase or only the abstract. responsibility, which simplifies the system while allowing
The organizer shall also be able to issue a call for papers from for easy updating (or possibly reimplementation) of a single
E-Conf. module.
During the submission phase, authors (i.e., anyone regis- 3) Extensibility: Taking into account the previous two
tered on the system) shall be able to submit their work, as requirements, E-Conf must be also be extensible, i.e., E-Conf
many articles as wanted. Authors shall also be able to edit should be ready to accept new functionality in the form on
their submissions during this phase. new modules. Additionally, E-Conf shall also be extensible
Organizers shall also be able to enable or disable editing in an administrative way, i.e., new rules or policies may be
outside of this period, if they want. applied.
3) Reviewing: While preparing this phase, an organizer 4) Platform compatibility: Given the nature of a CMS, E-
shall be able to define the start and end dates of the revision Conf will be accessed by many and different types of users.
phase, as well as for the bidding phase. Organizers shall be As nowadays there are many platforms and screen sizes, a user
able to invite reviewers for the conference through E-Conf, can access the system from any of them. So, E-Conf shall be
using their emails. easily accessed from any platform.
Regarding the review form, organizers shall be able to set 5) Reliability: Any data loss or down time could be critical
the marking scheme for papers, i.e., define the factors and their for E-Conf. So, the system shall try to avoid any down time
weights, for E-Conf to be able to calculate a score. and, mainly, try to prevent data from being lost.
Reviewers shall be able to choose the topics of their interest, 6) Security: As different users will be able to access a dif-
as well as be able to bid on submissions, choosing the ones ferent set of features, authorization should be made according
they are willing or more comfortable to review, during the to the user role. This also implies that the system shall identify
bidding phase. the user when accessing E-Conf.
 5

Fig. 1. E-Conf flow

7) Usability: Using the system must not require previous To add users to committees, an organizer or a committee chair
training from the users, not only the committees but specially shall be able to send an invite to an email address. When a
the researchers, who will use the system only one or two times user receives an invite, they may refuse it or accept it, creating
and should be able to submit their work easily. an account if needed. Both the committee and invite systems
shall be made available to the other modules. The organizing
C. General architecture committee shall be created by the main app.
Conference chairs shall be able to setup the conference
Some design options made from the above requirements and
details from the main app. The chair shall be able to define
the analysis are presented below.
naming and branding details and the conference dates, along
1) Web application: The best way to achieve the best
with the conference topics and session types.
platform compatibility is to develop E-Conf as a web app.
Most of the devices nowadays have a web browser, allowing
them to access the system, without the need to install a new
software. E. Submissions module
2) Multitenancy: Most of the CMSs are provided as a The submissions module is responsible for receiving work
service, and as a result a single installation of the software submissions by authors. It is also through this module that
is able to handle multiple conferences, independent from each chairs can configure the submission phase.
other. Also, most of the institutions / organizations that host a Acceptance is also handled through this module. Although
conference usually host more than one. So, E-Conf must be a most of the times acceptance is done considering the review
multitenant application: each conference has a dedicated share results, that might not happen or the review process may be
of the instance, including its data, configuration, etc. However, handled through an external service. As such, acceptance must
there may also be cases where this is clearly too much for the be handled by this module and not the reviews module.
user’s needs. So E-Conf must also be able to host a single
conference, with no further requirements.
3) Multi language: Conferences are usually open to authors F. Reviews module
(and attendants) all around the world, so it would be nice,
although not required, to be able to present E-Conf to users The reviews module handles the revision process, from
in their own language (or in the language they prefer). bidding to acceptance. Because this module is so dependent
4) Hooks: To make E-Conf extensible, a hooks system will on submissions, it requires the submissions module.
be used, like WordPress. This method places actions and hooks To manage authorization in this module, a program com-
through the code so that the system calls package functions at mittee, containing all the reviewers, shall be created.
specific times. The revision phase, and thus this module, is divided in three
E-Conf comprises four modules, as shown in figure 2. The parts: bidding, assignment and revision.
main app module is the foundation of the system, and as a Bidding: During the bidding phase, reviewers can bid on
result all other modules need it to work. submissions. Reviewers can also mark if there is a conflict of
interest, which are stored as a specific type of bid. The dates
D. Main app of the bidding period and which information will be presented
The main app will handle the functionality common to all to reviewers during this phase shall be configurable by chairs.
the steps, and as such is responsible for multitenancy, user The system shall be able to detect conflicts of interest
management and the conference setup. between submissions and reviewers, and these conflicts need
The system administrator must be able to decide if the to be marked as an actual conflict by a chair to be applied.
system will host a single conference or many events. If they Assignment: Assignment occurs between the bidding and
decide for multitenancy, a special type of user must exist, the revision phases, and is responsible of assigning the submis-
network manager, that does not belong to any conference and sions to the reviewers. Chairs should be able to assign in one
manages the conferences in the system. of two ways: manually or automatically. Even if assignment
Users shall be able to register through the conference is automatic, a chair shall be able to adjust the results.
website, and login to access their personal area, using their Assignments will be stored as a review with no score, i.e.,
email address and a password to login. To ensure proper a review to be done.
authorization, i.e., each user can only access what they are E-Conf shall include an assignment algorithm and be able
supposed to, users shall be able to be assigned to committees. to receive more easily, through the extension system.
 6

Fig. 2. System architecture overview

Revision: During this phase reviewers score the submis- is currently the most popular PHP framework [11], has a
sions, according to the parameters defined by the organization. large community and includes solutions to common problems,
A chair can define on which parameters submissions will be such as authentication, authorization, multi language, routing,
reviewed, along with their weights. debugging, events, templates, form validation, etc.
A reviewer can delegate a review to another user (a subre-
viewer) if they intend. This invitation will use the invite system B. Multitenancy
before. Multitenancy in E-Conf is optional, making it an uncommon
Acceptance: When the reviews module is present, it shall case. This features needs to be abstracted, so that E-Conf could
“take over” the acceptance screen, showing the score of each be programmed ignoring its presence. The implementation of
submission and ordering them accordingly. In this case, the this feature builds on Laravel features.
system shall highlight the top submissions according to the First, a configuration file defines if the installation will hold
number defined in the submissions module. one or multiple conferences. Then, to know which conference
is being accessed, a middleware runs before the majority
G. Program management of the routes, which analyses the first URL parameter - the
This module is responsible for managing and displaying the conference slug. If it is a known slug, the system stores
conference program. the conference details and removes the parameter from the
A chair shall be able to create venues and rooms. A chair query, allowing it to be parsed the same way independently of
shall then be able to create events, and set its details, as well as tenancy.
sessions. Each session has a corresponding event, keeping the Knowing which conference is being accessed, the main step
timetable logic in one place. The program shall then be shown is to alter the queries to the database, using query scopes and
on the conference public page, with details on the speakers and model events. This behaviour is packaged in a PHP trait that
works. is applied to the models that shall be multitenant.
If the submissions module is enabled, and similarly to
reviews, a chair shall be able to assign submissions to sessions, C. App module
either manually or automatically. Like reviews, E-Conf shall This module is the core of E-Conf, and all other mod-
include an algorithm. ules are installed on this app, adding them as dependen-
cies through Composer. It is a Laravel app (built from the
IV. I MPLEMENTATION laravel/laravel repository).
The app User Interface (UI) is split in two parts, public
This section describes the implementation details of E-Conf.
page and administration. The UI for both parts can be seen on
All the project’s code is available at https://gitlab.com/econf.
figure 3.
Public page: The public page part includes the user
A. Technologies management pages. The menu that allows to change E-Conf
As E-Conf is a web-app, its client-side needs to be built language is also on the public page.
using HTML, CSS and JavaScript. LESS was used to pre- Administration: The administration area is only acces-
process CSS, as it extends its syntax to include variables, sible to users who belong to a committee, and it is locked
mixins, nested rule sets, etc. through a middleware. The administration allows members
PHP was used to program E-Conf server-side, as it cur- of the organizing committee to configure the conference,
rently powers the vast majority of websites [10]. To make including branding. These options are stored using the Laravel
development easier, the Laravel framework was used. Laravel Settings package.
 7

(a) Program public page

(b) Administration reviewer assignment pageparo

Fig. 3. App User Interface
 8

Network administration: If multitenancy is enabled, the bids and the conflicts of interest and tries to maximize the
core app will have a third part, the network administration. reviewers’ satisfaction. The algorithm is described in figure
From this panel the network administrator can add and remove 4. New assignment algorithms can be added by external
conferences. packages.
Revision: After the submissions are assigned, reviewers
D. Submissions module must review their assigned submissions. A reviewer may
review the submissions assigned to them through the admin-
This module main responsibility is to receive work submis- istration panel. The review form allows reviewers to score the
sions from authors. submission from 1 to 5 on the parameters defined by a chair
Submissions can be done by any registered user through the and to write text comments to the program committee and the
conference public page, during the defined period. The main author.
component of this module is the submission form, which is Every time a review is saved, its score is computed and
customizable through the Submission Settings page. stored to the database, so they are not computed each time it
A submission can have as many authors as needed, as the is viewed.
forms allows to add and remove authors. The list of fields Reviewers may also delegate their reviews to other users
available for each author is configurable through the settings (subreviewers) through the reviews list. These users do not
page and extensible through a filter. need to be a part of the program committee.
The submissions are stored with a sequential ID, however,
showing this ID to the users can be a security breach. So, the
actual ID is masked as a Hashid1 and this is the one displayed F. Program management module
to end users. The program management module is responsible for man-
This module also handles acceptance. When the acceptance aging and showing the conference program. Organizers can
period is open - after the submission phase ended - the manage venues and rooms through the Program section of the
Acceptance page becomes accessible , allowing an organizer administration panel. Organizers can then create events, which
to pick the accepted submissions. have the start and end times, the room where they take place
The organizer can also choose to ask for additional infor- and other details.
mation when a submission is accepted, like a final document Organizers can also create sessions. Sessions are special
or the presenter. This data can then be filled through the public events where submissions will be presented. Each session has
page. a session type and a topic, along with its submissions. As
it was said before, each session has a corresponding event,
E. Reviews module allowing queries to the program to be more efficient.
E-Conf also creates a program page on the conference
The reviews module is, as stated before, divided in three
public page, shown on figure 3a.
parts: bidding, assignment and revision. These parts are de-
Assignment: Organizers can assign submissions to ses-
scribed below.
sions though this module. This assignment can be done
Bidding: Reviewers can bid on submissions through
manually or automatically.
the Bidding page in the administration panel. In this page,
The manual assignment can be done from the administration
reviewers view a list with all submissions and are able to
session page. From this page organizers can add submissions
choose their bid - High, Medium, Low or Conflict.
to a session, and order them.
E-Conf is, by default, able to detect conflicts between a
Automatic assignment works similarly to the reviews au-
submission authors and a reviewer by name and organization.
tomatic assignment. When the Automatic assignment page is
A hook placed in the conflict detection code allows packages
accessed, organizers are presented with a list of algorithms
to add new ways to detect conflicts.
and, after an algorithm is picked, the results of that algorithm
Assignment: The assignment phase occurs between the
are presented.
bidding and review phases. During this phase, the chairs can
E-Conf includes the Chronos algorithm, described in figure
assign the submissions to reviewers, through two new pages
5, which takes into account the scores and the topics. New
in the administration panel: manual and automatic assignment.
assignment algorithms can be added by external packages.
The manual assignment page lists all the submissions and,
for each submission, all the reviewers that can review it, along
with the value of their bid and the number of submissions V. E VALUATION
assigned to them. This page is shown on figure 3b. Like every other system, E-Conf needs to be tested and
The automatic assignment page lists all the available al- evaluated, to check if it is ready to use in a real world scenario.
gorithms. After choosing one, its results will be presented,
without saving them. A chair may then save the results and A. Algorithm tests
will after be able to adjust them.
E-Conf includes the AREA (Automatic REviewer Assign- Both the algorithms created, AREA and Chronos, were
ment) algorithm, which takes into account the reviewers’ tested and evaluated.
To make the results more accurate, the algorithms were
1 http://hashids.org tested with real data, from the 12th Conference on PhD
 9

1 Function AssignForValue (bidValue) 1 submissions ←

2 valueReviewers ← Submissions.accepted().groupBy(topic);
Bids.withV alue(bidV alue).reviewers 2 sessions ← Sessions.all();
.sortBy(numBids) foreach valueReviewers as 3 sessions submissions ← [];
reviewer do 4 while not submissions.isEmpty or not
3 reviewerBids ← sessions.isEmpty do
reviewer.bidsW ithV alue(bidV alue).submissions 5 submissions.sortByDesc(length);
.sortBy(numBids); 6 sessions.sortByDesc(remainingSubmissions);
4 foreach reviewerBids as submission do 7 session ← sessions.shif t();
5 subs revs[submission.id].add(reviewer.id); 8 chosen submissions ← submissions.f irst
.slice(0, session.remainingSubmissions);
6 revs subs[reviewer.id].add(submission.id); 9 submissions.remove(chosen submissions);
10 if session.f ull then
7 if subs revs.length > 11 sessions.remove(session);
reviews per submission then
8 submissions.remove(submission); 12 foreach sessions submissions as session do
9 if revs subs.length > 13 session.shuf f le();
submissions per reviewer then 14 return sessions submissions;
10 reviewers.remove(reviewer);
Fig. 5. Chronos algorithm
11 break

12 Function AssignRemainingSubmissions Research in Microelectronics and Electronics (PRIME 2016).

13 foreach submissions as submission do To import the data and compute the values, a fifth module was
14 submReviewers ← created (Evaluation), which imports data from a spreadsheet,
P rogramCommittee.canReview(submission) exported from EasyChair.
.sortBy(topicM atch); Reviewers assignment algorithm (AREA): Two metrics were
15 foreach submReviewers as reviewer do used to test this algorithm: reviewer satisfaction and topic
16 subs revs[submission.id].add(reviewer.id); matching. These metrics were computed after importing all the
data from PRIME and using AREA to assign submissions to
17 revs subs[reviewer.id].add(submission.id); reviewers, configured to assign three reviewers per submission.
The results were then compared with the assignments Easy-
18 if subs revs.length > Chair made for PRIME.
reviews per submission then Reviewers satisfaction: Reviewer satisfaction describes
19 submissions.remove(submission); how much a reviewer’s assignments match their bids. The
20 if revs subs.length > Evaluation module computes this metric for each reviewer and
submissions per reviewer then then uses that data to present some statistics.
21 reviewers.remove(reviewer); Taking into account only the reviewers who have done their
22 break bidding, the average reviewer satisfaction is 84.69%. While
this is a good rate, there is clearly space for improvement.
Topic matching: After assigning submissions to the users
23 submissions ← Submissions.all();
who bided, AREA assigns the remaining submissions match-
24 reviewers ← P rogramCommittee.members();
ing the submission and the reviewer topics.
25 bids ← Bids.all();
Considering all the reviewers who have not done their
26 subs revs ← [];
bidding, the average topic matching is 8.12%. However, if only
27 revs subs ← [];
reviewers who chose more than one topic are considered, the
28 reviews per submission ← getF romSettings();
average topic matching raises to 82.56%. These results show
29 submissions per reviewer ←
that the algorithm performs better if the reviewers choose more
ceil( submissions.length×reviews per submission
reviewers.length ); than one topic of interest, because submissions have usually
30 AssignF orV alue(High); more than one topic.
31 AssignF orV alue(M edium); Comparison with PRIME: Another interesting evaluation to
32 reviewers ← reviewers.sortBy(numAssignments); consider is the comparison between the AREA results and the
33 AssignRemainingSubmissions(); actual assignments for PRIME, both for the metrics above and
34 if not submissions.empty then the assignments done.
35 reviewers ← P rogramCommittee.members(); 55.86% of the assignments done by AREA were also done
36 AssignRemainingSubmissions(); for PRIME. This value, by itself, does not say much, although
37 return subs revs; it is still interesting that both methods share more than half of
the assignments.
Fig. 4. AREA algorithm
 10

To get the values for reviewers satisfaction and topic match- with a web browser. The clear separation of the features in
ing, the assignments from PRIME were imported and the same categories was used to group the functionality in modules,
computations were done. The reviewers satisfaction, 81.96%, one for the main app and one for each category.
is pretty close to the value obtained by AREA. However, The evaluation performed showed that E-Conf can fulfil its
the topic matching value is greatly inferior: 2.19% for all purpose. The AREA algorithm evaluation also showed positive
reviewers with no bids and only 10% for the reviewers with results (similar or better than PRIME’s) for the used metrics.
more than one topic. The Chronos algorithm, however, despite the good results
Session assignment algorithm (Chronos): The Chronos al- in the metrics used, clearly needs improvement - or to be
gorithm is a very simple algorithm that mainly takes into replaced.
account topics. So, the metric used to evaluate this algorithm
was the topic grouping, i.e., whether the submissions in a A. Future Work
session shared a topic.
To test the algorithm, the acceptance data from PRIME was The addition of new modules focusing on registration and
imported, a program similar to PRIME’s was created on E- public page would make E-Conf a CMS able to cover the
Conf and the submissions assigned using Chronos. whole organizing process, with no external tools required.
The average topic grouping for these assignments were The assignment algorithms are another area that could be
100%. Closer inspection of individual sessions shown that, for enhanced in the future. E-Conf includes two basic algorithms,
the majority of the sessions, the common topic was the same but a thorough study on these aspects could produce much
topic. This shows that there is clearly room for improvement, better algorithms.
even if taking only in account topics. For instance, topic pairs Regarding UI and customization, E-Conf could benefit from
could be considered. a theming engine, allowing chairs to easily change their public
Other improvements can be done to this algorithm, as stated page appearance. This is already possible if a module overrides
in VI-A. most of the templates, but this process could be made easier.
Another feature that would open many doors for E-Conf is
the creation of an Application Programming Interface (API).
B. Functionality tests
This could allow the integration of E-Conf in other systems
To test if the system could fulfil its main goal, it was run for and many other creative uses, e.g. a complete new interface
the whole organization process, from the set up to the program for the system.
generation.
To do these tests a mock conference was created - TEST
R EFERENCES
2016 - and organized using only E-Conf. All the needed
data was generated with the help of the Faker package 2 and [1] “Ex ordo — award winning abstract management software,” http:
//exordo.com/, (Visited on 14/Dec/2015).
inserted automatically, but all the remaining interactions, like [2] “Easychair,” http://www.easychair.org/, (Visited on 14/Dec/2015).
assigning submissions to reviewers or creating sessions, was [3] “Openconf peer-review, conference and abstract management software
done through the E-Conf UI. system,” http://www.openconf.com/, (Visited on 14/Dec/2015).
[4] “Edas: Editor’s assistant,” https://edas.info/doc/, (Visited on
Besides a struggle with time when storing the assignments, 14/Dec/2015).
E-Conf was able to handle the whole process with no prob- [5] “Conftool: Conference and event management software,” http://www.
lems, and requiring minimal input from the organizer. conftool.net/, (Visited on 14/Dec/2015).
[6] “Open source event manager,” http://osem.io/, (Visited on 14/Dec/2015).
[7] “Welcome to epapers!” http://www.epapers.org/main/, (Visited on
VI. C ONCLUSIONS 14/Dec/2015).
[8] C. Caldera, R. Berndt, and D. W. Fellner, “Comfy –a conference
Conferences are an essential channel for the exchange of management framework,” Inf. Serv. Use, vol. 33, no. 2, pp. 119–
information, and as result there are hundreds of thousands of 128, Apr. 2013. [Online]. Available: http://dl.acm.org/citation.cfm?id=
conferences every year. Organizing a conference is a long and 2596858.2596867
[9] N. D. Mauro, T. M. A. Basile, and S. Ferilli, “Grape: An expert
hard process, involving hundreds of people. Most of the tasks review assignment component for scientific conference management
are repetitive and, therefore, ideal to delegate to a computer. systems.” in IEA/AIE, ser. Lecture Notes in Computer Science, M. Ali
There are already several systems to solve this problem, and F. Esposito, Eds., vol. 3533. Springer, 2005, pp. 789–798.
[Online]. Available: http://dblp.uni-trier.de/db/conf/ieaaie/ieaaie2005.
Conference Management Systems, although they have some html#MauroBF05
restrictions. The work shown in this document proposes E- [10] “Usage of server-side programming languages for websites,” https:
Conf, a new CMS. //w3techs.com/technologies/overview/programming language/all, (Vis-
ited on 05/Oct/2016).
To find which features were required, other CMSs were [11] B. Skvorc, “The best php framework for 2015:
analysed. They share the same overall set of features, which Sitepoint survey results,” http://www.sitepoint.com/
can be grouped in five main categories - submission, review- best-php-framework-2015-sitepoint-survey-results/, Mar. 2015, (Visited
on 02/Jan/2016).
ing, pre-conference, registration and public page. The set of
features on E-Conf, based on this analysis, covers three of this
five areas.
E-Conf was implemented as a web-app, using the most used
web technologies today, making E-Conf accessible to anyone
2 https://github.com/fzaninotto/Faker