4 - COSMOS - Public - Description
4 - COSMOS - Public - Description
Version 1.1
Date 14 January 2011
Status Draft Final
Approval
Version Date Name Function Signature
Previous versions
Version Date Author Summary of changes
Related documents
Attachments
1 Summary ................................................................................................ 3
1 Summary
The CWE project parties have selected COSMOS as the algorithm to calculate daily market
coupling results. COSMOS is a branch-and-bound algorithm designed, in collaboration with
N-SIDE, to solve the problem of coupling spot markets including block orders. It naturally
treats all technical and product requirements set by the CWE project, including step and
interpolated orders, flow-based network under PTDF representation, ATC links and DC
cables (possible with ramping, tariffs and losses), profiles block orders, flexible blocks
orders and linked block orders.
COSMOS outputs net export positions and prices on each market and each hour, the set of
executed orders, and the congestion prices on each tight network element. These outputs
satisfy all requirements of a feasible solution, including congestion price properties and the
absence of Paradoxically Accepted Blocks.
This document only describes the features that are currently in use in the CWE context,
though COSMOS already integrates many additional features such as those to be expected
in a context of product and geographic extensions.
As such, COSMOS algorithm is also used for local and coupling auctions such as:
- EPEX SPOT for CH;
- HUPX, OTE and OKTE for CZ-HU-SK coupling.
In those market areas, the COSMOS market constraints are defined in local market rules.
Market coupling is both a mechanism for matching orders on power exchanges (PXs) and
an implicit cross-border capacity allocation mechanism. Market coupling optimizes the
economic efficiency of the coupled markets: all profitable deals resulting from the
matching of bids and offers in the coupled markets of the PXs are executed; matching
results are however subject to capacity constraints calculated by Transmission System
Operators (TSOs) which may limit the flows between the coupled markets.
Market prices and schedules of the connected markets are simultaneously determined with
the use of the available capacity defined by the TSOs. The transmission capacity is thereby
implicitly auctioned and the implicit cost of the transmission capacity is settled by the price
differences between the markets. In particular, if no transmission capacity constraint is
active, then there is no price difference between the markets and the implicit cost of the
transmission capacity is null.
Under ATC, Market coupling relies on the principle that the markets with the lowest prices
export electricity to the markets with the highest prices. Between two markets, two
situations are possible: either the ATC is large enough and the prices of both markets are
equalized (price convergence), or the ATC is not sufficient and the prices cannot be
equalized. These two cases are described in the following examples.
Suppose that, initially, the price of market A is lower than the price of market B. Market A
will therefore export to market B, the price of market A will increase whereas the price of
market B decreases. If the ATC from market A to market B is sufficiently large, a common
price in the market may be reached (PA* = PB*). This first case is illustrated in Figure 1.
The other case, illustrated in Figure 2, happens when the ATC is not sufficient to ensure
price harmonization between the two markets. The amount of electricity exchanged
between the two countries is then equal to the ATC and the prices PA* and PB* are given
by the intersection of the purchase and sale curves. Exported electricity is bought in the
export area at a price of PA* and is sold in the import area at a price of PB*. The
difference between the two prices multiplied by the exchanged volume – i.e. the ATC – is
called “congestion revenue”, and is collected and used pursuant to article 6.6 of the
Regulation (EC) N° 1228/2003 of the European Parliament and of the Council of 26 June
2003 on condition for access to the network for cross-border exchanges in electricity.
3 COSMOS in a nutshell
This section describes the model and the algorithm that have been chosen to solve the
problem associated with the coupling of the day-ahead power markets in the CWE region.
Market participants submit orders on their respective power exchange. The goal is to
decide which orders to execute and which to reject and publish prices such that:
1
• The social welfare generated by the executed orders is maximal.
• Orders and prices are coherent.
• The power flows induced by the executed orders, resulting in the net positions do
not exceed the capacity of the relevant network elements.
COSMOS has been initially developed by BELPEX, in collaboration with N-SIDE, a company
specialized in optimization solutions based on operations research & modeling. The
purpose of this algorithm is to deal with the CWE coupling problem in way that allows
considering more general aspects of market coupling such as constraints that would arise if
coupling with neighboring markets. COSMOS is currently co-owned by APX-ENDEX, BELPEX
and EPEX SPOT.
• naturally treats standard and “new” order types with all their requirements,
• naturally handles both Available Transmission Capacity (ATC) and Flow-Based (FB)
network representations as well as possible alternative models and HVDC cable
features,
• implements specific curtailment rules for those cases where price boundaries are
not harmonized
• is not limited by the number of markets, orders or network constraints,
• finds quickly (within seconds) a very good solution in all cases (even with problems
with 350000 hourly orders and 1800 block orders in more than 10 markets),
• continues improving this initial solution until the time limit (e.g. 10 min) is
reached,
• generating several feasible solutions during the course of its execution,
• unless it can show that the mathematically optimal solution has been found (which
is most often the case), in which case it stops before the time limit.
In the two following sections, we detail which products and network models can be
handled by COSMOS. Section 6 gives a high-level description of how COSMOS works, and
section 7 provides additional information related to the functionalities and behaviors of the
algorithm.
4 Market constraints
Market constraints2 are those applying to the orders submitted to the exchanges. The list
presented hereunder proposes a set of all products available in at least one CWE
Exchange.
1
Social welfare is defined as: consumer surplus + producer surplus + congestion revenue
across the region. It is the objective function of COSMOS (see “objective” in technical
appendix 1).
2
See also “Market constraints” section in appendix 1 for a mathematical formulation of
these constraints.
• An Hourly Offer is rejected when the Market Clearing Price is lower than the offer
(lowest) price limit.
• An Hourly Bid is rejected when the Market Clearing Price is higher than the bid
(highest) price limit.
• An Hourly Offer is executed when the Market Clearing Price is higher than the offer
(highest) price limit.
• An Hourly Bid is executed when the Market Clearing Price is lower than the bid
(lowest) price limit.
• An Hourly Order may be partially executed if and only if the Market Price is equal
to the price limit of that order / is between the two price limits of that order.
• An Hourly Order is not executed for a quantity in excess of the volume limit
specified in the Order.
Block orders are neither partially nor paradoxically executed. Therefore, all orders can only
be either executed fully, or rejected fully. Because of this constraint – called the “fill or kill
3
constraint” - some block orders can be rejected even if they are in the money , in which
case they are called Paradoxically Rejected Blocks (PRB). On the contrary, no block orders
can be executed paradoxically (i.e. executed even if out of the money).
• A Block Offer is not executed when the average of the rounded Market Clearing
Prices over the relevant hours and weighted by the corresponding volume limits is
lower than the price limit of this order.
• A Block Bid is not executed when the average of the rounded Market Clearing
Prices over the relevant hours and weighted by the corresponding volume limits is
higher than the price limit of this order.
• A Block Order can only be executed at all hours simultaneously, for a quantity
equal to the hourly volume limits specified in the order.
5 Network Constraints
COSMOS is able to tackle the network constraints associated with several network
configurations4 (ATC-Based and Flow-Based – as well as with HVDC cables and ramping
constraints in case of further extensions).
3
A supply (respectively demand) order is said to be in the money if the submission price
of the order is below (resp. above) the average market price.
4
See also “Network constraints” section in appendix 1 for a mathematical formulation of
these constraints.
• Whenever there is a price difference between two areas, the ATC between these, if
any, is congested in the direction of the high price area.
In a flow-based representation of the network, the flows on a set of given critical network
elements are equal to the product of a PTDF (Power Transfer Distribution Factor) matrix
with the vector of the areas’ net positions. These (unidirectional) flows are limited by the
corresponding transmission capacities provided for each hour. Such PTDF constraints allow
representing explicitly all critical elements and security constraints, but would also support
more simplified network models.
6 Functioning of COSMOS
In this section we describe how COSMOS selects orders to be executed or rejected, under
the Market and Networks Constraints. The objective of COSMOS is to maximize the social
welfare, i.e. the total market value of the day-ahead auction.
The main difficulty in determining which orders to execute or reject comes from the fact
that block orders must satisfy the “fill or kill” property.
Without those block orders, the problem is much simpler to solve. Indeed, the problem can
then naturally be modeled as a Quadratic Program (QP)5, which can be routinely solved by
off-the-shelf commercial solvers6. The use of a commercial solver to directly solve this
Quadratic Program would then be the most efficient solution.
The presence of block orders in the order book however makes the problem substantially
more difficult. The problem with block orders can be formulated as a Mixed Integer
Quadratic Program (MIQP) allowing modeling the fill or kill condition of block orders. The
state-of-the-art method used to solve MIQP is called branch-and-bound7. COSMOS has
been designed as a dedicated branch-and-bound algorithm for solving the CWE Market
Coupling problem (the mathematical model of this problem is proposed in appendix 1).
6.1. Algorithm
COSMOS proceeds step by step.
At the first step, COSMOS solves a market coupling QP without fill or kill constraints, hence
allowing all block orders to be partially executed. By chance, the solution of this problem
might satisfy the fill or kill condition for all block orders and is therefore a feasible solution
5
A Quadratic Program (QP) is an optimization problem where an objective (function) of
the second order is to be optimized under linear constraints.
6
COSMOS uses the CPLEX solver.
7
For a more extensive discussion on the branch and bound technique, see for instance:
Integer Programming, Wolsey, 1998
of the CWE market coupling problem. In this case, the solution that has been found is the
optimal solution.
Otherwise, COSMOS gradually forces the partially executed block orders to be either fully
rejected or fully executed in subsequent steps, in order to obtain a solution to the CWE
market coupling problem which respects all fill or kill constraints.
2. COSMOS has produced a solution in which all block orders are either fully executed
or fully rejected (even those that were not forced to). In this case, COSMOS must
still check whether there exist prices that are compatible with this solution and the
constraints (which is done by verifying that all market and network constraints are
satisfied). Two cases can occur:
- Sub-case 2a: If such prices exist, COSMOS has found a feasible
solution. If this solution is better than the best one found so far8, it
is marked as such.
- Sub-case 2b: If no such prices exist, then a new step is created
with a transformed problem containing additional constraints to
exclude this non feasible solution.
During the course of its execution, COSMOS might sometimes increase the number of
steps that it has yet to consider (e.g. sub-cases b) or reduce it (sub-cases a). When there
remains none, this means that COSMOS has finished and has found the best possible
solution. Possibly, COSMOS will reach the time limit although there remain some partial
solutions that were not analyzed. In this case, COSMOS will output the best solution found
so far without being able to prove whether it is the very best possible one.
8
Or if it is the first feasible solution found.
- First node
- Solution value 3500
- Blocks 23 and 54 fractional
Case1a
Case2a
- third node, block 23 executed
- Solution value 3440
- all blocks integral, no prices
constraints added
Case2b
- second node, block 23 rejected
- Solution value 3050
- fourth node, block 23 executed + constraints
- all block integral, there exist prices
Case1b - Solution value 3300
feasible solution found
- block 30 fractional
- fifth node, block 23 executed, block 30 rejected + constraints - sixth node, block 23 executed, block 30 executed + constraints
- Solution value 3100 - Solution value 3000
- all block integral, there exist prices there cannot exist better solutions here!
better solution found!
NB: rounding the results imposes to accept some tolerances on constraints. Typically, this
tolerance is equal to the sum the precision ticks of all rounded values divided by two. For
instance, the sum of the net positions of all bidding areas must be zero, with a tolerance of
0.65 MWh (the sum of the net position ticks of all markets divided by two) for the CWE
MC.
In order to accommodate technical price boundaries and to compute coherent prices (in
the sense that they respect market and network constraints), COSMOS guarantees on the
one hand that market and network constraints are satisfied with respect to unrounded
prices. On the other hand, COSMOS also ensures that market constraints are satisfied
using rounded and within bound prices. Hence some network constraints are not checked
against rounded and within bound prices, but only against unrounded and possibly out of
bounds prices. This allows computing coherent prices while respecting the local price
boundaries and is currently only relevant in flow-based simulations.
COSMOS algorithm selects the solutions with the largest welfare, but discards during its
computation the solutions with paradoxically rejected blocks that are very deep in the
money. This is implemented to guarantee fairness, as this could only happen with blocks of
small volume.
Due to the combinatorial complexity of the problem, this is obviously not true for the
computing time to obtain the optimal solutions. Nevertheless, most of the instances were
solved at optimality in less than 10 minutes (which is the maximal time allowed in the CWE
coupling process), the remaining showing quite small distances to optimality after this time
limit.
6.6. Transparency
Generally speaking, COSMOS is based on sound and robust concepts and has a good
degree of transparency. In particular, COSMOS is perfectly clear and transparent as to
what are feasibility and optimality. More precisely, COSMOS will typically consider all
feasible solutions and choose the best one according to the agreed criterion (welfare-
maximization).
Also, COSMOS optimizes the total welfare, so that the chosen results are well explainable
to the market participants: published solutions are the ones for which the market value is
the largest. In addition, in order to avoid undesirable solutions, COSMOS will not output
solutions in which blocks that are unduly deep in the money are rejected paradoxically.
Furthermore, COSMOS uses a very general method for solving the market coupling
problems with “fill or kill” constraints. The ability of the algorithm to handle new products
or new requirements is thus excellent as long as the constraints remain of the same type
(linear constraints, with possible fill or kill conditions).
8.1. Sets
Let us first introduce some sets, and their indices:
- Bidding areas : m
- Hours: h
- Hourly orders: o
- Profile block orders: b
- Unidirectional ATC lines: l
8.2. Data
Here are the input data of the Coupling problem:
- qo is the quantity of hourly order o; it is considered positive for
supply orders and negative for demand orders
- p0o is the price at which an hourly order starts to be accepted
- p1o is the price at which an hourly order is fully accepted
(in the case of step orders, p0o= p1o; in the case of interpolated
supply orders, p0o ≤ p1o; in the case of interpolated demand order
p0o ≥ p1o)
- qb,h is the quantity of profile block b on period h; it is considered
positive for supply orders and negative for demand orders
- bidding area (b) is the bidding area in which profiled block order
b originates
- hours(b) is the set of hours on which profile block order b spans
- bidding area(o) is the bidding area in which hourly order o b
originates
- hour(o) is the hour on which hourly order o spans
- from(l) is the bidding area from which line l is originating
- to(l) is the bidding area to which line l is leading
- capacityl,h is the capacity on the ATC line l on period h
8.3. Variables
- 0 ≤ ACCEPTo ≤ 1 Acceptance of the hourly order
- ACCEPTb∈{0,1} Acceptance of the block order
- 0 ≤ FLOWl,h Flow on the line l at period h
- MCPm,h Market clearing price at market m and period h
- 0 ≤ ATC_PRICEl,h Congestion price of the capacity constraint of line l at period h
- The bidding area m must be in balance on hour h, meaning that the sell and the
import volumes must match the purchase and the export volumes:
∑ ACCEPT
o ,bidding area ( o ) = m ,
o ⋅ qo + ∑
b ,bidding area ( b ) = m ,
q ACCEPT = ∑ FLOWl ,h − ∑ FLOWl ,h
b, h b l , from (l ) =m l ,to ( l ) = m
hour ( o ) = h h∈hours ( b )
- The maximum flow on an ATC line l on hour h must not exceed the available
capacity.
FLOWl ,h ≤ Capacity l ,h
- The congestion price of an ATC line l on hour h can be positive only if the line is
congested.
ATC _ PRICE l , h > 0 ⇒ FLOWl , h = Capacity l , h
- The (positive) congestion prices of the capacity constraints must be equal to the
price difference between the destination and the source bidding areas of the ATC
line l on hour h:
- ATC _ PRICE l , h = MCPto(l ), h − MCPfrom(l ), h
8.6. Objective
The objective of the Market Coupling Problem is to maximize the total welfare:
p 0 + po1 po0 − po1
max ∑ − qo ACCEPTo o + (1 − ACCEPTo ) − ∑ pb ACCEPTb ∑ qb ,h
o 2 2 b h∈hours ( b )
Here the welfare is defined as the difference between the cumulative amount that the
buyers are ready to pay and the cumulative amount that the sellers want to be paid
(quantities are signed) over all markets and hours. This difference corresponds to the sum
of the surplus of the producers and the consumers, plus the congestion revenue.
Note that the objective is quadratic in the ACCEPTo variables (for interpolated orders).
8.7. Summary
COSMOS is an algorithm that generates several (i.e. virtually all) solutions which
satisfy the constraints listed under “Market Constraints”, and “Network
Constraints” and chooses the best one according to the criteria formulated under
“Objective”.
There are cases where multiple solutions of the Market Coupling Problem have identical
block selections and identical welfares but different clearing prices. In the most frequent
cases, this arises when offer and demand curves overlap on a vertical segment. These
cases are called price indeterminacies. The aim of this appendix is to describe the rules
implemented in COSMOS to choose amongst these clearing prices.
UB
MCP*
LB
In a nutshell, COSMOS will pick the mid-point of the intersection, that is MCP* in the figure
above. However, in a realistic problem with several markets coupled over several hours,
this solution might be non-feasible. This would be the case for example if MCP* invalidates
the selection of block (i.e. the price acceptance of block orders are not compatible with
MCP*). In such a case, COSMOS will pick up the price which is closed to the mid point and
which respects all other price constraints.
This section explains in details how COSMOS selects prices in indeterminate cases9.
9.1. Notations
Let’s note:
LBm,h: the lowest possible price in the market m on hour h for a defined net position Q*.
UBm,h: the highest possible price in the market m on hour h for a defined net position Q*.
MCPm,h: the (unrounded) Market Clearing Price of the market m on hour h.
The mid point of the intersection defined by [LBm,h, UBm,h] is equal to (UBm,h+LBm,h)/2.
9
Note that these rules are independent of the ATC or FB network model
2
UBm, h + LBm,h
min ∑ MCPm, h −
m, h 2
Subject to the following constraints:
• All market and network constraints
• B_ACCEPTo = 1 if the block order o is executed in the feasible solution
= 0 otherwise
Note that market constraints are checked against these prices as well before being
published
9.4. Summary
COSMOS lifts potential price indeterminacies by selecting the prices at the middle
of the feasible interval, taking into account the price constraints arising from
block selection and network configuration. Then Cosmos rounds and enforces
bounds on these prices before publication.
MCV*
In a nutshell, COSMOS will choose the solutions for which the traded volumes are
maximal, that is MCV* in the figure above.
In addition, these volume indeterminacy rules encompass curtailment rules which are rules
for special volume indeterminacy situations, namely at minimum/maximum prices. Indeed,
a secondary objective of COSMOS is to avoid that price taking orders - i.e. orders at
extreme prices - are not fully executed, this situation being called curtailment10.
However, because of volume indeterminacy at extreme prices, hierarchical rules have been
implemented:
10
These rules are independent of the ATC or FB network model
• Firstly: avoid curtailment by temporarily matching price taking order locally (see
10.2)
• Secondly: share curtailment (see 10.3).In order to provide fairness, the algorithm
will try to equally share the “curtailed volumes” between all markets.
• Thirdly: maximize traded volume (see 10.4)
Figure 5 Removing price taking volume from Demand and Supply curves
Figure 6 Removing price taking volume from Demand and Supply from a market
initially in curtailment
In practice, COSMOS will force the acceptance of price taking hourly supply (respectively
demand) orders by matching them with hourly demand (resp. supply) orders. Let us call
such a constraint “LocalMatchingpmin” (resp. “LocalMatchingpmax”).
∑ qd
d ∈Demand
LocalMatching p min : ACCEPTSupply p min ≥ min 1;
∑ qs
s∈Supply p min
• For price taking demand orders:
∑ qs
s∈Supply
LocalMatching p max : ACCEPTDemand p max ≥ min 1;
∑ qd
d ∈Demand p max
Where
• Supplypmin is the set of all supply orders at minimum price (i.e. price taking),
• Demandpmax is the set of all demand orders at maximum price (i.e. price taking).
The local matching constraints are enforced for each market individually prior to the
welfare maximization process. This rule helps finding block selections where curtailment is
reduced. Upon lifting volume indeterminacies, the local matching constraint is dropped in
order to allow a fair sharing of curtailment.
2. The solution must be fair in apportioning the curtailment between two (or more)
markets initially in curtailment.
To this end, the objective is defined as a quadratic function of the acceptance of the
curtailed orders:
min ∑ qo (1 − ACCEPTo )
2
o∈C
where C is the set of price taking orders of markets of whom the curtailment has to be
apportioned and qo is the volume of these price taking orders. In plain English, we
minimize the sum of the non-acceptance of price taking orders, measuring non-acceptance
by the square of the rejected proportion of the price taking order times its volume. This
objective function has several desirable properties:
• it is strictly convex in the space of price taking orders of markets in curtailment,
thus guaranteeing uniqueness of solution for these orders
• It will apportion curtailment according to the size of initial curtailment. For
example suppose two markets initially in curtailment by X and Y MW are coupled
(with infinite capacities) with a third market that can offer Z MW before being itself
in curtailment. Then X/(X+Y)*Z will be given to reduce curtailment in market X
and Y/(X+Y)*Z will be used to reduce curtailment in market Y. Therefore, each
market will see its curtailment decrease by the same proportion Z/(X+Y). Thus
price taking orders in each curtailed market are treated equally.
If two markets are curtailed to a different degree, the market with the least severe
curtailment (in proportion) would help the other reducing its curtailment, so that both
markets end up with identical curtailment ratios.
o∈H
Subject to the following constraints:
• All market and network constraints
• B_ACCEPTo = 1 if the block order o is executed in the feasible solution
= 0 otherwise
where H is the set of supply and demand step curves that are exactly at the market
clearing price of their respective markets and are not at Pmin or Pmax. Again, we want to
minimize the rejection of all (demand & supply) orders and apportion this equally across
markets.
10.5. Summary
COSMOS lifts potential volume indeterminacies with the following priorities:
1. Avoiding curtailment of price taking orders whenever possible
2. Minimizing and sharing the curtailment when unavoidable
3. Maximizing the volume traded in other cases.