7.3 Callback Stages and Example

Matpower defines five stages in the execution of a simulation where custom code can be inserted to alter the behavior or data before proceeding to the next stage. This custom code is defined as a set of “callback” functions that are registered via add_userfcn for Matpower to call automatically at one of the five stages. Each stage has a name and, by convention, the name of a user-defined callback function ends with the name of the corresponding stage. For example, a callback for the formulation stage that modifies the OPF problem formulation to add reserve requirements could be registered with the following line of code.

mpc = add_userfcn(mpc, 'formulation', @userfcn_reserves_formulation);

The sections below will describe each stage and the input and output arguments for the corresponding callback function, which vary depending on the stage. An example that employs additional variables, constraints and costs will be used for illustration.

Consider the problem of jointly optimizing the allocation of both energy and reserves, where the reserve requirements are defined as a set of nrz  fixed zonal MW quantities. Let Z
  k  be the set of generators in zone k  and R
 k  be the MW reserve requirement for zone k  . A new set of variables r  are introduced representing the reserves provided by each generator. The value ri  , for generator i  , must be non-negative and is limited above by a user-provided upper bound rmiax  (e.g. a reserve offer quantity) as well as the physical ramp rate Δi  .

0 ≤ ri ≤ min (rmiax ,Δi),  i = 1...ng

If the vector c  contains the marginal cost of reserves for each generator, the user defined cost term from (6.34) is simply

f (ˆx) = cTr.

There are two additional sets of constraints needed. The first ensures that, for each generator, the total amount of energy plus reserve provided does not exceed the capacity of the unit.

pig + ri ≤ pig,max,  i = 1 ...ng

The second requires that the sum of the reserve allocated within each zone k  meets the stated requirements.

    ri ≥ Rk,  k = 1 ...nrz

Table 7-2:Names Used by Implementation of OPF with Reserves



Matpower case struct


additional field in mpc containing input parameters for zonal reserves in the following sub-fields:


ng × 1  vector of reserve costs, c  from (7.3)


ng × 1  vector of reserve quantity upper bounds, ith  element is rmiax


nrz × ng  matrix of reserve zone definitions

zones(k,j) = {
 1if gen j belongs to reserve zone k (j ∈ Zk)
 0otherwise (j∈∕Zk)


nrz × 1  vector of zonal reserve requirements,  th
k  element is Rk  from (7.5)


OPF model object, already includes standard OPF setup


OPF results struct, superset of mpc with additional fields for output data


ng  , number of generators


name for new reserve variable block, ith  element is ri


name for new capacity limit constraint set (7.4)


name for new reserve requirement constraint set (7.5)

Table 7-2 describes some of the variables and names that are used in the example callback function listings in the sections below.

7.3.1 ext2int Callback

Before doing any simulation of a case, Matpower performs some data conversion on the case struct in order to achieve a consistent internal structure, by calling the following.

mpc = ext2int(mpc, mpopt);

All isolated buses, out-of-service generators and branches are removed, along with any generators or branches connected to isolated buses and the buses are renumbered consecutively, beginning at 1. All of the related indexing information and the original data matrices are stored in an order field in the case struct to be used later by int2ext to perform the reverse conversions when the simulation is complete.

The first stage callback is invoked from within the ext2int function immediately after the case data has been converted. Inputs are a Matpower case struct (mpc) freshly converted to internal indexing, a Matpower options struct mpopt,49 and any (optional) args value supplied when the callback was registered via add_userfcn. Output is the (presumably updated) mpc. This is typically used to reorder any input arguments that may be needed in internal ordering by the formulation stage. The example shows how e2i_field can also be used, with a case struct that has already been converted to internal indexing, to convert other data structures by passing in 2 or 3 extra parameters in addition to the case struct. In this case, it automatically converts the input data in the qty, cost and zones fields of mpc.reserves to be consistent with the internal generator ordering, where off-line generators have been eliminated. Notice that it is the second dimension (columns) of mpc.reserves.zones that is being re-ordered. See the on-line help for e2i_field and e2i_data for more details on what all they can do.

function mpc = userfcn_reserves_ext2int(mpc, mpopt, args)

mpc = e2i_field(mpc, {'reserves', 'qty'}, 'gen');
mpc = e2i_field(mpc, {'reserves', 'cost'}, 'gen');
mpc = e2i_field(mpc, {'reserves', 'zones'}, 'gen', 2);

This stage is also a good place to check the consistency of any additional input data required by the extension and throw an error if something is missing or not as expected.

7.3.2 formulation Callback

This stage is called at the end of opf_setup after the OPF Model (om) object has been initialized with the standard OPF formulation, but before calling the solver. This is the ideal place for modifying the problem formulation with additional variables, constraints and costs, using the add_var, add_lin_constraint, add_nln_constraint, add_quad_cost, add_nln_cost and add_legacy_cost methods of the OPF Model object.50 Inputs are the om object, the Matpower options struct mpopt and any (optional) args supplied when the callback was registered via add_userfcn. Output is the updated om object.

The om object contains both the original Matpower case data as well as all of the indexing data for the variables and constraints of the standard OPF formulation. See the on-line help for opf_model and opt_model for more details on the OPF model object and the methods available for manipulating and accessing it.

In the example code, a new variable block named R with ng  elements and the limits from (7.2) is added to the model via the add_var method. Similarly, two linear constraint blocks named Pg_plus_R and Rreq, implementing (7.4) and (7.5), respectively, are added via the add_lin_constraint method. And finally, the add_quad_cost method is used to add to the model a quadratic (actually linear) cost block corresponding to (7.3).

Notice that the last argument to add_lin_constraint and add_quad_cost allows the constraints and costs to be defined only in terms of the relevant parts of the optimization variable ˆx  . For example, the A matrix for the Pg_plus_R constraint contains only columns corresponding to real power generation (Pg) and reserves (R) and need not bother with voltages, reactive power injections, etc. As illustrated in Figure 7-1, this allows the same code to be used with both the AC OPF, where xˆ  includes Vm  and Qg  , and the DC OPF where it does not. This code is also independent of any additional variables that may have been added by Matpower (e.g. y  variables from Matpower’s CCV handling of piece-wise linear costs) or by the user via previous formulation callbacks. Matpower will place the constraint and cost matrix blocks in the appropriate place when it constructs the aggregated constraint and cost matrices at run-time. This is an important feature that enables independently developed Matpower OPF extensions to work together.

function om = userfcn_reserves_formulation(om, mpopt, args)

%% initialize some things
mpc = om.get_mpc();
r = mpc.reserves;
ng  = size(mpc.gen, 1);             %% number of on-line gens

%% variable bounds
Rmin = zeros(ng, 1);                %% bound below by 0
Rmax = r.qty;                       %% bound above by stated max reserve qty ...
k = find(mpc.gen(:, RAMP_10) > 0 & mpc.gen(:, RAMP_10) < Rmax);
Rmax(k) = mpc.gen(k, RAMP_10);      %% ... and ramp rate
Rmax = Rmax / mpc.baseMVA;

%% constraints
I = speye(ng);                      %% identity matrix
Ar = [I I];
Pmax = mpc.gen(:, PMAX) / mpc.baseMVA;
lreq = r.req / mpc.baseMVA;

%% cost
Cw = r.cost * mpc.baseMVA;          %% per unit cost coefficients

%% add them to the model
om.add_var('R', ng, [], Rmin, Rmax);
om.add_lin_constraint('Pg_plus_R', Ar, [], Pmax, {'Pg', 'R'});
om.add_lin_constraint('Rreq', r.zones, lreq, [], {'R'});
om.add_quad_cost('Rcost', [], Cw, 0, {'R'});

Figure 7-1:Adding Constraints Across Subsets of Variables
7.3.3 int2ext Callback

After the simulation is complete and before the results are printed or saved, Matpower converts the case data in the results struct back to external indexing by calling the following.

results = int2ext(results, mpopt);

This conversion essentially undoes everything that was done by ext2int. Buses are restored to their original numbering and all out-of-service or isolated generators, branches and buses are restored.

This callback is invoked from int2ext immediately before the resulting case is converted from internal back to external indexing. At this point, the simulation has been completed and the results struct, a superset of the original Matpower case struct passed to the OPF, contains all of the results. This results struct is passed to the callback, along with the Matpower options struct mpopt,51 and any (optional) args supplied when the callback was registered via add_userfcn. The output of the callback is the updated results struct. This is typically used to convert any results to external indexing and populate any corresponding fields in the results struct.

The results struct contains, in addition to the standard OPF results, solution information related to all of the user-defined variables, constraints and costs. Table 7-3 summarizes where the various data is found. Each of the fields listed in the table is actually a struct whose fields correspond to the named sets created by add_var, add_lin_constraint, add_nln_constraint, add_quad_cost, add_nln_cost and add_legacy_cost.

Table 7-3:Results for User-Defined Variables, Constraints and Costs
name description
results.var.val final value of user-defined variables
results.var.mu.l shadow price on lower limit of user-defined variables
results.var.mu.u shadow price on upper limit of user-defined variables
results.lin.mu.l shadow price on lower (left-hand) limit of linear constraints
results.lin.mu.u shadow price on upper (right-hand) limit of linear constraints
results.nle.lambda shadow price on nonlinear equality constraints
results.nli.mu shadow price on nonlinear inequality constraints
results.cost final value of legacy user costs
results.nlc final value of general nonlinear costs
results.qdc final value of quadratic costs

In the example code below, the callback function begins by converting the reserves input data in the resulting case (qty, cost and zones fields of results.reserves) back to external indexing via calls to i2e_field. See the help for i2e_field and i2e_data for more details on how they can be used.

Then the reserves results of interest are extracted from the appropriate sub-fields of results.var, results.lin and results.cost, converted from per unit to per MW where necessary, and stored with external indexing for the end user in the chosen fields of the results struct.

function results = userfcn_reserves_int2ext(results, mpopt, args)

%%-----  convert stuff back to external indexing  -----
%% convert all reserve parameters (zones, costs, qty, rgens)
results = i2e_field(results, {'reserves', 'qty'}, 'gen');
results = i2e_field(results, {'reserves', 'cost'}, 'gen');
results = i2e_field(results, {'reserves', 'zones'}, 'gen', 2);

r = results.reserves;
ng  = size(results.gen, 1);     %% number of on-line gens (internal)
ng0 = size(results.order.ext.gen, 1);   %% number of gens (external)

%%-----  results post-processing  -----
%% get the results (per gen reserves, multipliers) with internal gen indexing
%% and convert from p.u. to per MW units
[R0, Rl, Ru] = results.om.params_var('R');
R       = results.var.val.R * results.baseMVA;
Rmin    = Rl * results.baseMVA;
Rmax    = Ru * results.baseMVA;
mu_l    = results.var.mu.l.R / results.baseMVA;
mu_u    = results.var.mu.u.R / results.baseMVA;
mu_Pmax = results.lin.mu.u.Pg_plus_R / results.baseMVA;

%% store in results in results struct
z = zeros(ng0, 1);
results.reserves.R       = i2e_data(results, R, z, 'gen');
results.reserves.Rmin    = i2e_data(results, Rmin, z, 'gen');
results.reserves.Rmax    = i2e_data(results, Rmax, z, 'gen');
results.reserves.mu.l    = i2e_data(results, mu_l, z, 'gen');
results.reserves.mu.u    = i2e_data(results, mu_u, z, 'gen');
results.reserves.mu.Pmax = i2e_data(results, mu_Pmax, z, 'gen');
results.reserves.prc     = z;
for k = 1:ng0
    iz = find(r.zones(:, k));
    results.reserves.prc(k) = sum(results.lin.mu.l.Rreq(iz)) / results.baseMVA;
results.reserves.totalcost = results.cost.Rcost;
7.3.4 printpf Callback

The pretty-printing of the standard OPF output is done via a call to printpf after the case has been converted back to external indexing. This callback is invoked from within printpf after the pretty-printing of the standard OPF output. Inputs are the results struct, the file descriptor to write to, a Matpower options struct, and any (optional) args supplied via add_userfcn. Output is the results struct. This is typically used for any additional pretty-printing of results.

In this example, the out.all flag in the options struct is checked before printing anything. If it is non-zero, the reserve quantities and prices for each unit are printed first, followed by the per-zone summaries. An additional table with reserve limit shadow prices might also be included.

function results = userfcn_reserves_printpf(results, fd, mpopt, args)

%% define named indices into data matrices
    QC2MIN, QC2MAX, RAMP_AGC, RAMP_10, RAMP_30, RAMP_Q, APF] = idx_gen;

%%-----  print results  -----
r = results.reserves;
ng = length(r.R);
nrz = size(r.req, 1);
if mpopt.out.all ~= 0
    fprintf(fd, '\n=======================================================');
    fprintf(fd, '\n|     Reserves                                        |');
    fprintf(fd, '\n=======================================================');
    fprintf(fd, '\n Gen   Bus   Status  Reserves   Price');
    fprintf(fd, '\n  #     #              (MW)     ($/MW)');
    fprintf(fd, '\n----  -----  ------  --------  --------');
    for k = 1:ng
        fprintf(fd, '\n%3d %6d     %2d ', k, results.gen(k, GEN_BUS), ...
                    results.gen(k, GEN_STATUS));
        if results.gen(k, GEN_STATUS) > 0 && abs(results.reserves.R(k)) > 1e-6
            fprintf(fd, '%10.2f', results.reserves.R(k));
            fprintf(fd, '       -  ');
        fprintf(fd, '%10.2f     ', results.reserves.prc(k));
    fprintf(fd, '\n                     --------');
    fprintf(fd, '\n            Total:%10.2f              Total Cost: $%.2f', ...
        sum(results.reserves.R(r.igr)), results.reserves.totalcost);
    fprintf(fd, '\n');

    fprintf(fd, '\nZone  Reserves   Price  ');
    fprintf(fd, '\n  #     (MW)     ($/MW) ');
    fprintf(fd, '\n----  --------  --------');
    for k = 1:nrz
        iz = find(r.zones(k, :));     %% gens in zone k
        fprintf(fd, '\n%3d%10.2f%10.2f', k, sum(results.reserves.R(iz)), ...
                    results.lin.mu.l.Rreq(k) / results.baseMVA);
    fprintf(fd, '\n');

    %% print binding reserve limit multipliers ...

7.3.5 savecase Callback

The savecase is used to save a Matpower case struct to an M-file, for example, to save the results of an OPF run. The savecase callback is invoked from savecase after printing all of the other data to the file. Inputs are the case struct, the file descriptor to write to, the variable prefix (typically 'mpc.') and any (optional) args supplied via add_userfcn. Output is the case struct. The purpose of this callback is to write any non-standard case struct fields to the case file.

In this example, the zones, req, cost and qty fields of mpc.reserves are written to the M-file. This ensures that a case with reserve data, if it is loaded via loadcase, possibly run, then saved via savecase, will not lose the data in the reserves field. This callback could also include the saving of the output fields if present. The contributed serialize function52 can be very useful for this purpose.

function mpc = userfcn_reserves_savecase(mpc, fd, prefix, args)
%   mpc = userfcn_reserves_savecase(mpc, fd, mpopt, args)
%   This is the 'savecase' stage userfcn callback that prints the M-file
%   code to save the 'reserves' field in the case file. It expects a
%   MATPOWER case struct (mpc), a file descriptor and variable prefix
%   (usually 'mpc.'). The optional args are not currently used.

r = mpc.reserves;

fprintf(fd, '\n%%%%-----  Reserve Data  -----%%%%\n');
fprintf(fd, '%%%% reserve zones, element i, j is 1 iff gen j is in zone i\n');
fprintf(fd, '%sreserves.zones = [\n', prefix);
template = '';
for i = 1:size(r.zones, 2)
    template = [template, '\t%d'];
template = [template, ';\n'];
fprintf(fd, template, r.zones.');
fprintf(fd, '];\n');

fprintf(fd, '\n%%%% reserve requirements for each zone in MW\n');
fprintf(fd, '%sreserves.req = [\t%g', prefix, r.req(1));
if length(r.req) > 1
    fprintf(fd, ';\t%g', r.req(2:end));
fprintf(fd, '\t];\n');

fprintf(fd, '\n%%%% reserve costs in $/MW for each gen\n');
fprintf(fd, '%sreserves.cost = [\t%g', prefix, r.cost(1));
if length(r.cost) > 1
    fprintf(fd, ';\t%g', r.cost(2:end));
fprintf(fd, '\t];\n');

if isfield(r, 'qty')
    fprintf(fd, '\n%%%% max reserve quantities for each gen\n');
    fprintf(fd, '%sreserves.qty = [\t%g', prefix, r.qty(1));
    if length(r.qty) > 1
        fprintf(fd, ';\t%g', r.qty(2:end));
    fprintf(fd, '\t];\n');

%% save output fields for solved case ...