Callback Stages and Example

Matpower deﬁnes ﬁve stages in the execution of a simulation where custom code can be inserted to alter the behavior or data before proceeding to the next stage. This custom code is deﬁned as a set of “callback” functions that are registered via add_userfcn for Matpower to call automatically at one of the ﬁve stages. Each stage has a name and, by convention, the name of a user-deﬁned callback function ends with the name of the corresponding stage. For example, a callback for the formulation stage that modiﬁes the OPF problem formulation to add reserve requirements could be registered with the following line of code.

mpc = add_userfcn(mpc, 'formulation', @userfcn_reserves_formulation);

The sections below will describe each stage and the input and output arguments for the corresponding callback function, which vary depending on the stage. An example that employs additional variables, constraints and costs will be used for illustration.

Consider the problem of jointly optimizing the allocation of both energy and reserves, where the reserve requirements are deﬁned as a set of $nrz$ ﬁxed zonal MW quantities. Let $Z k$ be the set of generators in zone $k$ and $R k$ be the MW reserve requirement for zone $k$ . A new set of variables $r$ are introduced representing the reserves provided by each generator. The value $ri$ , for generator $i$ , must be non-negative and is limited above by a user-provided upper bound $rmiax$ (e.g. a reserve oﬀer quantity) as well as the physical ramp rate $Δi$ .

If the vector $c$ contains the marginal cost of reserves for each generator, the user deﬁned cost term from (6.34) is simply

There are two additional sets of constraints needed. The ﬁrst ensures that, for each generator, the total amount of energy plus reserve provided does not exceed the capacity of the unit.

The second requires that the sum of the reserve allocated within each zone $k$ meets the stated requirements.

ng × 1 — Table 7-2:Names Used by Implementation of OPF with Reserves

Table 7-2:Names Used by Implementation of OPF with Reserves

Table 7-2 describes some of the variables and names that are used in the example callback function listings in the sections below.

7.3.1 ext2int Callback

Before doing any simulation of a case, Matpower performs some data conversion on the case struct in order to achieve a consistent internal structure, by calling the following.

mpc = ext2int(mpc, mpopt);

All isolated buses, out-of-service generators and branches are removed, along with any generators or branches connected to isolated buses and the buses are renumbered consecutively, beginning at 1. All of the related indexing information and the original data matrices are stored in an order ﬁeld in the case struct to be used later by int2ext to perform the reverse conversions when the simulation is complete.

The ﬁrst stage callback is invoked from within the ext2int function immediately after the case data has been converted. Inputs are a Matpower case struct (mpc) freshly converted to internal indexing, a Matpower options struct mpopt,⁴⁹ and any (optional) args value supplied when the callback was registered via add_userfcn. Output is the (presumably updated) mpc. This is typically used to reorder any input arguments that may be needed in internal ordering by the formulation stage. The example shows how e2i_field can also be used, with a case struct that has already been converted to internal indexing, to convert other data structures by passing in 2 or 3 extra parameters in addition to the case struct. In this case, it automatically converts the input data in the qty, cost and zones ﬁelds of mpc.reserves to be consistent with the internal generator ordering, where oﬀ-line generators have been eliminated. Notice that it is the second dimension (columns) of mpc.reserves.zones that is being re-ordered. See the on-line help for e2i_field and e2i_data for more details on what all they can do.

function mpc = userfcn_reserves_ext2int(mpc, mpopt, args)

mpc = e2i_field(mpc, {'reserves', 'qty'}, 'gen');
mpc = e2i_field(mpc, {'reserves', 'cost'}, 'gen');
mpc = e2i_field(mpc, {'reserves', 'zones'}, 'gen', 2);

This stage is also a good place to check the consistency of any additional input data required by the extension and throw an error if something is missing or not as expected.

7.3.2 formulation Callback

This stage is called at the end of opf_setup after the OPF Model (om) object has been initialized with the standard OPF formulation, but before calling the solver. This is the ideal place for modifying the problem formulation with additional variables, constraints and costs, using the add_var, add_lin_constraint, add_nln_constraint, add_quad_cost, add_nln_cost and add_legacy_cost methods of the OPF Model object.⁵⁰ Inputs are the om object, the Matpower options struct mpopt and any (optional) args supplied when the callback was registered via add_userfcn. Output is the updated om object.

The om object contains both the original Matpower case data as well as all of the indexing data for the variables and constraints of the standard OPF formulation. See the on-line help for opf_model and opt_model for more details on the OPF model object and the methods available for manipulating and accessing it.

In the example code, a new variable block named R with $ng$ elements and the limits from (7.2) is added to the model via the add_var method. Similarly, two linear constraint blocks named Pg_plus_R and Rreq, implementing (7.4) and (7.5), respectively, are added via the add_lin_constraint method. And ﬁnally, the add_quad_cost method is used to add to the model a quadratic (actually linear) cost block corresponding to (7.3).

Notice that the last argument to add_lin_constraint and add_quad_cost allows the constraints and costs to be deﬁned only in terms of the relevant parts of the optimization variable $ˆx$ . For example, the A matrix for the Pg_plus_R constraint contains only columns corresponding to real power generation (Pg) and reserves (R) and need not bother with voltages, reactive power injections, etc. As illustrated in Figure 7-1, this allows the same code to be used with both the AC OPF, where $xˆ$ includes $Vm$ and $Qg$ , and the DC OPF where it does not. This code is also independent of any additional variables that may have been added by Matpower (e.g. $y$ variables from Matpower’s CCV handling of piece-wise linear costs) or by the user via previous formulation callbacks. Matpower will place the constraint and cost matrix blocks in the appropriate place when it constructs the aggregated constraint and cost matrices at run-time. This is an important feature that enables independently developed Matpower OPF extensions to work together.

function om = userfcn_reserves_formulation(om, mpopt, args)

%% initialize some things
define_constants;
mpc = om.get_mpc();
r = mpc.reserves;
ng  = size(mpc.gen, 1);             %% number of on-line gens

%% variable bounds
Rmin = zeros(ng, 1);                %% bound below by 0
Rmax = r.qty;                       %% bound above by stated max reserve qty ...
k = find(mpc.gen(:, RAMP_10) > 0 & mpc.gen(:, RAMP_10) < Rmax);
Rmax(k) = mpc.gen(k, RAMP_10);      %% ... and ramp rate
Rmax = Rmax / mpc.baseMVA;

%% constraints
I = speye(ng);                      %% identity matrix
Ar = [I I];
Pmax = mpc.gen(:, PMAX) / mpc.baseMVA;
lreq = r.req / mpc.baseMVA;

%% cost
Cw = r.cost * mpc.baseMVA;          %% per unit cost coefficients

%% add them to the model
om.add_var('R', ng, [], Rmin, Rmax);
om.add_lin_constraint('Pg_plus_R', Ar, [], Pmax, {'Pg', 'R'});
om.add_lin_constraint('Rreq', r.zones, lreq, [], {'R'});
om.add_quad_cost('Rcost', [], Cw, 0, {'R'});

7.3.3 int2ext Callback

After the simulation is complete and before the results are printed or saved, Matpower converts the case data in the results struct back to external indexing by calling the following.

results = int2ext(results, mpopt);

This conversion essentially undoes everything that was done by ext2int. Buses are restored to their original numbering and all out-of-service or isolated generators, branches and buses are restored.

This callback is invoked from int2ext immediately before the resulting case is converted from internal back to external indexing. At this point, the simulation has been completed and the results struct, a superset of the original Matpower case struct passed to the OPF, contains all of the results. This results struct is passed to the callback, along with the Matpower options struct mpopt,⁵¹ and any (optional) args supplied when the callback was registered via add_userfcn. The output of the callback is the updated results struct. This is typically used to convert any results to external indexing and populate any corresponding ﬁelds in the results struct.

The results struct contains, in addition to the standard OPF results, solution information related to all of the user-deﬁned variables, constraints and costs. Table 7-3 summarizes where the various data is found. Each of the ﬁelds listed in the table is actually a struct whose ﬁelds correspond to the named sets created by add_var, add_lin_constraint, add_nln_constraint, add_quad_cost, add_nln_cost and add_legacy_cost.

Table 7-3:Results for User-Deﬁned Variables, Constraints and Costs

name	description
results.var.val	ﬁnal value of user-deﬁned variables
results.var.mu.l	shadow price on lower limit of user-deﬁned variables
results.var.mu.u	shadow price on upper limit of user-deﬁned variables
results.lin.mu.l	shadow price on lower (left-hand) limit of linear constraints
results.lin.mu.u	shadow price on upper (right-hand) limit of linear constraints
results.nle.lambda	shadow price on nonlinear equality constraints
results.nli.mu	shadow price on nonlinear inequality constraints
results.cost	ﬁnal value of legacy user costs
results.nlc	ﬁnal value of general nonlinear costs
results.qdc	ﬁnal value of quadratic costs

In the example code below, the callback function begins by converting the reserves input data in the resulting case (qty, cost and zones ﬁelds of results.reserves) back to external indexing via calls to i2e_field. See the help for i2e_field and i2e_data for more details on how they can be used.

Then the reserves results of interest are extracted from the appropriate sub-ﬁelds of results.var, results.lin and results.cost, converted from per unit to per MW where necessary, and stored with external indexing for the end user in the chosen ﬁelds of the results struct.

function results = userfcn_reserves_int2ext(results, mpopt, args)

%%-----  convert stuff back to external indexing  -----
%% convert all reserve parameters (zones, costs, qty, rgens)
results = i2e_field(results, {'reserves', 'qty'}, 'gen');
results = i2e_field(results, {'reserves', 'cost'}, 'gen');
results = i2e_field(results, {'reserves', 'zones'}, 'gen', 2);

r = results.reserves;
ng  = size(results.gen, 1);     %% number of on-line gens (internal)
ng0 = size(results.order.ext.gen, 1);   %% number of gens (external)

%%-----  results post-processing  -----
%% get the results (per gen reserves, multipliers) with internal gen indexing
%% and convert from p.u. to per MW units
[R0, Rl, Ru] = results.om.params_var('R');
R       = results.var.val.R * results.baseMVA;
Rmin    = Rl * results.baseMVA;
Rmax    = Ru * results.baseMVA;
mu_l    = results.var.mu.l.R / results.baseMVA;
mu_u    = results.var.mu.u.R / results.baseMVA;
mu_Pmax = results.lin.mu.u.Pg_plus_R / results.baseMVA;

%% store in results in results struct
z = zeros(ng0, 1);
results.reserves.R       = i2e_data(results, R, z, 'gen');
results.reserves.Rmin    = i2e_data(results, Rmin, z, 'gen');
results.reserves.Rmax    = i2e_data(results, Rmax, z, 'gen');
results.reserves.mu.l    = i2e_data(results, mu_l, z, 'gen');
results.reserves.mu.u    = i2e_data(results, mu_u, z, 'gen');
results.reserves.mu.Pmax = i2e_data(results, mu_Pmax, z, 'gen');
results.reserves.prc     = z;
for k = 1:ng0
    iz = find(r.zones(:, k));
    results.reserves.prc(k) = sum(results.lin.mu.l.Rreq(iz)) / results.baseMVA;
end
results.reserves.totalcost = results.cost.Rcost;

7.3.4 printpf Callback

The pretty-printing of the standard OPF output is done via a call to printpf after the case has been converted back to external indexing. This callback is invoked from within printpf after the pretty-printing of the standard OPF output. Inputs are the results struct, the ﬁle descriptor to write to, a Matpower options struct, and any (optional) args supplied via add_userfcn. Output is the results struct. This is typically used for any additional pretty-printing of results.

In this example, the out.all ﬂag in the options struct is checked before printing anything. If it is non-zero, the reserve quantities and prices for each unit are printed ﬁrst, followed by the per-zone summaries. An additional table with reserve limit shadow prices might also be included.

function results = userfcn_reserves_printpf(results, fd, mpopt, args)

%% define named indices into data matrices
[GEN_BUS, PG, QG, QMAX, QMIN, VG, MBASE, GEN_STATUS, PMAX, PMIN, ...
    MU_PMAX, MU_PMIN, MU_QMAX, MU_QMIN, PC1, PC2, QC1MIN, QC1MAX, ...
    QC2MIN, QC2MAX, RAMP_AGC, RAMP_10, RAMP_30, RAMP_Q, APF] = idx_gen;

%%-----  print results  -----
r = results.reserves;
ng = length(r.R);
nrz = size(r.req, 1);
if mpopt.out.all ~= 0
    fprintf(fd, '\n=======================================================');
    fprintf(fd, '\n|     Reserves                                        |');
    fprintf(fd, '\n=======================================================');
    fprintf(fd, '\n Gen   Bus   Status  Reserves   Price');
    fprintf(fd, '\n  #     #              (MW)     ($/MW)');
    fprintf(fd, '\n----  -----  ------  --------  --------');
    for k = 1:ng
        fprintf(fd, '\n%3d %6d     %2d ', k, results.gen(k, GEN_BUS), ...
                    results.gen(k, GEN_STATUS));
        if results.gen(k, GEN_STATUS) > 0 && abs(results.reserves.R(k)) > 1e-6
            fprintf(fd, '%10.2f', results.reserves.R(k));
        else
            fprintf(fd, '       -  ');
        end
        fprintf(fd, '%10.2f     ', results.reserves.prc(k));
    end
    fprintf(fd, '\n                     --------');
    fprintf(fd, '\n            Total:%10.2f              Total Cost: $%.2f', ...
        sum(results.reserves.R(r.igr)), results.reserves.totalcost);
    fprintf(fd, '\n');

    fprintf(fd, '\nZone  Reserves   Price  ');
    fprintf(fd, '\n  #     (MW)     ($/MW) ');
    fprintf(fd, '\n----  --------  --------');
    for k = 1:nrz
        iz = find(r.zones(k, :));     %% gens in zone k
        fprintf(fd, '\n%3d%10.2f%10.2f', k, sum(results.reserves.R(iz)), ...
                    results.lin.mu.l.Rreq(k) / results.baseMVA);
    end
    fprintf(fd, '\n');

    %% print binding reserve limit multipliers ...
end

7.3.5 savecase Callback

The savecase is used to save a Matpower case struct to an M-ﬁle, for example, to save the results of an OPF run. The savecase callback is invoked from savecase after printing all of the other data to the ﬁle. Inputs are the case struct, the ﬁle descriptor to write to, the variable preﬁx (typically 'mpc.') and any (optional) args supplied via add_userfcn. Output is the case struct. The purpose of this callback is to write any non-standard case struct ﬁelds to the case ﬁle.

In this example, the zones, req, cost and qty ﬁelds of mpc.reserves are written to the M-ﬁle. This ensures that a case with reserve data, if it is loaded via loadcase, possibly run, then saved via savecase, will not lose the data in the reserves ﬁeld. This callback could also include the saving of the output ﬁelds if present. The contributed serialize function⁵² can be very useful for this purpose.

function mpc = userfcn_reserves_savecase(mpc, fd, prefix, args)
%
%   mpc = userfcn_reserves_savecase(mpc, fd, mpopt, args)
%
%   This is the 'savecase' stage userfcn callback that prints the M-file
%   code to save the 'reserves' field in the case file. It expects a
%   MATPOWER case struct (mpc), a file descriptor and variable prefix
%   (usually 'mpc.'). The optional args are not currently used.

r = mpc.reserves;

fprintf(fd, '\n%%%%-----  Reserve Data  -----%%%%\n');
fprintf(fd, '%%%% reserve zones, element i, j is 1 iff gen j is in zone i\n');
fprintf(fd, '%sreserves.zones = [\n', prefix);
template = '';
for i = 1:size(r.zones, 2)
    template = [template, '\t%d'];
end
template = [template, ';\n'];
fprintf(fd, template, r.zones.');
fprintf(fd, '];\n');

fprintf(fd, '\n%%%% reserve requirements for each zone in MW\n');
fprintf(fd, '%sreserves.req = [\t%g', prefix, r.req(1));
if length(r.req) > 1
    fprintf(fd, ';\t%g', r.req(2:end));
end
fprintf(fd, '\t];\n');

fprintf(fd, '\n%%%% reserve costs in $/MW for each gen\n');
fprintf(fd, '%sreserves.cost = [\t%g', prefix, r.cost(1));
if length(r.cost) > 1
    fprintf(fd, ';\t%g', r.cost(2:end));
end
fprintf(fd, '\t];\n');

if isfield(r, 'qty')
    fprintf(fd, '\n%%%% max reserve quantities for each gen\n');
    fprintf(fd, '%sreserves.qty = [\t%g', prefix, r.qty(1));
    if length(r.qty) > 1
        fprintf(fd, ';\t%g', r.qty(2:end));
    end
    fprintf(fd, '\t];\n');
end

%% save output fields for solved case ...

name	description
mpc	Matpower case struct
reserves	additional ﬁeld in mpc containing input parameters for zonal reserves in the following sub-ﬁelds:
cost	$ng × 1$ vector of reserve costs, $c$ from (7.3)
qty	$ng × 1$ vector of reserve quantity upper bounds, $ith$ element is $rmiax$
zones	$nrz × ng$ matrix of reserve zone deﬁnitions
	zones(k,j) = ${ 1if gen j belongs to reserve zone k (j ∈ Zk) 0otherwise (j∈∕Zk)$
req	$nrz × 1$ vector of zonal reserve requirements, $th k$ element is $Rk$ from (7.5)
om	OPF model object, already includes standard OPF setup
results	OPF results struct, superset of mpc with additional ﬁelds for output data
ng	$ng$ , number of generators
R	name for new reserve variable block, $ith$ element is $ri$
Pg_plus_R	name for new capacity limit constraint set (7.4)
Rreq	name for new reserve requirement constraint set (7.5)

7.3 Callback Stages and Example

7.3.1 ext2int Callback

7.3.2 formulation Callback

7.3.3 int2ext Callback

7.3.4 printpf Callback

7.3.5 savecase Callback