In this post, I use “behaviour” to mean a simple component feature that serves as a building block. There are many different behaviours, including variable capacity, fixed capacity, variable cost, fixed cost, unit commitment, ramping etc. The exhaustive list is available in the appendix of this post.

In capacity expansion and dispatch models, non-trivial components result from the application of multiple behaviours¹. For instance, Nosy uses explicit composition of one model archetype² and as many behaviours as required. This architecture makes the interaction of behaviours more explicit than other architectures, but this interaction has to be handled in every capacity expansion and dispatch model, one way or another.

Behaviours encapsulate a single facet of how a component exists or interacts with the rest of the system. However, designing their scope is not always trivial. Some of the behaviours only target a single notion e.g. fixed capacity, while others manage a small bundle of parameters e.g. unit commitment also manages minimum uptimes and downtimes. I wanted behaviours to be as small and conceptually orthogonal as possible, to minimize the complexity of the building blocks of components - at least for the user. Of course, splitting functionality into small, simple behaviours does not reduce the mathematical complexity; instead it shifts the responsibility for coordinating those behaviours from the user to the internal model-building logic.

Before discussing how behaviours interact, I also need to define the concept of port: a Port is a connection point on a component; it carries a specific carrier and represents its flow as a time series. In Nosy, most behaviours actually target specific ports of a component. For instance, the capacity is not defined for the whole component; it is applied specifically to one of its ports.

Nosy currently has 16 user-facing behaviours. The full list is provided in the appendix; their documentation is available here.

Let’s consider the simple example of a PV field with optimisable capacity. We will model it as

• a ProfileSource model archetype associated with a time series, which defines something that outputs a flow following a predefined pattern;
• a VariableCapacity behaviour that will create a capacity variable and tie it to the component’s capacity;
• a FixedCost behaviour that will generate an annualized investment cost proportional to capacity.

Let be the installed PV capacity, the availability profile, the output at time , and the annualized investment cost per unit of capacity. Together, the archetype and behaviours add the following terms to the model:

The ProfileSource provides , VariableCapacity creates and uses it to define the output profile, and FixedCost retrieves the same to construct the investment cost. The capacity must therefore be built before the fixed cost.

The actual implementation of this component is detailed here in the examples section of the docs.

Components can be more complex than that, and in detailed studies like the system cost study of Sweden (available here), they can routinely include up to 20 behaviours, often with multiple instances of fixed and variable costs.

The dependency in the PV example is small, but it establishes the general problem: behaviours cannot always be built in the order supplied by the user. Nosy therefore needs a deliberate build order.

There are multiple ways to manage such interactions. One elegant way would have been a dependency-resolution mechanism. For instance, I could have implemented the dependencies below:

abstract type Dependency end
abstract type Capacity <: Dependency end
requires(::FixedCost) = (Capacity,)
provides(::FixedCost) = () # no provision
requires(::VariableCapacity) = () # no requirement
provides(::VariableCapacity) = (Capacity,)

I could then build a DAG (hopefully) and topologically sort it. I like it, it looks fancy, but it is clearly overkill for Nosy. I decided to go against it because the total number of behaviours is small enough that another layer of abstraction was not necessary - the dependency graph is simple enough that it can be explicitly written.

I hard-coded the build order shown below³. For all model archetypes except ProfileSource and ProfileSink, the build order is:

CapacityMultiplier → capacity behaviours → Duration → UnitCommitment → all remaining behaviours

This build order is quite simple and understandable. The general flow is that capacities are required to design unit commitment, and the other behaviours’ constructors⁴ are independent of one another but may depend on capacities and unit commitment. The build order’s maintenance cost is explicit: whenever a new behaviour is added, its interactions with the existing behaviours must be examined before it can be placed in the sequence.

However, the build order itself does not explain why one behaviour must precede another. To reason about these interactions, I find it useful to distinguish two kinds of dependencies: hard requirements and conditional requirements. A hard requirement means that a behaviour is invalid when another behaviour is missing; a conditional requirement means that the behaviour remains valid on its own, but changes when another behaviour is present.

The table below names these dependencies directly.

These dependencies are handled at different stages: most hard requirements are checked during construction, whereas most conditional requirements alter constraint generation. Julia’s multiple dispatch makes these interactions straightforward to implement.

There are some more requirements than the table shows. For instance, UnitCommitment requires that a capacity defines the unit size; in addition, most behaviours are not properties of the whole component but of a port in particular, and the requirements are actually bound to the port and not the component. But for a simple mental model, the table is accurate enough.

All in all, for capacity expansion and dispatch, a behaviour should not be designed in isolation. Adding a behaviour requires identifying what it needs from other behaviours, as well as how its equations change when other behaviours are present, and vice versa.

Composing components from small behaviours makes the interactions explicit. In Nosy, the dependency graph is sparse and largely structured around capacity and unit commitment. As such, a fixed build order is easier to understand and maintain than a dependency-resolution mechanism. The explicit build order approach might eventually stop scaling if the number of behaviours grows substantially and new behaviours interact heavily with the others, but I doubt that this will happen. For now, the explicit build order, coupled with the requirement matrix, seems to provide the right level of abstraction: it is simple enough to inspect, and structured enough to manage components composed of many interacting behaviours.

To my knowledge, most capacity expansion studies use this formalism. However, this simplification comes with assumptions that are sometimes overlooked. As the name indicates, this formalism is static: it does not consider the path leading to the target year. Depending on the system, the path might be costly, or create vulnerabilities. The formulation also omits the dynamic aspect of investment and capacity expansion.

One difficult question is the choice of cost reference year. Models consume cost data such as overnight / connection / O&M / fuel / decommissioning / etc., in addition to time-related inputs such as lead time. When modelling year , should the model incorporate cost data associated with year , or a prior date corresponding to the non-modelled year when capacity is actually deployed? And should this hypothetical date be the same for each technology¹?

In my opinion, more so than correctness, one of the biggest drawbacks of modelling snapshots is that they do not answer the question “What should I do now?”, or “What happens if I don’t do X now”. A normative snapshot of year 2050 does not answer that question about short-term priorities. It might give clues, or a general direction, but it does not prioritize actions. This is detrimental to the usefulness of the modelling effort: the reason many policy reports exist is because decision-makers need / want to know what they could do now. This drawback is reinforced by discounting: investment decisions in the near term have higher present-value weight. In other words, their relative importance is higher than that of distant-future investments. However, they are indistinguishable in a snapshot model. Of course, reports can use expert judgement or complementary analysis to try and answer the question of near-term investments, but there is often a large leap between the model outputs and the conclusions, and the near-term recommendations are usually not backed directly by a modelling effort.

The “pathway” formalism explicitly models the steps leading up to the target year. In general, the assumption is perfect foresight across years: the decisions taken at year are made with year in sight². This is the formulation used by EnergyPathway.jl, a pathway model built on top of the composable energy system modelling toolkit Nosy.jl.

Modelling a scenario under the pathway formalism starts with the definition of the decision years. Assuming you wish to model the transition between 2026 and 2040, a first attempt might be to explicitly model every year in between, which would mean 15 snapshots in total. As the number of nonzeros in the optimization matrix scales roughly linearly³ with the number of snapshot-years, the size of the pathway problem might quickly become too large.

An interesting concept is the decision year. It is generally possible to define a subset of the pathway years, and only model the snapshots in this subset. If a year is modelled as a snapshot, investment decisions can be made in that year; otherwise the year has no investment decision, and its capacity mix and other metrics are carried forward from the previous decision year: EnergyPathway.jl treats capacities and other parameters as piecewise constant between decision years. Using this concept, it is generally feasible to reduce the size of the problem. Also, using a higher density of decision years in the near-term future and reducing the density in the more distant future helps focus the model on short-term decisions.

Nevertheless, the introduction of decision years is not trivial. For instance, there might not be a decision year matching the exact end of a component’s lifetime. In this case, tricks must be used to minimize the impact. In EnergyPathway.jl, such tricks rely on evaluating residual economic value relative to the mismatch between the closest snapshot and the expected end of life. That is to say: pathway modelling is not exempt from assumptions either.

As mentioned above, the general assumption for pathway models is perfect foresight across years. This is a strong hypothesis. As with Snapshot models, sensitivity studies should be run in order to understand the most important parameters and their impact. Modelling to Generate Alternatives (MGA) and near-optimality analysis are also very useful for pathways. Pathways also interact with multi-stage stochastic programming in an intuitive manner: decision years can also be used to model years in which new information becomes available, even though the complexity of the optimization model grows quickly with the number of decision years.

Let’s illustrate both snapshot and pathway formalisms on the reduction of CO2 emissions of a facility that consumes electricity. The full example, including hypotheses and results, is available in EnergyPathway.jl’s examples. We assume a constant electricity consumption from the facility, associated with progressively more stringent CO2 emissions targets. The target year is 2040. The snapshot only models year 2040, the pathway models a subset of years: [2026, 2030, 2035, 2040]. Available technologies are: PV, battery storage, CCGT. For simplicity, the costs are assumed to be constant between 2026 and 2040. The optimization objective of the snapshot formulation is the annualized cost; the objective of the pathway approach is the discounted system cost from 2026 to 2060, 2060 being the economic end of the project.

The capacity expansion is shown in the graph below. After 2040, the pathway capacities are carried forward to 2060 to show the interpolation interval. The x-axis extends until 2060 because it is the pathway model’s economic horizon. However, the last decision year is 2040, so no additional capacity is deployed after 2040.

The capacities of all three technologies are different between the snapshot and the final pathway decision year. The PV and battery capacity are quite similar between both, but the CCGT capacity is larger in the end-state of the pathway than in the snapshot. The pathway minimizes the cost by deploying more CCGT when CO2 emissions constraint is looser, then retires some but not all of that additional capacity relative to the snapshot solution. The pathway’s last decision year, in isolation, is more costly than the snapshot, but is cost-effective under the dynamic approach.

It is not surprising that the end state of the pathway does not match the snapshot. The reason is that the capacity deployment has to be optimal for the whole pathway, not just for the end state. In particular, the weight of the end-state is not necessarily dominant because of discounting. The economic end date of the project (here 2060) can be used to tune the weight of the future against the present in the objective. In particular, extending the economic project end date gives more weight to the carried-forward final system, and would generally be expected to move the pathway result closer⁴ to the snapshot optimization, assuming the same assumptions are used to model the lifetime of components.

All in all, pathway modelling is a powerful tool to analyze the effects of a transition, including the management of existing capacity, the timing for deployment of new technologies based on cost and demand, and time-dependent policies. In particular, it is very helpful to identify the decisions that should be made in the near term. However, even more than snapshots, pathways are very data-intensive. The amount of assumptions and data required to build a pathway is very high. But this is not necessarily a disadvantage relative to snapshots: it is rather that the snapshot formalism, and the interpretation of snapshots, imply many implicit hypotheses that rarely are validated, whereas pathways force the modeller to explicitly formulate such hypotheses.

Let’s first define a node as an abstract, passive place where the balance of flows is enforced. A node can represent many things including a bidding zone in a regional power system, a hydrogen pipe in a hydrogen system, or a wire in an electric device. One node is only associated with one carrier. The active counterpart of the node is the component, which generates and/or consumes flows. There are many systems that require modelling of multiple nodes. In one system, multiple nodes can represent multiple geographic or market zones, multiple parts of a system supporting different carriers, etc.

A convention that I like, and that is applied in Nosy, is that components are connected only to nodes, and vice versa.

Let’s model an electricity and hydrogen system composed of PV, battery, electricity consumption, electrolyser, long-term hydrogen storage, and hydrogen consumption. The capacity of every component is an investment decision, except for consumption. Let’s also assume, for simplicity, that the electricity and hydrogen consumption profiles are flat. The full example is available in Nosy’s documentation.

The levels of the battery and hydrogen storage during the first week of June are shown below, both evaluated using hourly time meshes. The data is synthetic, so the shape of the curves is more regular than with real-world data, but it still captures the essence of the dynamics.

Let’s analyze the behaviour of storage in both nodes. On the electricity side, the characteristic time scale is shorter than a day, with large-amplitude hourly variations. In particular, the battery is used at full capacity every day. It is a well-known result that using a coarse time description when modelling systems that include PV leads to underestimating the optimal battery storage capacity, as shown in the previous post on heterogeneous time meshes.

On the hydrogen side, the H2 storage characteristic time is mostly seasonal, as it inherits from the seasonality of the PV capacity factor: hydrogen storage is charged in sunny seasons, and discharged in seasons with less sun. The intra-day variations exist, but are quite limited. In particular, slightly misrepresenting the intra-day variations would likely not lead to underestimating the optimal hydrogen storage capacity. The only times it may misrepresent the optimal capacity is when the level is close to zero or maximum level, which is expected to happen a very limited number of times per year, and may therefore have a reasonably low impact. Based on that, we can infer that representing with satisfactory precision the hydrogen node might not actually require a fine hourly time mesh. However, we do not wish to use a coarser mesh for the whole model, as that would misrepresent the role of battery storage.

Some models, including Nosy offer the possibility to use different time meshes. Nosy supports defining a different mesh to each component and node. The tricky part is enforcing node balance across meshes; it is enforced via a constraint: the components’ meshes must be at least as fine¹ as the meshes of the nodes they are connected to. Under Nosy’s formalism, the behaviour of components is not different from a single-mesh formalism: each component is associated with a single mesh, and all its flows, levels and inner variables will follow the same mesh. However, the behaviour of nodes is less intuitive, as they must manage the interaction of flows associated with multiple time meshes to apply the balance constraint. Effectively, the flows are projected on the node’s mesh. But the projection must make sense: it is defined so that the average flow is conserved for each timestep of the coarse mesh. Defining another projection e.g. sampling the finer mesh over the coarse would break the consistency as the flow balance seen from the component and the node sides would differ.

Nodes generally do not carry variables, but their mesh defines the way the node balance is enforced. To be more accurate, the gain is not coming from modelling nodes with a coarser mesh, as the number of non-zeros in the optimization matrix is not really impacted by modelling nodes on a coarse or fine mesh. The gain is coming from modelling components using a coarser mesh. A component using a coarser mesh generally has a lower number of optimization variables and constraints. The exact reduction depends on the nature of the component, and the way its behaviour is encoded, but the reduction of the number of variables is often almost proportional with the ratio of timesteps in the meshes.

In the present case, we can try using two meshes: is hourly and uses 4-hour time steps, and assign them the following way:

• for PV, battery, electricity consumption, electricity node and electrolyser.
• for H2 storage, H2 consumption and the H2 node.

In particular, the electrolyser uses the fine mesh, although it is connected to both nodes. This is an interesting feature, because it keeps the temporal consistency of everything happening on the electricity side, while being compatible with the coarser hydrogen node.

We run the model using the combination of fine and coarse meshes (denoted as “mixed”) and compare the results with a reference case using only the fine mesh. Solve time is based on the HiGHS solver.

First, the results show that capacities of PV, battery storage, and electrolyser are the same as for the fine mesh, within the reported precision. The hydrogen storage capacity evaluated using a coarse mesh for hydrogen, is reasonably close to the reference (-0.2%). The objective value also is very close to the reference case (-0.0001%). The number of variables and constraints is reduced thanks to the coarser mesh, which reduced the solve time by 37%.

Depending on the goals of the study, this can be a desirable simplification, with substantial reduction of the solve time at the cost of a slight optimistic bias in the objective value.

Using a coarser mesh for specific components and nodes is a viable strategy to improve the tractability of a model. Some systems are more favourable to this approximation than others. The main characteristics of components and nodes that are likely to be compatible with a coarser mesh are the domination of a long-term pattern over short-term variations. Using multiple meshes is also compatible with heterogeneous time meshes; using a combination of both takes advantage of the fact that resolution is more valuable at some times and in some parts of the system.

Solve time in MILP problems is difficult to estimate, in particular because they’re extremely difficult problems solved with extremely smart heuristics, and finding a solution feels like luck. Nevertheless, reducing the number of variables without introducing complexity generally decreases the solve time¹.

One large source of variables is the time scope. Modelling one full year with hourly resolution generates 8760 variables per series. 100 such series are almost sufficient to reach one million variables, which is one of the fuzzy limits to start calling a problem “difficult”.

In what follows, I will call “time mesh” the vector of time step durations. For instance, for the hourly modelling of a year, the time mesh will be with length 8760. There are cases where modelling time with a coarser time mesh is a good approximation, and there are multiple degrees of complexity associated with that task².

I will also call “collapsing” the time mesh the process of reducing the number of time steps while maintaining the total duration. The reflections below are based on Nosy, which uses the power formalism³, under which power (or other flows) are linearly interpolated over the intervals. In particular, for a model under power formalism, the effect of mesh collapsing is quite transparent: a time step longer than one hour will simply imply that power will be linearly interpolated for the duration above one hour.

The first example of time mesh reduction is using a uniformly coarser time mesh. For instance, modelling 1 hour time steps, one can model 2 hours time steps: with length 4380. It may be a valid approximation, however the precision of some phenomena at the hourly scale will be degraded, such as variability of PV production, or ramping constraints.

A more involved example detailed in Nosy’s docs is using heterogeneous time meshes. The system is composed of PV, battery storage and consumption with a flat profile, on a one-year horizon; it performs greenfield capacity expansion and dispatch to match consumption. It is modelled under three time meshes: one is a fine hourly resolution over the full year, the second is a coarser two-hour resolution and the last one is heterogeneous and uses 18 hourly time steps during the day and 3 two-hour time steps at night. The results are in the table below.

The results show that the coarse time mesh returns different results and objective from the fine time mesh, but the heterogeneous time mesh returns the same results (within reported precision) as the fine mesh (same objective value, same capacity expansion for battery and PV). This result is interesting because the problem with heterogeneous time mesh has a 12.5% lower number of variables, directly resulting in a reduced solve time⁴ (solve time based on using HiGHS solver).

A plausible reason is that at night, the variation from one hour to the next of PV production is zero, therefore the system works in a static manner at night, the only exception being the battery storage level getting lower. Static behavior is compatible with using larger time steps at some hours without degrading the results.

The example above was a very clear case of when to use heterogeneous time meshes. However, all problems are not simple enough to contain periods with static behavior. In particular, when modelling country-scale power systems, consumption is not flat, even at night. However, a static behavior is not a required condition for the time collapsing to work.

One target is to find when the system is most stable. But that alone is generally not sufficient. Let’s assume that the problem is a cost-optimization. When you collapse hours, you also introduce bias, because complex systems rarely are purely static. And the bias you introduce is related to the marginal cost of operating the system at the time of collapsing, or more precisely, the shadow price. Intuitively, the reason is that the shadow price is the local derivative of your system’s objective against variation of consumption minus production. Should your approximation misrepresent the actual consumption or production, the bias generated will, in the vicinity of the optimal solution, be proportional to the shadow price.

Price can be used as an indicator for when to collapse time steps. When the price is stable, it hints at good stability, and when the absolute value of the price is low, it hints at low impact of bias. Obviously, having access to the shadow price generally implies that you have already solved a version of the problem. But there are many workflows where the solution is not one-shot, but incrementally refined by screening out unused technologies, removing relaxation etc. If you can afford to solve one initial LP relaxation of the MILP problem, you can derive the (shadow) price from it and use it to feed a time mesh collapsing process that will facilitate the next stages of the workflow, e.g. solving the MILP with the heterogeneous time mesh.

This is generally not a straightforward process though. One reason is that there are likely to be multiple nodes with shadow prices, so it is necessary to design an aggregation method that works for the case. Also, since these are two distinct targets (lowest prices and most stable prices), you will need to find the correct recipe, which also is problem-dependent.

A practical workflow is:

1. Solve an LP relaxation at full hourly resolution.
2. Extract nodal shadow prices and relevant exogenous profiles.
3. Aggregate prices across nodes if the model has several relevant shadow prices.
4. Identify candidate periods to collapse, for instance adjacent hours in the 20% quantile of most stable prices while price remains under a threshold.
5. Collapse only those intervals, with a maximum allowed step length.
6. Solve the MILP on the resulting heterogeneous mesh.

Obviously validation of the results has to be part of the workflow too.

I will summarize this post by saying that the value of temporal resolution is uneven. Heterogeneous time meshes exploit that unevenness: they keep detail where the system is sensitive, and collapse periods where the approximation is unlikely to affect investment or dispatch decisions.

A few years back, at the OECD Nuclear Energy Agency, I built POSY¹, a capacity expansion and dispatch energy systems model written in Julia. My most serious programming project before that was a discrete event simulation written in Java. However, as I was new to linear programming models, I was advised by colleagues at another institution to take a look at the programming language Julia, in particular because of its very strong operations research framework JuMP. In fact, two of our closest partners at the time were using Julia.

I was initially confused by Julia’s deliberate avoidance of class-style inheritance: abstract types cannot hold fields, concrete types cannot be subtyped, and much of what inheritance would usually do is handled through multiple dispatch². I found an alternative to inheritance with traits³, even though their implementation felt slightly awkward to me.

Anyway, driven by my previous experiences, I started building POSY using a top-down approach. My target was to build a system where components interact, so I implemented each component type. I found some general patterns that helped me organize the components into families that component types would implement. For instance, the “storage” family would be used for both battery storage and large hydro reservoirs with natural intake. The “dispatchable” family would work for non-level-based plants whose output is an optimization variable, e.g. nuclear plants and gas plants.

At the beginning, that worked very well. I could rapidly prototype the first components and make them interact together in simple least-cost optimizations. So I decided to keep the model, improving it little by little when I needed a new component or a new constraint.

Unfortunately, I happened to need many more components, and I patched the oddities onto the edifice. Every family accumulated optional features until it stopped meaning one thing. For instance, the “storage” abstract type was initially designed for batteries, but soon enough I had to model large hydro reservoirs, so I added fields describing natural intake in addition to charging variables. Another example is that unit commitment became part of every dispatchable plant, but was quite often unused for tractability reasons. Some types became severely bloated, a substantial amount of the code was duplicated, and the model became difficult to maintain and expand without breaking things. My architecture was inadequate. In the end, the model did the job, the project was successful, but I was relieved that POSY would not have to grow any bigger.

Retrospectively, I believe that my mental model was suboptimal because it was overfitting my previous projects: the objects my models had to manipulate generally were more similar. I was used to dealing with complexity in algorithms, not heterogeneity of objects. Also, for some reason, “composition over inheritance” had never really clicked with me at that time.

A few years later, I needed to work again on a similar problem, except even more heterogeneous. In particular, the system I studied with POSY contained a single node (a single electricity market bidding zone, surrounded by neighbours modelled as interconnector capacity and price time series). This time, the system contained 4 nodes with capacity expansion, plus 10-15 other power nodes without capacity expansion, and just as many hydrogen nodes. After estimating the volume of work it would take to adapt POSY to this new problem, I made the decision to implement a different model. The new model would bear the same equations, but would have to be more flexible, more extendable.

Having worked for a few years with Julia, I decided to write the new model in a Julian way, by being composition-heavy. This time, I designed the model bottom-up: I wanted to isolate every little piece of business logic, and allow these pieces to interact at a small scale. This led me to design a new grammar for modelling components in energy systems (or any component-based commodity-flow graph, actually). The vocabulary of this grammar is: model archetype, joint flow, behavior, and component.

Model archetypes are a small family of simple models, each having only one function. They create the “core” variables. For instance: dispatchable source (one output flow that is a series of independent variables), profile source (one output flow is one variable multiplied by a time series), basic storage (one input flow, one output flow, one level, all are a series of independent variables), basic converter (convert an input flow into an output flow), etc. There are about ten model archetypes so far, and I have not felt the need to add more after multiple full-scale studies with the model. What’s important is that they don’t do anything complex: they just do one thing. They don’t have intrinsic capacity, cost, unit commitment, or any non-trivial feature.

Joint flows constitute the second element of components. Joint flows are additional flows that can be:

• Linked to other component flows via functions, such as CO2 emissions by a gas plant proportional to the energy output;
• Partly linked to other component flows, such as heat production by a CHP plant, that may be linked or not to the power output, depending on type;
• Independent from other component flows, such as constant consumption of electricity by a system, independently of its use.

Joint flows are useful because they make it easy to extend the interaction of a model. Their existence is one of the reasons why it was possible to keep the set of model archetypes small.

Model archetypes and their joint flows are complemented by behaviors. Behaviors are a family of mostly orthogonal features, which refine how the model operates. They do not add flows, but they may add constraints, auxiliary variables, etc. They will generally constrain how the flows behave. For instance, the “capacity” behaviors add a notion of capacity (fixed capacity generally bounds the flow, variable capacity ties it to a variable, etc.). UnitCommitment is another example of behavior; it exists to constrain one of the flows of a model to obey the unit commitment rules (state, startup and shutdown, min downtime and uptime etc.). Another family of behaviors is costs, which add different types of cost to a model (fixed cost relative to capacity, variable cost relative to flows).

So, to summarize, a component is made of the three following items:

• Exactly one model archetype, which accomplishes a basic, essential, minimal function.
• As many joint flows as necessary, which add more input or output flows to the component.
• As many behaviors as necessary, which refine how the component works.

As an example, one can define a gas plant fleet the following way, for a capacity expansion and dispatch study in a CO2-limited setting:

• DispatchableSource model archetype, generating the variables for variable output power.
• LinkedJointFlow generating a CO2 joint flow, proportional to the power output.
• VariableCapacity behavior, which defines that the capacity of the gas plant can be optimized within boundaries.
• If one needs accurate flexibility analysis, UnitCommitment behavior will give state as well as startup / shutdown durations to the plant; and Ramping (up and down) will constrain the ramping of the free output flow.
• Multiple FixedCost behaviors: an annualized investment cost, an annualized decommissioning cost, and a fixed O&M cost proportional to the capacity.
• Multiple VariableCost behaviors: a variable O&M cost proportional to the output flow, a fuel cost proportional to the output flow, and one targeting the CO2 flow to represent the impact of CO2 price.
• Optionally ReserveUp and ReserveDown to model contribution of the fleet to operational reserves.

This grammar is very helpful because it provides a very high level of freedom in defining components, and does so in a very contained way. Choosing composition over inheritance allowed this parsimonious approach: the component owns just as many refinements as necessary.

All is not perfectly simple though. In addition to interacting with flows, behaviors may also interact with other behaviors. This is not without challenges, because adding one behavior requires assessing the interaction with all existing behaviors.

A simple example of behaviors interacting is that capacity-related costs (e.g. overnight cost) will look for a capacity behavior matching the requirements of the cost behavior. If there is none, the behavior cannot be applied and throws an error.

A more complex example is how Ramping behavior equations depend on whether the target component bears a UnitCommitment behavior connected to the ramping target flow. If not, ramping is applied to the flow directly. But if it does, ramping must be applied to the free-flow of the unit commitment, not the flow itself.

The implementation of these interactions is based on precedence rules. The instantiation of behaviors to a component follows a pre-established order that guarantees that any behavior possibly influencing another behavior is instantiated before the behavior that is influenced.

So far all the behaviors required for modelling power systems or industrial systems with usual requirements have been implemented and the interactions seem clean: no production bugs related to behavior handling have happened in a long time.

I think that the current scope of model archetype, joint flows and behaviors works well for Nosy. One question remaining, at least conceptually, is how far that concept of separation could have been pushed. In particular, I think that model archetypes now have the least clearly defined scope. I mentioned that a model archetype is something simple with exactly one essential function. It is likely that the current model still is a suboptimal level of abstraction: perhaps it could have been something less “physical”, and more mathematical, such as a set of variables, e.g. “vector of independent variables”, or “one variable multiplied with a numeric vector”, that one would then connect to an empty box either as an input or an output. And some features currently implemented as model archetypes, e.g. “having a level” (for storage), could be shifted to behaviors. In the future I might experiment with that thought, even though the current API of Nosy has the advantage of being relatively intuitive to modelers.

NB: for the sake of simplicity, I deliberately omitted the concept of port from the description of components. Ports are the way flows are emitted or targeted. For more information, please refer to Nosy’s documentation.

For most of my career, I’ve had to optimize. I’ve changed domains (energy systems, nuclear materials management, supply chain) and method (LP/MILP, multiobjective, metaheuristics with surrogate modelling) multiple times, but optimization has stayed a constant.

Some disciplines are even seemingly shaped by optimization. For instance, energy systems economics revolves around optimization - in general cost optimization. Capacity expansion analysis studies how to deploy and use capacity so as to reach the least-cost strategy.

Optimization is conceptually simple: you’re looking for the minimum of a function. In general, one looks for a combination of parameters that will lead to minimizing a function. But sometimes, one mistakenly interprets that as looking for “the best” combination of parameters.

Mathematically, this is a mistake because optimization does not generally guarantee a unique argmin unless the problem has specific uniqueness properties. There is no guarantee that you have one “best” minimizer - you may have infinitely many. If you look at the function , you will immediately notice that all tuples will minimize the function.

This is an important point from a practical perspective too. Depending on the method you use, you may only obtain one result, that is an arbitrary minimizer, in our example – for instance (3,3). What the method said is that (3,3) minimizes the function. What one would incorrectly infer is that (3,3) is “the” (implicitly unique or significant) optimal tuple.

Some optimization methods tackle this point in particular. For instance, lexicographic optimization lets you define a ranked set of objectives , and minimizes , then minimizes among the minimizers of and so on. This method conceptually solves the problem of mathematically equivalent solutions - even though in practice making this claim is actually difficult, as you would need a full objective hierarchy to discriminate solution. In addition, to my knowledge, these methods generally don’t provide the full set of solutions at each step.

But stopping here would have us miss the more important point, which originates from the fact that systems modelling is full of bias and uncertainty. If you have two tuples of solutions to a problem, and one barely has an infinitesimally lower objective value than the other, should you legitimately exclude the worse solution?

There are many cases where your system has near-optimal solutions, with parameters quite different from the optimal solution. Depending on how you define near-optimality, a near-optimal solution is just as desirable as an optimal one. The existence of this near-optimal space is fundamental, because it characterizes choice: instead of just a tuple of optimal parameters, you may choose among a range of tuples.

Near-optimality is relative to your problem. Maybe you accept that 1% variation is good enough. Maybe the uncertainty of your input data is such that 2% variation on your objective value is well below the possible impact of uncertainty. Maybe you actively decide to trade some optimality for choice and flexibility.

In any case, the near-optimal set, with relative tolerance , is a valuable piece of information (for a strictly positive minimization objective):

A quick algorithm I generally try when studying a system is to scan the range of the near-optimal set, one meaningful variable at a time. Let’s assume I’m interested in optimizing a function , but I also want to know if I have some leeway when choosing , or if my system is quickly deoptimized when changes from its optimal solution value. First, I evaluate and obtain . Then, I allow a tolerance , and run the following optimizations:

and the same thing for max . Interestingly, this snippet is actually quite close to what lexicographic algorithms actually do under the hood, but with a different philosophy.

In my experience, this is enlightening when applied to capacity expansion problems. When you solve your problem, you obtain what is commonly referred to as the “optimal” capacity – actually a minimizer capacity. But because of uncertainty and modelling assumptions, costs with one percent difference are indistinguishable to me. For example, overnight costs tend to be both impactful parameters and highly uncertain ones. So after I optimize the cost, I generally ask next “what is the range of technology x so that cost is near-optimal”. What I’ve seen is that if your system is behaving smoothly, with progressively more expensive flexibility resources (in particular: if you’re modelling a well-interconnected network), and technologies with different advantages (e.g. baseload for one, lower LCOE for the other), then the range of near-optimal capacity for a competitive technology can be wide.

Recently, I had to evaluate least-cost capacity mixes at country scale, through a capacity expansion study of multiple scenarios. I applied this technique to check the range of the near-cost-optimal deployment of two technologies (nuclear and onshore wind for that case). Then, I fixed the capacity of one technology at sampled values within its near-optimal range, and re-optimized the system. This gave me the matching capacity of the other technology. It turns out that there was a substantially wider range of near-optimal solutions than expected. What was also quite interesting is that the system’s behavior would shift from net exporter to net importer while shifting from one technology to the other - which also showed that both being an importer or an exporter could be cost-optimal strategies, which leaves room for choice.

The concept of “choice” matters here. This type of choice is not about applying a subjective personal preference. It is the way one can factor in additional information not present in the modelling effort. For instance, for strategic reasons, the country would prefer being an exporter than an importer. Or they have uncertainty regarding future energy consumption by industry, and are looking for a mix compatible with a rapidly rising demand. They can also make a no-regrets decision to invest now up to the lower bound for that technology’s capacity in the near-optimal set, and decide whether to invest more or not later.

All in all, the point is that optimization is often too small an object to support real-world decisions. In systems with uncertainty, modelling assumptions, and strategic constraints outside the objective function, the near-optimal set is often where the useful information lives. Optimization tells you the minimum value of your function and gives you one minimizer, but near-optimality tells you what choices remain.

And then, based on the state, you can derive the output or any operation that the unit is allowed to perform.

In addition to its conceptual simplicity, what is also great about it is that it is very modular at its core, and many small bits can be added based on what you need to model:

• Clustered unit commitment (CUC) models a fleet of units, using numbers instead of booleans.
• Variable flow can be added, for instance when the unit has a minimum output level (e.g. minimum power is 20% of nominal power).
• Startup and shutdown procedures can be added, by adding flows at that are proportional to and .
• Startup cost or flows can be added, again proportional to .
• You can model other states than on and off, e.g. by replacing with a vector of different ways to with .
• Minimum uptime and downtime can be enforced by constraining, at each timestep , the switches that occurred during the preceding duration : with as the number of units in the fleet and (the formula is an approximation, the actual expression is a little too cumbersome to be presented here).

Unfortunately, UC also tends to make the optimization process significantly more difficult for the solver.

First, it adds many variables: at least 3 per timestep as you need to express the state and the state switches, that is 26280 variables if you assume one year of 8760 hours. Some of the unit commitment flavors add many constraints to your problem (in particular lengthy minimum uptime/downtime in my experience). Also, it is common to model multiple technologies using the UC formalism in the same problem. And finally, when you use unit commitment, you’re generally doing it because you need some degree of accuracy, which correlates it with using integer constraints.

However, there are ways to simplify the problem.

Let’s take the example of modelling a nuclear plant. The example I’m taking is simplified to focus on the relevant points, but I will still emphasize some strong hypotheses:

• No ramping for load following (not all operators perform load following with nuclear, although it’s technically feasible on various plant types).
• No short-duration shutdown for load-following purposes.
• Minimum downtime is 30 days (720 hours) for refuelling. We assume that the operator can schedule refuelling optimally during the year.
• Most large PWR reactors have to refuel every 1.5 years, that we interpret as “the sum of refuelling shutdowns over the year must be greater than or equal to 2/3 of the number of units”

One of the methods I recently implemented in the energy systems modelling toolkit Nosy is to reduce the number of timesteps at which you declare UC variables or enforce UC constraints. It consists in applying masks to the startup and shutdown variable vectors so that we only allow starting or shutting down at specific timesteps. For instance: instead of allowing your plants to shut down for refuelling at any hour, they only are allowed to do so once a week e.g. the first hour of every week.

Why are you allowed to use masks? There are multiple answers, depending on the problem but also on the results. Maybe it makes no sense for the operator to optimize the refuelling schedule on a sub-weekly basis: once teams and materials are ready, delaying the operations seems economically detrimental. Maybe because analysis of the results shows that the solution you obtain is reasonably close to not using masks, which constitutes an a posteriori justification of your hypothesis. Maybe you don’t need a perfect solution, and you will use masks as a tool to make an in-house heuristic to get an acceptable solution quick enough. One last reason is that maybe this is not your “reference” run, just an intermediate step that you are using to warm-start your non-simplified problem. A low-cost high-quality initialization gives you a head start for your non-simplified reference run. Still, using masks shrinks the feasible region: you need to make sure you have a valid reason for using it.

Let’s assume that we make a shutdown mask vector , length 8760 hours, so that for any multiple of 24*7 (shifted by one hour, so that is true, which will simplify our reasoning later), false otherwise. We only create shutdown variables when is true, meaning that we create 53 shutdown variables (h in {1, 169, …, 8737}) instead of 8760.

For the startup, it is slightly more complex. We assumed that the minimum downtime was 30*24=720 hours. We have to make sure that we can allow the reactors to start exactly 720 hours after shutdown. That gives a 720-hour shift of . We can also give some more freedom e.g. add mid-week startup slots, but not in this simple case. All in all, we now have 48 startup variables (h in {721, 889, …, 8617}).

So we’ve managed to reduce the number of variables for unit commitment from 26280 difficult and generally integer variables to 8861 (8760 for state + 53 for shutdown + 48 for startup), which is already a substantial reduction. However, we still have to apply the most interesting part of the mask simplification: the state propagation.

Let’s assume that we don’t even know the number of units, because that could also be a variable. At the first hour, the state is variable . We’ve made above the hypothesis that reactors are allowed to shut down at the first hour. Because of the equality , we don’t know the state at hour 2 before solving the problem, so let’s use variable to represent it. Now, the reactors aren’t allowed to start up or shut down at hour 2, because of the masks. This tells us that by definition, is equal to . Instead of a variable + constraint duo, we can just express this by referencing as , and never introduce a variable. Same thing for and so on, until you meet a timestep where reactors are either allowed to start up or shut down. If you assume that time is linear and not circular, that gives 102 different states (one plus as many as startup and shutdown variables, as they never switch at the same time), and the rest are simply propagated.

All in all, we’re now down to 203 variables (53+48+102) instead of 26280. We’ve also avoided many constraints. In particular, the extremely long expressions usually generated by the minimum downtime constraint have been substantially simplified, because many of the duplicate state switches have now transformed into just a coefficient before a small number of switches.

The implementation of the masks in Nosy is here.

When not to use this? There are many situations where you will need more granularity, so this method must always be used with reasonable impact assessment. The weekly mask is a particularly aggressive presolving tactic, which works well for demonstration purposes but may be ill-suited for your everyday problem. You can tune this method to adapt it to the shape of your problem and make it less blind and extreme. For instance, your business knowledge may give you a good idea of when it’s interesting to start up or shut down units - maybe you’ve inferred this from a preliminary LP solve, maybe you’ve read it in the price time series, or maybe you have another idea of when it’s possible or impossible to simplify the behavior of your unit.

Using masks and state propagation generally is one of the first steps I make when I need to make a difficult unit-commitment problem more tractable. Depending on your problem, it can make the problem significantly easier. However, it is not a free approximation, and will likely require efforts in both business logic and validation.