In this work, we perform an extensive theoretical and experimental analysis of the characteristics of five of the most prominent algebraic modelling languages (

Many real-world problems are routinely solved using modern optimization tools (e.g. Abhishek

Mathematical modelling is the process of translating real-world business problems into mathematical formulations whose theoretical and numerical analysis can provide insight, answers, and guidance beneficial for the originating application (Kallrath,

Models written in an AML are known for the high degree of similarity to the mathematical formulation. This aspect distinguishes AMLs from other types of modelling languages, like object-oriented (e.g.

From the late 1970s, many AMLs were created (e.g.,

The remainder of the paper is organized as follows. In Section

The first algebraic modelling languages, developed in the late 1970s, were game-changers. They allowed separating the model formulation from the implementation details (Kallrath,

Essential characteristics of a modern AML could be defined in the following way (Kallrath,

problems are represented in a declarative

Specifying the problem’s properties: space, set of constraints and optimality requirements.

way;there is a clear separation between problem definition and the solution process;

there is a clear separation between the problem structure and its data.

The algebraic expressions are useful in describing individual models and describing manipulations on models and transformations of data. Thus, almost as soon as AML became available, users started finding ways to adapt model notations to implement sophisticated solution strategies and iterative schemes. These efforts stimulated the evolution within AMLs of scripting features, including statements for looping, testing, and assignment (Fourer,

For this review, we have chosen five AMLs:

AMLs which won 2012 INFORMS Impact Prize award

the popularity of AMLs based on NEOS Server model input statistics for the year 2020;

NEOS Server. Solver Access Statistics:

open-source options that are attractive for the academic society or in situations where budgets are tight.

We have chosen to include

In the following section, we investigate how each of the chosen languages meets the requirements for a modern AML defined in the previous section. The websites of the AMLs and vendor documentation are used for this comparison. Any support of the identified features and capabilities are validated against the documentation the suppliers of the AMLs provide. Besides, an in-depth survey concluded by Robert Fourer in Linear Programming Software Survey (Fourer,

We start by analysing how selected AMLs satisfy the three essential characteristics defined in the previous Section

Overviewof AMLs features.

Feature | AIMMS | AMPL | GAMS | JuMP | Pyomo | |

Modelling | Independent | Yes | Yes | Yes | Yes | Yes |

Scripting | Yes | Limited | Limited | Yes | Yes | |

Data | Input | Yes | Limited | Limited | Yes | Yes |

Manipulation | Yes | No | No | Yes | Yes | |

Solvers | Total | 13 | 47 | 35 | 14 | 25 |

Global | 1 | 4 | 9 | 2 | 1 | |

LP | 8 | 17 | 21 | 9 | 10 | |

MCP | 2 | 1 | 5 | 1 | 1 | |

MINLP | 3 | 6 | 15 | 3 | 6 | |

MIP | 5 | 14 | 16 | 6 | 8 | |

MIQCP | 5 | 5 | 20 | 3 | 4 | |

NLP | 6 | 19 | 17 | 7 | 10 | |

QCP | 6 | 9 | 21 | 6 | 6 | |

Presolving | Yes | Yes | No | No | No | |

Visualization | Yes | No | No | No | No | |

License | General | Paid | Paid | Paid | Free | Free |

Academic | Paid | Free | Free | Free | Free |

Next, in Table

When it comes to solver support,

Presolving capabilities are only available in

It is important to conclude that

For the first practical comparison of the selected AMLs, a classical Dantzig Transportation Problem was chosen (Dantzig,

The transportation problem formulated as a model in all five considered AML is compared based on the following criteria:

model size in bytes;

model size in the number of code lines;

model size in the number of language primitives used;

model instance creation time.

Since the transportation problem is a linear programming (LP) type of problem, we have chosen to measure the model instance creation time as the time needed to export a concrete model instance to

AIMMS Wikipedia page;

AMPL model in GNU Linear Programming Kit;

GAMS Model Library;

JuMP Examples;

Pyomo Gallery.

Comparison of transportation problem models.

Criteria | AIMMS | AMPL | GAMS | JuMP | Pyomo |

Size in bytes | 2229 | 683 | 652 | 632 | 1235 |

Lines of code | 68 | 24 | 31 | 18 | 29 |

Primitives used | 9 | 5 | 8 | 4 | 6 |

A comparison of the sample Transportation Problem model’s characteristics in all reviewed AMLs is given in Table

all optional comments, explanatory texts, and documentation are removed;

all empty lines are excluded;

parts of the code responsible for calling the solver and displaying results are omitted;

while counting AML primitives generic functions (

It can be seen from Table

While comparing the number of language primitives required to create a model,

The creation time of the transportation problem model instance defined in each AMLs was used to measure a model loading. The process was done in the following steps:

loading model instance from a problem definition written in the native AML;

exporting model instance to MPS format;

measuring total execution time;

investigating characteristics of an instance model.

Characteristics of the created transportation model instances.

Characteristic | AMPL | GAMS | JuMP | Pyomo |

Constraints | 6 | 6 | 6 | 6 |

Non zero elements | 13 | 19 | 13 | 13 |

Variables | 7 | 7 | 7 | 7 |

The characteristics of the created model instances can be seen in Table

Total time of consecutive transportation model instance creation runs.

No. of runs | AMPL | GAMS | JuMP | Pyomo |

1 run | 30 ms | 170 ms | 28341 ms | 720 ms |

10 runs | 220 ms | 1730 ms | 32199 ms | 7280 ms |

100 runs | 2130 ms | 16490 ms | 58151 ms | 79600 ms |

In Table

Start-up cost consists of the precompilation and caching time required to prepare JuMP environment.

of a few seconds even for the smallest instances. In our case, only the initialization of theAll examined AMLs support all types of traditional optimization problems; however, it is unclear how efficiently each AML can handle large model loading and what optimizations are applied during model instance creation. It would also be of great value to analyse how each of the modelling languages performs within an area of the specific type of optimization problems (e.g. linear, quadratic, nonlinear, mixed-integer). To give such a comparison and thoroughly examine characteristics of AMLs, a more extensive benchmark involving much larger optimization problem models is needed. Therefore, a large and extensive library of sample optimization problems for the analysed AMLs has to be used.

We have chosen the GAMS Model Library

The generated library was used to determine the amount of time each modelling system requires to create a model instance of a particular problem. We wrote

We have chosen to exclude sample problems with conversion errors from the benchmark (more information about them in Appendix

Average model instance creation time.

Average large model instance creation time

Benchmark methodology, hardware, and software specifications can be found in our GitHub repository (Jusevičius and Paulavičius,

We have observed that the average difference between

Thus, we can conclude that out of the reviewed AMLs,

A similar time benchmark of the model instance creation has already been conducted (Dunning

First of all, our and their time benchmark methodologies differ. In comparison, we are trying to be solver independent and instruct AML to export the generated model instance to NL file, Dunning

In the following, we have conducted two benchmarks – one as described in the original article and the second one using our method of exporting to a NL file. Results of the benchmarks can be seen in Tables

JuMP benchmark using Dunning

Model | AMPL | GAMS | Pyomo | JuMP (DIRECT) | JuMP (CACHE) |

lqcp-500 | 2093 | 2271 | 17000 | 17388 | 37317 |

lqcp-1000 | 8075 | 11995 | 139201 | 24590 | 44575 |

lqcp-1500 | 18222 | 38813 | 322604 | 39370 | 66566 |

lqcp-2000 | 32615 | 93586 | 575406 | 57597 | 88833 |

fac-25 | 407 | 480 | 7442 | 17517 | 39245 |

fac-50 | 2732 | 2884 | 43106 | 21331 | 47735 |

fac-75 | 9052 | 12422 | 150550 | 31582 | 57432 |

fac-100 | 20998 | 29144 | 393200 | 61326 | 93129 |

Before running the benchmarks, we had to rewrite some parts of the sample

Additionally, we wanted to test JuMP’s new abstraction layer’s performance for working with solvers called MathOptInterface.jl (MOI). Therefore, we have tried both

JuMP benchmark using export to NL method (in milliseconds).

Model | AMPL | GAMS | Pyomo | JuMP |

lqcp-500 | 2716 | 3265 | 39988 | 20424 |

lqcp-1000 | 10503 | 14394 | 161404 | 80578 |

lqcp-1500 | 25402 | 49822 | 307121 | 483268 |

lqcp-2000 | 42780 | 125564 | >10 min | >10 min |

fac-25 | 409 | 502 | 9420 | 8163 |

fac-50 | 2837 | 2993 | 43087 | 31799 |

fac-75 | 10879 | 13457 | 143286 | 219548 |

fac-100 | 23474 | 32128 | 328170 | >10 min |

Overall, both benchmarks confirmed our observation that

The reported differences between our and the original benchmark (Dunning

Another performance-related feature of AMLs is the ability to presolve a problem before providing it to the solver. The presolver can preprocess problems and simplify, i.e. reduce the problem size or determine the unfeasible problem. Only two of the reviewed algebraic modelling languages provide presolving capabilities –

To assess

AMPL model presolving.

Type | # models | # infeasible | Presolved (%) | Constraints reduced (%) | Variables reduced (%) |

CNS | 4 | 0 | 100.00% | 14.63% | 31.39% |

DNLP | 5 | 0 | 20.00% | 0.00% | 7.41% |

LP | 57 | 0 | 36.84% | 17.81% | 9.66% |

MCP | 19 | 0 | 89.47% | 47.00% | 8.56% |

MINLP | 21 | 1 | 61.90% | 16.32% | 9.30% |

MIP | 61 | 0 | 60.66% | 19.06% | 11.50% |

MIQCP | 5 | 2 | 60.00% | 0.00% | 2.38% |

MPEC | 1 | 0 | 100.00% | 50.00% | 0.00% |

NLP | 101 | 2 | 47.52% | 9.71% | 11.55% |

QCP | 10 | 0 | 60.00% | 7.10% | 2.55% |

RMIQCP | 2 | 0 | 0.00% | 0.00% | 0.00% |

Total | 286 | 5 | 52.80% | 18.42% | 10.73% |

A detailed report of the presolving applied to the specific model can be seen in the benchmark section of our GitHub repository (Jusevičius and Paulavičius,

We can conclude that

To evaluate if

We have chosen to solve the models using

Gurobi Optimizer Reference Manual:

It is important to note that both

Detailed

Summary of AMPL presolve impact on solving.

Iteration-wise | Time-wise | Iteration-wise (%) | Time-wise (%) | |

Positive | 37 | 67 | 26.43% | 47.86% |

Neutral | 74 | 40 | 52.86% | 28.57% |

Negative | 29 | 33 | 20.71% | 23.57% |

During this benchmark, 6 models failed to be solved due to solver limitations. Two models were deemed to be not feasible, and two were solved during the

As mentioned earlier, both

AMPL presolve impact with Gurobi presolve on.

Iteration-wise | Time-wise | Iteration-wise (%) | Time-wise (%) | |

Positive | 18 | 39 | 28.57% | 61.90% |

Neutral | 34 | 0 | 53.97% | 0.00% |

Negative | 11 | 24 | 17.46% | 38.10% |

AMPL presolve impact with Gurobi presolve off.

Iteration-wise | Time-wise | Iteration-wise (%) | Time-wise (%) | |

Positive | 33 | 44 | 54.10% | 72.13% |

Neutral | 10 | 0 | 16.39% | 0.00% |

Negative | 18 | 17 | 29.51% | 27.87% |

As seen once comparing these results in Tables

As we can see from the benchmarks, presolving done by AML has inconclusive effects on the actual problem solving both iterations and time-wise. However, a positive impact is always more significant than the negative one, and it especially becomes evident once the solver does not have or use its problem presolving mechanisms. This allows us to conclude that the presolving capability of AML is an important feature of a modern algebraic modelling language. We can also advise choosing AML having presolving capabilities in cases the solver used to solve the problem does not have its presolving mechanism.

From the research, we can conclude that

We plan to continue our research in this area by including performance comparison on automatic differentiation, adding even more large problems to our test library, and exploring the potential of parallel model instance creation support by AMLs.

Data underlying this article can be accessed on Zenodo at

The automated shell script

The script has two execution modes – one for converting a single model and another for converting all GAMS Library models. An example of the transportation problem from GAMS Model Library

Transportation problem converted to GAMS scalar model.

Example of a GAMS Convert error.

At the time of writing, there were 423 models in the GAMS Model Library. Out of them, we eliminated 66 models using

Transportation problem defined in AMPL format.

Transportation problem defined in GAMS format.

Transportation problem defined in Pyomo format.

Transportation problem defined in JuMP format.