Projects / Programmes
Incorporating real-world problems into the benchmarking of multiobjective optimizers
| Code |
Science |
Field |
Subfield |
| 2.07.07 |
Engineering sciences and technologies |
Computer science and informatics |
Intelligent systems - software |
| Code |
Science |
Field |
| P170 |
Natural sciences and mathematics |
Computer science, numerical analysis, systems, control |
| Code |
Science |
Field |
| 1.02 |
Natural Sciences |
Computer and information sciences |
Algorithm benchmarking, real-world optimization problems, multiobjective optimization
Researchers (1)
| no. |
Code |
Name and surname |
Research area |
Role |
Period |
No. of publicationsNo. of publications |
| 1. |
24894 |
PhD Tea Tušar |
Computer science and informatics |
Head |
2017 - 2019 |
209 |
Organisations (1)
| no. |
Code |
Research organisation |
City |
Registration number |
No. of publicationsNo. of publications |
| 1. |
0106 |
Jožef Stefan Institute |
Ljubljana |
5051606000 |
90,695 |
Abstract
Most real-world optimization problems found in engineering, medicine and other domains are inherently multiobjective, for example, searching for trade-off solutions of high quality and low cost. Very often we have no insight into the functions to be optimized, which means we are faced with the so-called black-box optimization problems. Hundreds of algorithms for solving black-box multiobjective optimization problems exist and we can determine their value only through testing their performance on benchmark problems. COCO (Comparing Continuous Optimizers), an open-source platform for benchmarking optimization algorithms on a black-box setting, has been recently extended to include also bi-objective problems. This was an important advancement in the benchmarking methodology used in multiobjective optimization, since COCO employs the more interpretable fixed-target (instead of the fixed-budget) approach to performance assessment.
However, the test functions from COCO, like other available benchmark suites in multiobjective optimization, are at their core still synthetic and do not incorporate some important properties of real-world problems, such as mixed variables, constraints, expensive evaluations and asynchronous evaluations of objectives. Since only a few real-world multiobjective optimization problems are freely available for research purposes, there is an urgent need to collect real-world problems, models of real-world problems and more realistic synthetic benchmark problems into an open benchmark suite that could be used by any researcher in multiobjective optimization.
The idea of this project is to extend the COCO platform by incorporating real-world problems and their properties in order to bridge the gap between research and application in multiobjective optimization. More specifically, the project will: (1) extend COCO’s problem formulation and its fixed-target any-time performance assessment methodology to accommodate specificities of real-world problems, (2) provide a new multiobjective benchmark suite consisting of real-world problems, models of real-world problems and synthetic problems containing features of real-world problems, and (3) design an algorithm capable of solving problems from the new real-world benchmark suite and make its results available for future comparisons.
The new real-world benchmark suite will be included in the future BBOB (Black-box Optimization Benchmarking) workshops at renowned conferences from the evolutionary computation field, enabling a wide dissemination of project results.
Significance for science
The idea for this research project stems directly from the need to finally construct a real-world benchmark suite identified by the key researchers and practitioners in the multiobjective optimization community, which confirms the high relevance of the project.
COCO and the associated BBOB workshops are renowned in the international optimization community. So far, 169 algorithm variants have been described in 126 workshop papers authored by 93 researchers from 25 different countries. The papers describing the experimental setup and the COCO/BBOB benchmark functions have been cited more than 750 times in total (according to Google Scholar). Similarly, the DEMO algorithm is well-known in the community. Other researchers have extended it in numerous ways (DEMOwSA, AMS-DEMO, CFDE, DEMO-TDQL, AC-DEMO, GP-DEMO, etc.) and applied it to real-world problems. The two basic papers on the DEMO algorithm have been cumulatively cited more than 500 times (according to Google Scholar).
All this means that the project results will have a big impact on the research community. Not only will the project enhance benchmarking in multiobjective optimization, it will also stimulate the development of new, real-world-oriented optimization algorithms. Although the project addresses multiobjective optimization problems, most of the advances to the COCO platform (support of the mixed variables, constraints and expensive evaluations) are relevant also for singleobjective optimization, which further broadens the project impact.
Additionally, the project will trigger new research in exploratory landscape analysis, an emerging research topic that aims at automatizing the selection of the best available algorithm for a given problem by applying statistical analysis and machine learning tools on extensive data of algorithm performances in order to predict which of the available algorithms should be recommended in practice for a new problem. This project will provide the data needed for extending exploratory landscape analysis from singleobjective to multiobjective problems.
Significance for the country
The main objective of the proposed project is to bridge the gap between research and application in multiobjective optimization by moving away from the artificial test problems and bringing the focus to real-world optimization problems. This will directly influence the real-world optimization problems included in the new benchmark suite. Because they will be optimized by numerous good algorithms in the forthcoming BBOB workshops, it is reasonable to expect that the best known solutions to these problems will be greatly improved already in the short term. Moreover, the implemented advances to the COCO platform and the real-world benchmark suite will stimulate the inclusion of other real-world problems to the platform, increasing the quality of their solutions. Since an omnipresent objective in multiobjective optimization is that of reducing costs, the project will have also beneficial economic impact.
Some real-world problems to be included in COCO directly or through surrogate models could stem from Slovenian companies involved in past and current projects, such as Štore Steel from the SYNERGY project (continuous casting of steel), Kolektor from the COPCAMS project (manufacturing in automotive industry) and Domel from the SYNERGY project (optimization of electrical motor design). Consequently, the proposed project will enable Slovenian industry to take advantage of the cutting edge of research in multiobjective optimization.
Finally, in the long run, the project will enable new ranking of algorithms based on their performance on the real-world problems included in the platform, which will help select “the right algorithm for the right purpose”. This will enable practitioners to solve their real-world problems better and faster.
Most important scientific results
Interim report,
final report
Most important socioeconomically and culturally relevant results
Interim report,
final report
