DESDEO Web-API state and progress

This issue is to track and document the development of the DESDEO2 web-API. The items below will be divided into three
categories: models, routers, and functionalities. What we have, and what is needed is listed in each category, with the latter
accompanied by a priority marker (⭐). Currently, the web-API is most needed by a research project (ending at the end of this year), which will dictate many of the priorities listed below. However, if you are not tied to this project, we absolutely welcome contributions that are not marked as priorities. Everything listed will be needed eventually, in any case.

If some topic becomes too big to be discussed in a single issue, feel free to open a sub-issue. However, keep it connected to this issue, just to ease keeping track of things.

For running the API, check this README.md.

Models

Models refers to models implemented using SQLModel, which describe database entries or how to display/parse these. The models that describe the database structure are referred to as table models, often with a class name ModelName followed by DB. E.g., PreferenceDB. These models
often inherit from a ModelName + Base, and have the SQLModel-class argument table set to be True. E.g.:

class PreferenceDB(SQLModel, table=True):
    ...

Other model types can be thought of defining 'views' of the core model. I.e., when we want to fetch specific information about a certain database entry, we might not want to dump the whole table, but instead only relevant parts of it. Either to be efficient, or for security reasons. For example, the view UserPublic for the User model is defined as:

class UserPublic(UserBase):
    """The object to handle public user information."""

    id: int
    role: UserRole
    group: str

In which case when we fetch information on a specific user, we get that user's id, role, and group columns, and leave other information out.

Models (and views) are defined in desdeo/api/model/.

What we have

• User model (user.py): User model with the roles dm (decision maker), guest, analyst, admin. Comes with some views. Used to define and manage users in the API.
• States (state.py): A base definition of a state to manage the state of interactive methods. An specific state for the reference point method is defined and is functional.
• Session (session.py): A base definition for a session model. Sessions are for storing states that emerge during an interactive optimization process.
• Reference point method models (reference_point_method.py): A single model for defining a solve request for the reference point method. This is in essence all that is needed to operate the reference point method when it comes to communicating with the method.
• Problem model (problem.py): A model to store and represent multiobjective optimization problems, which is 1-to-1 with the Pydantic based schema in the core-logic of DESDEO.
• Preference models (preference.py): Currently, there are two preference models that can support preferences that can be represented as a reference point or as bounds (on the objective functions values).
• Archive model (archive.py): An archive model to store information about solutions found by interactive methods.

What is needed

Some of the requirements here should be implemented in a generic way, for example, saving solutions and computing intermediate solutions in NIMBUS is something we would like to access in other methods as well. However, we can start by implementing them with mainly NIMBUS mind, check that they work, then generalize them (this basically means renaming the state, if done correctly).

• ⭐ Models for NIMBUS: A state model needs to be defined, which can store information related to the different states of (Synchronous) NIMBUS. These models need to support requests required by NIMBUS: classification, computation of intermediate points, and saving solutions. (@StinaSoile working on)
  • NimbusBaseState
  • NimbusClassificationState
  • NimbusClassificationRequest
  • States related to solving intermediate solutions (generic).
• Models for NAUTILUS Navigator: A state model to represent the possible states of NAUTILUS Navigator needs to be defined. These models need to support the requests required by NAUTILUS Navigator: start navigation, stop navigation, set preference, go to a previous iteration. Note: the needed functionality for streaming information from the API might affect this task.
• ⭐ Models needed for group decision-making: Models that support the needs of group decision-making are needed. Particularly, a session model for handling group decision-making is critical. Details of these will be updated eventually.
• Solver models: Since many methods give the option to specify the solver (the optimization algorithms used to solve, e.g., a scalarized multiobjective optimization problem), a general model to handle these, including their parameters, is needed. Problems can be linked to these models, since these parameters are often problem specific.
• Models for other interactive methods: States to support other interactive methods in DESDEO are needed. Again, the main purpose of these is to ultimately support the requests related to the methods. Will be updated later.
  • E-NAUTILUS related models: Models to handle state and request/response are needed. (worked on by @gialmisi )
• Population management models: Especially with EMO (evolutionary) methods, models are needed to handle these. To be updated later.
• Explainability related models: Models to handle explainability, and related requests, are needed. To be updated later.
• Problem modeling models: Since we already have ProblemDB, we need only a request and response model for endpoints related to problem modeling, i.e., adding a problem model to the database.

Routers

Routers are HTTP-endpoints that define how users of the API can interact with DESDEO and the database. These are implemented using FastAPI. The best way to explore the available endpoints is to launch the API, and visit the automatically generated documentation of the endpoint. E.g., running the following commands (in the folder desdeo/api:

python db_init.py
uvicorn --app-dir=./ app:app --reload

And then visiting http:127.0.0.1:8000/docs (the url may vary, check the output of uvicorn).

Routers are defined in desdeo/api/routers. Note: old routers still exist for NAUTILUS/NAUTILUS Navigator, and NIMBUS. These are left as references, which could be helpful in implementing new routers. The file names with old routers are pre-fixed with an underscore, e.g.,
_NAUTILUS.py, _NAUTILUS_navigator.py, and _NIMBUS.py.

What we have

• User routers (user_authentication.py): Routes for logging in, managing JWT tokens, and getting user info.
• Problem routers (problem.py): Routes for fetching existing problems, getting problem info, and adding problems.
• Session routers (session.py): Routes for creating and fetching sessions.
• Reference point method routers (reference_point_method.py): Router for solving a problem using the reference point method.

What is needed

• ⭐ Additional User routers (user_authentication.py):
  • Routes for registering new users ( being worked on by @viljokass )
  • logging in as a guest
  • logging out.
• ⭐ NIMBUS routers: Routers, which allow implementing the NIMBUS method through the API. At least routers for iterating the NIMBUS method with given classifications, computing intermediate solutions between existing solutions, saving solutions to an archive (and fetching them). Computing intermediate solutions and fetching/saving solutions to an archive can be implemented as generic endpoints. (@StinaSoile working on)
  • Router for solving sub-problems based on classifications given (method/nimbus/solve)
  • Router for solving intermediate solutions (generic)
  • Router for saving solutions to an archive (generic)
  • Router for fetching solutions for the archive (generic)
• NAUTILUS Navigator: Routers, which allow implementing the NAUTILUS Navigator method through the API. E.g., endpoints for starting/continuing navigation, setting a reference point and bounds, going to a previous iteration.
• Routes for other methods: Routers to implement other methods are needed as well. Especially EMO methods, and group-variants of interactive methods.
  • E-NAUTILUS: E-NAUTILUS will need a route to enable the use of enautilus_step in the core-logic. (worked on by @gialmisi )
• Problem model related routers: we need routers for:
  • Adding a new problem to the database for a user.
  • Modifying an existing problem.
  • Deleting a problem(s) from the database.
  • Adding metadata to problems.
• General method related routers:
  • A route that returns all the available method.

Functionalities

Functionalities refers to everything else in the API that does not fall in the category of models or routers.

What we have

• JWT authentication: We use JWT access and refresh tokens to use for authentication between the client and the API. The tokens are assumed to be handled via cookies.
• Configuration management: We have the config.toml file that is used to handle all the parameters needed to operate the API. Naturally, when deploying, sensitive parameters, such as keys, should be defined and read as environment variables, and NOT defined in this file. For testing the API locally, without any sensitive information, defining all parameters in the file is fine.
• Scripts to initialize the database: We have a basic script for initializing the database of the API defined in db_init.py. This works for testing right now, i.e., running the API locally, but has a skeleton ready for deployment related functionalities as well.
• Testing: We have a solid base for testing most of the new API. To run these, execute pytest in the folder desdeo/api.

What is needed

• ⭐ Web sockets: Group decision-making needs longer lasting bidirectional connections with the API. For this purpose, web sockets are likely the best solution. These allows the client to get real-time updates from the API without having to separately prompt any endpoints. Very useful in situations, where multiple users would be solving the same problem using a group decision-making method. See WebSockets.
• ⭐ Further testing: As new features are implemented, tests for these should be written right away.
• Streaming: Navigation methods, and evolutionary methods would benefit greatly from the possibility to have routers that can stream information, which allows sending information in increments instead of at once. This allows, e.g., for sending information on a computed solution set with an evolutionary method as it is generated. For navigation methods, this is useful as well, since navigation happens in incremental steps. See StreamingResponse.
• Proper documentation: The API should be properly documented.
  • Explanation of the database structure and its schema (being worked on by @gialmisi)
  • OpenAPI documentation of the endpoints (automatically generated).
  • A reference of the database models (automatically generated) (being owrked on by @gialmisi)

Contributing

If you decide to contribute to any of the above needs, please make sure to reply to this issue with your intentions. If needed, we can also discuss your contribution plans, if you require additional information before starting. If needed, sub-issues can be created, but they must be linked to the corresponding needs listen in this issue. This is to document and keep track of the developments of the web-API for DESDEO.