Ox-orch: a state-driven orchestration engine for deterministic workflow

[thomas_oxylus]

Hi all,

I’d like to present you one of the projects I’ve been working on for several months. It is still in development but slowly reach a usable point.

Its name is ox-orch, you can check it here: GitHub - oxylus-tech/ox-orch: A state-driven orchestration engine for deterministic application lifecycle management. · GitHub (documentation: ox-orch - Read the Docs Community)

So what’s ox-orch? It is a state-driven orchestration engine for deterministic application lifecycle management. In a nutshell, it provides a structured way to define, execute and rollback complex workflows such as package installation, migration, and application reconciliation making its best effort to keep deployment state coherent and clean.

This project provides:

• A library that allows you to integrate it into your own projects;
• A command line tool using configuration files to run workflows and manage them;

And it features:

• A composable operation system for lifecycle workflows;
• Deterministic execution graph with dependency resolution;
• Explicit state tracking per operation, with forward and backward system for safe rollback (eg. on failure)
• Framework agnostic design though initially and supporting django.

So why ox-orch?

The initial reason was to allow application installation by end-user from a django running instance (we not there yet, though most of the work has been done to allow this ultimate goal). What it means is that we also need a way for deployment to run nice and being able to reverse it without breaking everything meanwhile – and you know how fast it may come.

First examples

A simple installation pipeline would look like this (for the CLI tool, but you can use python libraries too):

# install.yaml
name: Pip Install
operation: "install:pip"

Once the pipeline is defined, you have to provide parameters, as context file:

# context.yaml
install:
    packages:
        httpx: 0.28.1
        pyyaml: 6.0.3

To run:

    ox-orch run -c context.yaml apply install.yaml

Whenever the pipeline fails, we can ensure to rollback changes made by the tool.

You can also run sequential operations from within a plan:

# install.yaml
name: Pip Install
operation:
    __type_id__: plan
    operations:
    # Our installation
    - "pip:install"
    # Simple shell invokation
    - __type_id__: shell
      forward: ["echo", "Installed!"]
      backward: ["echo", "Rolling-back"]

Usage with django

Here is a basic Django application installation pipeline, just for the sake of example. Though you should check on the doc for an understanding of what is going on:

name: Django App install
operation:
  __type_id__: apps
  # Use UV to install packages
  install: install:uv
  operations:
  # Ensure installed packages are enabled
  - "django:enable"
  # Fork into a new subprocess ensuring django reinitialization
  - __type_id__: fork
    # This operation wraps: setup, migrate, collectstatic, compilemessages
    operation: 
      "django:reconciliation"

Context file:

# Django project configuration
django:
    project_path: "/path/to/the/project"
    settings_module: "my_project.settings"

# Apps management
apps:
    apps: ["ox-fin"]

    # Application store
    store_backend: file
    store_args:
        path: "./path/to/app_store.yaml"

    # Configure app state store
    state_store_backend: file
    state_store_args:
        path: "./path/to/app_state_store.yaml"

Whats next?

I’m still on the development of the project on my available time, and lots needs to happen.

Currently we have a working framework and command line tools, and multiple concepts have been validated. However we still have work to do:

• More operations, as database snapshot/restore, more UV and Poetry operations, launch and stop services (TBD, as for supervisorctl, docker or whatever)
• Documentation improvment:
• Provide examples;
• Some hidden TODOs and FIXMEs in the code.
• Improve the CLI tool: as allowing to continue an interrupted pipeline, replay and inspection, etc
• … and a running server to run pipelines through API (aka => allow to install Django applications and restart it).

[abhijeetmote]

Interesting project. Making rollback and state tracking first-class seems useful for application installation and reconciliation workflows, especially when several steps need to run in order.

One challenge I’d be curious about is interruption recovery. If an operation completes its side effect but the process stops before recording success, how does ox-orch decide whether to retry, reconcile, or mark the state as uncertain? That can be tricky for package installs and database migrations, where a backward operation may be a compensating action rather than a true rollback.

For the planned Django server/API, I’d also keep workflow execution out of the request-response path and run it through a background worker. Since plans can invoke shell commands, authentication, audit logging, and clear execution boundaries will matter.

Are operations expected to be idempotent, and how are retries handled after a partial failure?

[thomas_oxylus]

Hi there! Thanks for you message!

Yes, that actually is a difficult part of the design. The model is closer to state-driven reconciliation. An operation moves the current state toward a desired one, while state is persisted separately from the operation. After an interruption we thus have to distinguish different cases: applied, not applied, uncertain (requiring reconciliation).

Whenever possible, we try to delegate, as for example the dependency graph and migration state of Django’s MigrationExecutor, or for packages installation. We don’t maintain a separate implementation, rather call it and use for example the executor to inspect the current applied migrations.

So, operation should be idempotent or reconciliation-safe where possible. A rollback is then a transition toward a previous desired state, and can be a compensating action as you’ve highlighted, rather than being a universal inverse of it.

And here comes the difficult part: what happens when side effect succeed but persisting the state fails? Lets be honest: currently there is no way yet for Ox-Orch to safely assume that the operation failed or succeeded. This still needs to be implemented: inspect the external state and reconcile it before deciding whether to retry.

I’d distinguish three cases then:

• idempotent operation: safe to retry
• reconcilable operation: inspect the actual state and continue from there
• non-reconcilable: keep the execution uncertain and requires explicit handling instead of retrying automatically

This still has to be implemented, actually is part of the next steps for the project – as soon I’ve got more time to continue on this quest.

Django API / Async

Regarding the Django API, I totally agree, the workflow execution should be asynchronous. The HTTP layer should submit and inspect execution, not run the complete plan in the request-response cycle. Authentication, authorization, audit logging and execution boundaries shall be handled around the executor or worker (though I’d avoid in the first place to provide this as a whole for modularity).

The current execution system of operations actually is based on state generators. The methods Operation.apply and Operation.rollback will yield all states transition (including for nested ones). These are the public entry point, that handle state notifying state transition (running, failed, etc), and call the actual operation specific code which lies in the _apply and _rollback. Those two last can either be a regular method or generator one, this is handled at apply/rollback level.

This actually was thought to allow upstream (HTTP or cli, etc) to be able to notify end-user and handle state transition.

From there to async seems kinda simple at first sight, grosso modo: make apply/rollback async and handle async cases of _apply/_rollback method + upstream handling. However as you know or may have seen in bad implementations this is not a sufficient warranty of non-blocking:

• Calling blocking operation in async will loose the whole interest of it (or partially): it means blocking workers which surely is not what you want. For example, MigrationExecutor ain’t async, meaning you’ll block on apply or rollback.

• This implies to implement either counterpart async version of operations (as shell, file access, etc.) and potentially enforce usage of async upstream.

So here came the why I didn’t make async yet: is it necessary yet since we already have a way to feed back upstream? Maybe. But currently the need didn’t raised (I let the door open however). Here I’ll just cover two cases:

• Lets say you want to send controls command from API to the background worker, eg via HTTP. You will have an MPC way-ish to communicate between processes (HTTP ↔ Executor), and since you’re using generators, you can completely interrupt the flow if you want – but only between operation calls if sync operation stack. Which actually may seems a be better way for state handling.

• Another case is to allow multiple workflows to run concurrently without consuming too much workers, eg. lets scale up. To KISS environments and execution, we assume that ox-orch is run from within the environment which cover cases as Docker containers, which potentially can be sufficient (potentially: I didn’t study the case but I let the door open for later).

I’ve been working on Ox-Orch since January ±, and it took a long path to come here from the question: “How can I deploy safely from an API new Django application in my project?” (actually Oxylus, if you want to checkout => https://oxylus.app/ ) . Between this first question and the current result came also the realization that we can’t just run a simple script for it but need full state-safe transition.

I don’t currently expect for Ox-Orch to be known and used, though would love to, because it’d mean that I haven’t done this whole work for just myself but something useful for the community. If it becomes, then I’d go further on the development/improvement as async or more complex cases handling. I’d love too, but you know how’s life and time we can allocate to projects…