Enhanced support for uv dynamic virtual environments in Prefect agent executions

[eduardojandre]

Describe the current behavior

Currently, Prefect agents execute flows using the environment in which the agent process is running. While this works well in homogeneous environments, it complicates scenarios where different flows require distinct Python environments or dependency sets.

The current alternatives — using multiple agent configurations or dynamically instantiated containers with specialized images — introduce additional operational complexity, infrastructure overhead, and slower startup times.

Describe the proposed behavior

Proposed Behavior
Introduce a mechanism within the Prefect agent or flow runner that allows:

• Optional detection of a .venv or uv environment in the cloned repository.
• Invocation of the virtual environment's Python binary to execute the flow, instead of relying on the agent's global Python environment.

Potential Implementation Ideas

• Modify src/prefect/workers/process.py (ProcessWorker) to support automatic detection of a .venv or uv environment in the cloned repository.
  If such an environment exists, the worker can invoke the flow run using the Python binary within that virtual environment, instead of the agent's default sys.executable.

• Consider extending similar logic to other workers to support it, ensuring consistent behavior across different deployment types, including containerized and Kubernetes-based workers.

• Provide a configuration option (e.g., environment variable or deployment setting) to enable or disable this auto-detection and invocation of virtual environments, allowing for flexible adoption.

Example Use

An organization operates a Prefect agent on a shared server where multiple teams deploy flows with different and potentially conflicting dependencies.

With this feature, each team can commit their flow code, and use a Utility step on the pull action to create a dedicated .venv or uv environment within their repository. When the flow is deployed and executed by the agent:

1. The agent clones the repository
2. Agent runs Utility step to generate virtual env
3. The and detects the presence of the virtual environment, and instead of using the agent's global Python interpreter, it automatically invokes the flow using the python binary inside the detected environment.
4. This allows different flows to run with completely isolated dependencies without requiring multiple agents or complex container orchestration.
   For example:

• Flow A requires pandas==1.5.
• Flow B requires pandas==2.2.

Both flows can run independently on the same agent using their respective virtual environments, improving deployment flexibility and minimizing dependency conflicts.

Additional context

Benefits

Simplified and Faster Iteration

Switching Python environments via virtual environments (venv, uv, etc.) can be faster and simpler than building, pushing, and managing container images. Developers can adjust dependencies without modifying deployment infrastructure.

Additionally, execution startup time can be significantly reduced on some scenarios — invoking a subprocess using the environment's Python binary is can be faster than instantiating a new container. This makes the approach particularly well-suited for short-lived flows or frequent deployments.

Greater Flexibility in Deployment

Enables the use of multiple Python environments within a single worker, allowing diverse flows with different dependencies to coexist without requiring container orchestration.

Improved Compatibility

Supports scenarios where running inside a container is restricted,

[zzstoatzz]

hi @eduardojandre - thanks for the write-up! this is possible today by overriding the command job variable on a per deployment basis like this. you can also run arbitrary shell commands in a pull step instead to setup your flow run venv as needed.

Modify src/prefect/workers/process.py (ProcessWorker) to support automatic detection of a .venv or uv environment in the cloned repository.

we are unlikely to do this, because the above escape hatch already exists and there is a great deal of variation on what "automation detection" could mean here

additionally, you might be interested in ad-hoc infra submission.

docs: https://docs.prefect.io/v3/deploy/submit-flows-directly-to-dynamic-infrastructure
issue: #17194

given the optionality that exists today and the documented plans to expand ad-hoc infra over time, I think we can close this issue. what do you think?

[eduardojandre]

Hi @zzstoatzz, thank you very much for the super quick response!

I appreciate the flexibility that Prefect offers with command overrides and pull steps. I had actually been experimenting with something along those lines, but I ran into a limitation: the pull step seems to execute after the command, which makes it tricky for my use case.

Specifically, I was trying to set up something like this:

deployments:
  - name: test-labs-uv4
    ...
    pull:
      - prefect.deployments.steps.git_clone:
          id: clone-step
          repository: "https://dev.azure.com/prefect-demo/_git/prefect-demo"
          branch: "testeduardo"
          access_token: "{{ prefect.blocks.secret.azure-devops-token }}"
      - prefect.deployments.steps.run_shell_script:
          script: |
            uv venv .venv
            uv pip install -r requirements.txt
    work_pool:
      name: my-work-pool
      job_variables:
        command: "uv run -m prefect.engine"

But since the repo isn’t cloned yet when command runs, the environment isn’t available, so this approach doesn’t quite solve the problem I’m aiming to address.

I completely understand the concern about generalizing "automatic detection," especially given the variations in environments. I also looked into ad-hoc infrastructure submission as you suggested — while it’s a great feature, it doesn’t quite fit my use case either.

My use case is to support lightweight flows with slightly different dependency requirements, all running on the same worker, without having to manage and provision entire container images for each variation. I also understand what I proposed as a solution will not work either because of the order.
But I think the use case is still valid and we might find another way of doing. maybe a pre pull action where we download and install requirements.

Thanks again!

[zzstoatzz]

hi @eduardojandre - i think you have a good point that we could improve this. it might be a another worker type, built-in step but there is definitely a gap here

incomplete initial assessment

you shouldn't need both. you're correct about the sequencing, but I'm suggesting you could either

• override command if you want to point at a repo or existing package (use --with)
• setup a venv as needed in a pull step if you want to install from cloned requirements

in this case it sounds like you'd prefer the latter, so check out this example:

» prefect deployment run 'uses-pandas/deployment-3'
Creating flow run for deployment 'uses-pandas/deployment-3'...
Created flow run 'eccentric-caterpillar'.
└── UUID: b1e8875a-a358-4abc-9a4b-d9c1cc21dc6f
└── Parameters: {}
└── Job Variables: {}
└── Scheduled start time: 2025-05-28 16:18:33 CDT (now)
└── URL: http://127.0.0.1:4200/runs/flow-run/b1e8875a-a358-4abc-9a4b-d9c1cc21dc6f

» uv run --isolated prefect worker start --pool 'local' --run-once
Installed 134 packages in 239ms
Discovered type 'process' for work pool 'local'.
Worker 'ProcessWorker bae703d0-8d76-4078-b457-0e09c236f168' started!
16:18:46.965 | INFO    | prefect.flow_runs.worker - Worker 'ProcessWorker bae703d0-8d76-4078-b457-0e09c236f168' submitting flow run 'b1e8875a-a358-4abc-9a4b-d9c1cc21dc6f'
16:18:47.015 | INFO    | prefect.flow_runs.runner - Opening process...
16:18:47.037 | INFO    | prefect.flow_runs.worker - Completed submission of flow run 'b1e8875a-a358-4abc-9a4b-d9c1cc21dc6f'
16:18:47.612 | INFO    | Flow run 'eccentric-caterpillar' -  > Running git_clone step...
16:18:48.080 | INFO    | Flow run 'eccentric-caterpillar' -  > Running run_shell_script step...
Resolved 7 packages in 7ms
Installed 3 packages in 62ms
 + emoji==2.14.1
 + numpy==2.2.6
 + pandas==2.2.3
16:18:48.700 | INFO    | Flow run 'eccentric-caterpillar' - Beginning flow run 'eccentric-caterpillar' for flow 'uses-pandas'
16:18:48.700 | INFO    | Flow run 'eccentric-caterpillar' - View at http://127.0.0.1:4200/runs/flow-run/b1e8875a-a358-4abc-9a4b-d9c1cc21dc6f
16:18:55.115 | INFO    | Flow run 'eccentric-caterpillar' -    a  b
0  1  4
1  2  5
2  3  6
16:18:55.142 | INFO    | Flow run 'eccentric-caterpillar' - Finished in state Completed()
16:18:55.266 | INFO    | prefect.flow_runs.runner - Process for flow run 'eccentric-caterpillar' exited cleanly.
Worker 'ProcessWorker bae703d0-8d76-4078-b457-0e09c236f168' stopped!

» uv pip list | rg pandas

you should have full flexibility via uv with how you want to reuse or recreate a given virtual environment in a pull step

for example, you might want to define dependency-groups in your pyproject.toml and have each flow create a new venv like uv sync --group this-flow-group in your pull step

[NTC4818]

Checking in on this thread if anyone has found anything hacky but functional? I have tried the pair of uv sync --directory {repo} and gotten packages to download but uv run python -m prefect.engine doesn't invoke the new package environment. Was wondering if I needed to expand that pull step shell script to activate the environment or if @eduardojandre had a different solution so far?

[leo-selfee]

I'm also interested in a workaround. My use case is to upgrade my package from the cloned repo, in a next step, running in a process worker.
But with a concurrency limit > 1 there seems to be some intermittent race conditions happening resulting in import errors on my package for some runs. I suspect this happens when 2 concurrent runs upgrade step execute around the same time window

[torext]

@eduardojandre Is completely right, the current solutions using command or pull steps are not sufficient as in the latter case it can lead to race conditions (two flows simultaneously updating the env during their respective pull steps).

@zzstoatzz The point about the race condition above is why your "incomplete initial assessment" above is not a good enough solution, even temporarily. If two flows start simultaneously and run uv pip install they'll be affecting the same underlying Python environment simultaneously and there will be race conditions.

@NTC4818 @leo-selfee The only hacky solution I've found so far is to use an approach based on the command variable. It's hacky because you have limited control on how exactly the environment gets handled, only works with uv and not e.g. venv, and there is a certain redundant duplication of fetching of the repo. But it does work (save for some caveats)!

More precisely, I essentially specify the repo (which is uv-based, meaning has a uv.lock and a pyproject.toml) in the pull step as well as in the command. So overall my prefect.yaml looks as follows:

pull:
- prefect.deployments.steps.git_clone:
    repository: git@github.mycompany.com:myproject/myproject.git
    branch: "{{ $MYPROJECT_BRANCH }}"

definitions:
  deployments:
    local_pool_deployment: &local_pool_deployment
      work_pool:
        name: local
        job_variables:
          command: uv run --with git+ssh://git@github.mycompany.com/myproject/myproject.git@{{ $MYPROJECT_BRANCH }} python -m prefect.engine

deployments:
- <<: *local_pool_deployment
  name: some-deployment
  entrypoint: src/myproject/deployments/some_deployment.py:main
  schedules:
  - cron: 0 19 * * Mon-Fri
    timezone: US/Eastern

This pretty much works as one would like and there's no race conditions between flows running simultaneously as they will each create a temporary implicit virtual env on the fly thanks to the --with. However, this is still not a great solution and quite hacky because:

• We have to specify the repo twice; it gets fetched in the pull step and "on the fly" before the pull step even starts.
• The uv.lock file gets ignored in the command; the --with statement will only respect the pyproject.toml, thus not being an exact replica of the working repo state.
• Finally, to be sure this runs fairly cleanly one has to make sure that the actual virtual env housing the Prefect worker/server is bare, meaning it doesn't contain a possibly stale installation of the repo and its dependencies. This is maybe not strictly required but there's an obvious risk here.

Given all the above I agree with @eduardojandre very much that Prefect devs need to find a solution here. There has got to be a way to dynamically create and use Python environments independently for each flow run, especially nowadays that environment specifications are mostly declarative and can be committed to and pinned in repositories themselves. Not having this functionality makes the whole git pull step mechanism somewhat incomplete and pointless at this stage.