Make it easier to find all the "things" associated with a project with tags or groups or something

[j-carson]

This is a thread I was having with @zzstoatzz on slack and he suggested creating a discussion.

I had asked why wouldn't I just convert all my "variables" to "blocks" so that they'd be easier to find.

Could someone explain why we have both blocks and variables? I get that I can't use a variable when I need a block for encryption, but why not just use custom-blocks for your variables so that all your config is in one spot in the UI?

Nate explained the overhead is a bit higher with blocks and recommended:

use blocks when
you want to perform validation on get/set of the value with python code (since blocks are BaseModel subclasses and pydantic does validation nicely)
you have a secret value you want to be encrypted etc

use variables when
you just have a JSON value you want to load in your flows / read a lot

But, for me the overhead might be worth it just from a system management point of view. My reason for wanting to get rid of all my variables and just make blocks was really a small part of the "system config scatter" - one less kind of thing to worry about keeping track of.

It's more like, the system config scatter is annoying, the fewer object pages I have to visit the better. From a programming point of view, my only simple custom block was pretty easy to do.

If I could just "find everything with project name " and have the search bring up all my variables and blocks (which can't be tagged - why???) and flows and deployments in one spot instead of having to click around to understand all the connected "things", life would be easier.  And naming conventions! Why do blocks always have dashes and variables only accept underscores?  Makes remembering spellings harder...

I also suspect that I'm not using any of this stuff quite right -- My flows have parameters with block and variable names and look up the contents when they kick off. But, that means it's hard to draw a dependency graph between what flows depend on what.  Is that level of indirection how folks normally do it?  Or is there a way to directly have a project bucket where all your objects live together?

Nate's search for prior discussions: https://gist.github.com/zzstoatzz/dde156aaed51b28dcdd20fea1e418cc1

[omni-front]

Totally feel you. Ran into this myself when managing multiple projects in Prefect. It can be a bit of a headache when the system feels fragmented.

The distinction between blocks and variables can indeed seem like an unnecessary complication at times. Variables are great for lightweight configs, but if you're dealing with something that needs validation or encryption, blocks are worth the overhead. But yeah, consolidating everything as blocks just for the sake of simplicity in management makes sense, especially when dealing with complex projects.

Regarding the inability to tag blocks, it’s a limitation that’s been a pain point for many. Prefect's UI has room for improvement here. This is where naming conventions and consistent documentation can really help, even if they're a bit of a chore.

For now, one workaround could be to implement a naming scheme that inherently includes your "tags". For example, prefixing or suffixing your block and variable names with a project identifier. Something like:

# block name example: "projectA-config-block"
# variable name example: "projectA_config_variable"

As for the naming conventions (dashes vs underscores), it’s just one of those quirks. It helps to establish a mental model or a cheat sheet for your team to avoid mix-ups.

For the future, you might want to explore using Prefect's GraphQL API to build a custom dashboard that can fetch and display all the resources associated with a project. It's a bit of work upfront, but could save time in the long run. Here's a tiny snippet to get you started with querying blocks:

{
  block(where: {name: {_ilike: "projectA%"}}) {
    id
    name
    type
  }
}

Keep pushing for improvements on this front. Prefect's community is pretty responsive to feedback, and they can provide updates or workarounds too.

If you're looking for a more integrated solution, consider opening a feature request on the Prefect GitHub to advocate for tagging capabilities or integrated views. It’s something that others would likely benefit from too.