fix: capability detection for named compute (databricks_compute)

[trouze]

Resolves #1354

Description

Fixes a timing bug where table_format='iceberg' (and other capability-gated features) fail with "iceberg requires DBR 14.3+" when the model targets a named compute via databricks_compute model config, even though the compute supports the required DBR version.

Fix

One-line change: call _cache_dbr_capabilities() eagerly in _create_fresh_connection() before reading the cache. This ensures the DBR version is queried and cached at connection creation time rather than waiting for the lazy open(). The call is idempotent (guarded by if http_path not in cache), so for default compute with a warm cache it's a no-op.

Key changes

• dbt/adapters/databricks/connections.py: Added self._cache_dbr_capabilities(creds, conn.http_path) before conn.capabilities = self._get_capabilities_for_http_path(conn.http_path) in _create_fresh_connection()
• tests/unit/test_connection_manager.py: Added 3 unit tests validating eager caching for named compute, fallback behavior when cache is empty, and idempotency for default compute with a warm cache

Checklist

• I have run this code in development and it appears to resolve the stated issue
• This PR includes tests, or tests are not required/relevant for this PR
• I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

[sd-db]

I see failing tests. I think there is a subtle bag in the cache population logic which is why this is happening.

The call is idempotent (guarded by if http_path not in cache), so for default compute with a warm cache it's a no-op.

That's true when the cache is warm yes, but on a cold cache the eager _cache_dbr_capabilities() fires before any dbsql.connect() has happened. If this fails for any reason (cluster spinning up, credentials_manager not set yet, transient blip), the except Exception: pass eats the error and returns None.

The issue is that _cache_dbr_capabilities stores the result unconditionally:

  cls._dbr_capabilities_cache[http_path] = DBRCapabilities(
      dbr_version=dbr_version,  # None!
      ...
  )

When open() runs later the idempotency guard sees the path is already cached and skips it and every version-gated capability comes back False.

I think instead of calling _cache_dbr_capabilities eagerly,we should add a safe variant that only writes to the cache when the version query actually returns something.

@classmethod
  def _try_cache_dbr_capabilities(cls, creds, http_path):
      if http_path not in cls._dbr_capabilities_cache:
          is_cluster = is_cluster_http_path(http_path, creds.cluster_id)
          dbr_version = cls._query_dbr_version(creds, http_path)
          if dbr_version is not None:
              cls._dbr_capabilities_cache[http_path] = DBRCapabilities(
                  dbr_version=dbr_version,
                  is_sql_warehouse=not is_cluster,
              )

wdyt ? I tested this locally as well to verify and looks good (also if adding this please add unit tests as well, thx)

[trouze]

@sd-db great catch! I'm good with your approach, tested with my repro and worked well - I also added unit tests to ensure that _try_cache_dbt_capabilities() does a no-write on None version, write on known version, and skip when already cached.

[tejassp-db]

@trouze can you please run pre-commit to fix the lint issues ?

[trouze]

@tejassp-db apologies, forgot to run it. Fixed!

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
dbt/adapters/databricks
connections.py					229-230
Project Total

This report was generated by python-coverage-comment-action

[sd-db]

Changes look good, thanks for the contribution !!