Workspace API removes extension from Python files

[fferegrino]

We are running into an issue when uploading Python files using the method import_workspace_dir in the WorkspaceApi class. However, these files are not treated as Python source files, but as Databricks notebooks.

The only reason I managed to pinpoint this issue is the fact that the API removes the extension and as such they are no longer interpreted as Python modules but as Databricks notebooks.

databricks-cli/databricks_cli/workspace/api.py

Line 153 in 5c38791

cur_dst = cur_dst[:-len(ext)]

Is this intended behaviour? could this be changed by not removing the file extension, unless the file is actually a Databricks notebook?

[fferegrino]

Actually, the issue is not only with the extension but with the file format, according to the docs it is possible to upload arbitrary files (?) when the format is set to "AUTO", so that probably needs changing as well...

Now, it seems that the feature is in public preview, and as such this change has not made it to the CLI, rendering this and #618 a bit redundant.

[osuleiman]

I've just created a PR to help tackle this problem.

#643

[shadzic]

Hello, any update? It seems that the PR is not merged and the official documentation (databricks workspace import-dir --help
on v0.238.0) does not mention it either. I'm interested to know what happened to this feature. Thanks!

[shadzic]

Hello
If I am not mistaken, this is not only a limitation from the databricks-cli but also from the databricks REST API, as mentioned in the documentation: https://docs.databricks.com/api/workspace/workspace/import

[path](https://docs.databricks.com/api/workspace/workspace/import#path)
required
string
The absolute path of the object or directory. Importing a directory is only supported for the DBC and SOURCE formats.