Work around DLT issue with `$PYTHONPATH` not being set correctly (#999)

## Changes

DLT currently doesn't always set `$PYTHONPATH` correctly (ES-947370).
This restores the original workaround to make new pipelines work while
that issue is being addressed. The workaround was removed in #832.

Manually tested.
This commit is contained in:
Lennart Kats (databricks) 2023-11-20 20:25:43 +01:00 committed by GitHub
parent 1b7558cd7d
commit 92539d4b9b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 4 additions and 0 deletions

View File

@ -7,3 +7,6 @@ resources:
libraries:
- notebook:
path: ../src/dlt_pipeline.ipynb
configuration:
bundle.sourcePath: /Workspace/${workspace.file_path}/src

View File

@ -35,6 +35,7 @@
"# Import DLT and src/{{.project_name}}\n",
"import dlt\n",
"import sys\n",
"sys.path.append(spark.conf.get(\"bundle.sourcePath\", \".\"))\n",
"from pyspark.sql.functions import expr\n",
"from {{.project_name}} import main"
{{else}}