Commit Graph

3 Commits

Author SHA1 Message Date
Andrew Nester 86c30dd328
Fixed artifact file uploading on Windows and wheel execution on DBR 13.3 (#722)
## Changes
Fixed artifact file uploading on Windows and wheel execution on DBR 13.3

Fixes #719, #720

## Tests
Added regression test for Windows
2023-08-31 14:10:32 +00:00
Pieter Noordhuis ca2f1dc06c
Filter down to Python wheel tasks only for trampoline (#712)
## Changes

Fixes issue introduced in #635.

## Tests

Added new unit test to confirm correct behavior.

Manually deployed sample bundle.
2023-08-30 13:51:15 +00:00
Andrew Nester 12368e3382
Added transformation mutator for Python wheel task for them to work on DBR <13.1 (#635)
## Changes
***Note: this PR relies on sync.include functionality from here:
https://github.com/databricks/cli/pull/671***

Added transformation mutator for Python wheel task for them to work on
DBR <13.1

Using wheels upload to Workspace file system as cluster libraries is not
supported in DBR < 13.1

In order to make Python wheel work correctly on DBR < 13.1 we do the
following:
1. Build and upload python wheel as usual
2. Transform python wheel task into special notebook task which does the
following
a. Installs all necessary wheels with %pip magic
b. Executes defined entry point with all provided parameters
3. Upload this notebook file to workspace file system
4. Deploy transformed job task

This is also beneficial for executing on existing clusters because this
notebook always reinstall wheels so if there are any changes to the
wheel package, they are correctly picked up

## Tests
bundle.yml

```yaml
bundle:
  name: wheel-task

workspace:
  host: ****

resources:
  jobs:
    test_job:
      name: "[${bundle.environment}] My Wheel Job"
      tasks:
        - task_key: TestTask
          existing_cluster_id: "***"
          python_wheel_task:
            package_name: "my_test_code"
            entry_point: "run"
            parameters: ["first argument","first value","second argument","second value"]
          libraries:
          - whl: ./dist/*.whl
```

Output
```
andrew.nester@HFW9Y94129 wheel % databricks bundle run test_job
Run URL: ***

2023-08-03 15:58:04 "[default] My Wheel Job" TERMINATED SUCCESS 
Output:
=======
Task TestTask:
Hello from my func
Got arguments v1:
['python', 'first argument', 'first value', 'second argument', 'second value']

```
2023-08-30 12:21:39 +00:00