steve/databricks-cli
steve
/
databricks-cli
mirror of https://github.com/databricks/cli.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Enable `spark_jar_task` with local JAR libraries (#993) 

## Changes
Previously local JAR paths were transformed to remote path during
initialisation and thus artifact building logic did not recognise such
libraries as local to be handled and uploaded.

Now it's possible to use spark_jar_tasks with local JAR libraries on
14.1+ DBR clusters

Example configuration
```
bundle:
  name: spark-jar

workspace:
  host: ***

artifacts:
  my_java_code:
    path: ./sample-java
    build: "javac PrintArgs.java && jar cvfm PrintArgs.jar META-INF/MANIFEST.MF PrintArgs.class"
    files:
      - source: "/Users/andrew.nester/dabs/wheel/sample-java/PrintArgs.jar"

resources:
  jobs:
    print_args:
      name: "Print Args"
      tasks:
        - task_key: Print
          new_cluster:
            num_workers: 0
            spark_version: 14.2.x-scala2.12
            node_type_id: i3.xlarge
            spark_conf:
              "spark.databricks.cluster.profile": "singleNode"
              "spark.master": "local[*]"
            custom_tags:
              ResourceClass: "SingleNode"
          spark_jar_task:
            main_class_name: PrintArgs
          libraries:
            - jar: ./sample-java/PrintArgs.jar
```

## Tests
Manually running `bundle deploy and bundle run`
This commit is contained in:
Andrew Nester 2023-11-21 11:15:09 +01:00 committed by GitHub
parent 92539d4b9b
commit fa89db57e9
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
2 changed files with 4 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
bundle/config/mutator/translate_paths_jobs.go
View File

@ -46,7 +46,7 @@ func transformWhlLibrary(resource any, dir string) *transformer {
dir, dir,
&library.Whl, &library.Whl,
"libraries.whl", "libraries.whl",
translateNoOp, translateNoOp, // Does not convert to remote path but makes sure that nested paths resolved correctly
} }
} }
@ -88,7 +88,7 @@ func transformJarLibrary(resource any, dir string) *transformer {
dir, dir,
&library.Jar, &library.Jar,
"libraries.jar", "libraries.jar",
translateFilePath, translateNoOp, // Does not convert to remote path but makes sure that nested paths resolved correctly
} }
} }

4
bundle/config/mutator/translate_paths_test.go
View File

@ -238,7 +238,7 @@ func TestTranslatePaths(t *testing.T) {
) )
assert.Equal( assert.Equal(
t, t,
"/bundle/dist/task.jar", filepath.Join("dist", "task.jar"),
b.Config.Resources.Jobs["job"].Tasks[5].Libraries[0].Jar, b.Config.Resources.Jobs["job"].Tasks[5].Libraries[0].Jar,
) )
assert.Equal( assert.Equal(
@ -352,7 +352,7 @@ func TestTranslatePathsInSubdirectories(t *testing.T) {
) )
assert.Equal( assert.Equal(
t, t,
"/bundle/job/dist/task.jar", filepath.Join("job", "dist", "task.jar"),
b.Config.Resources.Jobs["job"].Tasks[1].Libraries[0].Jar, b.Config.Resources.Jobs["job"].Tasks[1].Libraries[0].Jar,
) )
assert.Equal( assert.Equal(