feat: Lakeflow Jobs Plugin by atilafassina · Pull Request #265 · databricks/appkit

atilafassina · 2026-04-10T20:40:56Z

Summary

Resource-scoped jobs plugin following the files plugin pattern. Jobs are configured as named resources discovered from environment variables at startup.

Design

Resource-scoped: Only configured jobs are accessible — this is not an open SDK wrapper
Env-var discovery: Jobs are discovered from DATABRICKS_JOB_<KEY> env vars (e.g. DATABRICKS_JOB_ETL=123)
Single-job shorthand: DATABRICKS_JOB_ID maps to the "default" key
Manifest declares job resources with CAN_MANAGE_RUN permission
Works with databricks apps init --features jobs

API

// Trigger a configured job
const { run_id } = await appkit.jobs("etl").runNow();

// Trigger and wait for completion
const run = await appkit.jobs("etl").runNowAndWait();

// OBO access
await appkit.jobs("etl").asUser(req).runNow();

// List recent runs
const runs = await appkit.jobs("etl").listRuns({ limit: 10 });

// Single-job shorthand
await appkit.jobs("default").runNow();

Files changed

plugins/jobs/manifest.json — declares job resource with CAN_MANAGE_RUN permission
plugins/jobs/types.ts — JobAPI, JobHandle, JobsExport, IJobsConfig types
plugins/jobs/plugin.ts — JobsPlugin with discoverJobs(), getResourceRequirements(), resource-scoped createJobAPI()
plugins/jobs/index.ts — barrel exports
connectors/jobs/client.ts — listRuns now respects limit parameter
plugins/jobs/tests/plugin.test.ts — 32 tests covering discovery, resource requirements, exports, OBO, multi-job, and auto-fill

Documentation safety checklist

Examples use least-privilege permissions
Sensitive values are obfuscated
No insecure patterns introduced

Reopened from #223 — moved head branch from fork to upstream to fix CI secrets access.

Jobs are configured as named resources (DATABRICKS_JOB_<KEY> env vars) and discovered at startup, following the files plugin pattern. API is scoped to configured jobs: appkit.jobs('etl').runNow() appkit.jobs('etl').runNowAndWait() appkit.jobs('etl').lastRun() appkit.jobs('etl').listRuns() appkit.jobs('etl').asUser(req).runNow() Single-job shorthand via DATABRICKS_JOB_ID env var. Supports OBO access via asUser(req). Co-authored-by: Isaac Signed-off-by: Evgenii Kniazev <evgenii.kniazev@databricks.com>

Co-authored-by: Isaac Signed-off-by: Atila Fassina <atila@fassina.eu>

…nfig Phase 1 of jobs plugin rework: establishes config foundation with TaskType union, three interceptor tiers (read/write/stream), and pure parameter mapping for all 6 task types. Co-authored-by: Isaac Signed-off-by: Atila Fassina <atila@fassina.eu>

…reaming Wave 2 of jobs plugin rework: - Fix connector protobuf types (Waiter access pattern) - Remove waitForRun from connector, add runAndWait async generator in plugin - Wrap reads in execute() with JOBS_READ_DEFAULTS, writes with JOBS_WRITE_DEFAULTS - Add Zod param validation and mapParams integration to runNow - Enrich clientConfig() with JSON Schema per job - Update JobAPI types and adapt existing tests Co-authored-by: Isaac Signed-off-by: Atila Fassina <atila@fassina.eu>

Phase 5 of jobs plugin rework: adds injectRoutes() with 4 endpoints: - POST /:jobKey/run (with ?stream=true SSE support) - GET /:jobKey/runs (paginated) - GET /:jobKey/runs/:runId (detail) - GET /:jobKey/status (latest run) All routes use OBO via asUser(req) and validate job keys. Co-authored-by: Isaac Signed-off-by: Atila Fassina <atila@fassina.eu>

- Fix ERR_HTTP_HEADERS_SENT crash when streaming errors occur after headers are flushed (check res.headersSent before setting status) - Fix spark_jar/sql param mapping to use correct SDK fields (jar_params, sql_params instead of parameters) - Forward abort signal from execute() interceptors to all connector calls for proper timeout/cancellation support - Use Zod result.data after validation to preserve schema transforms and defaults - Validate params even when omitted if a schema is configured - Fix lastRun return type from Run to BaseRun (matches listRuns) - Fix getJobId error message for default job key Co-authored-by: Isaac Signed-off-by: Atila Fassina <atila@fassina.eu>

The asUser() proxy wraps createJobAPI() in user context, but the returned JobAPI closures called this.client after the ALS scope exited, falling back to service principal. Fix by capturing the client and userId eagerly at creation time and passing them explicitly to execute() and connector calls. Co-authored-by: Isaac Signed-off-by: Atila Fassina <atila@fassina.eu>

Migrate streaming to executeStream(), route polling through execute(), use ValidationError, standardize eager captures, tighten types, expose taskType in clientConfig, remove .job() alias, update manifest docs. Co-authored-by: Isaac Signed-off-by: Atila Fassina <atila@fassina.eu>

…ms before streaming - Cap listRuns to [1, 100] in both connector and HTTP route to prevent unbounded memory materialization - Reject negative runId (<=0) in GET/DELETE run handlers - Validate Zod params before entering SSE stream branch so bad params return 400 JSON instead of a generic SSE error event Co-authored-by: Isaac Signed-off-by: Atila Fassina <atila@fassina.eu>