Bey­ond BI: Inter­act­ive Data Applic­a­tions with Dat­ab­ricks and Plotly Dash

The Shift from Pass­ive Report­ing to Act­ive Data Apps

Tra­di­tion­ally, data plat­forms based on Dat­ab­ricks or Snow­flake were designed for a one-way flow: data moves from sources to a ware­house, and then to a BI tool for visu­al­iz­a­tion and ana­lysis. In most cases, the devel­op­ment pro­cess involves busi­ness users provid­ing require­ments, a Busi­ness Analyst trans­lat­ing those into tech­nical spe­cific­a­tions, and a team of Data Engin­eers build­ing the solu­tion. While agile meth­od­o­lo­gies are often used to bridge these gaps, this struc­ture can struggle with more soph­ist­ic­ated needs.

Over the last dec­ade, busi­ness teams have become more tech­nical and inter­act­ive, and the need for fast changes in data ana­lysis and pro­cessing has increased. As a res­ult, we have iden­ti­fied a shift toward using Data Applic­a­tions instead of tra­di­tional report­ing. We use Plotly Dash for para­met­er­ized data deliv­ery solu­tions. How­ever, such a devel­op­ment also trig­gers sig­ni­fic­ant concerns:

• What hap­pens when a busi­ness user needs to do more than just view data?
• What if they need to trig­ger a com­plex recal­cu­la­tion, run a sim­u­la­tion, or gen­er­ate a cus­tom data mart on the fly?
• How can we integ­rate “power users” – those with both deep busi­ness and tech­nical expert­ise – dir­ectly into the devel­op­ment pro­cess, rather than rely­ing on a single Busi­ness Analyst?

In this post, we explore a frame­work for using Dat­ab­ricks as a power­ful backend for Plotly Dash as a web-based applic­a­tion. This setup, cre­ated by an inter­dis­cip­lin­ary team of tech­nical and busi­ness experts, enables a seam­less bridge between high-level busi­ness logic and scal­able data processing.

The Fron­tend: Plotly Dash & Dat­ab­ricks Connect

While Dat­ab­ricks note­books and pipelines are the stand­ard for data pro­cessing, we iden­ti­fied a power­ful way to build stan­dalone Python applic­a­tions that inter­act dir­ectly with Dat­ab­ricks resources. To make data truly action­able, you need a respons­ive inter­face. Plotly Dash provides this by offer­ing a Python-based frame­work to build ana­lyt­ical web applic­a­tions without requir­ing deep JavaS­cript knowledge.

The key ena­bler here is Spark Con­nect. It decouples the cli­ent (the Dash app) from the Spark server (Dat­ab­ricks), allow­ing the applic­a­tion to man­aging local work­loads inside the app and man­aging work­loads in com­bin­a­tion with the backend in Databricks.

Trig­ger Inter­act­ive Processing

We can run com­plex Spark jobs on a remote cluster dir­ectly from a UI com­pon­ent. When work­ing inter­act­ively with data that has inter­me­di­ate res­ults stored dur­ing pro­cessing, it is cru­cial not to wait for back­ground batch jobs. From this per­spect­ive, Dat­ab­ricks Con­nect makes it pos­sible to pro­cess data dir­ectly from a UI. The pro­cesses run on a Dat­ab­ricks cluster with dir­ect reads and writes to Dat­ab­ricks tables:

spark = DatabricksSession.builder.remote( 
host="https://adb-xxxx.x.azuredatabricks.net",
token="your-access-token", 
cluster_id="xxxx-xxxxxx-xxxxxx" ).getOrCreate()

pdf_info_version = spark.read.table(info_table).toPandas()

We can receive inform­a­tion on actual data from a Dat­ab­ricks table dir­ectly in the fron­tend. Usab­il­ity bene­fits for these use cases include:

• Com­bin­ing heavy, data-intens­ive work­loads com­pletely within Databricks.
• Work­ing on small data­sets and user-man­aged para­met­riz­a­tions dir­ectly in the app.
• Provid­ing pro­cess man­age­ment cap­ab­il­it­ies to review ana­lyt­ics and dir­ectly trig­ger para­met­rized workloads.

In Fig. 1, we present an example app for manual con­fig­ur­a­tion pro­cessing built with a Dash UI. It shows an over­view of all Dat­ab­ricks job runs, along with addi­tional tabs for data approvals and con­fig­ur­a­tion set­tings. Depend­ing on the pro­ject, it is pos­sible to com­bine both: manual, para­met­rized jobs on the one hand, and sched­uled, auto­mated Dat­ab­ricks jobs on the other.

Trig­ger­ing Work­loads: From the Dash UI, users can adjust para­met­ers and trig­ger Spark jobs that execute in the Dat­ab­ricks backend in real time (Fig­ure 1).

Bridge the Gap: Trans­ition seam­lessly from auto­mated job runs in a note­book to a pro­duct­ive web app using the same Python ecosystem.

Semi-Auto­ma­tion: This cre­ates a “human-in-the-loop” sys­tem where the fron­tend col­lects user input and the backend handles the heavy lift­ing. Data qual­ity and manual inputs can be man­aged by the user dir­ectly in the frontend.

Data Report­ing Con­fig­ur­a­tion in Plotly Dash

To handle com­plex work­loads in ana­lyt­ics use cases, we intro­duced an auto­mated ana­lysis con­fig­ur­a­tion approach. For example, when deal­ing with a P&L (Profit & Loss) table con­tain­ing income and expense data, busi­ness teams often need to define new ratio KPIs. While in most data pro­jects this intro­duces a new require­ment for the defined report, we provide a solu­tion with a con­fig­ur­a­tion table that defines cal­cu­la­tions for dif­fer­ent ratios between income and expenses.

With this approach, it is pos­sible to min­im­ize effort in pipeline devel­op­ment because busi­ness users can align the report­ing res­ults dir­ectly after newly adding data to the sil­ver layer. By defin­ing the job con­fig­ur­a­tion via a JSON file (Fig­ure 2) and adding new KPIs via an Excel con­fig­ur­a­tion, the report­ing is much more flex­ible than with a com­pletely code-based ETL pipeline for the data pre­par­a­tion for the report­ing layer.

The Engine: Com­plex Work­loads with Spark & SQL Notebooks

To trig­ger this con­fig­ur­a­tion, a user ini­ti­ates an action in the fron­tend, and the backend handles the con­figured report­ing cal­cu­la­tion and built respons­ive data model. By util­iz­ing Dat­ab­ricks Note­books in PyS­park, developers can:

Build Stand­ard­ized Work­loads: Cre­ate con­sist­ent logic for both sched­uled batch pro­cessing and real-time, user-triggered events.

Orches­trate Multi-Step Pro­cesses: Execute com­plex sequences and provide users with imme­di­ate feed­back by retriev­ing job status and metadata.

Lever­age Dis­trib­uted Com­pute: Use Spark’s engine to pro­cess massive data­sets that would oth­er­wise over­whelm a tra­di­tional web server.

Why Archi­tec­ture Matters

When trig­ger­ing Spark work­loads dir­ectly from an applic­a­tion, a scal­able and well-defined data struc­ture is essen­tial. Our frame­work com­bines large-scale data pro­cessing with inter­act­ive con­fig­ur­a­tions in Plotly Dash. How­ever, even the most advanced tech­nical setup requires a solid busi­ness con­text as its found­a­tion. This ensures that the data being manip­u­lated remains rel­ev­ant and provides actual value to the end user.

Gov­ernance and Dis­cov­ery: Power­ing Busi­ness Units via Unity Catalog

A data applic­a­tion is only as good as the data it con­sumes. Using Unity Cata­log, we ensure that the web applic­a­tion oper­ates within a strictly gov­erned environment:

Cent­ral­ized Access: Dif­fer­ent busi­ness units see only the data they are author­ized to see and to do ana­lysis on.

Lin­eage: Track how data flows from the raw source to the user-defined report in the Dash app.

The goal of data dis­cov­ery for all busi­ness units is to avoid unclean and incon­sist­ent data qual­ity. The main goal of applic­a­tions with a cent­ral­ized data lake­house is to provide a solu­tion to remove leg­acy applic­a­tions. Manual file hand­ling leads to incon­sist­ent data and untrace­able data changes. By provid­ing a con­nec­ted fron­tend solu­tion, we offer his­tor­ized and ver­sioned para­met­ers for all entry points in the data pro­cessing engine.

Self-Ser­vice Archi­tec­ture: Build­ing Gov­erned Data Marts

One of the most power­ful use cases for this concept is the cre­ation of user-defined Data Marts. Instead of wait­ing weeks for IT to build a new table, a busi­ness user can use the Dash fron­tend to:

Select spe­cific gov­erned data­sets from the Unity Catalog.

Apply busi­ness-spe­cific fil­ters or transformations.

Click “Gen­er­ate,” trig­ger­ing a Spark job that writes a new, gov­erned Data Mart back into the Lakehouse.

We provide a com­pletely sep­ar­ate Unity Cata­log for this use case to dis­tin­guish between cent­ral­ized and user-deployed work­loads. We cre­ate user-defined cus­tom­ized data marts. The main object­ive of busi­ness-led data ana­lysis is to accel­er­ate the deliv­ery of a reli­able data struc­ture for report­ing requirements.

In the past, when a data ana­lyt­ics team wanted to cre­ate a data mart, an entire sub-pro­ject had to be hos­ted. Data­base deploy­ments, schemas, and tables had to be deployed, and sub­sequently, sev­eral users cre­ated Excel exports of the res­ult­ing data­sets because they wanted to be able to imple­ment aggreg­a­tions and report­ing KPIs on their own. Now, we can deliver a fron­tend that can build a small Data Mart on top of our exist­ing frame­work by enabling app users to cre­ate a table on top of already exist­ing, stand­ard­ized datasets.

This turns the data plat­form into a self-ser­vice fact­ory, where the data remains gov­erned, but the agil­ity remains with the business.

Con­clu­sion: The Future is Data-Driven Apps

We com­bine the flex­ib­il­ity of Plotly Dash with the massive scale of Dat­ab­ricks. This makes it pos­sible to gov­ern data pro­cessing via Unity Cata­log. With this func­tion­al­ity, organ­iz­a­tions can move past static charts and shar­ing data via exports and file trans­fers between pro­jects. This archi­tec­ture allows you to build internal tools that don’t just report on the business—they run it.

In fol­low-up blog posts, we will provide a detailed over­view of build­ing an approach for busi­ness-cre­ated data pipelines and gov­erned tools to make the data more under­stand­able and usable.

Estim­ated read­ing time: 7 minutes