Bey­ond BI: Inter­ac­tive Data Appli­ca­ti­ons with Dat­ab­ricks and Plotly Dash

The Shift from Pas­sive Report­ing to Active Data Apps

Tra­di­tio­nally, data plat­forms based on Dat­ab­ricks or Snow­flake were desi­gned for a one-way flow: data moves from sources to a ware­house, and then to a BI tool for visua­liza­tion and ana­ly­sis. In most cases, the deve­lo­p­ment pro­cess invol­ves busi­ness users pro­vi­ding requi­re­ments, a Busi­ness Analyst trans­la­ting those into tech­ni­cal spe­ci­fi­ca­ti­ons, and a team of Data Engi­neers buil­ding the solu­tion. While agile metho­do­lo­gies are often used to bridge these gaps, this struc­ture can struggle with more sophisti­ca­ted needs.

Over the last decade, busi­ness teams have become more tech­ni­cal and inter­ac­tive, and the need for fast chan­ges in data ana­ly­sis and pro­ces­sing has increased. As a result, we have iden­ti­fied a shift toward using Data Appli­ca­ti­ons ins­tead of tra­di­tio­nal report­ing. We use Plotly Dash for para­me­ter­i­zed data deli­very solu­ti­ons. Howe­ver, such a deve­lo­p­ment also trig­gers signi­fi­cant concerns:

• What hap­pens when a busi­ness user needs to do more than just view data?
• What if they need to trig­ger a com­plex recal­cu­la­tion, run a simu­la­tion, or gene­rate a cus­tom data mart on the fly?
• How can we inte­grate «power users» – those with both deep busi­ness and tech­ni­cal exper­tise – directly into the deve­lo­p­ment pro­cess, rather than rely­ing on a sin­gle Busi­ness Analyst?

In this post, we explore a frame­work for using Dat­ab­ricks as a powerful backend for Plotly Dash as a web-based appli­ca­tion. This setup, crea­ted by an inter­di­sci­pli­nary team of tech­ni­cal and busi­ness experts, enables a seam­less bridge bet­ween high-level busi­ness logic and sca­lable data processing.

The Front­end: Plotly Dash & Dat­ab­ricks Connect

While Dat­ab­ricks note­books and pipe­lines are the stan­dard for data pro­ces­sing, we iden­ti­fied a powerful way to build stan­da­lone Python appli­ca­ti­ons that inter­act directly with Dat­ab­ricks resour­ces. To make data truly actionable, you need a respon­sive inter­face. Plotly Dash pro­vi­des this by offe­ring a Python-based frame­work to build ana­ly­ti­cal web appli­ca­ti­ons wit­hout requi­ring deep Java­Script knowledge.

The key enabler here is Spark Con­nect. It decou­ples the cli­ent (the Dash app) from the Spark ser­ver (Dat­ab­ricks), allo­wing the appli­ca­tion to mana­ging local workloads inside the app and mana­ging workloads in com­bi­na­tion with the backend in Databricks.

Trig­ger Inter­ac­tive Processing

We can run com­plex Spark jobs on a remote clus­ter directly from a UI com­po­nent. When working inter­ac­tively with data that has inter­me­diate results stored during pro­ces­sing, it is cru­cial not to wait for back­ground batch jobs. From this per­spec­tive, Dat­ab­ricks Con­nect makes it pos­si­ble to pro­cess data directly from a UI. The pro­ces­ses run on a Dat­ab­ricks clus­ter with direct reads and wri­tes to Dat­ab­ricks tables:

spark = DatabricksSession.builder.remote( 
host="https://adb-xxxx.x.azuredatabricks.net",
token="your-access-token", 
cluster_id="xxxx-xxxxxx-xxxxxx" ).getOrCreate()

pdf_info_version = spark.read.table(info_table).toPandas()

We can receive infor­ma­tion on actual data from a Dat­ab­ricks table directly in the front­end. Usa­bi­lity bene­fits for these use cases include:

• Com­bi­ning heavy, data-inten­sive workloads com­ple­tely within Databricks.
• Working on small data­sets and user-mana­ged para­me­triza­ti­ons directly in the app.
• Pro­vi­ding pro­cess manage­ment capa­bi­li­ties to review ana­ly­tics and directly trig­ger para­me­tri­zed workloads.

In Fig. 1, we pre­sent an exam­ple app for manual con­fi­gu­ra­tion pro­ces­sing built with a Dash UI. It shows an over­view of all Dat­ab­ricks job runs, along with addi­tio­nal tabs for data appr­ovals and con­fi­gu­ra­tion set­tings. Depen­ding on the pro­ject, it is pos­si­ble to com­bine both: manual, para­me­tri­zed jobs on the one hand, and sche­du­led, auto­ma­ted Dat­ab­ricks jobs on the other.

Trig­ge­ring Workloads: From the Dash UI, users can adjust para­me­ters and trig­ger Spark jobs that exe­cute in the Dat­ab­ricks backend in real time (Figure 1).

Bridge the Gap: Tran­si­tion seam­lessly from auto­ma­ted job runs in a note­book to a pro­duc­tive web app using the same Python ecosystem.

Semi-Auto­ma­tion: This crea­tes a «human-in-the-loop» sys­tem where the front­end coll­ects user input and the backend hand­les the heavy lif­ting. Data qua­lity and manual inputs can be mana­ged by the user directly in the frontend.

Data Report­ing Con­fi­gu­ra­tion in Plotly Dash

To handle com­plex workloads in ana­ly­tics use cases, we intro­du­ced an auto­ma­ted ana­ly­sis con­fi­gu­ra­tion approach. For exam­ple, when deal­ing with a P&L (Pro­fit & Loss) table con­tai­ning income and expense data, busi­ness teams often need to define new ratio KPIs. While in most data pro­jects this intro­du­ces a new requi­re­ment for the defi­ned report, we pro­vide a solu­tion with a con­fi­gu­ra­tion table that defi­nes cal­cu­la­ti­ons for dif­fe­rent ratios bet­ween income and expenses.

With this approach, it is pos­si­ble to mini­mize effort in pipe­line deve­lo­p­ment because busi­ness users can align the report­ing results directly after newly adding data to the sil­ver layer. By defi­ning the job con­fi­gu­ra­tion via a JSON file (Figure 2) and adding new KPIs via an Excel con­fi­gu­ra­tion, the report­ing is much more fle­xi­ble than with a com­ple­tely code-based ETL pipe­line for the data pre­pa­ra­tion for the report­ing layer.

The Engine: Com­plex Workloads with Spark & SQL Notebooks

To trig­ger this con­fi­gu­ra­tion, a user initia­tes an action in the front­end, and the backend hand­les the con­fi­gu­red report­ing cal­cu­la­tion and built respon­sive data model. By uti­li­zing Dat­ab­ricks Note­books in PySpark, deve­lo­pers can:

Build Stan­dar­di­zed Workloads: Create con­sis­tent logic for both sche­du­led batch pro­ces­sing and real-time, user-trig­ge­red events.

Orchest­rate Multi-Step Pro­ces­ses: Exe­cute com­plex sequen­ces and pro­vide users with imme­diate feed­back by retrie­ving job sta­tus and metadata.

Leverage Dis­tri­bu­ted Com­pute: Use Spark’s engine to pro­cess mas­sive data­sets that would other­wise over­whelm a tra­di­tio­nal web server.

Why Archi­tec­ture Matters

When trig­ge­ring Spark workloads directly from an appli­ca­tion, a sca­lable and well-defi­ned data struc­ture is essen­tial. Our frame­work com­bi­nes large-scale data pro­ces­sing with inter­ac­tive con­fi­gu­ra­ti­ons in Plotly Dash. Howe­ver, even the most advan­ced tech­ni­cal setup requi­res a solid busi­ness con­text as its foun­da­tion. This ensu­res that the data being mani­pu­la­ted remains rele­vant and pro­vi­des actual value to the end user.

Gover­nance and Dis­co­very: Powe­ring Busi­ness Units via Unity Catalog

A data appli­ca­tion is only as good as the data it con­su­mes. Using Unity Cata­log, we ensure that the web appli­ca­tion ope­ra­tes within a strictly gover­ned environment:

Cen­tra­li­zed Access: Dif­fe­rent busi­ness units see only the data they are aut­ho­ri­zed to see and to do ana­ly­sis on.

Lineage: Track how data flows from the raw source to the user-defi­ned report in the Dash app.

The goal of data dis­co­very for all busi­ness units is to avoid unclean and incon­sis­tent data qua­lity. The main goal of appli­ca­ti­ons with a cen­tra­li­zed data lake­house is to pro­vide a solu­tion to remove legacy appli­ca­ti­ons. Manual file hand­ling leads to incon­sis­tent data and untraceable data chan­ges. By pro­vi­ding a con­nec­ted front­end solu­tion, we offer his­to­ri­zed and ver­sio­ned para­me­ters for all entry points in the data pro­ces­sing engine.

Self-Ser­vice Archi­tec­ture: Buil­ding Gover­ned Data Marts

One of the most powerful use cases for this con­cept is the crea­tion of user-defi­ned Data Marts. Ins­tead of wai­ting weeks for IT to build a new table, a busi­ness user can use the Dash front­end to:

Sel­ect spe­ci­fic gover­ned data­sets from the Unity Catalog.

Apply busi­ness-spe­ci­fic fil­ters or transformations.

Click «Gene­rate,» trig­ge­ring a Spark job that wri­tes a new, gover­ned Data Mart back into the Lakehouse.

We pro­vide a com­ple­tely sepa­rate Unity Cata­log for this use case to distin­gu­ish bet­ween cen­tra­li­zed and user-deployed workloads. We create user-defi­ned cus­to­mi­zed data marts. The main objec­tive of busi­ness-led data ana­ly­sis is to acce­le­rate the deli­very of a relia­ble data struc­ture for report­ing requirements.

In the past, when a data ana­ly­tics team wan­ted to create a data mart, an entire sub-pro­ject had to be hos­ted. Data­base deploy­ments, sche­mas, and tables had to be deployed, and sub­se­quently, seve­ral users crea­ted Excel exports of the resul­ting data­sets because they wan­ted to be able to imple­ment aggre­ga­ti­ons and report­ing KPIs on their own. Now, we can deli­ver a front­end that can build a small Data Mart on top of our exis­ting frame­work by enab­ling app users to create a table on top of alre­ady exis­ting, stan­dar­di­zed datasets.

This turns the data plat­form into a self-ser­vice fac­tory, where the data remains gover­ned, but the agi­lity remains with the business.

Con­clu­sion: The Future is Data-Dri­ven Apps

We com­bine the fle­xi­bi­lity of Plotly Dash with the mas­sive scale of Dat­ab­ricks. This makes it pos­si­ble to govern data pro­ces­sing via Unity Cata­log. With this func­tion­a­lity, orga­niza­ti­ons can move past sta­tic charts and sha­ring data via exports and file trans­fers bet­ween pro­jects. This archi­tec­ture allows you to build inter­nal tools that don’t just report on the business—they run it.

In fol­low-up blog posts, we will pro­vide a detailed over­view of buil­ding an approach for busi­ness-crea­ted data pipe­lines and gover­ned tools to make the data more under­stan­da­ble and usable.

Esti­ma­ted rea­ding time: 7 minutes