4 Ways to Improve Data Security With Databricks Unity Catalog

This April, Databricks announced end-of-life for their Standard tier workspaces, meaning that Databricks customers will either need to upgrade to the Premium or Enterprise tier or migrate to a different platform by April 25, 2025. For current customers of the Standard tier Databricks plan, this means those who remain with the platform will need to embrace Unity Catalog, Databricks’ data governance solution. 

This is the third and final post in our series on what data teams need to know about the transition to Databricks Unity Catalog. In the first post in our Unity Catalog series, we gave an overview of Unity Catalog, including its main capabilities and limitations in the realm of data access control. The second post detailed the value of integrating Databricks Unity Catalog with a Data Security Platform (DSP) such as Satori. This final article is geared towards teams who are currently using Unity Catalog, or have made the decision to switch. We’ll go through 5 ways data teams using Unity Catalog are approaching data security and access control in their organizations. 

Access control is complicated enough for midsize companies with departments like marketing, sales, product, and R&D, all with different data access requirements. When you factor in the globally distributed teams and more complex business domains and relationships that come with large enterprises, data security and access control get exponentially more complex. Each country has its own compliance standards, each business domain may have multiple kinds of roles, all with different uses for data and thus different data security needs. 

In a media streaming company, for example, a data engineer on the recommender systems team in Ecuador has pretty different data access needs than a marketing analyst on the studio relations team in Germany. In short, data access is hard to manage in large enterprises. When data access is managed by data teams with more technical expertise, Unity Catalog can serve as a solid platform for implementing RBAC and RLS on Databricks environments. Even still, many data teams find themselves managing access with frustrating manual methods, using SQL, multiple views, or Excel spreadsheets. A data team might be technically able to manage data access with SQL, but doing so manually in large, diverse environments takes time away from more interesting and complex work. Being able to automate these processes frees up tons of resources for data teams.

On the other hand, data security and access are very often managed by less technical teams, such as audit and compliance teams. These teams may not have the SQL skills to manage data access using native Unity Catalog controls, especially when additional data sources and BI tools are involved. In this case, it’s especially helpful to use a centralized platform to manage access to multiple data stores, including Databricks, BI tools, and other data sources, without the need for SQL. Using a data security platform like Satori, teams can implement RLS, RBAC, ABAC, and temporary data access on a variety of complex use cases.

An increasing number of data teams are migrating data from legacy architectures such as SAP into cloud data warehouses like Amazon Redshift, Snowflake, and of course, Databricks. Migrating off of proprietary technologies, especially after many years of use, can come with complications. The migration itself can be an undertaking, with projects often taking months or years to complete. During the process, data teams’ productivity is hindered by significantly reduced ability to share data within the organization. At the same time, sensitive data is more exposed to security threats during a migration, affecting compliance. 

Clearly, speed, safety, and cost are major concerns in a migration project into Databricks. Satori helps data teams migrate data securely to Databricks by providing data governance that’s decoupled from the data plane. What this means is that data access is automated from the beginning of the migration, and can easily be shared between business domains. Satori also offers automatic classification, which automatically scans and applies security policies on sensitive data. Data teams have access to both Satori’s native classifiers and your organization’s custom classifiers. Adopting a DSP allows teams to easily protect sensitive data during and after a large migration project.

If access to Databricks is granted based on role, geographical location, or business domain, this should also be the case for your BI tools and operational databases. Tools like Power BI and Tableau contain rudimentary RBAC and RLS abilities, but they typically require specific knowledge of DAX, SQL, and/or Excel. For less technical teams, this is not the ideal way to manage data access. Even for technical teams, however, managing access to BI tools often requires manual configuration using separate SQL and Excel files, adding unnecessary overhead. If the team is managing access to Databricks, BI tools, and production databases, or other data stores, the manual work required increases significantly as changes must be continuously synchronized across the entire environment. Any data team would benefit from automating data access. Here’s how Satori helps data teams automate data security through the two most popular BI tools used today, Power BI and Tableau.

Unity Catalog users today have two options for setting up fine-grained access control in PowerBI. In Import mode, and often in Direct Query Mode, PowerBI authenticates to Databricks via a service account. In this case, RLS must be set in Power BI itself, as Databricks only has context on the service account rather than on the individual users accessing data. In this scenario, you would need to set security policies both in Databricks, for users querying directly in Databricks, and in Power BI. The other option, for teams that want to govern access control directly in Unity Catalog without having to also set it up in Power BI, is to use AAD passthrough. This option is only available with Direct Query.

Like in Power BI, today Tableau users have two options for setting up row-level security. On tables using live queries, you can set security policies in the database. This is possible on extracts as well, but requires teams to create and manage an entitlements file, which adds manual work and is difficult to scale. The alternative is to set manual or dynamic user filters within Tableau. This is the simplest solution, but comes with the tradeoff of being cumbersome to maintain with any more than a few users.

In PowerBI, native Unity Catalog controls give you two options: manage data access in Unity Catalog and Power BI, or only in Unity Catalog with AAD passthrough. Tableau provides the same channels for access control as Power BI, with similar tradeoffs. It ends up coming down to whether to set access controls in Unity Catalog, the BI tool, the production database, or some combination of the three.

This is where a Data Security Platform really makes an impact in reducing time spent manually managing data access. When you can manage all data access from a single portal, setting ABAC security policies on your data warehouses and lakes, BI tools, and production databases, there’s no more need to manage security on multiple tools. This saves significant time for technical and non-technical teams alike, with the added bonus of a more intuitive interface than Unity Catalog, which is geared towards hardcore data scientists and engineers. 

A common pain point we’ve seen in teams we work with is the need to meet requirements for different compliance standards. If you’re in a large enterprise that spans multiple geographical locations, it’s not trivial to general reports and answer questions about the locations, sources, permissions, and access patterns of sensitive data. While Unity Catalog does maintain audit logs for data access, they require SQL knowledge to query and get information from them. With a Data Security Platform like Satori, teams with less technical knowledge can easily create customized reports and identify behavior anomalies from all your data stores, including operational data stores. Satori also enriches audit logs with metadata about all data access, including identity data, types of sensitive data, and the application of any security policies.

Data consumers can occasionally run a query that’s costly and suboptimal. Satori provides complete visibility into all data access across your environment, including querying. This means that you can find information on queries, who made them, and block or limit them as needed. Since audit logs show the query duration on compute, it can be used to reduce Databricks costs immediately, instead of relying on training or manual methods of optimizing queries. 

As more data teams adopt Unity Catalog, we’ve seen an increasing number of enterprises coming to us in search of a way to extend the access control capabilities of the platform. Unity Catalog is great for implementing RBAC policies within Databricks, but falters when you need to apply data security to BI tools, production databases, or additional data warehouses. What’s great about Unity Catalog, however, is that it was built with extensibility in mind. When used in conjunction with a data security platform like Satori, you can take advantage of Unity Catalog’s native capabilities while also applying them to your other data stores and tools.