Data engineering teams are a crucial part of any data driven organization. With the vast improvements in cloud data processing and BI tools, many users in the organization can deliver a lot of value from data projects, reports, and insights.
Data engineering teams serve a key role enabling data availability with validated quality, enrichments, and reliability. This is often work that requires a high level of technical expertise, as well as business knowledge.
However, because some of the data platforms are both critical in terms of data security (holds sensitive data and enables users to access that data), and in the complete control of the data engineering teams, in many cases data engineering teams are spending more and more of their valuable time on data security, access control, compliance, and privacy projects and tasks. This, of course, comes at the expense of time spent helping the company make the best use of the data.
What is Cloud Data Security
Cloud data security is a specialized form of data security. It involves ensuring that data, particularly sensitive data, is protected within the cloud or when it moves between environments; including both internally- and third party-managed data.
The high volume and movement of data over multiple environments makes it increasingly difficult for organizations to protect and secure the information. While, cloud vendors and customers share the responsibility for protecting information; ultimately, the responsibility rests with the organization. Within the organization securing the cloud based data generally defaults to the data engineering team.
These security challenges are further compounded because organizations may not know the location of their data and applications in the cloud, who is accessing their data, the security capabilities of the cloud providers security, and the necessity of enforcing different security policies over a range and geography of different cloud environments.
The complexity of data security in the cloud would be better left to security teams. However, the security teams often lack access to the data platforms which are completely controlled by the data engineering teams and often have no other way to gain access but to ask data engineering teams.
To learn more about cloud security please read our guide on Cloud Data Security.
Why This Is A Problem
There are a number of problems that results from data engineering teams who spend an increasingly large amount of time on security, privacy, and compliance which we outline below. You can also read more about why data security projects keep data engineering teams away from their core responsibilities here.
1. Not Their Job
Data engineers want to solve challenges around…data engineering. This is the job that they trained for and the one that they enjoy.
They typically do not enjoy spending the majority of their time on mundane tasks like authorizing users to data (or “yet another GRANT request”), leaving the data engineers frustrated, as well as less productive on their actual tasks.
Data engineers who are unsatisfied are more likely to leave an organization resulting in high turnover costs.
2. Not Enough Time
There are only so many hours in a day. If data engineers are consumed with continual requests for access, data anonymization, and managing data access audits projects, they will have only very limited time to spend on other projects (which likely include things that they enjoy and are trained to do).
Since there is only a finite amount of time in a day, any time spent on security reduces the available time that the data engineer can spend on helping the business make use of the data in the most efficient and productive way.
3. Not Security Experts
Data engineers are not security experts. This is not their area of expertise and in some cases, their solutions could have been more efficient and elegant (security-wise) if it was designed and implemented by security engineers.
While, this may not seem like a significant issue, in order for a business to operate and reach its full potential, it is necessary to have all systems operate as efficiently as possible.
4. Not as Secure
When data engineers spend the majority of their time on these tasks (such as “yet another GRANT request”), they end up frustrated. This frustration can lead to either giving too many permissions to users or not revoking access when necessary; leaving the organization open to a security breach.
Alternatively, they may not grant access to data fast enough, slowing down the use and analysis of the data, thus the business cannot take full advantage of the available resources.
Towards A Solution
An ideal solution would be to move things that are not data engineering work away from data engineers. For example, many organizations build or use platforms that enable data owners or data stewards to manage access to data without the need of “data engineering hand-holding”.
As for security and privacy policies, these would ideally be managed by the relevant teams, in a centralized way, and in a way that’s decoupled from the data platforms themselves.
This is why we built Satori!
Satori is a data security platform that solves exactly this problem. We allow companies to manage their security and privacy policies from a single location, as well as enable an organization to have a self-service data portal. This is done to accelerate data use, as well as to free data engineers from the burden, to help them do great stuff.