Long short story, it’s a mix between DevOps and Data Engineering. Data Engineer that works like DevOps but is not too deep in managing infrastructure yet also understands how data pipelining works and some SQL logics. The idea comes when an information technology company grows beyond a medium-sized company1According to the European Commission, SMEs are enterprises that meet the following definition of staff headcount and either the turnover or balance sheet total definitions.. To serve such a large amount of internal stakeholders, the Data department (BigData, Data & Analytics, or whatever the name) needs a team to quickly respond to any infrastructure-related issues to everyone in the Data department.
This has become my thing since my previous position. I really thankful to my lead at that time to allow me to manage some cloud infrastructure in AWS. As I said before, the reason why my team needs people to manage the cloud infrastructure for our Data Engineering team was that we need quick action for our VMs. Also, as hungry as we were when we want to deploy Dremio back then, we want to fine-tune our needs for it. We realized that rather than waiting for our DevOps team’s workload to be lessened, we decided to ask for a piece of cake, no, part of their kitchen to be used directly by the Data team. And so, we started baking the cake on our own.
The work is pretty much much simpler than what DevOps did. Provisioning some VMs or service clusters, maintain the VMs and their services, deploy new stacks, maintain some network connection, and so on. That’s just covering the very basics of cloud infrastructure. Of course, the DevOps team, or rather the Cloud Infrastructure team and other teams like IT security are still in touch with our infrastructure for obvious reasons. Since the work is more basics, we can still be in touch with our own data pipeline and improving it to become more robust.
The title finally become official in the company structure in my current role2I’m the Data Infrastructure Engineering Lead in LinkAja as of this article written.. Yey~
The scope of work as Data Infrastructure isn’t pretty much different from my previous company. We do basic cloud infrastructure and the cloud infrastructure team and IT security team help us with advanced networking, monitoring platform, and infrastructure security. And since the Data Engineering team has infrastructure flexibility, we can supercharge our pipeline development from batching into streaming.