Data Mart to Data Mastery: How Tarun Parmar Transformed Manufacturing Data Pipelines

Subham Kumar

22 Aug 2022 — 3 min read

Tarun Parmar

In today’s digital world, in modern manufacturing, data is as critical as the materials that go into production. Data engineers try to make sure that the data is extracted, processed, and stored where it is supposed to be. One such data wizard named Tarun Parmar, has been important in guiding the handling of manufacturing data, simplifying access and application for technical and operational units.

Parmar was at the forefront of the batter manufacturing group's inception, He had to build the data backbone from scratch, which meant interviewing 30 candidates and bringing on three dedicated full-time data engineers. On top of this, he took young interns under his guidance, steering them towards becoming regular contributors. His daily work hovered around architecting cloud-operated information systems capable of handling mammoth manufacturing data sets, improving reporting accuracy, and accelerating decision-making processes.

By implementing a cloud-based data mart and OLAP system, he enabled engineers to query data more efficiently, reducing the time it took to retrieve critical insights from hours to minutes.

For the smooth processing of the data, he built batch-processing pipelines with Python, Apache Airflow, Presto SQL (Trino), and Kubernetes. These pipelines ensured that manufacturing data was processed at scale without delays. In parallel, he led the creation of dashboards that monitored quality metrics like Defective Parts Per Million (DPPM), which helped engineering teams reduce failure rates and inventory inefficiencies.

To further improve efficiency, he facilitated over 300 JIRA tickets, including multiple impactful JIRA Epics, and developed JMP scripts for process parameter analysis, accelerating data validation efforts, which was a critical step in preparing for the launch of a much-anticipated new product.

One of Parmar's major projects was the design and implementation of a cloud data mart consisting of over 300 database tables. This system consolidated manufacturing data from multiple sources, providing a single access point for engineers and analysts to retrieve information, which resulted in an improved data retrieval ability and data processing ability.

Another critical project was leading JIRA epics related to key performance indicator (KPI) reporting. These initiatives included dashboards for tracking defect rates, quality containment reporting, and manufacturing capability metrics like Cp and Cpk. All the insights were comprehensively documented for data pipelines, dashboards, and quality systems and were used across the company’s manufacturing departments to pass on the already established knowledge base.

There were also noteworthy results of these efforts. Data processing speed improved dramatically, reducing query times from hours to minutes. Enhanced Cp and Cpk reporting helped teams optimize manufacturing precision, leading to better product quality. Over 300 JIRA requests were addressed, streamlining workflows across departments. These improvements contributed to a more streamlined manufacturing environment where data-driven decisions could be made quickly.

While stumbling on these results, Parmar also had to navigate certain considerations. Such as the challenge of scaling data pipelines for over 300 tables, ensuring high availability and minimal processing delays. Another challenge was establishing a new team and infrastructure from scratch and managing the gaps between departments that had their distinct data needs and considerations. However, his passion for creating solutions for seamless data integrations and consistent availability of the required data, helped him to sail through the challenges.

Looking ahead at industry trends, Parmar notes a noticeable shift toward facilitating access to analytical data for non-technical teams. This allows them to comprehend and utilize the data without the need for engineer assistance. He also sees growth in real-time analysis to improve operational agility and predicts that cloud-originated technologies will persist in moulding the era of manufacturing data systems. Moreover, AI and machine learning are set to become commonplace for predictive analytics and abnormality recognition, aiding manufacturers in pre-emptively managing quality and productivity issues.

In line with the changing dynamics of production-related data collection, processing, and utilization for making decisions Parmar's contributions echo this metamorphosis. His efforts in constructing scalable models, promoting teamwork, and simplifying systems have established strong foundational elements for a data-guided production atmosphere that will continue to progress with future tech innovations.