Responsibility:
• Design and implement data ingestion pipelines, which processing both batch and streamed data in unstructured, semi- structured and structure form, using Snowflake, Apache Spark, Delta
lake, Delta Engine, Hive and other relevant tech stacks in cloud.
• Work with data source owner that understand the value delivery potential their data, and collaborate to harvest, land and prepare that data at scale.
• Collaborate with product owner to understand business needs and data demand, which can be reflected to build data warehousing data model.
• Work with fellow engineers and analysts to improve Data Quality and Data Management consistently.
• Using agile development practices, and continually improving development methods with the goal of automating the build, integration, deployment and monitoring of ingestion, enrichment,
data model and data pipelines.
Required Skills:
• Minimum B.S. degree in a technical discipline.
• Experience with Azure Cloud and Big Data Analytic solutions
• Expert, hands on experience in Azure - Data Factory, Data Lake store/Blob storage, SQL DB
• Experience in creating Big data Pipelines with Azure components