Data Ingestion Specialist

Mandatory requirements

Must have minimum 7-10 years of hands-on experience on building data ingestion highly robust and scalable pipelines using AWS native services Cloud Formation AWS Glue Crawlers Catalog Step Functions Event Bridge Lambda S3 SNS Kinesis other data related services

Must be proficient in Python Java

Must be proficient with SQL and HQL

Must have handled structured and unstructured data ingestion from different file formats TXT CSV JSON XML Parquet etc

Must have handled data streaming from Kafka topics using Apache Flink etc

Must have prior experience with open table formats like Apache Iceberg Hudi etc

Must have exposure to Cloudera data lake environment should have exposure to Snowflake environment

Must be proficient with Github versioning activities

Prior experience on CICD using Jenkins would be an added advantage

Should be able to envision the ingestion end to end while keeping up with all the data privacy lineage the cost optimization requirements

Must possess clear written verbal communication skills along with interpersonal skill to collaborate with the team is very important

Should be willing to coordinate on the delivery with the team located in India

Must have a positive attitude should be open for discussions stay hungry to learn try out something new

Must have prior hands-on working experience in Cloudera Hadoop implementation Should be able to reverse engineer the current processes running on it

Prior Apache Flink experience would be ideal