Job Information
Insight Global Sr Data Engineer in Chicago, Illinois
Job Description
The Data Engineer will report to a Manager of Data Engineering. The Data and Analytics team's primary mission is to develop analytics by centralizing and integrating high-quality, trusted corporate data in a performant and scalable cloud platform. The Data Engineer will design and develop data pipelines and products in a modern-day, cloud environment. You will work with SMEs, architects, analysts, data scientists and others to build solutions that integrate data from many of our enterprise data sources.
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.
Skills and Requirements
• 5+ years of experience in batch and streaming ETL using Spark, Python, Scala, Snowflake, or Databricks for Data Engineering or Machine Learning workloads. Snowflake and Databricks are a must.
• 5+ years orchestrating and implementing pipelines with workflow tools like Databricks Workflows, Apache Airflow, or Luigi
• 5+ years of experience prepping structured and unstructured data for data science models.
• 5+ years of experience with containerization and orchestration technologies and experience with shell scripting in Bash, Unix, or Windows shell is preferable.
• Implemented CI/CD with automated testing in Jenkins, Github Actions, or Gitlab CI/CD
• Demonstrated experience implementing data management life cycle, using data quality functions like standardization, transformation, rationalization, linking, and matching.