Job Information
Insight Global Lead Data Engineer in San Francisco, California
Job Description
Title: Lead Data Engineer
Location: Hybrid in SF (Tuesdays onsite)
Openings: 1
Work Schedule: (available until 12 am PT time to overlap with onshore team)
Follow-Up Meeting: After each interview is scheduled.
Contract Type: 12 months contract extensions
We are a company committed to creating diverse and inclusive environments where people can bring their full, authentic selves to work every day. We are an equal opportunity/affirmative action employer that believes everyone matters. Qualified candidates will receive consideration for employment regardless of their race, color, ethnicity, religion, sex (including pregnancy), sexual orientation, gender identity and expression, marital status, national origin, ancestry, genetic factors, age, disability, protected veteran status, military or uniformed service member status, or any other status or characteristic protected by applicable laws, regulations, and ordinances. If you need assistance and/or a reasonable accommodation due to a disability during the application or recruiting process, please send a request to HR@insightglobal.com.To learn more about how we collect, keep, and process your private information, please review Insight Global's Workforce Privacy Policy: https://insightglobal.com/workforce-privacy-policy/.
Skills and Requirements
Must Haves:
• At least 5 years of strong hands‑on expertise with Databricks and Datastage (or comparable ETL tools)
• Experience with migrations
○ Exs: Migrating SQL to cloud, SQL to Azure Databricks, (on prem to the cloud)
• 8–10 years of software development and data engineering experience with SQL, ADF, SSAS cubes, Cognos, Tableau, ThoughtSpot and other BI tools
• Advanced SQL skills for processing raw data, Kafka ingestions, ADF pipelines, data validation and QA with experience across SQL and NoSQL databases
• Programming experience with Python and/or Scala
• Experience ingesting data via APIs and streaming sources
• Experience building data pipelines, data warehouses, and analytics platforms
• Proficiency with Spark, Hive, Kafka, and cloud data services
• Experience with Azure, AWS or equivalent cloud platforms
Day-To-Day:
• Lead and support an offshore engineering team as the primary onshore technical owner
• Define and drive engineering strategy in partnership with product and technology leaders
• Own the migration from SQL/Datastage to Databricks, with full understanding of the data landscape
• Review existing Datastage pipelines and re‑engineer them in Databricks
• Provide hands‑on mentorship and technical guidance
• Develop and deploy solutions using cloud‑based platforms daily