Job Information
Northwell Health Part-Time Senior Data Platform Engineer - Azure Databricks & API Integration in Lake Success, New York
Req Number 181726
FlexStaff is seeking a Senior Data Platform Engineer - Azure Databricks & API Integration (PHI)
This is a temporary role, for a one -year assignment
Part-Time - approximately 20 hours/week
Only candidates who reside in New York State can be considered.
The Senior Data Platform Engineer will lead the operation, stability, and evolution of our client's Azure Databricks environment and the API integrations that support enterprise data exchange involving Protected Health Information (PHI). This role is a key senior individual contributor within the Data Engineering organization and works closely with architecture, security, and application teams to ensure a secure, scalable, and compliant data platform.
The ideal candidate brings deep hands-on experience with Azure Databricks, Azure-native services, and API development, along with a strong understanding of healthcare data security and compliance.
Key Responsibilities
Azure Databricks & Data Platform Ownership
*Own and maintain the Azure Databricks workspace(s), including clusters, jobs, pools, and libraries
*Design, build, and optimize Spark-based data pipelines using Databricks, Delta Lake, and PySpark/Scala
*Implement best practices for cluster sizing, autoscaling, job orchestration, and cost optimization
*Manage Databricks access controls using Azure AD, RBAC, and workspace permissions
*Support CI/CD pipelines for Databricks assets using Azure DevOps and Git integration
API Integration & Data Access
*Design, develop, and support secure RESTful APIs that enable data ingestion and consumption
*Integrate Azure Databricks with upstream and downstream systems using APIs, messaging, or batch interfaces
*Ensure API reliability through proper versioning, monitoring, logging, and alerting
*Collaborate with application and integration teams to meet data access and performance requirements
Security, Compliance & PHI
*Ensure all Databricks workloads and APIs meet HIPAA, Azure security, and enterprise compliance standards
*Implement encryption at rest and in transit, key management (Azure Key Vault), and secrets handling
*Enforce data protection controls including data masking, row/column-level security, and auditing
*Partner with security, privacy, and compliance teams on audits, risk assessments, and remediation
Azure & Platform Operations
*Leverage Azure services such as ADLS Gen2, Azure Data Factory, Event Hubs, Azure Monitor, and Log Analytics
*Monitor platform health, performance, and availability; troubleshoot and resolve production issues
*Create and maintain technical documentation, standards, and operational runbooks
*Provide technical guidance and mentorship within the Data Engineering team
Required Skills & Qualifications
Technical Skills
*Strong hands-on experience with Azure Databricks (administration and development)
*Advanced proficiency in Python (PySpark) and/or Scala
*Experience developing and supporting REST APIs (e.g., Python/FastAPI, .NET, or Java/Spring Boot)
*Deep experience with Azure data and integration services, including:
o Azure Data Lake Storage (ADLS Gen2)
o Azure Data Factory or Synapse pipelines
o Azure Event Hubs or Service Bus
*Experience with Azure DevOps, Git, and CI/CD automation
*Infrastructure as Code experience using Terraform or ARM/Bicep
*Strong understanding of cloud security, identity, and access management
Compliance & Domain Knowledge
*Proven experience working with PHI and regulated healthcare data
*Strong knowledge of HIPAA and data privacy best practices
*Experience implementing audit logging, access controls, and governance in Azure
Experience & Background
*7+ years of experience in data engineering, platform engineering, or backend engineering
*3-5+ years of hands-on Azure Databricks experience in production environments
*3+ years of experience designing and supporting APIs at enterprise scale
*Experience in healthcare, life sciences, or other highly regulated industries strongly preferred
*Bachelor's degree in Computer Science, Engineering, or equivalent practical experience
Preferred Qualifications
*Experience with Delta Live Tables, Unity Catalog, or Databricks Workflows
*Familiarity with FHIR, HL7, or healthcare interoperability standards
*Experience with streaming architectures and near-real-time data processing
*Azure certifications (e.g., Azure Data Engineer Associate, Azure Solutions Architect)
*Additional Salary Detail
The salary range and/or hourly rate listed is a good faith determination of potential base compensation that may be offered to a successful applicant for this position at the time of this job advertisement and may be modified in the future. When determining a team member's base salary and/or rate, several factors may be considered as applicable (e.g., location, specialty, service line, years of relevant experience, education, credentials, negotiated contracts, budget and internal equity).
The salary range for this position is $75.00-$95.00/hour
It is Northwell Health’s policy to provide equal employment opportunity and treat all applicants and employees equally regardless of their age, race, creed/religion, color, national origin, immigration status or citizenship status, sexual orientation, military or veteran status, sex/gender, gender identity, gender expression, disability, pregnancy, genetic information or genetic predisposition or carrier status, marital or familial status, partnership status, victim of domestic violence, sexual or other reproductive health decisions, or other characteristics protected by applicable law.