Data Engineer - Databricks (Mid Level) - US Citizens Only

---- Project requirements mandate role open only for US Citizens. IRS MBI Clearance a plus/ Active Secret or Top Secret a Plus. All candidates will have to go through Clearance process before being able to start on the project.--(No exceptions to this requirement)

Job Description

• Infobahn Solutions is hiring Databricks Data Engineering professionals in the Washington DC Metro Area for a US Government Federal Project with the Department of Treasury .
• The Data Engineers will be part of a Data Migration & Conversion Team on a large DataLake being implemented on AWS Gov Cloud .
• Data will be migrated from on premise Main Frame /Legacy database systems using Informatica PowerCenter to the AWS Landing Zone on S3.
• Further conversion will be done using Databricks (PySpark) in AWS.
• The Data Engineer should have prior Data Migration experience and understand all the intricacies required of developing data integration routines for moving data from multiple source systems to a new target system with a different data model.
• The Data Engineer should have experience in converting Oracle PL/SQL and/or Greenplum code to Databricks.
• Must have experience - Experience with Data Migrations and Conversion using Databricks .
• Experience of using Databricks on AWS and managing a Databricks production system is critical and a must have for the project.

What you’ll be doing:

• Databricks Environment Setup: Configure and maintain Databricks clusters, ensuring optimal performance and scalability for big data processing and analytics.
• ETL (Extract, Transform, Load): Design and implement ETL processes using Databricks notebooks or jobs to process and transform raw data into a usable format for analysis.
• Data Lake Integration: Work with data lakes and data storage systems to efficiently manage and access large datasets within the Databricks environment.
• Data Processing and Analysis: Develop and optimize Spark jobs for data processing, analysis, and machine learning tasks using Databricks notebooks.
• Collaboration: Collaborate with data scientists, data engineers, and other stakeholders to understand business requirements and implement solutions.
• Performance Tuning: Identify and address performance bottlenecks in Databricks jobs and clusters to optimize data processing speed and resource utilization.
• Security and Compliance: Implement and enforce security measures to protect sensitive data within the Databricks environment, ensuring compliance with relevant regulations.
• Documentation: Maintain documentation for Databricks workflows, configurations, and best practices to facilitate knowledge sharing and team collaboration.

Skills:

• Apache Spark: Strong expertise in Apache Spark, which is the underlying distributed computing engine in Databricks.
• Databricks Platform: In-depth knowledge of the Databricks platform, including its features, architecture, and administration.
• Programming Languages: Proficiency in languages such as Python or Scala for developing Spark applications within Databricks.
• SQL: Strong SQL skills for data manipulation, querying, and analysis within Databricks notebooks.
• ETL Tools: Experience with ETL tools and frameworks for efficient data processing and transformation.
• Data Lake and Storage: Familiarity with data lakes and storage systems, such as Delta Lake, AWS S3, or Azure Data Lake Storage.
• Collaboration and Communication: Effective communication and collaboration skills to work with cross-functional teams and stakeholders.
• Problem Solving: Strong problem-solving skills to troubleshoot issues and optimize Databricks workflows.
• Version Control: Experience with version control systems (e.g., Git) for managing and tracking changes to Databricks notebooks and code.

Role Requirements:

• Bachelor/Master’s degree in computer science, Engineering, or related field
• 7-8 plus years of development experience on ETL tools (4+ years of Databricks is a must have)
• 5+ years of experience as a Databricks Engineer or similar role.
• Strong expertise in Apache Spark and hands-on experience with Databricks.
• More than 7 years of experience performing data reconciliation, data validation, ETL testing, deploying ETL packages and automating ETL jobs, developing reconciliation reports.
• Working knowledge of message-oriented middleware/streaming data technologies such as Kafka, Confluent
• Proficiency in programming languages such as Python or Scala for developing Spark applications.
• Solid understanding of ETL processes and data modeling concepts.
• Experience with data lakes and storage systems, such as Delta Lake, AWS S3, or Azure Data Lake Storage.
• Strong SQL skills for data manipulation and analysis.
• Good experience in shell scripting, AutoSys
• Strong Data Modeling Skills
• Strong analytical skills applied to business software solutions maintenance and/or development
• Must be able to work with a team to write code, review code, and work on system operations.
• Past project experience with Data Conversion and Data Migration
• Communicate analysis, results and ideas to key decision makers including business and technical stakeholders.
• Experience in developing and deploying data ingestion, processing, and distribution systems with AWS technologies
• Experience with using AWS datastores, including RDS Postgres, S3, or DynamoDB
• Dev-ops experience using GIT, developing, deploying code to production
• Proficient in using AWS Cloud Services for Data Engineering tasks
• Proficient in programming in Python/shell or other scripting languages for the purpose of data movement
• Eligible for a US Government issued IRS MBI (candidates with active IRS MBIs will be preferred)
• Databricks industry certifications - Associate / Professional Level

Preferred Qualifications

• Cloud Data Migration and Conversion projects
• Experience on AWS

Job Types: Full-time, Contract Pay: $90,000.00 - $130,000.00 per year Benefits:

• Dental insurance
• Flexible schedule
• Health insurance
• Life insurance
• Paid time off
• Vision insurance

Education:

• Bachelor's (Preferred)

License/Certification:

• Databricks Certified Data Engineer Professional (Required)

Security clearance:

• Secret (Preferred)

Work Location: Remote Apply tot his job Apply To this Job