Big Data Platform Support Engineer

We are looking forBig Data Platform Support Engineer for our client in Reston, VA
Job Title: Big Data Platform Support Engineer
Job Location: Reston, VA
Job Type: Contract
Job Description:

• The Big Data / Cloudera Lead Engineer will lead the design, development, and support of large-scale data platforms, focusing on Cloudera, Hadoop, Kafka, and related technologies.
• This role involves ensuring high performance, scalability, and reliability of data pipelines and services while mentoring junior engineers and collaborating with cross-functional teams to optimize system performance and implement new solutions.

Responsibilities:

• Lead a team of software engineers in the design, development, and implementation of multiple applications.
• Perform detailed technical design, coding, code reviews, and testing of new and existing applications.
• Serve as a subject matter expert for specified systems and ensure compliance with DevSecOps practices and policies.
• Identify gaps in requirements and develop solutions to improve system performance and availability.
• Lead evaluations of risk, opportunities, and technology integration across projects.
• Support the Project Manager as the technical lead, estimating technical costs and defining system requirements.
• Participate in architectural planning, roadmap discussions, and transformation initiatives to new software architectures.
• Extend open-source libraries and systems to meet team requirements.
• Ensure adherence to code quality, deployment standards, and practices.
• Mentor junior developers and foster a culture of continuous improvement.

Experience:

• 8 years of software design, development, testing, and requirements management.
• Strong experience supporting Cloudera applications running in cloud environments.
• Hands-on experience with Cloudera Manager for monitoring and maintaining production clusters.
• Expertise in writing shell scripts for Hadoop daemon service health checks and automation tasks.
• Experience working with Hive, Impala, HBase, and SOLR, including query optimization and database management.
• Proficient in coordinating with application development, security, and platform teams for configuration and performance enhancements.

Skills Must Have:

• Cloudera administration and optimization (CDP, CDSW, Hive, Spark, NiFi).
• Cloud application support with MongoDB.
• Experience in cloud disaster recovery strategies.
• Strong Kafka administration and troubleshooting, including Streams API, integration with MQ, topic/offset management, and broker management.
• Apache NiFi administration, registry management, and integration with Kafka, HBase, and Solr.
• HBase administration, troubleshooting, and database management.
• Solr administration, managing shards, collections, logging levels, and query optimization.
• Flume integration with Kafka, NiFi, and IBM MQ.

Additional Skills:

• Perform critical data migrations from CDH to CDP.
• Design and implement big data pipelines and automated data flows using Python/R and NiFi.
• Perform incremental updates and upgrades to Cloudera environments.
• Manage job workflows using Oozie and Hue.
• Implement security policies across Hadoop clusters using Ranger.
• Configure and manage Cloudera Data Science Workbench (CDSW).
• Troubleshoot Kerberos, TLS/SSL, and model-related issues.
• Support Big Data/Hadoop databases through all development and production phases.
• Optimize database performance, troubleshoot replication, and manage connectivity/security issues.
• Create and implement automation scripts for maintenance and monitoring.
• Experience with Kafka ecosystem (Brokers, Connect, Zookeeper) in production environments.
• Support streaming technologies including Kafka, Spark, and Kudu.

Qualification And Education:

• Bachelor s degree in Information Technology or Computer Science.
• In lieu of a degree, an additional 4 years of relevant work experience is required.