Data Engineer (PySpark)
- Master’s degree in Computer Science or related fields with three years of relevant experience, or Bachelor’s degree in Computer Science or related fields and five years of relevant experience in lieu Master's.
- Extensive experience in software development, modern design patterns and web technologies under e-commerce enabled environments using multi-tier architectures.
- Experience as a Data Engineer, Software Engineer, Business Intelligence Engineer, Data Scientist, or similar specialty with a track record of manipulating, processing, productionizing and extracting value from large and complex datasets.
- Strong knowledge of DBMS concepts with hands-on architecting and data modeling (Relational, Dimensional, NoSql) experience.
- Strong knowledge and Experience working in projects involving building Data Warehouses, Data Lakes, Big data ecosystems (Hadoop, HDFS, Hive, and Hbase ) on a Cloud platform (AWS).
- Strong programming experience using Python, PySpark, Bash, SQL in Linux/UNIX environment to process and analyze large data sets on the Cloud.
- Strong Knowledge and Hands-on experience in building ETL systems and data-pipelines on Cloud (AWS Glue)
- Experience with Cloud technologies/Services - (AWS - EC2, S3, EMR, CodeCommit, Glue, Athena etc.) and Modern Data Processing/Analysis tools (zeppelin Notebooks etc)
- Experience with data visualization software (Tableau)
- Knowledge of object-oriented Programming languages and frameworks - Java/JEE, Spring, in development for modern data engineering systems.
- Experience with technologies such as Apache Spark, Kafka, Flume, Sqoop is a big plus.
- Experience working with modern tools in the Agile software development life cycle - Version Control Systems (Ex. git, github, Stash/BitBucket), Knowledge Management (Ex. Confluence, Google Docs), Development Workflow (Ex. Jira), Continuous Integration (Ex. Bamboo, Jenkins), Real Time Collaboration (Ex. Hipchat, Slack)
- Design, develop, implement, and test software modules and other technology components for consumerreports.org and related customer-facing web properties using agile software development practices.
- Develop and implement payment, billing and/or dynamic content delivery systems.
- Analyze business requirements, provide development estimates and feedback, and determine implementation/execution approaches.
- Work with other software engineers, database developers, infrastructure, operations and QA teams to design and implement quality software solutions.
- Support, troubleshoot, monitor and optimize existing software systems.
- Help streamline software development by contributing to implementation of a continual integration process, automating manual processes, and eliminating defects.
- Performs other duties as assigned but not at a higher level.