Sr. Data Engineer - Spark/Hadoop
Position: Senior Data Engineer
As a Senior Big Data Engineer, you will be responsible for building out all aspects relating to our client's ecosystem and moving products from R&D into production scale. A successful Senior Data Engineer will possess a natural curiosity about data, strong work ethic, and clear technical leadership ability. Understanding of the ad ecosystem and the large scale data problems that come with it are going to be the root of your every day work.
You will be both hands-on and strategic—with both a broad ecosystem-level understanding of our market space and the ability to work closely with engineering, data science and product teams to deliver software in an iterative, continual-release environment. This is a high-visibility position involving close collaboration across many functional groups as well as interaction with executive stakeholders.
What you’ll be doing...
- Work with internal stakeholders to design and develop components in the next generation of products
- Write clear documentation to convey plans and technical architecture
- Ensure all new and modified code and pipelines are tested and of the highest quality at delivery time
- Build out new pipelines as part of an ever-evolving set of requirements for the core data asset as our business needs grow
- Ensure high reliability of all maintained product offerings by building reporting and monitoring mechanisms into our infrastructure.
What we’re looking for...
- BS or Masters in Computer Science and 3+ years of working experience
- Worked in a start-up (less than 150 employees) within the past 4 years
- Proven experience with Hadoop or Spark or other large-scale data processing platforms
- Proven experience processing and aggregating over billions to trillions of rows
- Strong software design and development experience, preferably in Scala or Java or other compiled languages
- Strong experience with Apache Parquet, Avro, or similar technologies
- Understanding of cookies, mobile web traffic, and user behavior is a strong plus
- Understanding of cloud infrastructure (preferably AWS) is a strong plus
- Proven ability to solve problems using state of the art technology
- Proven ability to innovate when necessary, but not reinvent the wheel
- Ability to define standards and best practices for teams
- Able to handle ambiguous delivery goals and turn them into concrete output
- Ability to suggest technical direction when necessary to solve problems
- Experimental yet pragmatic ability to create something useful
- Hands on and not afraid to wear multiple hats
- Passion for career growth