Head of Development - Crawlera

Scrapinghub published a month ago


Senior, Lead, Manager, Full-time

No office location

Location Availability

m 2015 05 12 traveling tips for remote workers rel nofollow wherever you want a . have the opportunity to go to conferences and meet
pportunity to go to conferences and meet with the team from across the globe . get the chance to work with cutting edge open source t

Head of Development - Crawlera

Scrapinghub | No office location

About this job

Job type: Full-time
Experience level: Senior, Lead, Manager
Role: Backend Developer
Industry: Big Data, Data Science
Company size: 51–200 people
Company type: Private


Job description

About the Job:

Crawlera is a smart downloader designed specifically for web crawling and scraping, removing the headaches of proxy management. It is part of the Scrapinghub platform, the world’s most comprehensive web crawling stack which powers crawls of over 8 billion pages a month. We’re looking for a highly collaborative Head of Developement for Crawlera.

Leading Crawlera’s technical development through the exponential growth that we’re experiencing is just the start. You’ll help set overarching business strategy and goals, identify opportunities, and over time, together with Crawlera’s Product Manager, will have full responsibility for Crawlera as a business.

You should be comfortable not having all the answers but being the person that gets the right people to the table for discussions, and a strong bias towards taking action. Ideally you’ve done this job before, either in an early stage company or a new team/initiative, and have a concrete history of developing world-class team(s).

Job Responsibilities:

    • Identify and resolve performance and scalability issues with distributed crawling at scale.
    • Develop, maintain and support a high load distributed system.
    • Comfortable taking ownership in business critical situations; you enjoy being the “go-to” person.
    • Lead the team. You’ll have a fair amount of autonomy over this, but it will likely include:
    • Developing talent and maintaining a strong team culture.
    • Expanding and maintaining technical and quality standards, in conjunction with broader Engineering expectations.
    • Translating business intent and requirements into architectural design/decisions to achieve that intent.
    • Collaborating with other platform teams to provide Crawlera customers the best possible integration to the growing Scrapinghub platform.

Required Skills:

    • 5+ years of production experience with at least two of: Erlang (or other functional programming language), Python, and Go.
    • Strong knowledge of Linux/UNIX, HTTP and Networking.
    • Experience managing and growing a technical team: developing talent, creating and enforcing standards, resource allocation.
    • A strong history of achieving both technical and business goals.
    • Comfortable taking ownership in business critical situations; you enjoy being the “go-to” person.
    • Good communication in written English.

Desired Skills:

    • Familiarity with techniques and tools for crawling, extracting, and processing data.
    • Knowledge of ELK, Graylog, Docker and Mesos.
    • Good spoken English.

Life at Scrapinghub

About Scrapinghub

Scrapinghub is a fast growing and diverse technology business turning web content into useful data with a cloud-based web crawling platform, off-the-shelf datasets, and turn-key web scraping services.

We’re a globally distributed team of 180 Shubbers working from over 30 countries who are passionate about scraping, web crawling, and data science.

As a new Shubber, you will:

Become part of a self-motivated, progressive, multi-cultural team.

Autonomy to make the role your own, supported by great people

Join a team with huge opportunity to make a difference

Have the freedom and flexibility to work from wherever you want.

Have the opportunity to go to conferences and meet with the team from across the globe.

Get the chance to work with cutting-edge open source technologies and tools.


  • Flexible working hours
  • Remote working
  • Paid time off
  • Paid open source work
  • Global team meet ups
  • Learning & Development Opportunities

Joel Test

Source control
One-step build
Daily builds
Bug database
Bugs fixed before writing new code
Up-to-date schedule
Quiet working conditions
Best tools that money can buy
Code screening
Hallway usability testing
Learn more about Scrapinghub
Head of Development - Crawlera at Scrapinghub