Airbnb Engineering » Tech Talks http://nerds.airbnb.com Nerds Thu, 26 Feb 2015 15:33:56 +0000 en-US hourly 1 http://wordpress.org/?v=3.8.5 Surviving at Scale: Lessons from Airbnb and LinkedIn http://nerds.airbnb.com/surviving-scale-lessons-airbnb-linkedin/ http://nerds.airbnb.com/surviving-scale-lessons-airbnb-linkedin/#comments Thu, 29 Jan 2015 17:45:06 +0000 http://nerds.airbnb.com/?p=176594732 We are super excited to host our first tech talk of 2015! With the new year comes a new schedule of tech talks happening once a month. Each talk will be carefully curated with multiple speakers and a cohesive theme. The theme this month is scaling and infrastructure with Airbnb engineers, Ben Hughes and Jon […]

The post Surviving at Scale: Lessons from Airbnb and LinkedIn appeared first on Airbnb Engineering.

]]>
We are super excited to host our first tech talk of 2015! With the new year comes a new schedule of tech talks happening once a month. Each talk will be carefully curated with multiple speakers and a cohesive theme. The theme this month is scaling and infrastructure with Airbnb engineers, Ben Hughes and Jon Tai and the co-founder and head of engineering at Confluent, Neha Narkhede.

Scaling Things That Don’t Scale: Scalability and Reliability at Airbnb

Paul Graham says to do things that don’t scale, and this advice has served us very well at Airbnb. By 2012, guests had booked more than 10 million nights in 192 countries. Years of exponential growth were great for our business, but hard on our infrastructure.

In early 2013, scaling issues began to affect our user experience. As peak summer travel season ramped up, external services took longer to respond, background tasks overwhelmed our database, and memory usage grew unchecked. We started having downtime several days a week, and responding to incidents was consuming more and more engineers’ time.

A small group of us started on a journey to improve reliability and scalability. We added instrumentation, got scientific about timeouts, fixed bad queries, and threw a lot of hardware at the problem. By the end of the year, not only did we have more users and bookings than ever before, but we had improved our response time and uptime as well. We didn’t accomplish this by doing a major rewrite, removing functionality, or disrupting other teams; we accomplished this by dedicating a small team to work on application reliability, focusing on fundamentals, and working to better understand our systems.

In this talk we will share what we did to improve reliability and scalability, how we shifted from tactics to strategy, and the lessons we learned along the way.

Speaker Bio

Ben Hughes is a Software Engineer at Airbnb focusing on reliability and scalability. He was previously a cofounder of NabeWise, a neighborhood information startup that was acquired by Airbnb in 2012. Jon Tai is a Software Engineer on the Production Infrastructure team at Airbnb. Beyond scaling software systems, he works to build tooling and evangelize best practices among the engineering team. Prior to Airbnb, Jon was an engineer at IGN Entertainment, where he helped re-architect and scale out IGN’s media platform.

Real-time stream processing at scale using Apache Kafka and Samza

We are enjoying something of a renaissance in data infrastructure. The old workhorses like MySQL and Oracle still exist but they are complemented by new specialized distributed data systems like Cassandra, Redis, Druid, and Hadoop. At the same time what we consider data has changed too–user activity, stock tickers, gaming events, monitoring, logging and other event data are becoming first class citizens for data driven companies. Taking full advantage of all these systems and the relevant data creates a massive data integration problem. This problem is important to solve as these specialized systems are not very useful in the absence of a complete and reliable data flow.

One of the most powerful ways of solving this data integration problem is by restructuring your digital business logic around a centralized firehose of immutable events. Once a company’s data is captured and available as real-time streams, processing this data becomes the next challenge. Stream processing is an essential part of real-time data systems today, and is used for building news feeds, real-time analytics, metrics, alerts and monitoring.

At LinkedIn, we successfully moved virtually all data flow (500 billions events a day) to real-time structured logs using Apache Kafka. This architecture has influenced a lot of companies worldwide in doing the same and Apache Kafka is used in production at hundreds of companies [https://cwiki.apache.org/confluence/display/KAFKA/Powered+By] for similar use cases.

In this talk, I will share our experience of successfully building LinkedIn’s data pipeline infrastructure around real-time streams on top of Apache Kafka and Apache Samza. These lessons are hugely relevant to anyone building a data driven company.

Speaker Bio

Neha Narkhede is co-founder and head of engineering at Confluent. Previously, she was responsible for LinkedIn’s petabyte scale streaming infrastructure supporting hundreds of billions of events per day. She is also one of the initial authors of Apache Kafka and serves as a PMC member and committer for the project. In the past she has worked on search within the database at Oracle and holds a Masters in Computer Science from Georgia Tech.

The post Surviving at Scale: Lessons from Airbnb and LinkedIn appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/surviving-scale-lessons-airbnb-linkedin/feed/ 1
Payments at Airbnb http://nerds.airbnb.com/payments-airbnb/ http://nerds.airbnb.com/payments-airbnb/#comments Wed, 17 Dec 2014 20:29:18 +0000 http://nerds.airbnb.com/?p=176594675 Talk Abstract Payments at Airbnb is challenging because it operates at the core of a global marketplace to create trust. No single solution exists to meet our needs. Going global involves supporting multiple currencies and payment methods but it also involves nuanced complexities that are unique to local markets. It’s very common for our problems […]

The post Payments at Airbnb appeared first on Airbnb Engineering.

]]>
Talk Abstract

Payments at Airbnb is challenging because it operates at the core of a global marketplace to create trust. No single solution exists to meet our needs. Going global involves supporting multiple currencies and payment methods but it also involves nuanced complexities that are unique to local markets. It’s very common for our problems to cross different domains such as product, compliance, growth, finance, and engineering infrastructure. Airbnb’s two-sided marketplace includes guests and hosts who introduce two different payment problems: collecting and distributing money. Our payments platform also supports other activities in the marketplace such as photographers, translators, trip services, and more in the future.

We’ll talk about building end-to-end systems (upstream and downstream) that process billions of dollars in a reliable and scalable way. Want to know how to change tires while driving in a production environment that has significant money flow? Want to know how to confidently sprint instead of walk with code on a frequent basis? As a result of Airbnb’s rapid growth, its Payments platform has evolved quite a bit and we’d like to talk about it while also providing concrete tips based on our experience.

Speaker Bio

Ian Logan is an Engineering Manager at Airbnb. He joined Airbnb over 3 and a half years ago as the first payments engineer, bringing his experience working in investment banking at BlackRock, Barclays Global Investors, and Morgan Stanley to a new problem of payments in the sharing economy. During his time at Airbnb, Ian worked on all aspects of core payments technology and built its original foundation. Currently, Ian leads the engineering teams responsible for Payments, Trust & Safety, and Internal Products at Airbnb.

The post Payments at Airbnb appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/payments-airbnb/feed/ 0
Decentralized Payments with Bitcoin http://nerds.airbnb.com/decentralized-payments-bitcoin/ http://nerds.airbnb.com/decentralized-payments-bitcoin/#comments Wed, 10 Dec 2014 18:54:33 +0000 http://nerds.airbnb.com/?p=176594641 Talk Abstract So you’ve heard of Bitcoin, a global digital currency with an avid following, and a wildly fluctuating price. But the real power of Bitcoin stems from its underlying decentralized ledger, the blockchain. In this talk, Adrian will take a deep dive into the blockchain, and look at how transactions are generated and broadcast […]

The post Decentralized Payments with Bitcoin appeared first on Airbnb Engineering.

]]>
Talk Abstract

So you’ve heard of Bitcoin, a global digital currency with an avid following, and a wildly fluctuating price. But the real power of Bitcoin stems from its underlying decentralized ledger, the blockchain.

In this talk, Adrian will take a deep dive into the blockchain, and look at how transactions are generated and broadcast in a completely trustless peer-to-peer network. He will discuss what role miners play in this network, and whether the blockchain network could exist independent of the Bitcoin currency. Finally, he will look at some of the potential future applications of Bitcoin and the blockchain, including sidechains and decentralized asset tracking.

Speaker Bio

Adrian Macneil is Director of Engineering at Coinbase, the largest bitcoin company in the world. Originally from New Zealand, Adrian has previously been involved with several e-commerce and payments startups, and has seen first hand the difficulties faced by companies looking to innovate using the current financial networks. Adrian moved to the US in early 2014, and has since helped Coinbase grow from 5 to 20 engineers, and launch in 19 countries and 22 languages. In his spare time, Adrian can be found traveling, snowboarding, and spreading the word of bitcoin.

The post Decentralized Payments with Bitcoin appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/decentralized-payments-bitcoin/feed/ 1
Randomness and Fraud http://nerds.airbnb.com/randomness-fraud-2/ http://nerds.airbnb.com/randomness-fraud-2/#comments Wed, 03 Dec 2014 23:25:49 +0000 http://nerds.airbnb.com/?p=176594604 Talk Abstract Over the course of three years, we’ve built Stripe from scratch and scaled it to process billions of dollars a year in transaction volume by making it easy for merchants to get set up and start accepting payments. While the vast majority of the transactions we process are legitimate, Stripe does need to […]

The post Randomness and Fraud appeared first on Airbnb Engineering.

]]>
Talk Abstract

Over the course of three years, we’ve built Stripe from scratch and scaled it to process billions of dollars a year in transaction volume by making it easy for merchants to get set up and start accepting payments. While the vast majority of the transactions we process are legitimate, Stripe does need to protect its merchants and itself from rogue individuals and groups seeking to “test” or “cash” stolen credit cards. In this talk, I’ll discuss the two types of fraud Stripe faces—merchant fraud and transaction fraud—and how our approach to both has evolved over time. I’ll then demonstrate how “randomness” helps us combat fraud in the three stages of the model-building process. Specifically, I’ll focus on how fraud often appears less random than legitimate behavior (feature generation), how our in-house, distributed random forest learner allows us to build models on huge data sets with more control over how the random splits are made (model training), and how the introduction of randomness in the production scoring environment allows us to reason about counterfactuals (“what would have happened if we hadn’t intervened?”) and evaluate candidate models without production experiments (model evaluation).

Speaker Bio

Michael Manapat is a Software Engineer on Stripe’s Machine Learning team. He was previously a Software Engineer at Google, a Postdoctoral Fellow in and Lecturer on Applied Mathematics at Harvard, and a graduate student in mathematics at MIT.

The post Randomness and Fraud appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/randomness-fraud-2/feed/ 1
SDK Mindset http://nerds.airbnb.com/sdk-mindset/ http://nerds.airbnb.com/sdk-mindset/#comments Wed, 03 Dec 2014 23:25:30 +0000 http://nerds.airbnb.com/?p=176594599 Talk Abstract Writing software is difficult. Bugs happen. Users find weakness and hit them with their mallets repeatedly. Complexity hides and exponentiates with every new feature. When developing SDKs—that is, code we write to run in others’ systems—these problems are critical and acute. We will talk about a number of scenarios we’ve encountered to highlight […]

The post SDK Mindset appeared first on Airbnb Engineering.

]]>
Talk Abstract

Writing software is difficult. Bugs happen. Users find weakness and hit them with their mallets repeatedly. Complexity hides and exponentiates with every new feature. When developing SDKs—that is, code we write to run in others’ systems—these problems are critical and acute. We will talk about a number of scenarios we’ve encountered to highlight the difficulties of building and shipping SDKs: backwards compatibility, flexibility, abstraction, understandability, shippability, adoption, beta partners, communication, etc.

Speaker Bios

Ben Mills is a software developer who works on the Braintree SDKs primarialy on the back-end.

Mickey Reiss is an iOS developer who works on Braintree’s iOS SDK. Braintree is a payments company who’s goal is to provide developers with a foundation to build applications on.

The post SDK Mindset appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/sdk-mindset/feed/ 0
The Future of App Deployment http://nerds.airbnb.com/future-app-deployment/ http://nerds.airbnb.com/future-app-deployment/#comments Wed, 03 Dec 2014 23:25:07 +0000 http://nerds.airbnb.com/?p=176594596 Talk Abstract Writing an app is one thing, deploying it to a production ready environment and keeping it online in the face of countless potential scenarios of adversity is an entirely different beast. Not only does it currently still involve a fair bit of expertise when it comes to unix-fu, it also involves keeping an […]

The post The Future of App Deployment appeared first on Airbnb Engineering.

]]>
Talk Abstract

Writing an app is one thing, deploying it to a production ready environment and keeping it online in the face of countless potential scenarios of adversity is an entirely different beast. Not only does it currently still involve a fair bit of expertise when it comes to unix-fu, it also involves keeping an eye out on the latest software and configure them properly to combat things like security breaches. This is all generally considered tedious and cumbersome work, and often outside the domain of knowledge of developers. Wouldn’t it be great if we didn’t need to go through as many hoops as we need to do today and make it more developer friendly?

This talk will go over the most important steps currently involved in setting up a production environment for your a web app and will propose alternative approaches as well in the form of new software solutions developed by Phusion. This talk will focus on app deployment, monitoring and server provisioning, but will also touch upon topics such as UI design and UX as the latter plays an important part in making things more accessible. More specifically, we’ll discuss Docker, Polymer, Node, Rails, Phusion Passenger, Union Station and much more.

Speaker Bios

Hongli Lai is the co-founder and CTO of Phusion. His work involves all layers of an application stack. Be it from Phusion Passenger for deployment, to tinkering on Docker baseimage, his work has helped deploy hundreds of thousands of sites.

Tinco Andringa is a full stack software engineer at Phusion. During his time at Phusion he has built Ruby on Rails applications for clients and he is now leading development work on Phusion’s new monitoring solution Union Station. When he is not building Ruby web apps he works on open source software and tinkers on video games.

Goffert van Gool is a Full-Stack Engineer at Phusion, where he works on modern web applications, and high-availability services. He is often experimenting with new technology and how it can be used to improve application architecture.

The post The Future of App Deployment appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/future-app-deployment/feed/ 0
Feature Engineering for Real-Time Fraud Detection http://nerds.airbnb.com/feature-engineering-real-time-fraud-detection/ http://nerds.airbnb.com/feature-engineering-real-time-fraud-detection/#comments Wed, 03 Dec 2014 23:24:44 +0000 http://nerds.airbnb.com/?p=176594588 Talk Abstract This talk will explore some of the joys and pitfalls of applying machine learning to Internet activity. It’ll focus on the process that Sift Science uses to develop features for its realtime fraud detection service, covering some of the specific evaluation and modeling problems that are unique to this kind of data. Speaker […]

The post Feature Engineering for Real-Time Fraud Detection appeared first on Airbnb Engineering.

]]>
Talk Abstract

This talk will explore some of the joys and pitfalls of applying machine learning to Internet activity. It’ll focus on the process that Sift Science uses to develop features for its realtime fraud detection service, covering some of the specific evaluation and modeling problems that are unique to this kind of data.

Speaker Bio

Doug Beeferman is a software engineer at Sift Science, formerly at Google and Lycos. His focus throughout his career has been applications that analyze logs of user activity such as search and web logs. He is also interested in text and speech processing.

View the slides here.

The post Feature Engineering for Real-Time Fraud Detection appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/feature-engineering-real-time-fraud-detection/feed/ 0
How to Drive Growth: Lessons from Facebook, Dropbox, and More http://nerds.airbnb.com/drive-growth-lessons-facebook-dropbox/ http://nerds.airbnb.com/drive-growth-lessons-facebook-dropbox/#comments Tue, 01 Jul 2014 16:00:03 +0000 http://nerds.airbnb.com/?p=176594450 You’re inundated with tactics about how to grow your company. This talk will help you refine a framework for how to think about growth, and arm you with some tools to help hack through the bullshit. We’ll fly through a 50,000 foot view of the product development process used at Dropbox and Facebook to drive […]

The post How to Drive Growth: Lessons from Facebook, Dropbox, and More appeared first on Airbnb Engineering.

]]>
You’re inundated with tactics about how to grow your company. This talk will help you refine a framework for how to think about growth, and arm you with some tools to help hack through the bullshit.

We’ll fly through a 50,000 foot view of the product development process used at Dropbox and Facebook to drive growth.

We’ll dive into the nitty gritty: dozens of real examples of tests that worked and didn’t work.

Ivan Kirigin is an engineer and founder at YesGraph, a young startup focused on referral recruiting. Previous he lead growth at Dropbox, helping drive 12X growth in 2 years, and helped build Facebook Credits, and co-founded Tipjoy YC W08.

The post How to Drive Growth: Lessons from Facebook, Dropbox, and More appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/drive-growth-lessons-facebook-dropbox/feed/ 2
How to Search http://nerds.airbnb.com/search/ http://nerds.airbnb.com/search/#comments Sat, 28 Jun 2014 18:45:54 +0000 http://nerds.airbnb.com/?p=176594446 I have worked on search projects for around 7 years now in 3 companies and have had the privilege of working with smart search engineers from many other companies.Solving search problems together, we’ve found there are many ways to skin a cat. And we’ve formed opinions on the best ways among them. In this talk, […]

The post How to Search appeared first on Airbnb Engineering.

]]>
I have worked on search projects for around 7 years now in 3 companies and have had the privilege of working with smart search engineers from many other companies.Solving search problems together, we’ve found there are many ways to skin a cat. And we’ve formed opinions on the best ways among them.

In this talk, I’ll share some of these experiences and cover different approaches to deliver the best search results for our user. I’ll cover infrastructure (indexing strategies, sharding, efficient retrieval, etc.) as well as relevance (query rewriting, scoring, etc.). Finally, I’ll spend some time of frontend/product issues and ways to measure search quality.

Sriram Sankar is a Principal Staff Engineer at LinkedIn, where he is leading the development of their next-generation search infrastructure and search quality frameworks. Before that, he led Facebook’s search quality efforts for Graph Search, and was a key contributor to Unicorn, the index powering Graph Search. He previously worked at Google on search quality and ads infrastructure and has held senior technical roles at VMware, WebGain, and Sun. He is the author of JavaCC, one of the leading parser generators for Java. Sriram obtained his PhD from Stanford University, and a BS from the Indian Institute of Technology Kanpur.

The post How to Search appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/search/feed/ 1
Referrals at Airbnb: Driving Sustainable & Scalable Growth http://nerds.airbnb.com/referrals-airbnb-driving-sustainable-scalable-growth/ http://nerds.airbnb.com/referrals-airbnb-driving-sustainable-scalable-growth/#comments Sat, 28 Jun 2014 18:43:13 +0000 http://nerds.airbnb.com/?p=176594442 Word of mouth is the largest source of growth for Airbnb, in part because Airbnb experiences are so personal. People use Airbnb to unlock incredible experiences–anything from weekend getaways with friends, cultural exchanges, and once in a lifetime events like honeymoons. The Growth team builds products that helps users tell their stories. Our most successful […]

The post Referrals at Airbnb: Driving Sustainable & Scalable Growth appeared first on Airbnb Engineering.

]]>
Word of mouth is the largest source of growth for Airbnb, in part because Airbnb experiences are so personal. People use Airbnb to unlock incredible experiences–anything from weekend getaways with friends, cultural exchanges, and once in a lifetime events like honeymoons.

The Growth team builds products that helps users tell their stories. Our most successful program is Referrals which accounts for up to 30% of growth in certain markets. We launched Referrals in January on all three of our platforms: our website, our Android app, and our iOS app allowing users both to send and redeem referrals.

In this talk we will dive into the details of why we decided to build out Referrals, how we built it and the effect it have had on our growth trajectory. We’ll walk you through in-depth learnings and tips that we’ve acquired along the way. Finally, we’ll share what we’re working on now.

Jimmy Tang is an engineer on the Airbnb Growth team and built referrals for iOS. He works on all things growth from product, tools, and tracking. Previously, he cofounded Yozio, a platform for organic mobile growth.

Gustaf Alstromer is a product manager on the Airbnb Growth team and helped launched Referrals. Gustaf was previously Head of Growth at Voxer before he joined Airbnb in the end of 2012.

The post Referrals at Airbnb: Driving Sustainable & Scalable Growth appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/referrals-airbnb-driving-sustainable-scalable-growth/feed/ 1