Airbnb Engineering » Tech Talks http://nerds.airbnb.com Nerds Tue, 25 Aug 2015 16:41:56 +0000 en-US hourly 1 http://wordpress.org/?v=373 Developer Infrastructure with Airbnb and Bundler http://nerds.airbnb.com/developer-infrastructure-with-airbnb-and-bundler/ http://nerds.airbnb.com/developer-infrastructure-with-airbnb-and-bundler/#comments Thu, 20 Aug 2015 21:59:47 +0000 http://nerds.airbnb.com/?p=176595159 How does Bundler work, anyway? Bundler has turned out to be a super-useful tool for installing and managing dependencies, but many Rubyists don’t really have a handle on why it exists or how, exactly, it works. This talk aims to explain the huge mess that existed before Bundler, and then talk about the solutions to […]

The post Developer Infrastructure with Airbnb and Bundler appeared first on Airbnb Engineering.

]]>

How does Bundler work, anyway?

Bundler has turned out to be a super-useful tool for installing and managing dependencies, but many Rubyists don’t really have a handle on why it exists or how, exactly, it works. This talk aims to explain the huge mess that existed before Bundler, and then talk about the solutions to those problems that Bundler provides. The talk won’t spend a lot of time on code slides, but will include a fairly detailed explanation of Ruby’s require system, Rubygems, gem dependencies, dependency graph resolution, and how Bundler interacts with them. At the end, the talk will cover the Bundler “superpowers”, allowing rapid development and deployment in ways that were highly impractical before Bundler existed.

André Arko
André thinks Ruby is pretty neat. He leads the Bundler team, co-authored the third edition of The Ruby Way, and runs Ruby Together, the Ruby trade association. At his day job, he provides expert development, architecture, and teaching through Cloud City Development in San Francisco.

Automate the Boring Parts: Web Apps to Ship Web Apps

Delays, bureaucracy, angry release engineers: turning working software into production systems can be a drag. What if you could write software to automate away the pain, and have intelligent systems ship your code for you when it’s ready? Learn how Airbnb replaced manual release processes with code to make shipping smooth, fast, and simple.

Matt Baker
Matt is currently a software engineer at Airbnb on the Developer Infrastructure team, and was previously a game developer and computer graphics researcher. His work spans high-performance distributed systems and low-level rendering techniques. Having grown up across six states and two countries, he enjoys reading, ‘riting, ‘rithmetic, and road trips.

Igor Serebryany

The post Developer Infrastructure with Airbnb and Bundler appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/developer-infrastructure-with-airbnb-and-bundler/feed/ 0
Airbnb Price Tips Powered By Aerosolve Machine Learning http://nerds.airbnb.com/airbnb-machine-learning/ http://nerds.airbnb.com/airbnb-machine-learning/#comments Wed, 19 Aug 2015 20:11:58 +0000 http://nerds.airbnb.com/?p=176595153 Matching buyers with sellers depends a lot on price. Price too high and buyers are turned off. Price too low and the seller loses out. To increase the odds of matching buyers and sellers, Airbnb’s team of data scientists and engineers have worked hard to develop pricing technology, said Bar Ifrach, Airbnb data scientist, during […]

The post Airbnb Price Tips Powered By Aerosolve Machine Learning appeared first on Airbnb Engineering.

]]>

Matching buyers with sellers depends a lot on price. Price too high and buyers are turned off. Price too low and the seller loses out.

To increase the odds of matching buyers and sellers, Airbnb’s team of data scientists and engineers have worked hard to develop pricing technology, said Bar Ifrach, Airbnb data scientist, during a recent OpenAir 2015 talk.

“We’re trying to empower our hosts with tools to price their listings and get bookings seamlessly and effectively, so we have more hosts and stays on Airbnb and more matches on the platform,” Ifrach said.

Airbnb displays a host’s calendar. Days shown in white are available to book. Gray indicates days already booked, in the past, or that are days the host doesn’t want to book.

At the bottom of days in white, the host sees a color bar, which indicates how likely the host is to get a booking for that day at a given price. Green indicates a high likelihood. Yellow means there’s a medium chance of booking at the current price; red suggests a low probability of booking.

The technology behind this is divided into two parts, Ifrach said: Modeling and Aerosolve.

With modeling, Airbnb is “trying to predict for every day of year, for every listing, what will be the likelihood of getting a booking for any possible price,” Ifrach explained. “Then we can find a price that works best.” Airbnb modeling is doing this “on a huge, global scale” by looking at millions of derived features and over 5 billion training points, he said.

Three main considerations go into the pricing model:

  • Demand, or the impact of seasonality or special events (such as Austin’s SXSW conference) on an area. Airbnb’s model translates demand features into pricing predictions.
  • A listing’s location, such as the market, neighborhood, or street block. Example: San Francisco has many distinct neighborhoods that appeal to different crowds, and “we need to account for that” in pricing, said Ifrach. This is accomplished through grids and k-d trees.
  • A listing’s type and quality. Airbnb’s pricing model is “unique,” Ifrach said, because it incorporates such factors as a property’s size and specific qualities.

As an example, he compared a houseboat situated across from the Eiffel Tower in Paris and a private room with a view of the tower. To determine pricing for each, Airbnb’s model takes into account the differences in the two listing types (houseboat vs. a room), amenities (such as air conditioning), and other qualities. To quantify a listing’s quality, Airbnb’s modeling uses Dirac and Cubic splines to capture the effect of guest reviews.

Airbnb Hector Yee

The second part of Airbnb’s dynamic pricing technology is Aerosolve, “machine learning for humans,” said Airbnb engineer Hector Yee, who took the stage after Ifrach.

Airbnb introduced Aerosolve at OpenAir 2015 as an open-source machine learning library available on GitHub. Aerosolve is designed to interpret complex data sets so that humans can easily understand them. While Airbnb’s modeling suggests pricing to hosts, Aerosolve’s goal is to add context to the pricing recommendation so the host understands it. “You have to understand why the model makes it decisions. Maybe the host’s pricing is too high because they have no reviews or because it’s low season,” Yee explained.

The blog post “Aerosolve: Machine learning for humans” offers details about how Aerosolve works and how developers can use it.

The post Airbnb Price Tips Powered By Aerosolve Machine Learning appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/airbnb-machine-learning/feed/ 0
Building Periscope for Android http://nerds.airbnb.com/building-periscope-for-android/ http://nerds.airbnb.com/building-periscope-for-android/#comments Thu, 13 Aug 2015 18:56:16 +0000 http://nerds.airbnb.com/?p=176595128 Yesterday, Periscope announced that it reached 10 Million Periscope accounts. At OpenAir, Sara Haider detailed how she and her colleagues, faced multiple challenges in developing an Android version of the live video streaming app. There are two video livestreaming protocols, Haider explained: Real-Time Messaging Protocol (RTMP), which is suitable for low latency and real-time interaction […]

The post Building Periscope for Android appeared first on Airbnb Engineering.

]]>

Yesterday, Periscope announced that it reached 10 Million Periscope accounts. At OpenAir, Sara Haider detailed how she and her colleagues,
faced multiple challenges in developing an Android version of the live video streaming app.

There are two video livestreaming protocols, Haider explained: Real-Time Messaging Protocol (RTMP), which is suitable for low latency and real-time interaction among a few participants but isn’t widely supported; and HTTP Live Streaming (HLS), which sits atop the HTTP layer. HLS offers much higher latency, is ideal for non-interactive broadcasters, and is widely supported.
Low latency for Periscope is a must, however, so both protocols were needed to deliver the user experience on Android.

For playback, Periscope used ExoPlayer, Android’s application-level media player, which began supporting HLS earlier this year but lacks RTMP support. “So we built from scratch a completely custom RTMP broadcasting and playback stack for Periscope on Android,” Haider said.

Because RTMP isn’t widely scalable, Periscope relies on HLS to support scale. For example, when a Periscope video stream chatroom gains a lot of participants, Periscope limits the number of people who can comment to keep things moving smoothly for everyone. Those users get an HLS video stream that keeps them in sync while also enabling Periscope to scale the broadcast “to thousands of users,” Haider said. (Jump to 5:19 in the video to hear more.)

The chats accompanying a video feed are handled on their own Real-Time Chat (RTC) channel and come in instantly, though the video feed has variable latency. Also, Android clocks are “completely unreliable,” Haider said, with variations of as much as 15 seconds behind or ahead. To compensate for those variations, Periscope uses the Network Time Protocol (NTP).

The Periscope Android team consisted, at various times, of three app engineers, one video engineer, and one designer. The “bootstrapped stack” the team built consists of open-source libraries Retrofit, OkHttp, EventBus, Glide, and Spongy Castle. They used the standard tools DexGuard, Crashlytics (Twitter’s crash reporting system), Localytics, Android Studio, and Genymotion. As for application libraries, Periscope used PubNub for its chatroom channel in addition to the ExoPlayer media player. (Go to 14:21 in the video to hear more.)

The post Building Periscope for Android appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/building-periscope-for-android/feed/ 0
How Marketplace Search Differs from Traditional Search http://nerds.airbnb.com/how-marketplace-search-differs-from-traditional-search/ http://nerds.airbnb.com/how-marketplace-search-differs-from-traditional-search/#comments Tue, 04 Aug 2015 17:33:58 +0000 http://nerds.airbnb.com/?p=176595123 A marketplace search is when you go to a specific website, such as Airbnb or Amazon, to hunt for a particular product, service or, in Airbnb’s case, property. As Airbnb engineering manager Surabhi Gupta explained in her recent OpenAir 2015 talk, there are three main reasons why marketplace search is more challenging for engineers than […]

The post How Marketplace Search Differs from Traditional Search appeared first on Airbnb Engineering.

]]>

A marketplace search is when you go to a specific website, such as Airbnb or Amazon, to hunt for a particular product, service or, in Airbnb’s case, property. As Airbnb engineering manager Surabhi Gupta explained in her recent OpenAir 2015 talk, there are three main reasons why marketplace search is more challenging for engineers than traditional search (such as on Google or Bing):

Surabhi Gupta OpenAir 2

1. Conversion takes more than a click.

With traditional search, a user types a keyword or phrase, reviews the results, then clicks on the most relevant result. With a marketplace search on Airbnb, other steps sometimes must happen before a user ‘converts’ (makes a booking).

For example, a user may identify a listing she likes but wants to ask the host questions. The host can respond to the questions and accept the reservation; answer the questions but reject the booking; or not get the chance to answer the questions, and the guest moves on to another listing.

Airbnb’s search ranking uses machine learning to try and predict a final booking outcome, in order to help the guest easily find the best listing. To accomplish this, the Airbnb search team modeled the five intermediate states it cares most about: Impression (the displayed search results); Clicks (did the user like a result enough to click on it?); Accept (the host accepts the guest’s reservation); Reject (the dates don’t work out); and Booking.

Each state is given a score based on past user actions. By serving up search results influenced by how likely the user is to book the properties displayed, Airbnb has achieved “a huge booking gain,” Gupta said.

2. Decision making requires a lot of context.

Because Airbnb properties are unique, users need a lot of context to decide whether a listing is right for them. Airbnb’s engineering team has experimented with different ways of presenting information to users. For example, because location and price are the two most important requirements, Airbnb has given users information about neighborhood characteristics displayed on a city map as well as price histograms.

3. The supply is perishable.

Airbnb listings are “perishable” because they are unique properties whose availability comes and goes. To prevent users from reading about a property only to discover it’s not available when they want it, Airbnb built a real-time information infrastructure using MySQL databases, a centralized index, and Ruby on Rails. (Jump to 15:53 in the video for more details.)

Going forward, Airbnb wants to make it easier for users to pick up exactly where they left off when returning to the site; add more personalization options; and obtain a deeper understanding of its listings in order to give guests “the best possible experience” when matching them with hosts.

The post How Marketplace Search Differs from Traditional Search appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/how-marketplace-search-differs-from-traditional-search/feed/ 0
Netflix Algorithms Are Key to the ‘Future of Internet Television’ http://nerds.airbnb.com/netflix-algorithms-are-key-to-the-future-of-internet-television/ http://nerds.airbnb.com/netflix-algorithms-are-key-to-the-future-of-internet-television/#comments Fri, 24 Jul 2015 00:02:05 +0000 http://nerds.airbnb.com/?p=176595119 When Netflix’s 60 million subscribers log in to the streaming video service, their home page is populated with TV show and movie recommendations. The recommendations are key to Netflix’s success, as they drive two out of every three hours of video streamed.   The user’s home page “is where all our algorithms for recommending TV […]

The post Netflix Algorithms Are Key to the ‘Future of Internet Television’ appeared first on Airbnb Engineering.

]]>

When Netflix’s 60 million subscribers log in to the streaming video service, their home page is populated with TV show and movie recommendations. The recommendations are key to Netflix’s success, as they drive two out of every three hours of video streamed.

 

The user’s home page “is where all our algorithms for recommending TV shows and movies come together,” said Carlos Gomez Uribe, Netflix VP of Innovation. In his recent OpenAir 2015 talk, Gomez Uribe said over 100 engineers are focused on developing algorithms to help Netflix meet its business goal of “inventing the future of Internet television.”

 

One Netflix algorithm organizes the entire video catalog in a personalized way for users. Another looks for similarities between all content Netflix offers. A master algorithm “looks at all the other algorithms to decide which videos make it onto a user’s home page,” said Gomez Uribe.

 

Keyword searches drive 20 percent of video streaming hours, so Netflix’s search algorithm is tied into its recommendations and other algorithms. When users search for a title Netflix doesn’t have, the search results will display recommendations for similar shows. “We try to recommend movies related to a search, even though it’s not exactly what you wanted. All this requires a large number of algorithms,” Gomez Uribe said.

 

Personalization is important because it’s more likely to drive higher engagement with Netflix content vs. simply showing a user what’s popular. When Netflix organizes videos by popularity on user home pages, the “take rate” (the percentage of suggested videos that are actually watched) is “OK,” Gomez Uribe said. “But when we personalize recommendations, the take rate goes way up.” (Go to 5:00 in the video to hear more.)

 

Algorithms also help Netflix perform long-term A/B testing on its user interface, providing alternate ways to organize and display recommendations to users, said Gomez Uribe. In turn, the A/B testing can help Netflix measure subscriber cancellation rates more effectively. Cancellations are an easier metric to track than new member sign-ups because the latter are often fueled by word of mouth—which is notoriously difficult to track.

 

A/B testing has enabled Netflix to “stand our ground” on occasion, Gomez Uribe added. In 2011, Netflix.com unveiled a new user interface. A/B test results influenced the design, as the data showed that the new look-and-feel decreased cancellations and increased hours streamed.

 

Netflix was “so proud” of the new interface that it ran a blog post about it, “New Look and Feel for the Netflix Website” (June 8, 2011). But in short order, Netflix received a considerable number of snarky comments about the new look. Wrote one displeased subscriber: “Please inform your employers that a drunken dyslexic monkey would be a more acceptable design lead for your web concepts.”

 

Data from the A/B tests told Netflix that, despite the snark, “the majority of users were better off” with the new interface. And so, rather than rolling back to old the UI, Netflix moved forward with the new one, continuing to fine-tune it along the way. (Discussion begins around 10:40 in the video.)

The post Netflix Algorithms Are Key to the ‘Future of Internet Television’ appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/netflix-algorithms-are-key-to-the-future-of-internet-television/feed/ 0
Recap of OpenAir http://nerds.airbnb.com/recap-of-openair/ http://nerds.airbnb.com/recap-of-openair/#comments Mon, 06 Jul 2015 16:04:20 +0000 http://nerds.airbnb.com/?p=176595098 Three weeks ago we hosted OpenAir 2015, our second technology conference. We had an amazing turnout of bright minds from across the industry, more than doubling attendance from 2014. A new generation of companies are emerging whose customers aren’t judging them by their apps and websites but on the experiences and content the products connect […]

The post Recap of OpenAir appeared first on Airbnb Engineering.

]]>
Three weeks ago we hosted OpenAir 2015, our second technology conference. We had an amazing turnout of bright minds from across the industry, more than doubling attendance from 2014.

A new generation of companies are emerging whose customers aren’t judging them by their apps and websites but on the experiences and content the products connect them with. With that in mind the theme for OpenAir 2015 was scaling human connection and we focused on online to offline and the better matching that enables it.

Throughout the day we learned how Instagram helps their users discover new content that inspires them; how Stripe helps people transact across borders, how LinkedIn used data to power their social network, how Periscope came to life on Android, and of course, how Airbnb helps turns strangers into friends.

Behind all of these challenges there are central concepts that we as a tech industry need to understand better – trust, personalization and the data that enables both.

With that in mind, Airbnb open-sourced two new tools for wrangling data. The first is called Airflow which is a sophisticated tool to programmatically author, schedule and monitor data pipelines. People in the industry will know this work as ETL engineering. The second was Aerosolve. Aerosolve is a machine learning package for Apache Spark. It’s designed to combine high capacity to learn with an accessible workflow that encourages iteration and deep understanding of underlying patterns on a human level. Since we launched these tools they have gotten over 2000 stars on GitHub – we can’t wait to see how people use and contribute to them.

We also announced a new tool for our hosts called Price Tips, which is powered by Aerosolve. Price Tips creates ongoing tips for our hosts on how to price their listing, not just for one day, but for each day of the year. This pricing is fully dynamic — it takes into account demand, location, travel trends, amenities, type of home and much more. There are hundreds of signals that go into the model to produce each price tip. We believe that better pricing will be a great way to further empower our hosts to meet their personal goals through hosting.

Finally we closed out the opening keynote morning with the launch of our brand new Gift Cards website. Now anyone in the US can give their family, friends, colleagues, frenemies, whomever, the gift of travel on Airbnb. And for those lucky folks in the audience, we gave everyone a $100 gift card.

We will be following up with more videos from the event, so keep your eyes on this space.

The post Recap of OpenAir appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/recap-of-openair/feed/ 0
Android Frameworks from Airbnb and Square http://nerds.airbnb.com/android-frameworks-from-airbnb-and-square/ http://nerds.airbnb.com/android-frameworks-from-airbnb-and-square/#comments Thu, 16 Apr 2015 19:09:01 +0000 http://nerds.airbnb.com/?p=176594918 We wrapped up the month of March with with the theme of android development. It was exciting to host Pierre-Yves Ricau from Square and feature Airbnb engineers, Eric Petzel and Nick Adams. Mapstraction Many mobile applications today require some form of an interactive map as part of their user interface. Google provides a native package […]

The post Android Frameworks from Airbnb and Square appeared first on Airbnb Engineering.

]]>
We wrapped up the month of March with with the theme of android development. It was exciting to host Pierre-Yves Ricau from Square and feature Airbnb engineers, Eric Petzel and Nick Adams.

Mapstraction

Many mobile applications today require some form of an interactive map as part of their user interface. Google provides a native package and experience with Google Play Services to satisfy this need, but the questions remains of how one creates interactive maps for devices without Play Services.

In some countries, the majority of devices are sold without Google Play Services. Device manufacturers who ship their devices without Play Services are continuing to gain popularity worldwide. We have built a package that solves these problems, letting developers remain agnostic of manufacturer while providing a consistent map experience to users.

Speaker Bio

Eric Petzel: Software Engineer on the Android team at Airbnb where he focuses on building features for our hosts and guests, as well as tools to share with the Android community. Previously to Airbnb, he worked at Skype on their Android client

Nick Adams: Software Engineer on the Android team at Airbnb. He focuses on building features that improve the quality of the app and expanding to new form factors. Before Airbnb he was a student at the University of British Columbia in Canada.

Crash Fast: Square’s approach to Android crashes

The Square Register Android app has few crashes. Getting there requires a systematic approach: coding defensively, gathering information, measuring impact and improving architecture.

This talk presents our concrete steps towards lowering the crash rate, from the general philosophy to the tools we use, together with real crash examples.

Speaker Bio

Pierre-Yves Ricau: Android baker at Square. I started having fun with Java & Android as a consultant in Paris, then joined a startup in Barcelona and finally joined Square in San Francisco to work with some of the best engineers in the world. I like good wine & low entropy code.

The post Android Frameworks from Airbnb and Square appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/android-frameworks-from-airbnb-and-square/feed/ 0
CSS Frameworks and The Evolution of Airbnb’s Frontend http://nerds.airbnb.com/css-frameworks-evolution-airbnbs-frontend/ http://nerds.airbnb.com/css-frameworks-evolution-airbnbs-frontend/#comments Tue, 10 Mar 2015 23:50:54 +0000 http://nerds.airbnb.com/?p=176594802 The theme for the month of february was front-end engineering. Engineers, Spike Brehm and Fiona Tay spoke on behalf of Airbnb and we welcomed Jessica Dillon from Bugsnag. The Evolution of Airbnb’s Frontend Over the last few years, Airbnb’s frontend architecture has evolved to keep pace with the rapid advancement happening the JavaScript world. Starting […]

The post CSS Frameworks and The Evolution of Airbnb’s Frontend appeared first on Airbnb Engineering.

]]>
The theme for the month of february was front-end engineering. Engineers, Spike Brehm and Fiona Tay spoke on behalf of Airbnb and we welcomed Jessica Dillon from Bugsnag.

The Evolution of Airbnb’s Frontend

Over the last few years, Airbnb’s frontend architecture has evolved to keep pace with the rapid advancement happening the JavaScript world. Starting as a humble Rails 2 + Prototype.js app in 2008, the frontend stack powering airbnb.com has gone through a few revisions, including a push towards single-page app architecture with Backbone.js and Handlebars.js, an adventure into isomorphic JavaScript with Rendr (our library for using Node.js to server-render Backbone SPAs), and most recently, a move toward React.js and a re-envisioning of our build pipeline to take advantage of CommonJS, ES6, and a Node.js-based transform system. Spike Brehm, software engineer on the @AirbnbNerds team, will walk through how we approached and executed on these changes. Plus, get excited to see a preview of our new approach to isomorphic JavaScript, allowing us to server-render React components from our Rails app.

Speaker Bio

Spike Brehm is a software engineer at Airbnb who specializes in building rich web experiences. As a JavaScript nerd, he has spent the last few years shipping web apps and prototyping Airbnb’s front-end stack, experimenting with “isomorphic JavaScript” — apps that have the flexibility to run on both the client and sever using the same codebase.

O2: The Life of A CSS Framework

In 2012, a small startup added Bootstrap to our site as a stopgap to make things look better. Never could we have expected that in just 3 years, Airbnb would explode and our design would evolve dramatically. But what’s stayed constant, helping to keep our code sane, is our internal CSS framework. In this talk, I’ll discuss when and why we forked Bootstrap and how maintaining our own system enables beautiful, on-brand design. I’ll also discuss how we’ve rolled out drastic changes for a rebrand and later, a responsive overhaul of a desktop site, using versioning best practices.

Speaker Bio

Fiona Tay is an engineer on the Core Web team at Airbnb, where she develops strategy and build tooling for the web platform. She’s architected engineering strategy for our rebrand across web and email. She’s currently working on making airbnb.com responsive. She’s a fan of design, performance and testing.

Implementing a visual CSS testing framework

Working with large CSS codebases can be hard. Large-scale refactors, or even just tweaking styles on our more general elements, could end up having unintended consequences on the rest of the site. To catch these problems we would manually check every page on our site, which is a slow and error-prone approach. We needed a better way to test our CSS.

We looked up various ways to test CSS, including trying libraries like Huxley. Although some of it was what we needed, the frameworks ultimately didn’t end up integrating well enough for what we wanted to do. After looking at the minimal amount of support each framework was adding, we decided it would be best to rollout our own visual diffing system for our specific needs. We decided to outline a plan that included the components we’d need, and figure out from there the right tools for the job.

In this talk, I’ll walk through how we at Bugsnag implemented a visual CSS testing framework using RSpec & Selenium, using automatic screenshot comparison to catch style regressions.

Speaker Bio

Jessica Dillon is a self taught full-stack software engineer in the heart of San Francisco. She works at Bugsnag, the leading real-time crash detection service for mobile and web applications. Her expertise runs the gamut from developing rich in-browser applications to building service-oriented backends. When not coding or playing games, she likes to hang out with her dog named Boo.

The post CSS Frameworks and The Evolution of Airbnb’s Frontend appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/css-frameworks-evolution-airbnbs-frontend/feed/ 0
Surviving at Scale: Lessons from Airbnb and LinkedIn http://nerds.airbnb.com/surviving-scale-lessons-airbnb-linkedin/ http://nerds.airbnb.com/surviving-scale-lessons-airbnb-linkedin/#comments Thu, 29 Jan 2015 17:45:06 +0000 http://nerds.airbnb.com/?p=176594732 We are super excited to host our first tech talk of 2015! With the new year comes a new schedule of tech talks happening once a month. Each talk will be carefully curated with multiple speakers and a cohesive theme. The theme this month is scaling and infrastructure with Airbnb engineers, Ben Hughes and Jon […]

The post Surviving at Scale: Lessons from Airbnb and LinkedIn appeared first on Airbnb Engineering.

]]>
We are super excited to host our first tech talk of 2015! With the new year comes a new schedule of tech talks happening once a month. Each talk will be carefully curated with multiple speakers and a cohesive theme. The theme this month is scaling and infrastructure with Airbnb engineers, Ben Hughes and Jon Tai and the co-founder and head of engineering at Confluent, Neha Narkhede.

Scaling Things That Don’t Scale: Scalability and Reliability at Airbnb

Paul Graham says to do things that don’t scale, and this advice has served us very well at Airbnb. By 2012, guests had booked more than 10 million nights in 192 countries. Years of exponential growth were great for our business, but hard on our infrastructure.

In early 2013, scaling issues began to affect our user experience. As peak summer travel season ramped up, external services took longer to respond, background tasks overwhelmed our database, and memory usage grew unchecked. We started having downtime several days a week, and responding to incidents was consuming more and more engineers’ time.

A small group of us started on a journey to improve reliability and scalability. We added instrumentation, got scientific about timeouts, fixed bad queries, and threw a lot of hardware at the problem. By the end of the year, not only did we have more users and bookings than ever before, but we had improved our response time and uptime as well. We didn’t accomplish this by doing a major rewrite, removing functionality, or disrupting other teams; we accomplished this by dedicating a small team to work on application reliability, focusing on fundamentals, and working to better understand our systems.

In this talk we will share what we did to improve reliability and scalability, how we shifted from tactics to strategy, and the lessons we learned along the way.

Speaker Bio

Ben Hughes is a Software Engineer at Airbnb focusing on reliability and scalability. He was previously a cofounder of NabeWise, a neighborhood information startup that was acquired by Airbnb in 2012. Jon Tai is a Software Engineer on the Production Infrastructure team at Airbnb. Beyond scaling software systems, he works to build tooling and evangelize best practices among the engineering team. Prior to Airbnb, Jon was an engineer at IGN Entertainment, where he helped re-architect and scale out IGN’s media platform.

Real-time stream processing at scale using Apache Kafka and Samza

We are enjoying something of a renaissance in data infrastructure. The old workhorses like MySQL and Oracle still exist but they are complemented by new specialized distributed data systems like Cassandra, Redis, Druid, and Hadoop. At the same time what we consider data has changed too–user activity, stock tickers, gaming events, monitoring, logging and other event data are becoming first class citizens for data driven companies. Taking full advantage of all these systems and the relevant data creates a massive data integration problem. This problem is important to solve as these specialized systems are not very useful in the absence of a complete and reliable data flow.

One of the most powerful ways of solving this data integration problem is by restructuring your digital business logic around a centralized firehose of immutable events. Once a company’s data is captured and available as real-time streams, processing this data becomes the next challenge. Stream processing is an essential part of real-time data systems today, and is used for building news feeds, real-time analytics, metrics, alerts and monitoring.

At LinkedIn, we successfully moved virtually all data flow (500 billions events a day) to real-time structured logs using Apache Kafka. This architecture has influenced a lot of companies worldwide in doing the same and Apache Kafka is used in production at hundreds of companies [https://cwiki.apache.org/confluence/display/KAFKA/Powered+By] for similar use cases.

In this talk, I will share our experience of successfully building LinkedIn’s data pipeline infrastructure around real-time streams on top of Apache Kafka and Apache Samza. These lessons are hugely relevant to anyone building a data driven company.

Speaker Bio

Neha Narkhede is co-founder and head of engineering at Confluent. Previously, she was responsible for LinkedIn’s petabyte scale streaming infrastructure supporting hundreds of billions of events per day. She is also one of the initial authors of Apache Kafka and serves as a PMC member and committer for the project. In the past she has worked on search within the database at Oracle and holds a Masters in Computer Science from Georgia Tech.

The post Surviving at Scale: Lessons from Airbnb and LinkedIn appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/surviving-scale-lessons-airbnb-linkedin/feed/ 1
Payments at Airbnb http://nerds.airbnb.com/payments-airbnb/ http://nerds.airbnb.com/payments-airbnb/#comments Wed, 17 Dec 2014 20:29:18 +0000 http://nerds.airbnb.com/?p=176594675 Talk Abstract Payments at Airbnb is challenging because it operates at the core of a global marketplace to create trust. No single solution exists to meet our needs. Going global involves supporting multiple currencies and payment methods but it also involves nuanced complexities that are unique to local markets. It’s very common for our problems […]

The post Payments at Airbnb appeared first on Airbnb Engineering.

]]>
Talk Abstract

Payments at Airbnb is challenging because it operates at the core of a global marketplace to create trust. No single solution exists to meet our needs. Going global involves supporting multiple currencies and payment methods but it also involves nuanced complexities that are unique to local markets. It’s very common for our problems to cross different domains such as product, compliance, growth, finance, and engineering infrastructure. Airbnb’s two-sided marketplace includes guests and hosts who introduce two different payment problems: collecting and distributing money. Our payments platform also supports other activities in the marketplace such as photographers, translators, trip services, and more in the future.

We’ll talk about building end-to-end systems (upstream and downstream) that process billions of dollars in a reliable and scalable way. Want to know how to change tires while driving in a production environment that has significant money flow? Want to know how to confidently sprint instead of walk with code on a frequent basis? As a result of Airbnb’s rapid growth, its Payments platform has evolved quite a bit and we’d like to talk about it while also providing concrete tips based on our experience.

Speaker Bio

Ian Logan is an Engineering Manager at Airbnb. He joined Airbnb over 3 and a half years ago as the first payments engineer, bringing his experience working in investment banking at BlackRock, Barclays Global Investors, and Morgan Stanley to a new problem of payments in the sharing economy. During his time at Airbnb, Ian worked on all aspects of core payments technology and built its original foundation. Currently, Ian leads the engineering teams responsible for Payments, Trust & Safety, and Internal Products at Airbnb.

The post Payments at Airbnb appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/payments-airbnb/feed/ 0