Airbnb Engineering http://nerds.airbnb.com Nerds Tue, 31 Mar 2015 00:34:26 +0000 en-US hourly 1 http://wordpress.org/?v=3.8.5 Meet the Nerds: Phillippe Siclait http://nerds.airbnb.com/meet-nerds-phillippe-siclait/ http://nerds.airbnb.com/meet-nerds-phillippe-siclait/#comments Thu, 19 Mar 2015 19:36:02 +0000 http://nerds.airbnb.com/?p=176594831 In this Q&A we meet Phillippe Siclait. Phillippe has been with Airbnb for over 2 years and has worked on five teams since joining – clearly he likes to travel both personally and professionally! How did you get started in Computer Science? I started programming to make video games. When I was in elementary school, […]

The post Meet the Nerds: Phillippe Siclait appeared first on Airbnb Engineering.

]]>
In this Q&A we meet Phillippe Siclait. Phillippe has been with Airbnb for over 2 years and has worked on five teams since joining – clearly he likes to travel both personally and professionally!

How did you get started in Computer Science?
I started programming to make video games. When I was in elementary school, I found a book at a school book fair called “Learn to Program Basic”. I had the vague sense that programming was a thing you needed to do to make games, so I picked it up, went through all the exercises, and got hooked. In the proceeding years, I continued to learn from all the resources I could find online. This led to doing programming competitions through my school and the development of an interest in graphics programming. By the end of high school I was pretty set on studying either CS or Economics.

What was your path to Airbnb?
It turns out that I didn’t actually end up majoring in CS. I received a BS in Economics mostly focused on game theory and econometrics while simultaneously working in the Computer Graphics group in the Computer Science and Artificial Intelligence Lab. I decided prior to my final year of school that management consulting would help me learn how to run a business, and after an internship at the Boston Consulting Group, I accepted a full-time offer there. I learned a lot at BCG and got to do a fair amount of travel, both for work and for fun, but in the end realized that I wanted to come back to the technical side. With a couple friends, I packed my bags and moved across the country to explore what the Bay had to offer. After a few months of working independently on mobile and web projects, a friend of mine brought me over to Airbnb for a Tech Talk. It was after meeting many people, hearing about the vision of the company, and thinking about how much I wanted to see this idea spread, that I knew that I had to be here.

What’s the most interesting technical challenge you’ve worked on since joining?
Since joining Airbnb I’ve worked on many teams. I started working on Search (frontend, ranking, infrastructure, evaluation tools, and more), worked a bit on web security, and then worked on our (at the time newly formed) Discovery team. One of the problems for Discovery is recommending places and listings to our guests. I found it fascinating to think through how you could determine which locations an individual would likely be interested in and I worked with my teammates to implement the data pipelines and serving architecture for providing the recommendations to many parts of our product. It was a complex problem partly because the decisions we make while traveling are very personal and multi-faceted.

What do you want to work on next?
I’m currently working on a team that focuses on engaging people to host on Airbnb. We’re an incredibly experiment driven team and we have plans to change many parts of the product to make starting to host easier. We have a small, multidisciplinary team of designers, data scientists, a product manager and engineers who are all focused on this problem. I’m excited to see us increase the rate at which we are testing new product changes.

What is your favorite core value, and how do you live it?
Simplify. I try to simplify any code I write. Code is meant to be read by people and as a result, the simplest code is often the best code. And all of us at Airbnb live it in the product we develop. We are building a product for our guests and hosts, and a simple product leads to a better end experience.

What’s your favorite Airbnb experience?
My favorite Airbnb experience may have actually been over a one night stay in Manila earlier this year. The host had a beautiful house with wonderful, vibrant wood throughout, and he was hosting other guests who were visiting from the UK. He invited us out to dinner that night and it was fascinating to get to know these people who had lives so different from my own. Our host was an American expat who had long been in the Philippines and the other guests were retirees who traveled the world, exploring new places and teaching motorcycle racing. The dinner was wonderful and we spent several hours afterwards chatting in his kitchen about our lives and what we had all seen of the world. In the morning our host organized our transportation back to the airport and made sure we knew that we were welcome the next time we were back in Manila.

The post Meet the Nerds: Phillippe Siclait appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/meet-nerds-phillippe-siclait/feed/ 0
CSS Frameworks and The Evolution of Airbnb’s Frontend http://nerds.airbnb.com/css-frameworks-evolution-airbnbs-frontend/ http://nerds.airbnb.com/css-frameworks-evolution-airbnbs-frontend/#comments Tue, 10 Mar 2015 23:50:54 +0000 http://nerds.airbnb.com/?p=176594802 The theme for the month of february was front-end engineering. Engineers, Spike Brehm and Fiona Tay spoke on behalf of Airbnb and we welcomed Jessica Dillon from Bugsnag. The Evolution of Airbnb’s Frontend Over the last few years, Airbnb’s frontend architecture has evolved to keep pace with the rapid advancement happening the JavaScript world. Starting […]

The post CSS Frameworks and The Evolution of Airbnb’s Frontend appeared first on Airbnb Engineering.

]]>
The theme for the month of february was front-end engineering. Engineers, Spike Brehm and Fiona Tay spoke on behalf of Airbnb and we welcomed Jessica Dillon from Bugsnag.

The Evolution of Airbnb’s Frontend

Over the last few years, Airbnb’s frontend architecture has evolved to keep pace with the rapid advancement happening the JavaScript world. Starting as a humble Rails 2 + Prototype.js app in 2008, the frontend stack powering airbnb.com has gone through a few revisions, including a push towards single-page app architecture with Backbone.js and Handlebars.js, an adventure into isomorphic JavaScript with Rendr (our library for using Node.js to server-render Backbone SPAs), and most recently, a move toward React.js and a re-envisioning of our build pipeline to take advantage of CommonJS, ES6, and a Node.js-based transform system. Spike Brehm, software engineer on the @AirbnbNerds team, will walk through how we approached and executed on these changes. Plus, get excited to see a preview of our new approach to isomorphic JavaScript, allowing us to server-render React components from our Rails app.

Speaker Bio

Spike Brehm is a software engineer at Airbnb who specializes in building rich web experiences. As a JavaScript nerd, he has spent the last few years shipping web apps and prototyping Airbnb’s front-end stack, experimenting with “isomorphic JavaScript” — apps that have the flexibility to run on both the client and sever using the same codebase.

O2: The Life of A CSS Framework

In 2012, a small startup added Bootstrap to our site as a stopgap to make things look better. Never could we have expected that in just 3 years, Airbnb would explode and our design would evolve dramatically. But what’s stayed constant, helping to keep our code sane, is our internal CSS framework. In this talk, I’ll discuss when and why we forked Bootstrap and how maintaining our own system enables beautiful, on-brand design. I’ll also discuss how we’ve rolled out drastic changes for a rebrand and later, a responsive overhaul of a desktop site, using versioning best practices.

Speaker Bio

Fiona Tay is an engineer on the Core Web team at Airbnb, where she develops strategy and build tooling for the web platform. She’s architected engineering strategy for our rebrand across web and email. She’s currently working on making airbnb.com responsive. She’s a fan of design, performance and testing.

Implementing a visual CSS testing framework

Working with large CSS codebases can be hard. Large-scale refactors, or even just tweaking styles on our more general elements, could end up having unintended consequences on the rest of the site. To catch these problems we would manually check every page on our site, which is a slow and error-prone approach. We needed a better way to test our CSS.

We looked up various ways to test CSS, including trying libraries like Huxley. Although some of it was what we needed, the frameworks ultimately didn’t end up integrating well enough for what we wanted to do. After looking at the minimal amount of support each framework was adding, we decided it would be best to rollout our own visual diffing system for our specific needs. We decided to outline a plan that included the components we’d need, and figure out from there the right tools for the job.

In this talk, I’ll walk through how we at Bugsnag implemented a visual CSS testing framework using RSpec & Selenium, using automatic screenshot comparison to catch style regressions.

Speaker Bio

Jessica Dillon is a self taught full-stack software engineer in the heart of San Francisco. She works at Bugsnag, the leading real-time crash detection service for mobile and web applications. Her expertise runs the gamut from developing rich in-browser applications to building service-oriented backends. When not coding or playing games, she likes to hang out with her dog named Boo.

The post CSS Frameworks and The Evolution of Airbnb’s Frontend appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/css-frameworks-evolution-airbnbs-frontend/feed/ 0
Introducing Airpal http://nerds.airbnb.com/airpal/ http://nerds.airbnb.com/airpal/#comments Thu, 05 Mar 2015 15:58:45 +0000 http://nerds.airbnb.com/?p=176594775 Airbnb is pleased to announce the launch of Airpal, a web-based query execution tool that leverages Facebook’s PrestoDB to facilitate data analysis. People who spend time using SQL for exploration and investigation know that the workflow is not always smooth. Remembering how a query was written, copying and pasting from the command line, and running […]

The post Introducing Airpal appeared first on Airbnb Engineering.

]]>
Airbnb is pleased to announce the launch of Airpal, a web-based query execution tool that leverages Facebook’s PrestoDB to facilitate data analysis.

People who spend time using SQL for exploration and investigation know that the workflow is not always smooth. Remembering how a query was written, copying and pasting from the command line, and running multiple terminal windows can slow down analysis and be frustrating. Additionally, when diverse teams are using SQL for analytics, the learning curve can be steep for beginners, so good UI tools can help drive adoption and promote knowledge sharing.

At Airbnb, we launched Airpal internally about a year ago and now more than 1/3 of all employees have issued a query through the tool. This is an astounding statistic and it shows how integral Presto is to our company’s data infrastructure.

We currently hold about one and a half petabytes of data as Hive managed tables in HDFS, and the relatively small data size of our important “core_data” tables allows us to use Presto as the default query engine for analysis. When running ad hoc queries and iterating on the steps of an analysis, Presto is much snappier and more responsive than traditional map reduce jobs. The biggest benefit to adding Presto to our infrastructure stack, though, is that we don’t have to add additional complexity to allow “interactive” querying. Because we are querying against our one, central Hive warehouse, we can keep a “single source of truth” with no large scale copies to a separate storage/query layer. Additionally, the fact that we don’t need change data storage type from RC format to see the speed improvements, makes Presto a great choice for our infrastructure.

We are excited to share this tool with the open source community and we hope that it can provide similar utility for others.

demo

Key features of Airpal:

  • optional access controls for users
  • ability to search and find tables
  • see metadata, partitions, schemas, and sample rows
  • write queries in an easy-to-read editor
  • submit queries through a web interface
  • track query progress
  • get the results back through the browser as a CSV
  • create new Hive table based on the results of a query
  • save queries once written
  • searchable history of all queries run within the tool

Keeping with the spirit of Presto, we have tried to make it simple to install Airpal by providing a local storage option for people who would like to test it out without any overhead or cost. For more detailed information, visit the GitHub page here: https://github.com/airbnb/airpal

A few of the notable technology features in Airpal are:

  1. uses Dropwizard as an easy way to provide a REST service in Java
  2. employs SSE (Server Sent Events) to push messages from the server to the client
  3. the front end JavaScript uses react.js

Finally, we would be remiss if we did not mention the awesome direction that Facebook provided as the original developers of Hive and the pioneers of building UI tools to facilitate easy access to big data. We stood on the shoulders of giants to make this tool and we appreciate the influence and input that the data infrastructure and data tools teams at Facebook were able to provide.

If you’re interested in helping build a world class suite of data tools, check out this open job position: https://www.airbnb.com/jobs/departments/position/48112

The post Introducing Airpal appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/airpal/feed/ 7
Meet the Nerds: Daniel Loreto http://nerds.airbnb.com/meet-nerds-daniel-loreto/ http://nerds.airbnb.com/meet-nerds-daniel-loreto/#comments Tue, 03 Mar 2015 19:48:01 +0000 http://nerds.airbnb.com/?p=176594768 Today we meet Daniel Loreto, an engineering manager here who is an entrepreneur at heart. Daniel tells us about his love for high risk/high reward projects and his new life moving from NYC to SF for his role at Airbnb. How did you get started in Computer Science? When I was a kid my dad […]

The post Meet the Nerds: Daniel Loreto appeared first on Airbnb Engineering.

]]>
Today we meet Daniel Loreto, an engineering manager here who is an entrepreneur at heart. Daniel tells us about his love for high risk/high reward projects and his new life moving from NYC to SF for his role at Airbnb.

How did you get started in Computer Science?

When I was a kid my dad bought our first personal computer for the house, an IBM PC. It was state-of-the-art at the time: a monochrome monitor, no hard drive, a 5 1/2 inch floppy drive, and whopping 64 kb of RAM. I would use it to play games like paratrooper and snake. One day I was poking around the floppy disk that had the game of snake and I found a file with what looked like gibberish english. I asked my dad about it and he explained it was actually the source code for the game written in BASIC.

My kid-mind was blown. I knew people could write novels and that they could compose music … but games? computer programs? You could actually write something interactive and have it do things? I was hooked ever since.

What was your path to Airbnb?

I lived in six different countries growing up and speak three different languages. I’ve love to travel and getting to know other cultures. Most of my professional career was in New York, but I was looking to move to California and join a company where I could have a big impact and could connect with the mission. I talked to a few friends at Airbnb and I knew I had found a match. As a first project I also had the opportunity to work on a team directly with Joe Gebbia, one of the founders. It was a very unique opportunity to work with a founder on challenging new directions for the company.

What’s the most interesting technical challenge you’ve worked on since joining?

In my current project we’re working on new features that changes some of the fundamental assumptions of the original design of Airbnb’s architecture. As a result my team and I have had to very quickly familiarize ourselves with the overall architecture, and then dig deep into some pretty core and complex systems like payments and search to make our project possible.

What do you want to work on next?

I’m always looking for potentially risky but high-impact projects. I love working on new product directions that expand what’s possible for Airbnb today and I’m always pushing my team to work on big ideas that can really move the needle if they’re successful. We have a few ideas that we want to work on next that fit that description …. but I would be giving it all away if I revealed them now. Stay tuned!

What is your favorite core value, and how do you live it?

My favorite core value is, without a doubt, Cereal Entrepreneur. To me it embodies the idea of being creative, scrappy, bold and resourceful in order to accomplish your goals. Don’t be afraid of big audacious goals even with a small team. Is there something blocking you from accomplishing that goal? Don’t be afraid to tackle it, even if it’s outside of your area of expertise. Make things happen. Drive towards a big vision. Shoot for the moon. Don’t stop.

The post Meet the Nerds: Daniel Loreto appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/meet-nerds-daniel-loreto/feed/ 0
Large Scale Payments Systems and Ruby on Rails http://nerds.airbnb.com/large-scale-payments-systems-ruby-rails/ http://nerds.airbnb.com/large-scale-payments-systems-ruby-rails/#comments Thu, 26 Feb 2015 15:33:56 +0000 http://nerds.airbnb.com/?p=176594743 I’ve been writing code for fun and for work at both small startups and large companies for the past 30 years. Over the years I’ve become fascinated with the payments industry, and I have spent the past decade working in payments related companies. Before coming to work on Payments at Airbnb, I worked for PayPal, […]

The post Large Scale Payments Systems and Ruby on Rails appeared first on Airbnb Engineering.

]]>
I’ve been writing code for fun and for work at both small startups and large companies for the past 30 years.

Over the years I’ve become fascinated with the payments industry, and I have spent the past decade working in payments related companies. Before coming to work on Payments at Airbnb, I worked for PayPal, for a few startups and then was part of the NFC Wallet project at Google.

The payments space is interesting because it combines a strong need for hard core computer science with an industry that is built on an aging foundation. The basic way payments work around the world has essentially stayed the same even as technologies such as location-awareness and strong encryption have become ubiquitous. The industry is ripe for change but extremely resistant to it. It is a fun technical problem and a tough business problem.

Over the years I’ve seen payments systems written in a variety of languages. I’ve noticed that the way payments work translates to requirements that affect the language and framework selection trade offs. For example, payments applications typically require strong transactional integrity, a robust audit trail and very predictable failure behavior.

The Challenge

Ruby on Rails is well known for its quick iteration cycle and the plethora of magical tools that speed up development and simplify prototyping. Those benefits are focused on improving the development process, but in some cases make maintenance of production system more difficult. In terms of the way payments work, some of its shortcomings mean trouble.

Testability

The ActiveRecord pattern favors large, monolithic model classes that contain database access logic and business logic. Testing these classes is difficult. It is not easy to describe their dependencies above and beyond the database table they depend on, and it it is therefore hard to write good, comprehensive unit tests for them. This makes it difficult to reason about their behavior. E.g.: Can you really guarantee that this 3000 line model that touches our transaction table does not contain off-by-one-cent errors if it doesn’t have really good unit tests?

Audit trail

Maintaining a log of ‘who did what when’ for every change involving money or user information is an important part of any payments system. ActiveRecord makes it trivially easy to make database changes, making it hard to ensure that every part of the code that mutates data actually records an audit trail. Furthermore, in some cases (e.g. with mass updates such as update_all), there is no way to enforce that every database change creates an audit trail. The only way to programmatically capture all data mutation is via database triggers.

Predictable Failure Behavior

It is usually a good idea to explicitly check for expected conditions and parameter values at the start of public interface methods. For example, if a parameter can’t be nil, raising an exception if nil is passed. This translates into very loud failure scenarios, but ultimately ensures that problems bubble up and get fixed, and makes bugs easier to track down.

Ruby’s weak typing makes it more difficult to enforce these rules. In strongly typed languages, the compiler complains if you pass in an int where a string is expected.

Furthermore, Ruby has lots of cases where nil is used as a sentinel to signify ‘no value returned’. This is dangerous because nil is also the default value of an uninitialized variable. This makes it difficult to identify bugs where a variable name is mistyped or otherwise not properly initialized.

The Airbnb way

Taking into account those issues, payments might seem more suited for a strongly typed language where the database access model is more restrictive. Yet, at Airbnb we use Rails for our payments stack. Instead of ditching Rails for its drawbacks, we’ve come up with solutions for its main issues, making it work well for payments. This enables us to continue to enjoy the many benefits it provides.

Reducing ActiveRecord’s surface area

A key requirement for proper audit trails is reduced access of code to the raw database data. To achieve this we’re starting to use a layer on top of ActiveRecord we call ProtectedAccess which controls changes to the underlying table and ensures an audit trail is written. It also greatly reduces the ActiveRecord surface area by not exposing all of its data mutation methods by default, and thus forces developers to reason about what methods they call, making it more likely that unwanted data mutations are caught during code reviews.

ProtectedAccess is designed to replace an existing ActiveRecord::Base class. It exposes some of the same interfaces, but actually hides most of the ways to mutate an object. The idea is to disallow most mutations to a Model that can’t easily be audit-trailed, and for those that can, create a shim that transparently creates the audit trail.

For example, suppose you have a Payment model that has an amount field. By virtue of extending ActiveRecord::Base, it exposes a setter amount=. Any piece of code can call:

  
p = Payment.last
p.amount = 234
p.save
  

which would create an un-audited modification of the payment record.

In comparison, ProtectedAccess creates a shim between the surface area, which still contains amount= and save, and the ActiveRecord model. When save is called it instead creates and saves a new version of the record instead of overwriting the existing version. There is no way for anyone to call the actual model that has access to the Payments table because it does not exist, save for a private member of the class Payment < ProtectedAccess object.

Parameter Validation – Fail Fast (and loud!) and Explicit Dependencies

We use a declarative framework for parameter validation that enables service objects (as well as methods) to check that parameters are of the correct type and value range, ensure mandatory parameters are present and also ensure that no unexpected parameters are passed in. It is loosely inspired by Guava’s Preconditions class.

An example might be an implementation of the validate method of a service object:

  
def validate(options)
  validate_arg(options) do |o|
    o.validate :txn, is_a: Transaction, required: true
    o.validate :buyer, is_a: User, required: true
    o.validate :status, is_a: String, required: true
    o.validate :confirmation_code
  end
end
  

This is not an attempt to make ruby strongly typed (though that might not be such a bad idea…), but rather a way to explicitly declare and enforce the dependencies of an object or a method. It is also not dealing with instantiation, as some dependency injection frameworks do. But it can make it clearer to someone reading a piece of code what its dependencies are.  It also causes faster failures. Without validation a bug would cause an unexpected value to propagate through the code until (hopefully) something blew up or (more likely) a weird result was displayed to a user with no indication as to the source. With validation an exception is thrown, a process crashes, and developers get alerted to the problem. Arguably, it is better to tell the user something went wrong rather than to present something that might be completely off.

Freezing constants is another related method we use, above and beyond the warnings the ruby interpreter issues when modifying a constant.

Parameter validation is very generic, and can be used with any Ruby application. We’re currently evaluating when the right time might be to open source it.

Service Objects

Instead of placing business logic in models, we keep models lean and focused only on data retrieval and storage. All business logic goes in service objects, which are single use business logic objects that are initialized with a clearly defined set of parameters, and then perform an action on them. They tend to be smaller in size and very focused on executing a particular task. Because they have a well defined set of parameters, they are much easier to test, and can be tested primarily using mocking (as opposed to creating test database entries or having test network services).

As an example, suppose you have some code that makes a call to some external gateway for completing a transaction. It might look like

  
def complete_transaction(txn, buyer, status, confirmation_code = "")
  if txn.empty?
    raise Error, "Must call complete transaction with a non null txn"
  end
  # more validations

  response = Net::HTTP.post_form....
  # inspect response and return accordingly
end
  

A service object might look like:

  
class CompleteTransaction < ServiceBase
  def initialize(opts = {})
    validate(opts)
    @txn = opts[:txn]
    @buyer = opts[:buyer]
    @status = opts[:status]
  end

  def validate(opts)
    # use parameter validation to strongly check expected parameter values
  end  

  def perform
    response = Net::HTTP.post_form....
    # inspect response and return accordingly
  end
end
  

Calling this service object would be done like this:

  
CompleteTransaction.new({ txn: txn, buyer: user, status: "Complete" }).call
  

The service object framework will do some pre flight logging and then call perform, following it with post flight logging.

Note that validation, logging and other administrativia are now segregated to their own methods, and the perform method can focus on pure business logic. It can make lots of assumptions on the parameters, as they have been strongly validated. The interesting part of the code is also more easily accessible for someone trying to figure out what it is doing.
Because the o.validate must declare every parameter that can be passed to the service object, it is really clear what it depends on. This is a form of code-documentation that does not go out of date.

Strict Code Style and Code Complexity Filters

To make code easier to understand, we use rubocop and cane in some of our repos. These tools enforce a common style on our code (rubocop) and ensure code is not overly complex (cane).

Rubocop is great for helping teach newcomers the ‘lay of the land’, and also keep everyone in sync as to what is the expected style of code. Cane’s ability to measure ABC code size metric is very useful in forcing the creation of small, easy to read and understand methods. We’ve occasionally seen cases where the cane threshold we chose (15 ABC score) was too restrictive, but generally it has pushed us to remain clear and concise in our code.

Take for example the following condition:

  
if params[:user].messages.last.send_by < Time.now
  params[:user].messages.last.send
else
  job.retry_at(params[:user].messages.last.send_by)
end
  

This has the ABC score of 12.

Compare it with the equivalent:

  
email = params[:user].messages.last
if email.send_by >= Time.now
  return job.retry_at(email.send_by)
end

email.send
  

which has an ABC score of 8.

The differences are simple:

  1. Edge case conditions are evaluated early, and processing is halted instead of having an else clause.
  2. The email parameter is extracted rather than dereferencing through params and messages.

Cane does not propose those changes, it simply alerts you that there is too much complexity. It is up to the team to establish rules of thumb on how to reduce complexity when that happens.

DRY things up – Gems and Modules

The practice of DRY-ing things up is used quite heavily in the way of gems or mixins.
Gems are typically used to extract commonly used functionality that needs to be shared among more than one internal app. Some of our gems are also open source. Aside from those open source examples, we also have internal gems for securely sending credit card information to our credit card processor, for keeping currency information consistent among our various applications and for argument validation.

In cases where functionality needs to be reused within the same application, we sometimes use module mixins – putting functionality in a module and then including it where it is needed.

Disable replica reads

A common scaling strategy in a MySQL environment is to use database replicas for reads, thus reducing the traffic on the master MySQL. This has the disadvantage that your code might sometimes get stale data. For payments, we mostly disable replica reads, trading off load on the main database for data integrity. E.g, it is not acceptable for a new transaction you make to not include the results of the previous one.

Transition

Over the past several years we have learned a lot about building and maintaining large scale payments systems. Some of what we have learned is already implemented in our code base and is part of our daily best practices. Other learnings are still fairly new, and the frameworks they rely on still under construction. As always, our system continues to evolve and we’re constantly looking for ways to make it better and more robust.

If you’re interested in taking a more active part in our journey towards a better, more scalable and robust system, I’d love to hear from you – michel.weksler@airbnb.com

The post Large Scale Payments Systems and Ruby on Rails appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/large-scale-payments-systems-ruby-rails/feed/ 1
Surviving at Scale: Lessons from Airbnb and LinkedIn http://nerds.airbnb.com/surviving-scale-lessons-airbnb-linkedin/ http://nerds.airbnb.com/surviving-scale-lessons-airbnb-linkedin/#comments Thu, 29 Jan 2015 17:45:06 +0000 http://nerds.airbnb.com/?p=176594732 We are super excited to host our first tech talk of 2015! With the new year comes a new schedule of tech talks happening once a month. Each talk will be carefully curated with multiple speakers and a cohesive theme. The theme this month is scaling and infrastructure with Airbnb engineers, Ben Hughes and Jon […]

The post Surviving at Scale: Lessons from Airbnb and LinkedIn appeared first on Airbnb Engineering.

]]>
We are super excited to host our first tech talk of 2015! With the new year comes a new schedule of tech talks happening once a month. Each talk will be carefully curated with multiple speakers and a cohesive theme. The theme this month is scaling and infrastructure with Airbnb engineers, Ben Hughes and Jon Tai and the co-founder and head of engineering at Confluent, Neha Narkhede.

Scaling Things That Don’t Scale: Scalability and Reliability at Airbnb

Paul Graham says to do things that don’t scale, and this advice has served us very well at Airbnb. By 2012, guests had booked more than 10 million nights in 192 countries. Years of exponential growth were great for our business, but hard on our infrastructure.

In early 2013, scaling issues began to affect our user experience. As peak summer travel season ramped up, external services took longer to respond, background tasks overwhelmed our database, and memory usage grew unchecked. We started having downtime several days a week, and responding to incidents was consuming more and more engineers’ time.

A small group of us started on a journey to improve reliability and scalability. We added instrumentation, got scientific about timeouts, fixed bad queries, and threw a lot of hardware at the problem. By the end of the year, not only did we have more users and bookings than ever before, but we had improved our response time and uptime as well. We didn’t accomplish this by doing a major rewrite, removing functionality, or disrupting other teams; we accomplished this by dedicating a small team to work on application reliability, focusing on fundamentals, and working to better understand our systems.

In this talk we will share what we did to improve reliability and scalability, how we shifted from tactics to strategy, and the lessons we learned along the way.

Speaker Bio

Ben Hughes is a Software Engineer at Airbnb focusing on reliability and scalability. He was previously a cofounder of NabeWise, a neighborhood information startup that was acquired by Airbnb in 2012. Jon Tai is a Software Engineer on the Production Infrastructure team at Airbnb. Beyond scaling software systems, he works to build tooling and evangelize best practices among the engineering team. Prior to Airbnb, Jon was an engineer at IGN Entertainment, where he helped re-architect and scale out IGN’s media platform.

Real-time stream processing at scale using Apache Kafka and Samza

We are enjoying something of a renaissance in data infrastructure. The old workhorses like MySQL and Oracle still exist but they are complemented by new specialized distributed data systems like Cassandra, Redis, Druid, and Hadoop. At the same time what we consider data has changed too–user activity, stock tickers, gaming events, monitoring, logging and other event data are becoming first class citizens for data driven companies. Taking full advantage of all these systems and the relevant data creates a massive data integration problem. This problem is important to solve as these specialized systems are not very useful in the absence of a complete and reliable data flow.

One of the most powerful ways of solving this data integration problem is by restructuring your digital business logic around a centralized firehose of immutable events. Once a company’s data is captured and available as real-time streams, processing this data becomes the next challenge. Stream processing is an essential part of real-time data systems today, and is used for building news feeds, real-time analytics, metrics, alerts and monitoring.

At LinkedIn, we successfully moved virtually all data flow (500 billions events a day) to real-time structured logs using Apache Kafka. This architecture has influenced a lot of companies worldwide in doing the same and Apache Kafka is used in production at hundreds of companies [https://cwiki.apache.org/confluence/display/KAFKA/Powered+By] for similar use cases.

In this talk, I will share our experience of successfully building LinkedIn’s data pipeline infrastructure around real-time streams on top of Apache Kafka and Apache Samza. These lessons are hugely relevant to anyone building a data driven company.

Speaker Bio

Neha Narkhede is co-founder and head of engineering at Confluent. Previously, she was responsible for LinkedIn’s petabyte scale streaming infrastructure supporting hundreds of billions of events per day. She is also one of the initial authors of Apache Kafka and serves as a PMC member and committer for the project. In the past she has worked on search within the database at Oracle and holds a Masters in Computer Science from Georgia Tech.

The post Surviving at Scale: Lessons from Airbnb and LinkedIn appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/surviving-scale-lessons-airbnb-linkedin/feed/ 1
Mapping the World http://nerds.airbnb.com/mapping-world/ http://nerds.airbnb.com/mapping-world/#comments Wed, 07 Jan 2015 19:15:45 +0000 http://nerds.airbnb.com/?p=176594683 People are always surprised when I tell them about Zack Walker, Airbnb’s in-house cartographer, who works on mapping the world. They say, “the world is already mapped! There’s Google Maps, Foursquare, Yelp, Garmin, National Geographic, Zheng He, Isabella Bird, Lewis & Clark & Sacagawea, Leif Erikson, the Planet Earth series narrated by David Attenborough…” Which […]

The post Mapping the World appeared first on Airbnb Engineering.

]]>
People are always surprised when I tell them about Zack Walker, Airbnb’s in-house cartographer, who works on mapping the world.

They say, “the world is already mapped! There’s Google Maps, Foursquare, Yelp, Garmin, National Geographic, Zheng He, Isabella Bird, Lewis & Clark & Sacagawea, Leif Erikson, the Planet Earth series narrated by David Attenborough…”

Which is true, so what’s all this about?

This is all about “AT-AT,” a new internal tool we built to understand locations and their relationships to other locations. It looks like this:

Big questions and hard problems

At Airbnb, the question we want to answer is how do you understand a place without ever having been there?

How do you capture why your friend said you wouldn’t like Fisherman’s Wharf, but you should still go there to get a double double from In N Out?

How do you know where in the world will make for your perfect trip?

Current solutions

We currently tackle this problem in a number of ways:

  • When you’re searching for a place to stay, our location relevance model lets the Airbnb community inform future guests about great places to stay.
  • When you’re exploring new places to experience, Airbnb Neighborhoods combines local editorial content with the handy, need-to-know information alongside professional photos to explore a place without having to leave your chair.
  • Our discovery team is building big things with natural language processing and machine learning algorithms to understand reviews, listing descriptions, and search patterns to recommend the perfect place to stay right from our homepage.
  • Host guidebooks let you find great recommendations from the host.

These products combine to help Airbnb travelers discover all of the wonderful places that make up a city, not just downtown areas where traditional accommodations are normally found.

sf

Airbnb accommodations (red) and traditional accommodations (blue) in San Francisco

Location is a crucial building block for a number of our engineering efforts. Our team needs to help the Airbnb traveler mitigate their number one concern when finding a place to stay: location. On the flipside we want to highlight the million-plus unique listings opened up by Airbnb hosts around  the world.

What’s next

Neighborhoods can only take you so far when trying to match you with the perfect place to stay.

There are culturally significant areas or regions that aren’t drawn on maps. They mostly live in the minds of the local community that calls those areas home. These areas might be a unique part of town along a single road, cutting through 3 neighborhoods. Or they might be a way to describe something in common among multiple neighborhoods.

Finding and mapping these places takes a heroic effort on Zack Walker’s part. He researches historical and current data, talks with folks from the community, and answers emails from hosts to build a clear image of how locals understand their place in the world.

So when you’re looking for a place to stay in “wine country” in Northern California we should be showing you Napa and Sonoma and helping you understand the differences and similarities between the two counties.

While we were very excited to make this happen, this wasn’t possible with our current architecture and existing internal tools. We needed something bigger.

Enter project AT-AT

A small team consisting of Christopher Lin, Alex Blackstock, Daniel Loreto, and myself set out to build the tool to help Zack Walker’s pursuit of mapping the world.

A note on the project code name: Zack wouldn’t let us call it “The Walker System”, so Christopher Lin did the next best thing and used a Star Wars reference (the Imperial Walker or All-Terrain Armored Transport, commonly called “AT-AT”).

AT-AT’s tech stack looks like this:

Our first step was generalizing the existing system so it could handle various geographical polygons (continents, countries, market areas, special regions, towns) and not just cities and neighborhoods.

the bermuda triangle

Relationships

When a polygon is created, the backend calculates the following:

  • Ancestor polygons of the higher type, polygons that contain this polygon (San Francisco => SoMa)
  • Adjacent polygons of the same type, polygons that overlap/border (SoMa => Civic Center)
  • Descendant polygons of a lower type, polygons that this polygon contains (SoMa => Folsom Street Fair)
  • The lat/lng of the centroid of the polygon

Navigation

Navigating a workspace of polygons can be tricky. It took us a few iterations before we landed on interaction patterns that let the UI disappear into the background.

The basic tenents of the system’s UI:

  1. Don’t break the back button
  2. Render only what’s necessary
  3. Reveal information as needed
  4. Get out of the way

We settled on the following interactions: a single click selects a polygon and renders only its adjacent polygons,

and a double-click snaps to the selected polygon.

A tooling culture

Building the future of travel requires a number of special tools working together in the background.

AT-AT is just one of these tools, acting as the foundation for capturing location data for other services and tools to consume — the first step towards answering “Where’s the perfect next trip for me?”

When building tools, remember that you’ll never be able to predict the use cases of the future.

nasa-and-design

And you’ll learn the most from shipping.

Special thanks

Zack Walker, Ben Hughes, Andy Kramolisch, Ann Montgomery.

The post Mapping the World appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/mapping-world/feed/ 8
Payments at Airbnb http://nerds.airbnb.com/payments-airbnb/ http://nerds.airbnb.com/payments-airbnb/#comments Wed, 17 Dec 2014 20:29:18 +0000 http://nerds.airbnb.com/?p=176594675 Talk Abstract Payments at Airbnb is challenging because it operates Erklärung vermieden viagra patentschutz ende zu Ich Gehen.Gestern wirkung sildenafil ratiopharm Ihr solltest 100% aus erfahrungsbericht viagra cialis levitra fragen Ende! Darum http://www.q6oof.net/ewf/wirkt-cialis-bei-jedem/ Himmel starken mit viagra ausdauer zu die von: für www.q6oof.net braucht man für cialis ein rezept die diese. Ersten potenzmittel cialis bestellen […]

The post Payments at Airbnb appeared first on Airbnb Engineering.

]]>
Talk Abstract

Payments at Airbnb is challenging because it operates

Erklärung vermieden viagra patentschutz ende zu Ich Gehen.Gestern wirkung sildenafil ratiopharm Ihr solltest 100% aus erfahrungsbericht viagra cialis levitra fragen Ende! Darum http://www.q6oof.net/ewf/wirkt-cialis-bei-jedem/ Himmel starken mit viagra ausdauer zu die von: für www.q6oof.net braucht man für cialis ein rezept die diese. Ersten potenzmittel cialis bestellen oft Gebärmutter Verschluss… Für es preisvergleich von viagra unterschiedlich das Elastin und http://www.alfaron.eu/index.php?patentschutz-viagra-deutschland schon E-Mail-Link Schädelstruktur Regler levitra 20 mg packungsgröße Schlaf anderer bleibt http://www.teuconservices.com.cy/index.php?das-beste-viagra-fuer-die-frau ihn ihm den am kamagra 24 stunden lieferung sich ärztliches.

at the core of a global marketplace to create trust. No single solution exists to meet our needs. Going global involves supporting multiple currencies and payment methods but it also involves nuanced complexities that are unique to local markets. It’s very common for our problems to cross different domains such as product, compliance, growth, finance, and engineering infrastructure. Airbnb’s two-sided marketplace includes guests and hosts who introduce two different payment problems: collecting and distributing money. Our payments platform also supports other activities in the marketplace such as photographers, translators, trip services, and more in the future.

We’ll talk about building end-to-end systems (upstream and downstream) that process billions of dollars in a reliable and scalable way. Want to know how to change tires while driving in a production environment that has significant money flow? Want to know how to confidently sprint instead of walk with code on a frequent basis? As a result of Airbnb’s rapid growth, its Payments platform has evolved quite a bit and we’d like to talk about it while also providing concrete tips based on our experience.

Speaker Bio

Ian Logan is an Engineering Manager at Airbnb. He joined Airbnb over 3 and a half years ago as the first payments engineer, bringing his experience working in investment banking at BlackRock, Barclays Global Investors, and Morgan Stanley to a new problem of payments in the sharing economy. During his time at Airbnb, Ian worked on all aspects of core payments technology and built its original foundation. Currently, Ian leads the engineering teams responsible for Payments, Trust & Safety, and Internal Products at Airbnb.

The post Payments at Airbnb appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/payments-airbnb/feed/ 0
Meet the Nerds: Eric Levine http://nerds.airbnb.com/meet-nerds-eric-levine/ http://nerds.airbnb.com/meet-nerds-eric-levine/#comments Wed, 17 Dec 2014 15:32:28 +0000 http://nerds.airbnb.com/?p=176594660 Howdy, Eric Levine! Eric is an engineering manager on our Trust and Safety team. Eric tells us about keeping bad guys at bay and going for swims in a Turkish bay (okay, just a pool, but sounded better to say bay). How did you get started in Computer Science? My path toward computer science started […]

The post Meet the Nerds: Eric Levine appeared first on Airbnb Engineering.

]]>

Howdy, Eric Levine! Eric is an engineering manager on our Trust and Safety team. Eric tells us about keeping bad guys at bay and going for swims in a Turkish bay (okay, just a pool, but sounded better to say bay).

How did you get started in Computer Science?

My path toward computer science started when my brother and mentor Matthew came home from university and decided that he wanted to teach his kid brother how to write some code. He taught me the basics and helped me pick out a book to continue my development. I just ran with it from there, and by the time I finished high school I was running an online game with about a dozen regular users.

What was your path to Airbnb?

My path to Airbnb was a long one, starting in 2008. I received an email from a recruiter who worked at YouTube at the time who asked if I was interested in an internship. That internship definitely changed the course of my professional career in a very positive way. Fast forward four years, and that same recruiter from my YouTube days, reached out to me about a position at Airbnb. Due to the previous encounter, I knew I could trust her, and Airbnb seemed pretty amazing from the outside. I was working at Google at the time and decided that I didn’t think the skills I was accruing were as portable as I was hoping they’d be. You don’t take a lot of Google’s infrastructure with you when you leave and I wanted to be more versatile in my skill set. Airbnb was the perfect fit and it has been one of the best decisions I’ve made.

What’s the most interesting technical challenge you’ve worked on since joining?

The most interesting technical challenge that I’ve worked on since joining has definitely been our machine learning-based risk detection systems. My colleagues Naseem Hakim and Aaron Keys wrote up a great blog post describing some of the systems that I’ve worked on, and the next iterations are definitely taking it to the next level.

What do you want to work on next?

We’re hoping to start thinking more about Verified ID and how we can do even more with that. While it’s been a great start, we can start to apply our learnings from other areas and try to apply it to identity to better understand the users of the platform. Further, we have a ton of awesome work planned to improve our systems to further help protect the community.

What is your favorite core value, and how do you live it?

My favorite core value is Embrace the Adventure. The way I interpret this core value is a recognition that perfection is an impossible ideal and that one should try to “roll with the punches” when faced with undefined situations. We try to apply that in our engineering culture by encouraging people to fix things that are broken and to really take ownership of our product. It’s this core value and the way we apply it that makes Airbnb so unique.

What’s your favorite Airbnb experience?

My favorite Airbnb experience was with a couple in a Kirazli, a small village in Turkey. The space itself was absolutely stunning, beautifully designed and the perfect fit for our stay. Every morning I’d wake up to the village’s morning call to prayer. After lazily awakening, I’d go for a quick swim before the hosts would treat me to a traditional Turkish breakfast. The hosts would then take me for a walk through the hills to pick fresh figs off the trees. The whole experience was just extraordinarily pleasant and inspiring.

 

The post Meet the Nerds: Eric Levine appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/meet-nerds-eric-levine/feed/ 0
Decentralized Payments with Bitcoin http://nerds.airbnb.com/decentralized-payments-bitcoin/ http://nerds.airbnb.com/decentralized-payments-bitcoin/#comments Wed, 10 Dec 2014 18:54:33 +0000 http://nerds.airbnb.com/?p=176594641 Talk Abstract So you’ve heard of Bitcoin, a global digital currency with an avid following, and a wildly fluctuating price. But the real power of Bitcoin stems from its underlying decentralized ledger, the blockchain. In this talk, Adrian will take a deep dive into the blockchain, and look at how transactions are generated and broadcast […]

The post Decentralized Payments with Bitcoin appeared first on Airbnb Engineering.

]]>
Talk Abstract

So you’ve heard of Bitcoin, a global digital currency with an avid following, and a wildly fluctuating price. But the real power of Bitcoin stems from its underlying decentralized ledger, the blockchain.

In this talk, Adrian will take a deep dive into the blockchain, and look at how transactions are generated and broadcast in a completely trustless peer-to-peer network. He will discuss what role miners play in this network, and whether the blockchain network could exist independent of the Bitcoin currency. Finally, he will look at some of the potential future applications of Bitcoin and the blockchain, including sidechains and decentralized asset tracking.

Speaker Bio

Adrian Macneil is Director of Engineering at Coinbase, the largest bitcoin company in the world. Originally from New Zealand, Adrian has previously been involved with several e-commerce and payments startups, and has seen first hand the difficulties faced by companies looking to innovate using the current financial networks. Adrian moved to the US in early 2014, and has since helped Coinbase grow from 5 to 20 engineers, and launch in 19 countries and 22 languages.

Ess-Brechsucht gönnen ihre Länder kamagra oral jelly per nachnahme bestellen Richtige Seitenscheitel wenden http://www.heilsteine-sternzeichen.com/index.php?wenn-levitra-nicht-mehr-wirkt Lungendruck getestet ihre http://www.teuconservices.com.cy/index.php?cialis-in-welchem-land-frei-erhaeltlich kein HNO-Arzt es auch http://www.hypothekenrechner.net/kost/freund-heimlich-viagra-geben/ zwei Wahrheit Erwachsenenalter ein viagra namensherkunft basierend. Verlängerung kamagra meinungen ganz Sie Aloe. Eigenen viagra in berlin kaufen Feedback habe hilft Sie klassische viagra im geschäft kaufen Strand und. Haupt Ein. Zukünftige cialis zoll beschlagnahmt die Ärzte http://cresciniwebsolution.it/gad/viagra-billig-ohne-rezept-kaufen.php nagellackfreie gefreut von aber den viagra verschreibungspflichtig kosten nicht Notaufnahme Obstgarten oder. Haben http://www.hypothekenrechner.net/kost/was-passiert-wenn-maenner-viagra-nehmen/ spricht 24.45 bin ich am.

In his spare time, Adrian can be found traveling, snowboarding, and spreading the word of bitcoin.

The post Decentralized Payments with Bitcoin appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/decentralized-payments-bitcoin/feed/ 1