Airbnb Engineering http://nerds.airbnb.com Nerds Wed, 22 Apr 2015 17:43:02 +0000 en-US hourly 1 http://wordpress.org/?v=140 Meet The Nerds: Gary Wu http://nerds.airbnb.com/meet-nerds-gary-wu/ http://nerds.airbnb.com/meet-nerds-gary-wu/#comments Wed, 22 Apr 2015 17:43:02 +0000 http://nerds.airbnb.com/?p=176594938 Today we’re introducing you to Gary Wu, a traveler, family man, who values simplicity above most other things. How did you get started in Computer Science? I was a quiet kid, I loved math and electronics. It is not surprising that I fell in love with computers. As a teenager, I picked up Basic. At […]

The post Meet The Nerds: Gary Wu appeared first on Airbnb Engineering.

]]>
Today we’re introducing you to Gary Wu, a traveler, family man, who values simplicity above most other things.

How did you get started in Computer Science?

I was a quiet kid, I loved math and electronics. It is not surprising that I fell in love with computers. As a teenager, I picked up Basic. At first, it is just curiosity, but soon I liked the ability to create something fun with little investment. Any new tip learned from a book could be applied immediately to my naive programs. Soon, I was showing off my projects to parents and friends, and they gave me feedback on how to make it better. The process of continuous innovation was exciting. After graduation and working for tech companies, I also found that this repeated cycle of learn => build => feedback is not so much different from my own teenage experience. More importantly, the quicker the cycle runs, the faster the products will innovate and people will grow.

What was your path to Airbnb?

I am very interested in the sharing economy and love travel. The Internet and mobile technology has completely revamped the ways that we communicate, shop, and collaborate. I believe the relationship between people and services will have a revolutionary change as well in the near future, and this may have a profound long term impact to the human society. I followed and heard of many impressive Airbnb stories, but I haven’t considered changing job, as I was pretty happy with the previous company . Some friends approached me and shared a lot of internal stories, especially how Airbnb builds up teams based on its core values. This impressed me that Airbnb has a strong vision to completely change how everyone experiences the world. If Airbnb’s vision comes true, the world will be very different from today. This convinced me to have a try and be a part of the journey. I am glad that I made this career decision.

What’s the most interesting technical challenge you’ve worked on since joining?

After joining Airbnb, I focused on prototyping several early stage product ideas with a potential to significantly expand our market in the future. There are two sets of challenges both from the products and the infrastructure.

On the product side, we want to maximize our learning with minimum efforts, and so we develop MVPs (Minimum Viable Product). Building MVPs is easy to say but hard to execute right, because if the product is not appreciated by the customers, it is difficult to tell whether it is because of no enough engineering efforts or because of the wrong idea. Likely, we may over-emphasize the engineering execution instead of reevaluating the ideas.

On the infrastructure side, there may also have many pitfalls. Building MVP may unfortunately introduce technical debt that could be hard to extend or scale in the long run, especially under a complex business flow. Furthermore, unlike other mature technical companies, which have abundant resources and are easily able to handle 5-10X sudden load increase, Airbnb infrastructure is still at an early stage and doesn’t have enough cushion to deal with unusual resource usage pattern. Therefore, we have to be very thoughtful in developing MVPs, eliminate any possible system threats, and try to minimize any long term debt.

In summary, it is really a enjoyable and fast learning process. Airbnb has a plenty of these opportunities because there are so many areas to explore in the traveling space.

What do you want to work on next?

I would like to enable more micro-entrepreneurs to create services on top of Airbnb. We are grateful that many hosts leverage Airbnb to rent out their extra spaces for the travelers, and it is only a beginning of changing how we experience the world. To provide a magic experience for each traveller, there are a lot of opportunities for the local people to participate in, and Airbnb can be the prefect platform for them to contribute to the traveling industry.

What is your favorite core value, and how do you live it?

Simplify. I like a simplified way of thinking and doing things. First, simplification drives me to provide a simplified “interface” to others. When doing a presentation, no matter it is just 5 mins or 30 mins, I push myself to only make one sentence takeaway for the audiences. When discussing a comprehensive system design topic, I tried to summarize my points in a couple of bullet points. When writing programs, I do my best to make the function names and execution flow are so intuitive that other engineers can pick it up easily. Second, simplification makes me stay focused. It is important to be very productive, but to me, it is more important of avoid doing irrelevant things. Finally, simplification can help achieving a better software quality in the long run. I like to challenge myself to avoid unnecessary complexity for marginal improvements, but seek for architecture simplicity for future extension with a potential 10X improvement.

What’s your favorite Airbnb experience?

I spent a few days with my family living an amazing cabin inside the Sequoia national park early this year. The location of the cabin made me so close to the nature. We were surrounded by the great mountain views, hundred years old giant pine trees, and beautiful stony creeks. In addition, we totally lived in a pre-Interenet world, as there was no cable and the closest village to receive cellular signals was one hour driving distance away. We had a lot of fun with hiking, climbing, and photography. In the last day, an unexpected snow storm turned the entire mountain into white, and our cabin in the snow mountain was exactly my childhood fantasy. It was a magic experience that traditional hotel is probably impossible to offer.

The post Meet The Nerds: Gary Wu appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/meet-nerds-gary-wu/feed/ 0
Introducing AirMapView http://nerds.airbnb.com/airmapview/ http://nerds.airbnb.com/airmapview/#comments Mon, 20 Apr 2015 16:35:07 +0000 http://nerds.airbnb.com/?p=176594926 Many Android applications today require some form of an interactive map as part of their user interface. Google provides a native package and experience with Google Play Services to satisfy this need, but the question remains of how one creates interactive maps for devices without Play Services. In some countries, the majority of devices are […]

The post Introducing AirMapView appeared first on Airbnb Engineering.

]]>
Many Android applications today require some form of an interactive map as part of their user interface. Google provides a native package and experience with Google Play Services to satisfy this need, but the question remains of how one creates interactive maps for devices without Play Services.

In some countries, the majority of devices are sold without Google Play Services. Device manufacturers who ship their devices without Play Services are continuing to gain popularity worldwide. In order for our application to provide a truly internationalized experience, we can’t leave out a feature as critical as maps. And because we know other companies have this same issue, we’ve created and open sourced AirMapView.

AirMapView is a view abstraction that enables interactive maps for devices with and without Google Play Services. Devices that do have Google Play Services will use Google Maps V2, while devices without will use a web based implementation of Google Maps. This all comes as one single API that is designed after that of Google Maps V2 that most developers are used to.

AirMapView will choose by default the best map provider available for the device. By default it will use native Google Maps V2 if available and fallback to a WebView solution if Google Play Services are not available. The API is designed to be completely transparent to the user so that developers can use the same APIs that are currently used for Google Maps to gain the fallback functionality.

Native GoogleMap is implemented as a Fragment inside of the AirMapView providing the exact same functionality as using Google Maps V2 directly. Porting existing implementations from GoogleMap to AirMapView is as simple as replacing calls to GoogleMap with calls to AirMapView and implementing the correct callback classes for operations such as OnCameraChanged. The API is designed to be pluggable so developers can add their own providers for specific devices such as Amazon Maps for Amazon kindle fire devices.

The fallback webview map displays a Google Map inside of an Android WebView and uses the javascript bridge callbacks to allow dynamic interaction with the map. Due to it being a webview and not native code it isn’t as performant as the native GoogleMap but it only performed slightly worse in experiments in the Airbnb app.

Using the Javascript Bridge we are able to implement the same API in the web map so no client code changes are required to support the web map once AirMapView has been implemented for native maps.

The web map allows setting a location, centering, adding markers, dragging, tapping on the map and other common operations that are currently supported in the GoogleMap.

We’ve built AirMapView in such a way that allows us to easily add additional map providers in the future, such as Amazon Maps V2, Baidu, Mapbox, etc.

For more information take a look at our Github page here: https://github.com/airbnb/airmapview

The post Introducing AirMapView appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/airmapview/feed/ 0
Android Frameworks from Airbnb and Square http://nerds.airbnb.com/android-frameworks-from-airbnb-and-square/ http://nerds.airbnb.com/android-frameworks-from-airbnb-and-square/#comments Thu, 16 Apr 2015 19:09:01 +0000 http://nerds.airbnb.com/?p=176594918 We wrapped up the month of March with with the theme of android development. It was exciting to host Pierre-Yves Ricau from Square and feature Airbnb engineers, Eric Petzel and Nick Adams. Mapstraction Many mobile applications today require some form of an interactive map as part of their user interface. Google provides a native package […]

The post Android Frameworks from Airbnb and Square appeared first on Airbnb Engineering.

]]>
We wrapped up the month of March with with the theme of android development. It was exciting to host Pierre-Yves Ricau from Square and feature Airbnb engineers, Eric Petzel and Nick Adams.

Mapstraction

Many mobile applications today require some form of an interactive map as part of their user interface. Google provides a native package and experience with Google Play Services to satisfy this need, but the questions remains of how one creates interactive maps for devices without Play Services.

In some countries, the majority of devices are sold without Google Play Services. Device manufacturers who ship their devices without Play Services are continuing to gain popularity worldwide. We have built a package that solves these problems, letting developers remain agnostic of manufacturer while providing a consistent map experience to users.

Speaker Bio

Eric Petzel: Software Engineer on the Android team at Airbnb where he focuses on building features for our hosts and guests, as well as tools to share with the Android community. Previously to Airbnb, he worked at Skype on their Android client

Nick Adams: Software Engineer on the Android team at Airbnb. He focuses on building features that improve the quality of the app and expanding to new form factors. Before Airbnb he was a student at the University of British Columbia in Canada.

Crash Fast: Square’s approach to Android crashes

The Square Register Android app has few crashes. Getting there requires a systematic approach: coding defensively, gathering information, measuring impact and improving architecture.

This talk presents our concrete steps towards lowering the crash rate, from the general philosophy to the tools we use, together with real crash examples.

Speaker Bio

Pierre-Yves Ricau: Android baker at Square. I started having fun with Java & Android as a consultant in Paris, then joined a startup in Barcelona and finally joined Square in San Francisco to work with some of the best engineers in the world. I like good wine & low entropy code.

The post Android Frameworks from Airbnb and Square appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/android-frameworks-from-airbnb-and-square/feed/ 0
How Airbnb uses machine learning to detect host preferences http://nerds.airbnb.com/host-preferences/ http://nerds.airbnb.com/host-preferences/#comments Tue, 14 Apr 2015 09:15:45 +0000 http://nerds.airbnb.com/?p=176594906 At Airbnb we seek to match people who are looking for accommodation – guests — with those looking to rent out their place – hosts. Guests reach out to hosts whose listings they wish to stay in, however a match succeeds only if the host also wants to accommodate the guest. I first heard about […]

The post How Airbnb uses machine learning to detect host preferences appeared first on Airbnb Engineering.

]]>
At Airbnb we seek to match people who are looking for accommodation – guests — with those looking to rent out their place – hosts. Guests reach out to hosts whose listings they wish to stay in, however a match succeeds only if the host also wants to accommodate the guest.

I first heard about Airbnb in 2012 from a friend. He offered his nice apartment on the site when he traveled to see his family during our vacations from grad school. His main goal was to fit as many booked nights as possible into the 1-2 weeks when he was away. My friend would accept or reject requests depending on whether or not the request would help him to maximize his occupancy.

About two years later, I joined Airbnb as a Data Scientist. I remembered my friend’s behavior and was curious to discover what affects hosts’ decisions to accept accommodation requests and how Airbnb could increase acceptances and matches on the platform.

What started as a small research project resulted in the development of a machine learning model that learns our hosts’ preferences for accommodation requests based on their past behavior. For each search query that a guest enters on Airbnb’s search engine, our model computes the likelihood that relevant hosts will want to accommodate the guest’s request. Then, we surface likely matches more prominently in the search results. In our A/B testing the model showed about a 3.75% increase in booking conversion, resulting in many more matches on Airbnb. In this blog post I outline the process that brought us to this model.

What affects hosts’ acceptance decisions?

I kicked off my research into hosts’ acceptances by checking if other hosts maximized their occupancy like my friend. Every accommodation request falls in a sequence or in a window of available days in the calendar, such as on April 5-10 in the calendar shown below. The gray days surrounding the window are either blocked by the host or already booked. If accepted and booked, a request may leave the host with a sub-window before the check-in date (check-in gap — April 5-7) and/or a sub-window after the check-out (check-out gap — April 10).

A host looking to have a high occupancy will try to avoid such gaps. Indeed, when I plotted hosts’ tendency to accept over the sum of the check-in gap and the check-out gap (3+1= 4 in the example above), as in the next plot, I found the effect that I expected to see: hosts were more likely to accept requests that fit well in their calendar and minimize gap days.

But do all hosts try to maximize occupancy and prefer stays with short gaps? Perhaps some hosts are not interested in maximizing their occupancy and would rather host occasionally. And maybe hosts in big markets, like my friend, are different from hosts in smaller markets.

Indeed, when I looked at listings from big and small markets separately, I found that they behaved quite differently. Hosts in big markets care a lot about their occupancy — a request with no gaps is almost 6% likelier to be accepted than one with 7 gap nights. For small markets I found the opposite effect; hosts prefer to have a small number of nights between requests. So, hosts in different markets have different preferences, but it seems likely that even within a market hosts may prefer different stays.

A similar story revealed itself when I looked at hosts’ tendency to accept based on other characteristics of the accommodation request. For example, on average Airbnb hosts prefer accommodation requests that are at least a week in advance over last minute requests. But perhaps some hosts prefer short notice?

The plot below looks at the dispersion of hosts’ preferences for last minute stays (less than 7 days) versus far in advance stays (more than 7 days). Indeed, the dispersion in preferences reveals that some hosts like last minute stays better than far in advance stays — those in the bottom right — even though on average hosts prefer longer notice. I found similar dispersion in hosts’ tendency to accept other trip characteristics like the number of guests, whether it is a weekend trip etc.

All these findings pointed to the same conclusion: if we could promote in our search results hosts who would be more likely to accept an accommodation request resulting from that search query, we would expect to see happier guests and hosts and more matches that turned into fun vacations (or productive business trips).

In other words, we could personalize our search results, but not in the way you might expect. Typically personalized search results promote results that would fit the unique preferences of the searcher — the guest. At a two-sided marketplace like Airbnb, we also wanted to personalize search by the preference of the hosts whose listings would appear in the search results.

How to model host preferences?

Encouraged by my findings, I joined forces with another data scientist and a software engineer to create a personalized search signal. We set out to associate hosts’ prior acceptance and decline decisions by the following characteristics of the trip: check-in date, check-out date and number of guests. By adding host preferences to our existing ranking model capturing guest preferences, we hoped to enable more and better matches.

At first glance, this seems like a perfect case for collaborative filtering – we have users (hosts) and items (trips) and we want to understand the preference for those items by combining historical ratings (accept/decline) with statistical learning from similar hosts. However, the application does not fully fit in the collaborative filtering framework for two reasons.

  • First, no two trips are ever identical because behind each accommodation request there is a different guest with a unique human interaction that influences the host’s acceptance decision. This results in accept/decline labels that are noisier than, for example, the ratings of a movie or a song like in many collaborative filtering applications.
  • Taking this point one step further, a host can receive multiple accommodation requests for the same trip with different guests at different points in time and give those requests conflicting votes. A host may accept last minute stays that start on a Tuesday 2 out of 4 times, and it remains unclear whether the host prefers such stays.

With these points in mind, we decided to massage the problem into something resembling collaborative filtering. We used the multiplicity of responses for the same trip to reduce the noise coming from the latent factors in the guest-host interaction. To do so, we considered hosts’ average response to a certain trip characteristic in isolation. Instead of looking at the combination of trip length, size of guest party, size of calendar gap and so on, we looked at each of these trip characteristics by itself.

With this coarser structure of preferences we were able to resolve some of the noise in our data as well as the potentially conflicting labels for the same trip. We used the mean acceptance rate for each trip characteristic as a proxy for preference. Still our data-set was relatively sparse. On average, for each trip characteristic we could not determine the preference for about 26% of hosts, because they never received an accommodation request that met those trip characteristics. As a method of imputation, we smoothed the preference using a weight function that, for each trip characteristic, averages the median preference of hosts in the region with the host’s preference. The weight on the median preference is 1 when the host has no data points and goes to 0 monotonically the more data points the host has.

Using these newly defined preferences we created predictions for host acceptances using a L-2 regularized logistic regression. Essentially, we combine the preferences for different trip characteristics into a single prediction for the probability of acceptance. The weight the preference of each trip characteristic has on the acceptance decision is the coefficient that comes out of the logistic regression. To improve the prediction, we include a few more geographic and host specific features in the logistic regression.

This flow chart summarizes the modeling technique.

We ran this model on segments of hosts on our cluster using a user-generated-function (UDF) on Hive. The UDF is written in Python; its inputs are accommodation requests, hosts’ response to them and a few other host features. Depending on the flag passed to it, the UDF either builds the preferences for the different trip characteristics or trains the logistic regression model using scikit-learn.

Our main off-line evaluation metric for the model was mean squared error (MSE), which is more appropriate in a setting when we care about the predicted probability more than about classification. In our off-line evaluation of the model we were able to get a 10% decrease in MSE over our previous model that captured host acceptance probability. This was a promising result. But, we still had to test the performance of the model live on our site.

Experimenting with the model

To test the online performance of the model, we launched an experiment that used the predicted probability of host acceptance as a significant weight in our ranking algorithm that also includes many other features that capture guests’ preferences. Every time a guest in the treatment group entered a search query, our model predicted the probability of acceptance for all relevant hosts and influenced the order in which listings were presented to the guest, ranking likelier matches higher.

We evaluated the experiment by looking at multiple metrics, but the most important one was the likelihood that a guest requesting accommodation would get a booking (booking conversion). We found a 3.75% lift in our booking conversion and a significant increase in the number of successful matches between guests and hosts.

After concluding the initial experiment, we made a few more optimizations that improved conversion by approximately another 1% and then launched the experiment to 100% of users. This was an exciting outcome for our first full-fledged personalization search signal and a sizable contributor to our success.

Conclusions

First, this project taught us that in a two sided marketplace personalization can be effective on the buyer as well as the seller side.

Second, the project taught us that sometimes you have to roll up your sleeves and build a machine learning model tailored for your own application. In this case, the application did not quite fit in the collaborative filtering and a multilevel model with host fixed-effect was too computationally demanding and not suited for a sparse data-set. While building our own model took more time, it was a fun learning experience.

Finally, this project would not have succeeded without the fantastic work of Spencer de Mars and Lukasz Dziurzynski.

The post How Airbnb uses machine learning to detect host preferences appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/host-preferences/feed/ 10
Overcoming Missing Values In A Random Forest Classifier http://nerds.airbnb.com/overcoming-missing-values-in-a-rfc/ http://nerds.airbnb.com/overcoming-missing-values-in-a-rfc/#comments Tue, 07 Apr 2015 18:02:05 +0000 http://nerds.airbnb.com/?p=176594836 No Strangers Airbnb is trying to build a world where people can belong anywhere and there are no strangers. This helps hosts feel comfortable opening their homes and guests be confident traveling around the globe to stay with people they have never met before. While almost all members of the Airbnb community interact in good […]

The post Overcoming Missing Values In A Random Forest Classifier appeared first on Airbnb Engineering.

]]>
No Strangers

Airbnb is trying to build a world where people can belong anywhere and there are no strangers. This helps hosts feel comfortable opening their homes and guests be confident traveling around the globe to stay with people they have never met before.

While almost all members of the Airbnb community interact in good faith, there is an ever shrinking group of bad actors that seek to take advantage of the platform for profit. This problem is not unique to Airbnb: social networks battle with attempts to spam or phish users for their details; ecommerce sites try to prevent the use of stolen credit cards. The Trust and Safety team at Airbnb works tirelessly to remove bad actors from the Airbnb community and to help make the platform a safer and trustworthy place to experience belonging.

Missing Values In A Random Forest

We can train machine learning models to identify new bad actors (for more details see the previous blog post Architecting a Machine Learning System for Risk). One particular family of models we use is Random Forest Classifiers (RFCs). A RFC is a collection of trees, each independently grown using labeled and complete input training data. By complete we explicitly mean that there are no missing values i.e. NULL or NaN values. But in practice the data often can have (many) missing values. In particular, very predictive features do not always have values available so they must be imputed before a random forest can be trained.

Typically, random forest methods/packages encourage two ways of handling missing values: a) drop data points with missing values (not recommended); b) fill in missing values with the median (for numerical values) or mode (for categorical values). While a) does not use all the available information by dropping data points, b) can sometimes brush too broad a stroke for data sets with many gaps and significant structure.

There are alternative techniques for dealing with missing values, but most of these are computationally expensive e.g. repeated iteration of random forest training to compute proximities. What we propose in this post is a one-step pre-computation method which normalises features to construct a distance metric for filling in missing values with the median of their k-nearest neighbors.

All of the features in our fraud prediction models fall into two types: a) numerical and b) categorical. Boolean features can be thought of as a special case of categorical features. Since we work in the business of fraud detection, our labels are binary: 0 if the data point is not fraud and 1 if the data point is fraud. Below are some feature transformations we wish to compare for missing value treatment.

Transformations

  1. Lets first deal with numerical features. Write a numerical feature as a random variable \(X\) taking values in \(\mathbb{R}\). And denote the data labels by the random variable \(Y\) taking values in \(\{0,1\}\). Then for \(X=x\) the transformation \(F_n:\mathbb{R} \rightarrow [0,1]\) defined by$$
    x \mapsto F_n(x) := \mathbb{P}[X \leq x | Y=1]
    $$has the following properties:

    1. \(F_n\) is order preserving i.e. if \(a \leq b\) then \(F(a) \leq F(b)\)
    2. \(F_n\) is invertible i.e. given a value for \(y=F(x)\) we can find \(x=F^{-1}(y)\) (follows from i)
    3. \(F_n(X|{Y=1}) \sim U[0,1]\) i.e. \(F_n(X|{Y=1})\) is uniformly distributed on \([0,1]\).

     

  2. Similarly, consider a categorical feature represented by random variable \(X\) taking values in finite set \(\Omega = \{a,b,…\}\). Take ordering \(\preceq\) on \(\Omega\) according to the rate of fraud for each categorical value. Then for categorical \(X=x\) we can define the transformation \(F_c:\Omega \rightarrow [0,1]\) equivalently by$$
    x \mapsto F_c(x) := \mathbb{P}[X \preceq x | Y=1]
    $$which also has properties i), ii) and iii) from above with respect to the ordering \(\preceq\).
  3. A more common method for transforming a numerical feature \(X\) to the unit interval is given by \(G_n\) where$$
    x \mapsto G_n(x) := \mathbb{P}[X \leq x]
    $$which would also have properties i), ii) and iii) in the above.
  4. We can adapt the popular method \(g(x) = \mathbb{P}[Y=1 | X=x]\) for transforming a categorical feature \(X\) to a numerical value to give us \(G_c:\Omega \rightarrow [0,1]\) defined by$$
    x \mapsto G_c(x) := [ g(x) - \min g(x) ] / [\max g(x) - \min g(x)]
    $$The conditional probability transform \(g\) intuitively makes more sense, but violates properties i), ii), and iii) above. And without invertibility of \(G_c\) we cannot in the RFC easily (or practically) distinguish between two different categorical values \(a\) and \(b\) that have \(G_c(a) \approx G_c(b)\).

Interpretation

The aim of the scaling transforms \(F_n\) and \(F_c\) is two fold. First of all to make the transformation invertible so no information is lost. Secondly, to uniformly distribute the data points with fraud in the interval [0,1] for each feature. If we think of data as points in \(N\) dimensional space where \(N\) is the number of features, then the distance in each dimension between two data points becomes comparable. By comparable we mean that a distance of 0.4 in the the first dimension contains twice as many fraud data points as a distance of 0.2 in the second dimension. This enables better construction of distance metrics to identify fraud.

Imputation Using K-Nearest Neigbours

Suppose there are \(N\) features \(X_1,X_2,…,X_N\) and data points \(\mathbf{a},\mathbf{b},…\in \mathbf{D}\). After we have transformed the \(i\)th feature, by \(F_i\) say, for \(i=1,2,…,N\) we can construct a distance metric \(d\) between any two data points \(\mathbf{a}=(a_1,a_2,…,a_N)\) and \(\mathbf{b}=(b_1,b_2,…,b_N)\) as follows:

$$
d(\mathbf{a},\mathbf{b};\lambda) := \sum_{i=1}^{N} \lambda + \mathbf{1}_{a_i \neq NULL,b_i \neq NULL}(|F_i(a_i)-F_i(b_i)| – \lambda)
$$

where \(\lambda\) is a pre-chosen constant which determines how to weight the distance between two feature values when at least one of them is missing. By increasing \(\lambda\) we push out neighbours of a data point that have many missing values. Then, for a data point \(\mathbf{a}\) with missing value \(a_j\), say, we calculate the nearest neighbours missing value (NNMV) as:

$$
m(a_j;d,k) = \text{median} ( [\text{argmin}^{(k)}_{\mathbf{b} \in \mathbf{D}} d(\mathbf{a},\mathbf{b};\lambda)]_j )
$$

where \(k\) denotes how many nearest neigbhours we wish to use for calculating the median value of the \(j\)th feature. In other words, find the \(k\) closest neighbours and then in the \(j\)th dimension take the median of the \(k\) values.

Experiment

In order to see the effect of the above feature transforms, we use the adult dataset from the UCI Machine Learning Repository and assess the performance of the model under different feature transformations and proportions of missing values. The dataset containts 32,561 rows and 14 features, of which 8 are categorical and the remaining 4 are numerical. The boolean labels correspond to whether the income level of the adult is greater than or less than $50k per annum. We divide the dataset into a training and test set in the ratio of 4 to 1 respectively.

For the first experiment we compare different models using the methodology:

  1. Use the same training and test data set split
  2. Remove \(M\)% of values from the training and test data sets
  3. Either calculate the median/mode for the missing values or use \(m(.;d(.,.;\lambda),100)\) from the training set
  4. Fill in the missing values in both training and test set using the previous step’s calculations
  5. Train with the same number of trees (100)

Observe that \(M\) and \(\lambda\) are unspecified parameters – we will loop over different values of these during experimentation. The Performance of each model will be judged using Area Under Curve (AUC) scores which measures the area under the Reciever Operating Characteristic (ROC) graph (this is a plot of the true postive rate vs the false postitive rate). We will test the following nine models:

  1. Baseline model (no feature transformation and missing value imputation using median/mode)
  2. \(G_n\) and \(G_c\) with median imputation
  3. \(F_n\) and \(F_c\) with median imputation
  4. \(G_n\) and \(G_c\) with NNMV imputation
  5. \(G_n\) and \(F_c\) with NNMV imputation
  6. \(F_n\) and \(G_c\) with NNMV imputation
  7. \(O_n\) and \(F_c\) with NNMV imputation
  8. \(F_n\) and \(O_c\) with NNMV imputation
  9. \(F_n\) and \(F_c\) with NNMV imputation

where \(O_n(x) = O_c(x) := 0\), i.e. is the null map that takes all of a feature’s values to zero. The \(O_n\) and \(O_c\) maps help us verify that both the numerical and categorical features are contributing to the model.

In the second experiment we compare model 8) to model 0) for different combinations of number of trees and number of nearest neighbours.

Results

Models Comparison

First we consider how the models perform for different values of \(M\) and \(\lambda\) with a fixed number of trees (100) in the RFC training process and fixed number of nearest neighbours (100) for NNMV imputation.

rfc_mis_vals_cdf_transforms_13_0

The graphs above display interesting patterns, some intuitive and some surprising:

  1. As we increase the percentage of missing values, the performance of the baseline model deterioriates
  2. In the scenario where no missing values are added (top left graph) but the data set still has some missing values, the improvement from all models is roughly the same at 0.007
  3. For all scenarios, the models with median imputation (red and yellow lines) perform very similarly to each other and sometimes over 0.005 worse than the NNMV models
  4. Across all scenarios the ‘\(F_n\) and \(F_c\) + NNMV’ graph (light brown line) outperforms the median imputation models, with the improvement increasing as the percentage of missing values increases
  5. Across all scenarios the ‘\(F_n\) and \(F_c\) + NNMV’ graph (light brown line) outperforms the other NNMV graphs, by upto 0.003 in some cases (bottom right graph)
  6. Across all scenarios the \(O_n\) and \(O_c\) graphs (light blue and pink lines) reduce the power of the model considerably – as we would expect – so much so that they are outside of the scope of the graphs
  7. The results do not appear to be very sensitive to \(\lambda\) as long as \(\lambda>0\).

Robustness Checks

Having observed the outperformance of model 8) over the other candidates, we next check how model 8) compares to the baseline as we vary i) the number of trees in the RFC training and ii) the number of nearest neigbours in the NNMV imputation. We take the fourth scenario above – where 60% of values are missing – and we chose \(\lambda\)=0.5.

rfc_mis_vals_cdf_transforms_16_1

The left hand plot does not suggest the performance of model 8) improves with the number of nearest neighbours used in the NNMV imputation. However, there is a consistent pattern of improved performance as the number of trees increases, plateauing after about 100 trees. The right hand plot shows how much faster it is to train a RFC with the transformed feature set. This is to be expected as, instead of exploding categorical features to many binary features in the baseline model, we keep the number of features fixed in model 8).

Efficiency Implications

Consider the ROC curves for one of the scenarios above, say, where the number of trees is 100 and the number of nearest neighbours used is 100.

rfc_mis_vals_cdf_transforms_19_0

The improvement of the ROC curve suggests, for example, that holding recall fixed at 80%, say, the false positive rate falls from 26% to 24%. Suppose each day we are scoring 1 million events, 99% of which are non-fraud, each flagged event needs to be manually reviewed by a human, and each review takes 10 seconds. Then the aforementioned decrease in the false positive rate can save reviewing 1,000,000 x 0.99 x 0.02 = 19,800 events or 19,800 / (6 x 60) = 55 hours of reviewing per day! This is why even single digit or decimal digit improvements in the auc score of a RFC can have a dramatic effect on a department’s efficiency.

The post Overcoming Missing Values In A Random Forest Classifier appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/overcoming-missing-values-in-a-rfc/feed/ 0
Meet The Nerds: Barbara Raitz http://nerds.airbnb.com/meet-nerds-barbara-raitz/ http://nerds.airbnb.com/meet-nerds-barbara-raitz/#comments Fri, 03 Apr 2015 16:31:41 +0000 http://nerds.airbnb.com/?p=176594891 Meet Barbara Raitz! In today’s Q&A Barbara tells us about storytelling through code and how to be a magician. How did you get started in Computer Science? I grew up around computers and initially pursued a Computer Science degree for practical reasons. After working a few years in the industry, I discovered that I absolutely […]

The post Meet The Nerds: Barbara Raitz appeared first on Airbnb Engineering.

]]>
Meet Barbara Raitz! In today’s Q&A Barbara tells us about storytelling through code and how to be a magician.

How did you get started in Computer Science?

I grew up around computers and initially pursued a Computer Science degree for practical reasons. After working a few years in the industry, I discovered that I absolutely love it! There is beauty in well crafted code and elegant architectures. I love diving in deep, trying to understand the `story` of complex code, and then providing cleaner, simpler, and more powerful solutions.

If I could say anything to those considering this career path, I want to shout out that it is an amazing field! Though it can be initially daunting, there is such a rich, vast, nearly limitless set of opportunities with this skill set. I love quoting a colleague who said: “Do you want to watch the magic show, or do you want to become the magician?” I feel so lucky to have found a home where I can have such impact, while having so much fun.

What was your path to Airbnb?

I first heard of Airbnb on a podcast from The Commonwealth Club. I very quickly identified with the whole premise of Airbnb, but didn’t consider it much further until a recruiter reached out and contacted me. Because of that initial spark, I decided to come visit the office and meet the team, and was undeniably affected by the positive energy and excitement of the place! Even though it was an introductory visit, I knew I was changed, and every subsequent visit re-enforced that.

I joined Airbnb because it is a technical playground with many opportunities to dig in and make a difference; because the amazing people and culture absolutely blew me away; and because I felt a strong connection to the Airbnb mission. I am particularly drawn to the concept of unique, authentic, local travel that benefits the community, and, that in this ever virtual world, we are bringing people together again.

What’s the most interesting technical challenge you’ve worked on since joining?

I would like to share two!

Initially, I paired up with Spike Brehm to work on Rendr, a library that can re-use and render javascript code on either the client or server: original, powerful, fast — isomorphic javascript! This was a fascinating, ground-breaking, and richly challenging and rewarding project, and I am proud of our results!

From there, I moved to a completely different project that needed me most. I have spent much of my efforts diving deep into tangled legacy code and gradually moving it towards a cleaner, tested, decoupled service-oriented architecture. This process of refactoring mission critical code and data structures while still “in-flight” requires absolute attention to detail, small steps, and patience. Specific to the Calendar, it is rewarding to see powerful new features, such as seasonal availability rules, shine through.

What do you want to work on next?

Consistent with the mission of Simplify, there are other product areas that would greatly benefit from a deep-dive and refactor. Each has it’s own challenges, and is thus richly interesting. I will likely continue supporting opportunities that deliver the biggest impact in terms of decoupling code and processes, increasing stability and maintainability, and delivering powerful new capabilities and features.

In terms of learning something new, I have my eye on React.js — I instinctively like it and want to try it out myself sometime. I have also wanted to explore and experience writing a mobile client application. Maybe in my free time =)

What is your favorite core value, and how do you live it?


I am drawn to all the core values, and absolutely love that our company truly promotes and lives them! Though hard to choose a favorite, I suppose that I best embody “Champion the Mission”. In terms of my technical role, I passionately pursue real architecture changes that I feel will have the most impact to our fast-growing team, product, and company. And then I follow through with careful, persistent, patient hard work. In terms of something bigger, I truly love how Airbnb promotes “belonging anywhere”, in being gracious and welcoming hosts, and being your individual unique self. I love how Airbnb is beneficial to local communities and real individuals. I love how this form of travel encourages people to go outside their normal bounds to perhaps discover something new. These actually map to other core values, and I promote and champion them as well!

What’s your favorite Airbnb experience?

For New Year’s 2013, I wanted to go somewhere special with my boyfriend and we landed here, the experience was absolutely amazing! The space is truly beautiful, from the stunning landscaping and views, to the fabulous and uniquely crafted house itself. For us, it was magical. It was beautiful, relaxing, and creatively stimulating — it allowed us to both relax and go into hyper-creative mode, imagining all sorts of creative, crazy, inspiring futures — colorful ideas that are still with us today. But as any well-travelled person could tell you, the most meaningful memories are tied to the people and kindred spirits you meet along the way. Jeanne, our host, is one of the most gracious, welcoming people I’ve met. She went out of her way to make our stay uniquely special, arranging for a special boat on New Years eve to go to the center of the lake to watch ALL the fireworks. But truly, she opened her heart to us, and we shared real life stories, goals, and aspirations. It’s rare and beautiful when, unexpectedly, your life and perspective changes because you went somewhere or crossed paths and made friends with someone new. This was one of those memorable experiences.

Side note, this listing won Airbnb’s Most Unique Listing award in 2014.

The post Meet The Nerds: Barbara Raitz appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/meet-nerds-barbara-raitz/feed/ 0
Meet the Nerds: Phillippe Siclait http://nerds.airbnb.com/meet-nerds-phillippe-siclait/ http://nerds.airbnb.com/meet-nerds-phillippe-siclait/#comments Thu, 19 Mar 2015 19:36:02 +0000 http://nerds.airbnb.com/?p=176594831 In this Q&A we meet Phillippe Siclait. Phillippe has been with Airbnb for over 2 years and has worked on five teams since joining – clearly he likes to travel both personally and professionally! How did you get started in Computer Science? I started programming to make video games. When I was in elementary school, […]

The post Meet the Nerds: Phillippe Siclait appeared first on Airbnb Engineering.

]]>
In this Q&A we meet Phillippe Siclait. Phillippe has been with Airbnb for over 2 years and has worked on five teams since joining – clearly he likes to travel both personally and professionally!

How did you get started in Computer Science?
I started programming to make video games. When I was in elementary school, I found a book at a school book fair called “Learn to Program Basic”. I had the vague sense that programming was a thing you needed to do to make games, so I picked it up, went through all the exercises, and got hooked. In the proceeding years, I continued to learn from all the resources I could find online. This led to doing programming competitions through my school and the development of an interest in graphics programming. By the end of high school I was pretty set on studying either CS or Economics.

What was your path to Airbnb?
It turns out that I didn’t actually end up majoring in CS. I received a BS in Economics mostly focused on game theory and econometrics while simultaneously working in the Computer Graphics group in the Computer Science and Artificial Intelligence Lab. I decided prior to my final year of school that management consulting would help me learn how to run a business, and after an internship at the Boston Consulting Group, I accepted a full-time offer there. I learned a lot at BCG and got to do a fair amount of travel, both for work and for fun, but in the end realized that I wanted to come back to the technical side. With a couple friends, I packed my bags and moved across the country to explore what the Bay had to offer. After a few months of working independently on mobile and web projects, a friend of mine brought me over to Airbnb for a Tech Talk. It was after meeting many people, hearing about the vision of the company, and thinking about how much I wanted to see this idea spread, that I knew that I had to be here.

What’s the most interesting technical challenge you’ve worked on since joining?
Since joining Airbnb I’ve worked on many teams. I started working on Search (frontend, ranking, infrastructure, evaluation tools, and more), worked a bit on web security, and then worked on our (at the time newly formed) Discovery team. One of the problems for Discovery is recommending places and listings to our guests. I found it fascinating to think through how you could determine which locations an individual would likely be interested in and I worked with my teammates to implement the data pipelines and serving architecture for providing the recommendations to many parts of our product. It was a complex problem partly because the decisions we make while traveling are very personal and multi-faceted.

What do you want to work on next?
I’m currently working on a team that focuses on engaging people to host on Airbnb. We’re an incredibly experiment driven team and we have plans to change many parts of the product to make starting to host easier. We have a small, multidisciplinary team of designers, data scientists, a product manager and engineers who are all focused on this problem. I’m excited to see us increase the rate at which we are testing new product changes.

What is your favorite core value, and how do you live it?
Simplify. I try to simplify any code I write. Code is meant to be read by people and as a result, the simplest code is often the best code. And all of us at Airbnb live it in the product we develop. We are building a product for our guests and hosts, and a simple product leads to a better end experience.

What’s your favorite Airbnb experience?
My favorite Airbnb experience may have actually been over a one night stay in Manila earlier this year. The host had a beautiful house with wonderful, vibrant wood throughout, and he was hosting other guests who were visiting from the UK. He invited us out to dinner that night and it was fascinating to get to know these people who had lives so different from my own. Our host was an American expat who had long been in the Philippines and the other guests were retirees who traveled the world, exploring new places and teaching motorcycle racing. The dinner was wonderful and we spent several hours afterwards chatting in his kitchen about our lives and what we had all seen of the world. In the morning our host organized our transportation back to the airport and made sure we knew that we were welcome the next time we were back in Manila.

The post Meet the Nerds: Phillippe Siclait appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/meet-nerds-phillippe-siclait/feed/ 0
CSS Frameworks and The Evolution of Airbnb’s Frontend http://nerds.airbnb.com/css-frameworks-evolution-airbnbs-frontend/ http://nerds.airbnb.com/css-frameworks-evolution-airbnbs-frontend/#comments Tue, 10 Mar 2015 23:50:54 +0000 http://nerds.airbnb.com/?p=176594802 The theme for the month of february was front-end engineering. Engineers, Spike Brehm and Fiona Tay spoke on behalf of Airbnb and we welcomed Jessica Dillon from Bugsnag. The Evolution of Airbnb’s Frontend Over the last few years, Airbnb’s frontend architecture has evolved to keep pace with the rapid advancement happening the JavaScript world. Starting […]

The post CSS Frameworks and The Evolution of Airbnb’s Frontend appeared first on Airbnb Engineering.

]]>
The theme for the month of february was front-end engineering. Engineers, Spike Brehm and Fiona Tay spoke on behalf of Airbnb and we welcomed Jessica Dillon from Bugsnag.

The Evolution of Airbnb’s Frontend

Over the last few years, Airbnb’s frontend architecture has evolved to keep pace with the rapid advancement happening the JavaScript world. Starting as a humble Rails 2 + Prototype.js app in 2008, the frontend stack powering airbnb.com has gone through a few revisions, including a push towards single-page app architecture with Backbone.js and Handlebars.js, an adventure into isomorphic JavaScript with Rendr (our library for using Node.js to server-render Backbone SPAs), and most recently, a move toward React.js and a re-envisioning of our build pipeline to take advantage of CommonJS, ES6, and a Node.js-based transform system. Spike Brehm, software engineer on the @AirbnbNerds team, will walk through how we approached and executed on these changes. Plus, get excited to see a preview of our new approach to isomorphic JavaScript, allowing us to server-render React components from our Rails app.

Speaker Bio

Spike Brehm is a software engineer at Airbnb who specializes in building rich web experiences. As a JavaScript nerd, he has spent the last few years shipping web apps and prototyping Airbnb’s front-end stack, experimenting with “isomorphic JavaScript” — apps that have the flexibility to run on both the client and sever using the same codebase.

O2: The Life of A CSS Framework

In 2012, a small startup added Bootstrap to our site as a stopgap to make things look better. Never could we have expected that in just 3 years, Airbnb would explode and our design would evolve dramatically. But what’s stayed constant, helping to keep our code sane, is our internal CSS framework. In this talk, I’ll discuss when and why we forked Bootstrap and how maintaining our own system enables beautiful, on-brand design. I’ll also discuss how we’ve rolled out drastic changes for a rebrand and later, a responsive overhaul of a desktop site, using versioning best practices.

Speaker Bio

Fiona Tay is an engineer on the Core Web team at Airbnb, where she develops strategy and build tooling for the web platform. She’s architected engineering strategy for our rebrand across web and email. She’s currently working on making airbnb.com responsive. She’s a fan of design, performance and testing.

Implementing a visual CSS testing framework

Working with large CSS codebases can be hard. Large-scale refactors, or even just tweaking styles on our more general elements, could end up having unintended consequences on the rest of the site. To catch these problems we would manually check every page on our site, which is a slow and error-prone approach. We needed a better way to test our CSS.

We looked up various ways to test CSS, including trying libraries like Huxley. Although some of it was what we needed, the frameworks ultimately didn’t end up integrating well enough for what we wanted to do. After looking at the minimal amount of support each framework was adding, we decided it would be best to rollout our own visual diffing system for our specific needs. We decided to outline a plan that included the components we’d need, and figure out from there the right tools for the job.

In this talk, I’ll walk through how we at Bugsnag implemented a visual CSS testing framework using RSpec & Selenium, using automatic screenshot comparison to catch style regressions.

Speaker Bio

Jessica Dillon is a self taught full-stack software engineer in the heart of San Francisco. She works at Bugsnag, the leading real-time crash detection service for mobile and web applications. Her expertise runs the gamut from developing rich in-browser applications to building service-oriented backends. When not coding or playing games, she likes to hang out with her dog named Boo.

The post CSS Frameworks and The Evolution of Airbnb’s Frontend appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/css-frameworks-evolution-airbnbs-frontend/feed/ 0
Introducing Airpal http://nerds.airbnb.com/airpal/ http://nerds.airbnb.com/airpal/#comments Thu, 05 Mar 2015 15:58:45 +0000 http://nerds.airbnb.com/?p=176594775 Airbnb is pleased to announce the launch of Airpal, a web-based query execution tool that leverages Facebook’s PrestoDB to facilitate data analysis. People who spend time using SQL for exploration and investigation know that the workflow is not always smooth. Remembering how a query was written, copying and pasting from the command line, and running […]

The post Introducing Airpal appeared first on Airbnb Engineering.

]]>
Airbnb is pleased to announce the launch of Airpal, a web-based query execution tool that leverages Facebook’s PrestoDB to facilitate data analysis.

People who spend time using SQL for exploration and investigation know that the workflow is not always smooth. Remembering how a query was written, copying and pasting from the command line, and running multiple terminal windows can slow down analysis and be frustrating. Additionally, when diverse teams are using SQL for analytics, the learning curve can be steep for beginners, so good UI tools can help drive adoption and promote knowledge sharing.

At Airbnb, we launched Airpal internally about a year ago and now more than 1/3 of all employees have issued a query through the tool. This is an astounding statistic and it shows how integral Presto is to our company’s data infrastructure.

We currently hold about one and a half petabytes of data as Hive managed tables in HDFS, and the relatively small data size of our important “core_data” tables allows us to use Presto as the default query engine for analysis. When running ad hoc queries and iterating on the steps of an analysis, Presto is much snappier and more responsive than traditional map reduce jobs. The biggest benefit to adding Presto to our infrastructure stack, though, is that we don’t have to add additional complexity to allow “interactive” querying. Because we are querying against our one, central Hive warehouse, we can keep a “single source of truth” with no large scale copies to a separate storage/query layer. Additionally, the fact that we don’t need change data storage type from RC format to see the speed improvements, makes Presto a great choice for our infrastructure.

We are excited to share this tool with the open source community and we hope that it can provide similar utility for others.

demo

Key features of Airpal:

  • optional access controls for users
  • ability to search and find tables
  • see metadata, partitions, schemas, and sample rows
  • write queries in an easy-to-read editor
  • submit queries through a web interface
  • track query progress
  • get the results back through the browser as a CSV
  • create new Hive table based on the results of a query
  • save queries once written
  • searchable history of all queries run within the tool

Keeping with the spirit of Presto, we have tried to make it simple to install Airpal by providing a local storage option for people who would like to test it out without any overhead or cost. For more detailed information, visit the GitHub page here: https://github.com/airbnb/airpal

A few of the notable technology features in Airpal are:

  1. uses Dropwizard as an easy way to provide a REST service in Java
  2. employs SSE (Server Sent Events) to push messages from the server to the client
  3. the front end JavaScript uses react.js

Finally, we would be remiss if we did not mention the awesome direction that Facebook provided as the original developers of Hive and the pioneers of building UI tools to facilitate easy access to big data. We stood on the shoulders of giants to make this tool and we appreciate the influence and input that the data infrastructure and data tools teams at Facebook were able to provide.

If you’re interested in helping build a world class suite of data tools, check out this open job position: https://www.airbnb.com/jobs/departments/position/48112

The post Introducing Airpal appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/airpal/feed/ 8
Meet the Nerds: Daniel Loreto http://nerds.airbnb.com/meet-nerds-daniel-loreto/ http://nerds.airbnb.com/meet-nerds-daniel-loreto/#comments Tue, 03 Mar 2015 19:48:01 +0000 http://nerds.airbnb.com/?p=176594768 Today we meet Daniel Loreto, an engineering manager here who is an entrepreneur at heart. Daniel tells us about his love for high risk/high reward projects and his new life moving from NYC to SF for his role at Airbnb. How did you get started in Computer Science? When I was a kid my dad […]

The post Meet the Nerds: Daniel Loreto appeared first on Airbnb Engineering.

]]>
Today we meet Daniel Loreto, an engineering manager here who is an entrepreneur at heart. Daniel tells us about his love for high risk/high reward projects and his new life moving from NYC to SF for his role at Airbnb.

How did you get started in Computer Science?

When I was a kid my dad bought our first personal computer for the house, an IBM PC. It was state-of-the-art at the time: a monochrome monitor, no hard drive, a 5 1/2 inch floppy drive, and whopping 64 kb of RAM. I would use it to play games like paratrooper and snake. One day I was poking around the floppy disk that had the game of snake and I found a file with what looked like gibberish english. I asked my dad about it and he explained it was actually the source code for the game written in BASIC.

My kid-mind was blown. I knew people could write novels and that they could compose music … but games? computer programs? You could actually write something interactive and have it do things? I was hooked ever since.

What was your path to Airbnb?

I lived in six different countries growing up and speak three different languages. I’ve love to travel and getting to know other cultures. Most of my professional career was in New York, but I was looking to move to California and join a company where I could have a big impact and could connect with the mission. I talked to a few friends at Airbnb and I knew I had found a match. As a first project I also had the opportunity to work on a team directly with Joe Gebbia, one of the founders. It was a very unique opportunity to work with a founder on challenging new directions for the company.

What’s the most interesting technical challenge you’ve worked on since joining?

In my current project we’re working on new features that changes some of the fundamental assumptions of the original design of Airbnb’s architecture. As a result my team and I have had to very quickly familiarize ourselves with the overall architecture, and then dig deep into some pretty core and complex systems like payments and search to make our project possible.

What do you want to work on next?

I’m always looking for potentially risky but high-impact projects. I love working on new product directions that expand what’s possible for Airbnb today and I’m always pushing my team to work on big ideas that can really move the needle if they’re successful. We have a few ideas that we want to work on next that fit that description …. but I would be giving it all away if I revealed them now. Stay tuned!

What is your favorite core value, and how do you live it?

My favorite core value is, without a doubt, Cereal Entrepreneur. To me it embodies the idea of being creative, scrappy, bold and resourceful in order to accomplish your goals. Don’t be afraid of big audacious goals even with a small team. Is there something blocking you from accomplishing that goal? Don’t be afraid to tackle it, even if it’s outside of your area of expertise. Make things happen. Drive towards a big vision. Shoot for the moon. Don’t stop.

The post Meet the Nerds: Daniel Loreto appeared first on Airbnb Engineering.

]]>
http://nerds.airbnb.com/meet-nerds-daniel-loreto/feed/ 0