Natalia Silberstein

Combating User Fatigue with Frequency-Recency Features

Senior Data Scientist, TL of Data Science – Outbrain

Natalia Silberstein

Combating User Fatigue with Frequency-Recency Features

Senior Data Scientist, TL of Data Science – Outbrain

Bio

Natalia is a senior data scientist and data science team leader at Outbrain in the Recommendations group. She is involved in the design and improvements of algorithms that control personalized ad selection.

Before joining Outbrain, Natalia was a research scientist at Yahoo Research Haifa in the Native Ad Science group. Before joining that group, she was with the Mail Mining group, primarily analyzing and modeling mail data to devise novel mail features. 

Natalia completed her PhD on “Coding Theory and Projective Spaces” at the Technion, Computer Science Department, under the supervision of Prof. Tuvi Etzion in 2011. She then joined Prof. Sriram Vishwanath’s Wireless Networking and Communications Group at the Department of Electrical & Computer Engineering, University of Texas, Austin, as a postdoctoral fellow, focusing on coding for distributed storage systems

Bio

Natalia is a senior data scientist and data science team leader at Outbrain in the Recommendations group. She is involved in the design and improvements of algorithms that control personalized ad selection.

Before joining Outbrain, Natalia was a research scientist at Yahoo Research Haifa in the Native Ad Science group. Before joining that group, she was with the Mail Mining group, primarily analyzing and modeling mail data to devise novel mail features. 

Natalia completed her PhD on “Coding Theory and Projective Spaces” at the Technion, Computer Science Department, under the supervision of Prof. Tuvi Etzion in 2011. She then joined Prof. Sriram Vishwanath’s Wireless Networking and Communications Group at the Department of Electrical & Computer Engineering, University of Texas, Austin, as a postdoctoral fellow, focusing on coding for distributed storage systems

Abstract

Outbrain serves millions of users with personalized native ads and recommendations. One of the most challenging issues in such recommender systems is finding the optimal balance between user satisfaction and overall revenue. When users see the same ad repetitively, it hurts user experience the most. This problem leads to the phenomenon called “user fatigue,” expressed in a decrease of the click-through rate with every impression of the same ad.  However, naive solutions to this problem may come at the cost of reduced revenue.

An elegant solution incorporates new features based on recent historical interactions of users with ads into the click prediction model. Specifically, the approach combines information about the frequency and recency of past user-ad interactions in a sense that very recent interactions (that happen in recently) get higher importance than those that happen a few hours or days prior.

The features were implemented and tested thoroughly in online A/B testing systems and showed that this approach reduces user fatigue by diversification of displayed ads and increases revenue.

Abstract

Outbrain serves millions of users with personalized native ads and recommendations. One of the most challenging issues in such recommender systems is finding the optimal balance between user satisfaction and overall revenue. When users see the same ad repetitively, it hurts user experience the most. This problem leads to the phenomenon called “user fatigue,” expressed in a decrease of the click-through rate with every impression of the same ad.  However, naive solutions to this problem may come at the cost of reduced revenue.

An elegant solution incorporates new features based on recent historical interactions of users with ads into the click prediction model. Specifically, the approach combines information about the frequency and recency of past user-ad interactions in a sense that very recent interactions (that happen in recently) get higher importance than those that happen a few hours or days prior.

The features were implemented and tested thoroughly in online A/B testing systems and showed that this approach reduces user fatigue by diversification of displayed ads and increases revenue.

Discussion Points

  • First, how to decide whether a labeled data is a must? 
  • Different types of labeling challenges we’ve dealt with as data scientists (partial labels, noisy labels, etc.)
  • Academic approaches that discuss possible solutions to these problems
  • Practical solutions we eventually implemented 
  • Interesting case studies and results

Discussion Points

  • First, how to decide whether a labeled data is a must? 
  • Different types of labeling challenges we’ve dealt with as data scientists (partial labels, noisy labels, etc.)
  • Academic approaches that discuss possible solutions to these problems
  • Practical solutions we eventually implemented 
  • Interesting case studies and results

Planned Agenda

8:45 Reception
9:30 Opening words by WiDS TLV ambassador Nitzan Gado and by Lily Ben Ami, CEO of the Michal Sela Forum
9:50 Prof. Bracha Shapira – Data Challenges in Recommender Systems Research: Insights from Bundle Recommendation
10:20 Juan Liu – Accounting Automation: Making Accounting Easier So That People Can Forget About It
10:50 Break
11:00 Lightning talks
12:20 Lunch & poster session
13:20 Roundtable session & poster session
14:05 Roundtable closure
14:20 Break
14:30 Merav Mofaz – “Every Breath You Take and Every Move You Make…I'll Be Watching You:” The Sensitive Side of Smartwatches
14:50 Reut Yaniv – Ad Serving in the Online Geo Space Along Routes
15:10 Rachel Wities - It’s Not Just the Doctor’s Handwriting: Challenges and Opportunities in Healthcare NLP
15:30 Closing remarks
15:40 End