Implementing Multi-Armed Bandits to Improve Website Effectiveness


When: 12pm - 1pm
Where: UNC Charlotte Center City Building (320 E 9th St, Charlotte, NC 28202).

Lunch: A brown bag lunch may be brought in or Einstein’s Bagels is located in the lobby of Center City.
Parking: Parking will not be covered. We recommend using the 7th street parking deck. The pay to park lots near the building are full during week days. Parking is very limited in the area and we suggest that you walk, take the lightrail or carpool if possible. Please refrain from parking in UNCC's parking lots, as you could be ticketed.

For more information on the Center City Building, visit




Data Scientist at Red Ventures

Keith Williams is a data scientist at Red Ventures, where he is interested in using machine learning to automate decisions and visualization to communicate to humans how and what the algorithms learn. His current focus is on applying reinforcement learning algorithms to make personalized decisions for customers.


Sr Data Scientist at Red Ventures

Joe Tenini is a data scientist at Red Ventures, where he uses machine learning to help Red Ventures leverage its data to make real-time decisions, optimize processes, and personalize strategies.

His current area of specialization is in personalizing user treatments and optimizing actions in real time with things like contextual multi-armed bandit policies and recommender systems. He has implemented both batch learning systems on large distributed data sets as well as online learning algorithms that need to be executed in the sub 100ms range. He's a firm believer that successful data science projects require deep integration with both business and technology stakeholders, and has successfully directed such projects end to end.


Implementing Multi-Armed Bandits to Improve Website Effectiveness

As part of their digital marketing efforts for their partners, Red Ventures designs and hosts websites that are the destinations of their paid media efforts. These websites serve to build value for the brand and facilitate transactions. Historically, Red Ventures has used randomized control trials to test the effectiveness of these sites. While there are many benefits to testing effectiveness in this way, it leaves value on the table. For example, it assumes all decisions are made in the same context and optimizes to the average user. In this talk, Keith & Joe will present an alternative method for testing the effectiveness of a decision that address the weakness of randomized control trials. In particular, they will discuss a class of reinforcement learning algorithms called multi-armed bandits and their varieties that address different problem statements. Finally, we will discuss how we have implemented these algorithms into our technology that makes decisions about what website a customer should see as they arrive on our sites.

from the event