Bayesian A/B testing is not immune to peeking

2025-10-30T00:00:00+00:00

Introduction Over the last few months at RevenueCat I’ve been building a statistical framework to flag when an A/B test has reached statistical significance. I went through the usual literature, including Evan Miller’s posts. In his well known “How Not to Run an A/B Test” there’s a claim that with Bayesian experiment design you can stop at any time and still make valid inferences, and that you don’t need a fixed sample size to get a valid result. I’ve read this claim in other posts. The impression I got is that you can peek as often as you want, stop the moment the posterior clears a threshold (eg $P(A>B) > 0.95$), and you won’t inflate false positives. And this is not correct. If you’re an expert in Bayesian statistics this is probably obvious, but it wasn’t for me. So I decided to run some simulations to see what really happens, and I’m sharing the results here in case it can be useful for others.

Adding search to my static blog.

2025-08-11T00:00:00+00:00

How I added fast, client‑side search to this site with Lunr.js and a build-time index.

Who needs git when you have 1M context windows?

2025-07-28T00:00:00+00:00

Lately I’ve heard a lot of stories of AI accidentally deleting entire codebases or wiping production databases. But in my case it was the other way around. I removed some working code and the LLM helped me to recover it.

Semantic Unit Testing

2025-04-09T00:00:00+00:00

Left Wallapop a couple of weeks ago, heading to RevenueCat soon. In that classic ‘between jobs’ hacking window, I built suite: a Python library for semantic unit testing. What’s semantic unit testing? Think unit tests that understand context and meaning, not just assert obj == expected. Sound interesting? I’ll break down what semantic unit testing is, how suite works under the hood, and how you can integrate it.

Three symmetric math riddles

2025-03-19T00:00:00+00:00

I like problems that are easy to pose, and that seem difficult to solve at first glance, but that a slight change of perspective makes them simple and easy to solve. In this post, I will expose my 3 favorite problems of this type.

Optimizing Jupyter Notebooks for LLMs

2025-01-15T00:00:00+00:00

I’ve been using LLM-assisted coding for the last couple of months, and it has been a game-changer. After a couple of iterations, my setup consists in

Win your fantasy league using operations research

2024-07-15T00:00:00+00:00

I’ve never been good at playing football 1. I started playing again last year and I have scored more own goals than goals for my team. Also, I support the RCD Espanyol 2, which has spent last season in the second division and has miraculously ascended to the first division last week. And to be honest, I only watch games to spend time with my friends and family. So it’s not a secret that I’m not a football expert, and I have no shame in admitting it. But I’m very competitive. So when my friends invited me to a football fantasy league 3 some years ago I said “yes, and I’m going to wipe the floor with you, losers!”. I’m from Spain, so I’m speaking about the real and original football, the one that’s played with the foot and a ball, not with the hands and an egg. ↩ See point 54 in my 100 list. ↩ “A fantasy sport (also known less commonly as rotisserie or roto) is a game, often played using the Internet, where participants assemble imaginary or virtual teams composed of proxies of real players of a professional sport. These teams compete based on the statistical performance of those players in actual games.” Source. ↩

Something happens to everyone.

2024-07-05T00:00:00+00:00

In defense of Leetcode Interviews

2024-06-21T00:00:00+00:00

For the last weeks at Wallapop, we have been interviewing candidates for a Data Scientist position. Our current interview process is quite standard, but there are some things we would like to change about the process. We were talking about it during lunch, and I saw my opportunity to propose one of my hot takes: “We should start doing Leetcode interviews” 1. And, as expected, no one agreed with me. Their main arguments against my proposal were For those who don’t know, Leetcode interviews are technical interviews where the candidate has to solve Leetcode problems under time constraints. For example, the candidate needs to solve an easy and medium Leetcode problem in less than 30 minutes. ↩

Good code is rarely read

2024-06-06T00:00:00+00:00

The other day I was interviewing a developer for a position at Wallapop. The candidate was a little bit junior, but I enjoyed their technical assignment and the conversation was going great. One of the questions I had to ask was

You need more than p-values

2024-02-22T00:00:00+00:00

Explore the critical considerations and potential pitfalls of relying solely on A/B testing in making business decisions. While A/B testing is a valuable tool, this post challenges the assumption of exchangeability between past and future data, emphasizing the dynamic nature of the business environment. I propose two solutions (1) validating the exchangeability assumption through holdback groups and (2) advocating for a holistic decision-making approach that goes beyond statistical tests. Executives are urged not to blindly trust p-values, emphasizing the importance of intuition, market understanding, and forecasting in shaping successful business strategies. In conclusion, the post encourages a balanced approach, combining statistical rigor with real-world insights for effective decision-making.

hn-index

2024-02-12T00:00:00+00:00

h-index for HackerNews

A search engine in 80 lines of Python

2024-02-05T00:00:00+00:00

In this post I explain how I built a search engine from scratch using python. The resulting search engine is used to search in the posts of the blogs I follow.

ChatGPT knows things that Google doesn’t

2024-01-25T00:00:00+00:00

Here I explore a doubt I've had for the last 15 years. I vaguely remember a phrase attributed to Voltaire or Robespierre. The phrase was " I'm not a believer, but I prefer my barber to be a Christian, even more, when he's using his razor on my neck". For the last 15 years, Google has failed me to find who was the author of this phrase. But today ChatGPT has helped me to take a step forward in an answer. Who said that phrase? Does ChatGPT know more than Google?

Guide to onboarding in a new job

2024-01-18T00:00:00+00:00

Insights and tips for a smooth transition when starting a new job

2023 review

2024-01-01T00:00:00+00:00

I hate MFA

2023-12-10T00:00:00+00:00

I hate MFA, and in this rant post I explain why. It's basically because (1) is distracts me a lot, (2) it forces me to have a smartphone, (3) it's a leaky abstraction, and (4) it can be replaced with better solutions.

Conditioning is grouping by

2023-11-30T00:00:00+00:00

This insightful exploration draws parallels between mathematical formulations and practical implementations, showcasing how understanding conditioning as grouping can elevate your statistical insights. Discover the parallels between conditional expectations like $\mathbb{E}(y | X=x)$ and popular grouping techniques found in data analysis tools like pandas. Uncover the hidden synergy that exists between statistical conditioning and groupby operations, demystifying complex mathematical concepts with real-world applications. Whether you're a seasoned data scientist or a curious learner, this journey into the interconnected world of conditioning and grouping promises newfound clarity. Elevate your statistical understanding with practical examples, bridging the gap between theory and application. Embark on this enlightening exploration today and revolutionize your approach to statistical modeling. Uncover the simplicity behind complex conditional expressions through the power of grouping, transforming your data analysis skills along the way.

The least controversial movie

2023-11-14T00:00:00+00:00

tldr

Reading S3 data from local PySpark

2023-10-10T00:00:00+00:00

blabla

Stupid by choice

2023-09-18T00:00:00+00:00

Discover the surprising wisdom in embracing 'dumbness.' Learn from a remarkable grandmother-in-law who mastered the art of asking for help by appearing clueless. Explore the benefits of reducing cognitive load, saving time, and focusing on your priorities. Uncover when to apply this strategy and when it's best to rely on your expertise.

Different Types of Means

2023-09-14T00:00:00+00:00

Understand the different types of means and when to use them.

How far can you jump from a swing?

2023-08-18T00:00:00+00:00

In this post I derive the physics of pumping a swing and the physics of jumping from a swing. Then, using both pieces I compute the optimal strategy of jumping from a swing to maximize the traveled distance. I propose this game as a new Olympic sport.

Analyzing Gender Gap in Chess

2023-08-12T00:00:00+00:00

Does it exist a gender gap in chess? Why is it that the best players are men? Is there any fundamental difference between how men and women play this game?

Expected $k$ highest value from $n$ Gaussian draws.

2023-08-12T00:00:00+00:00

In this short note I compare Bilalic and Blom approximations to compute the k-th order statistics from a gaussian distribution.

Good Science, Good Engineering.

2023-08-01T00:00:00+00:00

In this text, I study the idea that science needs engineering as much as engineering needs science. I defend the position that doing good science without following best engineering practices is impossible.

You’re the best at something.

2023-07-25T00:00:00+00:00

In this text I use math to show that you have a rare skill combination, that makes you the best at something.

512KB Club

2023-07-20T00:00:00+00:00

Quick blog update. Since last Tuesday (2023-07-18) I’m a proud member of the 512KB club. I am aware it doesn’t sound very ambitious, but it serves as a small reward after having completely refactored my blog.

Automate your static blogroll.

2023-07-20T00:00:00+00:00

In this post I explain how I built my automatic blogroll using Github Action and Github Pages.

About math limitations.

2023-07-18T00:00:00+00:00

Nobody cares about your blog.

2023-07-15T00:00:00+00:00

Nobody cares about your blog, but you should keep writing!

Why did I make my blog uglier?

2023-07-13T00:00:00+00:00

Discover the reasons behind the transformation of my blog and the benefits it brings. Experience faster writing, simplified design, reduced distractions, and a sense of complete ownership. Explore the enhanced aesthetics that prioritize minimalist style and direct communication. Find out why this revamped blog reflects my unique vision.

A game for the next 15 years: counting license plates

2023-07-01T00:00:00+00:00

Since last October, I’ve been playing in a simple game all by myself, with just one rule - I have to spot all the license plates from 0000 to 9999 in sequential order. Where I live, license plates follow this format: XXXX - AAA, where XXXX represents 4 numbers, and AAA are three letters.

How much for your brain?

2023-06-13T00:00:00+00:00

In our modern era, we have witnessed a remarkable shift in the way we approach computation. We no longer rely solely on physical machines housed within our premises; instead, we rent computers in the ethereal realm of the cloud. It’s a convenient arrangement that grants us access to computational power without the hassle of maintenance or obsolescence. But the wonders of rental don’t stop there. We have taken a step further and begun leasing machine learning models for predictive tasks, tapping into the immense potential of algorithms to enhance our decision-making. OpenAI and its pre-trained models stand as prime examples of this fascinating trend.

Debunking the Myth of Dollar Cost Averaging

2023-06-07T00:00:00+00:00

Investing your money wisely is crucial for financial success. In this post, I delve into the popular strategy of dollar cost averaging (DCA) and compare it to the lump sum (LS) approach. By analyzing S&P 500 data from the past 40 years, I show that DCA is generally suboptimal, with LS outperforming it in 82% of cases, resulting in a 23% higher return on average. I also explore fine-tuning DCA by examining the impact of investment frequency and duration. Ultimately, the data suggests that investing all your money at once is a more favorable strategy. So, don't wait – start investing as soon as possible.

A randomized voting strategy

2023-05-23T00:00:00+00:00

Are you sick of having to choose which political party to vote for when none of them represents your beliefs 100%? In this post, I am going to explain a practice that will end your headaches when voting.

How to initialize your bias.

2023-02-24T00:00:00+00:00

Learn how initializing the correct bias in your neural network can speed up training process through analytical derivation and experiments.

The error of the error.

2023-02-16T00:00:00+00:00

The uncertainty of a measure is a statistical process. The standard approach to measure a quantity is to repeat the measure a number of times, compute the mean and its standard error. It is based on the assumption that measuring is an imperfect process, and therefore when measuring some quantity, an error is introduced. This post goes down the rabbit hole of computing the uncertainty of the uncertainty. Then I derive the results of the error of the error and applies the same logic when the distribution is not normal.

The Sweet Spot for Buying Used Cars.

2023-01-16T00:00:00+00:00

In this post, I show how can we leverage the extra information we get from knowing that a car has survived a certain amount of time. I formalize the idea mathematically and then build a framework to optimize the total costs.

Discovering the best Chess960 variation.

2023-01-11T00:00:00+00:00

In this post I analyze all the available Chess960 games played in Lichess. With this information I show that there are no starting positions that favor any of the players more than other positions.

2022 review

2023-01-01T00:00:00+00:00

Today I’m going to write about the most typical topic of the last days of the year: New Year’s resolutions. The usual schema for this post is to review one last year’s resolutions and write down the resolution for the starting year. However, I didn’t write any list for this ending year, so I’ll just do a quick summary of the year and write down the resolutions for 2023.

Can Random Forests overfit?

2022-12-26T00:00:00+00:00

Random Forest can't overfit. Or can they?

Mate with only pawns and no promotions.

2022-11-24T00:00:00+00:00

In this short post I show a position where's possible to mate using only pawns. Also, the mate is optimal, in the sense that the game can't we won using less movements.

Computability Theory (ii): uncomputable numbers.

2022-10-20T00:00:00+00:00

During my current incursion in computability theory I learnt about uncomputable numbers, this is, numbers that can’t be computed with arbitrary precision. This means that even given all the computational power in the universe you could not compute these numbers. Even if God himself came from Heaven, he could not compute these numbers. Uncomputable numbers are numbers that can’t be computed because of maths limitations. Just the idea that maths has some inherent limits amazed me. In my last post, I talked about the halting problem, which is a limitation of maths, but the halting problem is a little bit abstract in my opinion. In this post, instead, I’ll show that there are numbers for which we can know an approximate value but can’t be known with arbitrary precision.

Computability Theory (i): the Halting Problem.

2022-10-19T00:00:00+00:00

During these last few days I’ve been reading a little bit about computability theory, and I feel like a kid with a new toy, so I’m going to write some posts about this topic. I don’t pretend to explain anything new, and probably what I’m going to write has been written before, but I’ll write these posts for two reasons: (1) as future notes for myself, and (2) to help me clarify and organize the ideas.

Choose the smallest number not chosen yet.

2022-09-28T00:00:00+00:00

A game where you have to choose the smallest number that nobody has chosen yet.

TIL: constant folding in python.

2022-09-10T00:00:00+00:00

How does python optimize constant expressions.

L1 regularization from a bayesian perspective

2022-08-27T00:00:00+00:00

How to interpret L1 regularization using a Bayesian approach.

Field theory for recurrent mobility

2022-07-29T00:00:00+00:00

A review of my first -and for the moment the only- paper published in a top journal.

Your laptop is an implementation detail

2022-07-25T00:00:00+00:00

Treating your laptop as an implementation detail can help your productivity.

Vectorizing impossible operations: boolean algebra, sets, and filters

2022-07-13T00:00:00+00:00

How to use boolean algebra to vectorize complex operations and get x2000 speed-up.

Data science preflight checklist.

2022-07-06T00:00:00+00:00

List of checks to pass before starting a new data science project.

Why do we minimize the mean squared error?

2022-05-25T00:00:00+00:00

Developing a better understanding and intuition about what really means to minimize the mean squared error.

Training TensorFlow models with big tabular datasets (iii): preparing a warmup for TensorFlow models.

2022-05-24T00:00:00+00:00

How to create a warmup file for your tensorflow model using TFRecordWriter and tensorflow_serving.apis.

Why bureaucracy is fundamentally wrong.

2022-05-23T00:00:00+00:00

Why the combination of bureaucracy and computerization is breaking our society.

Feynman’s Restaurant Problem.

2022-05-09T00:00:00+00:00

Introduction and solution of the Feynman's Restaurant Problem.

Counterintuitive statistics (i): a fair coin game.

2022-05-04T00:00:00+00:00

A counterintuitive experiment with fair coins.

Training TensorFlow models with big tabular datasets (ii)

2022-04-25T00:00:00+00:00

Speeding up TensorFlow training.

Coding sucks.

2022-04-22T00:00:00+00:00

How to lose one hour of your time!

Training TensorFlow models with big tabular datasets (i).

2022-04-15T00:00:00+00:00

How to train a tensorflow model when your training dataset doesn't fit in memory?

Continuous Blackjack (ii): deriving basic equations from two other perspectives.

2022-03-24T00:00:00+00:00

Deriving same equations with different methods

Continuous Blackjack (i): introduction and first results.

2022-03-11T00:00:00+00:00

What is and how to play continuous blackjack?

Data Science and Software Engineering (I)

2022-02-07T00:00:00+00:00

I still remember when I started working as an intern data scientist and I had to open my first pull request. I was very happy with my work, so I just committed all my changes together, opened a pull request, assigned it to my supervisor, and started waiting for the compliments to arrive - spoiler: the compliments never arrived.

Dynamic Programming (i): Multiplying Matrices

2020-12-31T00:00:00+00:00

Learning how to multiply matrices in the best possible order, ie solving the Matrix Chaing Multiplication problem.

Alex Molas Blog