We have finally found a way to properly scale JOINs, with insane performance at multi-billion row scale, for our new analytics model. It's…

Thread by Jack Ellis@JackEllis · Sep 28, 2025 · 3 posts · 148 words · 1 min read

We have finally found a way to properly scale JOINs, with insane performance at multi-billion row scale, for our new analytics model. It's a huge moat for us as a company and required a genius database consultant to get us there. It was a lot of pain (seriously) but so worth it.
1/3
You know why a lot of analytics companies have maximum data rentention periods? Because the more data they have to analyze, the harder it is to do fast. So they chicken out and delete historical data. It's a super lame solution and it allows you to avoid the hard work of scaling.
2/3
You, the reader, instinctively want to ask "Have you tried X". Yeah, I've tried everything. This problem exists in all OLAP databases and you solve it via creative structuring specific to your application's structure. So yeah, your important, historical data will not be deleted.
3/3

Jack Ellis answered 4 questions

under this thread · top answer 6 likes

@dubaplex askedMost liked

Well I'm glad for fathom. I wonder what features you can add that competitors can't replicate without fast joins (besides keeping historical data).

Watch this space :D It's going to be solving at scale

Jack Ellis@JackEllis6 likes

@tplaner asked

Curious about the approach, did you actually find a way to keep the joins performant at query-time, or is this more along the lines of flattening / building rollups & aggregates (similar to how YouTube handles view counts)? I’m a bit skeptical since most systems solve it the latter way.

Zero rollups :)

Jack Ellis@JackEllis2 likes

@tplaner asked

Very cool that you got this working. I’m curious why go that route though. Architecturally, pushing multi-billion row joins to query-time usually isn’t the best approach. For most analytics, rollups/aggregates are faster and more pragmatic. What’s the actual gain here?

Yep rollups are solid but you introduce trade-offs. They're planned but being able to run joins at this scale allows us to do more with features that are on the fly vs "known rollups"

Jack Ellis@JackEllis2 likes

We have finally found a way to properly scale JOINs, with insane performance at multi-billion row scale, for our new analytics model. It's…

Thread by Jack Ellis@JackEllis · Sep 28, 2025 · 3 posts · 148 words · 1 min read

We have finally found a way to properly scale JOINs, with insane performance at multi-billion row scale, for our new analytics model. It's a huge moat for us as a company and required a genius database consultant to get us there. It was a lot of pain (seriously) but so worth it.
1/3
You know why a lot of analytics companies have maximum data rentention periods? Because the more data they have to analyze, the harder it is to do fast. So they chicken out and delete historical data. It's a super lame solution and it allows you to avoid the hard work of scaling.
2/3
You, the reader, instinctively want to ask "Have you tried X". Yeah, I've tried everything. This problem exists in all OLAP databases and you solve it via creative structuring specific to your application's structure. So yeah, your important, historical data will not be deleted.
3/3

Jack Ellis answered 4 questions

under this thread · top answer 6 likes

@dubaplex askedMost liked

Well I'm glad for fathom. I wonder what features you can add that competitors can't replicate without fast joins (besides keeping historical data).

Watch this space :D It's going to be solving at scale

Jack Ellis@JackEllis6 likes

@tplaner asked

Zero rollups :)

Jack Ellis@JackEllis2 likes

@tplaner asked

Yep rollups are solid but you introduce trade-offs. They're planned but being able to run joins at this scale allows us to do more with features that are on the fly vs "known rollups"

Jack Ellis@JackEllis2 likes