Your VP Is Doing a Rogue Analysis in Cursor Right Now — with Nell Thomas

Transcript#

This transcript was generated automatically and may contain errors.

Welcome to The Test Set. Here we talk with some of the brightest thinkers and tinkerers in statistical analysis, scientific computing, and machine learning. Digging into what makes them tick, plus the insights, experiments, and OMG moments that shape the field.

On this episode, we sit down with Nell Thomas, who has led data teams at Etsy, Facebook, and the Democratic National Committee, or DNC. And now she's VP of Data at Shopify, which has mandated that AI usage is a baseline expectation for every employee. She leads a team of over 400, and she'll be the first to tell you, nobody knows the right way to do any of this yet with AI. We talk about the modern data stack, building trust in data organizations, and why the smartest move right now might be resisting the urge to lock anything in with AI, but embracing exploration and discovery.

We also as a bonus talk a little bit about the movie Sneakers, so I feel like all you sneakerheads, the movie, rejoice, this one's for you.

Because actually, I think that's like the number one thing I've noticed is in organizations where people aren't actually paying attention to the data work. It's very hard for it to be amazing. Because you need like a great audience to appreciate it.

And ask questions and to push it and deepen it and challenge you. And so that's like another thing that I always want to make sure is happening.

You know, it's also it's kind of a bad word, right? Like, I think you hear the word scrutiny, you're like, yeah, yeah, it feels like it feels like, And like, no one, I mean, no one wants to feel like their work is constantly being questioned, or like, that's not a good feeling either, because it suggests a lack of trust.

It's more like people who can appreciate it, like maybe like thinking about like having a really great audience for, for like a play. We're doing a play with no one there watching it. Like, are you going to do your best work? Right? And but it's not just people clapping. It's also people like engaging with the material and being like receptive to it and like contributing back to it in a way that has a really has the ability to like, to make someone feel like they want to do better and better work, right?

So I think it may be scrutiny. I mean, I should think about a better word than scrutiny, but it I think it's about attention. Anyway, you asked me what it looks like, though, I would say it's making sure that that there are the right forums or rituals could be meetings could be like async things, but where people's work can be like reviewed and discussed and debated and and is part of the conversation.

The modern data stack, role by role

I mean, most of the great companies right now are data companies, and how well they leverage that data and the systems they build around it are one of the things that differentiates.

Yeah, I mean, some, you know, I like to think about the kind of the value chain of work of data from from raw data creation, all the way through to, like, how that data is being used. And one of the nice things about my current role is that, especially leading the data infrastructure team, I get to be pretty far left of on that sort of my spectrum, this is left of from raw data creation, like how that data is ingested, how it's processed, how we make sure that it is kind of prepared for downstream users, both for analytical purposes, or like kind of data science purposes, and for production purposes.

And that so kind of being really far left on that spectrum, is a place of immense leverage, because the better we can create really well, a make sure we're collecting the right data, which is always a fun problem. And making sure that's captured in a clear way and making sure that we're actually being cost efficient. That's increasingly an issue now that we see like how quickly costs can increase around LLM usage.

I think in terms of managing the appetite for that, which I think is the heart of your question, I see that as an incredibly exciting challenge. I'd always rather be working in that environment than trying to convince people why data matters, right? I think that right now we're in a moment where I don't need to convince anyone that they need to care about that spectrum of data. I think it's more managing the reality of what is possible.

Managing like the expectations around how quickly we can get what we need, and making sure that we have the strength in our foundations on that kind of left side of the spectrum so that we can trust the outputs. Because increasingly, as you see, it's very easy to work on the right side of the spectrum with vibe coding and lots of ways that you can very quickly do an analysis. Underlying data quality, underlying foundations make everything, it's only positive benefits downstream from that.

Sure. So, I mean, I think, you know, all the way on the left, it's usually you have your, like in my mind, this is just my experience, you have your production engineers who are like writing the code that hopefully generates the instrumentation that gathers the data you need, right? Whether that's writing to a database or firing an event that, you know, is sent off to some sort of log.

You know, then you have your infrastructure, your data infrastructure engineers that are ensuring you're ingesting that data correctly, right? So whether that's how the events are being fired into whatever Kafka topics or whether it's your replicating data from a database and then doing the work and building the data platform of processing that through a whole bunch of storage and compute layers that are doing some sort of modeling of that data, right?

And like usually, you know, at some point you're handing it over from like what is more of a data platform engineer of like creating the house to like the data, the analytics engineer who is then ensuring that the raw ingested data becomes model data that is hopefully some sort of canonical data asset that then the data scientist can query.

And just to check, is it at that point, like once it hits the analytics engineer, you're in something like BigQuery or... You're writing a sequel, you're writing a DBT, maybe you're using Airflow, you know, I mean, that's like the stack we currently use, but yes, that's very common. Maybe you're using Snowflake instead of BigQuery, but obviously flavors vary. But I think that sort of those big buckets are pretty universal. You have the team that's responsible for like maintaining the data house, and then those people are like working inside the house, as I like to say.

And then obviously at the very end, you have your presentation layers, whether that's Looker Studio and Google or Looker or, you know, there's some fun new ones, but, you know, whatever flavor of how that looks, they all have their limitations, in my opinion.

The other use case here that I kind of mentioned in the beginning is also like the diversion of some of that data out of the data science use case into sort of more of the production modeling use case. So like, hey, we're going to model this data to use it for recommendations, for search, for some sort of prediction, right? And those cases, usually the handoff isn't to an analytics engineer, it might be to a data engineer, to a machine learning engineer, kind of the path of coding for the machines as opposed to like coding for human input.

Yeah, I mean, I think the two, to me the differentiation there is a little bit, like one is latency. So like analytics engineers often working in batch, you know, might have, hey, it's totally acceptable to have a 8 hour, 12 hour, 24 hour delay and your data being fresh because you're counting things to, you know, do an experiment analysis or do a user behavior analysis of usage of a product or a feature changes over time, but usually have a longer scale.

Some of the production or operational use cases I'm talking about, you might, you have very low latency requirements, right? Like I want to be able to process data very quickly because I want to see what a buyer is doing on the site to be able to change what I'm showing them on the site, right?

You know, I think what's interesting, if I'm like driving this fork in my head, I guess one of my points here is that like the further left you go on that, the more unified it should be. And then it kind of starts to fork over time, but you don't want those forked the whole time because you end up with a bunch of data replicated, a lot of costs involved, and probably a lot of like lost knowledge and context and value because you're not getting like great reusable learnings between those two streams.

Technology choices and data culture

My number one recommendation to any person who's, like, new to a company is to get your hands on an end-to-end data flow diagram. And like, what are the technologies being used in every stack? And who owns them? And it really, I don't know, I think it's, to my point earlier about just, like, having that holistic view, I think it really helps connect dots about, like, where you're, like, it's kind of like a food chain.

Oh, and it's a huge topic. I mean, the question of there's the classic, like, build, burst, buy part of this, too. And there's the deprecation of legacy tools, which is really important, often doesn't happen. And there's the sort of fragmentation of the technologies people are using internally. And, like, how much do you, like, encourage and permit versus how much do you clamp down on that?

I remember when I was at Meta, it was, like, people used all sorts of different, like, personal data stacks. And the data sources was the same. But then some people were using Tableau and some people were using, like, Modes. Like, it was kind of a wild west of, like, what, you know.

When I was at Etsy, it was, Etsy was very big on, like, do-it-yourself. I mean, kind of per Etsy's vision as a company. Yeah, it's very DIY. It's a very DIY company. It was founded by a carpenter. I mean, like, our, like, our database were all, like, on-prem. Like, it was very, it was very, like, you know, build your own stack.

And we were writing, I mean, I was not writing, but, like, our data folks were writing, like, Scalding. Like, it was, yeah. We had a Vertica cluster. This is where we did our, yeah, which is also an old technology. Weirdly, I've done two Vertica migrations in my life. One at Etsy and one at DNC.

Yeah, obviously, pre-DBT, you know, people were writing a lot of, and some people were writing R, some people were writing Python, you know, people were writing, obviously, SQL. But a lot of the challenges we had there were not that dissimilar to, like, what I see now in terms of, like, where is the data for X? And is this the, am I using the right data?

And I think with LLMs, those are the same, they're still the same questions, right? Like, to your point earlier about it, like, can I trust what the LLM is sharing? Because I can interrogate what data source they're using, and are they holding it correctly? And there's so much, I mean, there's, unfortunately, there's a lot of art still in, like, are you kind of, like, using your data set correctly? Because you understand the, like, nuances and arcane details in a way that means you can leverage it correctly.

I mean, one of the things that I, you know, things I'm seeing discussed a lot lately is agents putting a lot more load and stress on data platforms, because essentially, like, you know, used to be like, you write a SQL query, run a SQL query, you look at the results in my list. So essentially, like, often, like the, you know, human in the loop, using the data platform or, like, using a dashboard, kind of limited by, like, you know, human in the loop doing stuff and thinking and using their mind to think about things.

But now you can have, you know, a bunch of agents, like, interrogating a data platform. And so, you know, in a way, it's almost like not differentiable from a DDoS attack in some cases, where it's just like, I run the SQL query over and over until I understand the problem.

Well, again, I think this is where some of the best practices that the industry has been talking about for decades are still the most important best practices, which is, like, you know, work out in the open. Like, when you're a data scientist and you're writing code, first of all, write code. Don't do your analysis in, like, Excel. So, like, write code. Like, check it into GitHub. Make it, like, observable by others. You know, have it code reviewed. Store the output of your analysis in a notebook. Like, have it all be visible and accessible so someone can walk through your chain of logic.

I think that, you know, we need LLMs to be able to follow that same path so that we can follow that, you know, we can follow their reasoning just as much. Like, I will, I do get, I get shared on screenshots of cursor outputs all the time. Mostly by non-data scientists, right? You know, it's like, you know, I have a, I love one of my fellow VPs of the company, but he loves to do a rogue analysis and send me a screenshot and be like, I found the number. And, you know, my first question is always like, what data source is it using? Like, can you know?

You know, it's like, you know, I have a, I love one of my fellow VPs of the company, but he loves to do a rogue analysis and send me a screenshot and be like, I found the number. And, you know, my first question is always like, what data source is it using? Like, can you know?

I mean, it really builds a lot of trust, right? Like when you can show, I mean, because I mean, and to kind of the point you made earlier, Weston, I kind of like a little bit challenge you on where I feel like there are a lot of people, you know, a lot of people will just like produce an answer. Like, oh, we ran the experiment and here is the result. And you get like a slide and a deck and it's like you then you're forced to just rely that you're okay. Trust this human is like doing everything.

But, hey, actually, if we can build a system and a set of tools where it makes all of that experiment work, like really easily auditable and like you can see it and interrogate it and you can like that, you know, inherently builds trust in the platform. And it also allows like the data person to do, I think, like higher order thinking a little bit. So I feel like that sort of when transparency in your work is the default, it just automatically breeds trust in the results of that work.

And it kind of frees people up from that loop of like questioning, like what assumptions did you make when you do this analysis? Or like what was the time period again? Or like, you know, what ways did you filter or not filter this audience? It bypasses all that so you can just actually have a conversation about what to do with it, right? Like, okay, now what do we do now that we have this answer?

And so I find that part of best practices like really compelling because it's not just like governance for governance sake or like, you know, dogmatic best practices for best practices sake, but it actually like speeds up the rate of kind of idea to answer and getting us to like insight faster.

Psychological safety and leadership at scale

I mean, first thing I'll say is that I have a huge amount of selection bias in what I know about. Like, this is one of my, it's one of the hardest things about being, I mean, overall, it's a joy and a delight to do my job and I feel like it's an honor. But it's a challenge of being in a like more senior role is that people don't always want to tell you the truth because they feel like they should be managing up in some way, which is a reality.

And I also, I talk to people who, you know, work for me directly the most. And I talk to like your kind of like average entry level data scientists the least. And so I'm always careful to how I answer a question like that because I'm like, I would much rather you had like, you know, someone from my team here to like talk about like how they think about it as opposed to like what I think they think about it.

I mean, I think one like just in terms of how to encourage a culture where people feel comfortable sharing and working transparently. I mean, to me, the number one rule is making sure that when people share their work, they aren't, the dialogue is like healthy and productive. So even if there's like an element of like critical review, it's from a place of like good human interaction. And like people don't want to work with people who are jerks.

They don't want to work with people who are like, you know, tough or demeaning or like being like critical of work just to be critical of work. And they want to do it if it's like, if it's making their work better and it's craft. So like an obvious thing, but like I think it's probably the most important thing is that you have to like to be kind of vulnerable and sharing your work openly. You have to trust that it's going to be received with good intent.

I think also sometimes people do get really like, like things become precious. You kind of start having these like tools that no one can touch, right, or or decisions. And I think one thing I actually really appreciate about Shopify, because it has a real focus on like our CEO founder, Toby, is very focused on like first principles and like that's what comes first.

And that it means that kind of every tool and technology and choice is kind of always up for discussion. And there's no kind of sunk cost fallacy of like, well, we can't talk about that because, you know, we've already invested X amount of time. So it's like the orientation around not like specific emotional choices around tools, but around just like, hey, these are the guardrails of what we want and how we want to get it really I think opens up more room for that freedom for people to not feel encumbered.

I think that it is a culture that is kind of, it is a cultural like artifact. I think that there have been places where there's more freedom to like interrogate and there's less. I think one thing Etsy did very well in terms of its psychological safety was they had a very, very big on blameless postmortems. And that idea of like, hey, we're going to do a retro and we're going to like make a point of going overboard about never ever having this be about an individual failure. It's always been a system failure. And how did like a person get caught up in that system failure?

You know, that spread around the company as a way of just thinking about mistakes generally. Right. And so that was a really important cultural touchstone. You know, I think that right now at Shopify, like Toby's investment in just like hiring people who are super high agency, who have like a high curiosity, a willingness to learn and teach learning like a muscle, like not something you do once, but something you do like every single day. And that's actually the most important skill set.

Political data work and the DNC

Campaigns are these like ephemeral moments, right? You have a campaign. It gets spun up. It lives as an organization and as a set of tooling choices for maybe at best if you're a presidential campaign and you make it through the primary, maybe it's 18 months. But like more often it might be three months, maybe nine months. And so, yeah, it's a lot of it can it can be a lot of quote unquote throwaway work and throw away org structures and throw away like and that is really hard on the overall it's hard on the outcome of wanting to be able to learn continuously and like leverage those learnings like cycle after cycle after cycle.

So the next campaign can be better at talking to voters than the prior campaign. And that's actually so it's one of the things that there are a lot of great organizations in the political space that try to create that like consistency across those cycles. DNC being one of them where it's like, hey, this is like a we can make an investment in our technology and our infrastructure that creates like a really strong foundation that these campaigns can build up on. And instead of the foundation being like this, the foundation is actually like this, where every year the foundation is getting better. And so the starting point is getting higher for every campaign.

I think the biggest difference is just scarcity of data. So like in political campaigns, mostly the outcomes you care about are getting people to vote. Voting is an offline behavior, whether you're in person or whether you're doing it by mail. It's also a very sporadic behavior. Maybe you're getting it once a year if you're lucky and you have a super motivated voter, which is usually not the type of voter you're trying to talk to, right?

And so, yeah, data about what people are buying is basically the very long-winded answer to your question. And I love your, it sounds like you do data infrastructure for the companies that is infrastructure for shopping. So it's like infrastructure all the way.

Shopify, AI mandates, and the future of data work

Yeah, no, I mean, it was a pretty big shift when like Toby wrote that memo, it was released internally, and then it found its way outside of the Shopify. And I would say I feel very lucky that I was at Shopify at this moment in time because that mandate created a lot of, you know, I say it took away a lot of the noise that was happening at the time of like, should one or shouldn't one? Is this like a direct assault to one's craft and one's discipline? Like, you know, what does this mean for the future of work? And I kind of just said, future is here, like figure out how to use it. And like, that's actually like fun, you know,

and when you shift from like the fear-based mentality of like, how is this encroaching on my job to a, this is another tool and how do I use it to its fullest potential to like do my work? I don't know, it just kind of really shifted the, I think, the feelings around it.

I was, one thing that I was incredibly proud of, like our, I think Anthropic announced like the model context protocol for MCPs in like November of 24. And my team had built one, when I say my team, but I was like, I had nothing to do with it, which is like the smart team did it. By like February, we had one for our data warehouse, which is like core. So very, very quickly, people could start just like doing all of the fun analysis. And I think that was again, like a good six months before a lot of other companies were doing something like that.

And so that's an example where I think it's creating a lot of like flay. And like, you know, and I know it's like a scary thing and it feels like uncertain and we don't know where it's going, but it's also like fun. And I think that it's kind of creating a lot of opportunities to just try different things out. And that is sometimes rare.

So I'm still grappling a little bit as a leader of a large team of how to like, watch all the fun and tinkering, but also make sure we're creating scale and leverage because I want to make sure everyone can kind of be along for this journey. And like, you know, there's a lot of variance right now. So I'm, that's one of the things I think about the loss the most. I don't have a great, easy answer right now of how I'm like walking that line.

I am thinking a lot about how to converge on like one default wave, leveraging it. But yeah, it's a hard question. I'm having a lot of fun with, with, with agentic coding. Like I was just watching, you know, Peter, Peter Steinberger is the creator of, you know, the now viral Claude bot. You know, he gave a talk late last year. The title was, you can just do things. And so, you know, it's kind of like, and I wrote a blog post that's called why not? And that was kind of like my new motto of like, Oh, you want to build a thing? You know, why not? Like, just try, you know, build something.

It doesn't work. It's not good. You can just throw it away and code. Code now has, you know, much less value, you know, used to be that like code artifacts would be like, Oh, this was the product of human labor. And like, you know, it's always funny. And like the, the code counting pools, like SLOC and CLOCK and SCC, they'd be like, Oh, this code base would have cost $3 million and three years to build and be like, I built it today with my agent.

And so it's, I think part of what's fun is just the, you know, not knowing what's the best, you know, like, what's what's the right and proper, proper way to do things and encouraging like that experimentation and creativity. And I feel like for me, like the almost like, for me, it's like it gives me what's happening right now. It gives me the same feeling that I got when I started to do Python, which was, you know, like in the late 2000s, where, yeah, it's like, I can just, I can just write code and do things.

I also kind of feel like the most of the best things in life, it's like they get more interesting, the more you engage with them. And I think that's true of like AI and LLMs right now. Like it's the first couple of times you do something with them, you're like, oh, this is like magical. But then it's like, the more you use, the more, almost the more magical they feel. And like that part of it is actually really exciting because it's like, you kind of want to keep going down the rabbit holes and pulling those threads.

I think that there's ways to like, you know, I think most of what I see because also Shopify is a pretty fast paced company with a pretty like, you know, rigorous set of milestones to hit. People are mostly doing all the creativity for the purposes of like impactful work. And so that right now, I don't worry too much about it being like a bunch of side quests. I think that they're all pretty aligned.

But I do think it's uncertain. I also think it, you know, if six months ago we like locked it and we're like, all right, now this is the way to like AI, we would have made a bunch of bad decisions. So I think it is probably is premature to be like, hey, here's the right way to do it. And I think also a lot of people are still just figuring it out. And so I do also think like demystifying the idea that there is a right way to do anything right now is important because no one knows. Like we're all just stumbling around.

Uncertainty, history of technology, and co-evolution

I mean, I do think this is for me the most uncertainty I've felt about how a technology might evolve. I was also like, I emerged like from college like post-internet. Like I don't know, people, I think some people compare it to like early internet days, which was like, I didn't quite experience that. But I think for my, for where I am in my career, I think this is the moment of the most uncertainty.

What I will say though, a fun fact is I studied the history of science and technology and there certainly are moments throughout history where there was immense periods of uncertainty about emerging technologies. So I don't think that this is like one for one in the history of how humans have used tools. I think it's a very big one though and certainly like the biggest in my lifetime so far.

But it's a pretty common pattern actually. When like a new technology emerges, whether it's like the car or like the fax machine or whatever, there's like a period of misuse of the product usually, right? Like, I mean, there's classic example of email, but where technology like is changed by the users.

I think like the sort of like simple narrative of the emergence of technologies is usually like the founder myth of like, hey, this one person invented X and they knew exactly how it should be and how it was used. And it's like, they get all the credit for inventing like the thing. But in many, many, many cases, the technologies get like fundamentally altered by the people who use them. And like that usage of those things like changes the trajectory of them in like unexpected ways.

And that it's a very much like a co-evolution once technology has emerged between the technical practitioners, the users of the technology, and often like regulation. So like that's also not unusual where you get people like how do governing bodies start to play into how the technology can or can't be used. So we're at that moment right now where like all of those things are a little bit up in the air .

And I think what's very cool is AI is something that is so clearly not like a founder myth. It's not like there's one person who's like, and yes, now you all can benefit from my great tool. And we're going to see all of these like little, you know, mini founder moments of people being like, I just developed that or this or that. But it's such a collaborative moment, really, when you think about it between like all the way different ways people are like sharing what they're doing and being able to kind of inspire each other.

And I think what's very cool is AI is something that is so clearly not like a founder myth. It's not like there's one person who's like, and yes, now you all can benefit from my great tool. And we're going to see all of these like little, you know, mini founder moments of people being like, I just developed that or this or that. But it's such a collaborative moment, really, when you think about it between like all the way different ways people are like sharing what they're doing and being able to kind of inspire each other.