ANGELA BUSHESKA: Yeah, absolutely. Thank you so much for having me. Super excited to be here and super excited to have spent three months—time flies!— as a researcher along with the project. So my name is Angela. I am originally from North Macedonia, a very small country next to Greece in the Balkans. And I spent my first, like, 16 years thinking that I would become a mathematician, really focused on math Olympiads. And then I moved to the capital city of my country, which is Skopje, North Macedonia, and that year, in 2019, we had the greatest pollution. We were, like, on the top of every single list in terms of pollution, but honestly, I didn’t care. I was like, I’m going to do, like, math Olympiads, win every medal. I moved in with my aunt, who was living very close to the city center, and she had a lot of cardiovascular difficulties. And then this air pollution clogged her blood, and she passed away. And that was the moment when I saw like, I love math, but this air pollution took my aunt, so I have to do something. So I made the hard decision that junior year going into senior year to cut my participation in math Olympiads after seven great years there and focus on the climate. I really just started to understand where is the pollution coming from in my area. I started to realize that fast fashion had a huge influence, and then that’s why I started, like, getting into this field, getting into research, started a climate tech nonprofit called EnRoute, where our mission is to debunk the fast fashion supply chain and let people but also corporations know what is behind their clothes. So we, kind of, help both sides. We help people with their shopping choices, but we help a lot of corporations and policy to understand either what’s behind their supply chain and change it or how to better communicate their supply chain because there’s a lot of greenwashing that is happening in this space. So it really started as a rag-tag group of teenagers, and this is my fifth year working at it. So I really had a passion for both understanding the research side of it and the nonprofit, like, getting people to learn more about it. So I think that marked really my presence. And then coming into the US in 2021, I started to study both electrical engineering and computer science just because I love this intersection of building and doing stuff in practice and a way to apply my math knowledge into real life.
RANGANATHAN: Wow, so early in life you have such amazing goals. That’s impressive. So, Angela, we connected after Madeleine Daepp, a researcher who was involved in the intern selection process, introduced your work and your passion for sustainability to me. I saw your passion for this area and, you know, your enthusiasm in the first conversation—we were already working on ideas, right! I knew I had to work with you. But for you, there are many other internship opportunities. I would love to know what excited you the most about Microsoft Research.
BUSHESKA: Absolutely. Yeah. So as I said, like, I started getting into this area of sustainability by myself. I really didn’t have a structured way to learn. So when I heard that there is an opportunity from people who have, like, previously worked in this area, are doctors in this area, to learn along them, that was definitely one. But also throughout my life, I had like entrepreneurship stints. I was selling lemonade when I was [LAUGHS] 6 years old; started a nonprofit. And this was a great opportunity for me to start a zero-to-one project while being an intern, which is, I would say, a once-in-a-lifetime chance. We had a chance to define the problem, see solution, try things. I was not part of just one project or building one feature. I had a chance to drive it along you and the team. So that was, I think, the main reason why.
RANGANATHAN: I’m glad we were able to attract your attention here. [LAUGHS] Before we get into specifics on, you know, exactly what we did this summer, I’d like for us to talk a bit about your internship setup, right. I think it allowed for some really unique experiences, and, you know, I would love to have you share your internship highlights. I think there is one in particular that I’m thinking about, which I really hope you’ll share.
BUSHESKA: Absolutely. I would say this was an internship like no other experience I’ve had in the past. So just for context, I was interning in New York while part of the team being in Brazil, part of the team—you and the other researchers—being in Redmond, a couple of other researchers, like, being technically in New York. So I had a chance to interact with all of these different time zones. Also for an additional context, I was a part of undergrad research internship group where they brought us to Seattle from New York—and also there are other interns from Boston—to spend like a week and learn more about Microsoft, Microsoft Research, the leadership. So that was an incredible opportunity. Also, I had a great opportunity prior to that to come to the team meeting to see farms in real life, interact with farmers, interact with the team in person, which was also another great opportunity. And I’ll come to the one that you’re referring [to] here [LAUGHS]. So the second time when I was, like, visiting Seattle, I remember a week prior, it was Friday, we were finishing up some stuff. You actually pointed out that I should definitely message some of the leadership people to interact with them and hear their insights. And it was EST time zone, somewhere around 8 when I left the office, and I shoot an email to the chief sustainability officer, Melanie Nakagawa. I was like, “Hey, my name is Angela. I’m a research intern. I have worked in, like, fast fashion in the past. Now I’m working with Vaishnavi and the team on this food project. I’d really love to meet you next week. [LAUGHS] I’m coming for like three days.” I sent that email. I was like, there’s no way I’m getting response from this. And then on Sunday, I traveled to Seattle. And I remember during the first day, assistant of the chief sustainability officer says, like, there is this Bloomberg Green Festival that is happening in Seattle on Thursday, and we would love for you to come and meet Melanie. It was, like, I don’t know. It was like real life or not. I really couldn’t … it took a lot of time to process. And after that, one thing additional that was a barrier was how to get into this festival because this festival had a ticket of like $300, $400. And I think that by that time everything was sold out. So I was like, I would love to, but I cannot just get into the Bloomberg Green Festival without a badge. So then I realized there is this sustainability community at Microsoft that is like 8,000 employees or something, and they got badges previously for the festival, and everyone who is working there can just get the basic badge just to get into. So I emailed a couple of people from there; they set [me up] with a badge. Huge props also to the undergraduate research team who extended my stay for a day because I was supposed to leave for New York the other day. So a couple of miracles happened there. I had a chance to go to the Bloomberg Green Festival, had a chance to meet with Melanie and her assistant, Tyler, who was working also in this area of sustainability in her presence on, like, these events. It really goes to say how an undergrad intern can, like, meet with leadership and learn from the leadership to the place and the opportunities we had during this internship. Thank you so much Melanie and Tyler for taking out the time at the Bloomberg Green Festival. I know it was a super packed day for both of you, but thank you so much for taking the time.
RANGANATHAN: Thank you for sharing that. I want to point out something. It’s amazing how the leadership makes time and they really value every person’s work, and that’s an internship which is accounted for now, right, and you know your work is meaningful to the company. I think Microsoft is so big that people don’t realize there’s this community of 8,000 people in sustainability often, so it’s amazing. It blows my mind every single time, and kudos to you for following up on my late-night comment. [LAUGHS] Um, Angela, so this internship falls under larger efforts within Microsoft Research to establish a sustainable agriculture and food supply chain, a project that we know as FoodVibes. In your work this summer, you focus specifically on enabling technology to meet the new European Union’s regulation around deforestation-free products. Could you tell us, in your words, about these regulations and how you think the work can aid in their implementation?
BUSHESKA: Absolutely, yeah. So the European Union’s regulation on deforestation-free products, also known as EUDR, is a legislation that will prevent food linked to deforestation to enter European Union borders[1]. But how will the border officials know if something is coming from a deforested area or not? On another side, it’s also the farmers from this area who don’t know what exactly is a deforested area, especially the ones who moved after maybe an area has been deforested. So there are a lot of questions around there, and I think that getting into and understanding the problem was a huge part to continue after that in building the solution.
RANGANATHAN: Yeah, and I believe that having a large team here really helped because we all often had discussions around many of these areas and we really valued your inputs and how you participated in these discussions, right. So through your internship, we’ve seen that there are a variety of tools, you know, and vegetation metrics at our disposal to help determine what exists on a plot of land because that’s what we want to identify. There’s, like, satellite imagery. We get the normalized difference vegetation index, and then there’s the USDA’s cropland data layer, which exists purely for the US. But, you know, we are getting an incomplete picture here, at least as it pertains to this particular use case. How do you think TerraTrace brings these pieces together, and what are some of the specific challenges or motivations of the core of your work?
BUSHESKA: Absolutely. Yeah, I think, like, a couple of the first weeks, we tried these great machine learning models, and one thing we realized is that when we have, like, orchards, and when we had pine trees, and then when we had, like, regular forest, all of them are classified the same, which was a really great light saying, like, yeah, these are great. These are models that are trained by millions of data. They still don’t work as we want to. And also, I think that in an era where everyone is, like, training big great models, we kind of decided to take another route and say like, models are great. What if we go step by step and maybe we don’t need a full model round. Maybe we can go step by step to a certain point to try to understand deeply. And I think I’m really grateful that we took this direction because we had a chance to understand at a granular level what is actually happening. Here we had, like, we were lucky at the beginning to work with farms in Washington. We had, like, an actual farmer who was in the team who we have, like, pictures and we had all the data. So that was a great ground truth for us to understand, OK, this is how the vegetation looks on a farm. This is how the vegetation looks on a forest across years. So we had these snapshots and then from then on, we just moved along to say, well, if we know about this, what’s happening in Washington, how we can scale now to other places in the US. We looked deeply into California just because it has a lot of, like, agricultural diversity. And I think that this step-by-step level brought us to create TerraTrace. TerraTrace is a platform where one can enter, like, the coordinates of a farm and see what has been happening on a given piece of land across time. Now, TerraTrace combines a couple of different things. It combines mathematical results, combines LLMs, and combines just, like, the basic information of risk. And I think this is good just because it’s not a heavy computational platform that we have. And if we need to use it on, like, a couple of years from now on a farm, we technically can. If we need to use it, like, in a legal office, we can. Just because it doesn’t depend a lot on data and it’s, like, easily portable, which is another plus, and differentiator compared to what exists outside.
RANGANATHAN: Do you want to share a little bit about the signature curves that you identified?
BUSHESKA: Absolutely. So when I just, like, realized … there is a metric called NDVI (Normalized Difference Vegetation Index), which is basically measuring vegetation. It is a time series of vegetation. What we did with NDVI is we tried to, like, measure it across time on different places. And I think in coffee, it had the greatest impact. Just because there was a coffee in Vietnam, coffee in Honduras, completely two different places. And then when we analyzed and saw the shapes, we were like, wow, they are completely two different places; however, they still follow the same trend and that was a great calling to say, like, this is a greater metric. Maybe if, like, vision computer models failed, this is something that is, like, very rudimentary. You wouldn’t expect that just a very simple index would calculate that, but we saw it working for a lot of different places, and I’m particularly grateful for the places that were not together at the same place.
RANGANATHAN: Yeah, and I think, like, looking at this, like, temporal, over-time picture is the unique aspect of what you built and that was what unlocked it for us, right. So you hinted at this. In developing the TerraTrace platform, you ended up combining large language models, or LLMs, and statistics. What did you like about this approach? What did each bring to the table? How did it contribute to your ultimate goal? Could you share a bit about?
BUSHESKA: As I said, like, at the beginning, I’m a huge math person, and I truly believe that math has more power than we credit today for. So one thing that I started building, when I started building, I was just, how can we get as much information as possible without LLMs and then give it to the LLMs? That was kind of the approach that I started. So we have this, as you mentioned previously, crop data layer that was the base truth. So I could always go back and compare to see if it has, like, if we are doing the same thing. Then we had the signature curves that we could compare to base signature curves to understand, OK, it’s coffee or it’s a farm. And now when we were having a farm, we had a problem because corn looked the same as wheat. Multiple crops look the same. So I needed something more to differentiate what exactly is there. So by math, I managed to understand like the growth rate, the fallout rate, what is the percentage when the curve is up, how much percentage is down so every insight that we could get by just basic statistics. And after that, all this combined knowledge was given to the LLM to, kind of, just confirm that the math is right and give us, OK, greater insight into, well, your math is good, and after that, based on your math and everything the LLM knows, well, there is this probability that it’s corn and after that we came to the crop data layer to say, yep, it’s corn. Referring to corn just because this was our example demo. Also, LLMs are changing very rapidly today, so like GPT-4, GPT-4 Turbo, expecting a GPT-5, so constant development needs to be done, and I think that if something is, like, changed and doesn’t work in the platform, one thing that will always work is the statistics. So having something to always refer to was a very interesting process.
RANGANATHAN: Yeah, I really appreciate, you know, how we had the focus on, hey, let’s do what we can with math first because, after all, this is a sustainability project, right?
BUSHESKA: Exactly.
RANGANATHAN: So LLMs were great at summarizing this, taking all the data and, like, giving you the outputs. I think that was a very interesting approach, too. So there are several existing pieces of work we’ve seen in this area. We’ve relied on a lot of literature in this specific application, right, that ended up havingauthors who were affiliated with Microsoft. Could you share a little bit about those works that we drew on and how we leveraged the Microsoft connections?
BUSHESKA: Absolutely. One thing that was something I was not expecting was reading the papers and then hopping on a call with the authors. So one thing that I would credit is first we understood … there was this, like, GeoLLM group. They actually had a presentation in our weekly meeting for the group. So it was, like, me just staying in the presentation and trying to learn. And the other day, they send out like, this is a research paper; take a look at it. And after, like, we realized that the majority of the authors were in the same building as you actually, we had a great chance to, like, meet, like, with them and understand how …
RANGANATHAN: This is SatCLIP, right?
BUSHESKA: … SatCLIP (opens in new tab) and GeoLLM (opens in new tab), both of them. Trying to understand, like, how they have built it, what they have built, why, how they are planning to continue, how I am able to use it. And, actually, because when you’re reading a paper and you’re reading GitHub repo, it’s one thing. [LAUGHTER] When you’re speaking with the author and seeing, like, this is why we’ve built it, here’s the limitations, be careful how you’re using it, it’s completely another thing. So I am really grateful for this situation. But also, not only, like, inside of Microsoft, we also had a chance to work with a lot of other folks. For example, we read a lot of papers from research groups in universities. And because our interns came from these universities, it was, like, great to understand from them also what they have built.
RANGANATHAN: Nice, and the other interns whom you worked with, which we’ll come to. So for this specific work, we were motivated by the use case. What were some of your findings, and also how are you envisioning this work as a foundation for other supply-chain or even, you know, broader sustainability scenarios and even beyond those fields that you can think of?
BUSHESKA: Yeah, so one thing that we, as a conclusion, came out from here is that signature curves are not just by look. When we tested that with a lot of different other signature curves, we realized, OK, this is true because in most of the cases, they worked. Obviously we had some failures, like citrus signature curves didn’t work because of other reasons that happened on the farm. But for most of them, we got same results on what’s happened with the signature curves with what happens actually on the farm, which is a great way to further this exploration here. Another thing that we realize from this project is now the group previously has worked on trying to understand supply chain, like having the tracking part and having this code to track along the journey, but we were missing the part on what happened on the land. And now that we have this additional way to understand what happened on the land, I think it is a full system of starting this is what happened on the land, this is what happened across the road, and this is where it is now, so that not only EUDR, but even, like, customers in the future will know where their food is coming from, which is that total part of sustainability that we want to get to. Because I think that at this point, we know the challenges, we know the climate change challenges that we are having, and the more information we have, the better decisions we and policymakers would be able to make.
RANGANATHAN: And I think we realized this and you can chime in, Angela, but I feel that the traceability is a vehicle for data. What you build is a means to take all that data and make it meaningful to people, right. Like what does NDVI mean? I have no idea as a layperson, so you can actually get that information out. There were also a few other use cases that you gleaned out of your data input, right. Like wildfire was one of them. Would you like to share a little bit about that?
BUSHESKA: Yes, so we were looking specifically about California, specifically about 2020, which was a season of very, very bad wildfires. And we could see by measuring this vegetation how, like, wildfires affected these regions. Like, you could see how vegetation was dropping very quickly from a great number of one to just, like, flat zero just because everything was there burned. So seeing firsthand on these curves and after that linking up to a lot of other background wildfire data was a good check-in. Like, you could see that it’s not a person deforestation; it’s a wildfire deforestation. And also it is really helpful for risk estimation as we go further along. We could see, like, this past year climate change effects are there. It’s not just like this fancy term in the future. It is there. So I think that having a chance to see in the past and having these models, it’s also giving us a preview for the future. Because ultimately we want this greater food yield so that more and more people can enjoy healthy food. So having a chance to predict the risk would help us to save more food and, yeah, care better about our planet and our people.
RANGANATHAN: I think there’s a lot of potential. The more information you have, the more applications in sustainability you can build. This is merely, like, a footstool to launch it off, right. You’ve said one of the things you’ll miss most from your internship is the density of smart people in the building and in the company. Over the course of your internship, what did you learn? How did your actual experience compare to, like, expectations you had for it?
BUSHESKA: I think it exceeded every single expectation that I had because I would say when you mentioned the density of smart people, it was really incredible to have—and I think that this … maybe my very interesting connection of being in New York while working with people from Seattle and Brazil allowed me to have a broad range of people that I had, like, had a chance to meet. So, like, the New York office, it is very specifically focused on economics and social aspects that I’m not an expert on, but having a lunch with these people every single day, I had a chance to learn a lot about how, like, human data is provisioned. They were building a lot of things for, like, Microsoft’s new projects and new products that are, like, the Copilot and the computers, so I had a chance to hear about those perspectives. Being in Seattle and meeting with interns and researchers in Seattle, I had a chance to learn not only about the farms perspective but also how other projects came to life. With our meetings, I realized how FarmVibes and FoodVibes were once just, like, a small idea and now are these huge projects. So having a chance to understand the history of all the things around me was a great way to see how I am able to build something for the future. And also maybe another thing is the collaboration that happens. As I said, I was, like, surrounded by a lot of socioeconomic people—that I am not really an expert in—but they were really great with providing me advice from an area I would never think of. So when I was speaking, like, from farms to someone who is in economics, they would raise another point on how this can hurt or help economy or to look at it from another side, which is all great perspectives to have when you’re creating a research from scratch. That’s why I really, like, referred to entrepreneurship previously because I feel like even though was an intern, I really had a chance to shape the project in a way that I would get all these insights from people and then obviously decide how to go further.
RANGANATHAN: Yeah, and I want to add summer with interns is the highest energy level at MSR (Microsoft Research) and the best time of my year at Microsoft. So having successfully completed your internship, what advice do you have for our audience, you know, when it comes to applying for an internship like this in industry and then getting the most out of that experience?
BUSHESKA: I would say one of the things that really maybe helped me through the process is, like, working obviously, in an area that I was really passionate about. But another thing, and especially this is coming specifically for the marginalized communities in tech—although I would say MSR is doing a great job in having this balance. But I’m coming from a liberal arts college where we have, like, two girls that are studying electrical engineering—so having this disbalance can in some ways, like, I don’t know, put you, “I’m not good enough” and all this imposter syndrome. So maybe one thing is definitely, like, reach out to people. And something that I really learned heavily from you is reach out. There is nothing you can lose. Most of the time, it is just … getting a no is maybe the hardest answer that you will get. So having a chance to ask is always a good idea. Obviously, don’t go to, like, every single executive to ask questions, even though it paid out—it paid out [LAUGHTER]—and just, like, ask for help from the people around you. Ask to learn. I realized, like, researchers really want to speak about what they have worked on previously. So whenever you get a chance to ask them about their past project, they’re super eager to tell you about like their grad days and how they have become what they are now. So that was also great. And in terms of applying, I think I would … when I applied for this, I was like, there’s no chance I will be able to get in. But for me, as an introvert turned into an extrovert—fun fact, I was a big introvert in the past who turned into an extrovert just because of my work—it’s like if you don’t apply, there’s a 100 percent chance that you’re not going to get in. So, like, when I interviewed, after my first interview, I was like, yeah, there is, like, no chance. Because, like, I presented the projects, but I was like, there are so many other students who I know that are working in these areas and then it happened and then I’m here recording this podcast. I remember it was Thanksgiving break when I applied for this internship. If I was like, no—if I self-rejected myself previously, yeah, this would never have happened. So I would say for all interns out there, just, like, don’t self-reject yourself. Let them reject you. [LAUGHTER] It’s my really advice.
RANGANATHAN: And, yeah, have more confidence. I’m glad you applied and you joined us because this was a highlight of the summer for us, this was, right. So your work with Microsoft Research is one of your several projects. You’ve been involved with many other things, right. What else do you have going on? How does it contribute to the impact you’re looking to have at this point in your life?
BUSHESKA: Yeah. So one thing that I realized in the past four years is that I started … so my, like, journey was very, I would say, interesting and not predictable just because when I, like, graduated from high school, I got trapped by all the COVID pandemic. I couldn’t fly into the US just because of all the visa regulations. It was like, OK, a gap year at home, which if you asked any of my friends before, I was never the person who would take a gap year. So staying that year really, like, without a school or official job was the chance for me to just look at and start my nonprofit, EnRoute. From then on, I worked with policy. We worked at the UN. We even went to Hollywood in March to speak about, like, fast fashion. So I realized there is no one way to make an impact. So I was really glad to join research, to join policy, and speak wherever I can. I think now I’m getting back into my senior year of college. Time flies. And something that I am, like, looking definitely into is how to continue my research in this area, where I most probably would love to be through grad school. Having a chance also to push further this project just because I think that speaking out and sharing research is one of the, like, greatest things we can do, like referring to all the other times when I was speaking on conferences. It will be magical for someone to respond with positive and negative feedback and maybe even can create a collaboration for us to move forward. So that is definitely something that I’m looking forward to. So definitely this area is something that I got in by mistake. I got in really just by … because I had that great time in my senior year to spend more time research, but it definitely something that marked my life at this point. And I think that, like, whatever route I pursue will be to fight climate change in any way or another.
RANGANATHAN: I want to point out we are also planning to open source this and you hope to work with people, you know, to get it in the hands of people who need it and could benefit from it most, right.
BUSHESKA: Absolutely.
RANGANATHAN: So that’s a great direction. I think every year our undergraduate researchers set a new bar in my life expectations. And I thank you for that. This year is another bump up. So, yeah, you are a senior, Angela. What’s next for you, and what’s next for your research and work?
BUSHESKA: Yeah, so this will be a fall really spent with a lot of applications because of, like, graduate school. Definitely something that I’m looking forward, especially graduate school in this area. So we kind of spent a lot of time living in an era of GPT wrappers, and I think when there is a product out there that is really based on research, it can make a tremendous impact. As I said, I am really a person who loves to experiment and do things. So something out there is definitely either, like, having a research that is spun out of this or having perhaps a startup down the line that is based in this area and helping millions of people around the world is definitely something I’m looking forward. Because I saw the impact when you’re helping other people, and I think that is the best thing that one person can do. So I was really excited to do it with all of you this summer and obviously continuing on to open source this project obviously for feedback but also for real use. And if we can help with something, that will be amazing. Hopefully getting these out to conferences around the world. And, yeah, see where it can take us.
RANGANATHAN: Yeah, I have one last question for you. You had some interesting intern mentors, myself included, and then Bruno Silva, who sits in Brazil, Peder Olsen, and Solon Barocas. Do you have anything to say to all of them? [LAUGHS]
BUSHESKA: Yeah, I would say a big thank-you for keeping up with my questions. As all the listeners can probably see, I’m a person who really speaks a lot, even more than needed sometimes. So I would say I had a great balance because I was in a time zone that was between Brazil and Redmond. So I would spend a lot of my mornings, especially in the beginning—huge shoutout to Bruno for dealing with all of my technical questions [LAUGHS]. It was my first time setting up a lot of research clusters. We should not forget also the people at Microsoft who are dealing with all of the GPUs and stuff because I think that managing all of these researchers for all the people out there is a hard job. So having a chance to communicate and also understand. Even though I study electrical engineering, I never had a chance to deeply understand how they’re used in real life. Speaking also with people from all different walks of life because Peder, my other mentor, was someone who has been really experienced in this area, so he would reference projects from I don’t know how many years, and it was, like, so fun to see the history, to see his previous experiences come to life. You don’t every single day get to have like three or four mentors from all different areas and aspects. So I’m really grateful that I had a chance to talk and also get called when things were not right so we can make them better and more correct for the people that will use this project.
RANGANATHAN: I’ll tell them you said thank you. So, Angela, I think we are coming to a close here. Thank you so much for taking the time to share your experience with us. It’s been a pleasure to work with you, and I really look forward to seeing what the future holds for you and what amazing things you’re going to achieve. So thank you. Closing out.
[MUSIC]
BUSHESKA: Thank you so much. Really appreciate to have this opportunity. And for everyone that is, like, working in this area, for every future intern that is doubting themselves, feel free to find me out there and, like, ask. Always happy to help people, always happy to chat sustainability with people. So thank you so much.
[MUSIC FADES]
