19 Sep 2020
I’m still working out what it means to be good at your job, but we can probably all agree that there’s a strong correlation between people who are good at their job and people who are constantly honing skills related to their job.
If that’s the case though, how does a PM hone their skills? The PM skillset is often so broad it’s hard to pin down any one thing to focus on that will make a tangible difference—you need to understand certain technical concepts, sure, but you also need to communicate well, learn what your customers are truly interested in, form opinions on what’s most important for your product, etc.
Software engineers and data scientists are lucky in that their day jobs rely on skills that they can hone outside of the 9-5. For SWEs, opportunities for side projects/open source projects abound. Data scientists can learn new methodologies, go deeper into statistical theories, heck, they can do Kaggle competitions. But what are parallels for a PM job?
Although there isn’t a straightforward answer, I’d say the best answer is starting a software business. What’s a better way to develop customer empathy, ruthlessly prioritize features, and develop a product vision than boostratpping a software company? I see a number of benefits:
- You develop an owner’s mindset
- You learn to obsess about a customer’s problems (because how else are you going to sell them a product?)
- You gain experience in multiple disciplines, like sales, marketing, finance, engineering, etc.
What do you think? Is there a better way for PMs to upskill?
06 Sep 2020
I’ve spent one year learning Chinese on Duolingo. It’s a well-designed product that effectively encourages users to build a habit around language learning. As any reasonable PM would say, though, no product is perfect—there is always room for improvement!
Here are some improvements I’d look into making if I were a PM at Duolingo.
Replacing Leagues with XP Challenges
There is a lot of gamification in Duolingo. There are a few different motivators Duolingo uses to increase app usage:
- XP: points earned for completing a lesson.
- Crowns: earned by completing five lessons from a particular skill/topic/vocabulary group.
- Streaks: earned by completing lessons . This used to be reaching a daily XP goal every day, but some time in the past year it changed.
- Gems/lingots: earned by reaching a daily XP goal or watching ads. Gems can be used to buy certain items, namely streak freezes (which allow you to keep your streak even if you miss a day) and outfits for the Duolingo mascot.
- Achievements: earned by doing certain tasks (e.g. uploading a profile photo, doing lessons on the weekend) or reaching a certain quantity of XP
Another motivator is leagues. Leagues are groups of 50 users who compete to get the highest XP totals over the course of a week. The top 10 users with the highest XP counts are promoted to the next league; the bottom 5 get demoted. There are 10 leagues in total (from lowest to highest: Bronze, Silver, Gold, Sapphire, Ruby, Emerald, Amethyst, Pearl, Obsidian, Diamond).
Problems and Resultant Anti-Patterns
Leagues have motivated me to binge the app before, but I suspect this sort of bingeing isn’t sustainable. There’s something incredibly frustrating about pouring your heart into a league for a week only to get bumped out of the top 10 on the last day by somebody else at the last second. Even worse is missing out on an achievement (i.e. the “Legendary” achievement, which requires you to get 1st place in the Diamond league) because somebody else just so happened to have more time on their hands.
When a friend started on Duolingo, they were curious about leagues. I told them they were a recipe for burnout—the higher your league, the more competitive your opponents will be, and the more you’ll have to do to keep up. There are strategies to get around the rat race, but they actually result in using the app less. That’s an anti-pattern.
Here’s an example: I wanted to get the Legendary achievement, because it’s conceivably the only remaining achievement that I won’t automatically get through continued use of the app (the remaining achievements I have to unlock are a function of XP and words learned, which I’ll passively get as I progress through the Chinese course). Given that the self-selecting group of people who make it to Diamond League are already pretty competitive, getting the top spot is uber-competitive, especially considering a rare achievement is on the line (don’t believe me? Read this post).
My workaround to avoid the rat race? Don’t complete any lessons between Monday and Saturday morning1, and then binge for the entire weekend. Passing on lessons for the majority of the week puts you into the less-active side of the Diamond-league user spectrum, so overall XP counts will be much lower (and therefore much easier to conquer with concentrated effort). This strategy worked really well for me (if we’re defining “well” as minimal effort for maximal reward): the 2nd place user in the Diamond league that I won finished the week with about 1600 XP. Compare that to stories on Reddit where greater than 1000 XP in Diamond is needed to not get demoted.
If one of a product’s KPIs is retention, and a feature rationally encourages less usage of the product, then chances are it’s a suboptimal feature. I’m guessing Duolingo has some data suggesting Leagues encourage retention, but my question is: “Over what time frame?”. I bet Leagues lead to short-term bingeing at the expense of long-term retention.
Are there any examples of features that encourage users to take a break in the short-term for the benefit of the long-term? Yes. Apple’s Screen Time is a perfect example:
Apple was not concerned with people using their devices less as a result of Screen Time, because it aims to provide the best usage experience, not the longest. “We don’t need to make you use it every minute of every day,” said Joswiak. “Our business model doesn’t depend on how much you use your devices.” (source)
Now, I think it is in Apple’s best interest for their users to use devices more, at least a little bit. Greater iPhone usage means more digital purchases for which Apple gets a 30% cut, plus an accelerated depreciation of the phone that leads to device repair or replacement. But what Apple is doing by introducing features like Screen Time is playing the long game—they think they can make more money from loyal customers by caring about their health and wellness compared to letting customers binge use products and grow disenchanted. In other words, they think sustained usage is better than binge-y usage.
For Duolingo, what would a good league replacement look like? Ideally, it would reward a user’s effort in a vacuum—that is, your reward would not be a function of how your effort compares to others around you. Additionally, it would encourage long-term retention instead of short-term binges in app usage.
I think something like “XP challenges” would fit this description: something where the user is encouraged to hit a certain amount of XP in a given week in order to advance to higher levels. You can still make hard-to-earn achievements with XP challenges—just set a wild XP target (5K, maybe?) for a given week. Users can go after the achievements when they want and not worry about gamesmanship from others.
Feature Parity Across Courses
When you read Duolingo forums or talk with friends who are learning a different language, one thing becomes clear: not all courses are created equal. Some features exist only in certain courses. Take Stories, for instance, which are a way to earn XP through answering questions associated with a (you guessed it) story.
As of this post’s writing, Stories are available in the Spanish, Portuguese, French, and German courses for English speakers, which reaches 54.4M speakers2 but not for the other 32 courses (who have a collective 47.7M users).
I’m sure there is a ton of work that goes into launching new experiences like this, and I trust that the Duolingo team prioritize languages which with the new experience will have the greatest impact. But hearing a certain feature isn’t available for your course leaves a bad taste in your mouth. It disincentivizes retention—which is why feature parity across courses is so important.
Side note: Stories are generally regarded as easy XP. If the previous section didn’t persuade you that leagues are bad, this point should help. If certain users have access to easy XP and others don’t, then users aren’t on a level playing field when competing to get the highest XP totals in a given week, which is what leagues are all about!
Standardizing the Advertising UI
Just like many other tech companies, Duolingo monetizes non-subscription users through ads. Optional viewing ads (i.e. ads that you can exit out of within a few seconds) show up at the end of each lesson. Users can also choose to view ads to double gem rewards, though these ads generally require viewership for 15 or 30 seconds before you can claim your reward.
Each ad I see on Duolingo has a slightly different UI: there is no consistency with where the video content, call-to-action, and cancel button are placed. I have a hunch this lowers click-through rates for ads—and thereby lowers ad revenue for Duolingo—because users have to take on the cognitive load of parsing the page before being persuaded by the content.
Duolingo doesn’t run their own ad network–they instead serve as a property for Google and Facebook-served ads—so there may not be much they can do about this. But it would be an interesting scenario to A/B test, if possible.
Normalizing Sizes for Matching Exercises
This recommendation is a nit-pick, but I may as well list it here. At least for Chinese, there are five different exercises Duolingo uses to teach:
- Foreign-native translation: translate a written Chinese sentence using an English word bank.
- Native-foreign translation: translate a written English sentence using a Chinese character bank.
- Character selection: choose the character for the presented pinyin.
- Character-pinyin matching: match the Chinese character to its pinyin spelling.
- Listening: write out what’s spoken in a Chinese audio clip out using a Chinese character bank.
- Speaking: recite a Chinese sentence.
My problem is with (4): users have to click on very small tiles to do the matching. There’s a ton of space left on the screen that would make the experience easier on the fingers and eyes—just take a look at this screenshot. I’m optimistic this will get fixed, though, as I’ve seen a re-designed experience a few times now. I’m guessing they’re A/B testing the re-design before releasing it for good.
Unify the Web and Mobile Experience
Duolingo has both a mobile app and a website. Each platform has a slightly different experience and feature set, which I bet leads to a lot of unnecessary overhead for the engineering team to support3.
One example: users have a different currency on each platform (lingots on the web compared to gems on mobile), each with different accumulation rates: gems are won in random quantities after you reach your daily XP goal, whereas you get one lingot for every multiple of ten you extend your streak to (i.e. a 370 day streak gets you 37 lingots).
Certain features of the web app aren’t fully built out either. For instance the web app has keyboard shortcuts for some of the exercise types listed in the previous section, but not all of them—that’s an accessibility problem. I’d be a die-hard web user if there were keyboard shortcuts for each exercise type. It’d let me iterate through lessons so much faster!
1 The league you’re placed in is determined by the time you finish your first lesson for the week. If you finish your first lesson for the week fewer than 24 hours prior to the league ending (sometime on Sunday evening), you’re not placed in a league that week. ↩
2 Portuguese seems like a strange language to prioritize for Stories, given that six other languages (Japanese, Italian, Korean, Chinese, Russian, and Arabic) have higher user counts. Maybe the lexical similarity between Spanish and Portuguese made it easy to port Stories over from one language to the other (this blog post suggests this is the right answer) .↩
3 What, you don’t think PMs care about engineering overhead? Of course we do! It’s all about opportunity cost. Time spent maintaing two distinct experiences is time spent not building features that will move the needle on the product’s KPIs. ↩
15 Sep 2019
I completed my thru-hike of the Appalachian Trail one year ago today. Here’s a quick-and-dirty guide for how to do it fast.
Have a plan
The only way you’re going to hike over 2,190 miles in sixteen weeks is if you have an idea of how far you need to hike each day. This naturally manifests itself in the form of an itinerary, which lists out where you plan to stay each night. You might know David “AWOL” Miller from his best-selling trail guide, but few people are aware he has itineraries on his website for free. You can give my itinerary a try as well.
Some people will say having an itinerary takes away the magic of thru-hiking. These are the same people who will be eating your dust.
The average thru-hikers takes between five and seven months to complete the A.T., with an average base weight of 19 pounds. If you want to go faster, you have to go lighter. Here’s an algorithm for cost-effectively getting a lighter pack:
- Weigh out everything you have in your pack, and find lighter alternatives on sites like OutdoorGearLab.
- Figure out how much weight you’d save if you bought the alternative, and divide it by price to get a savings-to-price ratio.
- Buy gear in descending order of the savings-to-price ratio.
- Stop when you don’t want to spend any more money.
My base weight was around 13.6 pounds. Get to a weight like that and you’re golden.
More than 80% of people who start an A.T. thru-hike don’t complete it. There are lots of reasons behind this: injury, improper budgeting, getting homesick. You can help yourself avoid some of these issues by being consistent. Learn the range for your pace and stick to it; set a budget and stick to it; make a plan for seeing friends and family and stick to it. If popular books are any indication, we’re just starting to realize how important habits are. By forming a good set of habits on the trail, knocking out miles will be the norm for you, not the exception.
Have a support system
If I can say one thing for certain, it’s that I wouldn’t have been able to complete the A.T. without my support system: my family. Here’s an incomplete list of what my family did to support me:
- My sister gave up her bed while I stayed at her home in Philadelphia.
- My brother drove 5 hours round-trip to drop off water purifier when I unexpectedly ran out.
- My parents drove [to North Carolina, New Jersey, New Hampshire, Maine] and took zero days with me.
- My brother let me stay at his place for a week while I recovered from illness.
Your support system needn’t be your family! It can be friends or relatives in various states, or even just some kind people in trail towns. Whoever it is, try and get help from them when you need it. You’ll be surprised how often people are willing to lend a hand.
Take help and be thankful
You may think that you’re completing the A.T. all by yourself, but the truth is you’ll be propelled to the terminus by a group of strangers.
Whether it’s a family that buys you a meal after a bear eats your food, or a day hiker that gives you a Clif Bar, or a woman that drives you to a gas station to re-supply, there will be lots of people that show up for you as trail angels. I recommend taking down their name and address and sending them a thank-you note when you complete your thru-hike (the photo to the left is all the thank-you notes I wrote when I finished). Not only will it make them smile, but it will also motivate you to keep going when times are tough—after all, you don’t want to disappoint them.
That’s all I have for now. If you’re considering hiking the A.T. and want to talk through any plans, feel free to reach out. My email’s at the top of my CV.
22 Mar 2019
I started my job at Microsoft five months ago today. I had the opportunity to write a small post about what I do for Wes Weimer, my former software engineering professor. I thought I’d share some of it here.
Hi everyone! I’m a program manager (PM)1 at Microsoft, working on high-performance computing. Professor Weimer asked me to write about my job to give you some industry perspective on what you’re learning right now.
Large software companies have an interesting problem: they have a lot of work to do on each of their products. Whether that’s implementing a new feature, improving performance, or working with another team to create a product integration, there’s often so much work going on that it’s hard to prioritize what to do next. That’s where PMs come in!
For every 5 to 15 software engineers writing code to build or improve a product, there’s one program manager figuring out what those software engineers should be working on next. With that comes a whole other bunch of responsibilities: specifying the behavior of new features, working with partner teams who have a stake in decisions you make; representing your product in meetings with finance, marketing, and documentation folks. The list goes on and on.
Everbody has a different definition of what PMs do. Many people say a PM’s job is to be “the voice of the customer”, but that makes software engineers sound ignorant of the customer, which they’re not. Software engineers and PMs both have the needs of the customer in mind–it’s just that PMs have a better understanding of what customers need most because they’re busy prioritizing everything that needs to get done to improve the product.
At least at Microsoft, PMs don’t actually tell software engineers what to do. Rather, program managers persuade software engineering managers2 that certain things need to get done. If our arguments are persuasive, software engineering managers will allocate their software engineers to work on what we think matters most. Software engineering managers often call this “funding” a certain piece of work (i.e. “we’ll fund that feature you seem to care about so much”).
From the perspective of a PM, slide 6 of the “Requirements and Specifications” lecture really drives home an important point: if a mistake happens and it’s not discovered until later, it becomes very expensive to fix. So expensive, in fact, that you won’t be able to deliver on all the other features you promised (and who likes breaking a promise?). As a result, priorities need to be reset, and somebody–you, a friend, that team you were working with–is going to be unhappy.
That’s why PMs are often assigned to write specs–a document that describes [in painstaking detail] the requirements for how a new feature or integration is going to work (see slide 33 of the “Elicitation, Validation and Risk” lecture). There’s no doubt that software engineers could write specs if they wanted to–it isn’t too hard. But why burden them with more work when they could be coding?3
Feel free to ask any questions in the follow-up section. I’ll answer as best as I can.
1 Depending on where you work, PM can mean program manager, product manager, or project manager. Each of those job titles means something different depending on who you ask. ↩
2 If you become a software engineer, your boss is a software engineering manager. At Microsoft, software engineering managers assign programming tasks (“work items”) to the engineers that report to them (their “direct reports” or “directs”). They’re often responsible for leading design and [re-]architecture efforts as well. ↩
3 Probably the easiest way to describe what a PM does is “anything that would get in the way of a software engineer from checking in code”. ↩