> Suddenly turning off services when the billing cap is reached is a big reliability risk for customers.
And yet this is exactly what happens if you exceed a quota limit, which aren't always well signposted and sometimes require days to get the limit raised. I've experienced multiple hours-long outages due to hitting GCP quotas we didn't know existed.
To provide a concrete example, this bit me in a typescript codebase:
type Option<T> = T | undefined
function f<T>(value: T): Option<T> { ... }
let thing: string | undefined = undefined;
let result = f(thing);
Now imagine the definition of Option is in some library or other file and you don't realize how it works. You are thinking of the Option as its own structure and expect f to return Option<string | undefined>. But Option<string | undefined> = string | undefined | undefined = string | undefined = Option<string>.
The mistake here is in how Option is defined, but it's a footgun you need to be aware of.
There was one in a previous AoC that I think stumped a lot of AI at the time because it involved something that was similar to poker with the same terminology but different rules. The AI couldn't help but fall into a "this is poker" trap and make a solution that follows the standard rules.
We all have our writing quirks, like how some people use shorthand for words where there is only a marginal difference (like "people" => "ppl"), or even people who capitalize the start of sentences, but not the start of their whole text.
There's plenty of prior work to go on. I mean, you could use a font ligature or one of the browser extensions (although I don't know if Chrome still lets you have a browser extension touch all text).
Change ChatGPT to 'my drunk uncle' while you're at it.
It affects a certain disposition for the writer; the information it contains isn't in the actual data they are expressing, but rather the state of mind that they express it from, which can be important context. Oftentimes it can indicate exasperation, which is an important social queue to be able to pick up on.
A little excerpt from Arlo Guthrie
"I mean, I mean, I mean that just, I'm sittin' here on the bench,
I mean I'm sittin here on the Group W bench, because you want to know if I'm moral enough to join the army, burn women, kids, houses and villages after being a litterbug."
Imagine that without the "I mean"s in it, and the importance of how they convey his stance on the situation.
Twitch puts the ads directly in the HLS stream, but as seperate segments from the content (a HLS stream is made of many small video files, on twitch they're about 2s long). They're trivial to recognize and filter out (they're actually explicitly tagged as ad segments) but it still won't serve you the actual stream you were trying to watch - the ad segments override it. The best you can do is just block until the first non-ad segment arrives.
Twitch is HLS, but they've tightened the buffers and shortened the segments (2s is standard) so that latencies of down to a couple of seconds is common. It's quite impressive, tbh.
In a weird way it kinda reminds me of `exec` in sh (which replaces the current process instead of creating a child process). Practically, there's little difference between these two scripts:
#!/bin/sh
foo
bar
vs
#!/bin/sh
foo
exec bar
And you could perhaps imagine a shell that does "tail process elimination" to automatically perform the latter when you write the former.
But the distinction can be important due to a variety of side effects and if you could only achieve it through carefully following a pattern that the shell might or might not recognize, that would be very limiting.
this is pretty much exactly how my "forth" handles tail call elimination, and it's the main thing that's added the quotes so far since it shifts the mental burden to being aware of this when writing code to manipulate the return stack.
as you imply towards the end, i'm not confident this is a trick you can get away with as easily without the constraints of concatenative programming to railroad you into it being an easily recognizable pattern for both the human and the interpreter.
I'm surprised they don't see more activity for polar orbits. You want to launch north-west or south-west and into ocean, northern scandanavia seems perfect for that.
Polar orbits have to go west, because they have to counter the earth's rotation if they aim to cross over the pole. Further north means less rotation to counter.
When I'm in charge of an on-call rotation I always try to make it very clear that this is not the expectation.
In my preferred model of on-call, you have a primary, then after 5min an escalation to secondary, then after 5min an escalation to something drastic (sometimes "everyone", sometimes a manager).
The expectation is that most of the time you should be able to respond within 5 minutes, but if you can't then that's what the secondary role is for - to catch you. This means it's perfectly acceptable to go for a run, go to a movie, etc.
You relax the responsibility on the individual and let a sensible amount of redundancy solve the problem instead. Everyone is less stressed, and sure you get the occasional 5min delay in response but I'm willing to bet that the overall MTTR is lower since people are well rested and happier to be on call to begin with.
We have a primary/backup setup and I would be pretty pissed if my primary just started going out for movies or a date night during their shift tbh. My job as a backup is to be there for unexpected events, ie they did not wake up or had an accident. Not be on call effectively 2 weeks in a row just because the primary doesn't take it seriously.
Yeah, going for a run or a dinner where you might be able to ack but not actually at keys for 10-20 minutes is one thing. Going to a movie or date where you might not even ack and won't be at keys for hours? Not cool at all.
I don’t see how this changes the problem where there is an expected guarantee of a rapid response except that now two people are expected to be available and would now need to directly coordinate in order to ensure one person’s going for a swim doesn’t interfere with the other’s WoW raid.
I guess to me that seems worse because that’d effectively double the number of off-hours accountability per teammate. Not only do you need to be first on call for your primary hours, therefore severely restricting the quality of your “free time” but now you ALSO have to be secondary on call for that irresponsible coworker that goes afk without properly communicating for 2 hours, dipping twice into your actual free time.
Out of 168 hours in a week, there are maybe up to 8 where I want to do something that interferes with being oncall. There's no downside real downside to being oncall for the other 160 hours. But I would get a lot of disutility from losing my freedom during those 8.
This is pretty much how it should be done. If the business demands more, they should have a properly manned 24x7 NOC.
You also need *ownership*. There is nothing worse than having to support somebody else's work and not being allowed (either via time or other restrictions) to do things "right" so that you're not always paged for fixable problems. Everywhere I worked where the techs had ownership (which varied from OPS people being allowed to override the backlog to fix issues or developers being given enough free reign to fix technical debt) has usually meant that oncall is barely an issue. My current gig I often forget I'm even on call at all and the main issues that do crop up are usually external.
Almost all the reliability issues I encounter is usually due to constraints ordered by people who don't have to deal with on-call.
Things like, running in AWS but you have to use a custom K8S install so they aren't dependent on AWS.
Using self managed Kafka so that you aren't dependent on proprietary tech.
It all sucks because they are always less reliable and generate their own errors and noise for on-calls.
If they had to deal with phone calls every time there's a firewall issue that had absolutely nothing to do with the application, they would soon change their tune.
So it takes 10 min until you've gone to the drastic solution? With this time-frame it would be risky to go the bathroom, not go to a movie. Also even the backup sounds like a primary in this scenario.
Sure, but the assumption here is that primary and backup (edit: probably, ie. they're not coordinating this) aren't going to the bathroom at the same time. It's also based on the idea that alerts are extremely rare to begin with. If you're expecting at least one page every rotation, that's way, way too often. Step one is to get alerts under control, step two is a sane on-call rotation.
We want to ack within five minutes, and be at a laptop within 30. So long as I'm within mobile signal when the page goes off, it doesn't really matter what I'm doing — an ack is a button press on a push notification. And I can stay within 30 minutes of my laptop and an Internet connection by carrying said laptop and my phone (with "unlimited" data).
If the primary (paid) on-call doesn't catch the notification, the secondary (unpaid) will be paged. And so on, down a couple more steps, to a senior manager. There's no expectation that anyone other than the primary would actually be available to ack the alert.
Having the primary/secondary rotation is arguably worse. In that model, from the perspective of any one participant, now they're on-call for two weeks each time around instead of one.
reply