Real-time Programming Talk
Follow along with this outline:
​
What is Real-Time Programming?
-
Suppose you have two browsers to choose from.
-
Browser A loads every page in 4s.
-
Browser B loads every page in 0.00001 seconds, except once in a month
when it takes 60 seconds.
-
-
Which browser would you use?
-
Now suppose A and B are not browsers but nuclear reactor control systems. Which one would you use?
-
Notice that we did not change the behavior of A and B.
-
A real-time program is when you care about the worst-case performance. Unlike general programming, where you care about the average case.
-
Sometimes people think that real-time programs are fast. They’re not. They’re predictable. In this example, A is more predictable but slower than B, so A is real-time.
-
When performance is part of correctness, that is, being late is as bad as giving the wrong answer, that’s a real-time program.
-
Real-time comes in hard and soft versions.
-
Hard real-time is like a plane that crashed because the software didn’t respond in time.
-
Soft real-time is like capturing a video with dropped frames, where a moving car appears to suddenly jump ahead.
-
I did only soft real-time.
Real-Time Programming is Very Hard
-
It took one quarter to make Futurecam work real-time after everything else was working correctly: Image capture, image processing, average case performance, edge case handling, everything. Just going from good average case performance with occasional hiccups to good worst-case performance took me one quarter.
-
Never underestimate real-time programming.
-
If you can find a way to avoid real-time programming, such as capturing a video and processing it afterward, instead of processing in real-time, take this option, even at the cost of UX.
Techniques Used in Real-Time Programming
-
Multithreading is mandatory. Having a single thread, even with async APIs like JavaScript, isn’t enough.
-
Sacrificing UX to some extent, like responsiveness.
-
In Futurecam, we had some processing that needed to be done in real time as the user is capturing a video. But the algorithm for that — let’s call it X — wasn’t fast enough to run in real-time. So I used another algorithm — let’s call it x — which did only part of the work, and the remaining work could be done after the user stopped recording, at which point it’s no longer a real-time problem. This post-processing algorithm Y took up time, so x + Y > X, but it was still worth it.
-
See if you can make just one part of the program real-time rather than the whole thing.
-
See if you can reduce the scope of the problem in any way possible. For example, Futurecam’s goal was to work perfectly on the latest iPhone. Dropping frames was okay on older iPhones.
-
Abstraction is important, because real-time programming is generally used for complex problems, and making it real-time makes it more complex. So you need abstraction to handle it.
-
Peeling under the covers of system abstractions to understand how they work. You can’t just treat them as black boxes. This is hard with closed-source components like iOS, but you still have to do it, learning the hard way from experience, experimentation, forming hypotheses and testing them (like a scientist), learning from other people’s experience, going back and validating, etc.
-
Warming / precomputing.
-
Time complexity analysis. The algorithms we used was linear in the number of frames, not quadratic.
-
Coming up with design principles, documenting them and sticking with them. For example, we used a video queue or thread to process the video. We had a rule that said that only real-time processing can be done in the video queue. The video queue cannot block for other queues, because it’s not real-time any more. Other queues can block for the video queue. You need to identify and come up with such architecture.
-
GPU and/or vector computing. In fact, in Futurecam, I built a framework that distributes work between the CPU and GPU.
-
Low-level integration with the OS, using Apple frames like Core Image for GPU computing, vImage for vector. Non-portable code. If you want to do cross-platform image processing, you can use a library like OpenCV, but that is three times as slow.
-
Strictly limiting of resources like memory, and reusing. The first version of code I wrote end up allocating 1GB of memory every second.
-
Timelines measured in years, not weeks. Flexible project management. Non-pointy haired boss, as opposed to a (project) manager who demands, “You said it will take 10 days, and it took 11 days and it’s still not done!”

Let's talk about what I can do for your startup
Outcomes-based engagement
Is your startup surpassing your milestones, but your tech is not keeping up?​​ If so, this engagement may be the solution for you:
​
I'll start by understanding the outcomes you want. We'll write them down, and order them by priority. We'll brainstorm and iterate on them, perhaps by creating two lists of outcomes and comparing them side by side to bring us clarity on what's truly important. We'll also seek input from your cofounders, advisors and functional leads. Once everything is clear, we'll lock in the outcomes, and start executing.​
​​
I'll work up to 32 hours a week, and the fee will be $20K for the first 5 months each, and nothing for the rest of the year.​​​​​​​​​​