top of page

Real-time Programming Talk

Follow along with this outline:

What is Real-Time Programming?
  • Suppose you have two browsers to choose from.

    • Browser A loads every page in 4s.

    • Browser B loads every page in 0.00001 seconds, except once in a month

      when it takes 60 seconds.

  • Which browser would you use?

  • Now suppose A and B are not browsers but nuclear reactor control systems. Which one would you use?

  • Notice that we did not change the behavior of A and B.

  • A real-time program is when you care about the worst-case performance. Unlike general programming, where you care about the average case.

  • Sometimes people think that real-time programs are fast. They’re not. They’re predictable. In this example, A is more predictable but slower than B, so A is real-time.

  • When performance is part of correctness, that is, being late is as bad as giving the wrong answer, that’s a real-time program.

  • Real-time comes in hard and soft versions.

  • Hard real-time is like a plane that crashed because the software didn’t respond in time.

  • Soft real-time is like capturing a video with dropped frames, where a moving car appears to suddenly jump ahead.

  • I did only soft real-time.

Real-Time Programming is Very Hard
  • It took one quarter to make Futurecam work real-time after everything else was working correctly: Image capture, image processing, average case performance, edge case handling, everything. Just going from good average case performance with occasional hiccups to good worst-case performance took me one quarter.

  • Never underestimate real-time programming.

  • If you can find a way to avoid real-time programming, such as capturing a video and processing it afterward, instead of processing in real-time, take this option, even at the cost of UX.


Techniques Used in Real-Time Programming
  • Multithreading is mandatory. Having a single thread, even with async APIs like JavaScript, isn’t enough.

  • Sacrificing UX to some extent, like responsiveness.

  • In Futurecam, we had some processing that needed to be done in real time as the user is capturing a video. But the algorithm for that — let’s call it X — wasn’t fast enough to run in real-time. So I used another algorithm — let’s call it x — which did only part of the work, and the remaining work could be done after the user stopped recording, at which point it’s no longer a real-time problem. This post-processing algorithm Y took up time, so x + Y > X, but it was still worth it.

  • See if you can make just one part of the program real-time rather than the whole thing.

  • See if you can reduce the scope of the problem in any way possible. For example, Futurecam’s goal was to work perfectly on the latest iPhone. Dropping frames was okay on older iPhones.

  • Abstraction is important, because real-time programming is generally used for complex problems, and making it real-time makes it more complex. So you need abstraction to handle it.

  • Peeling under the covers of system abstractions to understand how they work. You can’t just treat them as black boxes. This is hard with closed-source components like iOS, but you still have to do it, learning the hard way from experience, experimentation, forming hypotheses and testing them (like a scientist), learning from other people’s experience, going back and validating, etc.

  • Warming / precomputing.

  • Time complexity analysis. The algorithms we used was linear in the number of frames, not quadratic.

  • Coming up with design principles, documenting them and sticking with them. For example, we used a video queue or thread to process the video. We had a rule that said that only real-time processing can be done in the video queue. The video queue cannot block for other queues, because it’s not real-time any more. Other queues can block for the video queue. You need to identify and come up with such architecture.

  • GPU and/or vector computing. In fact, in Futurecam, I built a framework that distributes work between the CPU and GPU.

  • Low-level integration with the OS, using Apple frames like Core Image for GPU computing, vImage for vector. Non-portable code. If you want to do cross-platform image processing, you can use a library like OpenCV, but that is three times as slow.

  • Strictly limiting of resources like memory, and reusing. The first version of code I wrote end up allocating 1GB of memory every second.

  • Timelines measured in years, not weeks. Flexible project management. Non-pointy haired boss, as opposed to a (project) manager who demands, “You said it will take 10 days, and it took 11 days and it’s still not done!”

Let's talk about what I can do for your startup

Goal-based engagement

Is your startup not on track to achieve your next milestone? Are you unsure what to do to get back on track?

Wouldn't it be great if you could get a trusted expert to guide you throughout your journey, help you make critical decisions right, and give you visibility into the consequences of decisions before you make them, so that you can execute with confidence?

Work with me, and I’ll guide you to success.

I'd like to offer you a couple of pricing options:

Option 1: Guaranteed results: I’ll understand your goals, and commit to delivering them. In the unlikely event I fail to deliver, I’ll do whatever it takes to make things right, including redoing the work at no extra cost, giving you a partial refund, or even a full refund, whichever I feel is the appropriate resolution.

  • The fee will be $175/hour, subject to a minimum of $4K per month.

  • The bill fluctuates month to month. In the past I’ve billed as much as $20K in a single month.

  • I’ll inform you every $4K — 4, 8, 12, etc — to avoid bill shock.

  • A typical engagement costs $100K. I suggest you budget $100K to get started, and I’ll let you know if this changes as the work proceeds.

Option 2: Name your own price. It could be as low as $4k. We’ll divide it by the hourly rate of $275, and you’ll get a certain number of hours. I’ll use those hours to further the agreed-upon goals. When the hours are over, if work still remains, you can choose to repeat this process by naming another price for the next iteration, or switch to option 1.

Looking for something else?

bottom of page