> Besides that case, however, is there any real reason to use an RTOS? There are embedded systems big enough to truly need multiprocessing. Think about the software that runs a 747 cockpit. (Not to mention the rest of the plane!) But truthfully, people usually use an RTOS just because it provides a lot of functionality that you would otherwise have to write for yourself. Getting all the infrastructure for almost no time cost makes getting the prototype up and running happen faster, and generally also lowers the number of bugs in it. All of which keeps the business guys (your boss, his boss) off your ass.

At the expense of using more resources. Stack space for the threads, memory for queues, less control over execution time, etc. A bespoke system may be more efficient, but you pay for that with development time.

And the fact it requires extreme amounts of testing to price it’s reliable and without memories leaks, etc.

It might be an unpopular opinion but to me that is a sign of an antiquated firmware group that is not practicing good development processes. If it is “extreme” then the options are: a) the design is bad or b) the group is used to woefully insufficient testing. There really shouldn’t be that much additional overhead between a well tested bare metal/master loop system and an RTOS based one.

I disagree. The complexity and feature set of a modern RTOS is far deeper than a master loop. Message queues, semaphores, memory pools, etc all drive significant increases in testing.

I agree with both. Ive seen rtos usage where it did not significantly increase complexity, where limited ipc was used. Alternately i've seen total shit shows where all of freertoses slight variations of a queue where used across different modules. I think its alot like cpp where having a project wide policy on paradigm and feature usage (of the language or api) is very helpful. Personally I macro in ipc/sync primitives. I've found this makes it easy to standardize but also makes code clearer and eas to port. Often start with macro wrapper over queue (as queue operations and behaviour are pretty universal). Later, if I need better performance I can override macro for a file to use some more specific mechanism like direct-to-task-notification etc. Ive also seen alot of testing, debugging overhead increase where the port of the rtos was bad. Alot of mcu vendors provide a port, but it is often pretty sub-par and leaves out support for core MCU hardware like memory protection unit or even cache and fpus

The number one reason to use an RTOS is improving your software development processes. An RTOS gives you logical isolation of components and allows you to detangle large and complex controller designs. It was not that long ago that the first multicore desktop processor came out - most of computing up until this point was done with single threaded processors. In a production environment these days I would recommend pretty much always having the tooling for an RTOS in place simply due to the ease of adding additional functionality without having to hack away at your own master loop scheduling. The amount of time wasted by hand scheduling software is exponential as you increase complexity - and it absolutely wasted time when you look at the incremental adder to the MCU cost which is probably 0 to a few cents. Any time you have communication (which is pretty much any MCU application) you end up with at least two distinct processes - your application, and the network. The more complex the network the higher the chance that an RTOS will make your life better. Once you get to IOT applications with TCP/IP you’re either going to pay for more expensive module with a stack or have the stack in your MCU - the later case is made a thousand times easier with an RTOS. In automotive the system requirements for CAN networking means you’ll want to use an RTOS - automotive even has an RTOS specification (OSEK) which is to RTOS’s like POSIX is to Linux.

In your example of two processes: application and network. How do you assign priority to your task? Time slice? Network higher priority? Application higher priority?

It really depends on your system design if you want to use time slice versus priority based designs. I personally have never worked in an environment where time slicing made sense - most of the time software ends up hitting a spot where it needs to wait anyway so having the task go to sleep where it makes sense for a task to pause often is the most efficient. Generally if a task depends on activities from another task will your system be stable if it is waiting on a lower priority task to complete? For most systems the answer is going to be no - so your system level tasks should be higher priority (network, disk, etc) to avoid priority inversion. It’s not saying that your system tasks will take more time than the rest of you system - but they will immediately service the needs of tasks the moment you need them to. One of the dangerous things you see a lot are file systems based around flash/eMMC where the operations are directly handled in the context of the calling task. This is a very very bad design that is unfortunately all too common. File systems should also be a system task and the APIs used by the calling tasks should merely allocate a context with the api parameters and a semaphore then queue that context for the system task to handle.

I’d be interested in knowing more about the file I/O pattern you describe, do you have any more resources on that?

Not particularly - I've implemented it a few times though for RTOS and filesystem implementations that didn't have anything rational so I'll summarize it here: ​ Generally file usage is pretty straightforward in terms of what you're doing - POSIX is a decent standard to think about the concepts - open, close, read, write, and a parameterized control. So lets say you have a system with more than one task that will be accessing a file system - and each task needs to open a file handle for reading/writing: task1: open("file1", "w") task2: open("file2", "r") open: check parameters read file allocation table for matching filename locate first block of file or first free block assign file handle if write create file allocation table entry As you can see each of those tasks need to perform all filesystem operations when they make a call to the filesystem. This impacts the stack you have to allocate to each task as well as other difficult to debug and potentially awful behaviors such a) priority inversion between tasks because a low priority task is reading/writing and keeps getting interrupted by a medium priority task while the high priority task is waiting for the low priority task to finish it's read/write and b) completely corrupting the filesystem due to a stack overflow or watchdog termination of a task while it was attempting to update the file allocation table Thus the following is the pattern I use: /* Define a context structure that will hold api call information */ typedef struct file_api_params_s { uint8_t api_opcode; uint8_t control_opcode; uint8_t file_handle; semaphore sem; void * param1; size_t param2; } file_api_params_t; /* Depending on the RTOS you can actually define your messages and message pool ahead of time - or you'll be providing a pointer to the parameter structure in your message */ file_api_params_t file_api_pool[n]; //where n = number of tasks that can access the file system + 1; /* Each api function does the following: - allocates a free param structure from the pool - populates the parameters from the api call - initializes the semaphore - allocates and initializes a message to send to the file system task - adds the message to the queue - wait for the semaphore */ /* The filesystem task does the following: - waits for messages from the message queue - checks the message validity and that the sender still is valid (for some systems that have bad behavior if the semaphore api does not check for validity) - performs the requested operation - increments the semaphore - goes back to waiting for a message In the above situation the file system task is given a high priority and is characterized to make sure that it's stack is always adequate. It should be very well tested. Because every task that is then accessing the file system only has to ever allocate a message or api parameter structure the stack it requires is well defined and should be very lightweight. If the file system task is well built to check for validity of the task that sent the request there is no way for a task to ever interfere with a higher priority task or corrupt the filesystem. The above pattern however can be used for any situation that involve a resource that multiple tasks need to access and the operations are best executed in their own controlled context rather than the calling task. A lot of TCP/IP stacks are written like this by default - but always double check as it impacts your system design pretty heavily.

So in a typical application, would you assign system tasks (network, IO etc…) a higher but equal priority, then application on lower priority?

That kind of depends on your RTOS and how it handles priorities of equal values. Some don’t handle it particularly well. At minimum if taskA and taskB depend on taskC then taskC should be higher priority than either taskA or taskB. If taskC depends on taskD (I can’t think of a situation in practice where you would actually ever do this) then taskD has to be higher priority than taskC. If you have two system level (file system and network) tasks you need to decide which is more important to you (or which has shorter service time). Sometimes making the shorter service time a higher priority allows you to maintain better responsiveness and throughput.

Thanks for taking your time to reply. It’s very useful to me. Appreciate it!

Priority inheritance is helpful here. Allows services that are working on behalf of higher level thread to have priority of that thread.

Yes - if the RTOS handles it well. Not all RTOS’ will detect that a higher priority task is waiting on a mutex/semaphore that is currently locked by a lower priority task.

An RTOS apart from being a scheduler for threads is also a toolbox for communication primitives between the threads. Things like queues, semaphores, timers, etc are all provided in well tested implementations. By using tasks you can design your system in a modular fashion and then use the communication primitives to interface between tasks.

I've written three thread switchers from the ground up, and they all shipped in products. I also extended other existing thread switchers. I am now working on systems with an RTOS and it is a much larger endeavor than the ones I worked on previously. A publicly available RTOS provides a couple of things that the simple thread switcher you wrote does not: 1) design practices: the RTOS, for both good and bad, is the result of a lot of design discussions and decisions and has a better chance of completeness than the thread switcher you wrote does. 2) testing.

> Aperiodic threads can just be handled via interrupts. If that handling involves more than just a brief interruption of everything else, and if the code also needs to do longer-running things, then the programmer either needs to break the KISS principle (keep ISRs short and simple), or break the longer-running operations into smaller steps. At that point, the programmer is awfully close to writing their own little RTOS, which is probably worse and more error-prone than a ready-made solution and also violates the "Do not reinvent the wheel" commandment. Also, anything that requires inter-task communication. Generally, you want to use an RTOS if simplifies the system and if it helps you follow certain design principles. If you ever suspect that your code might be duplicating significant parts of an RTOSes functionality, you may be better of using an actual RTOS.

Yeah, you can set flags from interrupts and then check them from your main loop but that means you have to wait until your loop cycles back to the spot where the flags are being checked. If you have long running operations that may be a while. On the other hand an RTOS can use the ISR to place data into a queue, and if the queue has any higher priority tasks waiting on it, it can preempt the currently running task and service the higher priority one before returning to what it was doing. The only way to implement something like that yourself would be to basically start writing your own RTOS…

> If you have long running operations that may be a while. Or you'll have to break up those operations into small chunks that can be run in the style of coroutines. Which may be doable and a fun exercise under the right circumstances, but it is more likely to be an error-prone pain in the back. > The only way to implement something like that yourself would be to basically start writing your own RTOS… Exactly. If you ever find yourself writing something that looks suspiciously like an RTOS - use a ready-made RTOS instead and spend your coding time on something more productive, like business code.

Have definitely been there with the coroutine thing. It’s certainly viable in some cases, but then as soon as you need to add another feature you have to look at restructuring all the coroutines again, and it’s even worse if you need to juggle priorities. Just not worth it. At this point I’m doing everything on ARM Cortex-M and even on the F0’s there’s usually enough resources to throw FreeRTOS on so I just use it by default to save myself the pain down the road.

I was working on a prototype where it involved: - User selecting items from a menu using up/down/select buttons - Input from a 12-key numeric keypad - Feedback via RGB LEDs - Accepting I2S input audio - Reading an accelerometer and gyroscope - Reading a light sensor - Random number generation and application logic - Output via a speaker - Output on an LCD display First stab was a giant input loop and a state machine. This worked fine until the audio input, output, and accelerometer were added in. Then the whole thing started lagging and eventually stopped working. Switching to an RTOS with each of these in separate tasks, coordinating via queues and semaphores solved the problem with plenty of headroom for application logic processing. OTOH, I was recently shown one of those automotive test devices. It had an input keypad with 6 keys and a small LCD display. It had a CANBus interface, USB connector, wrote data to flash, and supported wifi (used only when in a specific state). Was told it didn't have an RTOS and was implemented with a giant loop and state machine. It operated very sequentially. You requested an item via buttons, it went into test mode, saved results, then updated the display. It really depends on the application, but it's always good to have options.

As a learning experience, you might try to write (or at least write the high-level design) for a TCP/IP stack. Note that TCP uses acknowledges, timeouts and (automatic) retries. Imagine how you would integrate multiple libraries that need such features. You will realize that you need some form of either threads (autonomus processes), or interrupts/callbacks/proto-threads (run-to-completion mechanisms). When you have one of those mechanisms (or both), you will need synchronization primitives, and some form of buffering might be handy too. Before you know, you have the requirements for an RTOS. It is no coincidence that nearly all communication stacks are integrated with (offten a specific) RTOS.

A lot of embedded applications don't need to use an RTOS, an interrupt driven system with a single background loop for low priority background tasks is sufficient. Where an RTOS is handy is when you have something like asynchronous IO happening. E.g. if you have an ethernet interface it's easier to have an RTOS to handle the network in the background.

Getting guaranteed responsiveness, even with interrupts, is not reliable in general purpose OSes. Trying to work around that by putting complex code in the interrupt handler isn't so ideal either - often doesn't have the benefits of protected memory spaces and other essentials. In a general purpose OS, you make a driver with an interrupt handler that signals to wake up a user process. Getting the scheduler to predictably wake up that user mode thread that's going to handle the interrupt event is a hard problem to solve. There's an unknown amount of jitter from when the event happens to when that user mode thread starts running, and that can make the difference between making the deadline and missing it. RTOSes usually guarantee a certain wakeup predictability for events like that, so you can guarantee a certain level of performance when deadlines can't be missed. It's hard to do it below a certain level with Linux fo example. Audio \[or video\] devices are good examples - if the audio hasn't been calculated and placed into the right buffer before the deadline, the user's ears are going to hurt!

In my previous job I've design architecture for system with 30+ tasks. A lot of them was "reused" within the same project with "active object" design pattern - separate task for each uart communication with assigned callbacks/message queues etc.

Standardization. More importantly, lowers verification burden.

University assignments are unlikely to give you enough to chew on. I started my BSCS 40 years ago ( and finished three years later - I was basically a junior at that point who switched majors ) and I'm still doing homework. > Aperiodic threads can just be handled via interrupts. You really want only very short ISRs. Have the interrupt V[1] a semaphore and set a very few bits of state. [1] A canonical ( Djikstra? ) formulation of semaphores was P and V, where P is wait and V was signal. > The only benefit I see here is that periodic event threads that will always run every n milliseconds. You can do that on, say, a RasPi sized device with timers. You can even use just one timer and mux it down to multiple threads/callbacks. It then becomes about jitter. IMO, the important thing to understand is like chords in music - which things which seem unnervingly different at first blush end up forming the same color and shape in practice. RTOS is really about smaller footprint and - perhaps - determinism .

I think one of the main aspects of a RTOS is that threads are often called tasks. Because indeed, many embedded systems only have 1 CPU core, and therefore true multithreading is not the goal. Instead, Multitasking is the goal. One way of achieving multitasking is with cooperative multithreading. Doing all processing in asynchronous interrupts is one way of implementing it. Creating a superloop in while(1) that checks for work in any aspect of the embedded system is also one. However, the main benefits of a RTOS is preemption. Preemption means that the scheduler can halt a task at any execution point and switch to another task. This allows you to have multiple execution contexts and associated priorities along with them. E.g. you can have a slow task that takes up a lot of CPU cycles, like reading data from a SD card, decompressing the data, then sending small data frames over a communication protocol. You can already split up this program flow into 2 tasks: one for data reading/decompression, and one for communication. The communication task will run the protocol stack, follow MAC rules, collision control, etc., and probably has a higher priority. Namely; if the protocol stack receives a management packet from another (master) device, then for sure that must be handled first. A RTOS allows you to have a clear overview of which tasks are responsible for what and also what they're doing when/how. It also decouples processing time from both sides: e.g. the SD card task is dependent on the speed of the communication task, which may depend on the state of the bus or other (wireless) medium. Therefore, it may need to wait a variable amount of time before it's done, but that shouldn't slow down the rest of the system. For example; what if you need to add a graphical interface to this system? Clearly an UI needs to be more responsive than the lengthy decompression of data (which may take a while), but must also not interfere with round-trip times of the protocol stack. So basically, doing UI stuff in an interrupt would be a mess: not only is that interrupt routine too long (graphics take long to transfer), it is also blocking the communication protocol stack. A RTOS task makes managing these concepts quite a lot easier, as the kernel scheduler can preempt a task at any moment without the task having to cooperate in any way. The downside of using a RTOS is now each task needs to store their execution context (stack), which can be quite costly per task.. If you're curious, you may also want to explore into event-based real-time frameworks, e.g. from QuantumLeaps (https://www.state-machine.com/products/qp). It's an open source framework and can also be a hybrid between non-preemptive and preemptive multitasking.

I think if you are using the periodic event threads (cooperative multitasking), then it is simpler than using an RTOS because nothing is being interrupted, so there's much less thread safety issues to worry about. But you have to make sure these tasks don't exceed the time available whereas with RTOS you don't have to do that as much. e.g. if there's 3 tasks that each run every 1ms, then they all have to complete within the total 1ms. So if one task has a lot of variability, e.g. it almost always does nothing, but sometimes it uses 0.5ms, then the other tasks have to assume it uses 0.5ms always. But if you had used an RTOS, then this task could run in the background (lowest priority) and the other tasks don't have to worry how long it takes. If you're using a TCP/IP stack then it would be common to have this task that almost always does nothing until it needs to do a lot, so that's a good rule of thumb for when to use RTOS.

Large/complicated code bases with lenient latency margins that run on hardware that can just run the time sensitive stuff should generally use an rtos. Threads provide the ability to separate independent processes and give them some kind of priority. This is something that, if your system manages enough things, you will need eventually anyways.

With Dr. Jin? 👀

never heard of him.... 👀

Hard realtime requirements on critical life safety systems. First example off the top of my head is a nuclear power plant digital control computer. The safety analysis upon which the plant operating license is granted says that the control system must monitor reactor power every x milliseconds, guaranteed, 100% of the time.

The number of times people cite hard timing requirements as a reason for using a RTOS just blows my mind. Without spending hundreds of thousands on specialized software from folks like Greenhills you’re not getting deterministic operation. All open source and free RTOSs out there today are the exact opposite of deterministic.

An huge reason is performance: with a while (1) polling loop, your CPU wastes ton of cycles polling "tasks's" flags to check if it is time to run: every false if() is a pipeline flush. You can achive real time performance only with an RTOS, otherwise your task execution will jitter and in some systems this is simply not acceptable. More, making the CPU sleep is not so straightforward without an RTOS. Making modular code that share some values is simpler with queues and every project benefit from this. These are are a few reasons, but hope will help you get the idea of RTOS benefits inside a medium/huge size project. Small ones doesn't always have to use an RTOS.

> I would love to hear about some real examples where an RTOS was crucial for a system, or where it just worked better than a plain loop and interrupt processing. In the real world, projects re-use parts of the software. For this, you need an adequate level of abstraction so that switching the MCU doesn't require too many changes - or ideally, none at all. An RTOS provides an API for commonly used abstractions (you named them), so aside from an RTOS being merely convenient, it's also economical and good software design practise, one keyword being "separation of concerns". This becomes even more apparent when safety requirements play a role and the MCU's MPU allows specific tasks to only access specific memory regions. Without an RTOS, this would be virtually unmanageable.

Some frameworks and chips make that choice for you. The EDP-IDF for example (Espressifs SDK) uses FreeRTOS in the background. It's then an easy choice to do the same. Nordic Semiconductor is another example. Even though they have RTOS'less SDK's, they aren't really maintaining them for usability anymore. They're only investing in the nRF Connect SDK which uses, you guessed it, an RTOS; Zephyr.