Is it possible to monitor how a process is scheduled real-time with Linux?

Question

I am trying to develop a multi-process system with RaspberryPi in which I want to monitor all major processes and how they are scheduled using Linux. That is, obtaining scheduler states, start time, release time etc. I have been messing with '/proc' folder in order to find such an information but I haven't able to find anything really useful so far.

For example, if you go into /proc/pid/task/pid you can see:

se.exec_start
se.vruntime
se.sum_exec_runtime
se.statistics.wait_start
se.statistics.sleep_start
se.statistics.block_start
se.statistics.sleep_max
se.statistics.block_max
se.statistics.iowait_sum
se.statistics.wakeups

and so on. Now, this looks like I am up to something, but not clear enough. I want to just see how processes are scheduled in cores. i.e, Process1 released at 0.30, then Process2 started at 0.70 (system timer values) etc.

Is something like this possible by monitoring kernel folders such as '/proc'? If not, is there a way to determine this using another tracing tools or scheduler tools for Linux?

Any guidance is greately appreciated. Please tell me if there is something else I need to provide.

Thanks in advance.

EDIT: using kernelshark,

I want to find out exactly, for other processes too, when does cpu complete one iteration for every task.

Answer 1

I want to just see how processes are scheduled in cores. i.e, Process1 released at 0.30, then Process2 started at 0.70 (system timer values) etc.

This is called tracing, and usually done in the kernel after request from user. There are several kernel event tracers in Linux. Try

• perf sched (man, lwn commit; also http://www.brendangregg.com/perf.html#SchedulerAnalysis) use perf sched record sleep 2 then perf sched script to get log
• some other perf command for tracing, like perf record -e 'sched:sched_process_*' -a sleep 2 + perf script (from http://www.brendangregg.com/perf.html)
• trace-cmd (man; based on ftrace - https://lwn.net/Articles/608497/ https://events.linuxfoundation.org/slides/2010/linuxcon_japan/linuxcon_jp2010_rostedt.pdf): trace-cmd record -e sched ./program / trace-cmd record -e sched_switch ..., trace-cmd report (or install and use kernelshark GUI - https://lwn.net/Articles/425583/ http://static.lwn.net/images/2011/ks-success.png)
• There are also special tracers like LTT/LTTng (wiki, website). It has GUI plugin for Eclipse (Trace Compass): https://wiki.eclipse.org/images/4/49/X-axis-alignment-full-histogram-axis-bottom.png with process graph and CPU graph over time
• I think, sysdig may trace scheduler too...

Gregg has some info of Linux tracing (with "pony-corn mascot" magic): http://www.brendangregg.com/blog/2015-07-08/choosing-a-linux-tracer.html (there should be some presentation about tracing in https://www.slideshare.net/brendangregg)...

Is something like this possible by monitoring kernel folders such as '/proc'

There is no inotify for /proc (it has no real directories or real files inside: https://stackoverflow.com/a/24898733), so you can't monitor for changes, you can only reread some /proc (or some /sys) periodically.