README.md
1Nvim core
2=========
3
4Module-specific details are documented at the top of each module (`terminal.c`,
5`screen.c`, …).
6
7See `:help dev` for guidelines.
8
9Filename conventions
10--------------------
11
12The source files use extensions to hint about their purpose.
13
14- `*.c`, `*.generated.c` - full C files, with all includes, etc.
15- `*.c.h` - parametrized C files, contain all necessary includes, but require
16 defining macros before actually using. Example: `typval_encode.c.h`
17- `*.h` - full headers, with all includes. Does *not* apply to `*.generated.h`.
18- `*.h.generated.h` - exported functions’ declarations.
19- `*.c.generated.h` - static functions’ declarations.
20
21Logs
22----
23
24Low-level log messages sink to `$NVIM_LOG_FILE`.
25
26UI events are logged at DEBUG level (`DEBUG_LOG_LEVEL`).
27
28 rm -rf build/
29 make CMAKE_EXTRA_FLAGS="-DMIN_LOG_LEVEL=0"
30
31Use `LOG_CALLSTACK()` (Linux only) to log the current stacktrace. To log to an
32alternate file (e.g. stderr) use `LOG_CALLSTACK_TO_FILE(FILE*)`. Requires
33`-no-pie` ([ref](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=860394#15)):
34
35 rm -rf build/
36 make CMAKE_EXTRA_FLAGS="-DMIN_LOG_LEVEL=0 -DCMAKE_C_FLAGS=-no-pie"
37
38Many log messages have a shared prefix, such as "UI" or "RPC". Use the shell to
39filter the log, e.g. at DEBUG level you might want to exclude UI messages:
40
41 tail -F ~/.cache/nvim/log | cat -v | stdbuf -o0 grep -v UI | stdbuf -o0 tee -a log
42
43Build with ASAN
44---------------
45
46Building Nvim with Clang sanitizers (Address Sanitizer: ASan, Undefined
47Behavior Sanitizer: UBSan, Memory Sanitizer: MSan, Thread Sanitizer: TSan) is
48a good way to catch undefined behavior, leaks and other errors as soon as they
49happen. It's significantly faster than Valgrind.
50
51Requires clang 3.4 or later, and `llvm-symbolizer` must be in `$PATH`:
52
53 clang --version
54
55Build Nvim with sanitizer instrumentation (choose one):
56
57 CC=clang make CMAKE_EXTRA_FLAGS="-DCLANG_ASAN_UBSAN=ON"
58 CC=clang make CMAKE_EXTRA_FLAGS="-DCLANG_MSAN=ON"
59 CC=clang make CMAKE_EXTRA_FLAGS="-DCLANG_TSAN=ON"
60
61Create a directory to store logs:
62
63 mkdir -p "$HOME/logs"
64
65Configure the sanitizer(s) via these environment variables:
66
67 # Change to detect_leaks=1 to detect memory leaks (slower).
68 export ASAN_OPTIONS="detect_leaks=0:log_path=$HOME/logs/asan"
69 # Show backtraces in the logs.
70 export UBSAN_OPTIONS=print_stacktrace=1
71 export MSAN_OPTIONS="log_path=${HOME}/logs/tsan"
72 export TSAN_OPTIONS="log_path=${HOME}/logs/tsan"
73
74Logs will be written to `${HOME}/logs/*san.PID` then.
75
76For more information: https://github.com/google/sanitizers/wiki/SanitizerCommonFlags
77
78Debug: Performance
79------------------
80
81### Profiling (easy)
82
83For debugging performance bottlenecks in any code, there is a simple (and very
84effective) approach:
85
861. Run the slow code in a loop.
872. Break execution ~5 times and save the stacktrace.
883. The cause of the bottleneck will (almost always) appear in most of the stacktraces.
89
90### Profiling (fancy)
91
92For more advanced profiling, consider `perf` + `flamegraph`.
93
94### USDT profiling (powerful)
95
96Or you can use USDT probes via `NVIM_PROBE` ([#12036](https://github.com/neovim/neovim/pull/12036)).
97
98> USDT is basically a way to define stable probe points in userland binaries.
99> The benefit of bcc is the ability to define logic to go along with the probe
100> points.
101
102Tools:
103- bpftrace provides an awk-like language to the kernel bytecode, BPF.
104- BCC provides a subset of C. Provides more complex logic than bpftrace, but takes a bit more effort.
105
106Example using bpftrace to track slow vim functions, and print out any files
107that were opened during the trace. At the end, it prints a histogram of
108function timing:
109
110 #!/usr/bin/env bpftrace
111
112 BEGIN {
113 @depth = -1;
114 }
115
116 tracepoint:sched:sched_process_fork /@pidmap[args->parent_pid]/ {
117 @pidmap[args->child_pid] = 1;
118 }
119
120 tracepoint:sched:sched_process_exit /@pidmap[args->pid]/ {
121 delete(@pidmap[args->pid]);
122 }
123
124 usdt:build/bin/nvim:neovim:eval__call_func__entry {
125 @pidmap[pid] = 1;
126 @depth++;
127 @funcentry[@depth] = nsecs;
128 }
129
130 usdt:build/bin/nvim:neovim:eval__call_func__return {
131 $func = str(arg0);
132 $msecs = (nsecs - @funcentry[@depth]) / 1000000;
133
134 @time_histo = hist($msecs);
135
136 if ($msecs >= 1000) {
137 printf("%u ms for %s\n", $msecs, $func);
138 print(@files);
139 }
140
141 clear(@files);
142 delete(@funcentry[@depth]);
143 @depth--;
144 }
145
146 tracepoint:syscalls:sys_enter_open,
147 tracepoint:syscalls:sys_enter_openat {
148 if (@pidmap[pid] == 1 && @depth >= 0) {
149 @files[str(args->filename)] = count();
150 }
151 }
152
153 END {
154 clear(@depth);
155 }
156
157 $ sudo bpftrace funcslower.bt
158 1527 ms for Slower
159 @files[/usr/lib/libstdc++.so.6]: 2
160 @files[/etc/fish/config.fish]: 2
161 <snip>
162
163 ^C
164 @time_histo:
165 [0] 71430 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
166 [1] 346 | |
167 [2, 4) 208 | |
168 [4, 8) 91 | |
169 [8, 16) 22 | |
170 [16, 32) 85 | |
171 [32, 64) 7 | |
172 [64, 128) 0 | |
173 [128, 256) 0 | |
174 [256, 512) 6 | |
175 [512, 1K) 1 | |
176 [1K, 2K) 5 | |
177
178Debug: TUI
179----------
180
181### TUI troubleshoot
182
183Nvim logs its internal terminfo state at 'verbose' level 3. This makes it
184possible to see exactly what terminfo values Nvim is using on any system.
185
186 nvim -V3log
187
188### TUI trace
189
190The ancient `script` command is still the "state of the art" for tracing
191terminal behavior. The libvterm `vterm-dump` utility formats the result for
192human-readability.
193
194Record a Nvim terminal session and format it with `vterm-dump`:
195
196 script foo
197 ./build/bin/nvim -u NONE
198 # Exit the script session with CTRL-d
199
200 # Use `vterm-dump` utility to format the result.
201 ./.deps/usr/bin/vterm-dump foo > bar
202
203Then you can compare `bar` with another session, to debug TUI behavior.
204
205### TUI redraw
206
207Set the 'writedelay' option to see where and when the UI is painted.
208
209 :set writedelay=1
210
211### Terminal reference
212
213- `man terminfo`
214- http://bazaar.launchpad.net/~libvterm/libvterm/trunk/view/head:/doc/seqs.txt
215- http://invisible-island.net/xterm/ctlseqs/ctlseqs.html
216
217Nvim lifecycle
218--------------
219
220Following describes how Nvim processes input.
221
222Consider a typical Vim-like editing session:
223
22401. Vim displays the welcome screen
22502. User types: `:`
22603. Vim enters command-line mode
22704. User types: `edit README.txt<CR>`
22805. Vim opens the file and returns to normal mode
22906. User types: `G`
23007. Vim navigates to the end of the file
23109. User types: `5`
23210. Vim enters count-pending mode
23311. User types: `d`
23412. Vim enters operator-pending mode
23513. User types: `w`
23614. Vim deletes 5 words
23715. User types: `g`
23816. Vim enters the "g command mode"
23917. User types: `g`
24018. Vim goes to the beginning of the file
24119. User types: `i`
24220. Vim enters insert mode
24321. User types: `word<ESC>`
24422. Vim inserts "word" at the beginning and returns to normal mode
245
246Note that we split user actions into sequences of inputs that change the state
247of the editor. While there's no documentation about a "g command mode" (step
24816), internally it is implemented similarly to "operator-pending mode".
249
250From this we can see that Vim has the behavior of an input-driven state machine
251(more specifically, a pushdown automaton since it requires a stack for
252transitioning back from states). Assuming each state has a callback responsible
253for handling keys, this pseudocode represents the main program loop:
254
255```py
256def state_enter(state_callback, data):
257 do
258 key = readkey() # read a key from the user
259 while state_callback(data, key) # invoke the callback for the current state
260```
261
262That is, each state is entered by calling `state_enter` and passing a
263state-specific callback and data. Here is a high-level pseudocode for a program
264that implements something like the workflow described above:
265
266```py
267def main()
268 state_enter(normal_state, {}):
269
270def normal_state(data, key):
271 if key == ':':
272 state_enter(command_line_state, {})
273 elif key == 'i':
274 state_enter(insert_state, {})
275 elif key == 'd':
276 state_enter(delete_operator_state, {})
277 elif key == 'g':
278 state_enter(g_command_state, {})
279 elif is_number(key):
280 state_enter(get_operator_count_state, {'count': key})
281 elif key == 'G'
282 jump_to_eof()
283 return true
284
285def command_line_state(data, key):
286 if key == '<cr>':
287 if data['input']:
288 execute_ex_command(data['input'])
289 return false
290 elif key == '<esc>'
291 return false
292
293 if not data['input']:
294 data['input'] = ''
295
296 data['input'] += key
297 return true
298
299def delete_operator_state(data, key):
300 count = data['count'] or 1
301 if key == 'w':
302 delete_word(count)
303 elif key == '$':
304 delete_to_eol(count)
305 return false # return to normal mode
306
307def g_command_state(data, key):
308 if key == 'g':
309 go_top()
310 elif key == 'v':
311 reselect()
312 return false # return to normal mode
313
314def get_operator_count_state(data, key):
315 if is_number(key):
316 data['count'] += key
317 return true
318 unshift_key(key) # return key to the input buffer
319 state_enter(delete_operator_state, data)
320 return false
321
322def insert_state(data, key):
323 if key == '<esc>':
324 return false # exit insert mode
325 self_insert(key)
326 return true
327```
328
329The above gives an idea of how Nvim is organized internally. Some states like
330the `g_command_state` or `get_operator_count_state` do not have a dedicated
331`state_enter` callback, but are implicitly embedded into other states (this
332will change later as we continue the refactoring effort). To start reading the
333actual code, here's the recommended order:
334
3351. `state_enter()` function (state.c). This is the actual program loop,
336 note that a `VimState` structure is used, which contains function pointers
337 for the callback and state data.
3382. `main()` function (main.c). After all startup, `normal_enter` is called
339 at the end of function to enter normal mode.
3403. `normal_enter()` function (normal.c) is a small wrapper for setting
341 up the NormalState structure and calling `state_enter`.
3424. `normal_check()` function (normal.c) is called before each iteration of
343 normal mode.
3445. `normal_execute()` function (normal.c) is called when a key is read in normal
345 mode.
346
347The basic structure described for normal mode in 3, 4 and 5 is used for other
348modes managed by the `state_enter` loop:
349
350- command-line mode: `command_line_{enter,check,execute}()`(`ex_getln.c`)
351- insert mode: `insert_{enter,check,execute}()`(`edit.c`)
352- terminal mode: `terminal_{enter,execute}()`(`terminal.c`)
353
354Async event support
355-------------------
356
357One of the features Nvim added is the support for handling arbitrary
358asynchronous events, which can include:
359
360- RPC requests
361- job control callbacks
362- timers
363
364Nvim implements this functionality by entering another event loop while
365waiting for characters, so instead of:
366
367```py
368def state_enter(state_callback, data):
369 do
370 key = readkey() # read a key from the user
371 while state_callback(data, key) # invoke the callback for the current state
372```
373
374Nvim program loop is more like:
375
376```py
377def state_enter(state_callback, data):
378 do
379 event = read_next_event() # read an event from the operating system
380 while state_callback(data, event) # invoke the callback for the current state
381```
382
383where `event` is something the operating system delivers to us, including (but
384not limited to) user input. The `read_next_event()` part is internally
385implemented by libuv, the platform layer used by Nvim.
386
387Since Nvim inherited its code from Vim, the states are not prepared to receive
388"arbitrary events", so we use a special key to represent those (When a state
389receives an "arbitrary event", it normally doesn't do anything other update the
390screen).
391
392Main loop
393---------
394
395The `Loop` structure (which describes `main_loop`) abstracts multiple queues
396into one loop:
397
398 uv_loop_t uv;
399 MultiQueue *events;
400 MultiQueue *thread_events;
401 MultiQueue *fast_events;
402
403`loop_poll_events` checks `Loop.uv` and `Loop.fast_events` whenever Nvim is
404idle, and also at `os_breakcheck` intervals.
405
406MultiQueue is cool because you can attach throw-away "child queues" trivially.
407For example `do_os_system()` does this (for every spawned process!) to
408automatically route events onto the `main_loop`:
409
410 Process *proc = &uvproc.process;
411 MultiQueue *events = multiqueue_new_child(main_loop.events);
412 proc->events = events;
413