1# Rocksdb Tuning Advisor 2 3## Motivation 4 5The performance of Rocksdb is contingent on its tuning. However, 6because of the complexity of its underlying technology and a large number of 7configurable parameters, a good configuration is sometimes hard to obtain. The aim of 8the python command-line tool, Rocksdb Advisor, is to automate the process of 9suggesting improvements in the configuration based on advice from Rocksdb 10experts. 11 12## Overview 13 14Experts share their wisdom as rules comprising of conditions and suggestions in the INI format (refer 15[rules.ini](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/rules.ini)). 16Users provide the Rocksdb configuration that they want to improve upon (as the 17familiar Rocksdb OPTIONS file — 18[example](https://github.com/facebook/rocksdb/blob/master/examples/rocksdb_option_file_example.ini)) 19and the path of the file which contains Rocksdb logs and statistics. 20The [Advisor](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/rule_parser_example.py) 21creates appropriate DataSource objects (for Rocksdb 22[logs](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/db_log_parser.py), 23[options](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/db_options_parser.py), 24[statistics](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/db_stats_fetcher.py) etc.) 25and provides them to the [Rules Engine](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/rule_parser.py). 26The Rules uses rules from experts to parse data-sources and trigger appropriate rules. 27The Advisor's output gives information about which rules were triggered, 28why they were triggered and what each of them suggests. Each suggestion 29provided by a triggered rule advises some action on a Rocksdb 30configuration option, for example, increase CFOptions.write_buffer_size, 31set bloom_bits to 2 etc. 32 33## Usage 34 35### Prerequisites 36The tool needs the following to run: 37* python3 38 39### Running the tool 40An example command to run the tool: 41 42```shell 43cd rocksdb/tools/advisor 44python3 -m advisor.rule_parser_example --rules_spec=advisor/rules.ini --rocksdb_options=test/input_files/OPTIONS-000005 --log_files_path_prefix=test/input_files/LOG-0 --stats_dump_period_sec=20 45``` 46 47### Command-line arguments 48 49Most important amongst all the input that the Advisor needs, are the rules 50spec and starting Rocksdb configuration. The configuration is provided as the 51familiar Rocksdb Options file (refer [example](https://github.com/facebook/rocksdb/blob/master/examples/rocksdb_option_file_example.ini)). 52The Rules spec is written in the INI format (more details in 53[rules.ini](https://github.com/facebook/rocksdb/blob/master/tools/advisor/advisor/rules.ini)). 54 55In brief, a Rule is made of conditions and is triggered when all its 56constituent conditions are triggered. When triggered, a Rule suggests changes 57(increase/decrease/set to a suggested value) to certain Rocksdb options that 58aim to improve Rocksdb performance. Every Condition has a 'source' i.e. 59the data source that would be checked for triggering that condition. 60For example, a log Condition (with 'source=LOG') is triggered if a particular 61'regex' is found in the Rocksdb LOG files. As of now the Rules Engine 62supports 3 types of Conditions (and consequently data-sources): 63LOG, OPTIONS, TIME_SERIES. The TIME_SERIES data can be sourced from the 64Rocksdb [statistics](https://github.com/facebook/rocksdb/blob/master/include/rocksdb/statistics.h) 65or [perf context](https://github.com/facebook/rocksdb/blob/master/include/rocksdb/perf_context.h). 66 67For more information about the remaining command-line arguments, run: 68 69```shell 70cd rocksdb/tools/advisor 71python3 -m advisor.rule_parser_example --help 72``` 73 74### Sample output 75 76Here, a Rocksdb log-based rule has been triggered: 77 78```shell 79Rule: stall-too-many-memtables 80LogCondition: stall-too-many-memtables regex: Stopping writes because we have \d+ immutable memtables \(waiting for flush\), max_write_buffer_number is set to \d+ 81Suggestion: inc-bg-flush option : DBOptions.max_background_flushes action : increase suggested_values : ['2'] 82Suggestion: inc-write-buffer option : CFOptions.max_write_buffer_number action : increase 83scope: col_fam: 84{'default'} 85``` 86 87## Running the tests 88 89Tests for the code have been added to the 90[test/](https://github.com/facebook/rocksdb/tree/master/tools/advisor/test) 91directory. For example, to run the unit tests for db_log_parser.py: 92 93```shell 94cd rocksdb/tools/advisor 95python3 -m unittest -v test.test_db_log_parser 96``` 97