10e4f07a6SMauro Carvalho Chehab.. _development_early_stage: 20e4f07a6SMauro Carvalho Chehab 30e4f07a6SMauro Carvalho ChehabEarly-stage planning 40e4f07a6SMauro Carvalho Chehab==================== 50e4f07a6SMauro Carvalho Chehab 60e4f07a6SMauro Carvalho ChehabWhen contemplating a Linux kernel development project, it can be tempting 70e4f07a6SMauro Carvalho Chehabto jump right in and start coding. As with any significant project, 80e4f07a6SMauro Carvalho Chehabthough, much of the groundwork for success is best laid before the first 90e4f07a6SMauro Carvalho Chehabline of code is written. Some time spent in early planning and 100e4f07a6SMauro Carvalho Chehabcommunication can save far more time later on. 110e4f07a6SMauro Carvalho Chehab 120e4f07a6SMauro Carvalho Chehab 130e4f07a6SMauro Carvalho ChehabSpecifying the problem 140e4f07a6SMauro Carvalho Chehab---------------------- 150e4f07a6SMauro Carvalho Chehab 160e4f07a6SMauro Carvalho ChehabLike any engineering project, a successful kernel enhancement starts with a 170e4f07a6SMauro Carvalho Chehabclear description of the problem to be solved. In some cases, this step is 180e4f07a6SMauro Carvalho Chehabeasy: when a driver is needed for a specific piece of hardware, for 190e4f07a6SMauro Carvalho Chehabexample. In others, though, it is tempting to confuse the real problem 200e4f07a6SMauro Carvalho Chehabwith the proposed solution, and that can lead to difficulties. 210e4f07a6SMauro Carvalho Chehab 220e4f07a6SMauro Carvalho ChehabConsider an example: some years ago, developers working with Linux audio 230e4f07a6SMauro Carvalho Chehabsought a way to run applications without dropouts or other artifacts caused 240e4f07a6SMauro Carvalho Chehabby excessive latency in the system. The solution they arrived at was a 250e4f07a6SMauro Carvalho Chehabkernel module intended to hook into the Linux Security Module (LSM) 260e4f07a6SMauro Carvalho Chehabframework; this module could be configured to give specific applications 270e4f07a6SMauro Carvalho Chehabaccess to the realtime scheduler. This module was implemented and sent to 280e4f07a6SMauro Carvalho Chehabthe linux-kernel mailing list, where it immediately ran into problems. 290e4f07a6SMauro Carvalho Chehab 300e4f07a6SMauro Carvalho ChehabTo the audio developers, this security module was sufficient to solve their 310e4f07a6SMauro Carvalho Chehabimmediate problem. To the wider kernel community, though, it was seen as a 320e4f07a6SMauro Carvalho Chehabmisuse of the LSM framework (which is not intended to confer privileges 330e4f07a6SMauro Carvalho Chehabonto processes which they would not otherwise have) and a risk to system 340e4f07a6SMauro Carvalho Chehabstability. Their preferred solutions involved realtime scheduling access 350e4f07a6SMauro Carvalho Chehabvia the rlimit mechanism for the short term, and ongoing latency reduction 360e4f07a6SMauro Carvalho Chehabwork in the long term. 370e4f07a6SMauro Carvalho Chehab 380e4f07a6SMauro Carvalho ChehabThe audio community, however, could not see past the particular solution 390e4f07a6SMauro Carvalho Chehabthey had implemented; they were unwilling to accept alternatives. The 400e4f07a6SMauro Carvalho Chehabresulting disagreement left those developers feeling disillusioned with the 410e4f07a6SMauro Carvalho Chehabentire kernel development process; one of them went back to an audio list 420e4f07a6SMauro Carvalho Chehaband posted this: 430e4f07a6SMauro Carvalho Chehab 440e4f07a6SMauro Carvalho Chehab There are a number of very good Linux kernel developers, but they 450e4f07a6SMauro Carvalho Chehab tend to get outshouted by a large crowd of arrogant fools. Trying 460e4f07a6SMauro Carvalho Chehab to communicate user requirements to these people is a waste of 470e4f07a6SMauro Carvalho Chehab time. They are much too "intelligent" to listen to lesser mortals. 480e4f07a6SMauro Carvalho Chehab 4993431e06SAlexander A. Klimov(https://lwn.net/Articles/131776/). 500e4f07a6SMauro Carvalho Chehab 510e4f07a6SMauro Carvalho ChehabThe reality of the situation was different; the kernel developers were far 520e4f07a6SMauro Carvalho Chehabmore concerned about system stability, long-term maintenance, and finding 530e4f07a6SMauro Carvalho Chehabthe right solution to the problem than they were with a specific module. 540e4f07a6SMauro Carvalho ChehabThe moral of the story is to focus on the problem - not a specific solution 550e4f07a6SMauro Carvalho Chehab- and to discuss it with the development community before investing in the 560e4f07a6SMauro Carvalho Chehabcreation of a body of code. 570e4f07a6SMauro Carvalho Chehab 580e4f07a6SMauro Carvalho ChehabSo, when contemplating a kernel development project, one should obtain 590e4f07a6SMauro Carvalho Chehabanswers to a short set of questions: 600e4f07a6SMauro Carvalho Chehab 610e4f07a6SMauro Carvalho Chehab - What, exactly, is the problem which needs to be solved? 620e4f07a6SMauro Carvalho Chehab 630e4f07a6SMauro Carvalho Chehab - Who are the users affected by this problem? Which use cases should the 640e4f07a6SMauro Carvalho Chehab solution address? 650e4f07a6SMauro Carvalho Chehab 660e4f07a6SMauro Carvalho Chehab - How does the kernel fall short in addressing that problem now? 670e4f07a6SMauro Carvalho Chehab 680e4f07a6SMauro Carvalho ChehabOnly then does it make sense to start considering possible solutions. 690e4f07a6SMauro Carvalho Chehab 700e4f07a6SMauro Carvalho Chehab 710e4f07a6SMauro Carvalho ChehabEarly discussion 720e4f07a6SMauro Carvalho Chehab---------------- 730e4f07a6SMauro Carvalho Chehab 740e4f07a6SMauro Carvalho ChehabWhen planning a kernel development project, it makes great sense to hold 750e4f07a6SMauro Carvalho Chehabdiscussions with the community before launching into implementation. Early 760e4f07a6SMauro Carvalho Chehabcommunication can save time and trouble in a number of ways: 770e4f07a6SMauro Carvalho Chehab 780e4f07a6SMauro Carvalho Chehab - It may well be that the problem is addressed by the kernel in ways which 790e4f07a6SMauro Carvalho Chehab you have not understood. The Linux kernel is large and has a number of 800e4f07a6SMauro Carvalho Chehab features and capabilities which are not immediately obvious. Not all 810e4f07a6SMauro Carvalho Chehab kernel capabilities are documented as well as one might like, and it is 820e4f07a6SMauro Carvalho Chehab easy to miss things. Your author has seen the posting of a complete 830e4f07a6SMauro Carvalho Chehab driver which duplicated an existing driver that the new author had been 840e4f07a6SMauro Carvalho Chehab unaware of. Code which reinvents existing wheels is not only wasteful; 850e4f07a6SMauro Carvalho Chehab it will also not be accepted into the mainline kernel. 860e4f07a6SMauro Carvalho Chehab 870e4f07a6SMauro Carvalho Chehab - There may be elements of the proposed solution which will not be 880e4f07a6SMauro Carvalho Chehab acceptable for mainline merging. It is better to find out about 890e4f07a6SMauro Carvalho Chehab problems like this before writing the code. 900e4f07a6SMauro Carvalho Chehab 910e4f07a6SMauro Carvalho Chehab - It's entirely possible that other developers have thought about the 920e4f07a6SMauro Carvalho Chehab problem; they may have ideas for a better solution, and may be willing 930e4f07a6SMauro Carvalho Chehab to help in the creation of that solution. 940e4f07a6SMauro Carvalho Chehab 950e4f07a6SMauro Carvalho ChehabYears of experience with the kernel development community have taught a 960e4f07a6SMauro Carvalho Chehabclear lesson: kernel code which is designed and developed behind closed 970e4f07a6SMauro Carvalho Chehabdoors invariably has problems which are only revealed when the code is 980e4f07a6SMauro Carvalho Chehabreleased into the community. Sometimes these problems are severe, 990e4f07a6SMauro Carvalho Chehabrequiring months or years of effort before the code can be brought up to 1000e4f07a6SMauro Carvalho Chehabthe kernel community's standards. Some examples include: 1010e4f07a6SMauro Carvalho Chehab 1020e4f07a6SMauro Carvalho Chehab - The Devicescape network stack was designed and implemented for 1030e4f07a6SMauro Carvalho Chehab single-processor systems. It could not be merged into the mainline 1040e4f07a6SMauro Carvalho Chehab until it was made suitable for multiprocessor systems. Retrofitting 1050e4f07a6SMauro Carvalho Chehab locking and such into code is a difficult task; as a result, the merging 1060e4f07a6SMauro Carvalho Chehab of this code (now called mac80211) was delayed for over a year. 1070e4f07a6SMauro Carvalho Chehab 1080e4f07a6SMauro Carvalho Chehab - The Reiser4 filesystem included a number of capabilities which, in the 1090e4f07a6SMauro Carvalho Chehab core kernel developers' opinion, should have been implemented in the 1100e4f07a6SMauro Carvalho Chehab virtual filesystem layer instead. It also included features which could 1110e4f07a6SMauro Carvalho Chehab not easily be implemented without exposing the system to user-caused 1120e4f07a6SMauro Carvalho Chehab deadlocks. The late revelation of these problems - and refusal to 1130e4f07a6SMauro Carvalho Chehab address some of them - has caused Reiser4 to stay out of the mainline 1140e4f07a6SMauro Carvalho Chehab kernel. 1150e4f07a6SMauro Carvalho Chehab 1160e4f07a6SMauro Carvalho Chehab - The AppArmor security module made use of internal virtual filesystem 1170e4f07a6SMauro Carvalho Chehab data structures in ways which were considered to be unsafe and 1180e4f07a6SMauro Carvalho Chehab unreliable. This concern (among others) kept AppArmor out of the 1190e4f07a6SMauro Carvalho Chehab mainline for years. 1200e4f07a6SMauro Carvalho Chehab 1210e4f07a6SMauro Carvalho ChehabIn each of these cases, a great deal of pain and extra work could have been 1220e4f07a6SMauro Carvalho Chehabavoided with some early discussion with the kernel developers. 1230e4f07a6SMauro Carvalho Chehab 1240e4f07a6SMauro Carvalho Chehab 1250e4f07a6SMauro Carvalho ChehabWho do you talk to? 1260e4f07a6SMauro Carvalho Chehab------------------- 1270e4f07a6SMauro Carvalho Chehab 1280e4f07a6SMauro Carvalho ChehabWhen developers decide to take their plans public, the next question will 1290e4f07a6SMauro Carvalho Chehabbe: where do we start? The answer is to find the right mailing list(s) and 1300e4f07a6SMauro Carvalho Chehabthe right maintainer. For mailing lists, the best approach is to look in 1310e4f07a6SMauro Carvalho Chehabthe MAINTAINERS file for a relevant place to post. If there is a suitable 1320e4f07a6SMauro Carvalho Chehabsubsystem list, posting there is often preferable to posting on 1330e4f07a6SMauro Carvalho Chehablinux-kernel; you are more likely to reach developers with expertise in the 1340e4f07a6SMauro Carvalho Chehabrelevant subsystem and the environment may be more supportive. 1350e4f07a6SMauro Carvalho Chehab 1360e4f07a6SMauro Carvalho ChehabFinding maintainers can be a bit harder. Again, the MAINTAINERS file is 1370e4f07a6SMauro Carvalho Chehabthe place to start. That file tends to not always be up to date, though, 1380e4f07a6SMauro Carvalho Chehaband not all subsystems are represented there. The person listed in the 1390e4f07a6SMauro Carvalho ChehabMAINTAINERS file may, in fact, not be the person who is actually acting in 1400e4f07a6SMauro Carvalho Chehabthat role currently. So, when there is doubt about who to contact, a 1410e4f07a6SMauro Carvalho Chehabuseful trick is to use git (and "git log" in particular) to see who is 1420e4f07a6SMauro Carvalho Chehabcurrently active within the subsystem of interest. Look at who is writing 1430e4f07a6SMauro Carvalho Chehabpatches, and who, if anybody, is attaching Signed-off-by lines to those 1440e4f07a6SMauro Carvalho Chehabpatches. Those are the people who will be best placed to help with a new 1450e4f07a6SMauro Carvalho Chehabdevelopment project. 1460e4f07a6SMauro Carvalho Chehab 1470e4f07a6SMauro Carvalho ChehabThe task of finding the right maintainer is sometimes challenging enough 1480e4f07a6SMauro Carvalho Chehabthat the kernel developers have added a script to ease the process: 1490e4f07a6SMauro Carvalho Chehab 1500e4f07a6SMauro Carvalho Chehab:: 1510e4f07a6SMauro Carvalho Chehab 1520e4f07a6SMauro Carvalho Chehab .../scripts/get_maintainer.pl 1530e4f07a6SMauro Carvalho Chehab 1540e4f07a6SMauro Carvalho ChehabThis script will return the current maintainer(s) for a given file or 1550e4f07a6SMauro Carvalho Chehabdirectory when given the "-f" option. If passed a patch on the 1560e4f07a6SMauro Carvalho Chehabcommand line, it will list the maintainers who should probably receive 157f1a69399SKrzysztof Kozlowskicopies of the patch. This is the preferred way (unlike "-f" option) to get the 158f1a69399SKrzysztof Kozlowskilist of people to Cc for your patches. There are a number of options 159f1a69399SKrzysztof Kozlowskiregulating how hard get_maintainer.pl will search for maintainers; please be 160f1a69399SKrzysztof Kozlowskicareful about using the more aggressive options as you may end up including 161f1a69399SKrzysztof Kozlowskidevelopers who have no real interest in the code you are modifying. 1620e4f07a6SMauro Carvalho Chehab 1630e4f07a6SMauro Carvalho ChehabIf all else fails, talking to Andrew Morton can be an effective way to 1640e4f07a6SMauro Carvalho Chehabtrack down a maintainer for a specific piece of code. 1650e4f07a6SMauro Carvalho Chehab 1660e4f07a6SMauro Carvalho Chehab 1670e4f07a6SMauro Carvalho ChehabWhen to post? 1680e4f07a6SMauro Carvalho Chehab------------- 1690e4f07a6SMauro Carvalho Chehab 1700e4f07a6SMauro Carvalho ChehabIf possible, posting your plans during the early stages can only be 1710e4f07a6SMauro Carvalho Chehabhelpful. Describe the problem being solved and any plans that have been 1720e4f07a6SMauro Carvalho Chehabmade on how the implementation will be done. Any information you can 1730e4f07a6SMauro Carvalho Chehabprovide can help the development community provide useful input on the 1740e4f07a6SMauro Carvalho Chehabproject. 1750e4f07a6SMauro Carvalho Chehab 1760e4f07a6SMauro Carvalho ChehabOne discouraging thing which can happen at this stage is not a hostile 1770e4f07a6SMauro Carvalho Chehabreaction, but, instead, little or no reaction at all. The sad truth of the 1780e4f07a6SMauro Carvalho Chehabmatter is (1) kernel developers tend to be busy, (2) there is no shortage 1790e4f07a6SMauro Carvalho Chehabof people with grand plans and little code (or even prospect of code) to 1800e4f07a6SMauro Carvalho Chehabback them up, and (3) nobody is obligated to review or comment on ideas 1810e4f07a6SMauro Carvalho Chehabposted by others. Beyond that, high-level designs often hide problems 182d4306db1SJunio C Hamanowhich are only revealed when somebody actually tries to implement those 1830e4f07a6SMauro Carvalho Chehabdesigns; for that reason, kernel developers would rather see the code. 1840e4f07a6SMauro Carvalho Chehab 1850e4f07a6SMauro Carvalho ChehabIf a request-for-comments posting yields little in the way of comments, do 1860e4f07a6SMauro Carvalho Chehabnot assume that it means there is no interest in the project. 1870e4f07a6SMauro Carvalho ChehabUnfortunately, you also cannot assume that there are no problems with your 1880e4f07a6SMauro Carvalho Chehabidea. The best thing to do in this situation is to proceed, keeping the 1890e4f07a6SMauro Carvalho Chehabcommunity informed as you go. 1900e4f07a6SMauro Carvalho Chehab 1910e4f07a6SMauro Carvalho Chehab 1920e4f07a6SMauro Carvalho ChehabGetting official buy-in 1930e4f07a6SMauro Carvalho Chehab----------------------- 1940e4f07a6SMauro Carvalho Chehab 1950e4f07a6SMauro Carvalho ChehabIf your work is being done in a corporate environment - as most Linux 1960e4f07a6SMauro Carvalho Chehabkernel work is - you must, obviously, have permission from suitably 1970e4f07a6SMauro Carvalho Chehabempowered managers before you can post your company's plans or code to a 1980e4f07a6SMauro Carvalho Chehabpublic mailing list. The posting of code which has not been cleared for 1990e4f07a6SMauro Carvalho Chehabrelease under a GPL-compatible license can be especially problematic; the 2000e4f07a6SMauro Carvalho Chehabsooner that a company's management and legal staff can agree on the posting 2010e4f07a6SMauro Carvalho Chehabof a kernel development project, the better off everybody involved will be. 2020e4f07a6SMauro Carvalho Chehab 2030e4f07a6SMauro Carvalho ChehabSome readers may be thinking at this point that their kernel work is 2040e4f07a6SMauro Carvalho Chehabintended to support a product which does not yet have an officially 2050e4f07a6SMauro Carvalho Chehabacknowledged existence. Revealing their employer's plans on a public 2060e4f07a6SMauro Carvalho Chehabmailing list may not be a viable option. In cases like this, it is worth 2070e4f07a6SMauro Carvalho Chehabconsidering whether the secrecy is really necessary; there is often no real 2080e4f07a6SMauro Carvalho Chehabneed to keep development plans behind closed doors. 2090e4f07a6SMauro Carvalho Chehab 2100e4f07a6SMauro Carvalho ChehabThat said, there are also cases where a company legitimately cannot 2110e4f07a6SMauro Carvalho Chehabdisclose its plans early in the development process. Companies with 2120e4f07a6SMauro Carvalho Chehabexperienced kernel developers may choose to proceed in an open-loop manner 2130e4f07a6SMauro Carvalho Chehabon the assumption that they will be able to avoid serious integration 2140e4f07a6SMauro Carvalho Chehabproblems later. For companies without that sort of in-house expertise, the 2150e4f07a6SMauro Carvalho Chehabbest option is often to hire an outside developer to review the plans under 2160e4f07a6SMauro Carvalho Chehaba non-disclosure agreement. The Linux Foundation operates an NDA program 2170e4f07a6SMauro Carvalho Chehabdesigned to help with this sort of situation; more information can be found 2180e4f07a6SMauro Carvalho Chehabat: 2190e4f07a6SMauro Carvalho Chehab 22099702304SJonathan Corbet https://www.linuxfoundation.org/nda/ 2210e4f07a6SMauro Carvalho Chehab 2220e4f07a6SMauro Carvalho ChehabThis kind of review is often enough to avoid serious problems later on 2230e4f07a6SMauro Carvalho Chehabwithout requiring public disclosure of the project. 224