1b28a2367SKishon Vijay Abraham I.. SPDX-License-Identifier: GPL-2.0 2b28a2367SKishon Vijay Abraham I 3b28a2367SKishon Vijay Abraham I=================================================================== 4b28a2367SKishon Vijay Abraham IPCI Non-Transparent Bridge (NTB) Endpoint Function (EPF) User Guide 5b28a2367SKishon Vijay Abraham I=================================================================== 6b28a2367SKishon Vijay Abraham I 7b28a2367SKishon Vijay Abraham I:Author: Kishon Vijay Abraham I <kishon@ti.com> 8b28a2367SKishon Vijay Abraham I 9b28a2367SKishon Vijay Abraham IThis document is a guide to help users use pci-epf-ntb function driver 10b28a2367SKishon Vijay Abraham Iand ntb_hw_epf host driver for NTB functionality. The list of steps to 11b28a2367SKishon Vijay Abraham Ibe followed in the host side and EP side is given below. For the hardware 12b28a2367SKishon Vijay Abraham Iconfiguration and internals of NTB using configurable endpoints see 13b28a2367SKishon Vijay Abraham IDocumentation/PCI/endpoint/pci-ntb-function.rst 14b28a2367SKishon Vijay Abraham I 15b28a2367SKishon Vijay Abraham IEndpoint Device 16b28a2367SKishon Vijay Abraham I=============== 17b28a2367SKishon Vijay Abraham I 18b28a2367SKishon Vijay Abraham IEndpoint Controller Devices 19b28a2367SKishon Vijay Abraham I--------------------------- 20b28a2367SKishon Vijay Abraham I 21b28a2367SKishon Vijay Abraham IFor implementing NTB functionality at least two endpoint controller devices 22b28a2367SKishon Vijay Abraham Iare required. 23b28a2367SKishon Vijay Abraham I 24b28a2367SKishon Vijay Abraham ITo find the list of endpoint controller devices in the system:: 25b28a2367SKishon Vijay Abraham I 26b28a2367SKishon Vijay Abraham I # ls /sys/class/pci_epc/ 27b28a2367SKishon Vijay Abraham I 2900000.pcie-ep 2910000.pcie-ep 28b28a2367SKishon Vijay Abraham I 29b28a2367SKishon Vijay Abraham IIf PCI_ENDPOINT_CONFIGFS is enabled:: 30b28a2367SKishon Vijay Abraham I 31b28a2367SKishon Vijay Abraham I # ls /sys/kernel/config/pci_ep/controllers 32b28a2367SKishon Vijay Abraham I 2900000.pcie-ep 2910000.pcie-ep 33b28a2367SKishon Vijay Abraham I 34b28a2367SKishon Vijay Abraham I 35b28a2367SKishon Vijay Abraham IEndpoint Function Drivers 36b28a2367SKishon Vijay Abraham I------------------------- 37b28a2367SKishon Vijay Abraham I 38b28a2367SKishon Vijay Abraham ITo find the list of endpoint function drivers in the system:: 39b28a2367SKishon Vijay Abraham I 40b28a2367SKishon Vijay Abraham I # ls /sys/bus/pci-epf/drivers 41b28a2367SKishon Vijay Abraham I pci_epf_ntb pci_epf_ntb 42b28a2367SKishon Vijay Abraham I 43b28a2367SKishon Vijay Abraham IIf PCI_ENDPOINT_CONFIGFS is enabled:: 44b28a2367SKishon Vijay Abraham I 45b28a2367SKishon Vijay Abraham I # ls /sys/kernel/config/pci_ep/functions 46b28a2367SKishon Vijay Abraham I pci_epf_ntb pci_epf_ntb 47b28a2367SKishon Vijay Abraham I 48b28a2367SKishon Vijay Abraham I 49b28a2367SKishon Vijay Abraham ICreating pci-epf-ntb Device 50b28a2367SKishon Vijay Abraham I---------------------------- 51b28a2367SKishon Vijay Abraham I 52b28a2367SKishon Vijay Abraham IPCI endpoint function device can be created using the configfs. To create 53b28a2367SKishon Vijay Abraham Ipci-epf-ntb device, the following commands can be used:: 54b28a2367SKishon Vijay Abraham I 55b28a2367SKishon Vijay Abraham I # mount -t configfs none /sys/kernel/config 56b28a2367SKishon Vijay Abraham I # cd /sys/kernel/config/pci_ep/ 57b28a2367SKishon Vijay Abraham I # mkdir functions/pci_epf_ntb/func1 58b28a2367SKishon Vijay Abraham I 59b28a2367SKishon Vijay Abraham IThe "mkdir func1" above creates the pci-epf-ntb function device that will 60b28a2367SKishon Vijay Abraham Ibe probed by pci_epf_ntb driver. 61b28a2367SKishon Vijay Abraham I 62b28a2367SKishon Vijay Abraham IThe PCI endpoint framework populates the directory with the following 63b28a2367SKishon Vijay Abraham Iconfigurable fields:: 64b28a2367SKishon Vijay Abraham I 65b28a2367SKishon Vijay Abraham I # ls functions/pci_epf_ntb/func1 66b28a2367SKishon Vijay Abraham I baseclass_code deviceid msi_interrupts pci-epf-ntb.0 67b28a2367SKishon Vijay Abraham I progif_code secondary subsys_id vendorid 68b28a2367SKishon Vijay Abraham I cache_line_size interrupt_pin msix_interrupts primary 69b28a2367SKishon Vijay Abraham I revid subclass_code subsys_vendor_id 70b28a2367SKishon Vijay Abraham I 71b28a2367SKishon Vijay Abraham IThe PCI endpoint function driver populates these entries with default values 72b28a2367SKishon Vijay Abraham Iwhen the device is bound to the driver. The pci-epf-ntb driver populates 73b28a2367SKishon Vijay Abraham Ivendorid with 0xffff and interrupt_pin with 0x0001:: 74b28a2367SKishon Vijay Abraham I 75b28a2367SKishon Vijay Abraham I # cat functions/pci_epf_ntb/func1/vendorid 76b28a2367SKishon Vijay Abraham I 0xffff 77b28a2367SKishon Vijay Abraham I # cat functions/pci_epf_ntb/func1/interrupt_pin 78b28a2367SKishon Vijay Abraham I 0x0001 79b28a2367SKishon Vijay Abraham I 80b28a2367SKishon Vijay Abraham I 81b28a2367SKishon Vijay Abraham IConfiguring pci-epf-ntb Device 82b28a2367SKishon Vijay Abraham I------------------------------- 83b28a2367SKishon Vijay Abraham I 84b28a2367SKishon Vijay Abraham IThe user can configure the pci-epf-ntb device using its configfs entry. In order 85b28a2367SKishon Vijay Abraham Ito change the vendorid and the deviceid, the following 86b28a2367SKishon Vijay Abraham Icommands can be used:: 87b28a2367SKishon Vijay Abraham I 88b28a2367SKishon Vijay Abraham I # echo 0x104c > functions/pci_epf_ntb/func1/vendorid 89b28a2367SKishon Vijay Abraham I # echo 0xb00d > functions/pci_epf_ntb/func1/deviceid 90b28a2367SKishon Vijay Abraham I 91*70b3740fSDamien Le MoalThe PCI endpoint framework also automatically creates a sub-directory in the 92*70b3740fSDamien Le Moalfunction attribute directory. This sub-directory has the same name as the name 93*70b3740fSDamien Le Moalof the function device and is populated with the following NTB specific 94*70b3740fSDamien Le Moalattributes that can be configured by the user:: 95b28a2367SKishon Vijay Abraham I 96b28a2367SKishon Vijay Abraham I # ls functions/pci_epf_ntb/func1/pci_epf_ntb.0/ 97b28a2367SKishon Vijay Abraham I db_count mw1 mw2 mw3 mw4 num_mws 98b28a2367SKishon Vijay Abraham I spad_count 99b28a2367SKishon Vijay Abraham I 100b28a2367SKishon Vijay Abraham IA sample configuration for NTB function is given below:: 101b28a2367SKishon Vijay Abraham I 102b28a2367SKishon Vijay Abraham I # echo 4 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/db_count 103b28a2367SKishon Vijay Abraham I # echo 128 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/spad_count 104b28a2367SKishon Vijay Abraham I # echo 2 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/num_mws 105b28a2367SKishon Vijay Abraham I # echo 0x100000 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/mw1 106b28a2367SKishon Vijay Abraham I # echo 0x100000 > functions/pci_epf_ntb/func1/pci_epf_ntb.0/mw2 107b28a2367SKishon Vijay Abraham I 108b28a2367SKishon Vijay Abraham IBinding pci-epf-ntb Device to EP Controller 109b28a2367SKishon Vijay Abraham I-------------------------------------------- 110b28a2367SKishon Vijay Abraham I 111b28a2367SKishon Vijay Abraham INTB function device should be attached to two PCI endpoint controllers 112b28a2367SKishon Vijay Abraham Iconnected to the two hosts. Use the 'primary' and 'secondary' entries 113b28a2367SKishon Vijay Abraham Iinside NTB function device to attach one PCI endpoint controller to 114b28a2367SKishon Vijay Abraham Iprimary interface and the other PCI endpoint controller to the secondary 115b28a2367SKishon Vijay Abraham Iinterface:: 116b28a2367SKishon Vijay Abraham I 117b28a2367SKishon Vijay Abraham I # ln -s controllers/2900000.pcie-ep/ functions/pci-epf-ntb/func1/primary 118b28a2367SKishon Vijay Abraham I # ln -s controllers/2910000.pcie-ep/ functions/pci-epf-ntb/func1/secondary 119b28a2367SKishon Vijay Abraham I 120b28a2367SKishon Vijay Abraham IOnce the above step is completed, both the PCI endpoint controllers are ready to 121b28a2367SKishon Vijay Abraham Iestablish a link with the host. 122b28a2367SKishon Vijay Abraham I 123b28a2367SKishon Vijay Abraham I 124b28a2367SKishon Vijay Abraham IStart the Link 125b28a2367SKishon Vijay Abraham I-------------- 126b28a2367SKishon Vijay Abraham I 127b28a2367SKishon Vijay Abraham IIn order for the endpoint device to establish a link with the host, the _start_ 128b28a2367SKishon Vijay Abraham Ifield should be populated with '1'. For NTB, both the PCI endpoint controllers 129b28a2367SKishon Vijay Abraham Ishould establish link with the host:: 130b28a2367SKishon Vijay Abraham I 131b28a2367SKishon Vijay Abraham I # echo 1 > controllers/2900000.pcie-ep/start 132b28a2367SKishon Vijay Abraham I # echo 1 > controllers/2910000.pcie-ep/start 133b28a2367SKishon Vijay Abraham I 134b28a2367SKishon Vijay Abraham I 135b28a2367SKishon Vijay Abraham IRootComplex Device 136b28a2367SKishon Vijay Abraham I================== 137b28a2367SKishon Vijay Abraham I 138b28a2367SKishon Vijay Abraham Ilspci Output 139b28a2367SKishon Vijay Abraham I------------ 140b28a2367SKishon Vijay Abraham I 141b28a2367SKishon Vijay Abraham INote that the devices listed here correspond to the values populated in 142b28a2367SKishon Vijay Abraham I"Creating pci-epf-ntb Device" section above:: 143b28a2367SKishon Vijay Abraham I 144b28a2367SKishon Vijay Abraham I # lspci 145b28a2367SKishon Vijay Abraham I 0000:00:00.0 PCI bridge: Texas Instruments Device b00d 146b28a2367SKishon Vijay Abraham I 0000:01:00.0 RAM memory: Texas Instruments Device b00d 147b28a2367SKishon Vijay Abraham I 148b28a2367SKishon Vijay Abraham I 149b28a2367SKishon Vijay Abraham IUsing ntb_hw_epf Device 150b28a2367SKishon Vijay Abraham I----------------------- 151b28a2367SKishon Vijay Abraham I 152b28a2367SKishon Vijay Abraham IThe host side software follows the standard NTB software architecture in Linux. 153b28a2367SKishon Vijay Abraham IAll the existing client side NTB utilities like NTB Transport Client and NTB 154b28a2367SKishon Vijay Abraham INetdev, NTB Ping Pong Test Client and NTB Tool Test Client can be used with NTB 155b28a2367SKishon Vijay Abraham Ifunction device. 156b28a2367SKishon Vijay Abraham I 157b28a2367SKishon Vijay Abraham IFor more information on NTB see 158b28a2367SKishon Vijay Abraham I:doc:`Non-Transparent Bridge <../../driver-api/ntb>` 159