14ac8c8e5SFrank Li.. SPDX-License-Identifier: GPL-2.0 24ac8c8e5SFrank Li 34ac8c8e5SFrank Li=================================================================== 44ac8c8e5SFrank LiPCI Non-Transparent Bridge (NTB) Endpoint Function (EPF) User Guide 54ac8c8e5SFrank Li=================================================================== 64ac8c8e5SFrank Li 74ac8c8e5SFrank Li:Author: Frank Li <Frank.Li@nxp.com> 84ac8c8e5SFrank Li 94ac8c8e5SFrank LiThis document is a guide to help users use pci-epf-vntb function driver 104ac8c8e5SFrank Liand ntb_hw_epf host driver for NTB functionality. The list of steps to 114ac8c8e5SFrank Libe followed in the host side and EP side is given below. For the hardware 124ac8c8e5SFrank Liconfiguration and internals of NTB using configurable endpoints see 134ac8c8e5SFrank LiDocumentation/PCI/endpoint/pci-vntb-function.rst 144ac8c8e5SFrank Li 154ac8c8e5SFrank LiEndpoint Device 164ac8c8e5SFrank Li=============== 174ac8c8e5SFrank Li 184ac8c8e5SFrank LiEndpoint Controller Devices 194ac8c8e5SFrank Li--------------------------- 204ac8c8e5SFrank Li 214ac8c8e5SFrank LiTo find the list of endpoint controller devices in the system:: 224ac8c8e5SFrank Li 234ac8c8e5SFrank Li # ls /sys/class/pci_epc/ 244ac8c8e5SFrank Li 5f010000.pcie_ep 254ac8c8e5SFrank Li 264ac8c8e5SFrank LiIf PCI_ENDPOINT_CONFIGFS is enabled:: 274ac8c8e5SFrank Li 284ac8c8e5SFrank Li # ls /sys/kernel/config/pci_ep/controllers 294ac8c8e5SFrank Li 5f010000.pcie_ep 304ac8c8e5SFrank Li 314ac8c8e5SFrank LiEndpoint Function Drivers 324ac8c8e5SFrank Li------------------------- 334ac8c8e5SFrank Li 344ac8c8e5SFrank LiTo find the list of endpoint function drivers in the system:: 354ac8c8e5SFrank Li 364ac8c8e5SFrank Li # ls /sys/bus/pci-epf/drivers 374ac8c8e5SFrank Li pci_epf_ntb pci_epf_test pci_epf_vntb 384ac8c8e5SFrank Li 394ac8c8e5SFrank LiIf PCI_ENDPOINT_CONFIGFS is enabled:: 404ac8c8e5SFrank Li 414ac8c8e5SFrank Li # ls /sys/kernel/config/pci_ep/functions 424ac8c8e5SFrank Li pci_epf_ntb pci_epf_test pci_epf_vntb 434ac8c8e5SFrank Li 444ac8c8e5SFrank Li 454ac8c8e5SFrank LiCreating pci-epf-vntb Device 464ac8c8e5SFrank Li---------------------------- 474ac8c8e5SFrank Li 484ac8c8e5SFrank LiPCI endpoint function device can be created using the configfs. To create 494ac8c8e5SFrank Lipci-epf-vntb device, the following commands can be used:: 504ac8c8e5SFrank Li 514ac8c8e5SFrank Li # mount -t configfs none /sys/kernel/config 524ac8c8e5SFrank Li # cd /sys/kernel/config/pci_ep/ 534ac8c8e5SFrank Li # mkdir functions/pci_epf_vntb/func1 544ac8c8e5SFrank Li 554ac8c8e5SFrank LiThe "mkdir func1" above creates the pci-epf-ntb function device that will 564ac8c8e5SFrank Libe probed by pci_epf_vntb driver. 574ac8c8e5SFrank Li 584ac8c8e5SFrank LiThe PCI endpoint framework populates the directory with the following 594ac8c8e5SFrank Liconfigurable fields:: 604ac8c8e5SFrank Li 614ac8c8e5SFrank Li # ls functions/pci_epf_ntb/func1 624ac8c8e5SFrank Li baseclass_code deviceid msi_interrupts pci-epf-ntb.0 634ac8c8e5SFrank Li progif_code secondary subsys_id vendorid 644ac8c8e5SFrank Li cache_line_size interrupt_pin msix_interrupts primary 654ac8c8e5SFrank Li revid subclass_code subsys_vendor_id 664ac8c8e5SFrank Li 674ac8c8e5SFrank LiThe PCI endpoint function driver populates these entries with default values 684ac8c8e5SFrank Liwhen the device is bound to the driver. The pci-epf-vntb driver populates 694ac8c8e5SFrank Livendorid with 0xffff and interrupt_pin with 0x0001:: 704ac8c8e5SFrank Li 714ac8c8e5SFrank Li # cat functions/pci_epf_vntb/func1/vendorid 724ac8c8e5SFrank Li 0xffff 734ac8c8e5SFrank Li # cat functions/pci_epf_vntb/func1/interrupt_pin 744ac8c8e5SFrank Li 0x0001 754ac8c8e5SFrank Li 764ac8c8e5SFrank Li 774ac8c8e5SFrank LiConfiguring pci-epf-vntb Device 784ac8c8e5SFrank Li------------------------------- 794ac8c8e5SFrank Li 804ac8c8e5SFrank LiThe user can configure the pci-epf-vntb device using its configfs entry. In order 814ac8c8e5SFrank Lito change the vendorid and the deviceid, the following 824ac8c8e5SFrank Licommands can be used:: 834ac8c8e5SFrank Li 844ac8c8e5SFrank Li # echo 0x1957 > functions/pci_epf_vntb/func1/vendorid 854ac8c8e5SFrank Li # echo 0x0809 > functions/pci_epf_vntb/func1/deviceid 864ac8c8e5SFrank Li 87*70b3740fSDamien Le MoalThe PCI endpoint framework also automatically creates a sub-directory in the 88*70b3740fSDamien Le Moalfunction attribute directory. This sub-directory has the same name as the name 89*70b3740fSDamien Le Moalof the function device and is populated with the following NTB specific 90*70b3740fSDamien Le Moalattributes that can be configured by the user:: 914ac8c8e5SFrank Li 924ac8c8e5SFrank Li # ls functions/pci_epf_vntb/func1/pci_epf_vntb.0/ 934ac8c8e5SFrank Li db_count mw1 mw2 mw3 mw4 num_mws 944ac8c8e5SFrank Li spad_count 954ac8c8e5SFrank Li 964ac8c8e5SFrank LiA sample configuration for NTB function is given below:: 974ac8c8e5SFrank Li 984ac8c8e5SFrank Li # echo 4 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/db_count 994ac8c8e5SFrank Li # echo 128 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/spad_count 1004ac8c8e5SFrank Li # echo 1 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/num_mws 1014ac8c8e5SFrank Li # echo 0x100000 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/mw1 1024ac8c8e5SFrank Li 103*70b3740fSDamien Le MoalA sample configuration for virtual NTB driver for virtual PCI bus:: 1044ac8c8e5SFrank Li 1054ac8c8e5SFrank Li # echo 0x1957 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/vntb_vid 1064ac8c8e5SFrank Li # echo 0x080A > functions/pci_epf_vntb/func1/pci_epf_vntb.0/vntb_pid 1074ac8c8e5SFrank Li # echo 0x10 > functions/pci_epf_vntb/func1/pci_epf_vntb.0/vbus_number 1084ac8c8e5SFrank Li 1094ac8c8e5SFrank LiBinding pci-epf-ntb Device to EP Controller 1104ac8c8e5SFrank Li-------------------------------------------- 1114ac8c8e5SFrank Li 1124ac8c8e5SFrank LiNTB function device should be attached to PCI endpoint controllers 1134ac8c8e5SFrank Liconnected to the host. 1144ac8c8e5SFrank Li 1154ac8c8e5SFrank Li # ln -s controllers/5f010000.pcie_ep functions/pci-epf-ntb/func1/primary 1164ac8c8e5SFrank Li 1174ac8c8e5SFrank LiOnce the above step is completed, the PCI endpoint controllers are ready to 1184ac8c8e5SFrank Liestablish a link with the host. 1194ac8c8e5SFrank Li 1204ac8c8e5SFrank Li 1214ac8c8e5SFrank LiStart the Link 1224ac8c8e5SFrank Li-------------- 1234ac8c8e5SFrank Li 1244ac8c8e5SFrank LiIn order for the endpoint device to establish a link with the host, the _start_ 1254ac8c8e5SFrank Lifield should be populated with '1'. For NTB, both the PCI endpoint controllers 1264ac8c8e5SFrank Lishould establish link with the host (imx8 don't need this steps):: 1274ac8c8e5SFrank Li 1284ac8c8e5SFrank Li # echo 1 > controllers/5f010000.pcie_ep/start 1294ac8c8e5SFrank Li 1304ac8c8e5SFrank LiRootComplex Device 1314ac8c8e5SFrank Li================== 1324ac8c8e5SFrank Li 1334ac8c8e5SFrank Lilspci Output at Host side 1349458c27aSBagas Sanjaya------------------------- 1354ac8c8e5SFrank Li 1364ac8c8e5SFrank LiNote that the devices listed here correspond to the values populated in 1374ac8c8e5SFrank Li"Creating pci-epf-ntb Device" section above:: 1384ac8c8e5SFrank Li 1394ac8c8e5SFrank Li # lspci 1404ac8c8e5SFrank Li 00:00.0 PCI bridge: Freescale Semiconductor Inc Device 0000 (rev 01) 1414ac8c8e5SFrank Li 01:00.0 RAM memory: Freescale Semiconductor Inc Device 0809 1424ac8c8e5SFrank Li 1434ac8c8e5SFrank LiEndpoint Device / Virtual PCI bus 1444ac8c8e5SFrank Li================================= 1454ac8c8e5SFrank Li 1464ac8c8e5SFrank Lilspci Output at EP Side / Virtual PCI bus 1474ac8c8e5SFrank Li----------------------------------------- 1484ac8c8e5SFrank Li 1494ac8c8e5SFrank LiNote that the devices listed here correspond to the values populated in 1504ac8c8e5SFrank Li"Creating pci-epf-ntb Device" section above:: 1514ac8c8e5SFrank Li 1524ac8c8e5SFrank Li # lspci 1534ac8c8e5SFrank Li 10:00.0 Unassigned class [ffff]: Dawicontrol Computersysteme GmbH Device 1234 (rev ff) 1544ac8c8e5SFrank Li 1554ac8c8e5SFrank LiUsing ntb_hw_epf Device 1564ac8c8e5SFrank Li----------------------- 1574ac8c8e5SFrank Li 1584ac8c8e5SFrank LiThe host side software follows the standard NTB software architecture in Linux. 1594ac8c8e5SFrank LiAll the existing client side NTB utilities like NTB Transport Client and NTB 1604ac8c8e5SFrank LiNetdev, NTB Ping Pong Test Client and NTB Tool Test Client can be used with NTB 1614ac8c8e5SFrank Lifunction device. 1624ac8c8e5SFrank Li 1634ac8c8e5SFrank LiFor more information on NTB see 1644ac8c8e5SFrank Li:doc:`Non-Transparent Bridge <../../driver-api/ntb>` 165