PCIe example does not work.

[ciniml]

Hi.

I'm trying to run the PCIe demo, the bitstream in https://github.com/sipeed/TangMega-138KPro-example/blob/main/pcie/PCIe2.0_dma_demo/pcie_dma_demo.gar , but the board is not recognized as PCIe device at all.

I've tried both

1. program to SRAM and reboot the PCIe host PC (warm boot)
2. program to flash and turn on the PCIe host PC (cold boot)

but nothing worked for me.

After some investigation, I've found that the physical constraint in the archived project seems for GOWIN PCIe DVK, not for Tang Mega 138K.

So, could you provide the prebuilt bitstream and correct physical constraint to check my board works correctly?

[kyonmiriam]

Perhaps U could try this pre-built bitstream first, It is verified on Tang Mega 138K,
pcie_dma_demo.zip

[ciniml]

Thanks for providing the pre-build bitstream.

I've modified the physical constraint and comment out define dvk_brd line in top_define.v this morning and the board got recognized as a PCIe device.

Here is the modified physical constraint file I've confirmed to work in my environment.
dvk_cfg_v2.zip

But there are another problem about the DMA transfer demo. The demo hangs after around 250 transfer when I run the loop.sh script.

I'll try the pre-build bitstream and check that the demo runs well.

[kyonmiriam]

Thanks for UR physical constraint file, I will verify it.
In addition, what does the 250 transfer mean? Could U let me know Is it 250 times or what other parameters?

[ciniml]

The number shown by this line of loop.sh. I think it means number of runs of DMA transfer test.

https://github.com/sipeed/TangMega-138KPro-example/blob/main/pcie/PCIe2.0_dma_demo/gowin_pcie_demo/gowin_pcie_demo/loop.sh#L11

I confirmed that my board was recognized as a PCIe device with the pre-build bitstream.

I had another TangMega138K user check whether the DMA demo works with both the bitstream I built and the pre-built bitstream.
He reported that it runs without any issues.
It seems like the problem might be specific to my environment, so I will look into it.

[kko-smol]

Hi. I have too issues with this example.
I have 2 devices:

1. Jetson TX2 - doesn't detect the board. I tried pre built bitstream, build my-own it stream, add pauses to pcie-controller driver(suggestion from Jetson forum). But TX2 doesn't see device.
2. x86_64 with ubuntu(6.2.0-26-generic #26~22.04.1-Ubuntu SMP) - it may detect pcie device sometimes, but it is totally unstable. I have to make a lot of actions like reset board, run rescan, reboot PC and so on to make it visible. But when PC detected card - demo application doesn't work. irq counter in /proc/interrupts doesn't increase and demo application got stall.

What would you recommend to check?

In gowin ide I see timings error: div_clk is 100Mhz, but timing analysis say it can work only at ~82Mhz. Do you have same and design works?

[kyonmiriam]

Hi. I have too issues with this example. I have 2 devices:

1. Jetson TX2 - doesn't detect the board. I tried pre built bitstream, build my-own it stream, add pauses to pcie-controller driver(suggestion from Jetson forum). But TX2 doesn't see device.

2. x86_64 with ubuntu(6.2.0-26-generic #26~22.04.1-Ubuntu SMP) - it may detect pcie device sometimes, but it is totally unstable. I have to make a lot of actions like reset board, run rescan, reboot PC and so on to make it visible. But when PC detected card - demo application doesn't work. irq counter in /proc/interrupts doesn't increase and demo application got stall.

What would you recommend to check?

In gowin ide I see timings error: div_clk is 100Mhz, but timing analysis say it can work only at ~82Mhz. Do you have same and design works?

Here is reply:

1. Unfortunately, in my own tests, this DEMO is not working when connecting the Raspberry Pi CM4 or 5, and it may be attributed to the compatibility of the ARM PCIe driver.
2. For your x86_64 PC, maybe you should try to disable the Secure Boot in BIOS first. And don't forget to refer to the PCIe DMA Demo User Guide to modify boot arguments for your ubuntu. In fact, our test environment is Ubuntu 20.04 LTS, The updated system and kernel have not been tested yet. When GOWIN released a new update for this DEMO, I will test on the new system such as Ubuntu 22.04 ASAP.
3. Since this demo is too old, it seams could not be correct synthesis in the 1.9.9 version of the GOWIN IDE. So try the prebuilt bitstream first, This demo should got a update soon.

[ZYTek2024]

Perhaps U could try this pre-built bitstream first, It is verified on Tang Mega 138K, pcie_dma_demo.zip
@kyonmiriam I ask if it is possible to upload the complete project?

[ZYTek2024]

Hi.

I'm trying to run the PCIe demo, the bitstream in https://github.com/sipeed/TangMega-138KPro-example/blob/main/pcie/PCIe2.0_dma_demo/pcie_dma_demo.gar , but the board is not recognized as PCIe device at all.

I've tried both

1. program to SRAM and reboot the PCIe host PC (warm boot)
2. program to flash and turn on the PCIe host PC (cold boot)

but nothing worked for me.

After some investigation, I've found that the physical constraint in the archived project seems for GOWIN PCIe DVK, not for Tang Mega 138K.

So, could you provide the prebuilt bitstream and correct physical constraint to check my board works correctly?

@ciniml May I ask if it is possible to upload the complete project? program to SRAM and reboot the PCIe host PC (warm boot)，the only work.


[VVpupkin69]

I'm having trouble https://github.com/sipeed/TangMega-138KPro-example/blob/main/pcie/PCIe2.0_dma_demo
I'm building a driver for ubuntu, and I'm doing everything according to the instructions. If the card is connected to the PCI slot at the time of PC startup, everything works. But if you connect the board after starting the PC, the example does not work. The analysis showed that the operating system finds the device, the status registers and the rest are read through the terminal and their values are valid. But DMA doesn't work gwbar->ctrl always becomes 0xffffffff. And all other fields of the gwbar structure are also 0xffffffff.
Is there any way to connect the board to an already running PC?

[kyonmiriam]

@VVpupkin69

Hi, thank you for your feedback

This demo and driver currently support a cold boot scenario—meaning the device must be connected at system startup. When the PCIe device is connected after boot, the OS detects it, but the necessary hardware initialization (like mapping DMA registers) doesn't occur, which is why registers such as gwbar->ctrl return 0xffffffff.

Although PCIe hot plug is possible, it requires additional support from the hardware, OS, and driver. For now, please connect the device before booting. If you need hot plug functionality, additional development will be required.

[VVpupkin69]

Thank you for your reply. Could you please clarify if the modification needs to be made to the "gowin_demo.ko" file or to the device's firmware?

[kevpatt]

I opened three issues today related to this demo, which include patches, solutions, and workarounds. Some of them may apply to one or more of the issues in this thread. I am able to build and run the demo (from flash) with the latest IDE, and both the individual and loop tests run stable with no problems.

[cklam12345]

anyone got a window driver I can install? I am using RW to test the pcie 8G demo, the BAR 1 at 0x0000004 is writable. does that mean it work. the window driver even after I enabled the test mode complain in the device manager as Code 10.

[cklam12345]

btw, I am using 138K dock, pcie demo example

[Friend-zva]

After spending about six months studying and trying to run the demo project, I decided to share my experience with the community.

Gen3 Project

I saw many questions related to the initial setup and board visibility in lspci. In my case, it helped that I used the Gen3 project and a laptop to load the bitstream. The board was not detected if it was connected to an already running host.

I also encountered issues caused by incorrect SerDes IP configuration: after changing the line width or link speed, the Gen3 project stopped working. At the same time, Gen2 did not work out of the box (which is also mentioned in the readme). If you encounter similar issues, it may be worth checking and adjusting the SerDes IP settings in Gowin IDE.

The following issues also helped me: #16, #17, #18. I encountered the same problems on a newer kernel version and applied the fixes mentioned there. With the kernel version used by the developers of the host demo, these fixes were not required.

PCIe SGDMA

While following Gowin updates, I discovered the new PCIe SGDMA IP, which internally handles TLP routing and processing and exposes an AXI-Stream interface on the output side. The link contains the User Guide, HDL, and Windows Driver. Note that you will need to update Gowin IDE to the latest version.

Using the PCIe SGDMA IP, I implemented h2c&c2h PCIe communication and DDR data storage. The implementation is available in this repository.

Troubleshooting

During implementation, I encountered an endian-related issue that I could not find mentioned in the PCIe SGDMA IP User Guide. It may also be worth checking this in the current repository demo. Although the demo appeared to run successfully, no actual data transfer was happening: the reported transfer speed was around 1700%, while the dumped data at the destination address consisted entirely of zeros.

You can also check the modified version of this demo in the tlp branch (or review the git history). In the end, it did not work correctly, but after fixing the endian issue, it likely would have behaved as expected.