Today, we are going to talk about the work we did with our friends from Thales in order to support CVA6 on an Agilex7 development kit (DK-DEV-AGF014EA), both in the HW platform, and the Linux layers that run on it (i.e. Buildroot and Yocto). This targets the 32-bit and 64-bit CVA6 configurations already supported in Genesys 2 board. Given that the software is already supported for CVA6 on Genesys 2 platform, the work is mainly on configuring and supporting the peripherals that differ between the boards. Easy, right?

Port the existing “CVA6 APU” to Quartus

The first step was to migrate the existing FPGA design in OpenHW repository to Altera technology and Quartus software. This design is depicted in Figure 1.

Figure 1. Original CVA6 APU design for Xilinx FPGA

Porting IPs and Primitives used

The existing design is using a combination of open source and Xilinx proprietary IPs. Additionally, it is also using some specific Xilinx primitives. The initial assumption was that all the open source modules could be reused, and Xilinx proprietary ones needed to be updated, some with open source IPs, and some with Altera/Intel proprietary IPs.

• The Xilinx generated IPs that were replaced by open source IPs are the following:
  • GPIO: https://github.com/pulp-platform/gpio.git
  • SPI: https://github.com/pulp-platform/axi_spi_master.git
  • AXI data width converter: Verilog AXI components for FPGA implementation
    (https://github.com/alexforencich/verilog-axi, MIT license)
• The Xilinx generated IPs/Primitives that were replaced by Altera/Intel proprietary IPs/Primitives are the following:
  • Xilinx DDR3 controller → Intel EMIF IP (DDR4 configuration)
  • Clocks
    • PLL REF: 200 MHz → 100 MHz (input from board)
    • DDR clock: 900 MHz → 1200 MHz (generated by Intel EMIF IP)
    • Main clock: 50 MHz –> 100 MHz (generated by the Altera PLL)
  • I/O buffers
    • Xilinx IOBUF → Intel GPIO IP in bidirectional mode, used in ariane_peripherals, for the Ethernet IP (open source)
    • Xilinx IODDR2 → Intel GPIO IP in DDR mode, both input and output, used in the Ethernet IP
  • Memory primitives (in APU, not about optimizations yet)
    • Xilinx RAMB16 → Intel Altsyncram dual port, used in the Ethernet IP

Placing the design in the HW platform

After migrating the IPs and primitives, the next step was to create the new pin mapping using the interfaces available in the Altera FPGA Board. However, we could not use the interfaces as planned. It is not possible to connect the HPS I/O pins to the FPGA fabric and use them there, and there are no GPIOs in the FPGA accessible externally in the board (see Figure 2), as the pins are connected to the HPS(Hard Processor System), not to the FPGA fabric.
This meant that our initial assumption “all the open source modules could be reused” was not 100% correct. The cores are there, can be synthesized and placed in the FPGA fabric; however, connecting the cores with the corresponding interfaces on the board was not feasible!

 

Figure 2. Intel Agilex 7 Block Diagram (Figure taken from Intel website)

IIn our design, the blocks affected by this limitation are: debug module, UART, Ethernet and SPI.

Luckily, it is possible to access the FPGA from the JTAG connector through another FPGA (MAX10).
The solution to use this bypass is to use Altera/Intel IPs that provide these connections.

• For UART, we use Intel JTAG UART IP.
• For debug module, we use a modified version of the DMI JTAG (DMI vJTAG) that connects CVA6 with Intel vJTAG IP (done by Thales).

After these modifications and the previous migration of IPs, the design looks like this.

Figure 3. First version of CVA6 APU migrated to Altera Technology

• For SD card and Ethernet, the story is different. Since they are not connected to the FPGA, it is required to make the connection through the HPS.

Figure 4. HPS block diagram, with connection from FPGA to SD card and Ethernet highlighted in red
(Figure taken from Intel website)

As seen in the figure, the connection is feasible, but it is not simple, since it traverses several modules inside HPS. From the figure it could seem that a simpler connection is the one on the right from the Secure Device Manager (SDM) in the FPGA to L3 interconnect. However, it can’t be used for this purpose. Given that accessing the CPU peripherals from the FPGA is not the main use case of this platform, we could not find examples or documentation on how to do it. In order to be able to solve this problem, we contacted the technical support, and worked together to have a reference design for this board (which did not exist) and add the support to enable this connection. Apart from configuring the HW, in order for this bridge to be enabled, the HPS core must boot until u-boot stage and execute some commands at this level. With the help of Altera technical support, we could enable this path and successfully control the SD Card controller from CVA6 inside the FPGA.

With this, the complete APU design for Altera board looks like Figure 5. Note that the HPS has its own EMIF IP to access the DDR. Since the board has 4 DDR slots, one is used by the HPS and the others are used by CVA6 on the FPGA without any conflicts.

Figure 5. Complete CVA6 APU for Agilex7

Once the path was enabled, we developed the Boot ROM code to copy the content on the SD-Card to the main memory of CVA6, to finally boot from there. This was done using the u-boot driver for MMC cards as baseline, and adapting it to baremetal code. It was not possible to reuse the driver from Genesys 2 because the connection there was through an SPI, and this is not the case in Agilex 7. With this, we have a minimalistic version of this Boot ROM, which is able to transfer data to the main memory in order to start the boot process!

This HW modifications would be enough for Buildroot support, since the whole image is loaded in RAM and the Linux file system is mounted there, meaning that the only use of the SD card is to store the images that will be loaded into memory at the beginning of the boot. In Yocto, this is different, because the file system is mounted in the SD card. At HW level, this means that the interrupt from the SDMMC controller needs to be connected to CVA6 PLIC.

After this, HW is ready and we can move on with SW modifications! The complete configuration of HPS subsystem is automated via a TCL script.

Adapting the SW layers to Altera platform

The adaptation of SW to the new HW required changes at the following layers:

• OpenSBI
• U-boot
• Buildroot
• Yocto

Both buildroot and yocto rely on OpenSBI and U-boot to boot the kernel, so the adaptation of these is usable for both layers.

OpenSBI

The main change required here is the introduction of a new HW Platform under platform/fpga/cva6-altera which is based on the platform/fpga/cva6 that was created for Genesys 2 support.

A second difference is the use of the JTAG UART instead of the regular uart8250. Since the differences between these UARTs are actually small, we created a patch that makes the following changes to the base uart8250:

• include new putc and getc functions: The main change here is which register provides information of ready and how the information is extracted

Figure 6. New functions for Altera JTAG UART management in OpenSBI

• include a simplified init function: in this case the HW is pre-configured and no software config is needed (nor possible).

Figure 7. Simplified Altera JTAG UART init function

The patch created to add the new cva6-altera platform and make the required UART modifications is available here.

U-boot

The main changes here are related to the introduction of a new board under board/openhwgroup/cva6_agilex7, a new config file “cv(XLEN)a6_agilex7_defconfig” with XLEN=32 or 64, and the corresponding device tree files.

• Differences in config files are mostly about changing the driver used to access the SD card. In Genesys 2 that is an SPI Xilinx driver, in the case of Agilex7 it is the Synopsis DW MMC driver.

Figure 8. Change of driver to access SD card in u-boot configuration

• Similarly, in the device tree the driver used to access the SD card and its configuration differs. There are also other small modifications like the frequency of operation.

Figure 9. Change of driver to access SD card in u-boot device tree

• Apart from this, again the UART driver (ns16550 in this case) requires modification of putc, getc and init functions, similar to the changes done for OpenSBI UART.
• Last, the snps_dw_mmc is also modified via patch in order to not repeat the card initialization, which was done by HPS.

All the details of the modifications done for Agilex7 support in u-boot are available here.

Buildroot

• In this case, one more aspect of the UART driver must be adapted, which is the register mapping of interrupts.
• In the Linux configuration, SPI related items and Xilinx specific drivers are removed. There is no need to configure the MMC access here, because Buildroot does not use the SD card.

And that’s it! Now we can build the buildroot image for Agilex 7!

Yocto

• For Yocto, new machine configurations are created for both cv32a6-agilex7 and cv64a6-agilex7 which are based on the Genesys 2 ones. This is done to handle the build of the Yocto layer.
• In the BSP, the OpenSBI and u-boot modifications mentioned above are included.
• At kernel layer:
  • New device trees are created for Agilex7 board, with same changes as in u-boot (Figure 13).
  • The UART driver must be adapted similarly to what has been done for the Buildroot version.
  • For the SD card, the driver used in this case is “snps,dw-mshc”. Some modifications are also required here:
    • FIFO mode is used, no DMA 
    • 
      Figure 10. Modify dw_mmc driver to not use DMA
    • Read and Write transactions must be done in SINGLE block.
    • When writing to the SD card, an issue was identified where not all the data sent via AXI to the SDMMC controller arrived to the corresponding FIFO. We have not been able to identify the root cause for this, but it seems data is being lost inside the HPS. A workaround is to add a small delay between write operations by reading the status register ensuring the FIFO is not full. With this addition, data is no longer lost and write operations can be successfully performed.

Figure 11. Workaround to avoid data loss when writing to SD controller FIFO

• The kernel configuration is also updated to select the DW MMC driver instead of SPI one.

Figure 12. Selection of MMC DW driver in kernel configuration

With these modifications a complete boot of Yocto on CVA6 on Agilex7 is possible. One last remaining issue was that the console was continuously dumping unknown characters. This was found to be related to the console configuration. Now, instead of configuring the console in the config file with SERIAL_CONSOLES = “115200;ttyS0”, it is done in the busybox inittab with console::respawn:/sbin/getty -L console 0 vt100 # GENERIC_SERIAL.

With this, the images are ready to boot Yocto on Agilex 7 board!

Preparing the SD card

The last step to be able to boot either buildroot or yocto on cva6 on the agilex7 board is to prepare an SD card with the images. This is again different between Genesys 2 and Agilex 7, but why? Remember that in order to enable the HW path, the HPS must boot until u-boot stage and execute some commands that enable the bridge between the FPGA and the HPS peripherals (red path in Figure 4). Therefore, the SD card for Agilex must contain not only the images for CVA6, but also for the HPS. The layout of partitions for the two platforms in buildroot and yocto is shown in the next Figure.

Figure 13. SD card partitions for different layers and platforms

In order to generate the HPS u-boot, we followed the instructions from Altera here. The modifications we did in order to automatically enable the FPGA bridge are available here.

The makefile in both repos takes care of this for us, so now we can finally see the boot of Buildroot or Yocto on the board!

Figure 14. Buildroot boot on CVA6 on Agilex7

Figure 15. Yocto boot on CVA6 on Agilex7

Acknowledgements

Thanks to our friends from Thales for supporting us on this work, Jérôme Quevremont, Christophe Biquard, Jullien Mallet, Nicolas Ventroux and also to Altera support Sunil Kallur Ramegowda. It’s a pleasure to work with you guys!