Annual: 2019

AS013 »
Cloud deployable FPGA configuration
📁Internet of Things
👤Paul DeCarlo
 (Microsoft)
📅Oct 17, 2019
Regional Final

0


👀 278   💬 1

AS013 » Cloud deployable FPGA configuration

Description

Azure IoT Edge enables developers to deploy containerized modules to internet connected devices which allows for maintaining a desired state of running services through cloud-configured deployment configurations. This mechanism also offers the ability to securely update running modules at runtime on remote devices via changes to this configuration.

This IoT Edge module will allow the user to configure the FPGA portion of the Cyclone V SoC from Linux within an IoT Edge module, allowing for a robust deployment mechanism for shipping FPGA configurations to remote devices at scale.

This can allow for FPGA enabled devices to be dynamically reprogrammed in the field, without physical access. As such, we can now deliver hardware configurable updates over the air to enable a wide variety of never before seen use cases including fixing issues without need for a physical recall, updating configurations to be more performant or add new features, and allowing devices to be bootstrap with remotely delivered FPGA configurations on first run.

Demo Video

  • URL: https://youtu.be/puXh0V_DZ4Y

  • Project Proposal

    1. High-level Project Description

    Project Description

    Azure IoT Edge enables developers to deploy containerized modules to internet connected devices which allows for maintaining a desired state of running services through cloud-configured deployment configurations. This mechanism also offers the ability to securely update running modules at runtime on remote devices via changes to this configuration.

    This IoT Edge module will allow the user to configure the FPGA portion of the Cyclone V SoC from Linux within an IoT Edge module, allowing for a robust deployment mechanism for shipping FPGA configurations to remote devices at scale. 

    This can allow for FPGA enabled devices to  be dynamically reprogrammed in the field, without physical access.  As such, we can now deliver hardware configurable updates over the air to enable a wide variety of never before seen use cases including fixing issues without need for a physical recall, updating configurations to be more performant or add new features, and allowing devices to be bootstrap with remotely delivered FPGA configurations on first run.

    Project Repository

    The project is currently open source and hosted on Github

    Video Demonstration

    A video demonstration of the project can be viewed on Youtube 

    2. Block Diagram

     

    3. Intel FPGA virtues in Your Project

    Our project allows us to demonstrate the Intel FPGA Virtues in a real-world solution, with an open-ended ability to deliver on these virtues in a wide variety of use cases.  

    Adapt to Change
    Our strongest virtue is exemplified in our ability to deliver any arbitrary FPGA configuration remotely using a desired state configuration specified in the Microsoft Azure cloud.  This allows us to adapt to change by giving developers the ability to ship new FPGA configurations in response to changes in requirements, security updates, or performance improvements.  Furthermore, by leveraging the Azure IoT Edge runtime, we are able to enahance reliability by allowing for remote monitoring of the status of our module through device telemetry reported back to the cloud.  This allows us to safely and securely allow for changes to our FPGA design at any point in the distribution process, even while deployed in a potentially intermittently connected production environment, and give us the ability to monitor the health of our device in real-time.

    Boosts Performance
    We are able to boost performance of arbitrary workloads by providing an aveneue for communicating directly with the onboard FPGA from our deployed IoT Edge module and then share those results between any additionally deployed modules.  This can allow us to pass a processd result from the FPGA to be used in a secondary module for things like aggregation and machine learning.  This orchestration of modules can be deployed to a bare device using a desired state configuration specified in the cloud allowing for manuevarability and expandability.

    Expands I/O
    Finally, due to our ability to change configurations at runtime, we are able to employ a solution which allows for dynamic configuration of the FPGA to allow for communication with a variety of connected devices that may not have been forseen in initial production.  We are able to adapt to technically any device or protocol which can be provided using an FPGA configuration.  For example, if a device is deployed to a factory floor and using Modbus protocol and is later updated to a new communication standard, we can ship an update to the device as it is deployed in the field to allow it to expand to whatever I/O device may be required.
     

    4. Design Introduction

    How it works

    Azure IoT Edge enables developers to deploy containerized modules to internet connected devices which allows for maintaining a desired state of running services through cloud-configured deployment configurations. This mechanism also offers the ability to securely update running modules on devices remotely via changes to this configuration.

    This IoT Edge module configures the FPGA portion of the Cyclone V SoC from Linux within an Iot Edge module, allowing for a robust deployment mechanism for shipping FPGA configurations to remote devices at scale.

    This is accomplished using direct register access to take control of the FPGA Manager device which is used by the HPS to configure the FPGA. A multi-stage Dockerfile builds this component using SoCEDS-18.1.0.625 and provides it to an IoT Edge module which maps /dev/mem from the host to allow for interaction with the FPGA from the containerized module.
     

    Kernel / OS Requirements

    An IoT Edge Compatible kernel / OS needs to be installed onto the target device. Microsoft officially provides .deb packages for IoTEdge for ARM, as such your OS choice needs to be Debian compatible.

    You will want to ensure that your device is connected to internet via the ethernet port and obtain a serial connection by following these instructions.

    Once connected to your device, ensure that is up to date and has curl installed with:

    sudo apt update
    sudo apt upgrade
    sudo apt install curl
    

    IoT Edge itself depends on an installation of Moby, to verify that your FPGA device's kernel can support this, run and view the output of:

    curl -sSL https://raw.githubusercontent.com/moby/moby/master/contrib/check-config.sh | bash

    You will want to ensure that you have all of the items under Generally Necessary and Network Drivers: - "overlay" marked as enabled.

    If these are not enabled, then you will need to refer to the source code of your kernel to enable these features.

    If you need to compile and setup a new kernel / OS from source, there are instructions avaialable @ https://www.digikey.com/eewiki/display/linuxonarm/DE10-Nano+Kit. Please note that following these steps without modification will produce an image that lacks CONFIG_BRIDGE, CONFIG_VETH, CONFIG_VXLAN modules which are required by Moby.

    The missing modules can be enabled during the execution of ./build_kernel.sh in the associated steps for building the Linux Kernel. This will produce a menuconfig prompt that allows for enabling / disabling key kernel configuration options

    CONFIG_BRIDGE can be enabled under Networking support > Networking options - 802.1d Ethernet Bridging

    CONFIG_VETH can be enabled under Device Drivers > Network device support - Virtual ethernet pair device

    CONFIG_VXLAN can be enabled under Device Drivers > Network device support - Virtual eXtensible Local Area Network (VXLAN)

    When you have confirmed that you have satisfied the requirements for Moby, install it onto your device with the following:

    # You can copy the entire text from this code block and 
    # paste in terminal. The comment lines will be ignored.
    
    # Download and install the moby-engine
    curl -L https://aka.ms/moby-engine-armhf-latest -o moby_engine.deb && sudo dpkg -i ./moby_engine.deb
    
    # Download and install the moby-cli
    curl -L https://aka.ms/moby-cli-armhf-latest -o moby_cli.deb && sudo dpkg -i ./moby_cli.deb
    
    # Run apt-get fix
    sudo apt-get install -f
    

    After a few minutes you should be able to verify that Moby has installed properly by running:

    systemctl status docker

    5. Function Description

    Requirements:

    Clone the project repo then open the project folder in Visual Studio Code and create a deployment for the IoT Edge device by right-clicking deployment.template.json and select Generate IoT Edge Deployment Manifest. This will create a file under the config folder named deployment.arm32v7.json, right-click that file and select Create Deployment for Single Device then select the registered device in your IoT Hub which represents the DE10-Nano device.

    The project ships with an .rbf file (fpga_config_file.rbf) that acts as a basic XOR gate using the physical switches on the DE10-Nano. When SW0 or SW1 are exclusively switched on, an onboard LED will light up. If you are curious how this .rbf was created, you may refer to this tutorial.

    To deploy your own .rbf, edit the DE10Nano_RBF_Loader/module.json to point to a docker repostory that you control, next overwrite the existing fpga_config_file.rbf with your intended .rbf file, then right-click deployment.template.json and select Build and Push IoT Edge Solution. This will update the deployment file under the config folder named deployment.arm32v7.json, right-click that file and select Create Deployment for Single Device then select the registered device in your IoT Hub which represents the DE10-Nano device. Once the IoT Edge deployment completes, your .rbf will load into the FPGA on the target device.

    You can confirm that the FPGA has been configured by running:

    sudo iotedge logs DE10Nano_RBF_Loader

    This should produce output similar to the following:

    ******************************************************
    MSEL Pin Config..... 0xa
    FPGA State.......... Powered Off
    cfgwdth Register.... 0x1
    cdratio Register.... 0x0
    axicfgen Register... 0x0
    Nconfig pull reg.... 0x0
    CONF DONE........... 0x0
    Ctrl.en?............ 0x0
    ******************************************************
    Turning FPGA Off.
    Setting cdratio with 0x3.
    Turning FPGA On.
    Loading rbf file.
    EOF reached.
    ******************************************************
    MSEL Pin Config..... 0xa
    FPGA State.......... User Phase
    cfgwdth Register.... 0x1
    cdratio Register.... 0x3
    axicfgen Register... 0x0
    Nconfig pull reg.... 0x0
    CONF DONE........... 0x0
    Ctrl.en?............ 0x0
    ******************************************************
    
    

    You can also view the results of the deployment by visiting the Azure Portal to confirm that the DE10Nano_RBF_Loader module has been started and is running on the device.

    6. Performance Parameters

    Performance of FPGA workloads loaded through the DE10Nano_RBF_Loader are comparable to workloads loaded directly to the FPGA.  This is due to the choice of mechanism for starting the FPGA workload on the device via direct configuration of the FPGA through Register access as demonstraed below.  Performance is not compromised in our cloud-driven configuration.
     

    #include "main.h"
    
    #define FPGA_MANAGER_ADD (0xff706000) // FPGA MANAGER MAIN REGISTER ADDRESS
    #define STAT_OFFSET      (0x000)
    #define CTRL_OFFSET      (0x004)
    #define GPIO_INTSTATUS   (0x840)
    
    #define FPGA_MANAGER_DATA_ADD (0xffb90000) // FPGA MANAGER DATA REGISTER ADDRESS
    
    int fd; // file descriptor for memory access
    void * virtualbase; // puntero genérico con map de userspace a hw
    char rbf_file [32] = "fpga_config_file.rbf";
    
    int main(int argc, const char * argv[])
    {
      const size_t largobytes = 1;
    
      fd = open("/dev/mem", (O_RDWR|O_SYNC));
    
      virtualbase = mmap(NULL, largobytes,
       (PROT_READ|PROT_WRITE), MAP_SHARED, fd, FPGA_MANAGER_ADD);
      // MAP_SHARED comparte la memoria con otras apicaciones
      // PROT READ y PROT WRITE, para lectura y escritura
    
      if(argc > 1) {
        if(strcmp(argv[1], "report_status") == 0)     report_status();
        else if(strcmp(argv[1], "status") == 0)       report_status();
        else if(strcmp(argv[1], "set_cdratio") == 0)  set_cdratio();
        else if(strcmp(argv[1], "reset_fpga") == 0)   reset_fpga();
        else if(strcmp(argv[1], "reset") == 0)        reset_fpga();
        else if(strcmp(argv[1], "config_fpga") == 0)  config_fpga();
        else if(strcmp(argv[1], "set_axicfgen") == 0) set_axicfgen(atoi(argv[2]));
        else if(strcmp(argv[1], "fpga_off") == 0)     fpga_off();
        else if(strcmp(argv[1], "fpga_on") == 0)      fpga_on();
        else if(strcmp(argv[1], "set_ctrl_en") == 0)  set_ctrl_en(atoi(argv[2]));
        else if(strcmp(argv[1], "set_nconfigpull") == 0)
          set_nconfigpull(atoi(argv[2]));
      }
    
      else {
        // Default action is to program fpga with default rbf file.
        config_routine();
      }
    
      close(fd);
      return 0;
    }
    
    void report_status()
    // Status reg report MSEL (RO) config and FPGA current state (RW).
    // Also report cfgwdth, cdratio registers and other useful registers.
    {
      uint8_t status = alt_read_byte(virtualbase + STAT_OFFSET);
      uint8_t mode_mask = 0b111;
      uint8_t msel_mask = 0b11111 << 3;
    
      uint8_t mode = status & mode_mask;
      uint8_t msel = (status & msel_mask) >> 3;
    
      uint16_t control_reg = alt_read_hword(virtualbase + CTRL_OFFSET);
      uint16_t cfgwdth_mask = (0b1 << 9);
      uint16_t cdratio_mask = (0b11 << 6);
    
      uint8_t cfgwdth  = (control_reg & cfgwdth_mask) >> 9;
      uint8_t cdratio  = (control_reg & cdratio_mask) >> 6;
      uint8_t axicfgen = BIT(control_reg, 8);
      uint8_t ctrl_en  = BIT(control_reg, 0);
      uint8_t nconfigpull = BIT(control_reg, 2);
    
    
      uint16_t gpio_intstatus_reg  = alt_read_hword(virtualbase + GPIO_INTSTATUS);
      uint8_t cd  = BIT(gpio_intstatus_reg, 1); // configuration done register
    
      printf("%s\n",      "******************************************************");
      printf("%s 0x%x\n", "MSEL Pin Config.....", msel);
      printf("%s %s\n",   "FPGA State..........", status_code(mode) );
      printf("%s 0x%x\n", "cfgwdth Register....", cfgwdth);
      printf("%s 0x%x\n", "cdratio Register....", cdratio);
      printf("%s 0x%x\n", "axicfgen Register...", axicfgen);
      printf("%s 0x%x\n", "Nconfig pull reg....", nconfigpull);
      printf("%s 0x%x\n", "CONF DONE...........", cd);
      printf("%s 0x%x\n", "Ctrl.en?............", ctrl_en);
      printf("%s\n",      "******************************************************");
    }
    
    uint8_t fpga_state()
    {
      uint8_t status = alt_read_byte(virtualbase + STAT_OFFSET);
      uint8_t mode_mask = 0b111;
      uint8_t mode = status & mode_mask;
    
      return mode;
    }
    
    void set_cdratio()
    // This should match your MSEL Pin configuration.
    // This is a config for MSEL[4..0] = 01010
    {
      uint16_t control_reg  = alt_read_hword(virtualbase + CTRL_OFFSET);
      uint16_t cdratio_mask = (0b11 << 6);
      uint8_t  cdratio      = 0x3;
    
      control_reg = INSERT_BITS(control_reg, cdratio_mask, cdratio, 6);
      alt_write_hword(virtualbase + CTRL_OFFSET, control_reg);
    
      printf("%s 0x%x.\n", "Setting cdratio with", cdratio);
    }
    
    void reset_fpga()
    {
      uint8_t status = alt_read_byte(virtualbase + STAT_OFFSET);
      uint8_t mode_mask = 0b111;
    
      status = INSERT_BITS(status, mode_mask, 0x1, 0);
    
      alt_write_byte(virtualbase + STAT_OFFSET, status);
    
      printf("%s.\n", "Resetting FPGA");
    }
    
    void config_fpga()
    // Main routine to transfer rbf file to fpga manager data register.
    {
      // Memory map the fpga data register
      void * data_mmap = mmap(NULL, 4,
       (PROT_READ|PROT_WRITE), MAP_SHARED, fd, FPGA_MANAGER_DATA_ADD);
    
      // Open rbf file.
      int rbf = open(rbf_file, (O_RDONLY|O_SYNC));
      if (rbf < 0) {
        // Some error happened...
        printf("\n%s\n\n",
          "Error opening file. Check for an appropiate fpga_config_file.rbf file.");
        exit(-1);
      }
    
      // Set buffer to read rbf files and copy to fpga manager data register.
      uint8_t * data_buffer = (uint8_t*)malloc(sizeof(uint8_t) * 4);
      memset(data_buffer, 0, 4); // set initial data to 0.
    
      // Loop to read rbf and write to fpga data address.
      // We advancse every 4 bytes (32 bits).
    
      bool run_while = true;
      printf("%s\n", "Loading rbf file.");
    
      while(run_while) {
        ssize_t read_result = read(rbf, data_buffer, 4);
        if (read_result < 4) {
          printf("%s\n", "EOF reached.");
          run_while = false;
        }
    
        // Follow format expected by fpga manager.
        uint32_t format_data = *(data_buffer)          << 0;
        format_data = format_data | *(data_buffer + 1) << 8;
        format_data = format_data | *(data_buffer + 2) << 16;
        format_data = format_data | *(data_buffer + 3) << 24;
    
        alt_write_word(data_mmap, format_data);
        memset(data_buffer, 0, 4); // reset data to 0.
      }
    
      close(rbf);
    }
    
    void set_axicfgen(uint8_t value)
    {
      uint16_t control_reg  = alt_read_hword(virtualbase + CTRL_OFFSET);
      uint16_t axicfgen_mask = 1 << 8;
      uint8_t axicfgen = value & 1; // binary values
    
      control_reg = INSERT_BITS(control_reg, axicfgen_mask, axicfgen, 8);
    
      alt_write_hword(virtualbase + CTRL_OFFSET, control_reg);
    }
    
    void set_ctrl_en(uint8_t value)
    {
      uint16_t control_reg  = alt_read_hword(virtualbase + CTRL_OFFSET);
      uint16_t ctrl_en_mask = 1 << 0;
      uint8_t  ctrl_en = value & 1; // binary values
    
      control_reg = INSERT_BITS(control_reg, ctrl_en_mask, ctrl_en, 0);
    
      alt_write_hword(virtualbase + CTRL_OFFSET, control_reg);
    }
    
    void set_nconfigpull(uint8_t value)
    {
      uint16_t control_reg  = alt_read_hword(virtualbase + CTRL_OFFSET);
      uint16_t nconfigpull_mask = 1 << 2;
      uint8_t  nconfigpull = value & 1; // binary values
    
      control_reg = INSERT_BITS(control_reg, nconfigpull_mask, nconfigpull, 2);
    
      alt_write_hword(virtualbase + CTRL_OFFSET, control_reg);
    }
    
    void fpga_off()
    {
      set_nconfigpull(1);
      printf("%s.\n", "Turning FPGA Off");
    }
    
    void fpga_on()
    {
      set_nconfigpull(0);
      printf("%s.\n", "Turning FPGA On");
    }
    
    // ****************************************************************************
    // *                            Auxiliary functions                           *
    // ****************************************************************************
    
    char * status_code(uint8_t code)
    {
      char * description = (char *)malloc(sizeof(char) * 30);
    
      if (code == 0x0)      description = "Powered Off";
      else if (code == 0x1) description = "Reset Phase";
      else if (code == 0x2) description = "Configuration Phase";
      else if (code == 0x3) description = "Initialization Phase";
      else if (code == 0x4) description = "User Phase";
      else                  description = "Undetermined (error ?)";
    
      return description;
    }
    
    // ****************************************************************************
    // *                          Complete config routine                         *
    // ****************************************************************************
    
    void config_routine()
    {
      report_status(); // Check registers... could be accessed using "status" argument.
      set_ctrl_en(1);  // Activate control by HPS.
      fpga_off();      // Reset State for fpga.
      while(1) if(fpga_state() == 0x1) break;
    
      set_cdratio();   // Set corresponding cdratio (check fpga manager docs).
      fpga_on();       // Should be on config state.
      while(1) if(fpga_state() == 0x2) break;
    
      set_axicfgen(1); // Activate AXI configuration data transfers.
      config_fpga();   // Load rbf config file to fpga manager data register.
      while(1) if(fpga_state() == 0x4) break;
    
      set_axicfgen(0); // Turn off AXI configuration data transfers..
      set_ctrl_en(0);  // Disable control by HPS (so JTAG can load new fpga configs).
      report_status(); // Should report "User mode".
    }

    7. Design Architecture

    Design Architecture

    Deployment is specified in Microsoft Azure which spawns a containerized IoT Edge workload that bootstraps an .rbf configuration to the onboard Cyclone V FPGA present on the De10-Nano.  The diagram below demonstrates this workflow.



     



    1 Comments

    Mandy Lei
    Looking forward to your final work!
    🕒 Jun 27, 2019 08:57 PM

    Please login to post a comment.