http://bee2.eecs.berkeley.edu/img/BEE2_logo_mini.png

Overview

BORPH Operating System


Created: 2006-06-21


  1. Introduction
  2. Designing for BORPH HowTo
  3. Obtaining BORPH
  4. Paper about BORPH

Introduction


The purpose of this document is to give an overview of the BORPH Operating system used by the BEE2. In addition to a brief overview and philosophy behind BORPH, this document also describes how to build a BORPH compatible system on the BEE2.

Overview


BORPH is an extended Linux kernel that treats FPGA resources as native computational resources on reconfigurable computers such as BEE2. As such, it is more than just a way to configure an FPGA. It also provides integral operating system supports for FPGA designs, such as the ability for an FPGA design to read/write to the standard Linux file system.

A user process in BORPH, can therefore either be a software program running on a processor, or a hardware design running on a FPGA. A hardware design that is running on a FPGA is called a hardware process.

BORPH uses regions of FPGA fabric as computation units to spawn hardware processes. Each reconfigurable region is defined as a hardware region (hwr). Logically, it is the smallest unit of a RC that is managed by BORPH. Physically, it can be implemented as an entire FPGA in a multi-FPGA system, or a partially reconfigurable region within a FPGA. On a BEE2 module, there is only one hwr type defined, the b2fpga, which corresponds to one user FPGA.

Starting a Hardware Process


A user starts a hardware process by executing a BORPH Object File (BOF) file as if it is any other Linux executable, such as ELF or A.OUT. When a BOF file is executed, the kernel examines hardware configurations encapsulated in that file. Based on this information, the kernel chooses and configures one or more suitable hwr’s in the system for this BOF file. On a BEE2, since each hwr corresponds to one user FPGA, it means the kernel will choose a suitable user FPGA to spawn this BOF file. The spawning of BOF files are handled natively by the kernel using standard fork and exec syscall.

A running hardware process behaves almost identically to any other software process in a Linux system. Users can, for example, check the status of hardware processes using command such as ps. A hardware process can be terminated by command like kill or by simply pressing Ctrl-C. The following shows a typical session of executing a BOF file, checking the value of an ioreg, and terminating the process.

  1:bash% ./counter.bof &
  [1] 2458
  2:bash% ps
  PID TTY TIME CMD
  2456 pts/4 00:00:00 bash
  2458 pts/4 00:00:00 counter.bof
  2507 pts/4 00:00:00 ps
  3:bash% cat /proc/2458/hw/ioreg/cntval
  A3B498E0
  4:bash% cat /proc/2458/hw/ioreg/cntval
  B289E906
  5:bash% kill -9 2458
  [1]+ Killed counter.bof
  6:bash%

Communicating with a Hardware Process


There are two primary methods to communicate with a hardware process: the ioreg interface described here, and standard Linux file I/O described in next subsection.

The ioreg Interface

BORPH’s ioreg interface encapsulates conventional memory mapped I/O concept with a virtual file system interface.

Communication between hardware and software usually involves defining a set of special hardware registers and shared memory regions that are mapped into memory space of a processor. Instead of requiring the FPGA designer to redesign a driver interface for each FPGA design, BORPH encapsulate this common design practice by supporting it systematically via its ioreg interface.

The ioreg interface extends a conventional Linux proc file system. A new hw directory is populated for each hardware process with information specific to a hardware design. For a hardware process with pid of <pid>, listing the directory /proc/<pid>/hw/ will shows the following three files:

 bash% cat /proc/123/hw/hardware_region
 Region 0
   Address: 0x1
 bash%

It shows hardware process 123 is currently running at hardware region address 1. On a BEE2, it means it is physically running at user FPGA2.

 bash% cat /proc/123/hw/ioreg_mode
 0
 bash% echo 1 > /proc/123/hw/ioreg_mode
 bash% cat /proc/123/hw/ioreg_mode
 1
 bash%

Reading and writing to a virtual file in this directory will be translated into reading and writing to the corresponding hardware resource in a hardware process. The detail of communication is documented in [Designing for BORPH HowTo]. Here, the usage of these files is first described.

An ioreg virtual file can represent one of the following physical resources:

  1. Single 32-bit word register

  2. On-chip Block RAM (BRAM)

  3. Off-chip memory connected to a user FPGA

Let's take a single word register as an example to explain how an ioreg virtual file work. For example, we have a register, cnt_val that stores the current value of a counter in a hardware process. To read the content of this register, a user may do the following:

 bash% cat /proc/123/hw/ioreg/cnt_val
 00AF0123
 bash%

Note two things: First, the value of the register is read live from the running hardware. The value is read when the moment the kernel receives a read syscall from the command cat. Second, the value returned is represented as a 8-digit hexadecimal number using ASCII representation. This behavior is controlled by the value of ioreg_mode, which by default is "0", meaning it is in ASCII mode. If it is changed to "1", the value 11,469,091 is returned in a single 32-bit word with big endian encoding. This is usually not what you want to use in a shell because this value is not printable. However, if a software program is reading or writing to ioreg files, binary mode communication is usually preferred as it eliminates the need for ASCII conversion.

An ioreg virtual file that corresponds to a memory (either on-chip or off-chip) resource maps all contents of the memory in the file. As result, reading or writing into n bytes offset from the beginning of the file corresponds to accessing the n-th byte of the memory.

The General File System Interface

The second method a hardware process can communicate with the rest of the system is through access to the general file system. Similar to a normal Linux process, when a hardware process is started, the kernel open standard input and standard output for the process by default. A hardware process can therefore read from stdin, which is usually connected to the shell keyboard input. Alternatively, a hardware process can print to stdout, performing simple debugging by printing. A hardware process communicates with the kernel through a predefined packet network described in [Designing for BORPH HowTo].


Designing for BORPH HowTo


Befor you read on ...

The best way to get a taste of how to use BORPH is to follow these step-by-step BorphTutorials. Most of them are designed around our Simulink based MSSGE design flow. Howerver, since BORPH simply provides OS level support for BEE2, it does not depend on any language. More tutorial on how to make a BORPH compatible binary will be available soon. For now, the following sections give you some idea about the low level BORPH kernel interaction with a user design.

Preparing the hardware

BORPH uses the Selectmap programming/communication interface to program the userFPGAs and then communicate with them. This means that an opb_selectmap core (latest version) has to be present in the Control FPGA design and correctly hooked-up to the PowerPC running BORPH Linux. On the userFPGA side, an opb_selectmap_fifo core is required and should be correctly attached to the processor (PowerPC or Microblaze).

Preparing the software

  1. A BORPH Linux has to be compiled to run on the Control FPGA. It is compiled the exact same way a regular linux kernel is (see Bee2LinuxKernel). It is also self contained and once running it is able to start any BORPH file. An ACE file for the control FPGA with precompiled kernel and bitstream can be found here: borph.ace.

  2. On the User FPGA, the following code has to be modified and compiled to run on the processor with EDK. The two main functions to modify are "size_loc" and "map_loc", that are associating the "location" number for a ressource to its size and address. Each hardware ressource in the user FPGA has to be assigned a unique ID, coded on 24 bits and called the "location" of this ressource. This ID is going to be used by BORPH during each access request to identify the file or ressource it is talking to.

/* ************************************ */
/* *                                  * */
/* * BEE2 BORPH communication example * */
/* *                                  * */
/* ************************************ */

/* 2006 Pierre-Yves droz */

/* Main file */

#include "xparameters.h"
#include "xbasic_types.h"
#include "xcache_l.h"
#include <stdio.h>
#include <stdlib.h>

#define MAX_PACKET_SIZE 1000
#define MAX_LOC         100

/*****************************************************************
 * Data macros
 *****************************************************************/

#define SELECTMAP_FIFO_NUM_WORDS 128

static inline Xuint32 GET_STATUS() 
{ 
  return XIo_In32(XPAR_OPB_SELECTMAP_FIFO_0_BASEADDR); 
}
static inline Xuint8  SELECTMAP_STATUS(Xuint32 word)
{ 
  return (Xuint8)((word >> 24) & 0xff); 
}

#define SELECTMAP_RFIFO_CNT(word) (Xuint8)((word >> 16) & 0xff)
#define SELECTMAP_WFIFO_CNT(word) (Xuint8)((word >> 8)  & 0xff)

/* Prototypes */
/* ********************************* */
void add_cmds();
void outbyte(unsigned char c);
int  inbyte();
void intr_selectmap(void);
void send_writeack(unsigned location, int val);
void send_selectmap_byte(unsigned char c);
unsigned char receive_selectmap_byte();
unsigned int receive_selectmap_int3();
void send_selectmap_int3(unsigned int data);
unsigned int receive_selectmap_int4();
void send_selectmap_int4(unsigned int data);

/* Globals */
/* ********************************* */
unsigned char last_byte_written;

/* Main thread */
/* ********************************* */

int main(void)
{
  int i;

/* activate the cache on the PPC */
  XCache_EnableICache(0x00000001);
  XCache_EnableDCache(0x00000001);

/* wait for greet from the BORPH linux */
  i = 0;
  do {
    i <<= 8;
    i |= receive_selectmap_byte();
  } while (i != 0x10FFAA55);

/* display welcome message */
  xil_printf("\n\r");
  xil_printf("***************\n\r");
  xil_printf("* Hello World *\n\r");
  xil_printf("***************\n\r");

/* loop waiting for chars and echo them */
  while(1) {
    outbyte(inbyte());
  }

}

/* The size_loc function associates a specific location number to its actual size in bytes */
/* ********************************* */
int size_loc(int location) {
        /* To be implemented by the user */;
}

/* The map_loc functio associates a specific location number to its actual address */
/* ********************************* */
Xuint32 map_loc(int location) {
        /* To be implemented by the user */;
}

/* The outbyte function implements stdout to the BORPH linux */
/* ********************************* */

void outbyte(unsigned char c)
{

  /* memorizes the last byte sent */
  last_byte_written = c;

  /* send a "write at location 1" packet to the control FPGA */
  /* Write command */
  send_selectmap_byte(3);
  /* location = 1 */
  send_selectmap_int3(1);
  /* offset = 0 */
  send_selectmap_int4(0);
  /* size = 1 */
  send_selectmap_int4(1);
  /* payload is the character */
  send_selectmap_byte(c);
  /* interrupt */
  intr_selectmap();
}

/* The inbyte function implements stdin from BORPH linux, it is also in charge of answering transfer requests from BORPH linux */
/* ********************************* */

int inbyte()
{
  unsigned char cmd;
  unsigned int location,size,offset,block_size,transfer_size;
  int i;
  Xuint32 data;
  core current_core;

  /* send a "read at location 0" packet to the control FPGA */
  /* Read command */
  send_selectmap_byte(1);
  /* location = 0 */
  send_selectmap_int3(0);
  /* offset = 0 */
  send_selectmap_int4(0);
  /* size = 1 */
  send_selectmap_int4(1);
  /* wait for packets from the control FPGA */
  /* HS: but we need to interrupt control FPGA to get this request */
  intr_selectmap();
  while(1) {
    cmd = receive_selectmap_byte();
    switch(cmd) {
      case 1 : /* read */
        /* get the location */
        location = receive_selectmap_int3();
        /* get the offset */
        offset = receive_selectmap_int4();
        /* get the size */
        size = receive_selectmap_int4();
        /* if location is 0, then output the location table */
        if(location == 0) {
            /* -- future implementation of location table output here -- */
        } else {
          /* check if location is greater than max location */
          if(location >= MAX_LOC) {
              /* -- future implementation of error handling here -- */
              /* -- error wrong location -- */
          } else {
            block_size = size_loc(location);
            if(offset+size>block_size) {
              size = block_size - offset;
            }
            if(offset>=block_size) {
              /* -- future implementation of error handling here -- */
              /* -- wrong seek -- */
            }
            offset += map_loc(location);
            data = XIo_In32(offset & 0xFFFFFFFC);
            while(size != 0) {
              if(size > MAX_PACKET_SIZE-10)
                transfer_size = MAX_PACKET_SIZE-10;
              else
                transfer_size = size;
              size -= transfer_size;

              /* Send read ack command */
              send_selectmap_byte(2);
              /* location */
              send_selectmap_int3(location);
              /* size */
              send_selectmap_int4(transfer_size);
              /* payload */
              for(;transfer_size != 0;transfer_size--) {
                send_selectmap_byte(((unsigned char *) &data)[offset % 4]);
                offset++;
                if(offset % 4 == 0)
                data = XIo_In32(offset);
              }
              intr_selectmap();

            }
          } 
        }
        break;
      case 2 : /* read ack */
        /* get the location. It should be 0 only as we don't read from anything else*/
        location = receive_selectmap_int3();
        if(location != 0) xil_printf("Error: Wrong location\n\r");
        /* get the size in bytes. If it is one, it means that one of our read succedded, and we return the payload */
        size = receive_selectmap_int4();
        if(size == 1) return receive_selectmap_byte();
        /* if we received an error code, we send a new request */
        /* HS should wait a bit before stalling cntrl fpga */
        usleep(100000);
        /* send a "read at location 0" packet to the control FPGA */
        /* Read command */
        send_selectmap_byte(1);
        /* location = 0 */
        send_selectmap_int3(0);
        /* offset = 0 */
        send_selectmap_int4(0);
        /* size = 1 */
        send_selectmap_int4(1);
        /* interrupt */
        intr_selectmap();
        break;
      case 3 : /* write */
        /* get the location */
        location = receive_selectmap_int3();
        /* get the offset */
        offset = receive_selectmap_int4();
        /* get the size */
        size = receive_selectmap_int4();
        /* if location is 0, then this is an error */
        if(location == 0) {
          send_writeack(location, -2);
          /* -- future implementation of error handling here -- */
          /* -- error unwritable -- */
        } else {
          /* check if location is greater than max loc */
          if(location >= MAX_LOC) {
            /* -- future implementation of error handling here -- */
            /* -- error wrong location -- */
            send_writeack(location, -3);
          } else {
            block_size = size_loc(location);
            if(offset+size>block_size) {
              size = block_size - offset;
            }
            if(offset>=block_size) {
                 /* -- future implementation of error handling here -- */
                 /* -- wrong seek -- */
            }
            offset += map_loc(location);
            /* prefetch data for the first read modify write operation */
            data = XIo_In32(offset & 0xFFFFFFFC);
            /* payload processing */
            for(transfer_size = 0;transfer_size < size;transfer_size++) {
              ((unsigned char *) &data)[offset % 4] = receive_selectmap_byte();
              offset++;
              if(offset % 4 == 0) {
                XIo_Out32(offset-4,data);
                /* save some cycles */
                if (size - transfer_size <= 4) {
                  data = XIo_In32(offset & 0xFFFFFFFC);
                }
              }
            }
            if (offset % 4 != 0) {
              XIo_Out32(offset & 0xFFFFFFFC, data);
            }
            /* Send write ack command */
            send_selectmap_byte(4);
            /* location */
            send_selectmap_int3(location);
            /* size */
            send_selectmap_int4(size);
            /* interrupt */
            intr_selectmap();
          }
        }
      case 4 : /* write ack */
        /* get the location */
        location = receive_selectmap_int3();
        /* get the size */
        size = receive_selectmap_int4();
        /* check the size */
        if(size==0) {
          /* last write failed, send it again */
          /* send a "write at location 1" packet to the control FPGA */
          /* Write command */
          send_selectmap_byte(3);
          /* location = 1 */
          send_selectmap_int3(1);
          /* offset = 0 */
          send_selectmap_int4(0);
          /* size = 1 */
          send_selectmap_int4(1);
          /* payload is the character */
          send_selectmap_byte(last_byte_written);
          /* interrupt */
          intr_selectmap();

        }
        break;
      case 17: /* bye */
          /* remove magic from the fifo */
          receive_selectmap_byte();
          receive_selectmap_byte();
          receive_selectmap_byte();
          data = 0x30000000;
          asm("mtdbcr0 %0" :: "r"(data));
        break;
      default : /* unknown packet type */
          /* -- future implementation of error handling here -- */
          /* -- error unknown packet -- */
        break;
    }
  }
}

void send_writeack(unsigned location, int val)
{    
    /* Send write ack command */
    send_selectmap_byte(4);
    /* location */
    send_selectmap_int3(location);
    /* size */
    send_selectmap_int4(val);
    /* interrupt */
    intr_selectmap();
}

void intr_selectmap(void)
{
    XIo_Out32(XPAR_OPB_SELECTMAP_FIFO_0_BASEADDR, 1);
}

void send_selectmap_byte(unsigned char c)
{
  int retry = 0;

  /* block on fifo full */
  while (!SELECTMAP_WFIFO_CNT(GET_STATUS())) {
    if (!(retry & 0xFFFF)) {
      intr_selectmap();
    }
    retry += 1;
  }
  /* send the byte */
  XIo_Out8(XPAR_OPB_SELECTMAP_FIFO_0_BASEADDR+4, c);
}

unsigned char receive_selectmap_byte()
{
  /* block on fifo empty */
  while (!SELECTMAP_RFIFO_CNT(GET_STATUS())) {

  }
  /* read the byte */
  return XIo_In8(XPAR_OPB_SELECTMAP_FIFO_0_BASEADDR+4);
}

unsigned int receive_selectmap_int3()
{
  unsigned int data = 0;
  
  data = receive_selectmap_byte();
  data <<= 8;
  data |= receive_selectmap_byte();
  data <<= 8;
  data |= receive_selectmap_byte();

  return data;
}

void send_selectmap_int3(unsigned int data)
{
  send_selectmap_byte(data >> 16);
  send_selectmap_byte(data >>  8);
  send_selectmap_byte(data >>  0);
}

unsigned int receive_selectmap_int4()
{
  unsigned int data = 0;

  data = receive_selectmap_byte();
  data <<= 8;
  data |= receive_selectmap_byte();
  data <<= 8;
  data |= receive_selectmap_byte();
  data <<= 8;
  data |= receive_selectmap_byte();

  return data;
}

void send_selectmap_int4(unsigned int data)
{
  send_selectmap_byte(data >> 24);
  send_selectmap_byte(data >> 16);
  send_selectmap_byte(data >>  8);
  send_selectmap_byte(data >>  0);
}

Compiling a design

Once the UserFPGA design has been compiled with EDK, the bitfile has to be transformed into a BORPH executable file using the command "mkbof". The syntax of makebof is as follows:

mkbof -o <output_file> -s <symbol_file> [-p <hard_location>] <bitfile>

where:

  1. <output_file> is the name of the BORPH executable to be produced

  2. <symbol_file> is the file indicating the matching between a ressource and its name, size, and permission. Inside the user has to write a list of TAB separated values (one line per ressource) indicating all the needed data.

    •   <Ressource name>\TAB<Permission>\TAB<Location>\TAB<Size>
        

      <Ressource_name> is the name of the file that will be associated with the ressource. <Permission> is 1 for Read only, 2 for Write Only, 3 for Read Write. <Location> is the unique ID of the ressource (it has to match the one defined in the UserFPGA software). <Size> is the size in bytes of the ressource.

  3. <hard_location> is optional and can be 0, 1, 2 or 3. If defined it forces the BORPH executable to be started on a specific FPGA instead. If not defined, the design is floating and BORPH will automatically find a free FPGA to start the design.

  4. <bitfile> is the .bit file compiled by EDK


Obtaining BORPH


The BORPH source tree will be available through the BEE2 CVS soon. In the mean time, you can download the latest snapshot directly from here:


Paper about BORPH


Bee2OperatingSystem (last edited 2006-12-30 16:05:49 by localhost)