Skip to content
Snippets Groups Projects
user avatar
add hardware submodule
Mikhail Khalilov authored
41e9ded1
Select Git revision
  • main default
  • pspin-master
  • dma_congestion
3 results
Compare History Find file
Name Last commit Last update
common_cells @ dcf84ce1
pspin @ 230583d8
.gitmodules
Dockerfile
README.md
docker-compose.yml
README.md

OSMOSIS: Enabling Multi-Tenancy in Datacenter SmartNICs

The paper will be presented at the USENIX ATC 24.

Repo structure

  • OSMOSIS/PsPIN Verilator simulation environment in Docker:

    • ./Dockerfile and ./docker-compose.yml

  • OSMOSIS/PsPIN source code package:

    • ./pspin/

  • OSMOSIS source code:

    • ./pspin/hw/verilator_model/src/AXIMaster.hpp - C++ model of hardware DMA AXI fragmentation
    • ./pspin/hw/verilator_model/src/FMQEngine.hpp - C++ model of WLBVT FMQ scheduler
    • ./pspin/examples/osmosis - OSMOSIS host/kernel C API/runtine
    • ./pspin/examples/mt_apps/handlers - evaluated SmartNIC kernels
    • ./pspin/examples/mt_apps/driver - deployment scenarios (e.g., tenant mixtures)

  • Experiment infrastructure:

    • ./pspin/examples/mt_apps/scripts/*.sh - scripts to batch experiments
    • ./pspin/examples/mt_apps/scripts/*.py - scripts to parse/visualize experimental logs
    • ./pspin/examples/mt_apps/traces - various input traffic packet traces with different number of tenants and packet sizes
    • ./pspin/examples/mt_apps/scripts/tracegen.py - packet trace generator using statistical parameters

  • OSMOSIS hardware blocks for usage with ASIC/FPGA:

    • These hardware blocks are used for OSMOSIS ASIC area estimations in the paper and can be syntesized using Synopsis Design Compiler NXT.
    • ./common_cells/src/bvt_arb_tree.sv - WLBVT hardware scheduler prototype written in SystemVerilog.
    • iDMA - DMA engine written in SystemVerilog with support of high-performance hardware protocol for AXI stream fragmentation.

A. Verilator environment setup

1. Set up the Docker container

This step takes approximately 15-20 minutes to build from scratch.

  1. host$ git clone git@spclgitlab.ethz.ch:mkhalilov/pspin-osmosis.git
  2. host$ cd ./pspin-osmosis/ && git submodule update --init --recursive
  3. host$ docker-compose -p osmosis-ae up -d

As a result container osmosis-ae will be built and run in the background.

2. Build PsPIN/OSMOSIS simulation core inside of container (30 minutes)

  1. host $ docker ps # get the containerID
  2. host $ docker exec -it <containerID> bash
  3. osmosis-ae # cd /opt/pspin/
  4. osmosis-ae # source ./sourceme.sh
  5. osmosis-ae # cd ./hw/verilator_model/
  6. osmosis-ae # VERILATOR_COMPILER_WORKERS=$(nproc) make release - this step needs around 30 minutes and 128-256 GB of RAM.

B. Experiments

Setup simulation environment

  1. In case you stopped the container, re-run docker-compose as shown in Step A1
  2. (Re)-attach to the running osmosis-ae container by re-running commands 1 - 4 from the Step A2
  3. osmosis-ae # cd /opt/pspin/examples/mt_apps/ - change to the root OSMOSIS experiments working directory
  4. osmosis-ae # make SPIN_KERNEL_NAME=kernels deploy - compile SmartNIC application kernels

Experiment pipeline

The experiment pipeline contains the following steps:

  1. osmosis-ae # make SPIN_APP_NAME=<experiment_name> osmosis - compile experiment scenario from ./driver directory, so the experiment binary sim_<experiment_name> will be generated in the current directory.
  2. osmosis-ae # nohup bash ./scripts/run_<experiment_name>.sh $(pwd)/traces/ $(pwd)/logs/ & - run batched simulation to perform all measurements (we control sim_<experiment_name> binary config through environment variables) and input traffic trace. Raw simulation logs will be stored in ./logs/ directory. Pre-generated traces are stored in ./traces/ directory.
  3. osmosis-ae # python3 ./scripts/postprocess_<experiment_name>.py $(pwd)/logs - post-process experiment logs to data *.csv in ./logs/ directory.
  4. osmosis-ae # python3 ./scripts/plot_<experiment_name>.py - run plotting script on *.csv data to visualize it in *.pdf format in the ./figures/ directory.
  • The figure is located at the $host <AE-repo-root-dir>/pspin/examples/mt_apps/figures/ path in the host $ file-system, and can be opened outside of osmosis-ae $ container, e.g., to open it in the PDF reader.

Experiment 1: Fairness of HPU utilization

  • The experiment takes approximately 2 minutes.
  • We recommend to use this experiment to go through basic functionality of the experiment pipeline (e.g., AE kick-the-tires stage).
  • The experiment demonstrates that the OSMOSIS WLBVT scheduler achieves fair share of SmartNIC compute engine between two tenants when compared to the Round Robin scheduler (e.g., see Figure 9 and Figure 4 in the paper).
  1. osmosis-ae # nohup bash ./scripts/run_hpu_contention.sh $(pwd)/traces/ $(pwd)/logs/ &
  2. osmosis-ae # python3 ./scripts/postprocess_hpu_contention.py $(pwd)/logs
  3. osmosis-ae # python3 ./scripts/plot_hpu_contention.py- produces figures/hpu_occupation.pdf

Experiment 2: DMA engine contention

  • The experiment takes approximately 2-3 hours.
  • OSMOSIS relies on the DMA request fragmentation to avoid small request HoL-blocking. The experiment evaluates the DMA congestor throughput and the DMA victim kernel completion time as a function of the DMA congestor request size and various DMA fragment sizes (see Figure 10 and Figure 5 in the paper).
  1. osmosis-ae # make SPIN_APP_NAME=io_contention osmosis
  2. osmosis-ae # nohup bash ./scripts/run_io_contention.sh $(pwd)/traces/ $(pwd)/logs/ &
  3. osmosis-ae # python3 ./scripts/postprocess_io_contention.py $(pwd)/logs
  4. osmosis-ae # python3 ./scripts/plot_io_contention.py

Experiment 3: Application throuhgput overheads

  • The experiment takes approximately 4-6 hours.
  • The experiment shows relative packet throughput of common datacenter workloads run in a standalone mode as a function of packet size with their raw performance in million packets per second (Mpps) at the top of the bars (see Figure 11 in the paper). The OSMOSIS schedulers doesn't introduce signinficant performance overheads in the hardware PsPIN datapath.
  1. osmosis-ae # make SPIN_APP_NAME=raw_tput osmosis
  2. osmosis-ae # nohup bash ./scripts/run_raw_tput.sh $(pwd)/traces/ $(pwd)/logs/ &
  3. osmosis-ae # python3 ./scripts/postprocess_raw_tput.py $(pwd)/logs
  4. osmosis-ae # python3 ./scripts/plot_raw_tput.py

Experiment 4: Application mixes

  • The experiment takes approximately 4 hours.
  • The experiment studies fairness of the compute and IO resource utilization under the mixture of different IO/compute-bound workloads (see Figure 12a/b in the paper). OSMOSIS demonstrates significant reductions in the FCT and fairly allocates SmartNIC resources.
  1. osmosis-ae # make SPIN_APP_NAME=compute_mix osmosis
  2. osmosis-ae # make SPIN_APP_NAME=io_mix osmosis
  3. osmosis-ae # nohup bash ./scripts/run_mixes.sh $(pwd)/traces/ $(pwd)/logs/ &
  4. osmosis-ae # python3 ./scripts/postprocess_mixes.py $(pwd)/logs
  5. osmosis-ae # python3 ./scripts/plot_mixes_compute.py
  6. osmosis-ae # python3 ./scripts/plot_mixes_io.py