CREASE Project Blog

CREASE project blog: CREASE on the SPHERE Testbed

by Van Anderson on 7 Aug 2026.

Up to this point, CREASE has been developed on and for the FABRIC testbed; however, a key goal of the project is to establish a general framework for packet provenance monitoring. To explore the generality of CREASE, I prototyped a CREASE instance on the SPHERE testbed as a proof-of-concept. This also helped us gather insight about the porting process to other testbeds.

Topology. The first factor I attempted to emulate from FABRIC on SPHERE was the topology of the monitored link, as it was necessary for the tool and provides a small test case; SPHERE requires declarations of the topology and host structure prior to any actual allocation of resources. While the interface was different in SPHERE, it was easy to successfully replicate the general four-host model we used for testing on FABRIC and update it as needed later in the process.

Fig 1. The test topology as shown in SPHERE’s in-browser GUI. `mon` is the Crinkle monitor, and `ana` the analyzer; h1 and h2 are Ubuntu VMs sending traffic. This topology provides an evaluation that the monitor and analyzer function correctly on SPHERE.

Hugepage Allocation. One assumption that carried over from FABRIC was the image script to configure the packet monitoring hosts. However, I discovered that changes to the grub configuration did not persist across reboots, specifically the allocation of hugepages. Suspecting it was the hypervisor reloading the image each boot, I attempted to create a custom image to subvert the issue, but attempting creation of the new image resulted in an SSH error. This new issue was resolved promptly by reaching out to SPHERE’s technical support. Yet, the configuration still didn't apply on reboots even with the custom image and more investigation is still needed on this effort.

The next best thing is to attempt to reserve hugepages as soon as possible after the VM is spun up. While this doesn't guarantee huge pages will be allocated, I haven't encountered any problems with this approach yet and DPDK has been allowed to operate as expected.

A key takeaway is that the amount of control we have available varies between networks and our reliance on capabilities like persistent grub changes created friction where SPHERE's environment diverged from FABRIC's. Thankfully, the SPHERE personnel were great to work with and quickly helped me resolve my initial issue. Hopefully, the process will be more seamless in the future.

DPDK Device Binding. Another implementation difference was found in DPDK’s ability to bind to network devices. On FABRIC, I was able to share the virtio_pci devices with the kernel. However, using SPHERE, DPDK was unable to bind to the virtio_pci devices. Instead I rebound the interfaces to uio_pci_generic, allowing full usage by DPDK.

It's important to frame that this alternative solution results in downstream effects that must be addressed differently in the following section.

Fig 2. The code used to create the test topology, using SPHERE’s Python API. In a full CREASE port, `m` and `a` would be created automatically through a custom API: `m` via a custom net.connect() call, and `a` via some call like Analyzer, akin to the paradigm developed for CREASE on FABRIC. In this diagram, `m` and `a` correspond to `h1` and `mon` used earlier in the text.

Dropped Packets Problem. On FABRIC the network devices were still managed by the kernel yet shared with DPDK, meaning that the MAC filter on the NIC still applied to each packet that arrived. Since these packets were addressed to the host on the far side of the monitor rather than to the monitor's own NIC, the MAC filter would drop them before DPDK could receive and forward them. Enabling promiscuous mode on the interfaces using ip link .. promisc on would solve this issue, as the MAC filter would no longer apply. However, SPHERE didn’t allow me to share the virtio_pci device with the kernel. Instead, I bound the NIC to a uio_pci_generic driver which didn’t seem to apply the MAC filter that caused issues on FABRIC. The prior decision to rebind the interface device created an environment where enabling promiscuous mode was no longer a consideration or requirement.

Subnet Enforcement/Bump in the Wire Functionality. However, while the MAC filter no longer applied at the monitor, packets still never reached the opposing endpoint. The hypervisor was still populating a packet with the monitor's MAC address since it was still viewed as a router between the two endpoints rather than the hosts appearing transparently as neighbors. To fix this on SPHERE, I manually set the endpoints on the same 10.0.1.0/24 subnet and set the topology's link "layer" to 2. That is, the corresponding edge in that topology consists of an instance of a data link in OSI, and SPHERE's way of expressing this is shown on line 12 of Fig 2 which creates an edge between `x` and `m`, and specifies that that edge is a data link. Similarly, L2Bridge instances were used on FABRIC. After these changes, packets did pass through successfully.

On FABRIC, there’s still another configuration adjustment to ensure successful travel through the monitor even though the packet dropping problem is resolved. If the two endpoints and the monitor share the same worker (physical host), the packets are dropped by the virtualization service since the monitor is not the expected destination (by the MAC address). To work around this, monitors on FABRIC are explicitly placed on a separate worker to force a direct pass through the monitor. The main difference is that the worker separation required on FABRIC wasn’t required on SPHERE, though the functionality is available if it’s ever needed.

SPADE. Our final step of the process, the utilization of SPADE for the collection, aggregation, and querying of packet provenance data, was identical on FABRIC and SPHERE and required no difference in implementation or configuration.

Future Work. While this singular example is reproducible via a series of scripts, the monitored link is not yet ready to be produced multiple times in a single topology on SPHERE. In this prototype, the script makes an assumption about the names of the hosts to be able to access them and run certain scripts at the very least, though it’s certainly possible to expand this capability to insert a monitor between any IPv4 links in a SPHERE topology. Future work can generalize this behavior to port more of CREASE’s automation to SPHERE. Additionally, further work is needed to utilize SPHERE’s custom image feature to allow easier deployment of the monitored link.

Acknowledgements. Thanks to Nik, Alexander, and other members of the networking lab at IIT for the incredible support throughout this project. I also wanted to thank the SPHERE staff, who responded promptly to questions and handled concerns quickly. AI was sparingly used in the creation of this blog post, having only assisted in ideation efforts and minor grammatical corrections. Thank you also goes to Ashish Gehani for providing feedback on this post.