A network block device, or NBD, is a cool Linux tool that you’ve probably never heard of. As its name implies, it lets you make a block device (or, if desired, a device image) available over the network.
Below, I’ll discuss why NBD is so useful, then explain how to set it up on a Linux system.
Why NBD?
In our age of ubiquitous scaled-out, distributed, network-connected storage devices, you might be wondering why anyone would want to mount a block device over the network. Isn’t a network file system a better solution?
Well, yes, if all you need to do is read and write files from a file system. But sometimes you may want to complete operations on a storage device at a lower level than a network file system would support.
For example, maybe you want to format the device. Or you want to modify or copy entire partitions. These tasks would be impossible to accomplish with a network file system, because they would require you to have the file system unmounted in order to perform them — and if you unmount your network file system, it’s no longer connected.
But if your remote storage device is mounted as a block device, you can do anything to it that you’d be able to do to a local block device. In other words, with NBD, you can take a device like /dev/sda on one machine and make it available to another machine as if it were a local device there connected via a SCSI or SATA cable, even though in actuality it is connected over the network.
Getting started with NBD
NBD works according to a client/server architecture. You use the server to make a volume available as a network block device from a host, then run the client to connect to it from another host.
Installation
NBD is not installed on most systems by default, so you’ll have to add it yourself. On Ubuntu, you can grab the NBD server package with:
apt-get install nbd-server
After installation, insert the NBD module with:
modprobe nbd
Export a device or file
With the module inserted, the next step is to export a device with the NBD server using the nbd-server utility. The basic syntax is simple and follows the form:
nbd-server port-number file-name
So, if we want to export the local device /dev/sdb on port 9999, we’d run:
nbd-server 9999 /dev/sdb
You can also export image files as block devices. This can be useful if you’re working with virtual disk images, for example. The syntax here is basically the same. To export the image example.img on port 9999, we use this command:
nbd-server 9999 example.img
There are other arguments that you can pass to nbd-server to control things like writability. Check out man nbd-server for details.
Note, too, that NBD has a configuration file, which by default lives at /etc/nbd-server/config (on my Ubuntu system, at least). You can specify configuration variables there, if desired, instead of passing them on the command line.
Connect to an export
On the client machine that we want to use to connect to the NBD export we just created, we first need to install the NBD client package with:
apt-get install nbd-client
We also have to insert the NBD module on the client with:
modprobe nbd-client
The basic nbd-client syntax looks like this:
nbd-client server-host port-number nbd-device
(nbd-device is the device name that we want to assign to the connection on the client machine.)
So, if we want to connect to an NBD export hosted on the server server.com on port 9999, and map it to /dev/nbd0, we use:
nbd-client server.com 9999 /dev/nbd0
Using NBD exports
Now, we can start doing cool things with the NBD export from the client machine by using /dev/nbd0 as the target. You could resize partitions. You could create a filesystem. The list goes on.
As long as the export that the client is using at /dev/nbd0 is mapped to a device like /dev/sda on the server, operations from the client on /dev/nbd0 will take effect on the server just as they would if you were running them locally on the server with /dev/sda as the target.
Warnings
As I noted, I think NBD is cool because it lets you perform low-level device operations over the network. However, lest you start deploying NBD in a production environment without understanding the drawbacks, I should note that it’s not perfect.
I have found NBD to be a bit buggy. It occasionally segfaults, for reasons that are not always clear. If you want to use it for production, you should make sure that you test everything very thoroughly first.
I have also found that NBD does not seem to do well if you are working with exports over a flaky network connection. It does not appear to have been designed to cope with intermittent losses of connectivity or fluctuating bandwidth.
Bear in mind, too, that you should be careful when exposing a block device from one host to another over the network because the NBD client won’t have any safeguards in place to protect you from, say, accidentally formatting /dev/sda on the host and wiping out the file system there. That makes working with NBD exports different from working with local block devices, since on the local system you won’t be able to reformat a block device that is already mounted. In contrast, NBD isn’t smart enough to know whether an operation performed by the client could have irreparably bad consequences for the host.
Last but not least, anyone who uses NBD should understand the security implications of what we did above. Making a block device (or file image) from one host available to another in this way opens up all sorts of avenues for potential security abuse from the client machine, the host machine and over the network. If I were going to use NBD in production, I would absolutely not export devices over a public network, and I would make sure that user accounts on both the server and the client were locked down in order to help prevent abuse.
Conclusion
All of the above said, NBD is a cool tool. It lets you do things that would otherwise not be possible. It doesn’t get much press these days (which is not surprising because NBD dates all the way back to the early 2000s, and hasn’t ever been important commercially), but it may be just the tool you need to solve some of the strange challenges that can arise in the life of a sysadmin.
