nvme Fabrics connections over Ipv6 not working with nvme-cli 2.11 and libnvme 1.11.1

[sukhi61166]

We are trying to connect using the IPv6 address over RoCEv2 protocol and it is not working with the latest nvme-cli/libnvme versions. We tried the same setup on the same server with nvme-cli 1.16 and connections are made. We must stay at the latest version in order to use other features. This is a blocking issue for us. Please correct me in case the arguments I am using are not correct.

Discover does work fine but the connect or connect-all does not even try to reach the target over the wire. We see no activity on the network trace.
This is all we see after the connect command is issued "could not add new controller: failed to write to nvme-fabrics device"
In the dmesg it shows "nvme nvme0: rdma connection establishment failed (-104)"
Failing nvme-versions are:
nvme version 2.11 (git 2.11+)
libnvme version 1.11.1 (git 1.11.1)
nvme version 2.10.2 (git 2.10.2)
libnvme version 1.10 (git 1.10)
Command used to connect/discover with 2.10/2.11
nvme discover -t rdma -a fe80::020c:caff:fe12:24c3%enp193s0f0np0 -s 4420
nvme connect-all -t rdma -a fe80::020c:caff:fe12:24c3%enp193s0f0np0 -s 4420 -i 1

Command used with Passing nvme version is 1.16
nvme connect-all -t rdma -a fe80::020c:caff:fe12:24c3%enp193s0f0np0 -s 4420 -i 1 -w fe80::966d:aeff:fe38:772e%enp193s0f0np0

[sukhi61166]

Just another data point that connection with TCP protocol works to the same target from same host with nvme-cli 2.11 using the following commands

nvme connect -t tcp -s 4420 -a fe80:0000:0000:0000:020c:caff:fe12:66ff -f enp193s0f0np0  -i 1  -n nqn.2015-09.com.wdc:nvme.1
nvme connect-all -t tcp -s 4420 -a fe80:0000:0000:0000:020c:caff:fe12:66ff -f enp193s0f0np0  -i 1

[igaw]

Could provide the output the nvme connect-all with the additional -vv options? This shows the connect arguments for the kernel. As nvme connect works, it looks like the problem is that the discovery code doesn't fill out the correct command string.

Also please provide the output from the nvme discovery command.

[martin-belanger]

@sukhi61166 - If I understand correctly, it works when you specify the interface separately with the -f iface option, but it does not work when the interface is specified with the -a option using the addr%iface syntax. Is this correct?

If that's the case that would indicate that the addr%iface does not get split properly.

[sukhi61166]

@martin-belanger
-f iface option ONLY works (and is only valid) with TCP not RoCEv2
connect works by adding the addr%iface at the end of target IPV6 address.
But connect-all does not work with the same arguments.

@igaw
Please see attached output of discovery, connect-all and connect commands and dmesg.
dmesg1.txt
cmds_output.txt

[ksingh-ospo]

These work as we've figured out through experimentation. It would be great if working examples could be added to the nvme-cli documentation. We didn't find any examples in the included docs. Also, it would help if IPv6 addresses worked consistently between TCP & RDMA.

TCP: (included -f and removed -w)

sudo ~/tmp/nvme-cli-v2.11/.build/nvme connect -t tcp -s 4420 -a fe80:0000:0000:0000:020c:caff:fe12:2a0c -f enp129s0f0np0 -i 2 -n nqn.2015-09.com.wdc:nvme.1

RDMA: (-f not allowed, so removed it. included host-iface with target addr; squished 0's in IPv6 addr unlike in TCP)

sudo ~/tmp/nvme-cli-v2.11/.build/nvme connect -t rdma -s 4420 -a fe80::020c:caff:fe12:2a0c%enp129s0f0np0 -i 2 -n nqn.2015-09.com.wdc:nvme.1

[martin-belanger]

I should have paid more attention. I did not realize that you had both rdma and tcp examples. In that case, you are correct that -f is only for TCP.

I'm no RDMA expert, but I don't think that the addr%iface syntax is needed (or supported) for RDMA. It's been a few years since I've looked at the kernel code, but IIRC the kernel uses the infiniband driver for RDMA and that driver requires addresses (no %iface) to determine how to make a connection. More precisely, one must provide both the -a and -w options and only supply addresses.

BTW, nvme-cli 1.16 is pretty old. Looking at the releases I don't even see 1.16. Are you running on an older system? Which distro? And what is the kernel version? Thanks.

[martin-belanger]

Clarification for TCP transport:

Scoped addresses (i.e. addr%iface) as defined in RFC4007 is only specified for IPv6. It is not supported for IPv4. That is why the option -f was added for TCP transport. This allows to specify an interface for IPv4. The -f option can also be used fort IPv6 or a scoped IPv6 address can be used instead of the -f option.

The -w option is used to specify a different source address of a TCP socket. It is not used to specify the interface. Interfaces can have more than one IP addresses assigned to them. There is primary IP address, which is used by default when making a connection. There may also be secondary addresses assigned to an interface, and if a user wants to use one of those secondary addresses as the source address of the connection, then they must tell the kernel with the -w option.

[sukhi61166]

I should have paid more attention. I did not realize that you had both rdma and tcp examples. In that case, you are correct that -f is only for TCP.

I'm no RDMA expert, but I don't think that the addr%iface syntax is needed (or supported) for RDMA. It's been a few years since I've looked at the kernel code, but IIRC the kernel uses the infiniband driver for RDMA and that driver requires addresses (no %iface) to determine how to make a connection. More precisely, one must provide both the -a and -w options and only supply addresses.

BTW, nvme-cli 1.16 is pretty old. Looking at the releases I don't even see 1.16. Are you running on an older system? Which distro? And what is the kernel version? Thanks.
@martin-belanger
Its not an old server. 1.16 is the last stable nvme-cli version we used with our previous product. So we use that as reference to cross check the changes.
We are using Ubuntu 24.04 kernel is 6.8.12-u6.8-50.51

[igaw]

These work as we've figured out through experimentation. It would be great if working examples could be added to the nvme-cli documentation. We didn't find any examples in the included docs. Also, it would help if IPv6 addresses worked consistently between TCP & RDMA.

Please provide some documentation in this case. I'm glad to add it.

[sukhi61166]

Are we still looking into the connect-all issue?

Yes, we didnt find any examples or documentation on IPV6 connect command arguments. It will be nice to add it to the documentation

[ksingh-ospo]

These work as we've figured out through experimentation. It would be great if working examples could be added to the nvme-cli documentation. We didn't find any examples in the included docs. Also, it would help if IPv6 addresses worked consistently between TCP & RDMA.

Please provide some documentation in this case. I'm glad to add it.

@igaw, that's what I listed in my previous reply, i.e. the examples for TCP & RDMA that worked. If you could please include those in the documentation that would help.

At the same time, we should update the tool's help text for some of the options mentioned above like -f, -w and scoped addresses, clearly mentioning allowed/not-allowed for each protocol. Based on my observations with nvme-cli v2.11 (+libnvme 1.11) here's my understanding so far. If the tool is modified to fix an issue the help text shown below should be updated accordingly.

[ --traddr=, -a ] --- transport address. Use scoped address for RDMA over IPv6, e.g. fe80::020c:caff:fe12:2a0c%enp129s0f0np0
[ --host-iface=, -f ] --- host interface (for tcp transport only)
[ --host-traddr=, -w ] --- host traddr (e.g. FC WWN's). Do not use for RDMA over IPv6

RDMA over IPv6:
[works]
sudo ~/tmp/nvme-cli-v2.11/.build/nvme connect -t rdma -s 4420 -a fe80::020c:caff:fe12:2a0c%enp129s0f0np0 -i 2 -n nqn.2015-09.com.wdc:nvme.1

[does not work]

1. Without scoped addresses and with -f for host-iface // FAILS
   sudo ~/tmp/nvme-cli-v2.11/.build/nvme connect -t rdma -s 4420 -a fe80::020c:caff:fe12:2a0c -f enp129s0f0np0 -i 2 -n nqn.2015-09.com.wdc:nvme.1
   // kernel error: invalid parameter 'host_iface=%s'

2. Adding -w to 1 above fails in the same way as 1

3. With scoped address & -w // FAILS
   sudo ~/tmp/nvme-cli-v2.11/.build/nvme connect -t rdma -s 4420 -a fe80::020c:caff:fe12:2a0c%enp129s0f0np0 -i 2 -n nqn.2015-09.com.wdc:nvme.1 -w fe80::eaeb:d3ff:feb4:75a2
   // kernel msg w/ my instrumentation of src & ctrl addresses: nvme nvme0: rdma_resolve_addr failed (-22): src:fe80::eaeb:d3ff:feb4:75a2, ctrl:fe80::20c:caff:fe12:2a0c
   // nvme-cli error: could not add new controller: invalid arguments/configuration

[igaw]

I think this could go into the wiki https://github.com/linux-nvme/nvme-cli/wiki as I don't think this should be in the man pages. Please add it to the wiki, I just don't have the bandwidth to do everything.

[ksingh-ospo]

@igaw,
I've started a new Wiki page for "connect" and added it to the home you linked above.
https://github.com/linux-nvme/nvme-cli/wiki/How-to-connect-to-a-target

Will update it with TCP example when I get a chance.

[igaw]

I've also added some bits to the wiki, e.g. Martin's explanation on the scoped interface. Anyway all are invited to improve the wiki. Thanks a lot!

[sukhi61166]

@igaw @martin-belanger
#2648 (comment)

The connect-all not working even with the correct parameters is still an issue for us. Did anyone get a chance to look at the output I provided with verbosity enabled?

[igaw]

The command lines contain some weird stuff, not easy to parse:

[2025-01-07 11:39:12.373] [mlx5/mlx5/i40e/i40e/i40e/i40e]root@perf-256-2:/home/kazan/suki# nvme connect-all -t rdma -a fe80::020c:caff:fe12:66ff%enp193s0f0np0 -s 4420 -i 1 -vv discover -t rdma -a fe80::020c:caff:fe12:66ff%enp193s0f0np0 -s 4420--version sudo nvme disconnect-allnvme connect -t rdma -a fe80::020c:caff:fe12:66ff%enp193s0f0np0 -s 4420 -i 1 -n nqn.2015-09.com.wdc:nvme.1 -vv   
[2025-01-07 11:39:25.441] warning: using auto generated hostid and hostnqn

Note that libnvme will auto generate the hostnqn as there is no /etc/nvme/{hostnqn,hostid}, this should obviously match with your target configuration.

Anyway, the resulting connect command to the kernel looks okay for the traddr, it matches what seems to be provided on the command line:

connect ctrl, 'nqn=nqn.2015-09.com.wdc:nvme.1,transport=rdma,traddr=fe80::020c:caff:fe12:66ff%enp193s0f0np0,trsvcid=4420,hostnqn=nqn.2014-08.org.nvmexpress:uuid:a9ff9600-3ac5-11eb-8000-3cecef1fc322,hostid=a9ff9600-3ac5-11eb-8000-3cecef1fc322,nr_io_queues=1,ctrl_loss_tmo=600'

The rdma transport is using the same parser as the tcp transport for splitting traddr into ctrl->addr . This code looks identically. I don't see the problem.