[iPadOS] USB 4 port unusable after driver uninstallation

Has this been observed before with PCIDriverKit or DriverKit in general? Is this some kind of a known limitation in iPadOS 17 in particular, or does this have something to do with the iPad model too? Is there any workaround that can be followed to prevent this from happening?

I'd need more data to be sure about what's going on, but there were a few bugs in the driver management process around that time which would match what you’re describing what you're describing. A few questions:

• Have you verified through the I/O Registry that your DEXT is unloading correctly at uninstall?

• What does the I/O Registry show after you unplug the device?

• What does the I/O Registry show when you plug your device back in (and it remains non-functional)?

• What happens if you plug an entirely different accessory, particularly a very simple configuration like a USB3 adaptor and keyboard? Do they work, and does your accessory return to normal operation after that?

• Have you looked at the system console at all three transitions? What is it logging?

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

@DTS Engineer

Hi Kevin,

Have you verified through the I/O Registry that your DEXT is unloading correctly at uninstall?

What does the I/O Registry show after you unplug the device?

Yes, I have verified this. When a device is connected and the driver is active, I have my driver instance and IOUserServer instance in the IORegistry. Whenever the device is removed or on uninstallation of the app, I can see that these entries are not present.

What does the I/O Registry show when you plug your device back in (and it remains non-functional)?

When I plug my device back in (non-functional state), there is no driver instance in the IORegistry.

What happens if you plug an entirely different accessory, particularly a very simple configuration like a USB3 adaptor and keyboard? Do they work, and does your accessory return to normal operation after that?

I have tried connecting some other USB3 accessory and it seems to function just fine. When I try to connect my USB4 accessory back again, it is still non-functional.

Have you looked at the system console at all three transitions? What is it logging?

Yes, I looked at the system console during uninstallation. I have attached the logs from both iPadOS 17 (non-functional) and iPadOS 18 (functional).

To add, I have also tried having a very minimal implementation in my Start/Stop methods like the one in here. I can see from the logs that IOPCIDevice::ClientCrashed is getting called in both the OS versions. iPadOS 18 recovers and is able to call Stop successfully whereas 17 fails to do so. I open an IOUserClient to the driver during app launch when an accessory gets connected and close it when the accessory is detached. So, I have also tried the below just to verify in iPadOS 17 if IOUserClient being open might be the case:

1. Install app. Do not launch.
2. Go to settings and enable driver.
3. Attach accessory. Driver matched and loaded in IORegistry.
4. Uninstall the app.

I can see the same crash from the console even in this case.

Yes, I looked at the system console during uninstallation. I have attached the logs from both iPadOS 17 (non-functional) and iPadOS 18 (functional).

Is it possible you flipped the two logs you posted? The one you've labeled "18" shows DriverKit attempting to load your DEXT after it had been deleted, which seems incorrect.

However, if that's occurring, then it might explain this:

When I try to connect my USB4 accessory back again, it is still non-functional.

...as it repeatedly tried to initialize the driver that had already been deleted.

One thing I'd look for here is the I/O registry state of your kernel driver after you uninstall the app and then hot plug the accessory without reinstalling your app. Your DEXT IOKitPersonalties dictionary entry ends up in the kernel during matching, so if your DEXT is still "partially" active, you should see signs of that in the properties of your kernel support driver.

Alternatively, what does your driver "do" to your hardware during teardown? The iOS 18 log shows the driver being relaunched and then completing the full cycle from Start-> Free. IF your DEXT fails to complete that cycle, is there some way it could have been "left" in a state where it couldn't reestablish control?

What happens if you uninstall your app/DEXT while your hardware ISN'T attached? If you see the same failure in that case, then that's a pretty strong indication that this is a bug in your DEXT uninstall logic, which would put this well outside your control.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Is it possible you flipped the two logs you posted? The one you've labeled "18" shows DriverKit attempting to load your DEXT after it had been deleted, which seems incorrect.

No, I checked and the logs are named correctly. As you mentioned, in the iPadOS 18 log, the driver is re-loaded and goes through the Start phase again which is not the case if you look at the iPadOS 17 log. Also, I can see Stop getting called only in the 18 log maybe because of this reload.

One thing I'd look for here is the I/O registry state of your kernel driver after you uninstall the app and then hot plug the accessory without reinstalling your app.

Could you please guide me on what you mean by the kernel driver here? I have already checked the IORegistry to see whether any instances of my driver is still present after a hot plug and I can't seem to find any. Also, it doesn't help if I reinstall the app, enable the driver and hot plug the accessory (still non-functional).

Alternatively, what does your driver "do" to your hardware during teardown?

In my actual implementation, I stop ongoing transfers if any and close the queues which were created in my Start(). I also suspected whether I was doing something wrong in my Stop() implementation first. However, I later changed my Start and Stop to be minimal (just open the PCI device and enable/disable bus mastering) to verify this. I get the same kind of logs even with this.

What happens if you uninstall your app/DEXT while your hardware ISN'T attached?

Whenever my hardware gets detached, Stop() is successful without any crashes. Doing an uninstall in this case seems to be of no harm.

I went through the IOPCIDevice::ClientCrashed_Impl which is getting called in both the logs here. Below is the code excerpt from the link.

kern_return_t IOPCIDevice::ClientCrashed_Impl(IOService *client, uint64_t options)
{
	DLOG(ALWAYS_ON, "IOPCIDevice::ClientCrashed_Impl() for client %s\n", (client) ? client->getName() : "unknown");

    // only reset the device if the driver potentially changed the state of the device
    if(isOpen(client) == true)
    {
        DLOG(ALWAYS_ON, "%s: PCIDriverKit client, %s, crashed for device %s[" BDF() "], attempting to recover\n",
              __PRETTY_FUNCTION__,
              (client != NULL) ? client->getName() : "unknown",
              getName(),
              PCI_ADDRESS_TUPLE(this));

        IOReturn ret = parent->childClientCrashRecovery(this);

		// rdar://122664049 (Remove terminate-on-crash option from PCIDriverKit crash recovery)
		if (ret == kIOReturnSuccess && getProperty("terminate-on-crash"))
		{
			launchReprobeThread();
		}
    }

This exact log inside the if-block is getting printed in my case and I suspect whether the childClientCrashRecovery() and launchReprobeThread() functions are actually the ones triggering the reload in the case of 18.

I have not got a chance to investigate deeper into this. The only difference between the two logs is the recovery/reprobe mechanism which happens in 18 but fails to do so in 17. Can it be something where the recovery mechanism with launchReprobeThread() was added starting from iPadOS 18 (maybe Driverkit 24) and not present in iPadOS 17 altogether?

If this would help, I have another driver for matching some USB3 accessories (USBDriverKit) which seems to follow the same cycle of [Uninstall -> Recovery/Reprobe -> Driver Start/Init -> Driver Stop] but across all versions of iPadOS without any issues.

Could you please guide me on what you mean by the kernel driver here?

To start with, I'd recommend reading through my "Basic introduction to DEXT Matching and Loading" post as a general overview of the relationship between IOKit and DriverKit. In any case, the key point here is that when your device is attached, "something" always happens in IOKit regardless of whether or not your DEXT was installed. Moving to here:

I have already checked the IORegistry to see whether any instances of my driver are still present after a hot plug, and I can't seem to find any.

Even without your DEXT installed, you should be able to find IOKit drivers for your device; in particular, the IOProviderClass for your accessory should be present, since that's the connection "point" your kernel support driver would attach at, which would then launch your DEXT. If your provider ISN'T present, that's definitely worth closer investigation.

However, keep in mind that what IS there can be just as important as what ISN'T- if you've got an IOPCIDevice present even without the accessory attached, then there's been a teardown failure, and that could interfere with future hot plugs.

Also, it doesn't help if I reinstall the app, enable the driver, and hot plug the accessory (still non-functional).

Just to clarify, is your hardware bus or wall powered, and, if wall powered, have you tried power cycling the device?

Can it be something where the recovery mechanism with launchReprobeThread() was added starting from iPadOS 18 (maybe DriverKit 24) and not present in iPadOS 17 altogether?

Yes. For your reference, iPadOS 17 was ~IOPCIFamily-617 while iPadOS 18 was ~IOPCIFamily-664. If you compare the two versions, you'll find that clientCrashedThreadCall was added in the later release. However, I tracked down the origin of this change (r.106513773) and this was actually driven by performance, not stability or behavior. Relaunching the client like this allows us to avoid terminating the IOPCIDevice (which is what iOS 17 did) which makes resetting the driver stack significantly faster.

If this would help, I have another driver for matching some USB3 accessories (USBDriverKit) which seems to follow the same cycle of [Uninstall -> Recovery/Reprobe -> Driver Start/Init -> Driver Stop] but across all versions of iPadOS without any issues.

Unfortunately, that doesn't really mean much. The USB family is basically "built" around hot plug; it's always been a core feature, and the family has never been NEARLY as performance sensitive as the PCI family.

More to the point, its long-standing user client support (IOUSBLib.h and IOUSBHost framework) means that it's actually supported out of process "control" for a VERY long time [1]. The basic driver stack treats a DriverKit client crash in exactly the same way it would treat any of its other user client disconnects and simply resets the device.

[1] USBDriverKit's control model is very similar to the IOUSBHostFamily and, in fact, the key feature DriverKit provides to USB isn't device control [2], but the ability to inject new IOKit objects back into the kernel.

[2] On macOS, the standard (and superior) solution for vendor-specific devices is the IOUSBHostFamily, NOT DriverKit.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware