FIX: Adalora ranknum loaded on wrong device (#1852)

Locally, multiple AdaLoRA-related tests are failing. We did not catch
this in the nightly run because the tests were missing the necessary
pytest marker.

The issue is related to the change in #1742, which made it possible to
load different adapters on different devices.

Although that PR itself was sound, the issue is that for AdaLoRA, one of
its parameters, ranknum, was not listed in the other_param_names and was
thus not moved to the correct device. This oversight is now fixed and
the GPU tests are now passing locally for me.

This PR also adds the missing pytest marker to the test class that was
missing it, so that these errors should be caught by our nightly CI in
the future.

src/peft/tuners/adalora/layer.py

@@ -35,7 +35,8 @@ class AdaLoraLayer(LoraLayer):
  # List all names of layers that may contain adapter weights
  # Note: ranknum doesn't need to be included as it is not an nn.Module
  adapter_layer_names = ("lora_A", "lora_B", "lora_E", "lora_embedding_A", "lora_embedding_B")
- # other_param_names is defined in LoraLayer
+ # All names of other parameters that may contain adapter-related parameters
+ other_param_names = ("r", "lora_alpha", "scaling", "lora_dropout", "ranknum")
 
  def __init__(self, base_layer: nn.Module) -> None:
  super().__init__(base_layer)

tests/test_common_gpu.py

@@ -1085,6 +1085,7 @@ def test_8bit_dora_merging(self):
 
 
 @pytest.mark.skipif(not torch.cuda.is_available(), reason="test requires a CUDA GPU")
+@pytest.mark.single_gpu_tests
 class TestSameAdapterDifferentDevices:
  # 1639
  # The original issue comes down to the following problem: If the user has a base layer on CUDA, moves the adapter to