This document gives details of the various PCIe capabilities that exerciser device supports and how the exerciser is supposed to behave.
-
Introduction to PCIe Exerciser Endpoint Device
-
Execution model and flow control for following PCIe Features:
- DMA Transaction
- MSI Generation and Service
- Legacy Interrupt Generation and Service
- ATS Transaction
- Transaction Monitoring
PCIe Exerciser is a client device wrapped up by PCIe Endpoint. This device was created to meet SBSA (Server Based System Architecture) requirements for various PCIe capability validation tests. Exerciser's BAR0 space contains following control and support registers to support various PCIe capabilities in exerciser:
Register Name Description Address offset
------------- ----------- --------------
MSI Control Register Configure and trigger MSIs. 0x0
Legacy Interrupt Control Configure and trigger legacy interrupts. 0x4
Register
DMA Control Register Configure and trigger DMA transactions(DMA_CTL_REG) 0x8
DMA Offset Register Configure DMA transaction access offset (in bytes) 0xc
from base of exerciser memory.
Bus Address Register Configure bus address that is destination/source 0x10
for DMA or untranslated input address for ATSRequest.
DMA Length Register Configure DMA transaction length. 0x18
DMA Status Register Status of last DMA transaction. 0x1c
PASID Value Register Store the pasid value passed for initiating DMA/ATS 0x20
transaction.
ATS Control Register Configure and trigger ATS transaction(ATS_CTL_REG) 0x24
ATS Reply Translated Address Hold the translated address returned from an 0x28
Register ATSRequest.
ATS Translated Range Size Size of the ATS translated address region in bytes. 0x30
Register
ATS Reply Permission Permissions returned for translated memory address 0x38
Register region as a result of ATSRequest.
Illegal Requestor ID This register is DANGEROUS, as it will cause illegal 0x3c
Control Register transactions to be generated by exerciser.
Use case is when SW wants exerciser to issue a
transaction with custom/illegal requestor id,
rather than the one determined through PCI enumeration.
Transaction Trace Database Holds information about the config and memory 0x40
Register transactions. Each read will return one 32 bit entry
previously recorded.
Transaction Trace Control Control bit to start/stop transaction monitoring. 0x44
Register
DMA Transaction
DMA Control Register Bit Description r/w Value at reset
-------------------- --- ----------- --- --------------
dmatxntrig 3:0 Trigger DMA for given configuration RW 0h
Set to 1h to trigger DMA. This is reset
to 0h after transaction completes.
values 2h – Fh are reserved.
Dmatxndir 4 DMA transaction type 0b : Read, 1b: Write RW 0b
dmatxnsnoop 5 Snoop attribute for DMA transaction RW 0b
0: Snoop
1: No-snoop
dmapasiden 6 PASID Enable bit RW 0b
0: PASID disabled
1: PASID enabled
dmaIsPrivileged 7 Is Privileged access mode bit RW 0b
0: User access mode
1: Privileged access mode
(dmapasiden bit must be 1)
dmaIsInstruction 8 Is Instruction access bit RW 0b
0: Data access
1: Instruction access (dmapasiden bit
must be 1)
dmaUseATCforTranslation 9 Use ATC for input address translation RW 0b
0: do no translation mapping.
1: If true then, if DMA input address is
in ATC input address range, then DMA address
will be mapped to output translated address
as per translation response of last ATSRequest
dmaAddressType 11:10 DMA bus address type RW 0h
0: Default (treated as Untranslated)
1: Untranslated
2: Translated
3: Reserved
Note: Any transaction with Reserved address
type will fault by logging error in UR bit in
device status register in upstream rootport.
DMA status register will also reflect internal
error.
dmactlres 31:12 Reserved RO 0h
DMA Status Register Bit Description r/w Value at reset
------------------- -- - ----------- --- --------------
dmastatus 1:0 DMA error codes RO 0h
0h: No error. DMA successful
1h: DMA range out of bounds
2h: DMA internal error
3h: Reserved
dmaStatusClear 2 Write 1 to clear DMA status WO 0b
dmastatres 31:3 Reserved RO 0h
Pasid Value Register Bit Description r/w Value at reset
-------------------- --- ----------- --- --------------
PASIDValue 19:0 PASID value for DMA/ATSRequest RW 0h
PASID_Reserved 31:20 Reserved RO 0h
DMA Requestor ID Register Bit Description r/w Value at reset
------------------------ --- ----------- --- --------------
RequestorID 15:0 Custom Requestor ID RW 0h
for DMA(ReqIDValid must be 1)
ReqIDReserved 30:16 Reserved RO 0h
ReqIDValid 31 Requestor ID Valid bit RW 0b
0: Requestor ID is not valid
1: Requestor ID passed is valid
HOW TO USE DMA
-
Before triggering DMA all the required DMA attribute fields like DMA bus address, dmatxnsnoop etc., fields should be correctly set in the following registers:
- DMA Control Register
- Bus Address Register
- DMA Length Register
- DMA Requestor ID Register
- DMA Transaction Offset Register
- PASID Value Register
-
If user wants to do a Pure Translated transaction for a virtual address which is in ATC's input address range then dmaUseATCforTranslation bit must be set to true, so that input virtual address will be mapped to translated address as per ATC fields and translated address will go on the bus for DMA.
-
User can set custom requestor ID for an DMA transaction by using Illegal Requestor ID Control Register. Note: This is illegal as per PCIe specification as requestor ID should be assigned during PCIe enumeration.
-
If passed address for DMA transaction is already translated one, then dmaAddressType must be set to translated. Note: This bypasses the ATC and so is also illegal as per PCIe specification as only a transaction translated by the ATC is allowed to go out on the bus as a Translated Transaction. Also, if dmaAddressType is set to translated and dmaUseATCforTranslation bit is set, then exerciser will fault by logging an error message while initiating DMA, as translation mapping of an already translated address might go wrong.
-
DMAPasidEn bit must be set to 1 when a valid PASID is passed for DMA transaction. If dmaIsPrivileged or dmaIsInstruction is 1, then DMAPasidEn bit must be set to 1 for correct DMA response.
-
Status of the last DMA request can be known from DMA Status Register.
MSI Generation and Service
MSI Control Register Bit Description r/w Value at reset
-------------------- --- ----------- --- --------------
MSICTLID 10:0 Index of MSI to be triggered RW 0h
(0 to 2047)
MSICTLRES 30:11 Reserved RO 0h
MSICTLTRG 31 Set to 1 to trigger MSI. RW 0b
This bit is reset after MSI
is sent.
HOW TO TRIGGER MSI:
- Poll until (MSI Control Register).MSICTLTRG == 0.
- Write MSI Control Register with MSICTLTRG == 1 and MSICTLID == desired MSI Index to initiate MSI from exerciser device.
Legacy Interrupt Generation and Service
Legacy Interrupt Control Register Bit Description r/w Value at reset
--------------------------------- ---- ----------- --- --------------
intxctltrg 0 Trigger legacy interrupt RW 0b
Set to 1 to trigger
interrupt Reset this bit
to clear interrupt
intxctlres 31:1 Reserved RO 0h
HOW TO TRIGGER LEGACY/PIN INTERRUPT:
- Write 1 into intxctltrg to trigger legacy pin interrupt from exerciser device.
ATS Transaction
ATS_CTL_REG Bit Description r/w Value at reset
----------- --- ----------- --- --------------
ATSRequestTrigger 0 Trigger ATS Request for given WO 0b
configuration Set to 1h to
trigger DMA. This gets reset to
0h after transaction completes.
ATSRequestIsPrivileged 1 Access mode for ATS Request is RW 0b
privileged or not.
0: user access
1: Privileged access
(ATSPasidenabled bit must be 1)
ATSRequestNoWriteRequested 2 Read permission requested for RW 0b
ATS translated address region
0: Read & Write permission
requested
1: Only Read permission requested
ATSPasidEnabled 3 PASID Enable bit for ATSRequest RW 0b
0: PASID disabled
1: PASID enabled
ATSExcutePermissionRequested 4 Is Execute permission Requested RW 0b
or not:
0: execute permission not
requested
1: execute permission Requested
(ATSPasidenabled bit must be 1)
ATSInvalidate/ClearATC 5 Write 1 to clear ATC and W1C 0b
last ATS translated Results.
ATSIsInFlight 6 ATS Request is in flight or not RO 0b
0: ATS Request is not in flight
1: ATS Request is in flight
ATSTranslationStatus 7 ATS Translation status RO 0b
0: Translation failed
1: Translation Successful
ATSTranslationCacheable 8 ATS Translation is cacheable RO 0b
or not.
0: Not Cacheable, means
RW permission is 0
1: Cacheable, RW!=2’b00.
ATCInvalidated 9 Whether ATC was invalidated RO 0b
or not.
0: ATC not invalidated
1: ATC was invalidated
ATSReserved 31:10 Reserved RO 0h
ATSPermissionReg Bit Description r/w Value at reset
---------------- --- ----------- --- --------------
ExecutePermission 0 Whether execute permission is RO 0b
given for translated address or
not
0: Execute permission not given
1: Execute permission given
WritePermission 1 Whether write permission is RO 0b
given for translated address
or not.
0: Write permission not given
1: Write permission given
ReadPermission 2 Whether Read permission is RO 0b
given for translated address
or not
0: Read permission not given
1: Read permission given
ExcutePrivileged 3 If 1 Execute permission was RO 0b
given when ATSRequest was
privileged.
WritePrivileged 4 If 1 Write permission was RO 0b
given when ATSRequest was
privileged.
ReadPrivileged 5 If 1 Read permission was given RO 0b
when ATSRequest was privileged.
ReplyPermissionReserved 31:6 Reserved RO 0h
HOW TO USE ATS:
-
Before starting an ATS request, untranslated input address for ATSRequest must be written onto Bus Address Register, with all related bit (is_read, etc.) set in ATS_CTL_REG. SW should enable ATS and PASID capability for exerciser (which will set exerciser_ats_enabled and exerciser_ats_pasid_enabled, where exerciser_ats_enabled will enable client side ATS Transaction capability for exerciser and exerciser_ats_pasid_enabled will enable PASID based ATSRequest for exerciser). Then ATS request can be triggered. Registers to be used for ATS:
- ATS Control Register
- PASID Value Register
- ATS Reply Translated Address Register
- ATS Reply Permission Register
- ATS Translated Range Size Register
-
If there is any fault occurred during the translation, then error log will get reflected in command line as MSG_INFO and status of ATSRequest remains as 0 (failed - by default). User can detect whether error has occurred during the translation when following condition is true:
- ATSRequest is not in flight (ATS_CTL_REG[6] == 0) and ATS transaction status bit (ATS_CTL_REG[7]) is 0.
-
If ATSRequest response result is cacheable (RW!= 2'b00 : Read/Write permission returned as ATSRequest result), then ATS response cacheable bit (ATS_CTL_REG[8]) is set to true.
-
If a matching ATC invalidation comes while ATS request is in flight, translation will be retried, discarding the current translation result.
-
If there is no error and translation response is cacheable, then translation result will be stored in ATC along with permissions returned.
-
Translation result will also get reflected in ATS Reply Translated Address Register, ATS Reply Permission Register, ATS Translated Range Size Register
-
DMA transaction's range must be fully covered by the ATC entry or it will fail.
-
Whenever any DMA transaction want to use ATC based translated address, it must set dmaUseATCforTranslation flag (DMA_CTL_REG[11]) to true, so that translated address will go into the bus after being remapped to translated output address if it has all the necessary read/write permissions.
-
Currently exerciser can store maximum 1 ATS request result, mappings are stored in ATC till invalidated by SMMU or user writes 1 to clear ATC bit (ATS_CTL_REG[5]) which will clear/invalidate all ATC mappings and fields.
-
ATC invalidation happens in following ways when an ATCInvalidate Request received by the exerciser:
-
If ATC is empty, then there is nothing to be invalidated, ATC invalidation call gets returned immediately.
-
Some form of invalidation will be done when invalidation range contains ATC's untranslated input address and either of the following is true:
- Invalidation range contains address for which any ongoing ATSRequest is in progress.
- SubstreamID is valid for both atc_invalidate request and atc_entry and either ATC invalidate's substreamID matches with atc_entry's substreamID or G-Global PASID bit is set.
-
If there is any ongoing ATS translation request with the address that is in atc_invalidation_address_range, then current translation result will be discarded, ATC will be invalidated and then current ATS translation request will be retried again.
-
If any DMA PCIe Translated transaction is using any address which is in atc_invalidation_address_range, then we wait for the DMA transaction to complete and then ATC invalidation is done.
-
If no PCIe Translated transaction is using any address which is in atc_invalidation_address_range, then we invalidate the ATC immediately if invalidation_range intersects/matches with atc_entry's untranslated address.
-
-
ATC can be invalidated by writing 1 to clear ATC bit (ATS_CTL_REG[5]), which will eventually invalidate ATC and discard all the mappings for last ATSRequest.
-
Note: the following ATS behaviours are illegal under the PCIe specification, but implemented in the model for validation purposes only:
- reading the ATC entry
- issuing a translated transaction without using the ATC
Transaction Monitoring
Transaction monitoring capabilities in the exerciser provides the ability to record the incoming transactions, for both config and memory transactions. This includes,
-
config read and write transactions serviced in PCIe endpoints.
-
memory transactions serviced in PCIe endpoint BARs.
-
memory transactions which are forwarded from PCIe endpoint to device(like exerciser).
Each recorded transaction has the following information:
-
a 32 bit entry indicating transaction attributes. It has bitfields for request type, R/W transaction, cfg/mem transaction and transaction byte size(bitfields given below)
-
a 32 bit entry indicating lower 32 bits of config or memory address
-
a 32 bit entry indicating upper 32 bits of config or memory address
-
a 32 bit entry indicating lower 32 bits of transaction data read or written to
-
a 32 bit entry indicating upper 32 bits of transaction data read or written to
Note: Each beat in a burst transaction is recorded as a single separate transaction.
________________________________
| TX ATTRIBUTES |
| CFG/MEM ADDRESS_LO |
| CFG/MEM ADDRESS_HI |
| DATA_LO |
| DATA_HI |
|________________________________|
Recorded Information Format
Registers specific to transaction monitoring
TransactionTraceReg Bit Description R/W Value at reset
------------------- --- ----------- --- --------------
Transaction Info. 31:0 Returns one 32 bit entry
mentioned above on each RO 0xFFFFffff
read.
For transaction attributes (TX_ATTRIBUTES), lower 16 bits indicates the type, R/W and cfg/mem information.
Upper 16 bits of transaction attributes indicates transaction byte size. The bit indicating the
transaction byte size is calculated as log2(bytes). For example, if size is 8, bit3 of the upper 16 bits
will be set.
------------------------------------------------------------------------------------------------
31:16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 0
------------------------------------------------------------------------------------------------
| Transaction byte size | Reserved C/M R/W Type
Bit0: Request type(defaults to zero).
Bit1: 1 for Read and 0 for Write.
Bit2: 1 for CFG transaction and 0 for MEM transaction.
Bits 31-16: Log2 of transaction size, in bytes.
TransactionTraceControlReg Bit Description R/W Value at reset
-------------------------- --- ----------- --- --------------
Start/stop monitoring 0 Control bit to start or stop
transaction monitoring. RW 0x0
HOW TO USE TRANSACTION MONITORING:
- Number of transactions recorded depends on the exerciser parameter 'max_transaction_trace_entries'. The maximum entries supported is 32, defaults to 16, minimum 1.
- To start transaction monitoring, write 0x1 to transaction trace control register(offset 0x44).
- To stop transaction monitoring, write 0x0 to transaction trace control register(offset 0x44).
- Read the transaction trace database register(offset 0x40). Each read returns one 32 bit entry mentioned above, starting from first transaction. A value of 0xFFFFffff indicates invalid entry.
Copyright (c) 2023, Arm Limited and Contributors. All rights reserved.