Merge branch 'ionic-diet'
Shannon Nelson says:
====================
ionic: putting ionic on a diet
Building on the performance work done in the previous patchset
[Link] https://lore.kernel.org/netdev/20240229193935.14197-1-shannon.nelson@amd.com/
this patchset puts the ionic driver on a diet, decreasing the memory
requirements per queue, and simplifies a few more bits of logic.
We trimmed the queue management structs and gained some ground, but
the most savings came from trimming the individual buffer descriptors.
The original design used a single generic buffer descriptor for Tx, Rx and
Adminq needs, but the Rx and Adminq descriptors really don't need all the
info that the Tx descriptors track. By splitting up the descriptor types
we can significantly reduce the descriptor sizes for Rx and Adminq use.
There is a small reduction in the queue management structs, saving about
3 cachelines per queuepair:
ionic_qcq:
Before: /* size: 2176, cachelines: 34, members: 23 */
After: /* size: 2048, cachelines: 32, members: 23 */
We also remove an array of completion descriptor pointers, or about
8 Kbytes per queue.
But the biggest savings came from splitting the desc_info struct into
queue specific structs and trimming out what was unnecessary.
Before:
ionic_desc_info:
/* size: 496, cachelines: 8, members: 10 */
After:
ionic_tx_desc_info:
/* size: 496, cachelines: 8, members: 6 */
ionic_rx_desc_info:
/* size: 224, cachelines: 4, members: 2 */
ionic_admin_desc_info:
/* size: 8, cachelines: 1, members: 1 */
In a 64 core host the ionic driver will default to 64 queuepairs of
1024 descriptors for Rx, 1024 for Tx, and 80 for Adminq and Notifyq.
The total memory usage for 64 queues:
Before:
65 * sizeof(ionic_qcq) 141,440
+ 64 * 1024 * sizeof(ionic_desc_info) 32,505,856
+ 64 * 1024 * sizeof(ionic_desc_info) 32,505,856
+ 64 * 1024 * 2 * sizeof(ionic_qc_info) 16,384
+ 1 * 80 * sizeof(ionic_desc_info) 39,690
----------
65,201,038
After:
65 * sizeof(ionic_qcq) 133,120
+ 64 * 1024 * sizeof(ionic_tx_desc_info) 32,505,856
+ 64 * 1024 * sizeof(ionic_rx_desc_info) 14,680,064
+ (removed) 0
+ 1 * 80 * sizeof(ionic_admin desc_info) 640
----------
47,319,680
This saves us approximately 18 Mbytes per port in a 64 core machine,
a 28% savings in our memory needs.
In addition, this improves our simple single thread / single queue
iperf case on a 9100 MTU connection from 86.7 to 95 Gbits/sec.
====================
Signed-off-by: 
David S. Miller <davem@davemloft.net>
Showing
Please register or sign in to comment