syzbot

map[Benign:true Confident:true Explanation:The data race occurs between the update of `vq->last_used_idx` in the buffer
reaping path and its read in the interrupt handler.

### Race Summary
`virtqueue_get_buf_ctx_split` (CPU 0) vs `more_used_split` (CPU 1) on
`vq->last_used_idx`.

### Synchronization Context
- **CPU 0:** Executing `virtnet_poll_tx` -> `virtnet_free_old_xmit` ->
`virtqueue_get_buf`. The driver ensures serialization of this path using the
transmit queue lock (`txq->_xmit_lock`) or NAPI context.
- **CPU 1:** Executing `vring_interrupt` -> `more_used`. This is the hard
interrupt handler for the virtqueue, which runs without acquiring the driver's
serialization locks.

The race exists because `vring_interrupt` performs a lockless check of the queue
state to avoid unnecessary callback calls.

### Final Classification: **BENIGN**

### Reasoning
The race is structurally tolerant to stale or torn reads for the following
reasons:
1.  **Heuristic Nature:** The read in `more_used_split` is used as a performance
optimization to decide whether to call the driver's callback. If it sees a stale
value of `last_used_idx`, it will likely think there is more work to do and call
the callback. The callback (e.g., `skb_xmit_done`) simply schedules NAPI or
wakes the queue, both of which are safe to call redundantly.
2.  **Livelock Prevention:** If the interrupt handler sees a "new" value of
`last_used_idx` and incorrectly concludes there is no work (`IRQ_NONE`), no work
is actually lost. This is because `last_used_idx` is only incremented by the
driver *after* it has already processed a buffer. If `last_used_idx ==
used->idx`, the driver has already caught up with the device's used ring.
3.  **Atomicity:** The racing variable is a 16-bit aligned integer (`u16`). On
almost all architectures supported by Linux, 16-bit aligned accesses are atomic,
preventing torn reads.
4.  **Consistency with Packed Ring:** The newer "packed" ring implementation in
the same file (`virtio_ring.c`) already uses `READ_ONCE()` and `WRITE_ONCE()`
for the same variable in the same logic (see `more_used_packed` and
`update_last_used_idx_packed`), indicating that the developers recognize this as
a benign race that simply requires marking.

### Recommended Annotations
To satisfy KCSAN and prevent potential compiler optimizations, the accesses to
`vq->last_used_idx` in the split ring implementation should be marked:
- Wrap the read in `more_used_split` and `more_used_split_in_order` with
`READ_ONCE()`.
- Wrap the increment in `virtqueue_get_buf_ctx_split` and
`virtqueue_get_buf_ctx_split_in_order` with `WRITE_ONCE(vq->last_used_idx,
vq->last_used_idx + 1)`.]

map[KernelSrc:/app/workdir/cache/src/993bb6115a6fd76b2e7e72455dca0bd2cc6d123e]

map[KernelObj:/app/workdir/cache/build/0435539076637b53b8068bf3e07b55b5ec59ec42]

map[Index:codesearch-index]

gemini-3-flash-preview

map[Benign:true Confident:true]

# KCSAN Data Race Severity Analysis Protocol

You are an expert Linux kernel concurrency engineer analyzing a Kernel
Concurrency Sanitizer (KCSAN) data race report to determine if it is
**BENIGN** or **HARMFUL**.

## 1. CLASSIFICATIONS

### **BENIGN (Truly Benign)**
The logic is sound and structurally tolerant to compiler optimizations or
stale/torn reads.

- **Diagnostics/Stats:** Reads used only for '/proc', '/sys', counters, or
  'pr_info'.
- **Heuristic Hints:** A "hint" flag where an old value only causes a
  slightly delayed update or a sub-optimal but safe fast-path.
- **Single-Writer Flag Updates:** A single writer updating flags where the
  concurrent read is a simple bitwise check (e.g., 'flags & MASK'). These are
  historically tolerated, assuming neither "Fused Accesses" nor "Ordering
  Violations" are relevant in this context.
- **Marked Reloads:** A load feeding into a 'cmpxchg()' loop or checked
  against a later 'READ_ONCE()' reload.
- **Safe Overwrites:** Writing the same value already present.

### **HARMFUL (Logic Bug or Marking Required)**
The race causes incorrect behavior due to a synchronization failure or
because missing annotations allow the compiler to break the algorithm.

**Marking Required for Correctness:**
The algorithm is logically sound but requires annotations ('READ_ONCE()',
'WRITE_ONCE()', 'smp_load_acquire()', 'smp_store_release()', etc.) to be safe.
- **Fused Accesses:** The compiler might merge accesses or hoist a load out
  of a loop, breaking polling/wait loops (livelocks).
- **Torn Accesses:** A large access (e.g., 64-bit on 32-bit arch) might be
  split into multiple non-atomic accesses. Note that 'READ_ONCE()' does **not**
  guarantee atomicity for 64-bit variables on 32-bit architectures.
- **Ordering Violations:** The race breaks a "happens-before" relationship
  (requires primitives with implied or explicit memory barriers).

**Logic Bugs:**
A fundamental synchronization failure. Marking accesses will **not** fix it;
the logic itself must change.
- **Pointers/Lifecycle:** The racing variable is a pointer being dereferenced
  or a refcount governing object lifecycle (Use-After-Free risk).
- **Control Flow:** The variable guards a critical section, memory allocation,
  or hardware command.
- **Bitfields:** Concurrent writes to different bits in the same word.
  Compilers often use non-atomic read-modify-write sequences, meaning a
  write to 'bit_A' can "clobber" a concurrent write to 'bit_B'. However,
  do not blindly assume all bitfield accesses are harmful; you must prove
  that a concurrent write actually clobbers another in a way that breaks
  logic.
- **Complex Structures:** Races on shared lists, trees, or hashmaps.
- **Lossy Updates:** Concurrent plain RMW operations (e.g., 'var++') on
  non-diagnostic variables where every increment must be preserved.
- **State Machines:** Races allowing a state machine to bypass transitions
  or enter an invalid state.
- **Adjacent Unsynchronized Operations:** Consider races happening at the
  same time. For example, if both threads execute 'struct->has_elements = true;
  list_add(node, &struct->list);', the race on 'has_elements' implies an
  adjacent race on 'list_head', which is HARMFUL.

## 2. RESEARCH & ANALYSIS WORKFLOW

1.  **Locate the Race:** Find the exact variables and functions in the stack
    traces. **Use codesearch tools to read the actual source code and
    confirm all assumptions.** Do not speculate about hypothetical compiler
    behaviors or theoretical dangers (e.g., dismissing something as
    "fundamentally unsafe") without tracing the actual data flow to a crash.
2.  **Contextualize:** Identify held locks, RCU sections, or interrupt
    contexts. Explain why the race is possible (e.g., "Thread A holds 'lock',
    but Thread B is a lockless reader").
3.  **Data Flow:** Follow the racing variable into its subsequent uses. If
    the reader sees a stale or torn value, what branch is taken? Does it lead
    to a pointer dereference or 'BUG_ON()'? **You must prove harm via this
    data flow.**
4.  **Failure Sequencing (if HARMFUL):** Construct a two-column execution trace
    demonstrating the failure.

    *Example format:*

    CPU0                                   CPU1
    
    function_a()
      // lockless read
      ptr = global_ptr
    
    <PREEMPT>
                                           function_b()
                                             lock(&my_lock)
                                             global_ptr = NULL
                                             unlock(&my_lock)
                                             kfree(ptr)
    
      if (ptr)
        *ptr = 1; // -> Use-After-Free!

## 3. OUTPUT FORMAT

- **Race Summary:** '[Function A]' vs '[Function B]' on '[Variable/Field]'.
- **Synchronization context:** List held locks and explain the lack of mutual
  exclusion.
- **Final Classification:** **BENIGN** or **HARMFUL**.

**If BENIGN:**
- **Reasoning:** Briefly explain why the race is structurally tolerant to stale
  or torn reads.
- **Recommended Annotations:** Suggest appropriate annotations (e.g.,
  'READ_ONCE()', 'data_race()').

**If HARMFUL:**
- **Failure Reasoning:** Explain and provide a two-column interleaving showing
  exactly how the race leads to a failure such as corruption or kernel crash.
- **Recommended Fix (ONLY if fix is trivial):** Suggest a structural fix (e.g.,
  "Hold 'mapping->i_pages' lock", "Convert to 'atomic_t'") or required memory
  ordering annotations (e.g., "Wrap in 'READ_ONCE()'", "Use
  'smp_load_acquire()'").

Prefer calling several tools at the same time to save round-trips.


Use set-results tool to provide results of the analysis.
It must be called exactly once before the final reply.
Ignore results of this tool.

The data race report is:

==================================================================
BUG: KCSAN: data-race in virtqueue_get_buf_ctx / vring_interrupt

read-write to 0xffff88810224b25c of 2 bytes by interrupt on cpu 0:
 virtqueue_get_buf_ctx_split drivers/virtio/virtio_ring.c:959 [inline]
 virtqueue_get_buf_ctx+0x607/0xdb0 drivers/virtio/virtio_ring.c:3086
 virtqueue_get_buf+0x1f/0x30 drivers/virtio/virtio_ring.c:3092
 __free_old_xmit+0x53/0x340 drivers/net/virtio_net.c:588
 virtnet_free_old_xmit+0x39/0x1b0 drivers/net/virtio_net.c:629
 free_old_xmit drivers/net/virtio_net.c:958 [inline]
 virtnet_poll_tx+0x2de/0xca0 drivers/net/virtio_net.c:3239
 __napi_poll+0x61/0x300 net/core/dev.c:7730
 napi_poll net/core/dev.c:7793 [inline]
 net_rx_action+0x452/0x930 net/core/dev.c:7950
 handle_softirqs+0xb9/0x280 kernel/softirq.c:622
 __do_softirq kernel/softirq.c:656 [inline]
 invoke_softirq kernel/softirq.c:496 [inline]
 __irq_exit_rcu+0x42/0xd0 kernel/softirq.c:735
 common_interrupt+0x83/0x90 arch/x86/kernel/irq.c:326
 asm_common_interrupt+0x26/0x40 arch/x86/include/asm/idtentry.h:688
 __preempt_count_dec_and_test arch/x86/include/asm/preempt.h:95 [inline]
 __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:179 [inline]
 _raw_spin_unlock_irqrestore+0x1a/0x30 kernel/locking/spinlock.c:198
 spin_unlock_irqrestore include/linux/spinlock.h:408 [inline]
 __skb_try_recv_datagram+0x123/0x320 net/core/datagram.c:267
 __unix_dgram_recvmsg+0x25a/0x870 net/unix/af_unix.c:2587
 unix_dgram_recvmsg+0x7e/0x90 net/unix/af_unix.c:2686
 sock_recvmsg_nosec+0xc2/0xf0 net/socket.c:1137
 ____sys_recvmsg+0x26f/0x280 net/socket.c:2916
 ___sys_recvmsg+0x11f/0x3b0 net/socket.c:2960
 do_recvmmsg+0x1ef/0x560 net/socket.c:3055
 __sys_recvmmsg net/socket.c:3129 [inline]
 __do_sys_recvmmsg net/socket.c:3152 [inline]
 __se_sys_recvmmsg net/socket.c:3145 [inline]
 __x64_sys_recvmmsg+0xe5/0x170 net/socket.c:3145
 x64_sys_call+0x80f/0x3020 arch/x86/include/generated/asm/syscalls_64.h:300
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0x12c/0x3b0 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f

read to 0xffff88810224b25c of 2 bytes by interrupt on cpu 1:
 more_used_split drivers/virtio/virtio_ring.c:906 [inline]
 more_used drivers/virtio/virtio_ring.c:3218 [inline]
 vring_interrupt+0x48/0x310 drivers/virtio/virtio_ring.c:3233
 __handle_irq_event_percpu+0x8b/0x480 kernel/irq/handle.c:209
 handle_irq_event_percpu kernel/irq/handle.c:246 [inline]
 handle_irq_event+0x64/0xf0 kernel/irq/handle.c:263
 handle_edge_irq+0x154/0x450 kernel/irq/chip.c:856
 generic_handle_irq_desc include/linux/irqdesc.h:186 [inline]
 handle_irq arch/x86/kernel/irq.c:262 [inline]
 call_irq_handler arch/x86/kernel/irq.c:-1 [inline]
 __common_interrupt+0x60/0xb0 arch/x86/kernel/irq.c:333
 common_interrupt+0x7e/0x90 arch/x86/kernel/irq.c:326
 asm_common_interrupt+0x26/0x40 arch/x86/include/asm/idtentry.h:688
 __preempt_count_dec_and_test arch/x86/include/asm/preempt.h:95 [inline]
 __raw_spin_unlock_irqrestore include/linux/spinlock_api_smp.h:179 [inline]
 _raw_spin_unlock_irqrestore+0x1a/0x30 kernel/locking/spinlock.c:198
 spin_unlock_irqrestore include/linux/spinlock.h:408 [inline]
 avc_reclaim_node security/selinux/avc.c:488 [inline]
 avc_alloc_node+0x21c/0x280 security/selinux/avc.c:507
 avc_insert security/selinux/avc.c:618 [inline]
 avc_compute_av+0xb0/0x430 security/selinux/avc.c:993
 avc_perm_nonode+0x5e/0xe0 security/selinux/avc.c:1117
 avc_has_perm_noaudit+0xf2/0x130 security/selinux/avc.c:1160
 avc_has_perm+0x60/0x190 security/selinux/avc.c:1195
 may_create+0x455/0x4a0 security/selinux/hooks.c:1880
 selinux_inode_symlink+0x22/0x30 security/selinux/hooks.c:3092
 security_inode_symlink+0x75/0xb0 security/security.c:1698
 vfs_symlink+0x8e/0x220 fs/namei.c:5635
 filename_symlinkat+0xe8/0x2b0 fs/namei.c:5668
 __do_sys_symlinkat fs/namei.c:5688 [inline]
 __se_sys_symlinkat+0x43/0x1b0 fs/namei.c:5683
 __x64_sys_symlinkat+0x43/0x50 fs/namei.c:5683
 x64_sys_call+0x2b7d/0x3020 arch/x86/include/generated/asm/syscalls_64.h:267
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0x12c/0x3b0 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f

value changed: 0x25f1 -> 0x25f2

Reported by Kernel Concurrency Sanitizer on:
CPU: 1 UID: 0 PID: 8527 Comm: syz-executor Tainted: G        W           syzkaller #0 PREEMPT(full) 
Tainted: [W]=WARN
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 04/18/2026
==================================================================

The data race occurs between the update of `vq->last_used_idx` in the buffer reaping path and its read in the interrupt handler.

### Race Summary
`virtqueue_get_buf_ctx_split` (CPU 0) vs `more_used_split` (CPU 1) on `vq->last_used_idx`.

### Synchronization Context
- **CPU 0:** Executing `virtnet_poll_tx` -> `virtnet_free_old_xmit` -> `virtqueue_get_buf`. The driver ensures serialization of this path using the transmit queue lock (`txq->_xmit_lock`) or NAPI context.
- **CPU 1:** Executing `vring_interrupt` -> `more_used`. This is the hard interrupt handler for the virtqueue, which runs without acquiring the driver's serialization locks.

The race exists because `vring_interrupt` performs a lockless check of the queue state to avoid unnecessary callback calls.

### Final Classification: **BENIGN**

### Reasoning
The race is structurally tolerant to stale or torn reads for the following reasons:
1.  **Heuristic Nature:** The read in `more_used_split` is used as a performance optimization to decide whether to call the driver's callback. If it sees a stale value of `last_used_idx`, it will likely think there is more work to do and call the callback. The callback (e.g., `skb_xmit_done`) simply schedules NAPI or wakes the queue, both of which are safe to call redundantly.
2.  **Livelock Prevention:** If the interrupt handler sees a "new" value of `last_used_idx` and incorrectly concludes there is no work (`IRQ_NONE`), no work is actually lost. This is because `last_used_idx` is only incremented by the driver *after* it has already processed a buffer. If `last_used_idx == used->idx`, the driver has already caught up with the device's used ring.
3.  **Atomicity:** The racing variable is a 16-bit aligned integer (`u16`). On almost all architectures supported by Linux, 16-bit aligned accesses are atomic, preventing torn reads.
4.  **Consistency with Packed Ring:** The newer "packed" ring implementation in the same file (`virtio_ring.c`) already uses `READ_ONCE()` and `WRITE_ONCE()` for the same variable in the same logic (see `more_used_packed` and `update_last_used_idx_packed`), indicating that the developers recognize this as a benign race that simply requires marking.

### Recommended Annotations
To satisfy KCSAN and prevent potential compiler optimizations, the accesses to `vq->last_used_idx` in the split ring implementation should be marked:
- Wrap the read in `more_used_split` and `more_used_split_in_order` with `READ_ONCE()`.
- Wrap the increment in `virtqueue_get_buf_ctx_split` and `virtqueue_get_buf_ctx_split_in_order` with `WRITE_ONCE(vq->last_used_idx, vq->last_used_idx + 1)`.

gemini-3-flash-preview

							input: 6261
							output: 23
							thoughts: 284
						

**Data Race Investigation: `virtqueue_get_buf_ctx_split` vs. `more_used_split`**

Okay, so the report flagged a data race. My initial instinct is to dig right into the specifics. It looks like we have a race condition between `virtqueue_get_buf_ctx_split` (called indirectly through `virtqueue_get_buf`) and `more_used_split` (called through `vring_interrupt`). The racing variable is at `0xffff88810224b25c` and is only 2 bytes in size. Seems to have incremented from `0x25f1` to `0x25f2`, which confirms a 16-bit increment.

My next step is to examine the source code. I know these functions are in `drivers/virtio/virtio_ring.c`, so I'll go there. Let me quickly find the relevant lines. Oh, here they are, I see `virtqueue_get_buf_ctx_split` at line 959 and `more_used_split` at line 906. Wait, wait, it says they are inline. I'll need to remember that when I examine the code. Let's get to those lines.

map[Dir:drivers/virtio/]

map[Files:[virtio.c virtio_anchor.c virtio_balloon.c virtio_debug.c virtio_dma_buf.c virtio_input.c virtio_mem.c virtio_mmio.c virtio_pci_admin_legacy_io.c virtio_pci_common.c virtio_pci_common.h virtio_pci_legacy.c virtio_pci_legacy_dev.c virtio_pci_modern.c virtio_pci_modern_dev.c virtio_ring.c virtio_rtc_arm.c virtio_rtc_class.c virtio_rtc_driver.c virtio_rtc_internal.h virtio_rtc_ptp.c virtio_vdpa.c] Subdirs:<nil>]

gemini-3-flash-preview

							input: 7102
							output: 26
							thoughts: 23
						

Here's my take on the thought process, as if it were my own:

**Diving into the Virtio Ring Implementation**

Alright, let's get down to brass tacks. I need to understand what's going on with the virtio ring buffer in this specific context. Given the problem at hand, I'm going to start by focusing on the `drivers/virtio/virtio_ring.c` file, specifically around the lines indicated. That's the core of the data transfer mechanism, and that's likely where the root of the problem will be found. I'll need to refresh my understanding of how the descriptors, the available and used rings, and the interrupt handling all interact within this file. I'll be looking for any potential issues with synchronization, descriptor management, or data consistency. I'm expecting this file to have a lot of complex interactions, so I'll need to go slow, and double-check my assumptions at every step.

map[File:drivers/virtio/virtio_ring.c]

map[Contents:// SPDX-License-Identifier: GPL-2.0-or-later
/* Virtio ring implementation.
 *
 *  Copyright 2007 Rusty Russell IBM Corporation
 */
#include <linux/virtio.h>
#include <linux/virtio_ring.h>
#include <linux/virtio_config.h>
#include <linux/device.h>
#include <linux/slab.h>
#include <linux/module.h>
#include <linux/hrtimer.h>
#include <linux/dma-mapping.h>
#include <linux/kmsan.h>
#include <linux/spinlock.h>
#include <xen/xen.h>

#ifdef DEBUG
/* For development, we want to crash whenever the ring is screwed. */
#define BAD_RING(_vq, fmt, args...)				\
	do {							\
		dev_err(&(_vq)->vq.vdev->dev,			\
			"%s:"fmt, (_vq)->vq.name, ##args);	\
		BUG();						\
	} while (0)
/* Caller is supposed to guarantee no reentry. */
#define START_USE(_vq)						\
	do {							\
		if ((_vq)->in_use)				\
			panic("%s:in_use = %i\n",		\
			      (_vq)->vq.name, (_vq)->in_use);	\
		(_vq)->in_use = __LINE__;			\
	} while (0)
#define END_USE(_vq) \
	do { BUG_ON(!(_vq)->in_use); (_vq)->in_use = 0; } while(0)
#define LAST_ADD_TIME_UPDATE(_vq)				\
	do {							\
		ktime_t now = ktime_get();			\
								\
		/* No kick or get, with .1 second between?  Warn. */ \
		if ((_vq)->last_add_time_valid)			\
			WARN_ON(ktime_to_ms(ktime_sub(now,	\
				(_vq)->last_add_time)) > 100);	\
		(_vq)->last_add_time = now;			\
		(_vq)->last_add_time_valid = true;		\
	} while (0)
#define LAST_ADD_TIME_CHECK(_vq)				\
	do {							\
		if ((_vq)->last_add_time_valid) {		\
			WARN_ON(ktime_to_ms(ktime_sub(ktime_get(), \
				      (_vq)->last_add_time)) > 100); \
		}						\
	} while (0)
#define LAST_ADD_TIME_INVALID(_vq)				\
	((_vq)->last_add_time_valid = false)
#else
#define BAD_RING(_vq, fmt, args...)				\
	do {							\
		dev_err(&_vq->vq.vdev->dev,			\
			"%s:"fmt, (_vq)->vq.name, ##args);	\
		(_vq)->broken = true;				\
	} while (0)
#define START_USE(vq)
#define END_USE(vq)
#define LAST_ADD_TIME_UPDATE(vq)
#define LAST_ADD_TIME_CHECK(vq)
#define LAST_ADD_TIME_INVALID(vq)
#endif

enum vq_layout {
	VQ_LAYOUT_SPLIT = 0,
	VQ_LAYOUT_PACKED,
	VQ_LAYOUT_SPLIT_IN_ORDER,
	VQ_LAYOUT_PACKED_IN_ORDER,
};

struct vring_desc_state_split {
	void *data;			/* Data for callback. */

	/* Indirect desc table and extra table, if any. These two will be
	 * allocated together. So we won't stress more to the memory allocator.
	 */
	struct vring_desc *indir_desc;
	u32 total_in_len;
};

struct vring_desc_state_packed {
	void *data;			/* Data for callback. */

	/* Indirect desc table and extra table, if any. These two will be
	 * allocated together. So we won't stress more to the memory allocator.
	 */
	struct vring_packed_desc *indir_desc;
	u16 num;			/* Descriptor list length. */
	u16 last;			/* The last desc state in a list. */
	u32 total_in_len;		/* In length for the skipped buffer. */
};

struct vring_desc_extra {
	dma_addr_t addr;		/* Descriptor DMA addr. */
	u32 len;			/* Descriptor length. */
	u16 flags;			/* Descriptor flags. */
	u16 next;			/* The next desc state in a list. */
};

struct vring_virtqueue_split {
	/* Actual memory layout for this queue. */
	struct vring vring;

	/* Last written value to avail->flags */
	u16 avail_flags_shadow;

	/*
	 * Last written value to avail->idx in
	 * guest byte order.
	 */
	u16 avail_idx_shadow;

	/* Per-descriptor state. */
	struct vring_desc_state_split *desc_state;
	struct vring_desc_extra *desc_extra;

	/* DMA address and size information */
	dma_addr_t queue_dma_addr;
	size_t queue_size_in_bytes;

	/*
	 * The parameters for creating vrings are reserved for creating new
	 * vring.
	 */
	u32 vring_align;
	bool may_reduce_num;
};

struct vring_virtqueue_packed {
	/* Actual memory layout for this queue. */
	struct {
		unsigned int num;
		struct vring_packed_desc *desc;
		struct vring_packed_desc_event *driver;
		struct vring_packed_desc_event *device;
	} vring;

	/* Driver ring wrap counter. */
	bool avail_wrap_counter;

	/* Avail used flags. */
	u16 avail_used_flags;

	/* Index of the next avail descriptor. */
	u16 next_avail_idx;

	/*
	 * Last written value to driver->flags in
	 * guest byte order.
	 */
	u16 event_flags_shadow;

	/* Per-descriptor state. */
	struct vring_desc_state_packed *desc_state;
	struct vring_desc_extra *desc_extra;

	/* DMA address and size information */
	dma_addr_t ring_dma_addr;
	dma_addr_t driver_event_dma_addr;
	dma_addr_t device_event_dma_addr;
	size_t ring_size_in_bytes;
	size_t event_size_in_bytes;
};

struct vring_virtqueue;

struct virtqueue_ops {
	int (*add)(struct vring_virtqueue *vq, struct scatterlist *sgs[],
		   unsigned int total_sg, unsigned int out_sgs,
		   unsigned int in_sgs,	void *data,
		   void *ctx, bool premapped, gfp_t gfp,
		   unsigned long attr);
	void *(*get)(struct vring_virtqueue *vq, unsigned int *len, void **ctx);
	bool (*kick_prepare)(struct vring_virtqueue *vq);
	void (*disable_cb)(struct vring_virtqueue *vq);
	bool (*enable_cb_delayed)(struct vring_virtqueue *vq);
	unsigned int (*enable_cb_prepare)(struct vring_virtqueue *vq);
	bool (*poll)(const struct vring_virtqueue *vq,
		     unsigned int last_used_idx);
	void *(*detach_unused_buf)(struct vring_virtqueue *vq);
	bool (*more_used)(const struct vring_virtqueue *vq);
	int (*resize)(struct vring_virtqueue *vq, u32 num);
	void (*reset)(struct vring_virtqueue *vq);
};

struct vring_virtqueue {
	struct virtqueue vq;

	/* Is DMA API used? */
	bool use_map_api;

	/* Can we use weak barriers? */
	bool weak_barriers;

	/* Other side has made a mess, don't try any more. */
	bool broken;

	/* Host supports indirect buffers */
	bool indirect;

	/* Host publishes avail event idx */
	bool event;

	enum vq_layout layout;

	/*
	 * Without IN_ORDER it's the head of free buffer list. With
	 * IN_ORDER and SPLIT, it's the next available buffer
	 * index. With IN_ORDER and PACKED, it's unused.
	 */
	unsigned int free_head;

	/*
	 * With IN_ORDER, once we see an in-order batch, this stores
	 * this last entry, and until we return the last buffer.
	 * After this, id is set to UINT_MAX to mark it invalid.
	 * Unused without IN_ORDER.
	 */
	struct used_entry {
		u32 id;
		u32 len;
	} batch_last;

	/* Number we've added since last sync. */
	unsigned int num_added;

	/* Last used index  we've seen.
	 * for split ring, it just contains last used index
	 * for packed ring:
	 * bits up to VRING_PACKED_EVENT_F_WRAP_CTR include the last used index.
	 * bits from VRING_PACKED_EVENT_F_WRAP_CTR include the used wrap counter.
	 */
	u16 last_used_idx;

	/* With IN_ORDER and SPLIT, last descriptor id we used to
	 * detach buffer.
	 */
	u16 last_used;

	/* Hint for event idx: already triggered no need to disable. */
	bool event_triggered;

	union {
		/* Available for split ring */
		struct vring_virtqueue_split split;

		/* Available for packed ring */
		struct vring_virtqueue_packed packed;
	};

	/* How to notify other side. FIXME: commonalize hcalls! */
	bool (*notify)(struct virtqueue *vq);

	/* DMA, allocation, and size information */
	bool we_own_ring;

	union virtio_map map;

#ifdef DEBUG
	/* They're supposed to lock for us. */
	unsigned int in_use;

	/* Figure out if their kicks are too delayed. */
	bool last_add_time_valid;
	ktime_t last_add_time;
#endif
};

static struct vring_desc_extra *vring_alloc_desc_extra(unsigned int num);
static void vring_free(struct virtqueue *_vq);

/*
 * Helpers.
 */

#define to_vvq(_vq) container_of_const(_vq, struct vring_virtqueue, vq)


static inline bool virtqueue_is_packed(const struct vring_virtqueue *vq)
{
	return vq->layout == VQ_LAYOUT_PACKED ||
	       vq->layout == VQ_LAYOUT_PACKED_IN_ORDER;
}

static inline bool virtqueue_is_in_order(const struct vring_virtqueue *vq)
{
	return vq->layout == VQ_LAYOUT_SPLIT_IN_ORDER ||
	       vq->layout == VQ_LAYOUT_PACKED_IN_ORDER;
}

static bool virtqueue_use_indirect(const struct vring_virtqueue *vq,
				   unsigned int total_sg)
{
	/*
	 * If the host supports indirect descriptor tables, and we have multiple
	 * buffers, then go indirect. FIXME: tune this threshold
	 */
	return (vq->indirect && total_sg > 1 && vq->vq.num_free);
}

/*
 * Modern virtio devices have feature bits to specify whether they need a
 * quirk and bypass the IOMMU. If not there, just use the DMA API.
 *
 * If there, the interaction between virtio and DMA API is messy.
 *
 * On most systems with virtio, physical addresses match bus addresses,
 * and it doesn't particularly matter whether we use the DMA API.
 *
 * On some systems, including Xen and any system with a physical device
 * that speaks virtio behind a physical IOMMU, we must use the DMA API
 * for virtio DMA to work at all.
 *
 * On other systems, including SPARC and PPC64, virtio-pci devices are
 * enumerated as though they are behind an IOMMU, but the virtio host
 * ignores the IOMMU, so we must either pretend that the IOMMU isn't
 * there or somehow map everything as the identity.
 *
 * For the time being, we preserve historic behavior and bypass the DMA
 * API.
 *
 * TODO: install a per-device DMA ops structure that does the right thing
 * taking into account all the above quirks, and use the DMA API
 * unconditionally on data path.
 */

static bool vring_use_map_api(const struct virtio_device *vdev)
{
	if (!virtio_has_dma_quirk(vdev))
		return true;

	/* Otherwise, we are left to guess. */
	/*
	 * In theory, it's possible to have a buggy QEMU-supposed
	 * emulated Q35 IOMMU and Xen enabled at the same time.  On
	 * such a configuration, virtio has never worked and will
	 * not work without an even larger kludge.  Instead, enable
	 * the DMA API if we're a Xen guest, which at least allows
	 * all of the sensible Xen configurations to work correctly.
	 */
	if (xen_domain())
		return true;

	return false;
}

static bool vring_need_unmap_buffer(const struct vring_virtqueue *vring,
				    const struct vring_desc_extra *extra)
{
	return vring->use_map_api && (extra->addr != DMA_MAPPING_ERROR);
}

size_t virtio_max_dma_size(const struct virtio_device *vdev)
{
	size_t max_segment_size = SIZE_MAX;

	if (vring_use_map_api(vdev)) {
		if (vdev->map) {
			max_segment_size =
				vdev->map->max_mapping_size(vdev->vmap);
		} else
			max_segment_size =
				dma_max_mapping_size(vdev->dev.parent);
	}

	return max_segment_size;
}
EXPORT_SYMBOL_GPL(virtio_max_dma_size);

static void *vring_alloc_queue(struct virtio_device *vdev, size_t size,
			       dma_addr_t *map_handle, gfp_t flag,
			       union virtio_map map)
{
	if (vring_use_map_api(vdev)) {
		return virtqueue_map_alloc_coherent(vdev, map, size,
						    map_handle, flag);
	} else {
		void *queue = alloc_pages_exact(PAGE_ALIGN(size), flag);

		if (queue) {
			phys_addr_t phys_addr = virt_to_phys(queue);
			*map_handle = (dma_addr_t)phys_addr;

			/*
			 * Sanity check: make sure we dind't truncate
			 * the address.  The only arches I can find that
			 * have 64-bit phys_addr_t but 32-bit dma_addr_t
			 * are certain non-highmem MIPS and x86
			 * configurations, but these configurations
			 * should never allocate physical pages above 32
			 * bits, so this is fine.  Just in case, throw a
			 * warning and abort if we end up with an
			 * unrepresentable address.
			 */
			if (WARN_ON_ONCE(*map_handle != phys_addr)) {
				free_pages_exact(queue, PAGE_ALIGN(size));
				return NULL;
			}
		}
		return queue;
	}
}

static void vring_free_queue(struct virtio_device *vdev, size_t size,
			     void *queue, dma_addr_t map_handle,
			     union virtio_map map)
{
	if (vring_use_map_api(vdev))
		virtqueue_map_free_coherent(vdev, map, size,
					    queue, map_handle);
	else
		free_pages_exact(queue, PAGE_ALIGN(size));
}

/*
 * The DMA ops on various arches are rather gnarly right now, and
 * making all of the arch DMA ops work on the vring device itself
 * is a mess.
 */
static struct device *vring_dma_dev(const struct vring_virtqueue *vq)
{
	return vq->map.dma_dev;
}

static int vring_mapping_error(const struct vring_virtqueue *vq,
			       dma_addr_t addr)
{
	struct virtio_device *vdev = vq->vq.vdev;

	if (!vq->use_map_api)
		return 0;

	if (vdev->map)
		return vdev->map->mapping_error(vq->map, addr);
	else
		return dma_mapping_error(vring_dma_dev(vq), addr);
}

/* Map one sg entry. */
static int vring_map_one_sg(const struct vring_virtqueue *vq, struct scatterlist *sg,
			    enum dma_data_direction direction, dma_addr_t *addr,
			    u32 *len, bool premapped, unsigned long attr)
{
	if (premapped) {
		*addr = sg_dma_address(sg);
		*len = sg_dma_len(sg);
		return 0;
	}

	*len = sg->length;

	if (!vq->use_map_api) {
		/*
		 * If DMA is not used, KMSAN doesn't know that the scatterlist
		 * is initialized by the hardware. Explicitly check/unpoison it
		 * depending on the direction.
		 */
		kmsan_handle_dma(sg_phys(sg), sg->length, direction);
		*addr = (dma_addr_t)sg_phys(sg);
		return 0;
	}

	/*
	 * We can't use dma_map_sg, because we don't use scatterlists in
	 * the way it expects (we don't guarantee that the scatterlist
	 * will exist for the lifetime of the mapping).
	 */
	*addr = virtqueue_map_page_attrs(&vq->vq, sg_page(sg),
					 sg->offset, sg->length,
					 direction, attr);

	if (vring_mapping_error(vq, *addr))
		return -ENOMEM;

	return 0;
}

static dma_addr_t vring_map_single(const struct vring_virtqueue *vq,
				   void *cpu_addr, size_t size,
				   enum dma_data_direction direction)
{
	if (!vq->use_map_api)
		return (dma_addr_t)virt_to_phys(cpu_addr);

	return virtqueue_map_single_attrs(&vq->vq, cpu_addr,
					  size, direction, 0);
}

static void virtqueue_init(struct vring_virtqueue *vq, u32 num)
{
	vq->vq.num_free = num;

	if (virtqueue_is_packed(vq))
		vq->last_used_idx = 0 | (1 << VRING_PACKED_EVENT_F_WRAP_CTR);
	else
		vq->last_used_idx = 0;

	vq->last_used = 0;

	vq->event_triggered = false;
	vq->num_added = 0;

#ifdef DEBUG
	vq->in_use = false;
	vq->last_add_time_valid = false;
#endif
}


/*
 * Split ring specific functions - *_split().
 */

static unsigned int vring_unmap_one_split(const struct vring_virtqueue *vq,
					  struct vring_desc_extra *extra)
{
	u16 flags;

	flags = extra->flags;

	if (flags & VRING_DESC_F_INDIRECT) {
		if (!vq->use_map_api)
			goto out;
	} else if (!vring_need_unmap_buffer(vq, extra))
		goto out;

	virtqueue_unmap_page_attrs(&vq->vq,
				   extra->addr,
				   extra->len,
				   (flags & VRING_DESC_F_WRITE) ?
				   DMA_FROM_DEVICE : DMA_TO_DEVICE,
				   0);

out:
	return extra->next;
}

static struct vring_desc *alloc_indirect_split(struct vring_virtqueue *vq,
					       unsigned int total_sg,
					       gfp_t gfp)
{
	struct vring_desc_extra *extra;
	struct vring_desc *desc;
	unsigned int i, size;

	/*
	 * We require lowmem mappings for the descriptors because
	 * otherwise virt_to_phys will give us bogus addresses in the
	 * virtqueue.
	 */
	gfp &= ~__GFP_HIGHMEM;

	size = sizeof(*desc) * total_sg + sizeof(*extra) * total_sg;

	desc = kmalloc(size, gfp);
	if (!desc)
		return NULL;

	extra = (struct vring_desc_extra *)&desc[total_sg];

	for (i = 0; i < total_sg; i++)
		extra[i].next = i + 1;

	return desc;
}

static inline unsigned int virtqueue_add_desc_split(struct vring_virtqueue *vq,
						    struct vring_desc *desc,
						    struct vring_desc_extra *extra,
						    unsigned int i,
						    dma_addr_t addr,
						    unsigned int len,
						    u16 flags, bool premapped)
{
	struct virtio_device *vdev = vq->vq.vdev;
	u16 next;

	desc[i].flags = cpu_to_virtio16(vdev, flags);
	desc[i].addr = cpu_to_virtio64(vdev, addr);
	desc[i].len = cpu_to_virtio32(vdev, len);

	extra[i].addr = premapped ? DMA_MAPPING_ERROR : addr;
	extra[i].len = len;
	extra[i].flags = flags;

	next = extra[i].next;

	desc[i].next = cpu_to_virtio16(vdev, next);

	return next;
}

static inline int virtqueue_add_split(struct vring_virtqueue *vq,
				      struct scatterlist *sgs[],
				      unsigned int total_sg,
				      unsigned int out_sgs,
				      unsigned int in_sgs,
				      void *data,
				      void *ctx,
				      bool premapped,
				      gfp_t gfp,
				      unsigned long attr)
{
	struct vring_desc_extra *extra;
	struct scatterlist *sg;
	struct vring_desc *desc;
	unsigned int i, n, avail, descs_used, err_idx, sg_count = 0;
	/* Total length for in-order */
	unsigned int total_in_len = 0;
	int head;
	bool indirect;

	START_USE(vq);

	BUG_ON(data == NULL);
	BUG_ON(ctx && vq->indirect);

	if (unlikely(vq->broken)) {
		END_USE(vq);
		return -EIO;
	}

	LAST_ADD_TIME_UPDATE(vq);

	BUG_ON(total_sg == 0);

	head = vq->free_head;

	if (virtqueue_use_indirect(vq, total_sg))
		desc = alloc_indirect_split(vq, total_sg, gfp);
	else {
		desc = NULL;
		WARN_ON_ONCE(total_sg > vq->split.vring.num && !vq->indirect);
	}

	if (desc) {
		/* Use a single buffer which doesn't continue */
		indirect = true;
		/* Set up rest to use this indirect table. */
		i = 0;
		descs_used = 1;
		extra = (struct vring_desc_extra *)&desc[total_sg];
	} else {
		indirect = false;
		desc = vq->split.vring.desc;
		extra = vq->split.desc_extra;
		i = head;
		descs_used = total_sg;
	}

	if (unlikely(vq->vq.num_free < descs_used)) {
		pr_debug("Can't add buf len %i - avail = %i\n",
			 descs_used, vq->vq.num_free);
		/* FIXME: for historical reasons, we force a notify here if
		 * there are outgoing parts to the buffer.  Presumably the
		 * host should service the ring ASAP. */
		if (out_sgs)
			vq->notify(&vq->vq);
		if (indirect)
			kfree(desc);
		END_USE(vq);
		return -ENOSPC;
	}

	for (n = 0; n < out_sgs; n++) {
		for (sg = sgs[n]; sg; sg = sg_next(sg)) {
			dma_addr_t addr;
			u32 len;
			u16 flags = 0;

			if (++sg_count != total_sg)
				flags |= VRING_DESC_F_NEXT;

			if (vring_map_one_sg(vq, sg, DMA_TO_DEVICE, &addr, &len,
					     premapped, attr))
				goto unmap_release;

			/* Note that we trust indirect descriptor
			 * table since it use stream DMA mapping.
			 */
			i = virtqueue_add_desc_split(vq, desc, extra, i, addr,
						     len, flags, premapped);
		}
	}
	for (; n < (out_sgs + in_sgs); n++) {
		for (sg = sgs[n]; sg; sg = sg_next(sg)) {
			dma_addr_t addr;
			u32 len;
			u16 flags = VRING_DESC_F_WRITE;

			if (++sg_count != total_sg)
				flags |= VRING_DESC_F_NEXT;

			if (vring_map_one_sg(vq, sg, DMA_FROM_DEVICE, &addr, &len,
					     premapped, attr))
				goto unmap_release;

			/* Note that we trust indirect descriptor
			 * table since it use stream DMA mapping.
			 */
			i = virtqueue_add_desc_split(vq, desc, extra, i, addr,
						     len, flags, premapped);
			total_in_len += len;
		}
	}

	if (indirect) {
		/* Now that the indirect table is filled in, map it. */
		dma_addr_t addr = vring_map_single(
			vq, desc, total_sg * sizeof(struct vring_desc),
			DMA_TO_DEVICE);
		if (vring_mapping_error(vq, addr))
			goto unmap_release;

		virtqueue_add_desc_split(vq, vq->split.vring.desc,
					 vq->split.desc_extra,
					 head, addr,
					 total_sg * sizeof(struct vring_desc),
					 VRING_DESC_F_INDIRECT, false);
	}

	/* We're using some buffers from the free list. */
	vq->vq.num_free -= descs_used;

	/* Update free pointer */
	if (virtqueue_is_in_order(vq)) {
		vq->free_head += descs_used;
		if (vq->free_head >= vq->split.vring.num)
			vq->free_head -= vq->split.vring.num;
		vq->split.desc_state[head].total_in_len = total_in_len;
	} else if (indirect)
		vq->free_head = vq->split.desc_extra[head].next;
	else
		vq->free_head = i;

	/* Store token and indirect buffer state. */
	vq->split.desc_state[head].data = data;
	if (indirect)
		vq->split.desc_state[head].indir_desc = desc;
	else
		vq->split.desc_state[head].indir_desc = ctx;

	/* Put entry in available array (but don't update avail->idx until they
	 * do sync). */
	avail = vq->split.avail_idx_shadow & (vq->split.vring.num - 1);
	vq->split.vring.avail->ring[avail] = cpu_to_virtio16(vq->vq.vdev, head);

	/* Descriptors and available array need to be set before we expose the
	 * new available array entries. */
	virtio_wmb(vq->weak_barriers);
	vq->split.avail_idx_shadow++;
	vq->split.vring.avail->idx = cpu_to_virtio16(vq->vq.vdev,
						vq->split.avail_idx_shadow);
	vq->num_added++;

	pr_debug("Added buffer head %i to %p\n", head, vq);
	END_USE(vq);

	/* This is very unlikely, but theoretically possible.  Kick
	 * just in case. */
	if (unlikely(vq->num_added == (1 << 16) - 1))
		virtqueue_kick(&vq->vq);

	return 0;

unmap_release:
	err_idx = i;

	if (indirect)
		i = 0;
	else
		i = head;

	for (n = 0; n < total_sg; n++) {
		if (i == err_idx)
			break;

		i = vring_unmap_one_split(vq, &extra[i]);
	}

	if (indirect)
		kfree(desc);

	END_USE(vq);
	return -ENOMEM;
}

static bool virtqueue_kick_prepare_split(struct vring_virtqueue *vq)
{
	u16 new, old;
	bool needs_kick;

	START_USE(vq);
	/* We need to expose available array entries before checking avail
	 * event. */
	virtio_mb(vq->weak_barriers);

	old = vq->split.avail_idx_shadow - vq->num_added;
	new = vq->split.avail_idx_shadow;
	vq->num_added = 0;

	LAST_ADD_TIME_CHECK(vq);
	LAST_ADD_TIME_INVALID(vq);

	if (vq->event) {
		needs_kick = vring_need_event(virtio16_to_cpu(vq->vq.vdev,
					vring_avail_event(&vq->split.vring)),
					      new, old);
	} else {
		needs_kick = !(vq->split.vring.used->flags &
					cpu_to_virtio16(vq->vq.vdev,
						VRING_USED_F_NO_NOTIFY));
	}
	END_USE(vq);
	return needs_kick;
}

static void detach_indirect_split(struct vring_virtqueue *vq,
				  unsigned int head)
{
	struct vring_desc_extra *extra = vq->split.desc_extra;
	struct vring_desc *indir_desc = vq->split.desc_state[head].indir_desc;
	unsigned int j;
	u32 len, num;

	/* Free the indirect table, if any, now that it's unmapped. */
	if (!indir_desc)
		return;
	len = vq->split.desc_extra[head].len;

	BUG_ON(!(vq->split.desc_extra[head].flags &
			VRING_DESC_F_INDIRECT));
	BUG_ON(len == 0 || len % sizeof(struct vring_desc));

	num = len / sizeof(struct vring_desc);

	extra = (struct vring_desc_extra *)&indir_desc[num];

	if (vq->use_map_api) {
		for (j = 0; j < num; j++)
			vring_unmap_one_split(vq, &extra[j]);
	}

	kfree(indir_desc);
	vq->split.desc_state[head].indir_desc = NULL;
}

static unsigned detach_buf_split_in_order(struct vring_virtqueue *vq,
					  unsigned int head,
					  void **ctx)
{
	struct vring_desc_extra *extra;
	unsigned int i;
	__virtio16 nextflag = cpu_to_virtio16(vq->vq.vdev, VRING_DESC_F_NEXT);

	/* Clear data ptr. */
	vq->split.desc_state[head].data = NULL;

	extra = vq->split.desc_extra;

	/* Put back on free list: unmap first-level descriptors and find end */
	i = head;

	while (vq->split.vring.desc[i].flags & nextflag) {
		i = vring_unmap_one_split(vq, &extra[i]);
		vq->vq.num_free++;
	}

	vring_unmap_one_split(vq, &extra[i]);

	/* Plus final descriptor */
	vq->vq.num_free++;

	if (vq->indirect)
		detach_indirect_split(vq, head);
	else if (ctx)
		*ctx = vq->split.desc_state[head].indir_desc;

	return i;
}

static void detach_buf_split(struct vring_virtqueue *vq, unsigned int head,
			     void **ctx)
{
	unsigned int i = detach_buf_split_in_order(vq, head, ctx);

	vq->split.desc_extra[i].next = vq->free_head;
	vq->free_head = head;
}

static bool virtqueue_poll_split(const struct vring_virtqueue *vq,
				 unsigned int last_used_idx)
{
	return (u16)last_used_idx != virtio16_to_cpu(vq->vq.vdev,
			vq->split.vring.used->idx);
}

static bool more_used_split(const struct vring_virtqueue *vq)
{
	return virtqueue_poll_split(vq, vq->last_used_idx);
}

static bool more_used_split_in_order(const struct vring_virtqueue *vq)
{
	if (vq->batch_last.id != UINT_MAX)
		return true;

	return virtqueue_poll_split(vq, vq->last_used_idx);
}

static void *virtqueue_get_buf_ctx_split(struct vring_virtqueue *vq,
					 unsigned int *len,
					 void **ctx)
{
	void *ret;
	unsigned int i;
	u16 last_used;

	START_USE(vq);

	if (unlikely(vq->broken)) {
		END_USE(vq);
		return NULL;
	}

	if (!more_used_split(vq)) {
		pr_debug("No more buffers in queue\n");
		END_USE(vq);
		return NULL;
	}

	/* Only get used array entries after they have been exposed by host. */
	virtio_rmb(vq->weak_barriers);

	last_used = (vq->last_used_idx & (vq->split.vring.num - 1));
	i = virtio32_to_cpu(vq->vq.vdev,
			vq->split.vring.used->ring[last_used].id);
	*len = virtio32_to_cpu(vq->vq.vdev,
			vq->split.vring.used->ring[last_used].len);

	if (unlikely(i >= vq->split.vring.num)) {
		BAD_RING(vq, "id %u out of range\n", i);
		return NULL;
	}
	if (unlikely(!vq->split.desc_state[i].data)) {
		BAD_RING(vq, "id %u is not a head!\n", i);
		return NULL;
	}

	/* detach_buf_split clears data, so grab it now. */
	ret = vq->split.desc_state[i].data;
	detach_buf_split(vq, i, ctx);
	vq->last_used_idx++;
	/* If we expect an interrupt for the next entry, tell host
	 * by writing event index and flush out the write before
	 * the read in the next get_buf call. */
	if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT))
		virtio_store_mb(vq->weak_barriers,
				&vring_used_event(&vq->split.vring),
				cpu_to_virtio16(vq->vq.vdev, vq->last_used_idx));

	LAST_ADD_TIME_INVALID(vq);

	END_USE(vq);
	return ret;
}

static void *virtqueue_get_buf_ctx_split_in_order(struct vring_virtqueue *vq,
						  unsigned int *len,
						  void **ctx)
{
	void *ret;
	unsigned int num = vq->split.vring.num;
	unsigned int num_free = vq->vq.num_free;
	u16 last_used, last_used_idx;

	START_USE(vq);

	if (unlikely(vq->broken)) {
		END_USE(vq);
		return NULL;
	}

	last_used = vq->last_used & (num - 1);
	last_used_idx = vq->last_used_idx & (num - 1);

	if (vq->batch_last.id == UINT_MAX) {
		if (!more_used_split_in_order(vq)) {
			pr_debug("No more buffers in queue\n");
			END_USE(vq);
			return NULL;
		}

		/*
		 * Only get used array entries after they have been
		 * exposed by host.
		 */
		virtio_rmb(vq->weak_barriers);

		vq->batch_last.id = virtio32_to_cpu(vq->vq.vdev,
				    vq->split.vring.used->ring[last_used_idx].id);
		vq->batch_last.len = virtio32_to_cpu(vq->vq.vdev,
				     vq->split.vring.used->ring[last_used_idx].len);
	}

	if (vq->batch_last.id == last_used) {
		vq->batch_last.id = UINT_MAX;
		*len = vq->batch_last.len;
	} else {
		*len = vq->split.desc_state[last_used].total_in_len;
	}

	if (unlikely(!vq->split.desc_state[last_used].data)) {
		BAD_RING(vq, "id %u is not a head!\n", last_used);
		return NULL;
	}

	/* detach_buf_split clears data, so grab it now. */
	ret = vq->split.desc_state[last_used].data;
	detach_buf_split_in_order(vq, last_used, ctx);

	vq->last_used_idx++;
	vq->last_used += (vq->vq.num_free - num_free);
	/* If we expect an interrupt for the next entry, tell host
	 * by writing event index and flush out the write before
	 * the read in the next get_buf call. */
	if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT))
		virtio_store_mb(vq->weak_barriers,
				&vring_used_event(&vq->split.vring),
				cpu_to_virtio16(vq->vq.vdev, vq->last_used_idx));

	LAST_ADD_TIME_INVALID(vq);

	END_USE(vq);
	return ret;
}

static void virtqueue_disable_cb_split(struct vring_virtqueue *vq)
{
	if (!(vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT)) {
		vq->split.avail_flags_shadow |= VRING_AVAIL_F_NO_INTERRUPT;

		/*
		 * If device triggered an event already it won't trigger one again:
		 * no need to disable.
		 */
		if (vq->event_triggered)
			return;

		if (vq->event)
			/* TODO: this is a hack. Figure out a cleaner value to write. */
			vring_used_event(&vq->split.vring) = 0x0;
		else
			vq->split.vring.avail->flags =
				cpu_to_virtio16(vq->vq.vdev,
						vq->split.avail_flags_shadow);
	}
}

static unsigned int virtqueue_enable_cb_prepare_split(struct vring_virtqueue *vq)
{
	u16 last_used_idx;

	START_USE(vq);

	/* We optimistically turn back on interrupts, then check if there was
	 * more to do. */
	/* Depending on the VIRTIO_RING_F_EVENT_IDX feature, we need to
	 * either clear the flags bit or point the event index at the next
	 * entry. Always do both to keep code simple. */
	if (vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT) {
		vq->split.avail_flags_shadow &= ~VRING_AVAIL_F_NO_INTERRUPT;
		if (!vq->event)
			vq->split.vring.avail->flags =
				cpu_to_virtio16(vq->vq.vdev,
						vq->split.avail_flags_shadow);
	}
	vring_used_event(&vq->split.vring) = cpu_to_virtio16(vq->vq.vdev,
			last_used_idx = vq->last_used_idx);
	END_USE(vq);
	return last_used_idx;
}

static bool virtqueue_enable_cb_delayed_split(struct vring_virtqueue *vq)
{
	u16 bufs;

	START_USE(vq);

	/* We optimistically turn back on interrupts, then check if there was
	 * more to do. */
	/* Depending on the VIRTIO_RING_F_USED_EVENT_IDX feature, we need to
	 * either clear the flags bit or point the event index at the next
	 * entry. Always update the event index to keep code simple. */
	if (vq->split.avail_flags_shadow & VRING_AVAIL_F_NO_INTERRUPT) {
		vq->split.avail_flags_shadow &= ~VRING_AVAIL_F_NO_INTERRUPT;
		if (!vq->event)
			vq->split.vring.avail->flags =
				cpu_to_virtio16(vq->vq.vdev,
						vq->split.avail_flags_shadow);
	}
	/* TODO: tune this threshold */
	bufs = (u16)(vq->split.avail_idx_shadow - vq->last_used_idx) * 3 / 4;

	virtio_store_mb(vq->weak_barriers,
			&vring_used_event(&vq->split.vring),
			cpu_to_virtio16(vq->vq.vdev, vq->last_used_idx + bufs));

	if (unlikely((u16)(virtio16_to_cpu(vq->vq.vdev, vq->split.vring.used->idx)
					- vq->last_used_idx) > bufs)) {
		END_USE(vq);
		return false;
	}

	END_USE(vq);
	return true;
}

static void *virtqueue_detach_unused_buf_split(struct vring_virtqueue *vq)
{
	unsigned int i;
	void *buf;

	START_USE(vq);

	for (i = 0; i < vq->split.vring.num; i++) {
		if (!vq->split.desc_state[i].data)
			continue;
		/* detach_buf_split clears data, so grab it now. */
		buf = vq->split.desc_state[i].data;
		if (virtqueue_is_in_order(vq))
			detach_buf_split_in_order(vq, i, NULL);
		else
			detach_buf_split(vq, i, NULL);
		vq->split.avail_idx_shadow--;
		vq->split.vring.avail->idx = cpu_to_virtio16(vq->vq.vdev,
				vq->split.avail_idx_shadow);
		END_USE(vq);
		return buf;
	}
	/* That should have freed everything. */
	BUG_ON(vq->vq.num_free != vq->split.vring.num);

	END_USE(vq);
	return NULL;
}

static void virtqueue_vring_init_split(struct vring_virtqueue_split *vring_split,
				       struct vring_virtqueue *vq)
{
	struct virtio_device *vdev;

	vdev = vq->vq.vdev;

	vring_split->avail_flags_shadow = 0;
	vring_split->avail_idx_shadow = 0;

	/* No callback?  Tell other side not to bother us. */
	if (!vq->vq.callback) {
		vring_split->avail_flags_shadow |= VRING_AVAIL_F_NO_INTERRUPT;
		if (!vq->event)
			vring_split->vring.avail->flags = cpu_to_virtio16(vdev,
					vring_split->avail_flags_shadow);
	}
}

static void virtqueue_reset_split(struct vring_virtqueue *vq)
{
	int num;

	num = vq->split.vring.num;

	vq->split.vring.avail->flags = 0;
	vq->split.vring.avail->idx = 0;

	/* reset avail event */
	vq->split.vring.avail->ring[num] = 0;

	vq->split.vring.used->flags = 0;
	vq->split.vring.used->idx = 0;

	/* reset used event */
	*(__virtio16 *)&(vq->split.vring.used->ring[num]) = 0;

	virtqueue_init(vq, num);

	virtqueue_vring_init_split(&vq->split, vq);
}

static void virtqueue_vring_attach_split(struct vring_virtqueue *vq,
					 struct vring_virtqueue_split *vring_split)
{
	vq->split = *vring_split;

	/* Put everything in free lists. */
	vq->free_head = 0;
	vq->batch_last.id = UINT_MAX;
}

static int vring_alloc_state_extra_split(struct vring_virtqueue_split *vring_split)
{
	struct vring_desc_state_split *state;
	struct vring_desc_extra *extra;
	u32 num = vring_split->vring.num;

	state = kmalloc_objs(struct vring_desc_state_split, num);
	if (!state)
		goto err_state;

	extra = vring_alloc_desc_extra(num);
	if (!extra)
		goto err_extra;

	memset(state, 0, num * sizeof(struct vring_desc_state_split));

	vring_split->desc_state = state;
	vring_split->desc_extra = extra;
	return 0;

err_extra:
	kfree(state);
err_state:
	return -ENOMEM;
}

static void vring_free_split(struct vring_virtqueue_split *vring_split,
			     struct virtio_device *vdev,
			     union virtio_map map)
{
	vring_free_queue(vdev, vring_split->queue_size_in_bytes,
			 vring_split->vring.desc,
			 vring_split->queue_dma_addr,
			 map);

	kfree(vring_split->desc_state);
	kfree(vring_split->desc_extra);
}

static int vring_alloc_queue_split(struct vring_virtqueue_split *vring_split,
				   struct virtio_device *vdev,
				   u32 num,
				   unsigned int vring_align,
				   bool may_reduce_num,
				   union virtio_map map)
{
	void *queue = NULL;
	dma_addr_t dma_addr;

	/* We assume num is a power of 2. */
	if (!is_power_of_2(num)) {
		dev_warn(&vdev->dev, "Bad virtqueue length %u\n", num);
		return -EINVAL;
	}

	/* TODO: allocate each queue chunk individually */
	for (; num && vring_size(num, vring_align) > PAGE_SIZE; num /= 2) {
		queue = vring_alloc_queue(vdev, vring_size(num, vring_align),
					  &dma_addr,
					  GFP_KERNEL | __GFP_NOWARN | __GFP_ZERO,
					  map);
		if (queue)
			break;
		if (!may_reduce_num)
			return -ENOMEM;
	}

	if (!num)
		return -ENOMEM;

	if (!queue) {
		/* Try to get a single page. You are my only hope! */
		queue = vring_alloc_queue(vdev, vring_size(num, vring_align),
					  &dma_addr, GFP_KERNEL | __GFP_ZERO,
					  map);
	}
	if (!queue)
		return -ENOMEM;

	vring_init(&vring_split->vring, num, queue, vring_align);

	vring_split->queue_dma_addr = dma_addr;
	vring_split->queue_size_in_bytes = vring_size(num, vring_align);

	vring_split->vring_align = vring_align;
	vring_split->may_reduce_num = may_reduce_num;

	return 0;
}

static const struct virtqueue_ops split_ops;

static struct virtqueue *__vring_new_virtqueue_split(unsigned int index,
					       struct vring_virtqueue_split *vring_split,
					       struct virtio_device *vdev,
					       bool weak_barriers,
					       bool context,
					       bool (*notify)(struct virtqueue *),
					       void (*callback)(struct virtqueue *),
					       const char *name,
					       union virtio_map map)
{
	struct vring_virtqueue *vq;
	int err;

	vq = kmalloc_obj(*vq);
	if (!vq)
		return NULL;

	vq->vq.callback = callback;
	vq->vq.vdev = vdev;
	vq->vq.name = name;
	vq->vq.index = index;
	vq->vq.reset = false;
	vq->we_own_ring = false;
	vq->notify = notify;
	vq->weak_barriers = weak_barriers;
#ifdef CONFIG_VIRTIO_HARDEN_NOTIFICATION
	vq->broken = true;
#else
	vq->broken = false;
#endif
	vq->map = map;
	vq->use_map_api = vring_use_map_api(vdev);

	vq->indirect = virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) &&
		!context;
	vq->event = virtio_has_feature(vdev, VIRTIO_RING_F_EVENT_IDX);
	vq->layout = virtio_has_feature(vdev, VIRTIO_F_IN_ORDER) ?
		     VQ_LAYOUT_SPLIT_IN_ORDER : VQ_LAYOUT_SPLIT;

	if (virtio_has_feature(vdev, VIRTIO_F_ORDER_PLATFORM))
		vq->weak_barriers = false;

	err = vring_alloc_state_extra_split(vring_split);
	if (err) {
		kfree(vq);
		return NULL;
	}

	virtqueue_vring_init_split(vring_split, vq);

	virtqueue_init(vq, vring_split->vring.num);
	virtqueue_vring_attach_split(vq, vring_split);

	spin_lock(&vdev->vqs_list_lock);
	list_add_tail(&vq->vq.list, &vdev->vqs);
	spin_unlock(&vdev->vqs_list_lock);
	return &vq->vq;
}

static struct virtqueue *vring_create_virtqueue_split(
	unsigned int index,
	unsigned int num,
	unsigned int vring_align,
	struct virtio_device *vdev,
	bool weak_barriers,
	bool may_reduce_num,
	bool context,
	bool (*notify)(struct virtqueue *),
	void (*callback)(struct virtqueue *),
	const char *name,
	union virtio_map map)
{
	struct vring_virtqueue_split vring_split = {};
	struct virtqueue *vq;
	int err;

	err = vring_alloc_queue_split(&vring_split, vdev, num, vring_align,
				      may_reduce_num, map);
	if (err)
		return NULL;

	vq = __vring_new_virtqueue_split(index, &vring_split, vdev, weak_barriers,
				   context, notify, callback, name, map);
	if (!vq) {
		vring_free_split(&vring_split, vdev, map);
		return NULL;
	}

	to_vvq(vq)->we_own_ring = true;

	return vq;
}

static int virtqueue_resize_split(struct vring_virtqueue *vq, u32 num)
{
	struct vring_virtqueue_split vring_split = {};
	struct virtio_device *vdev = vq->vq.vdev;
	int err;

	err = vring_alloc_queue_split(&vring_split, vdev, num,
				      vq->split.vring_align,
				      vq->split.may_reduce_num,
				      vq->map);
	if (err)
		goto err;

	err = vring_alloc_state_extra_split(&vring_split);
	if (err)
		goto err_state_extra;

	vring_free(&vq->vq);

	virtqueue_vring_init_split(&vring_split, vq);

	virtqueue_init(vq, vring_split.vring.num);
	virtqueue_vring_attach_split(vq, &vring_split);

	return 0;

err_state_extra:
	vring_free_split(&vring_split, vdev, vq->map);
err:
	virtqueue_reset_split(vq);
	return -ENOMEM;
}


/*
 * Packed ring specific functions - *_packed().
 */
static bool packed_used_wrap_counter(u16 last_used_idx)
{
	return !!(last_used_idx & (1 << VRING_PACKED_EVENT_F_WRAP_CTR));
}

static u16 packed_last_used(u16 last_used_idx)
{
	return last_used_idx & ~(-(1 << VRING_PACKED_EVENT_F_WRAP_CTR));
}

static void vring_unmap_extra_packed(const struct vring_virtqueue *vq,
				     const struct vring_desc_extra *extra)
{
	u16 flags;

	flags = extra->flags;

	if (flags & VRING_DESC_F_INDIRECT) {
		if (!vq->use_map_api)
			return;
	} else if (!vring_need_unmap_buffer(vq, extra))
		return;

	virtqueue_unmap_page_attrs(&vq->vq,
				   extra->addr, extra->len,
				   (flags & VRING_DESC_F_WRITE) ?
				   DMA_FROM_DEVICE : DMA_TO_DEVICE,
				   0);
}

static struct vring_packed_desc *alloc_indirect_packed(unsigned int total_sg,
						       gfp_t gfp)
{
	struct vring_desc_extra *extra;
	struct vring_packed_desc *desc;
	int i, size;

	/*
	 * We require lowmem mappings for the descriptors because
	 * otherwise virt_to_phys will give us bogus addresses in the
	 * virtqueue.
	 */
	gfp &= ~__GFP_HIGHMEM;

	size = (sizeof(*desc) + sizeof(*extra)) * total_sg;

	desc = kmalloc(size, gfp);
	if (!desc)
		return NULL;

	extra = (struct vring_desc_extra *)&desc[total_sg];

	for (i = 0; i < total_sg; i++)
		extra[i].next = i + 1;

	return desc;
}

static int virtqueue_add_indirect_packed(struct vring_virtqueue *vq,
					 struct scatterlist *sgs[],
					 unsigned int total_sg,
					 unsigned int out_sgs,
					 unsigned int in_sgs,
					 void *data,
					 bool premapped,
					 gfp_t gfp,
					 u16 id,
					 unsigned long attr)
{
	struct vring_desc_extra *extra;
	struct vring_packed_desc *desc;
	struct scatterlist *sg;
	unsigned int i, n, err_idx, len, total_in_len = 0;
	u16 head;
	dma_addr_t addr;

	head = vq->packed.next_avail_idx;
	desc = alloc_indirect_packed(total_sg, gfp);
	if (!desc)
		return -ENOMEM;

	extra = (struct vring_desc_extra *)&desc[total_sg];

	if (unlikely(vq->vq.num_free < 1)) {
		pr_debug("Can't add buf len 1 - avail = 0\n");
		kfree(desc);
		END_USE(vq);
		return -ENOSPC;
	}

	i = 0;

	for (n = 0; n < out_sgs + in_sgs; n++) {
		for (sg = sgs[n]; sg; sg = sg_next(sg)) {
			if (vring_map_one_sg(vq, sg, n < out_sgs ?
					     DMA_TO_DEVICE : DMA_FROM_DEVICE,
					     &addr, &len, premapped, attr))
				goto unmap_release;

			desc[i].flags = cpu_to_le16(n < out_sgs ?
						0 : VRING_DESC_F_WRITE);
			desc[i].addr = cpu_to_le64(addr);
			desc[i].len = cpu_to_le32(len);

			if (unlikely(vq->use_map_api)) {
				extra[i].addr = premapped ? DMA_MAPPING_ERROR : addr;
				extra[i].len = len;
				extra[i].flags = n < out_sgs ?  0 : VRING_DESC_F_WRITE;
			}

			if (n >= out_sgs)
				total_in_len += len;
			i++;
		}
	}

	/* Now that the indirect table is filled in, map it. */
	addr = vring_map_single(vq, desc,
			total_sg * sizeof(struct vring_packed_desc),
			DMA_TO_DEVICE);
	if (vring_mapping_error(vq, addr))
		goto unmap_release;

	vq->packed.vring.desc[head].addr = cpu_to_le64(addr);
	vq->packed.vring.desc[head].len = cpu_to_le32(total_sg *
				sizeof(struct vring_packed_desc));
	vq->packed.vring.desc[head].id = cpu_to_le16(id);

	if (vq->use_map_api) {
		vq->packed.desc_extra[id].addr = addr;
		vq->packed.desc_extra[id].len = total_sg *
				sizeof(struct vring_packed_desc);
		vq->packed.desc_extra[id].flags = VRING_DESC_F_INDIRECT |
						  vq->packed.avail_used_flags;
	}

	/*
	 * A driver MUST NOT make the first descriptor in the list
	 * available before all subsequent descriptors comprising
	 * the list are made available.
	 */
	virtio_wmb(vq->weak_barriers);
	vq->packed.vring.desc[head].flags = cpu_to_le16(VRING_DESC_F_INDIRECT |
						vq->packed.avail_used_flags);

	/* We're using some buffers from the free list. */
	vq->vq.num_free -= 1;

	/* Update free pointer */
	n = head + 1;
	if (n >= vq->packed.vring.num) {
		n = 0;
		vq->packed.avail_wrap_counter ^= 1;
		vq->packed.avail_used_flags ^=
				1 << VRING_PACKED_DESC_F_AVAIL |
				1 << VRING_PACKED_DESC_F_USED;
	}
	vq->packed.next_avail_idx = n;
	if (!virtqueue_is_in_order(vq))
		vq->free_head = vq->packed.desc_extra[id].next;

	/* Store token and indirect buffer state. */
	vq->packed.desc_state[id].num = 1;
	vq->packed.desc_state[id].data = data;
	vq->packed.desc_state[id].indir_desc = desc;
	vq->packed.desc_state[id].last = id;
	vq->packed.desc_state[id].total_in_len = total_in_len;

	vq->num_added += 1;

	pr_debug("Added buffer head %i to %p\n", head, vq);
	END_USE(vq);

	return 0;

unmap_release:
	err_idx = i;

	for (i = 0; i < err_idx; i++)
		vring_unmap_extra_packed(vq, &extra[i]);

	kfree(desc);

	END_USE(vq);
	return -ENOMEM;
}

static inline int virtqueue_add_packed(struct vring_virtqueue *vq,
				       struct scatterlist *sgs[],
				       unsigned int total_sg,
				       unsigned int out_sgs,
				       unsigned int in_sgs,
				       void *data,
				       void *ctx,
				       bool premapped,
				       gfp_t gfp,
				       unsigned long attr)
{
	struct vring_packed_desc *desc;
	struct scatterlist *sg;
	unsigned int i, n, c, descs_used, err_idx, len;
	__le16 head_flags, flags;
	u16 head, id, prev, curr, avail_used_flags;
	int err;

	START_USE(vq);

	BUG_ON(data == NULL);
	BUG_ON(ctx && vq->indirect);

	if (unlikely(vq->broken)) {
		END_USE(vq);
		return -EIO;
	}

	LAST_ADD_TIME_UPDATE(vq);

	BUG_ON(total_sg == 0);

	if (virtqueue_use_indirect(vq, total_sg)) {
		id = vq->free_head;
		BUG_ON(id == vq->packed.vring.num);
		err = virtqueue_add_indirect_packed(vq, sgs, total_sg, out_sgs,
						    in_sgs, data, premapped, gfp,
						    id, attr);
		if (err != -ENOMEM) {
			END_USE(vq);
			return err;
		}

		/* fall back on direct */
	}

	head = vq->packed.next_avail_idx;
	avail_used_flags = vq->packed.avail_used_flags;

	WARN_ON_ONCE(total_sg > vq->packed.vring.num && !vq->indirect);

	desc = vq->packed.vring.desc;
	i = head;
	descs_used = total_sg;

	if (unlikely(vq->vq.num_free < descs_used)) {
		pr_debug("Can't add buf len %i - avail = %i\n",
			 descs_used, vq->vq.num_free);
		END_USE(vq);
		return -ENOSPC;
	}

	id = vq->free_head;
	BUG_ON(id == vq->packed.vring.num);

	curr = id;
	c = 0;
	for (n = 0; n < out_sgs + in_sgs; n++) {
		for (sg = sgs[n]; sg; sg = sg_next(sg)) {
			dma_addr_t addr;

			if (vring_map_one_sg(vq, sg, n < out_sgs ?
					     DMA_TO_DEVICE : DMA_FROM_DEVICE,
					     &addr, &len, premapped, attr))
				goto unmap_release;

			flags = cpu_to_le16(vq->packed.avail_used_flags |
				    (++c == total_sg ? 0 : VRING_DESC_F_NEXT) |
				    (n < out_sgs ? 0 : VRING_DESC_F_WRITE));
			if (i == head)
				head_flags = flags;
			else
				desc[i].flags = flags;

			desc[i].addr = cpu_to_le64(addr);
			desc[i].len = cpu_to_le32(len);
			desc[i].id = cpu_to_le16(id);

			if (unlikely(vq->use_map_api)) {
				vq->packed.desc_extra[curr].addr = premapped ?
					DMA_MAPPING_ERROR : addr;
				vq->packed.desc_extra[curr].len = len;
				vq->packed.desc_extra[curr].flags =
					le16_to_cpu(flags);
			}
			prev = curr;
			curr = vq->packed.desc_extra[curr].next;

			if ((unlikely(++i >= vq->packed.vring.num))) {
				i = 0;
				vq->packed.avail_used_flags ^=
					1 << VRING_PACKED_DESC_F_AVAIL |
					1 << VRING_PACKED_DESC_F_USED;
			}
		}
	}

	if (i <= head)
		vq->packed.avail_wrap_counter ^= 1;

	/* We're using some buffers from the free list. */
	vq->vq.num_free -= descs_used;

	/* Update free pointer */
	vq->packed.next_avail_idx = i;
	vq->free_head = curr;

	/* Store token. */
	vq->packed.desc_state[id].num = descs_used;
	vq->packed.desc_state[id].data = data;
	vq->packed.desc_state[id].indir_desc = ctx;
	vq->packed.desc_state[id].last = prev;

	/*
	 * A driver MUST NOT make the first descriptor in the list
	 * available before all subsequent descriptors comprising
	 * the list are made available.
	 */
	virtio_wmb(vq->weak_barriers);
	vq->packed.vring.desc[head].flags = head_flags;
	vq->num_added += descs_used;

	pr_debug("Added buffer head %i to %p\n", head, vq);
	END_USE(vq);

	return 0;

unmap_release:
	err_idx = i;
	i = head;
	curr = vq->free_head;

	vq->packed.avail_used_flags = avail_used_flags;

	for (n = 0; n < total_sg; n++) {
		if (i == err_idx)
			break;
		vring_unmap_extra_packed(vq, &vq->packed.desc_extra[curr]);
		curr = vq->packed.desc_extra[curr].next;
		i++;
		if (i >= vq->packed.vring.num)
			i = 0;
	}

	END_USE(vq);
	return -EIO;
}

static inline int virtqueue_add_packed_in_order(struct vring_virtqueue *vq,
						struct scatterlist *sgs[],
						unsigned int total_sg,
						unsigned int out_sgs,
						unsigned int in_sgs,
						void *data,
						void *ctx,
						bool premapped,
						gfp_t gfp,
						unsigned long attr)
{
	struct vring_packed_desc *desc;
	struct scatterlist *sg;
	unsigned int i, n, sg_count, err_idx, total_in_len = 0;
	__le16 head_flags, flags;
	u16 head, avail_used_flags;
	bool avail_wrap_counter;
	int err;

	START_USE(vq);

	BUG_ON(data == NULL);
	BUG_ON(ctx && vq->indirect);

	if (unlikely(vq->broken)) {
		END_USE(vq);
		return -EIO;
	}

	LAST_ADD_TIME_UPDATE(vq);

	BUG_ON(total_sg == 0);

	if (virtqueue_use_indirect(vq, total_sg)) {
		err = virtqueue_add_indirect_packed(vq, sgs, total_sg, out_sgs,
						    in_sgs, data, premapped, gfp,
						    vq->packed.next_avail_idx,
						    attr);
		if (err != -ENOMEM) {
			END_USE(vq);
			return err;
		}

		/* fall back on direct */
	}

	head = vq->packed.next_avail_idx;
	avail_used_flags = vq->packed.avail_used_flags;
	avail_wrap_counter = vq->packed.avail_wrap_counter;

	WARN_ON_ONCE(total_sg > vq->packed.vring.num && !vq->indirect);

	desc = vq->packed.vring.desc;
	i = head;

	if (unlikely(vq->vq.num_free < total_sg)) {
		pr_debug("Can't add buf len %i - avail = %i\n",
			 total_sg, vq->vq.num_free);
		END_USE(vq);
		return -ENOSPC;
	}

	sg_count = 0;
	for (n = 0; n < out_sgs + in_sgs; n++) {
		for (sg = sgs[n]; sg; sg = sg_next(sg)) {
			dma_addr_t addr;
			u32 len;

			flags = 0;
			if (++sg_count != total_sg)
				flags |= cpu_to_le16(VRING_DESC_F_NEXT);
			if (n >= out_sgs)
				flags |= cpu_to_le16(VRING_DESC_F_WRITE);

			if (vring_map_one_sg(vq, sg, n < out_sgs ?
					     DMA_TO_DEVICE : DMA_FROM_DEVICE,
					     &addr, &len, premapped, attr))
				goto unmap_release;

			flags |= cpu_to_le16(vq->packed.avail_used_flags);

			if (i == head)
				head_flags = flags;
			else
				desc[i].flags = flags;

			desc[i].addr = cpu_to_le64(addr);
			desc[i].len = cpu_to_le32(len);
			desc[i].id = cpu_to_le16(head);

			if (unlikely(vq->use_map_api)) {
				vq->packed.desc_extra[i].addr = premapped ?
				      DMA_MAPPING_ERROR : addr;
				vq->packed.desc_extra[i].len = len;
				vq->packed.desc_extra[i].flags =
					le16_to_cpu(flags);
			}

			if ((unlikely(++i >= vq->packed.vring.num))) {
				i = 0;
				vq->packed.avail_used_flags ^=
					1 << VRING_PACKED_DESC_F_AVAIL |
					1 << VRING_PACKED_DESC_F_USED;
				vq->packed.avail_wrap_counter ^= 1;
			}

			if (n >= out_sgs)
				total_in_len += len;
		}
	}

	/* We're using some buffers from the free list. */
	vq->vq.num_free -= total_sg;

	/* Update free pointer */
	vq->packed.next_avail_idx = i;

	/* Store token. */
	vq->packed.desc_state[head].num = total_sg;
	vq->packed.desc_state[head].data = data;
	vq->packed.desc_state[head].indir_desc = ctx;
	vq->packed.desc_state[head].total_in_len = total_in_len;

	/*
	 * A driver MUST NOT make the first descriptor in the list
	 * available before all subsequent descriptors comprising
	 * the list are made available.
	 */
	virtio_wmb(vq->weak_barriers);
	vq->packed.vring.desc[head].flags = head_flags;
	vq->num_added += total_sg;

	pr_debug("Added buffer head %i to %p\n", head, vq);
	END_USE(vq);

	return 0;

unmap_release:
	err_idx = i;
	i = head;
	vq->packed.avail_used_flags = avail_used_flags;
	vq->packed.avail_wrap_counter = avail_wrap_counter;

	for (n = 0; n < total_sg; n++) {
		if (i == err_idx)
			break;
		vring_unmap_extra_packed(vq, &vq->packed.desc_extra[i]);
		i++;
		if (i >= vq->packed.vring.num)
			i = 0;
	}

	END_USE(vq);
	return -EIO;
}

static bool virtqueue_kick_prepare_packed(struct vring_virtqueue *vq)
{
	u16 new, old, off_wrap, flags, wrap_counter, event_idx;
	bool needs_kick;
	union {
		struct {
			__le16 off_wrap;
			__le16 flags;
		};
		u32 u32;
	} snapshot;

	START_USE(vq);

	/*
	 * We need to expose the new flags value before checking notification
	 * suppressions.
	 */
	virtio_mb(vq->weak_barriers);

	old = vq->packed.next_avail_idx - vq->num_added;
	new = vq->packed.next_avail_idx;
	vq->num_added = 0;

	snapshot.u32 = *(u32 *)vq->packed.vring.device;
	flags = le16_to_cpu(snapshot.flags);

	LAST_ADD_TIME_CHECK(vq);
	LAST_ADD_TIME_INVALID(vq);

	if (flags != VRING_PACKED_EVENT_FLAG_DESC) {
		needs_kick = (flags != VRING_PACKED_EVENT_FLAG_DISABLE);
		goto out;
	}

	off_wrap = le16_to_cpu(snapshot.off_wrap);

	wrap_counter = off_wrap >> VRING_PACKED_EVENT_F_WRAP_CTR;
	event_idx = off_wrap & ~(1 << VRING_PACKED_EVENT_F_WRAP_CTR);
	if (wrap_counter != vq->packed.avail_wrap_counter)
		event_idx -= vq->packed.vring.num;

	needs_kick = vring_need_event(event_idx, new, old);
out:
	END_USE(vq);
	return needs_kick;
}

static void detach_buf_packed_in_order(struct vring_virtqueue *vq,
				       unsigned int id, void **ctx)
{
	struct vring_desc_state_packed *state = NULL;
	struct vring_packed_desc *desc;
	unsigned int i, curr;

	state = &vq->packed.desc_state[id];

	/* Clear data ptr. */
	state->data = NULL;

	vq->vq.num_free += state->num;

	if (unlikely(vq->use_map_api)) {
		curr = id;
		for (i = 0; i < state->num; i++) {
			vring_unmap_extra_packed(vq,
						 &vq->packed.desc_extra[curr]);
			curr = vq->packed.desc_extra[curr].next;
		}
	}

	if (vq->indirect) {
		struct vring_desc_extra *extra;
		u32 len, num;

		/* Free the indirect table, if any, now that it's unmapped. */
		desc = state->indir_desc;
		if (!desc)
			return;

		if (vq->use_map_api) {
			len = vq->packed.desc_extra[id].len;
			num = len / sizeof(struct vring_packed_desc);

			extra = (struct vring_desc_extra *)&desc[num];

			for (i = 0; i < num; i++)
				vring_unmap_extra_packed(vq, &extra[i]);
		}
		kfree(desc);
		state->indir_desc = NULL;
	} else if (ctx) {
		*ctx = state->indir_desc;
	}
}

static void detach_buf_packed(struct vring_virtqueue *vq,
			      unsigned int id, void **ctx)
{
	struct vring_desc_state_packed *state = &vq->packed.desc_state[id];

	vq->packed.desc_extra[state->last].next = vq->free_head;
	vq->free_head = id;

	detach_buf_packed_in_order(vq, id, ctx);
}

static inline bool is_used_desc_packed(const struct vring_virtqueue *vq,
				       u16 idx, bool used_wrap_counter)
{
	bool avail, used;
	u16 flags;

	flags = le16_to_cpu(vq->packed.vring.desc[idx].flags);
	avail = !!(flags & (1 << VRING_PACKED_DESC_F_AVAIL));
	used = !!(flags & (1 << VRING_PACKED_DESC_F_USED));

	return avail == used && used == used_wrap_counter;
}

static bool virtqueue_poll_packed(const struct vring_virtqueue *vq,
				  unsigned int off_wrap)
{
	bool wrap_counter;
	u16 used_idx;

	wrap_counter = off_wrap >> VRING_PACKED_EVENT_F_WRAP_CTR;
	used_idx = off_wrap & ~(1 << VRING_PACKED_EVENT_F_WRAP_CTR);

	return is_used_desc_packed(vq, used_idx, wrap_counter);
}

static bool more_used_packed(const struct vring_virtqueue *vq)
{
	return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx));
}

static void update_last_used_idx_packed(struct vring_virtqueue *vq,
					u16 id, u16 last_used,
					u16 used_wrap_counter)
{
	last_used += vq->packed.desc_state[id].num;
	if (unlikely(last_used >= vq->packed.vring.num)) {
		last_used -= vq->packed.vring.num;
		used_wrap_counter ^= 1;
	}

	last_used = (last_used | (used_wrap_counter << VRING_PACKED_EVENT_F_WRAP_CTR));
	WRITE_ONCE(vq->last_used_idx, last_used);

	/*
	 * If we expect an interrupt for the next entry, tell host
	 * by writing event index and flush out the write before
	 * the read in the next get_buf call.
	 */
	if (vq->packed.event_flags_shadow == VRING_PACKED_EVENT_FLAG_DESC)
		virtio_store_mb(vq->weak_barriers,
				&vq->packed.vring.driver->off_wrap,
				cpu_to_le16(vq->last_used_idx));
}

static bool more_used_packed_in_order(const struct vring_virtqueue *vq)
{
	if (vq->batch_last.id != UINT_MAX)
		return true;

	return virtqueue_poll_packed(vq, READ_ONCE(vq->last_used_idx));
}

static void *virtqueue_get_buf_ctx_packed_in_order(struct vring_virtqueue *vq,
						   unsigned int *len,
						   void **ctx)
{
	unsigned int num = vq->packed.vring.num;
	u16 last_used, last_used_idx;
	bool used_wrap_counter;
	void *ret;

	START_USE(vq);

	if (unlikely(vq->broken)) {
		END_USE(vq);
		return NULL;
	}

	last_used_idx = vq->last_used_idx;
	used_wrap_counter = packed_used_wrap_counter(last_used_idx);
	last_used = packed_last_used(last_used_idx);

	if (vq->batch_last.id == UINT_MAX) {
		if (!more_used_packed_in_order(vq)) {
			pr_debug("No more buffers in queue\n");
			END_USE(vq);
			return NULL;
		}
		/* Only get used elements after they have been exposed by host. */
		virtio_rmb(vq->weak_barriers);
		vq->batch_last.id =
			le16_to_cpu(vq->packed.vring.desc[last_used].id);
		vq->batch_last.len =
			le32_to_cpu(vq->packed.vring.desc[last_used].len);
	}

	if (vq->batch_last.id == last_used) {
		vq->batch_last.id = UINT_MAX;
		*len = vq->batch_last.len;
	} else {
		*len = vq->packed.desc_state[last_used].total_in_len;
	}

	if (unlikely(last_used >= num)) {
		BAD_RING(vq, "id %u out of range\n", last_used);
		return NULL;
	}
	if (unlikely(!vq->packed.desc_state[last_used].data)) {
		BAD_RING(vq, "id %u is not a head!\n", last_used);
		return NULL;
	}

	/* detach_buf_packed clears data, so grab it now. */
	ret = vq->packed.desc_state[last_used].data;
	detach_buf_packed_in_order(vq, last_used, ctx);

	update_last_used_idx_packed(vq, last_used, last_used,
				    used_wrap_counter);

	LAST_ADD_TIME_INVALID(vq);

	END_USE(vq);
	return ret;
}

static void *virtqueue_get_buf_ctx_packed(struct vring_virtqueue *vq,
					  unsigned int *len,
					  void **ctx)
{
	unsigned int num = vq->packed.vring.num;
	u16 last_used, id, last_used_idx;
	bool used_wrap_counter;
	void *ret;

	START_USE(vq);

	if (unlikely(vq->broken)) {
		END_USE(vq);
		return NULL;
	}

	if (!more_used_packed(vq)) {
		pr_debug("No more buffers in queue\n");
		END_USE(vq);
		return NULL;
	}

	/* Only get used elements after they have been exposed by host. */
	virtio_rmb(vq->weak_barriers);

	last_used_idx = READ_ONCE(vq->last_used_idx);
	used_wrap_counter = packed_used_wrap_counter(last_used_idx);
	last_used = packed_last_used(last_used_idx);
	id = le16_to_cpu(vq->packed.vring.desc[last_used].id);
	*len = le32_to_cpu(vq->packed.vring.desc[last_used].len);

	if (unlikely(id >= num)) {
		BAD_RING(vq, "id %u out of range\n", id);
		return NULL;
	}
	if (unlikely(!vq->packed.desc_state[id].data)) {
		BAD_RING(vq, "id %u is not a head!\n", id);
		return NULL;
	}

	/* detach_buf_packed clears data, so grab it now. */
	ret = vq->packed.desc_state[id].data;
	detach_buf_packed(vq, id, ctx);

	update_last_used_idx_packed(vq, id, last_used, used_wrap_counter);

	LAST_ADD_TIME_INVALID(vq);

	END_USE(vq);
	return ret;
}

static void virtqueue_disable_cb_packed(struct vring_virtqueue *vq)
{
	if (vq->packed.event_flags_shadow != VRING_PACKED_EVENT_FLAG_DISABLE) {
		vq->packed.event_flags_shadow = VRING_PACKED_EVENT_FLAG_DISABLE;

		/*
		 * If device triggered an event already it won't trigger one again:
		 * no need to disable.
		 */
		if (vq->event_triggered)
			return;

		vq->packed.vring.driver->flags =
			cpu_to_le16(vq->packed.event_flags_shadow);
	}
}

static unsigned int virtqueue_enable_cb_prepare_packed(struct vring_virtqueue *vq)
{
	START_USE(vq);

	/*
	 * We optimistically turn back on interrupts, then check if there was
	 * more to do.
	 */

	if (vq->event) {
		vq->packed.vring.driver->off_wrap =
			cpu_to_le16(vq->last_used_idx);
		/*
		 * We need to update event offset and event wrap
		 * counter first before updating event flags.
		 */
		virtio_wmb(vq->weak_barriers);
	}

	if (vq->packed.event_flags_shadow == VRING_PACKED_EVENT_FLAG_DISABLE) {
		vq->packed.event_flags_shadow = vq->event ?
				VRING_PACKED_EVENT_FLAG_DESC :
				VRING_PACKED_EVENT_FLAG_ENABLE;
		vq->packed.vring.driver->flags =
				cpu_to_le16(vq->packed.event_flags_shadow);
	}

	END_USE(vq);
	return vq->last_used_idx;
}

static bool virtqueue_enable_cb_delayed_packed(struct vring_virtqueue *vq)
{
	u16 used_idx, wrap_counter, last_used_idx;
	u16 bufs;

	START_USE(vq);

	/*
	 * We optimistically turn back on interrupts, then check if there was
	 * more to do.
	 */

	if (vq->event) {
		/* TODO: tune this threshold */
		bufs = (vq->packed.vring.num - vq->vq.num_free) * 3 / 4;
		last_used_idx = READ_ONCE(vq->last_used_idx);
		wrap_counter = packed_used_wrap_counter(last_used_idx);

		used_idx = packed_last_used(last_used_idx) + bufs;
		if (used_idx >= vq->packed.vring.num) {
			used_idx -= vq->packed.vring.num;
			wrap_counter ^= 1;
		}

		vq->packed.vring.driver->off_wrap = cpu_to_le16(used_idx |
			(wrap_counter << VRING_PACKED_EVENT_F_WRAP_CTR));

		/*
		 * We need to update event offset and event wrap
		 * counter first before updating event flags.
		 */
		virtio_wmb(vq->weak_barriers);
	}

	if (vq->packed.event_flags_shadow == VRING_PACKED_EVENT_FLAG_DISABLE) {
		vq->packed.event_flags_shadow = vq->event ?
				VRING_PACKED_EVENT_FLAG_DESC :
				VRING_PACKED_EVENT_FLAG_ENABLE;
		vq->packed.vring.driver->flags =
				cpu_to_le16(vq->packed.event_flags_shadow);
	}

	/*
	 * We need to update event suppression structure first
	 * before re-checking for more used buffers.
	 */
	virtio_mb(vq->weak_barriers);

	last_used_idx = READ_ONCE(vq->last_used_idx);
	wrap_counter = packed_used_wrap_counter(last_used_idx);
	used_idx = packed_last_used(last_used_idx);
	if (is_used_desc_packed(vq, used_idx, wrap_counter)) {
		END_USE(vq);
		return false;
	}

	END_USE(vq);
	return true;
}

static void *virtqueue_detach_unused_buf_packed(struct vring_virtqueue *vq)
{
	unsigned int i;
	void *buf;

	START_USE(vq);

	for (i = 0; i < vq->packed.vring.num; i++) {
		if (!vq->packed.desc_state[i].data)
			continue;
		/* detach_buf clears data, so grab it now. */
		buf = vq->packed.desc_state[i].data;
		if (virtqueue_is_in_order(vq))
			detach_buf_packed_in_order(vq, i, NULL);
		else
			detach_buf_packed(vq, i, NULL);
		END_USE(vq);
		return buf;
	}
	/* That should have freed everything. */
	BUG_ON(vq->vq.num_free != vq->packed.vring.num);

	END_USE(vq);
	return NULL;
}

static struct vring_desc_extra *vring_alloc_desc_extra(unsigned int num)
{
	struct vring_desc_extra *desc_extra;
	unsigned int i;

	desc_extra = kmalloc_objs(struct vring_desc_extra, num);
	if (!desc_extra)
		return NULL;

	memset(desc_extra, 0, num * sizeof(struct vring_desc_extra));

	for (i = 0; i < num - 1; i++)
		desc_extra[i].next = i + 1;

	desc_extra[num - 1].next = 0;

	return desc_extra;
}

static void vring_free_packed(struct vring_virtqueue_packed *vring_packed,
			      struct virtio_device *vdev,
			      union virtio_map map)
{
	if (vring_packed->vring.desc)
		vring_free_queue(vdev, vring_packed->ring_size_in_bytes,
				 vring_packed->vring.desc,
				 vring_packed->ring_dma_addr,
				 map);

	if (vring_packed->vring.driver)
		vring_free_queue(vdev, vring_packed->event_size_in_bytes,
				 vring_packed->vring.driver,
				 vring_packed->driver_event_dma_addr,
				 map);

	if (vring_packed->vring.device)
		vring_free_queue(vdev, vring_packed->event_size_in_bytes,
				 vring_packed->vring.device,
				 vring_packed->device_event_dma_addr,
				 map);

	kfree(vring_packed->desc_state);
	kfree(vring_packed->desc_extra);
}

static int vring_alloc_queue_packed(struct vring_virtqueue_packed *vring_packed,
				    struct virtio_device *vdev,
				    u32 num, union virtio_map map)
{
	struct vring_packed_desc *ring;
	struct vring_packed_desc_event *driver, *device;
	dma_addr_t ring_dma_addr, driver_event_dma_addr, device_event_dma_addr;
	size_t ring_size_in_bytes, event_size_in_bytes;

	ring_size_in_bytes = num * sizeof(struct vring_packed_desc);

	ring = vring_alloc_queue(vdev, ring_size_in_bytes,
				 &ring_dma_addr,
				 GFP_KERNEL | __GFP_NOWARN | __GFP_ZERO,
				 map);
	if (!ring)
		goto err;

	vring_packed->vring.desc         = ring;
	vring_packed->ring_dma_addr      = ring_dma_addr;
	vring_packed->ring_size_in_bytes = ring_size_in_bytes;

	event_size_in_bytes = sizeof(struct vring_packed_desc_event);

	driver = vring_alloc_queue(vdev, event_size_in_bytes,
				   &driver_event_dma_addr,
				   GFP_KERNEL | __GFP_NOWARN | __GFP_ZERO,
				   map);
	if (!driver)
		goto err;

	vring_packed->vring.driver          = driver;
	vring_packed->event_size_in_bytes   = event_size_in_bytes;
	vring_packed->driver_event_dma_addr = driver_event_dma_addr;

	device = vring_alloc_queue(vdev, event_size_in_bytes,
				   &device_event_dma_addr,
				   GFP_KERNEL | __GFP_NOWARN | __GFP_ZERO,
				   map);
	if (!device)
		goto err;

	vring_packed->vring.device          = device;
	vring_packed->device_event_dma_addr = device_event_dma_addr;

	vring_packed->vring.num = num;

	return 0;

err:
	vring_free_packed(vring_packed, vdev, map);
	return -ENOMEM;
}

static int vring_alloc_state_extra_packed(struct vring_virtqueue_packed *vring_packed)
{
	struct vring_desc_state_packed *state;
	struct vring_desc_extra *extra;
	u32 num = vring_packed->vring.num;

	state = kmalloc_objs(struct vring_desc_state_packed, num);
	if (!state)
		goto err_desc_state;

	memset(state, 0, num * sizeof(struct vring_desc_state_packed));

	extra = vring_alloc_desc_extra(num);
	if (!extra)
		goto err_desc_extra;

	vring_packed->desc_state = state;
	vring_packed->desc_extra = extra;

	return 0;

err_desc_extra:
	kfree(state);
err_desc_state:
	return -ENOMEM;
}

static void virtqueue_vring_init_packed(struct vring_virtqueue_packed *vring_packed,
					bool callback)
{
	vring_packed->next_avail_idx = 0;
	vring_packed->avail_wrap_counter = 1;
	vring_packed->event_flags_shadow = 0;
	vring_packed->avail_used_flags = 1 << VRING_PACKED_DESC_F_AVAIL;

	/* No callback?  Tell other side not to bother us. */
	if (!callback) {
		vring_packed->event_flags_shadow = VRING_PACKED_EVENT_FLAG_DISABLE;
		vring_packed->vring.driver->flags =
			cpu_to_le16(vring_packed->event_flags_shadow);
	}
}

static void virtqueue_vring_attach_packed(struct vring_virtqueue *vq,
					  struct vring_virtqueue_packed *vring_packed)
{
	vq->packed = *vring_packed;

	if (virtqueue_is_in_order(vq)) {
		vq->batch_last.id = UINT_MAX;
	} else {
		/*
		 * Put everything in free lists. Note that
		 * next_avail_idx is sufficient with IN_ORDER so
		 * free_head is unused.
		 */
		vq->free_head = 0;
	}
}
static void virtqueue_reset_packed(struct vring_virtqueue *vq)
{
	memset(vq->packed.vring.device, 0, vq->packed.event_size_in_bytes);
	memset(vq->packed.vring.driver, 0, vq->packed.event_size_in_bytes);

	/* we need to reset the desc.flags. For more, see is_used_desc_packed() */
	memset(vq->packed.vring.desc, 0, vq->packed.ring_size_in_bytes);
	virtqueue_init(vq, vq->packed.vring.num);
	virtqueue_vring_init_packed(&vq->packed, !!vq->vq.callback);
}

static const struct virtqueue_ops packed_ops;

static struct virtqueue *__vring_new_virtqueue_packed(unsigned int index,
					       struct vring_virtqueue_packed *vring_packed,
					       struct virtio_device *vdev,
					       bool weak_barriers,
					       bool context,
					       bool (*notify)(struct virtqueue *),
					       void (*callback)(struct virtqueue *),
					       const char *name,
					       union virtio_map map)
{
	struct vring_virtqueue *vq;
	int err;

	vq = kmalloc_obj(*vq);
	if (!vq)
		return NULL;

	vq->vq.callback = callback;
	vq->vq.vdev = vdev;
	vq->vq.name = name;
	vq->vq.index = index;
	vq->vq.reset = false;
	vq->we_own_ring = false;
	vq->notify = notify;
	vq->weak_barriers = weak_barriers;
#ifdef CONFIG_VIRTIO_HARDEN_NOTIFICATION
	vq->broken = true;
#else
	vq->broken = false;
#endif
	vq->map = map;
	vq->use_map_api = vring_use_map_api(vdev);

	vq->indirect = virtio_has_feature(vdev, VIRTIO_RING_F_INDIRECT_DESC) &&
		!context;
	vq->event = virtio_has_feature(vdev, VIRTIO_RING_F_EVENT_IDX);
	vq->layout = virtio_has_feature(vdev, VIRTIO_F_IN_ORDER) ?
		     VQ_LAYOUT_PACKED_IN_ORDER : VQ_LAYOUT_PACKED;

	if (virtio_has_feature(vdev, VIRTIO_F_ORDER_PLATFORM))
		vq->weak_barriers = false;

	err = vring_alloc_state_extra_packed(vring_packed);
	if (err) {
		kfree(vq);
		return NULL;
	}

	virtqueue_vring_init_packed(vring_packed, !!callback);

	virtqueue_init(vq, vring_packed->vring.num);
	virtqueue_vring_attach_packed(vq, vring_packed);

	spin_lock(&vdev->vqs_list_lock);
	list_add_tail(&vq->vq.list, &vdev->vqs);
	spin_unlock(&vdev->vqs_list_lock);
	return &vq->vq;
}

static struct virtqueue *vring_create_virtqueue_packed(
	unsigned int index,
	unsigned int num,
	unsigned int vring_align,
	struct virtio_device *vdev,
	bool weak_barriers,
	bool may_reduce_num,
	bool context,
	bool (*notify)(struct virtqueue *),
	void (*callback)(struct virtqueue *),
	const char *name,
	union virtio_map map)
{
	struct vring_virtqueue_packed vring_packed = {};
	struct virtqueue *vq;

	if (vring_alloc_queue_packed(&vring_packed, vdev, num, map))
		return NULL;

	vq = __vring_new_virtqueue_packed(index, &vring_packed, vdev, weak_barriers,
					context, notify, callback, name, map);
	if (!vq) {
		vring_free_packed(&vring_packed, vdev, map);
		return NULL;
	}

	to_vvq(vq)->we_own_ring = true;

	return vq;
}

static int virtqueue_resize_packed(struct vring_virtqueue *vq, u32 num)
{
	struct vring_virtqueue_packed vring_packed = {};
	struct virtio_device *vdev = vq->vq.vdev;
	int err;

	if (vring_alloc_queue_packed(&vring_packed, vdev, num, vq->map))
		goto err_ring;

	err = vring_alloc_state_extra_packed(&vring_packed);
	if (err)
		goto err_state_extra;

	vring_free(&vq->vq);

	virtqueue_vring_init_packed(&vring_packed, !!vq->vq.callback);

	virtqueue_init(vq, vring_packed.vring.num);
	virtqueue_vring_attach_packed(vq, &vring_packed);

	return 0;

err_state_extra:
	vring_free_packed(&vring_packed, vdev, vq->map);
err_ring:
	virtqueue_reset_packed(vq);
	return -ENOMEM;
}

static const struct virtqueue_ops split_ops = {
	.add = virtqueue_add_split,
	.get = virtqueue_get_buf_ctx_split,
	.kick_prepare = virtqueue_kick_prepare_split,
	.disable_cb = virtqueue_disable_cb_split,
	.enable_cb_delayed = virtqueue_enable_cb_delayed_split,
	.enable_cb_prepare = virtqueue_enable_cb_prepare_split,
	.poll = virtqueue_poll_split,
	.detach_unused_buf = virtqueue_detach_unused_buf_split,
	.more_used = more_used_split,
	.resize = virtqueue_resize_split,
	.reset = virtqueue_reset_split,
};

static const struct virtqueue_ops packed_ops = {
	.add = virtqueue_add_packed,
	.get = virtqueue_get_buf_ctx_packed,
	.kick_prepare = virtqueue_kick_prepare_packed,
	.disable_cb = virtqueue_disable_cb_packed,
	.enable_cb_delayed = virtqueue_enable_cb_delayed_packed,
	.enable_cb_prepare = virtqueue_enable_cb_prepare_packed,
	.poll = virtqueue_poll_packed,
	.detach_unused_buf = virtqueue_detach_unused_buf_packed,
	.more_used = more_used_packed,
	.resize = virtqueue_resize_packed,
	.reset = virtqueue_reset_packed,
};

static const struct virtqueue_ops split_in_order_ops = {
	.add = virtqueue_add_split,
	.get = virtqueue_get_buf_ctx_split_in_order,
	.kick_prepare = virtqueue_kick_prepare_split,
	.disable_cb = virtqueue_disable_cb_split,
	.enable_cb_delayed = virtqueue_enable_cb_delayed_split,
	.enable_cb_prepare = virtqueue_enable_cb_prepare_split,
	.poll = virtqueue_poll_split,
	.detach_unused_buf = virtqueue_detach_unused_buf_split,
	.more_used = more_used_split_in_order,
	.resize = virtqueue_resize_split,
	.reset = virtqueue_reset_split,
};

static const struct virtqueue_ops packed_in_order_ops = {
	.add = virtqueue_add_packed_in_order,
	.get = virtqueue_get_buf_ctx_packed_in_order,
	.kick_prepare = virtqueue_kick_prepare_packed,
	.disable_cb = virtqueue_disable_cb_packed,
	.enable_cb_delayed = virtqueue_enable_cb_delayed_packed,
	.enable_cb_prepare = virtqueue_enable_cb_prepare_packed,
	.poll = virtqueue_poll_packed,
	.detach_unused_buf = virtqueue_detach_unused_buf_packed,
	.more_used = more_used_packed_in_order,
	.resize = virtqueue_resize_packed,
	.reset = virtqueue_reset_packed,
};

static int virtqueue_disable_and_recycle(struct virtqueue *_vq,
					 void (*recycle)(struct virtqueue *vq, void *buf))
{
	struct vring_virtqueue *vq = to_vvq(_vq);
	struct virtio_device *vdev = vq->vq.vdev;
	void *buf;
	int err;

	if (!vq->we_own_ring)
		return -EPERM;

	if (!vdev->config->disable_vq_and_reset)
		return -ENOENT;

	if (!vdev->config->enable_vq_after_reset)
		return -ENOENT;

	err = vdev->config->disable_vq_and_reset(_vq);
	if (err)
		return err;

	while ((buf = virtqueue_detach_unused_buf(_vq)) != NULL)
		recycle(_vq, buf);

	return 0;
}

static int virtqueue_enable_after_reset(struct virtqueue *_vq)
{
	struct vring_virtqueue *vq = to_vvq(_vq);
	struct virtio_device *vdev = vq->vq.vdev;

	if (vdev->config->enable_vq_after_reset(_vq))
		return -EBUSY;

	return 0;
}

/*
 * Generic functions and exported symbols.
 */

#define VIRTQUEUE_CALL(vq, op, ...)					\
	({								\
	typeof(vq) __VIRTQUEUE_CALL_vq = (vq);				\
	typeof(split_ops.op(__VIRTQUEUE_CALL_vq, ##__VA_ARGS__)) ret;	\
									\
	switch (__VIRTQUEUE_CALL_vq->layout) {				\
	case VQ_LAYOUT_SPLIT:						\
		ret = split_ops.op(__VIRTQUEUE_CALL_vq, ##__VA_ARGS__);	\
		break;							\
	case VQ_LAYOUT_PACKED:						\
		ret = packed_ops.op(__VIRTQUEUE_CALL_vq, ##__VA_ARGS__);\
		break;							\
	case VQ_LAYOUT_SPLIT_IN_ORDER:					\
		ret = split_in_order_ops.op(vq, ##__VA_ARGS__);		\
		break;							\
	case VQ_LAYOUT_PACKED_IN_ORDER:					\
		ret = packed_in_order_ops.op(vq, ##__VA_ARGS__);	\
		break;							\
	default:							\
		BUG();							\
		break;							\
	}								\
	ret;								\
})

#define VOID_VIRTQUEUE_CALL(vq, op, ...)				\
	({								\
	typeof(vq) __VIRTQUEUE_CALL_vq = (vq);				\
									\
	switch (__VIRTQUEUE_CALL_vq->layout) {				\
	case VQ_LAYOUT_SPLIT:						\
		split_ops.op(__VIRTQUEUE_CALL_vq, ##__VA_ARGS__);	\
		break;							\
	case VQ_LAYOUT_PACKED:						\
		packed_ops.op(__VIRTQUEUE_CALL_vq, ##__VA_ARGS__);	\
		break;							\
	case VQ_LAYOUT_SPLIT_IN_ORDER:					\
		split_in_order_ops.op(vq, ##__VA_ARGS__);		\
		break;							\
	case VQ_LAYOUT_PACKED_IN_ORDER:					\
		packed_in_order_ops.op(vq, ##__VA_ARGS__);		\
		break;							\
	default:							\
		BUG();							\
		break;							\
	}								\
})

static inline int virtqueue_add(struct virtqueue *_vq,
				struct scatterlist *sgs[],
				unsigned int total_sg,
				unsigned int out_sgs,
				unsigned int in_sgs,
				void *data,
				void *ctx,
				bool premapped,
				gfp_t gfp,
				unsigned long attr)
{
	struct vring_virtqueue *vq = to_vvq(_vq);

	return VIRTQUEUE_CALL(vq, add, sgs, total_sg,
			      out_sgs, in_sgs, data,
			      ctx, premapped, gfp, attr);
}

/**
 * virtqueue_add_sgs - expose buffers to other end
 * @_vq: the struct virtqueue we're talking about.
 * @sgs: array of terminated scatterlists.
 * @out_sgs: the number of scatterlists readable by other side
 * @in_sgs: the number of scatterlists which are writable (after readable ones)
 * @data: the token identifying the buffer.
 * @gfp: how to do memory allocations (if necessary).
 *
 * Caller must ensure we don't call this with other virtqueue operations
 * at the same time (except where noted).
 *
 * Returns zero or a negative error (ie. ENOSPC, ENOMEM, EIO).
 *
 * NB: ENOSPC is a special code that is only returned on an attempt to add a
 * buffer to a full VQ. It indicates that some buffers are outstanding and that
 * the operation can be retried after some buffers have been used.
 */
int virtqueue_add_sgs(struct virtqueue *_vq,
		      struct scatterlist *sgs[],
		      unsigned int out_sgs,
		      unsigned int in_sgs,
		      void *data,
		      gfp_t gfp)
{
	unsigned int i, total_sg = 0;

	/* Count them first. */
	for (i = 0; i < out_sgs + in_sgs; i++) {
		struct scatterlist *sg;

		for (sg = sgs[i]; sg; sg = sg_next(sg))
			total_sg++;
	}
	return virtqueue_add(_vq, sgs, total_sg, out_sgs, in_sgs,
			     data, NULL, false, gfp, 0);
}
EXPORT_SYMBOL_GPL(virtqueue_add_sgs);

/**
 * virtqueue_add_outbuf - expose output buffers to other end
 * @vq: the struct virtqueue we're talking about.
 * @sg: scatterlist (must be well-formed and terminated!)
 * @num: the number of entries in @sg readable by other side
 * @data: the token identifying the buffer.
 * @gfp: how to do memory allocations (if necessary).
 *
 * Caller must ensure we don't call this with other virtqueue operations
 * at the same time (except where noted).
 *
 * Returns zero or a negative error (ie. ENOSPC, ENOMEM, EIO).
 */
int virtqueue_add_outbuf(struct virtqueue *vq,
			 struct scatterlist *sg, unsigned int num,
			 void *data,
			 gfp_t gfp)
{
	return virtqueue_add(vq, &sg, num, 1, 0, data, NULL, false, gfp, 0);
}
EXPORT_SYMBOL_GPL(virtqueue_add_outbuf);

/**
 * virtqueue_add_outbuf_premapped - expose output buffers to other end
 * @vq: the struct virtqueue we're talking about.
 * @sg: scatterlist (must be well-formed and terminated!)
 * @num: the number of entries in @sg readable by other side
 * @data: the token identifying the buffer.
 * @gfp: how to do memory allocations (if necessary).
 *
 * Caller must ensure we don't call this with other virtqueue operations
 * at the same time (except where noted).
 *
 * Return:
 * Returns zero or a negative error (ie. ENOSPC, ENOMEM, EIO).
 */
int virtqueue_add_outbuf_premapped(struct virtqueue *vq,
				   struct scatterlist *sg, unsigned int num,
				   void *data,
				   gfp_t gfp)
{
	return virtqueue_add(vq, &sg, num, 1, 0, data, NULL, true, gfp, 0);
}
EXPORT_SYMBOL_GPL(virtqueue_add_outbuf_premapped);

/**
 * virtqueue_add_inbuf - expose input buffers to other end
 * @vq: the struct virtqueue we're talking about.
 * @sg: scatterlist (must be well-formed and terminated!)
 * @num: the number of entries in @sg writable by other side
 * @data: the token identifying the buffer.
 * @gfp: how to do memory allocations (if necessary).
 *
 * Caller must ensure we don't call this with other virtqueue operations
 * at the same time (except where noted).
 *
 * Returns zero or a negative error (ie. ENOSPC, ENOMEM, EIO).
 */
int virtqueue_add_inbuf(struct virtqueue *vq,
			struct scatterlist *sg, unsigned int num,
			void *data,
			gfp_t gfp)
{
	return virtqueue_add(vq, &sg, num, 0, 1, data, NULL, false, gfp, 0);
}
EXPORT_SYMBOL_GPL(virtqueue_add_inbuf);

/**
 * virtqueue_add_inbuf_cache_clean - expose input buffers with cache clean
 * @vq: the struct virtqueue we're talking about.
 * @sg: scatterlist (must be well-formed and terminated!)
 * @num: the number of entries in @sg writable by other side
 * @data: the token identifying the buffer.
 * @gfp: how to do memory allocations (if necessary).
 *
 * Same as virtqueue_add_inbuf but passes DMA_ATTR_DEBUGGING_IGNORE_CACHELINES
 * to indicate that the CPU will not dirty any cacheline overlapping this buffer
 * while it is available, and to suppress overlapping cacheline warnings in DMA
 * debug builds.
 *
 * Caller must ensure we don't call this with other virtqueue operations
 * at the same time (except where noted).
 *
 * Returns zero or a negative error (ie. ENOSPC, ENOMEM, EIO).
 */
int virtqueue_add_inbuf_cache_clean(struct virtqueue *vq,
				    struct scatterlist *sg, unsigned int num,
				    void *data,
				    gfp_t gfp)
{
	return virtqueue_add(vq, &sg, num, 0, 1, data, NULL, false, gfp,
			     DMA_ATTR_DEBUGGING_IGNORE_CACHELINES);
}
EXPORT_SYMBOL_GPL(virtqueue_add_inbuf_cache_clean);

/**
 * virtqueue_add_inbuf_ctx - expose input buffers to other end
 * @vq: the struct virtqueue we're talking about.
 * @sg: scatterlist (must be well-formed and terminated!)
 * @num: the number of entries in @sg writable by other side
 * @data: the token identifying the buffer.
 * @ctx: extra context for the token
 * @gfp: how to do memory allocations (if necessary).
 *
 * Caller must ensure we don't call this with other virtqueue operations
 * at the same time (except where noted).
 *
 * Returns zero or a negative error (ie. ENOSPC, ENOMEM, EIO).
 */
int virtqueue_add_inbuf_ctx(struct virtqueue *vq,
			struct scatterlist *sg, unsigned int num,
			void *data,
			void *ctx,
			gfp_t gfp)
{
	return virtqueue_add(vq, &sg, num, 0, 1, data, ctx, false, gfp, 0);
}
EXPORT_SYMBOL_GPL(virtqueue_add_inbuf_ctx);

/**
 * virtqueue_add_inbuf_premapped - expose input buffers to other end
 * @vq: the struct virtqueue we're talking about.
 * @sg: scatterlist (must be well-formed and terminated!)
 * @num: the number of entries in @sg writable by other side
 * @data: the token identifying the buffer.
 * @ctx: extra context for the token
 * @gfp: how to do memory allocations (if necessary).
 *
 * Caller must ensure we don't call this with other virtqueue operations
 * at the same time (except where noted).
 *
 * Return:
 * Returns zero or a negative error (ie. ENOSPC, ENOMEM, EIO).
 */
int virtqueue_add_inbuf_premapped(struct virtqueue *vq,
				  struct scatterlist *sg, unsigned int num,
				  void *data,
				  void *ctx,
				  gfp_t gfp)
{
	return virtqueue_add(vq, &sg, num, 0, 1, data, ctx, true, gfp, 0);
}
EXPORT_SYMBOL_GPL(virtqueue_add_inbuf_premapped);

/**
 * virtqueue_dma_dev - get the dma dev
 * @_vq: the struct virtqueue we're talking about.
 *
 * Returns the dma dev. That can been used for dma api.
 */
struct device *virtqueue_dma_dev(struct virtqueue *_vq)
{
	struct vring_virtqueue *vq = to_vvq(_vq);

	if (vq->use_map_api && !_vq->vdev->map)
		return vq->map.dma_dev;
	else
		return NULL;
}
EXPORT_SYMBOL_GPL(virtqueue_dma_dev);

/**
 * virtqueue_kick_prepare - first half of split virtqueue_kick call.
 * @_vq: the struct virtqueue
 *
 * Instead of virtqueue_kick(), you can do:
 *	if (virtqueue_kick_prepare(vq))
 *		virtqueue_notify(vq);
 *
 * This is sometimes useful because the virtqueue_kick_prepare() needs
 * to be serialized, but the actual virtqueue_notify() call does not.
 */
bool virtqueue_kick_prepare(struct virtqueue *_vq)
{
	struct vring_virtqueue *vq = to_vvq(_vq);

	return VIRTQUEUE_CALL(vq, kick_prepare);
}
EXPORT_SYMBOL_GPL(virtqueue_kick_prepare);

/**
 * virtqueue_notify - second half of split virtqueue_kick call.
 * @_vq: the struct virtqueue
 *
 * This does not need to be serialized.
 *
 * Returns false if host notify failed or queue is broken, otherwise true.
 */
bool virtqueue_notify(struct virtqueue *_vq)
{
	struct vring_virtqueue *vq = to_vvq(_vq);

	if (unlikely(vq->broken))
		return false;

	/* Prod other side to tell it about changes. */
	if (!vq->notify(_vq)) {
		vq->broken = true;
		return false;
	}
	return true;
}
EXPORT_SYMBOL_GPL(virtqueue_notify);

/**
 * virtqueue_kick - update after add_buf
 * @vq: the struct virtqueue
 *
 * After one or more virtqueue_add_* calls, invoke this to kick
 * the other side.
 *
 * Caller must ensure we don't call this with other virtqueue
 * operations at the same time (except where noted).
 *
 * Returns false if kick failed, otherwise true.
 */
bool virtqueue_kick(struct virtqueue *vq)
{
	if (virtqueue_kick_prepare(vq))
		return virtqueue_notify(vq);
	return true;
}
EXPORT_SYMBOL_GPL(virtqueue_kick);

/**
 * virtqueue_get_buf_ctx - get the next used buffer
 * @_vq: the struct virtqueue we're talking about.
 * @len: the length written into the buffer
 * @ctx: extra context for the token
 *
 * If the device wrote data into the buffer, @len will be set to the
 * amount written.  This means you don't need to clear the buffer
 * beforehand to ensure there's no data leakage in the case of short
 * writes.
 *
 * Caller must ensure we don't call this with other virtqueue
 * operations at the same time (except where noted).
 *
 * Returns NULL if there are no used buffers, or the "data" token
 * handed to virtqueue_add_*().
 */
void *virtqueue_get_buf_ctx(struct virtqueue *_vq, unsigned int *len,
			    void **ctx)
{
	struct vring_virtqueue *vq = to_vvq(_vq);

	return VIRTQUEUE_CALL(vq, get, len, ctx);
}
EXPORT_SYMBOL_GPL(virtqueue_get_buf_ctx);

void *virtqueue_get_buf(struct virtqueue *_vq, unsigned int *len)
{
	return virtqueue_get_buf_ctx(_vq, len, NULL);
}
EXPORT_SYMBOL_GPL(virtqueue_get_buf);
/**
 * virtqueue_disable_cb - disable callbacks
 * @_vq: the struct virtqueue we're talking about.
 *
 * Note that this is not necessarily synchronous, hence unreliable and only
 * useful as an optimization.
 *
 * Unlike other operations, this need not be serialized.
 */
void virtqueue_disable_cb(struct virtqueue *_vq)
{
	struct vring_virtqueue *vq = to_vvq(_vq);

	VOID_VIRTQUEUE_CALL(vq, disable_cb);
}
EXPORT_SYMBOL_GPL(virtqueue_disable_cb);

/**
 * virtqueue_enable_cb_prepare - restart callbacks after disable_cb
 * @_vq: the struct virtqueue we're talking about.
 *
 * This re-enables callbacks; it returns current queue state
 * in an opaque unsigned value. This value should be later tested by
 * virtqueue_poll, to detect a possible race between the driver checking for
 * more work, and enabling callbacks.
 *
 * Caller must ensure we don't call this with other virtqueue
 * operations at the same time (except where noted).
 */
unsigned int virtqueue_enable_cb_prepare(struct virtqueue *_vq)
{
	struct vring_virtqueue *vq = to_vvq(_vq);

	if (vq->event_triggered)
		vq->event_triggered = false;

	return VIRTQUEUE_CALL(vq, enable_cb_prepare);
}
EXPORT_SYMBOL_GPL(virtqueue_enable_cb_prepare);

/**
 * virtqueue_poll - query pending used buffers
 * @_vq: the struct virtqueue we're talking about.
 * @last_used_idx: virtqueue state (from call to virtqueue_enable_cb_prepare).
 *
 * Returns "true" if there are pending used buffers in the queue.
 *
 * This does not need to be serialized.
 */
bool virtqueue_poll(struct virtqueue *_vq, unsigned int last_used_idx)
{
	struct vring_virtqueue *vq = to_vvq(_vq);

	if (unlikely(vq->broken))
		return false;

	virtio_mb(vq->weak_barriers);

	return VIRTQUEUE_CALL(vq, poll, last_used_idx);
}
EXPORT_SYMBOL_GPL(virtqueue_poll);

/**
 * virtqueue_enable_cb - restart callbacks after disable_cb.
 * @_vq: the struct virtqueue we're talking about.
 *
 * This re-enables callbacks; it returns "false" if there are pending
 * buffers in the queue, to detect a possible race between the driver
 * checking for more work, and enabling callbacks.
 *
 * Caller must ensure we don't call this with other virtqueue
 * operations at the same time (except where noted).
 */
bool virtqueue_enable_cb(struct virtqueue *_vq)
{
	unsigned int last_used_idx = virtqueue_enable_cb_prepare(_vq);

	return !virtqueue_poll(_vq, last_used_idx);
}
EXPORT_SYMBOL_GPL(virtqueue_enable_cb);

/**
 * virtqueue_enable_cb_delayed - restart callbacks after disable_cb.
 * @_vq: the struct virtqueue we're talking about.
 *
 * This re-enables callbacks but hints to the other side to delay
 * interrupts until most of the available buffers have been processed;
 * it returns "false" if there are many pending buffers in the queue,
 * to detect a possible race between the driver checking for more work,
 * and enabling callbacks.
 *
 * Caller must ensure we don't call this with other virtqueue
 * operations at the same time (except where noted).
 */
bool virtqueue_enable_cb_delayed(struct virtqueue *_vq)
{
	struct vring_virtqueue *vq = to_vvq(_vq);

	if (vq->event_triggered)
		data_race(vq->event_triggered = false);

	return VIRTQUEUE_CALL(vq, enable_cb_delayed);
}
EXPORT_SYMBOL_GPL(virtqueue_enable_cb_delayed);

/**
 * virtqueue_detach_unused_buf - detach first unused buffer
 * @_vq: the struct virtqueue we're talking about.
 *
 * Returns NULL or the "data" token handed to virtqueue_add_*().
 * This is not valid on an active queue; it is useful for device
 * shutdown or the reset queue.
 */
void *virtqueue_detach_unused_buf(struct virtqueue *_vq)
{
	struct vring_virtqueue *vq = to_vvq(_vq);

	return VIRTQUEUE_CALL(vq, detach_unused_buf);
}
EXPORT_SYMBOL_GPL(virtqueue_detach_unused_buf);

static inline bool more_used(const struct vring_virtqueue *vq)
{
	return VIRTQUEUE_CALL(vq, more_used);
}

/**
 * vring_interrupt - notify a virtqueue on an interrupt
 * @irq: the IRQ number (ignored)
 * @_vq: the struct virtqueue to notify
 *
 * Calls the callback function of @_vq to process the virtqueue
 * notification.
 */
irqreturn_t vring_interrupt(int irq, void *_vq)
{
	struct vring_virtqueue *vq = to_vvq(_vq);

	if (!more_used(vq)) {
		pr_debug("virtqueue interrupt with no work for %p\n", vq);
		return IRQ_NONE;
	}

	if (unlikely(vq->broken)) {
#ifdef CONFIG_VIRTIO_HARDEN_NOTIFICATION
		dev_warn_once(&vq->vq.vdev->dev,
			      "virtio vring IRQ raised before DRIVER_OK");
		return IRQ_NONE;
#else
		return IRQ_HANDLED;
#endif
	}

	/* Just a hint for performance: so it's ok that this can be racy! */
	if (vq->event)
		data_race(vq->event_triggered = true);

	pr_debug("virtqueue callback for %p (%p)\n", vq, vq->vq.callback);
	if (vq->vq.callback)
		vq->vq.callback(&vq->vq);

	return IRQ_HANDLED;
}
EXPORT_SYMBOL_GPL(vring_interrupt);

struct virtqueue *vring_create_virtqueue(
	unsigned int index,
	unsigned int num,
	unsigned int vring_align,
	struct virtio_device *vdev,
	bool weak_barriers,
	bool may_reduce_num,
	bool context,
	bool (*notify)(struct virtqueue *),
	void (*callback)(struct virtqueue *),
	const char *name)
{
	union virtio_map map = {.dma_dev = vdev->dev.parent};

	if (virtio_has_feature(vdev, VIRTIO_F_RING_PACKED))
		return vring_create_virtqueue_packed(index, num, vring_align,
				vdev, weak_barriers, may_reduce_num,
				context, notify, callback, name, map);

	return vring_create_virtqueue_split(index, num, vring_align,
			vdev, weak_barriers, may_reduce_num,
			context, notify, callback, name, map);
}
EXPORT_SYMBOL_GPL(vring_create_virtqueue);

struct virtqueue *vring_create_virtqueue_map(
	unsigned int index,
	unsigned int num,
	unsigned int vring_align,
	struct virtio_device *vdev,
	bool weak_barriers,
	bool may_reduce_num,
	bool context,
	bool (*notify)(struct virtqueue *),
	void (*callback)(struct virtqueue *),
	const char *name,
	union virtio_map map)
{

	if (virtio_has_feature(vdev, VIRTIO_F_RING_PACKED))
		return vring_create_virtqueue_packed(index, num, vring_align,
				vdev, weak_barriers, may_reduce_num,
				context, notify, callback, name, map);

	return vring_create_virtqueue_split(index, num, vring_align,
			vdev, weak_barriers, may_reduce_num,
			context, notify, callback, name, map);
}
EXPORT_SYMBOL_GPL(vring_create_virtqueue_map);

/**
 * virtqueue_resize - resize the vring of vq
 * @_vq: the struct virtqueue we're talking about.
 * @num: new ring num
 * @recycle: callback to recycle unused buffers
 * @recycle_done: callback to be invoked when recycle for all unused buffers done
 *
 * When it is really necessary to create a new vring, it will set the current vq
 * into the reset state. Then call the passed callback to recycle the buffer
 * that is no longer used. Only after the new vring is successfully created, the
 * old vring will be released.
 *
 * Caller must ensure we don't call this with other virtqueue operations
 * at the same time (except where noted).
 *
 * Returns zero or a negative error.
 * 0: success.
 * -ENOMEM: Failed to allocate a new ring, fall back to the original ring size.
 *  vq can still work normally
 * -EBUSY: Failed to sync with device, vq may not work properly
 * -ENOENT: Transport or device not supported
 * -E2BIG/-EINVAL: num error
 * -EPERM: Operation not permitted
 *
 */
int virtqueue_resize(struct virtqueue *_vq, u32 num,
		     void (*recycle)(struct virtqueue *vq, void *buf),
		     void (*recycle_done)(struct virtqueue *vq))
{
	struct vring_virtqueue *vq = to_vvq(_vq);
	int err, err_reset;

	if (num > vq->vq.num_max)
		return -E2BIG;

	if (!num)
		return -EINVAL;

	if (virtqueue_get_vring_size(_vq) == num)
		return 0;

	err = virtqueue_disable_and_recycle(_vq, recycle);
	if (err)
		return err;
	if (recycle_done)
		recycle_done(_vq);

	err = VIRTQUEUE_CALL(vq, resize, num);

	err_reset = virtqueue_enable_after_reset(_vq);
	if (err_reset)
		return err_reset;

	return err;
}
EXPORT_SYMBOL_GPL(virtqueue_resize);

/**
 * virtqueue_reset - detach and recycle all unused buffers
 * @_vq: the struct virtqueue we're talking about.
 * @recycle: callback to recycle unused buffers
 * @recycle_done: callback to be invoked when recycle for all unused buffers done
 *
 * Caller must ensure we don't call this with other virtqueue operations
 * at the same time (except where noted).
 *
 * Returns zero or a negative error.
 * 0: success.
 * -EBUSY: Failed to sync with device, vq may not work properly
 * -ENOENT: Transport or device not supported
 * -EPERM: Operation not permitted
 */
int virtqueue_reset(struct virtqueue *_vq,
		    void (*recycle)(struct virtqueue *vq, void *buf),
		    void (*recycle_done)(struct virtqueue *vq))
{
	struct vring_virtqueue *vq = to_vvq(_vq);
	int err;

	err = virtqueue_disable_and_recycle(_vq, recycle);
	if (err)
		return err;
	if (recycle_done)
		recycle_done(_vq);

	VOID_VIRTQUEUE_CALL(vq, reset);

	return virtqueue_enable_after_reset(_vq);
}
EXPORT_SYMBOL_GPL(virtqueue_reset);

struct virtqueue *vring_new_virtqueue(unsigned int index,
				      unsigned int num,
				      unsigned int vring_align,
				      struct virtio_device *vdev,
				      bool weak_barriers,
				      bool context,
				      void *pages,
				      bool (*notify)(struct virtqueue *vq),
				      void (*callback)(struct virtqueue *vq),
				      const char *name)
{
	struct vring_virtqueue_split vring_split = {};
	union virtio_map map = {.dma_dev = vdev->dev.parent};

	if (virtio_has_feature(vdev, VIRTIO_F_RING_PACKED)) {
		struct vring_virtqueue_packed vring_packed = {};

		vring_packed.vring.num = num;
		vring_packed.vring.desc = pages;
		return __vring_new_virtqueue_packed(index, &vring_packed,
						    vdev, weak_barriers,
						    context, notify, callback,
						    name, map);
	}

	vring_init(&vring_split.vring, num, pages, vring_align);
	return __vring_new_virtqueue_split(index, &vring_split, vdev, weak_barriers,
				     context, notify, callback, name,
				     map);
}
EXPORT_SYMBOL_GPL(vring_new_virtqueue);

static void vring_free(struct virtqueue *_vq)
{
	struct vring_virtqueue *vq = to_vvq(_vq);

	if (vq->we_own_ring) {
		if (virtqueue_is_packed(vq)) {
			vring_free_queue(vq->vq.vdev,
					 vq->packed.ring_size_in_bytes,
					 vq->packed.vring.desc,
					 vq->packed.ring_dma_addr,
					 vq->map);

			vring_free_queue(vq->vq.vdev,
					 vq->packed.event_size_in_bytes,
					 vq->packed.vring.driver,
					 vq->packed.driver_event_dma_addr,
					 vq->map);

			vring_free_queue(vq->vq.vdev,
					 vq->packed.event_size_in_bytes,
					 vq->packed.vring.device,
					 vq->packed.device_event_dma_addr,
					 vq->map);

			kfree(vq->packed.desc_state);
			kfree(vq->packed.desc_extra);
		} else {
			vring_free_queue(vq->vq.vdev,
					 vq->split.queue_size_in_bytes,
					 vq->split.vring.desc,
					 vq->split.queue_dma_addr,
					 vq->map);
		}
	}
	if (!virtqueue_is_packed(vq)) {
		kfree(vq->split.desc_state);
		kfree(vq->split.desc_extra);
	}
}

void vring_del_virtqueue(struct virtqueue *_vq)
{
	struct vring_virtqueue *vq = to_vvq(_vq);

	spin_lock(&vq->vq.vdev->vqs_list_lock);
	list_del(&_vq->list);
	spin_unlock(&vq->vq.vdev->vqs_list_lock);

	vring_free(_vq);

	kfree(vq);
}
EXPORT_SYMBOL_GPL(vring_del_virtqueue);

u32 vring_notification_data(struct virtqueue *_vq)
{
	struct vring_virtqueue *vq = to_vvq(_vq);
	u16 next;

	if (virtqueue_is_packed(vq))
		next = (vq->packed.next_avail_idx &
				~(-(1 << VRING_PACKED_EVENT_F_WRAP_CTR))) |
			vq->packed.avail_wrap_counter <<
				VRING_PACKED_EVENT_F_WRAP_CTR;
	else
		next = vq->split.avail_idx_shadow;

	return next << 16 | _vq->index;
}
EXPORT_SYMBOL_GPL(vring_notification_data);

/* Manipulates transport-specific feature bits. */
void vring_transport_features(struct virtio_device *vdev)
{
	unsigned int i;

	for (i = VIRTIO_TRANSPORT_F_START; i < VIRTIO_TRANSPORT_F_END; i++) {
		switch (i) {
		case VIRTIO_RING_F_INDIRECT_DESC:
			break;
		case VIRTIO_RING_F_EVENT_IDX:
			break;
		case VIRTIO_F_VERSION_1:
			break;
		case VIRTIO_F_ACCESS_PLATFORM:
			break;
		case VIRTIO_F_RING_PACKED:
			break;
		case VIRTIO_F_ORDER_PLATFORM:
			break;
		case VIRTIO_F_NOTIFICATION_DATA:
			break;
		case VIRTIO_F_IN_ORDER:
			break;
		default:
			/* We don't understand this bit. */
			__virtio_clear_bit(vdev, i);
		}
	}
}
EXPORT_SYMBOL_GPL(vring_transport_features);

/**
 * virtqueue_get_vring_size - return the size of the virtqueue's vring
 * @_vq: the struct virtqueue containing the vring of interest.
 *
 * Returns the size of the vring.  This is mainly used for boasting to
 * userspace.  Unlike other operations, this need not be serialized.
 */
unsigned int virtqueue_get_vring_size(const struct virtqueue *_vq)
{

	const struct vring_virtqueue *vq = to_vvq(_vq);

	return virtqueue_is_packed(vq) ? vq->packed.vring.num :
				      vq->split.vring.num;
}
EXPORT_SYMBOL_GPL(virtqueue_get_vring_size);

/*
 * This function should only be called by the core, not directly by the driver.
 */
void __virtqueue_break(struct virtqueue *_vq)
{
	struct vring_virtqueue *vq = to_vvq(_vq);

	/* Pairs with READ_ONCE() in virtqueue_is_broken(). */
	WRITE_ONCE(vq->broken, true);
}
EXPORT_SYMBOL_GPL(__virtqueue_break);

/*
 * This function should only be called by the core, not directly by the driver.
 */
void __virtqueue_unbreak(struct virtqueue *_vq)
{
	struct vring_virtqueue *vq = to_vvq(_vq);

	/* Pairs with READ_ONCE() in virtqueue_is_broken(). */
	WRITE_ONCE(vq->broken, false);
}
EXPORT_SYMBOL_GPL(__virtqueue_unbreak);

bool virtqueue_is_broken(const struct virtqueue *_vq)
{
	const struct vring_virtqueue *vq = to_vvq(_vq);

	return READ_ONCE(vq->broken);
}
EXPORT_SYMBOL_GPL(virtqueue_is_broken);

/*
 * This should prevent the device from being used, allowing drivers to
 * recover.  You may need to grab appropriate locks to flush.
 */
void virtio_break_device(struct virtio_device *dev)
{
	struct virtqueue *_vq;

	spin_lock(&dev->vqs_list_lock);
	list_for_each_entry(_vq, &dev->vqs, list) {
		struct vring_virtqueue *vq = to_vvq(_vq);

		/* Pairs with READ_ONCE() in virtqueue_is_broken(). */
		WRITE_ONCE(vq->broken, true);
	}
	spin_unlock(&dev->vqs_list_lock);
}
EXPORT_SYMBOL_GPL(virtio_break_device);

/*
 * This should allow the device to be used by the driver. You may
 * need to grab appropriate locks to flush the write to
 * vq->broken. This should only be used in some specific case e.g
 * (probing and restoring). This function should only be called by the
 * core, not directly by the driver.
 */
void __virtio_unbreak_device(struct virtio_device *dev)
{
	struct virtqueue *_vq;

	spin_lock(&dev->vqs_list_lock);
	list_for_each_entry(_vq, &dev->vqs, list) {
		struct vring_virtqueue *vq = to_vvq(_vq);

		/* Pairs with READ_ONCE() in virtqueue_is_broken(). */
		WRITE_ONCE(vq->broken, false);
	}
	spin_unlock(&dev->vqs_list_lock);
}
EXPORT_SYMBOL_GPL(__virtio_unbreak_device);

dma_addr_t virtqueue_get_desc_addr(const struct virtqueue *_vq)
{
	const struct vring_virtqueue *vq = to_vvq(_vq);

	BUG_ON(!vq->we_own_ring);

	if (virtqueue_is_packed(vq))
		return vq->packed.ring_dma_addr;

	return vq->split.queue_dma_addr;
}
EXPORT_SYMBOL_GPL(virtqueue_get_desc_addr);

dma_addr_t virtqueue_get_avail_addr(const struct virtqueue *_vq)
{
	const struct vring_virtqueue *vq = to_vvq(_vq);

	BUG_ON(!vq->we_own_ring);

	if (virtqueue_is_packed(vq))
		return vq->packed.driver_event_dma_addr;

	return vq->split.queue_dma_addr +
		((char *)vq->split.vring.avail - (char *)vq->split.vring.desc);
}
EXPORT_SYMBOL_GPL(virtqueue_get_avail_addr);

dma_addr_t virtqueue_get_used_addr(const struct virtqueue *_vq)
{
	const struct vring_virtqueue *vq = to_vvq(_vq);

	BUG_ON(!vq->we_own_ring);

	if (virtqueue_is_packed(vq))
		return vq->packed.device_event_dma_addr;

	return vq->split.queue_dma_addr +
		((char *)vq->split.vring.used - (char *)vq->split.vring.desc);
}
EXPORT_SYMBOL_GPL(virtqueue_get_used_addr);

/* Only available for split ring */
const struct vring *virtqueue_get_vring(const struct virtqueue *vq)
{
	return &to_vvq(vq)->split.vring;
}
EXPORT_SYMBOL_GPL(virtqueue_get_vring);

/**
 * virtqueue_map_alloc_coherent - alloc coherent mapping
 * @vdev: the virtio device we are talking to
 * @map: metadata for performing mapping
 * @size: the size of the buffer
 * @map_handle: the pointer to the mapped address
 * @gfp: allocation flag (GFP_XXX)
 *
 * return virtual address or NULL on error
 */
void *virtqueue_map_alloc_coherent(struct virtio_device *vdev,
				   union virtio_map map,
				   size_t size, dma_addr_t *map_handle,
				   gfp_t gfp)
{
	if (vdev->map)
		return vdev->map->alloc(map, size,
					map_handle, gfp);
	else
		return dma_alloc_coherent(map.dma_dev, size,
					  map_handle, gfp);
}
EXPORT_SYMBOL_GPL(virtqueue_map_alloc_coherent);

/**
 * virtqueue_map_free_coherent - free coherent mapping
 * @vdev: the virtio device we are talking to
 * @map: metadata for performing mapping
 * @size: the size of the buffer
 * @vaddr: the virtual address that needs to be freed
 * @map_handle: the mapped address that needs to be freed
 *
 */
void virtqueue_map_free_coherent(struct virtio_device *vdev,
				 union virtio_map map, size_t size, void *vaddr,
				 dma_addr_t map_handle)
{
	if (vdev->map)
		vdev->map->free(map, size, vaddr,
				map_handle, 0);
	else
		dma_free_coherent(map.dma_dev, size, vaddr, map_handle);
}
EXPORT_SYMBOL_GPL(virtqueue_map_free_coherent);

/**
 * virtqueue_map_page_attrs - map a page to the device
 * @_vq: the virtqueue we are talking to
 * @page: the page that will be mapped by the device
 * @offset: the offset in the page for a buffer
 * @size: the buffer size
 * @dir: mapping direction
 * @attrs: mapping attributes
 *
 * Returns mapped address. Caller should check that by virtqueue_map_mapping_error().
 */
dma_addr_t virtqueue_map_page_attrs(const struct virtqueue *_vq,
				    struct page *page,
				    unsigned long offset,
				    size_t size,
				    enum dma_data_direction dir,
				    unsigned long attrs)
{
	const struct vring_virtqueue *vq = to_vvq(_vq);
	struct virtio_device *vdev = _vq->vdev;

	if (vdev->map)
		return vdev->map->map_page(vq->map,
					   page, offset, size,
					   dir, attrs);

	return dma_map_page_attrs(vring_dma_dev(vq),
				  page, offset, size,
				  dir, attrs);
}
EXPORT_SYMBOL_GPL(virtqueue_map_page_attrs);

/**
 * virtqueue_unmap_page_attrs - map a page to the device
 * @_vq: the virtqueue we are talking to
 * @map_handle: the mapped address
 * @size: the buffer size
 * @dir: mapping direction
 * @attrs: unmapping attributes
 */
void virtqueue_unmap_page_attrs(const struct virtqueue *_vq,
				dma_addr_t map_handle,
				size_t size, enum dma_data_direction dir,
				unsigned long attrs)
{
	const struct vring_virtqueue *vq = to_vvq(_vq);
	struct virtio_device *vdev = _vq->vdev;

	if (vdev->map)
		vdev->map->unmap_page(vq->map,
				      map_handle, size, dir, attrs);
	else
		dma_unmap_page_attrs(vring_dma_dev(vq), map_handle,
				     size, dir, attrs);
}
EXPORT_SYMBOL_GPL(virtqueue_unmap_page_attrs);

/**
 * virtqueue_map_single_attrs - map DMA for _vq
 * @_vq: the struct virtqueue we're talking about.
 * @ptr: the pointer of the buffer to do dma
 * @size: the size of the buffer to do dma
 * @dir: DMA direction
 * @attrs: DMA Attrs
 *
 * The caller calls this to do dma mapping in advance. The DMA address can be
 * passed to this _vq when it is in pre-mapped mode.
 *
 * return mapped address. Caller should check that by virtqueue_map_mapping_error().
 */
dma_addr_t virtqueue_map_single_attrs(const struct virtqueue *_vq, void *ptr,
				      size_t size,
				      enum dma_data_direction dir,
				      unsigned long attrs)
{
	const struct vring_virtqueue *vq = to_vvq(_vq);

	if (!vq->use_map_api) {
		kmsan_handle_dma(virt_to_phys(ptr), size, dir);
		return (dma_addr_t)virt_to_phys(ptr);
	}

	/* DMA must never operate on areas that might be remapped. */
	if (dev_WARN_ONCE(&_vq->vdev->dev, is_vmalloc_addr(ptr),
			  "rejecting DMA map of vmalloc memory\n"))
		return DMA_MAPPING_ERROR;

	return virtqueue_map_page_attrs(&vq->vq, virt_to_page(ptr),
					offset_in_page(ptr), size, dir, attrs);
}
EXPORT_SYMBOL_GPL(virtqueue_map_single_attrs);

/**
 * virtqueue_unmap_single_attrs - unmap map for _vq
 * @_vq: the struct virtqueue we're talking about.
 * @addr: the dma address to unmap
 * @size: the size of the buffer
 * @dir: DMA direction
 * @attrs: DMA Attrs
 *
 * Unmap the address that is mapped by the virtqueue_map_* APIs.
 *
 */
void virtqueue_unmap_single_attrs(const struct virtqueue *_vq,
				  dma_addr_t addr,
				  size_t size, enum dma_data_direction dir,
				  unsigned long attrs)
{
	const struct vring_virtqueue *vq = to_vvq(_vq);

	if (!vq->use_map_api)
		return;

	virtqueue_unmap_page_attrs(_vq, addr, size, dir, attrs);
}
EXPORT_SYMBOL_GPL(virtqueue_unmap_single_attrs);

/**
 * virtqueue_map_mapping_error - check dma address
 * @_vq: the struct virtqueue we're talking about.
 * @addr: DMA address
 *
 * Returns 0 means dma valid. Other means invalid dma address.
 */
int virtqueue_map_mapping_error(const struct virtqueue *_vq, dma_addr_t addr)
{
	const struct vring_virtqueue *vq = to_vvq(_vq);

	return vring_mapping_error(vq, addr);
}
EXPORT_SYMBOL_GPL(virtqueue_map_mapping_error);

/**
 * virtqueue_map_need_sync - check a dma address needs sync
 * @_vq: the struct virtqueue we're talking about.
 * @addr: DMA address
 *
 * Check if the dma address mapped by the virtqueue_map_* APIs needs to be
 * synchronized
 *
 * return bool
 */
bool virtqueue_map_need_sync(const struct virtqueue *_vq, dma_addr_t addr)
{
	const struct vring_virtqueue *vq = to_vvq(_vq);
	struct virtio_device *vdev = _vq->vdev;

	if (!vq->use_map_api)
		return false;

	if (vdev->map)
		return vdev->map->need_sync(vq->map, addr);
	else
		return dma_need_sync(vring_dma_dev(vq), addr);
}
EXPORT_SYMBOL_GPL(virtqueue_map_need_sync);

/**
 * virtqueue_map_sync_single_range_for_cpu - map sync for cpu
 * @_vq: the struct virtqueue we're talking about.
 * @addr: DMA address
 * @offset: DMA address offset
 * @size: buf size for sync
 * @dir: DMA direction
 *
 * Before calling this function, use virtqueue_map_need_sync() to confirm that
 * the DMA address really needs to be synchronized
 *
 */
void virtqueue_map_sync_single_range_for_cpu(const struct virtqueue *_vq,
					     dma_addr_t addr,
					     unsigned long offset, size_t size,
					     enum dma_data_direction dir)
{
	const struct vring_virtqueue *vq = to_vvq(_vq);
	struct virtio_device *vdev = _vq->vdev;

	if (!vq->use_map_api)
		return;

	if (vdev->map)
		vdev->map->sync_single_for_cpu(vq->map,
					       addr + offset, size, dir);
	else
		dma_sync_single_range_for_cpu(vring_dma_dev(vq),
					      addr, offset, size, dir);
}
EXPORT_SYMBOL_GPL(virtqueue_map_sync_single_range_for_cpu);

/**
 * virtqueue_map_sync_single_range_for_device - map sync for device
 * @_vq: the struct virtqueue we're talking about.
 * @addr: DMA address
 * @offset: DMA address offset
 * @size: buf size for sync
 * @dir: DMA direction
 *
 * Before calling this function, use virtqueue_map_need_sync() to confirm that
 * the DMA address really needs to be synchronized
 */
void virtqueue_map_sync_single_range_for_device(const struct virtqueue *_vq,
						dma_addr_t addr,
						unsigned long offset, size_t size,
						enum dma_data_direction dir)
{
	const struct vring_virtqueue *vq = to_vvq(_vq);
	struct virtio_device *vdev = _vq->vdev;

	if (!vq->use_map_api)
		return;

	if (vdev->map)
		vdev->map->sync_single_for_device(vq->map,
						  addr + offset,
						  size, dir);
	else
		dma_sync_single_range_for_device(vring_dma_dev(vq), addr,
						 offset, size, dir);
}
EXPORT_SYMBOL_GPL(virtqueue_map_sync_single_range_for_device);

MODULE_DESCRIPTION("Virtio ring implementation");
MODULE_LICENSE("GPL");
]

gemini-3-flash-preview

							input: 45988
							output: 25
							thoughts: 627
						

**Analyzing a Race Condition in the Virtio-Net Driver**

Okay, so I've got this crash report, and it looks like a race condition is brewing around `0xffff88810224b25c`, specifically, around the `vq->last_used_idx` variable.  The value changed from `0x25f1` to `0x25f2`, which immediately points me towards the virtqueue's used index.

The report highlights line 959 in `virtqueue_get_buf_ctx_split`, and I just double-checked the source to confirm, yes, that's where the increment happens: `vq->last_used_idx++`.  Now, the other crucial piece of information is line 906 in `more_used_split`, which is called by `vring_interrupt`: `return virtqueue_poll_split(vq, vq->last_used_idx);`. So, it's reading the index.

Therefore, the race seems to be between two CPUs:  CPU 0, which is incrementing `vq->last_used_idx` in `virtqueue_get_buf_ctx_split`, and CPU 1, which is reading `vq->last_used_idx` in `more_used_split`.  We know `virtqueue_get_buf_ctx_split` is called from `virtnet_poll_tx` -> `virtnet_free_old_xmit` -> `virtqueue_get_buf`.  That part is fine.

But `vring_interrupt` is called when the device signals that there are used buffers, so it's a separate path. `virtqueue_get_buf` is usually called by the driver to reap used buffers.  The driver is supposed to serialize this call. Hmmm, from the NAPI context in `virtnet_poll_tx`.  That means the call is happening from a NAPI context.

Now, I need to go back and check how `virtnet_poll_tx` is handling that serialization, because this suggests a potential violation of the expected locking or synchronization mechanisms within that NAPI context. I need to dig into the `virtnet_poll_tx` code in `drivers/net/virtio_net.c`.

map[File:drivers/net/virtio_net.c]

map[Contents:// SPDX-License-Identifier: GPL-2.0-or-later
/* A network driver using virtio.
 *
 * Copyright 2007 Rusty Russell <rusty@rustcorp.com.au> IBM Corporation
 */
//#define DEBUG
#include <linux/netdevice.h>
#include <linux/etherdevice.h>
#include <linux/ethtool.h>
#include <linux/module.h>
#include <linux/virtio.h>
#include <linux/virtio_net.h>
#include <linux/bpf.h>
#include <linux/bpf_trace.h>
#include <linux/scatterlist.h>
#include <linux/if_vlan.h>
#include <linux/slab.h>
#include <linux/cpu.h>
#include <linux/average.h>
#include <linux/filter.h>
#include <linux/kernel.h>
#include <linux/dim.h>
#include <net/route.h>
#include <net/xdp.h>
#include <net/net_failover.h>
#include <net/netdev_rx_queue.h>
#include <net/netdev_queues.h>
#include <net/xdp_sock_drv.h>
#include <net/page_pool/helpers.h>

static int napi_weight = NAPI_POLL_WEIGHT;
module_param(napi_weight, int, 0444);

static bool csum = true, gso = true, napi_tx = true;
module_param(csum, bool, 0444);
module_param(gso, bool, 0444);
module_param(napi_tx, bool, 0644);

#define VIRTIO_OFFLOAD_MAP_MIN	46
#define VIRTIO_OFFLOAD_MAP_MAX	47
#define VIRTIO_FEATURES_MAP_MIN	65
#define VIRTIO_O2F_DELTA	(VIRTIO_FEATURES_MAP_MIN - \
				 VIRTIO_OFFLOAD_MAP_MIN)

static bool virtio_is_mapped_offload(unsigned int obit)
{
	return obit >= VIRTIO_OFFLOAD_MAP_MIN &&
	       obit <= VIRTIO_OFFLOAD_MAP_MAX;
}

static unsigned int virtio_offload_to_feature(unsigned int obit)
{
	return virtio_is_mapped_offload(obit) ? obit + VIRTIO_O2F_DELTA : obit;
}

/* FIXME: MTU in config. */
#define GOOD_PACKET_LEN (ETH_HLEN + VLAN_HLEN + ETH_DATA_LEN)
#define GOOD_COPY_LEN	128

#define VIRTNET_RX_PAD (NET_IP_ALIGN + NET_SKB_PAD)

/* Separating two types of XDP xmit */
#define VIRTIO_XDP_TX		BIT(0)
#define VIRTIO_XDP_REDIR	BIT(1)

/* RX packet size EWMA. The average packet size is used to determine the packet
 * buffer size when refilling RX rings. As the entire RX ring may be refilled
 * at once, the weight is chosen so that the EWMA will be insensitive to short-
 * term, transient changes in packet size.
 */
DECLARE_EWMA(pkt_len, 0, 64)

#define VIRTNET_DRIVER_VERSION "1.0.0"

static const unsigned long guest_offloads[] = {
	VIRTIO_NET_F_GUEST_TSO4,
	VIRTIO_NET_F_GUEST_TSO6,
	VIRTIO_NET_F_GUEST_ECN,
	VIRTIO_NET_F_GUEST_UFO,
	VIRTIO_NET_F_GUEST_CSUM,
	VIRTIO_NET_F_GUEST_USO4,
	VIRTIO_NET_F_GUEST_USO6,
	VIRTIO_NET_F_GUEST_HDRLEN,
	VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO_MAPPED,
	VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO_CSUM_MAPPED,
};

#define GUEST_OFFLOAD_GRO_HW_MASK ((1ULL << VIRTIO_NET_F_GUEST_TSO4) | \
			(1ULL << VIRTIO_NET_F_GUEST_TSO6) | \
			(1ULL << VIRTIO_NET_F_GUEST_ECN)  | \
			(1ULL << VIRTIO_NET_F_GUEST_UFO)  | \
			(1ULL << VIRTIO_NET_F_GUEST_USO4) | \
			(1ULL << VIRTIO_NET_F_GUEST_USO6) | \
			(1ULL << VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO_MAPPED) | \
			(1ULL << VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO_CSUM_MAPPED))

struct virtnet_stat_desc {
	char desc[ETH_GSTRING_LEN];
	size_t offset;
	size_t qstat_offset;
};

struct virtnet_sq_free_stats {
	u64 packets;
	u64 bytes;
	u64 napi_packets;
	u64 napi_bytes;
	u64 xsk;
};

struct virtnet_sq_stats {
	struct u64_stats_sync syncp;
	u64_stats_t packets;
	u64_stats_t bytes;
	u64_stats_t xdp_tx;
	u64_stats_t xdp_tx_drops;
	u64_stats_t kicks;
	u64_stats_t tx_timeouts;
	u64_stats_t stop;
	u64_stats_t wake;
};

struct virtnet_rq_stats {
	struct u64_stats_sync syncp;
	u64_stats_t packets;
	u64_stats_t bytes;
	u64_stats_t drops;
	u64_stats_t xdp_packets;
	u64_stats_t xdp_tx;
	u64_stats_t xdp_redirects;
	u64_stats_t xdp_drops;
	u64_stats_t kicks;
};

#define VIRTNET_SQ_STAT(name, m) {name, offsetof(struct virtnet_sq_stats, m), -1}
#define VIRTNET_RQ_STAT(name, m) {name, offsetof(struct virtnet_rq_stats, m), -1}

#define VIRTNET_SQ_STAT_QSTAT(name, m)				\
	{							\
		name,						\
		offsetof(struct virtnet_sq_stats, m),		\
		offsetof(struct netdev_queue_stats_tx, m),	\
	}

#define VIRTNET_RQ_STAT_QSTAT(name, m)				\
	{							\
		name,						\
		offsetof(struct virtnet_rq_stats, m),		\
		offsetof(struct netdev_queue_stats_rx, m),	\
	}

static const struct virtnet_stat_desc virtnet_sq_stats_desc[] = {
	VIRTNET_SQ_STAT("xdp_tx",       xdp_tx),
	VIRTNET_SQ_STAT("xdp_tx_drops", xdp_tx_drops),
	VIRTNET_SQ_STAT("kicks",        kicks),
	VIRTNET_SQ_STAT("tx_timeouts",  tx_timeouts),
};

static const struct virtnet_stat_desc virtnet_rq_stats_desc[] = {
	VIRTNET_RQ_STAT("drops",         drops),
	VIRTNET_RQ_STAT("xdp_packets",   xdp_packets),
	VIRTNET_RQ_STAT("xdp_tx",        xdp_tx),
	VIRTNET_RQ_STAT("xdp_redirects", xdp_redirects),
	VIRTNET_RQ_STAT("xdp_drops",     xdp_drops),
	VIRTNET_RQ_STAT("kicks",         kicks),
};

static const struct virtnet_stat_desc virtnet_sq_stats_desc_qstat[] = {
	VIRTNET_SQ_STAT_QSTAT("packets", packets),
	VIRTNET_SQ_STAT_QSTAT("bytes",   bytes),
	VIRTNET_SQ_STAT_QSTAT("stop",	 stop),
	VIRTNET_SQ_STAT_QSTAT("wake",	 wake),
};

static const struct virtnet_stat_desc virtnet_rq_stats_desc_qstat[] = {
	VIRTNET_RQ_STAT_QSTAT("packets", packets),
	VIRTNET_RQ_STAT_QSTAT("bytes",   bytes),
};

#define VIRTNET_STATS_DESC_CQ(name) \
	{#name, offsetof(struct virtio_net_stats_cvq, name), -1}

#define VIRTNET_STATS_DESC_RX(class, name) \
	{#name, offsetof(struct virtio_net_stats_rx_ ## class, rx_ ## name), -1}

#define VIRTNET_STATS_DESC_TX(class, name) \
	{#name, offsetof(struct virtio_net_stats_tx_ ## class, tx_ ## name), -1}


static const struct virtnet_stat_desc virtnet_stats_cvq_desc[] = {
	VIRTNET_STATS_DESC_CQ(command_num),
	VIRTNET_STATS_DESC_CQ(ok_num),
};

static const struct virtnet_stat_desc virtnet_stats_rx_basic_desc[] = {
	VIRTNET_STATS_DESC_RX(basic, packets),
	VIRTNET_STATS_DESC_RX(basic, bytes),

	VIRTNET_STATS_DESC_RX(basic, notifications),
	VIRTNET_STATS_DESC_RX(basic, interrupts),
};

static const struct virtnet_stat_desc virtnet_stats_tx_basic_desc[] = {
	VIRTNET_STATS_DESC_TX(basic, packets),
	VIRTNET_STATS_DESC_TX(basic, bytes),

	VIRTNET_STATS_DESC_TX(basic, notifications),
	VIRTNET_STATS_DESC_TX(basic, interrupts),
};

static const struct virtnet_stat_desc virtnet_stats_rx_csum_desc[] = {
	VIRTNET_STATS_DESC_RX(csum, needs_csum),
};

static const struct virtnet_stat_desc virtnet_stats_tx_gso_desc[] = {
	VIRTNET_STATS_DESC_TX(gso, gso_packets_noseg),
	VIRTNET_STATS_DESC_TX(gso, gso_bytes_noseg),
};

static const struct virtnet_stat_desc virtnet_stats_rx_speed_desc[] = {
	VIRTNET_STATS_DESC_RX(speed, ratelimit_bytes),
};

static const struct virtnet_stat_desc virtnet_stats_tx_speed_desc[] = {
	VIRTNET_STATS_DESC_TX(speed, ratelimit_bytes),
};

#define VIRTNET_STATS_DESC_RX_QSTAT(class, name, qstat_field)			\
	{									\
		#name,								\
		offsetof(struct virtio_net_stats_rx_ ## class, rx_ ## name),	\
		offsetof(struct netdev_queue_stats_rx, qstat_field),		\
	}

#define VIRTNET_STATS_DESC_TX_QSTAT(class, name, qstat_field)			\
	{									\
		#name,								\
		offsetof(struct virtio_net_stats_tx_ ## class, tx_ ## name),	\
		offsetof(struct netdev_queue_stats_tx, qstat_field),		\
	}

static const struct virtnet_stat_desc virtnet_stats_rx_basic_desc_qstat[] = {
	VIRTNET_STATS_DESC_RX_QSTAT(basic, drops,         hw_drops),
	VIRTNET_STATS_DESC_RX_QSTAT(basic, drop_overruns, hw_drop_overruns),
};

static const struct virtnet_stat_desc virtnet_stats_tx_basic_desc_qstat[] = {
	VIRTNET_STATS_DESC_TX_QSTAT(basic, drops,          hw_drops),
	VIRTNET_STATS_DESC_TX_QSTAT(basic, drop_malformed, hw_drop_errors),
};

static const struct virtnet_stat_desc virtnet_stats_rx_csum_desc_qstat[] = {
	VIRTNET_STATS_DESC_RX_QSTAT(csum, csum_valid, csum_unnecessary),
	VIRTNET_STATS_DESC_RX_QSTAT(csum, csum_none,  csum_none),
	VIRTNET_STATS_DESC_RX_QSTAT(csum, csum_bad,   csum_bad),
};

static const struct virtnet_stat_desc virtnet_stats_tx_csum_desc_qstat[] = {
	VIRTNET_STATS_DESC_TX_QSTAT(csum, csum_none,  csum_none),
	VIRTNET_STATS_DESC_TX_QSTAT(csum, needs_csum, needs_csum),
};

static const struct virtnet_stat_desc virtnet_stats_rx_gso_desc_qstat[] = {
	VIRTNET_STATS_DESC_RX_QSTAT(gso, gso_packets,           hw_gro_packets),
	VIRTNET_STATS_DESC_RX_QSTAT(gso, gso_bytes,             hw_gro_bytes),
	VIRTNET_STATS_DESC_RX_QSTAT(gso, gso_packets_coalesced, hw_gro_wire_packets),
	VIRTNET_STATS_DESC_RX_QSTAT(gso, gso_bytes_coalesced,   hw_gro_wire_bytes),
};

static const struct virtnet_stat_desc virtnet_stats_tx_gso_desc_qstat[] = {
	VIRTNET_STATS_DESC_TX_QSTAT(gso, gso_packets,        hw_gso_packets),
	VIRTNET_STATS_DESC_TX_QSTAT(gso, gso_bytes,          hw_gso_bytes),
	VIRTNET_STATS_DESC_TX_QSTAT(gso, gso_segments,       hw_gso_wire_packets),
	VIRTNET_STATS_DESC_TX_QSTAT(gso, gso_segments_bytes, hw_gso_wire_bytes),
};

static const struct virtnet_stat_desc virtnet_stats_rx_speed_desc_qstat[] = {
	VIRTNET_STATS_DESC_RX_QSTAT(speed, ratelimit_packets, hw_drop_ratelimits),
};

static const struct virtnet_stat_desc virtnet_stats_tx_speed_desc_qstat[] = {
	VIRTNET_STATS_DESC_TX_QSTAT(speed, ratelimit_packets, hw_drop_ratelimits),
};

#define VIRTNET_Q_TYPE_RX 0
#define VIRTNET_Q_TYPE_TX 1
#define VIRTNET_Q_TYPE_CQ 2

struct virtnet_interrupt_coalesce {
	u32 max_packets;
	u32 max_usecs;
};

/* Internal representation of a send virtqueue */
struct send_queue {
	/* Virtqueue associated with this send _queue */
	struct virtqueue *vq;

	/* TX: fragments + linear part + virtio header */
	struct scatterlist sg[MAX_SKB_FRAGS + 2];

	/* Name of the send queue: output.$index */
	char name[16];

	struct virtnet_sq_stats stats;

	struct virtnet_interrupt_coalesce intr_coal;

	struct napi_struct napi;

	/* Record whether sq is in reset state. */
	bool reset;

	struct xsk_buff_pool *xsk_pool;

	dma_addr_t xsk_hdr_dma_addr;
};

/* Internal representation of a receive virtqueue */
struct receive_queue {
	/* Virtqueue associated with this receive_queue */
	struct virtqueue *vq;

	struct napi_struct napi;

	struct bpf_prog __rcu *xdp_prog;

	struct virtnet_rq_stats stats;

	/* The number of rx notifications */
	u16 calls;

	/* Is dynamic interrupt moderation enabled? */
	bool dim_enabled;

	/* Used to protect dim_enabled and inter_coal */
	struct mutex dim_lock;

	/* Dynamic Interrupt Moderation */
	struct dim dim;

	u32 packets_in_napi;

	struct virtnet_interrupt_coalesce intr_coal;

	/* Chain pages by the private ptr. */
	struct page *pages;

	/* Average packet length for mergeable receive buffers. */
	struct ewma_pkt_len mrg_avg_pkt_len;

	struct page_pool *page_pool;

	/* True if page_pool handles DMA mapping via PP_FLAG_DMA_MAP */
	bool use_page_pool_dma;

	/* RX: fragments + linear part + virtio header */
	struct scatterlist sg[MAX_SKB_FRAGS + 2];

	/* Min single buffer size for mergeable buffers case. */
	unsigned int min_buf_len;

	/* Name of this receive queue: input.$index */
	char name[16];

	struct xdp_rxq_info xdp_rxq;

	struct xsk_buff_pool *xsk_pool;

	/* xdp rxq used by xsk */
	struct xdp_rxq_info xsk_rxq_info;

	struct xdp_buff **xsk_buffs;
};

/* Control VQ buffers: protected by the rtnl lock */
struct control_buf {
	struct virtio_net_ctrl_hdr hdr;
	virtio_net_ctrl_ack status;
};

struct virtnet_info {
	struct virtio_device *vdev;
	struct virtqueue *cvq;
	struct net_device *dev;
	struct send_queue *sq;
	struct receive_queue *rq;
	unsigned int status;

	/* Max # of queue pairs supported by the device */
	u16 max_queue_pairs;

	/* # of queue pairs currently used by the driver */
	u16 curr_queue_pairs;

	/* # of XDP queue pairs currently used by the driver */
	u16 xdp_queue_pairs;

	/* xdp_queue_pairs may be 0, when xdp is already loaded. So add this. */
	bool xdp_enabled;

	/* I like... big packets and I cannot lie! */
	bool big_packets;

	/* number of sg entries allocated for big packets */
	unsigned int big_packets_num_skbfrags;

	/* Host will merge rx buffers for big packets (shake it! shake it!) */
	bool mergeable_rx_bufs;

	/* Host supports rss and/or hash report */
	bool has_rss;
	bool has_rss_hash_report;
	u8 rss_key_size;
	u16 rss_indir_table_size;
	u32 rss_hash_types_supported;
	u32 rss_hash_types_saved;

	/* Has control virtqueue */
	bool has_cvq;

	/* Lock to protect the control VQ */
	struct mutex cvq_lock;

	/* Host can handle any s/g split between our header and packet data */
	bool any_header_sg;

	/* Packet virtio header size */
	u8 hdr_len;

	/* UDP tunnel support */
	bool tx_tnl;

	bool rx_tnl;

	bool rx_tnl_csum;

	/* Work struct for config space updates */
	struct work_struct config_work;

	/* Work struct for setting rx mode */
	struct work_struct rx_mode_work;

	/* OK to queue work setting RX mode? */
	bool rx_mode_work_enabled;

	/* Does the affinity hint is set for virtqueues? */
	bool affinity_hint_set;

	/* CPU hotplug instances for online & dead */
	struct hlist_node node;
	struct hlist_node node_dead;

	struct control_buf *ctrl;

	/* Ethtool settings */
	u8 duplex;
	u32 speed;

	/* Is rx dynamic interrupt moderation enabled? */
	bool rx_dim_enabled;

	/* Interrupt coalescing settings */
	struct virtnet_interrupt_coalesce intr_coal_tx;
	struct virtnet_interrupt_coalesce intr_coal_rx;

	unsigned long guest_offloads;
	unsigned long guest_offloads_capable;

	/* failover when STANDBY feature enabled */
	struct failover *failover;

	u64 device_stats_cap;

	struct virtio_net_rss_config_hdr *rss_hdr;

	/* Must be last as it ends in a flexible-array member. */
	TRAILING_OVERLAP(struct virtio_net_rss_config_trailer, rss_trailer, hash_key_data,
		u8 rss_hash_key_data[NETDEV_RSS_KEY_LEN];
	);
};
static_assert(offsetof(struct virtnet_info, rss_trailer.hash_key_data) ==
	      offsetof(struct virtnet_info, rss_hash_key_data));

struct padded_vnet_hdr {
	struct virtio_net_hdr_v1_hash hdr;
	/*
	 * hdr is in a separate sg buffer, and data sg buffer shares same page
	 * with this header sg. This padding makes next sg 16 byte aligned
	 * after the header.
	 */
	char padding[12];
};

struct virtio_net_common_hdr {
	union {
		struct virtio_net_hdr hdr;
		struct virtio_net_hdr_mrg_rxbuf	mrg_hdr;
		struct virtio_net_hdr_v1_hash hash_v1_hdr;
		struct virtio_net_hdr_v1_hash_tunnel tnl_hdr;
	};
};

static struct virtio_net_common_hdr xsk_hdr;

static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf);
static void virtnet_sq_free_unused_buf_done(struct virtqueue *vq);
static int virtnet_xdp_handler(struct bpf_prog *xdp_prog, struct xdp_buff *xdp,
			       struct net_device *dev,
			       unsigned int *xdp_xmit,
			       struct virtnet_rq_stats *stats);
static void virtnet_receive_done(struct virtnet_info *vi, struct receive_queue *rq,
				 struct sk_buff *skb, u8 flags);
static struct sk_buff *virtnet_skb_append_frag(struct receive_queue *rq,
					       struct sk_buff *head_skb,
					       struct sk_buff *curr_skb,
					       struct page *page, void *buf,
					       int len, int truesize);
static void virtnet_xsk_completed(struct send_queue *sq, int num);
static void free_unused_bufs(struct virtnet_info *vi);
static void virtnet_del_vqs(struct virtnet_info *vi);

enum virtnet_xmit_type {
	VIRTNET_XMIT_TYPE_SKB,
	VIRTNET_XMIT_TYPE_SKB_ORPHAN,
	VIRTNET_XMIT_TYPE_XDP,
	VIRTNET_XMIT_TYPE_XSK,
};

static size_t virtnet_rss_hdr_size(const struct virtnet_info *vi)
{
	u16 indir_table_size = vi->has_rss ? vi->rss_indir_table_size : 1;

	return struct_size(vi->rss_hdr, indirection_table, indir_table_size);
}

static size_t virtnet_rss_trailer_size(const struct virtnet_info *vi)
{
	return struct_size(&vi->rss_trailer, hash_key_data, vi->rss_key_size);
}

/* We use the last two bits of the pointer to distinguish the xmit type. */
#define VIRTNET_XMIT_TYPE_MASK (BIT(0) | BIT(1))

#define VIRTIO_XSK_FLAG_OFFSET 2

static enum virtnet_xmit_type virtnet_xmit_ptr_unpack(void **ptr)
{
	unsigned long p = (unsigned long)*ptr;

	*ptr = (void *)(p & ~VIRTNET_XMIT_TYPE_MASK);

	return p & VIRTNET_XMIT_TYPE_MASK;
}

static void *virtnet_xmit_ptr_pack(void *ptr, enum virtnet_xmit_type type)
{
	return (void *)((unsigned long)ptr | type);
}

static int virtnet_add_outbuf(struct send_queue *sq, int num, void *data,
			      enum virtnet_xmit_type type)
{
	return virtqueue_add_outbuf(sq->vq, sq->sg, num,
				    virtnet_xmit_ptr_pack(data, type),
				    GFP_ATOMIC);
}

static u32 virtnet_ptr_to_xsk_buff_len(void *ptr)
{
	return ((unsigned long)ptr) >> VIRTIO_XSK_FLAG_OFFSET;
}

static void sg_fill_dma(struct scatterlist *sg, dma_addr_t addr, u32 len)
{
	sg_dma_address(sg) = addr;
	sg_dma_len(sg) = len;
}

static void __free_old_xmit(struct send_queue *sq, struct netdev_queue *txq,
			    bool in_napi, struct virtnet_sq_free_stats *stats)
{
	struct xdp_frame *frame;
	struct sk_buff *skb;
	unsigned int len;
	void *ptr;

	while ((ptr = virtqueue_get_buf(sq->vq, &len)) != NULL) {
		switch (virtnet_xmit_ptr_unpack(&ptr)) {
		case VIRTNET_XMIT_TYPE_SKB:
			skb = ptr;

			pr_debug("Sent skb %p\n", skb);
			stats->napi_packets++;
			stats->napi_bytes += skb->len;
			napi_consume_skb(skb, in_napi);
			break;

		case VIRTNET_XMIT_TYPE_SKB_ORPHAN:
			skb = ptr;

			stats->packets++;
			stats->bytes += skb->len;
			napi_consume_skb(skb, in_napi);
			break;

		case VIRTNET_XMIT_TYPE_XDP:
			frame = ptr;

			stats->packets++;
			stats->bytes += xdp_get_frame_len(frame);
			xdp_return_frame(frame);
			break;

		case VIRTNET_XMIT_TYPE_XSK:
			stats->bytes += virtnet_ptr_to_xsk_buff_len(ptr);
			stats->xsk++;
			break;
		}
	}
	netdev_tx_completed_queue(txq, stats->napi_packets, stats->napi_bytes);
}

static void virtnet_free_old_xmit(struct send_queue *sq,
				  struct netdev_queue *txq,
				  bool in_napi,
				  struct virtnet_sq_free_stats *stats)
{
	__free_old_xmit(sq, txq, in_napi, stats);

	if (stats->xsk)
		virtnet_xsk_completed(sq, stats->xsk);
}

/* Converting between virtqueue no. and kernel tx/rx queue no.
 * 0:rx0 1:tx0 2:rx1 3:tx1 ... 2N:rxN 2N+1:txN 2N+2:cvq
 */
static int vq2txq(struct virtqueue *vq)
{
	return (vq->index - 1) / 2;
}

static int txq2vq(int txq)
{
	return txq * 2 + 1;
}

static int vq2rxq(struct virtqueue *vq)
{
	return vq->index / 2;
}

static int rxq2vq(int rxq)
{
	return rxq * 2;
}

static int vq_type(struct virtnet_info *vi, int qid)
{
	if (qid == vi->max_queue_pairs * 2)
		return VIRTNET_Q_TYPE_CQ;

	if (qid % 2)
		return VIRTNET_Q_TYPE_TX;

	return VIRTNET_Q_TYPE_RX;
}

static inline struct virtio_net_common_hdr *
skb_vnet_common_hdr(struct sk_buff *skb)
{
	return (struct virtio_net_common_hdr *)skb->cb;
}

/*
 * private is used to chain pages for big packets, put the whole
 * most recent used list in the beginning for reuse
 */
static void give_pages(struct receive_queue *rq, struct page *page)
{
	struct page *end;

	/* Find end of list, sew whole thing into vi->rq.pages. */
	for (end = page; end->private; end = (struct page *)end->private);
	end->private = (unsigned long)rq->pages;
	rq->pages = page;
}

static struct page *get_a_page(struct receive_queue *rq, gfp_t gfp_mask)
{
	struct page *p = rq->pages;

	if (p) {
		rq->pages = (struct page *)p->private;
		/* clear private here, it is used to chain pages */
		p->private = 0;
	} else
		p = alloc_page(gfp_mask);
	return p;
}

static void virtnet_rq_free_buf(struct virtnet_info *vi,
				struct receive_queue *rq, void *buf)
{
	if (!rq->page_pool)
		give_pages(rq, buf);
	else
		page_pool_put_page(rq->page_pool, virt_to_head_page(buf), -1, false);
}

static void enable_rx_mode_work(struct virtnet_info *vi)
{
	rtnl_lock();
	vi->rx_mode_work_enabled = true;
	rtnl_unlock();
}

static void disable_rx_mode_work(struct virtnet_info *vi)
{
	rtnl_lock();
	vi->rx_mode_work_enabled = false;
	rtnl_unlock();
}

static void virtqueue_napi_schedule(struct napi_struct *napi,
				    struct virtqueue *vq)
{
	if (napi_schedule_prep(napi)) {
		virtqueue_disable_cb(vq);
		__napi_schedule(napi);
	}
}

static bool virtqueue_napi_complete(struct napi_struct *napi,
				    struct virtqueue *vq, int processed)
{
	int opaque;

	opaque = virtqueue_enable_cb_prepare(vq);
	if (napi_complete_done(napi, processed)) {
		if (unlikely(virtqueue_poll(vq, opaque)))
			virtqueue_napi_schedule(napi, vq);
		else
			return true;
	} else {
		virtqueue_disable_cb(vq);
	}

	return false;
}

static void virtnet_tx_wake_queue(struct virtnet_info *vi,
				struct send_queue *sq)
{
	unsigned int index = vq2txq(sq->vq);
	struct netdev_queue *txq = netdev_get_tx_queue(vi->dev, index);

	if (netif_tx_queue_stopped(txq)) {
		u64_stats_update_begin(&sq->stats.syncp);
		u64_stats_inc(&sq->stats.wake);
		u64_stats_update_end(&sq->stats.syncp);
		netif_tx_wake_queue(txq);
	}
}

static void skb_xmit_done(struct virtqueue *vq)
{
	struct virtnet_info *vi = vq->vdev->priv;
	unsigned int index = vq2txq(vq);
	struct send_queue *sq = &vi->sq[index];
	struct napi_struct *napi = &sq->napi;

	/* Suppress further interrupts. */
	virtqueue_disable_cb(vq);

	if (napi->weight)
		virtqueue_napi_schedule(napi, vq);
	else
		virtnet_tx_wake_queue(vi, sq);
}

#define MRG_CTX_HEADER_SHIFT 22
static void *mergeable_len_to_ctx(unsigned int truesize,
				  unsigned int headroom)
{
	return (void *)(unsigned long)((headroom << MRG_CTX_HEADER_SHIFT) | truesize);
}

static unsigned int mergeable_ctx_to_headroom(void *mrg_ctx)
{
	return (unsigned long)mrg_ctx >> MRG_CTX_HEADER_SHIFT;
}

static unsigned int mergeable_ctx_to_truesize(void *mrg_ctx)
{
	return (unsigned long)mrg_ctx & ((1 << MRG_CTX_HEADER_SHIFT) - 1);
}

static int check_mergeable_len(struct net_device *dev, void *mrg_ctx,
			       unsigned int len)
{
	unsigned int headroom, tailroom, room, truesize;

	truesize = mergeable_ctx_to_truesize(mrg_ctx);
	headroom = mergeable_ctx_to_headroom(mrg_ctx);
	tailroom = headroom ? sizeof(struct skb_shared_info) : 0;
	room = SKB_DATA_ALIGN(headroom + tailroom);

	if (len > truesize - room) {
		pr_debug("%s: rx error: len %u exceeds truesize %lu\n",
			 dev->name, len, (unsigned long)(truesize - room));
		DEV_STATS_INC(dev, rx_length_errors);
		return -1;
	}

	return 0;
}

static struct sk_buff *virtnet_build_skb(void *buf, unsigned int buflen,
					 unsigned int headroom,
					 unsigned int len)
{
	struct sk_buff *skb;

	skb = build_skb(buf, buflen);
	if (unlikely(!skb))
		return NULL;

	skb_reserve(skb, headroom);
	skb_put(skb, len);

	return skb;
}

/* Called from bottom half context */
static struct sk_buff *page_to_skb(struct virtnet_info *vi,
				   struct receive_queue *rq,
				   struct page *page, unsigned int offset,
				   unsigned int len, unsigned int truesize,
				   unsigned int headroom)
{
	struct sk_buff *skb;
	struct virtio_net_common_hdr *hdr;
	unsigned int copy, hdr_len, hdr_padded_len;
	struct page *page_to_free = NULL;
	int tailroom, shinfo_size;
	char *p, *hdr_p, *buf;

	p = page_address(page) + offset;
	hdr_p = p;

	hdr_len = vi->hdr_len;
	if (vi->mergeable_rx_bufs)
		hdr_padded_len = hdr_len;
	else
		hdr_padded_len = sizeof(struct padded_vnet_hdr);

	buf = p - headroom;
	len -= hdr_len;
	offset += hdr_padded_len;
	p += hdr_padded_len;
	tailroom = truesize - headroom  - hdr_padded_len - len;

	shinfo_size = SKB_DATA_ALIGN(sizeof(struct skb_shared_info));

	if (!NET_IP_ALIGN && len > GOOD_COPY_LEN && tailroom >= shinfo_size) {
		skb = virtnet_build_skb(buf, truesize, p - buf, len);
		if (unlikely(!skb))
			return NULL;
		/* Big packets mode chains pages via page->private, which is
		 * incompatible with the way page_pool uses page->private.
		 * Currently, big packets mode doesn't use page pools.
		 */
		if (!rq->page_pool) {
			page = (struct page *)page->private;
			if (page)
				give_pages(rq, page);
		}

		goto ok;
	}

	/* copy small packet so we can reuse these pages for small data */
	skb = napi_alloc_skb(&rq->napi, GOOD_COPY_LEN);
	if (unlikely(!skb))
		return NULL;

	/* Copy all frame if it fits skb->head, otherwise
	 * we let virtio_net_hdr_to_skb() and GRO pull headers as needed.
	 */
	if (len <= skb_tailroom(skb))
		copy = len;
	else
		copy = ETH_HLEN;
	skb_put_data(skb, p, copy);

	len -= copy;
	offset += copy;

	if (vi->mergeable_rx_bufs) {
		if (len)
			skb_add_rx_frag(skb, 0, page, offset, len, truesize);
		else
			page_to_free = page;
		goto ok;
	}

	BUG_ON(offset >= PAGE_SIZE);
	while (len) {
		unsigned int frag_size = min((unsigned)PAGE_SIZE - offset, len);
		skb_add_rx_frag(skb, skb_shinfo(skb)->nr_frags, page, offset,
				frag_size, truesize);
		len -= frag_size;
		page = (struct page *)page->private;
		offset = 0;
	}

	if (page)
		give_pages(rq, page);

ok:
	hdr = skb_vnet_common_hdr(skb);
	memcpy(hdr, hdr_p, hdr_len);
	if (page_to_free)
		page_pool_put_page(rq->page_pool, page_to_free, -1, true);

	return skb;
}

static void *virtnet_rq_get_buf(struct receive_queue *rq, u32 *len, void **ctx)
{
	BUG_ON(!rq->page_pool);

	return virtqueue_get_buf_ctx(rq->vq, len, ctx);
}

static void virtnet_rq_unmap_free_buf(struct virtqueue *vq, void *buf)
{
	struct virtnet_info *vi = vq->vdev->priv;
	struct receive_queue *rq;
	int i = vq2rxq(vq);

	rq = &vi->rq[i];

	if (rq->xsk_pool) {
		xsk_buff_free((struct xdp_buff *)buf);
		return;
	}

	virtnet_rq_free_buf(vi, rq, buf);
}

static void free_old_xmit(struct send_queue *sq, struct netdev_queue *txq,
			  bool in_napi)
{
	struct virtnet_sq_free_stats stats = {0};

	virtnet_free_old_xmit(sq, txq, in_napi, &stats);

	/* Avoid overhead when no packets have been processed
	 * happens when called speculatively from start_xmit.
	 */
	if (!stats.packets && !stats.napi_packets)
		return;

	u64_stats_update_begin(&sq->stats.syncp);
	u64_stats_add(&sq->stats.bytes, stats.bytes + stats.napi_bytes);
	u64_stats_add(&sq->stats.packets, stats.packets + stats.napi_packets);
	u64_stats_update_end(&sq->stats.syncp);
}

static bool is_xdp_raw_buffer_queue(struct virtnet_info *vi, int q)
{
	if (q < (vi->curr_queue_pairs - vi->xdp_queue_pairs))
		return false;
	else if (q < vi->curr_queue_pairs)
		return true;
	else
		return false;
}

static bool tx_may_stop(struct virtnet_info *vi,
			struct net_device *dev,
			struct send_queue *sq)
{
	int qnum;

	qnum = sq - vi->sq;

	/* If running out of space, stop queue to avoid getting packets that we
	 * are then unable to transmit.
	 * An alternative would be to force queuing layer to requeue the skb by
	 * returning NETDEV_TX_BUSY. However, NETDEV_TX_BUSY should not be
	 * returned in a normal path of operation: it means that driver is not
	 * maintaining the TX queue stop/start state properly, and causes
	 * the stack to do a non-trivial amount of useless work.
	 * Since most packets only take 1 or 2 ring slots, stopping the queue
	 * early means 16 slots are typically wasted.
	 */
	if (sq->vq->num_free < MAX_SKB_FRAGS + 2) {
		struct netdev_queue *txq = netdev_get_tx_queue(dev, qnum);

		netif_tx_stop_queue(txq);
		u64_stats_update_begin(&sq->stats.syncp);
		u64_stats_inc(&sq->stats.stop);
		u64_stats_update_end(&sq->stats.syncp);

		return true;
	}

	return false;
}

static void check_sq_full_and_disable(struct virtnet_info *vi,
				      struct net_device *dev,
				      struct send_queue *sq)
{
	bool use_napi = sq->napi.weight;
	int qnum;

	qnum = sq - vi->sq;

	if (tx_may_stop(vi, dev, sq)) {
		struct netdev_queue *txq = netdev_get_tx_queue(dev, qnum);

		if (use_napi) {
			if (unlikely(!virtqueue_enable_cb_delayed(sq->vq)))
				virtqueue_napi_schedule(&sq->napi, sq->vq);
		} else if (unlikely(!virtqueue_enable_cb_delayed(sq->vq))) {
			/* More just got used, free them then recheck. */
			free_old_xmit(sq, txq, false);
			if (sq->vq->num_free >= MAX_SKB_FRAGS + 2) {
				netif_start_subqueue(dev, qnum);
				u64_stats_update_begin(&sq->stats.syncp);
				u64_stats_inc(&sq->stats.wake);
				u64_stats_update_end(&sq->stats.syncp);
				virtqueue_disable_cb(sq->vq);
			}
		}
	}
}

/* Note that @len is the length of received data without virtio header */
static struct xdp_buff *buf_to_xdp(struct virtnet_info *vi,
				   struct receive_queue *rq, void *buf,
				   u32 len, bool first_buf)
{
	struct xdp_buff *xdp;
	u32 bufsize;

	xdp = (struct xdp_buff *)buf;

	/* In virtnet_add_recvbuf_xsk, we use part of XDP_PACKET_HEADROOM for
	 * virtio header and ask the vhost to fill data from
	 *         hard_start + XDP_PACKET_HEADROOM - vi->hdr_len
	 * The first buffer has virtio header so the remaining region for frame
	 * data is
	 *         xsk_pool_get_rx_frame_size()
	 * While other buffers than the first one do not have virtio header, so
	 * the maximum frame data's length can be
	 *         xsk_pool_get_rx_frame_size() + vi->hdr_len
	 */
	bufsize = xsk_pool_get_rx_frame_size(rq->xsk_pool);
	if (!first_buf)
		bufsize += vi->hdr_len;

	if (unlikely(len > bufsize)) {
		pr_debug("%s: rx error: len %u exceeds truesize %u\n",
			 vi->dev->name, len, bufsize);
		DEV_STATS_INC(vi->dev, rx_length_errors);
		xsk_buff_free(xdp);
		return NULL;
	}

	if (first_buf) {
		xsk_buff_set_size(xdp, len);
	} else {
		xdp_prepare_buff(xdp, xdp->data_hard_start,
				 XDP_PACKET_HEADROOM - vi->hdr_len, len, 1);
		xdp->flags = 0;
	}

	xsk_buff_dma_sync_for_cpu(xdp);

	return xdp;
}

static struct sk_buff *xsk_construct_skb(struct receive_queue *rq,
					 struct xdp_buff *xdp)
{
	unsigned int metasize = xdp->data - xdp->data_meta;
	struct sk_buff *skb;
	unsigned int size;

	size = xdp->data_end - xdp->data_hard_start;
	skb = napi_alloc_skb(&rq->napi, size);
	if (unlikely(!skb)) {
		xsk_buff_free(xdp);
		return NULL;
	}

	skb_reserve(skb, xdp->data_meta - xdp->data_hard_start);

	size = xdp->data_end - xdp->data_meta;
	memcpy(__skb_put(skb, size), xdp->data_meta, size);

	if (metasize) {
		__skb_pull(skb, metasize);
		skb_metadata_set(skb, metasize);
	}

	xsk_buff_free(xdp);

	return skb;
}

static struct sk_buff *virtnet_receive_xsk_small(struct net_device *dev, struct virtnet_info *vi,
						 struct receive_queue *rq, struct xdp_buff *xdp,
						 unsigned int *xdp_xmit,
						 struct virtnet_rq_stats *stats)
{
	struct bpf_prog *prog;
	u32 ret;

	ret = XDP_PASS;
	rcu_read_lock();
	prog = rcu_dereference(rq->xdp_prog);
	if (prog)
		ret = virtnet_xdp_handler(prog, xdp, dev, xdp_xmit, stats);
	rcu_read_unlock();

	switch (ret) {
	case XDP_PASS:
		return xsk_construct_skb(rq, xdp);

	case XDP_TX:
	case XDP_REDIRECT:
		return NULL;

	default:
		/* drop packet */
		xsk_buff_free(xdp);
		u64_stats_inc(&stats->drops);
		return NULL;
	}
}

static void xsk_drop_follow_bufs(struct net_device *dev,
				 struct receive_queue *rq,
				 u32 num_buf,
				 struct virtnet_rq_stats *stats)
{
	struct xdp_buff *xdp;
	u32 len;

	while (num_buf-- > 1) {
		xdp = virtqueue_get_buf(rq->vq, &len);
		if (unlikely(!xdp)) {
			pr_debug("%s: rx error: %d buffers missing\n",
				 dev->name, num_buf);
			DEV_STATS_INC(dev, rx_length_errors);
			break;
		}
		u64_stats_add(&stats->bytes, len);
		xsk_buff_free(xdp);
	}
}

static int xsk_append_merge_buffer(struct virtnet_info *vi,
				   struct receive_queue *rq,
				   struct sk_buff *head_skb,
				   u32 num_buf,
				   struct virtio_net_hdr_mrg_rxbuf *hdr,
				   struct virtnet_rq_stats *stats)
{
	struct sk_buff *curr_skb;
	struct xdp_buff *xdp;
	u32 len, truesize;
	struct page *page;
	void *buf;

	curr_skb = head_skb;

	while (--num_buf) {
		buf = virtqueue_get_buf(rq->vq, &len);
		if (unlikely(!buf)) {
			pr_debug("%s: rx error: %d buffers out of %d missing\n",
				 vi->dev->name, num_buf,
				 virtio16_to_cpu(vi->vdev,
						 hdr->num_buffers));
			DEV_STATS_INC(vi->dev, rx_length_errors);
			return -EINVAL;
		}

		u64_stats_add(&stats->bytes, len);

		xdp = buf_to_xdp(vi, rq, buf, len, false);
		if (!xdp)
			goto err;

		buf = napi_alloc_frag(len);
		if (!buf) {
			xsk_buff_free(xdp);
			goto err;
		}

		memcpy(buf, xdp->data, len);

		xsk_buff_free(xdp);

		page = virt_to_page(buf);

		truesize = len;

		curr_skb  = virtnet_skb_append_frag(rq, head_skb, curr_skb, page,
						    buf, len, truesize);
		if (!curr_skb) {
			put_page(page);
			goto err;
		}
	}

	return 0;

err:
	xsk_drop_follow_bufs(vi->dev, rq, num_buf, stats);
	return -EINVAL;
}

static struct sk_buff *virtnet_receive_xsk_merge(struct net_device *dev, struct virtnet_info *vi,
						 struct receive_queue *rq, struct xdp_buff *xdp,
						 unsigned int *xdp_xmit,
						 struct virtnet_rq_stats *stats)
{
	struct virtio_net_hdr_mrg_rxbuf *hdr;
	struct bpf_prog *prog;
	struct sk_buff *skb;
	u32 ret, num_buf;

	hdr = xdp->data - vi->hdr_len;
	num_buf = virtio16_to_cpu(vi->vdev, hdr->num_buffers);

	ret = XDP_PASS;
	rcu_read_lock();
	prog = rcu_dereference(rq->xdp_prog);
	if (prog) {
		/* TODO: support multi buffer. */
		if (num_buf == 1)
			ret = virtnet_xdp_handler(prog, xdp, dev, xdp_xmit,
						  stats);
		else
			ret = XDP_ABORTED;
	}
	rcu_read_unlock();

	switch (ret) {
	case XDP_PASS:
		skb = xsk_construct_skb(rq, xdp);
		if (!skb)
			goto drop_bufs;

		if (xsk_append_merge_buffer(vi, rq, skb, num_buf, hdr, stats)) {
			dev_kfree_skb(skb);
			goto drop;
		}

		return skb;

	case XDP_TX:
	case XDP_REDIRECT:
		return NULL;

	default:
		/* drop packet */
		xsk_buff_free(xdp);
	}

drop_bufs:
	xsk_drop_follow_bufs(dev, rq, num_buf, stats);

drop:
	u64_stats_inc(&stats->drops);
	return NULL;
}

static void virtnet_receive_xsk_buf(struct virtnet_info *vi, struct receive_queue *rq,
				    void *buf, u32 len,
				    unsigned int *xdp_xmit,
				    struct virtnet_rq_stats *stats)
{
	struct net_device *dev = vi->dev;
	struct sk_buff *skb = NULL;
	struct xdp_buff *xdp;
	u8 flags;

	len -= vi->hdr_len;

	u64_stats_add(&stats->bytes, len);

	xdp = buf_to_xdp(vi, rq, buf, len, true);
	if (!xdp)
		return;

	if (unlikely(len < ETH_HLEN)) {
		pr_debug("%s: short packet %i\n", dev->name, len);
		DEV_STATS_INC(dev, rx_length_errors);
		xsk_buff_free(xdp);
		return;
	}

	flags = ((struct virtio_net_common_hdr *)(xdp->data - vi->hdr_len))->hdr.flags;

	if (!vi->mergeable_rx_bufs)
		skb = virtnet_receive_xsk_small(dev, vi, rq, xdp, xdp_xmit, stats);
	else
		skb = virtnet_receive_xsk_merge(dev, vi, rq, xdp, xdp_xmit, stats);

	if (skb)
		virtnet_receive_done(vi, rq, skb, flags);
}

static int virtnet_add_recvbuf_xsk(struct virtnet_info *vi, struct receive_queue *rq,
				   struct xsk_buff_pool *pool, gfp_t gfp)
{
	struct xdp_buff **xsk_buffs;
	dma_addr_t addr;
	int err = 0;
	u32 len, i;
	int num;

	xsk_buffs = rq->xsk_buffs;

	num = xsk_buff_alloc_batch(pool, xsk_buffs, rq->vq->num_free);
	if (!num) {
		if (xsk_uses_need_wakeup(pool)) {
			xsk_set_rx_need_wakeup(pool);
			/* Return 0 instead of -ENOMEM so that NAPI is
			 * descheduled.
			 */
			return 0;
		}

		return -ENOMEM;
	} else {
		xsk_clear_rx_need_wakeup(pool);
	}

	len = xsk_pool_get_rx_frame_size(pool) + vi->hdr_len;

	for (i = 0; i < num; ++i) {
		/* Use the part of XDP_PACKET_HEADROOM as the virtnet hdr space.
		 * We assume XDP_PACKET_HEADROOM is larger than hdr->len.
		 * (see function virtnet_xsk_pool_enable)
		 */
		addr = xsk_buff_xdp_get_dma(xsk_buffs[i]) - vi->hdr_len;

		sg_init_table(rq->sg, 1);
		sg_fill_dma(rq->sg, addr, len);

		err = virtqueue_add_inbuf_premapped(rq->vq, rq->sg, 1,
						    xsk_buffs[i], NULL, gfp);
		if (err)
			goto err;
	}

	return num;

err:
	for (; i < num; ++i)
		xsk_buff_free(xsk_buffs[i]);

	return err;
}

static void *virtnet_xsk_to_ptr(u32 len)
{
	unsigned long p;

	p = len << VIRTIO_XSK_FLAG_OFFSET;

	return virtnet_xmit_ptr_pack((void *)p, VIRTNET_XMIT_TYPE_XSK);
}

static int virtnet_xsk_xmit_one(struct send_queue *sq,
				struct xsk_buff_pool *pool,
				struct xdp_desc *desc)
{
	struct virtnet_info *vi;
	dma_addr_t addr;

	vi = sq->vq->vdev->priv;

	addr = xsk_buff_raw_get_dma(pool, desc->addr);
	xsk_buff_raw_dma_sync_for_device(pool, addr, desc->len);

	sg_init_table(sq->sg, 2);
	sg_fill_dma(sq->sg, sq->xsk_hdr_dma_addr, vi->hdr_len);
	sg_fill_dma(sq->sg + 1, addr, desc->len);

	return virtqueue_add_outbuf_premapped(sq->vq, sq->sg, 2,
					      virtnet_xsk_to_ptr(desc->len),
					      GFP_ATOMIC);
}

static int virtnet_xsk_xmit_batch(struct send_queue *sq,
				  struct xsk_buff_pool *pool,
				  unsigned int budget,
				  u64 *kicks)
{
	struct xdp_desc *descs = pool->tx_descs;
	bool kick = false;
	u32 nb_pkts, i;
	int err;

	budget = min_t(u32, budget, sq->vq->num_free);

	nb_pkts = xsk_tx_peek_release_desc_batch(pool, budget);
	if (!nb_pkts)
		return 0;

	for (i = 0; i < nb_pkts; i++) {
		err = virtnet_xsk_xmit_one(sq, pool, &descs[i]);
		if (unlikely(err)) {
			xsk_tx_completed(sq->xsk_pool, nb_pkts - i);
			break;
		}

		kick = true;
	}

	if (kick && virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq))
		(*kicks)++;

	return i;
}

static bool virtnet_xsk_xmit(struct send_queue *sq, struct xsk_buff_pool *pool,
			     int budget)
{
	struct virtnet_info *vi = sq->vq->vdev->priv;
	struct virtnet_sq_free_stats stats = {};
	struct net_device *dev = vi->dev;
	u64 kicks = 0;
	int sent;

	/* Avoid to wakeup napi meanless, so call __free_old_xmit instead of
	 * free_old_xmit().
	 */
	__free_old_xmit(sq, netdev_get_tx_queue(dev, sq - vi->sq), true, &stats);

	if (stats.xsk)
		xsk_tx_completed(sq->xsk_pool, stats.xsk);

	sent = virtnet_xsk_xmit_batch(sq, pool, budget, &kicks);

	if (!is_xdp_raw_buffer_queue(vi, sq - vi->sq))
		check_sq_full_and_disable(vi, vi->dev, sq);

	if (sent) {
		struct netdev_queue *txq;

		txq = netdev_get_tx_queue(vi->dev, sq - vi->sq);
		txq_trans_cond_update(txq);
	}

	u64_stats_update_begin(&sq->stats.syncp);
	u64_stats_add(&sq->stats.packets, stats.packets);
	u64_stats_add(&sq->stats.bytes,   stats.bytes);
	u64_stats_add(&sq->stats.kicks,   kicks);
	u64_stats_add(&sq->stats.xdp_tx,  sent);
	u64_stats_update_end(&sq->stats.syncp);

	if (xsk_uses_need_wakeup(pool))
		xsk_set_tx_need_wakeup(pool);

	return sent;
}

static void xsk_wakeup(struct napi_struct *napi, struct virtqueue *vq)
{
	if (napi_if_scheduled_mark_missed(napi))
		return;

	local_bh_disable();
	virtqueue_napi_schedule(napi, vq);
	local_bh_enable();
}

static int virtnet_xsk_wakeup(struct net_device *dev, u32 qid, u32 flag)
{
	struct virtnet_info *vi = netdev_priv(dev);

	if (!netif_running(dev))
		return -ENETDOWN;

	if (qid >= vi->curr_queue_pairs)
		return -EINVAL;

	if (flag & XDP_WAKEUP_TX) {
		struct send_queue *sq = &vi->sq[qid];

		xsk_wakeup(&sq->napi, sq->vq);
	}

	if (flag & XDP_WAKEUP_RX) {
		struct receive_queue *rq = &vi->rq[qid];

		xsk_wakeup(&rq->napi, rq->vq);
	}

	return 0;
}

static void virtnet_xsk_completed(struct send_queue *sq, int num)
{
	xsk_tx_completed(sq->xsk_pool, num);

	/* If this is called by rx poll, start_xmit and xdp xmit we should
	 * wakeup the tx napi to consume the xsk tx queue, because the tx
	 * interrupt may not be triggered.
	 */
	xsk_wakeup(&sq->napi, sq->vq);
}

static int __virtnet_xdp_xmit_one(struct virtnet_info *vi,
				   struct send_queue *sq,
				   struct xdp_frame *xdpf)
{
	struct virtio_net_hdr_mrg_rxbuf *hdr;
	struct skb_shared_info *shinfo;
	u8 nr_frags = 0;
	int err, i;

	if (unlikely(xdpf->headroom < vi->hdr_len))
		return -EOVERFLOW;

	if (unlikely(xdp_frame_has_frags(xdpf))) {
		shinfo = xdp_get_shared_info_from_frame(xdpf);
		nr_frags = shinfo->nr_frags;
	}

	/* In wrapping function virtnet_xdp_xmit(), we need to free
	 * up the pending old buffers, where we need to calculate the
	 * position of skb_shared_info in xdp_get_frame_len() and
	 * xdp_return_frame(), which will involve to xdpf->data and
	 * xdpf->headroom. Therefore, we need to update the value of
	 * headroom synchronously here.
	 */
	xdpf->headroom -= vi->hdr_len;
	xdpf->data -= vi->hdr_len;
	/* Zero header and leave csum up to XDP layers */
	hdr = xdpf->data;
	memset(hdr, 0, vi->hdr_len);
	xdpf->len   += vi->hdr_len;

	sg_init_table(sq->sg, nr_frags + 1);
	sg_set_buf(sq->sg, xdpf->data, xdpf->len);
	for (i = 0; i < nr_frags; i++) {
		skb_frag_t *frag = &shinfo->frags[i];

		sg_set_page(&sq->sg[i + 1], skb_frag_page(frag),
			    skb_frag_size(frag), skb_frag_off(frag));
	}

	err = virtnet_add_outbuf(sq, nr_frags + 1, xdpf, VIRTNET_XMIT_TYPE_XDP);
	if (unlikely(err))
		return -ENOSPC; /* Caller handle free/refcnt */

	return 0;
}

/* when vi->curr_queue_pairs > nr_cpu_ids, the txq/sq is only used for xdp tx on
 * the current cpu, so it does not need to be locked.
 *
 * Here we use marco instead of inline functions because we have to deal with
 * three issues at the same time: 1. the choice of sq. 2. judge and execute the
 * lock/unlock of txq 3. make sparse happy. It is difficult for two inline
 * functions to perfectly solve these three problems at the same time.
 */
#define virtnet_xdp_get_sq(vi) ({                                       \
	int cpu = smp_processor_id();                                   \
	struct netdev_queue *txq;                                       \
	typeof(vi) v = (vi);                                            \
	unsigned int qp;                                                \
									\
	if (v->curr_queue_pairs > nr_cpu_ids) {                         \
		qp = v->curr_queue_pairs - v->xdp_queue_pairs;          \
		qp += cpu;                                              \
		txq = netdev_get_tx_queue(v->dev, qp);                  \
		__netif_tx_acquire(txq);                                \
	} else {                                                        \
		qp = cpu % v->curr_queue_pairs;                         \
		txq = netdev_get_tx_queue(v->dev, qp);                  \
		__netif_tx_lock(txq, cpu);                              \
	}                                                               \
	v->sq + qp;                                                     \
})

#define virtnet_xdp_put_sq(vi, q) {                                     \
	struct netdev_queue *txq;                                       \
	typeof(vi) v = (vi);                                            \
									\
	txq = netdev_get_tx_queue(v->dev, (q) - v->sq);                 \
	if (v->curr_queue_pairs > nr_cpu_ids)                           \
		__netif_tx_release(txq);                                \
	else                                                            \
		__netif_tx_unlock(txq);                                 \
}

static int virtnet_xdp_xmit(struct net_device *dev,
			    int n, struct xdp_frame **frames, u32 flags)
{
	struct virtnet_info *vi = netdev_priv(dev);
	struct virtnet_sq_free_stats stats = {0};
	struct receive_queue *rq = vi->rq;
	struct bpf_prog *xdp_prog;
	struct send_queue *sq;
	int nxmit = 0;
	int kicks = 0;
	int ret;
	int i;

	/* Only allow ndo_xdp_xmit if XDP is loaded on dev, as this
	 * indicate XDP resources have been successfully allocated.
	 */
	xdp_prog = rcu_access_pointer(rq->xdp_prog);
	if (!xdp_prog)
		return -ENXIO;

	sq = virtnet_xdp_get_sq(vi);

	if (unlikely(flags & ~XDP_XMIT_FLAGS_MASK)) {
		ret = -EINVAL;
		goto out;
	}

	/* Free up any pending old buffers before queueing new ones. */
	virtnet_free_old_xmit(sq, netdev_get_tx_queue(dev, sq - vi->sq),
			      false, &stats);

	for (i = 0; i < n; i++) {
		struct xdp_frame *xdpf = frames[i];

		if (__virtnet_xdp_xmit_one(vi, sq, xdpf))
			break;
		nxmit++;
	}
	ret = nxmit;

	if (!is_xdp_raw_buffer_queue(vi, sq - vi->sq))
		check_sq_full_and_disable(vi, dev, sq);

	if (flags & XDP_XMIT_FLUSH) {
		if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq))
			kicks = 1;
	}
out:
	u64_stats_update_begin(&sq->stats.syncp);
	u64_stats_add(&sq->stats.bytes, stats.bytes);
	u64_stats_add(&sq->stats.packets, stats.packets);
	u64_stats_add(&sq->stats.xdp_tx, n);
	u64_stats_add(&sq->stats.xdp_tx_drops, n - nxmit);
	u64_stats_add(&sq->stats.kicks, kicks);
	u64_stats_update_end(&sq->stats.syncp);

	virtnet_xdp_put_sq(vi, sq);
	return ret;
}

static void put_xdp_frags(struct receive_queue *rq, struct xdp_buff *xdp)
{
	struct skb_shared_info *shinfo;
	struct page *xdp_page;
	int i;

	if (xdp_buff_has_frags(xdp)) {
		shinfo = xdp_get_shared_info_from_buff(xdp);
		for (i = 0; i < shinfo->nr_frags; i++) {
			xdp_page = skb_frag_page(&shinfo->frags[i]);
			page_pool_put_page(rq->page_pool, xdp_page, -1, true);
		}
	}
}

static int virtnet_xdp_handler(struct bpf_prog *xdp_prog, struct xdp_buff *xdp,
			       struct net_device *dev,
			       unsigned int *xdp_xmit,
			       struct virtnet_rq_stats *stats)
{
	struct xdp_frame *xdpf;
	int err;
	u32 act;

	act = bpf_prog_run_xdp(xdp_prog, xdp);
	u64_stats_inc(&stats->xdp_packets);

	switch (act) {
	case XDP_PASS:
		return act;

	case XDP_TX:
		u64_stats_inc(&stats->xdp_tx);
		xdpf = xdp_convert_buff_to_frame(xdp);
		if (unlikely(!xdpf)) {
			netdev_dbg(dev, "convert buff to frame failed for xdp\n");
			return XDP_DROP;
		}

		err = virtnet_xdp_xmit(dev, 1, &xdpf, 0);
		if (unlikely(!err)) {
			xdp_return_frame_rx_napi(xdpf);
		} else if (unlikely(err < 0)) {
			trace_xdp_exception(dev, xdp_prog, act);
			return XDP_DROP;
		}
		*xdp_xmit |= VIRTIO_XDP_TX;
		return act;

	case XDP_REDIRECT:
		u64_stats_inc(&stats->xdp_redirects);
		err = xdp_do_redirect(dev, xdp, xdp_prog);
		if (err)
			return XDP_DROP;

		*xdp_xmit |= VIRTIO_XDP_REDIR;
		return act;

	default:
		bpf_warn_invalid_xdp_action(dev, xdp_prog, act);
		fallthrough;
	case XDP_ABORTED:
		trace_xdp_exception(dev, xdp_prog, act);
		fallthrough;
	case XDP_DROP:
		return XDP_DROP;
	}
}

static unsigned int virtnet_get_headroom(struct virtnet_info *vi)
{
	return vi->xdp_enabled ? XDP_PACKET_HEADROOM : 0;
}

/* We copy the packet for XDP in the following cases:
 *
 * 1) Packet is scattered across multiple rx buffers.
 * 2) Headroom space is insufficient.
 *
 * This is inefficient but it's a temporary condition that
 * we hit right after XDP is enabled and until queue is refilled
 * with large buffers with sufficient headroom - so it should affect
 * at most queue size packets.
 * Afterwards, the conditions to enable
 * XDP should preclude the underlying device from sending packets
 * across multiple buffers (num_buf > 1), and we make sure buffers
 * have enough headroom.
 */
static struct page *xdp_linearize_page(struct net_device *dev,
				       struct receive_queue *rq,
				       int *num_buf,
				       struct page *p,
				       int offset,
				       int page_off,
				       unsigned int *len)
{
	int tailroom = SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
	struct page *page;

	if (page_off + *len + tailroom > PAGE_SIZE)
		return NULL;

	page = page_pool_alloc_pages(rq->page_pool, GFP_ATOMIC);
	if (!page)
		return NULL;

	memcpy(page_address(page) + page_off, page_address(p) + offset, *len);
	page_off += *len;

	/* Only mergeable mode can go inside this while loop. In small mode,
	 * *num_buf == 1, so it cannot go inside.
	 */
	while (--*num_buf) {
		unsigned int buflen;
		void *buf;
		void *ctx;
		int off;

		buf = virtnet_rq_get_buf(rq, &buflen, &ctx);
		if (unlikely(!buf))
			goto err_buf;

		p = virt_to_head_page(buf);
		off = buf - page_address(p);

		if (rq->use_page_pool_dma)
			page_pool_dma_sync_for_cpu(rq->page_pool, p,
						   off, buflen);

		if (check_mergeable_len(dev, ctx, buflen)) {
			page_pool_put_page(rq->page_pool, p, -1, true);
			goto err_buf;
		}

		/* guard against a misconfigured or uncooperative backend that
		 * is sending packet larger than the MTU.
		 */
		if ((page_off + buflen + tailroom) > PAGE_SIZE) {
			page_pool_put_page(rq->page_pool, p, -1, true);
			goto err_buf;
		}

		memcpy(page_address(page) + page_off,
		       page_address(p) + off, buflen);
		page_off += buflen;
		page_pool_put_page(rq->page_pool, p, -1, true);
	}

	/* Headroom does not contribute to packet length */
	*len = page_off - XDP_PACKET_HEADROOM;
	return page;
err_buf:
	page_pool_put_page(rq->page_pool, page, -1, true);
	return NULL;
}

static struct sk_buff *receive_small_build_skb(struct virtnet_info *vi,
					       unsigned int xdp_headroom,
					       void *buf,
					       unsigned int len,
					       unsigned int buflen)
{
	unsigned int header_offset;
	unsigned int headroom;
	struct sk_buff *skb;

	header_offset = VIRTNET_RX_PAD + xdp_headroom;
	headroom = vi->hdr_len + header_offset;

	skb = virtnet_build_skb(buf, buflen, headroom, len);
	if (unlikely(!skb))
		return NULL;

	buf += header_offset;
	memcpy(skb_vnet_common_hdr(skb), buf, vi->hdr_len);

	return skb;
}

static struct sk_buff *receive_small_xdp(struct net_device *dev,
					 struct virtnet_info *vi,
					 struct receive_queue *rq,
					 struct bpf_prog *xdp_prog,
					 void *buf,
					 unsigned int xdp_headroom,
					 unsigned int len,
					 unsigned int buflen,
					 unsigned int *xdp_xmit,
					 struct virtnet_rq_stats *stats)
{
	unsigned int header_offset = VIRTNET_RX_PAD + xdp_headroom;
	unsigned int headroom = vi->hdr_len + header_offset;
	struct virtio_net_hdr_mrg_rxbuf *hdr = buf + header_offset;
	struct page *page = virt_to_head_page(buf);
	struct page *xdp_page;
	struct xdp_buff xdp;
	struct sk_buff *skb;
	unsigned int metasize = 0;
	u32 act;

	if (unlikely(hdr->hdr.gso_type))
		goto err_xdp;

	/* Partially checksummed packets must be dropped. */
	if (unlikely(hdr->hdr.flags & VIRTIO_NET_HDR_F_NEEDS_CSUM))
		goto err_xdp;

	if (unlikely(xdp_headroom < virtnet_get_headroom(vi))) {
		int offset = buf - page_address(page) + header_offset;
		unsigned int tlen = len + vi->hdr_len;
		int num_buf = 1;

		xdp_headroom = virtnet_get_headroom(vi);
		header_offset = VIRTNET_RX_PAD + xdp_headroom;
		headroom = vi->hdr_len + header_offset;
		buflen = SKB_DATA_ALIGN(GOOD_PACKET_LEN + headroom) +
			SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
		xdp_page = xdp_linearize_page(dev, rq, &num_buf, page,
					      offset, header_offset,
					      &tlen);
		if (!xdp_page)
			goto err_xdp;

		buf = page_address(xdp_page);
		page_pool_put_page(rq->page_pool, page, -1, true);
		page = xdp_page;
	}

	xdp_init_buff(&xdp, buflen, &rq->xdp_rxq);
	xdp_prepare_buff(&xdp, buf + VIRTNET_RX_PAD + vi->hdr_len,
			 xdp_headroom, len, true);

	act = virtnet_xdp_handler(xdp_prog, &xdp, dev, xdp_xmit, stats);

	switch (act) {
	case XDP_PASS:
		/* Recalculate length in case bpf program changed it */
		len = xdp.data_end - xdp.data;
		metasize = xdp.data - xdp.data_meta;
		break;

	case XDP_TX:
	case XDP_REDIRECT:
		goto xdp_xmit;

	default:
		goto err_xdp;
	}

	skb = virtnet_build_skb(buf, buflen, xdp.data - buf, len);
	if (unlikely(!skb))
		goto err;

	if (metasize)
		skb_metadata_set(skb, metasize);

	skb_mark_for_recycle(skb);

	return skb;

err_xdp:
	u64_stats_inc(&stats->xdp_drops);
err:
	u64_stats_inc(&stats->drops);
	page_pool_put_page(rq->page_pool, page, -1, true);
xdp_xmit:
	return NULL;
}

static struct sk_buff *receive_small(struct net_device *dev,
				     struct virtnet_info *vi,
				     struct receive_queue *rq,
				     void *buf, void *ctx,
				     unsigned int len,
				     unsigned int *xdp_xmit,
				     struct virtnet_rq_stats *stats)
{
	unsigned int xdp_headroom = mergeable_ctx_to_headroom(ctx);
	unsigned int buflen = mergeable_ctx_to_truesize(ctx);
	struct page *page = virt_to_head_page(buf);
	struct sk_buff *skb;

	/* We passed the address of virtnet header to virtio-core,
	 * so truncate the padding.
	 */
	buf -= VIRTNET_RX_PAD + xdp_headroom;

	len -= vi->hdr_len;
	u64_stats_add(&stats->bytes, len);

	if (unlikely(len > GOOD_PACKET_LEN)) {
		pr_debug("%s: rx error: len %u exceeds max size %d\n",
			 dev->name, len, GOOD_PACKET_LEN);
		DEV_STATS_INC(dev, rx_length_errors);
		goto err;
	}

	if (unlikely(vi->xdp_enabled)) {
		struct bpf_prog *xdp_prog;

		rcu_read_lock();
		xdp_prog = rcu_dereference(rq->xdp_prog);
		if (xdp_prog) {
			skb = receive_small_xdp(dev, vi, rq, xdp_prog, buf,
						xdp_headroom, len, buflen,
						xdp_xmit, stats);
			rcu_read_unlock();
			return skb;
		}
		rcu_read_unlock();
	}

	skb = receive_small_build_skb(vi, xdp_headroom, buf, len, buflen);
	if (likely(skb)) {
		skb_mark_for_recycle(skb);
		return skb;
	}

err:
	u64_stats_inc(&stats->drops);
	page_pool_put_page(rq->page_pool, page, -1, true);
	return NULL;
}

static struct sk_buff *receive_big(struct net_device *dev,
				   struct virtnet_info *vi,
				   struct receive_queue *rq,
				   void *buf,
				   unsigned int len,
				   struct virtnet_rq_stats *stats)
{
	struct page *page = buf;
	struct sk_buff *skb;

	/* Make sure that len does not exceed the size allocated in
	 * add_recvbuf_big.
	 */
	if (unlikely(len > (vi->big_packets_num_skbfrags + 1) * PAGE_SIZE)) {
		pr_debug("%s: rx error: len %u exceeds allocated size %lu\n",
			 dev->name, len,
			 (vi->big_packets_num_skbfrags + 1) * PAGE_SIZE);
		goto err;
	}

	skb = page_to_skb(vi, rq, page, 0, len, PAGE_SIZE, 0);
	u64_stats_add(&stats->bytes, len - vi->hdr_len);
	if (unlikely(!skb))
		goto err;

	return skb;

err:
	u64_stats_inc(&stats->drops);
	give_pages(rq, page);
	return NULL;
}

static void mergeable_buf_free(struct receive_queue *rq, int num_buf,
			       struct net_device *dev,
			       struct virtnet_rq_stats *stats)
{
	struct page *page;
	void *buf;
	int len;

	while (num_buf-- > 1) {
		buf = virtnet_rq_get_buf(rq, &len, NULL);
		if (unlikely(!buf)) {
			pr_debug("%s: rx error: %d buffers missing\n",
				 dev->name, num_buf);
			DEV_STATS_INC(dev, rx_length_errors);
			break;
		}
		u64_stats_add(&stats->bytes, len);
		page = virt_to_head_page(buf);
		page_pool_put_page(rq->page_pool, page, -1, true);
	}
}

/* Why not use xdp_build_skb_from_frame() ?
 * XDP core assumes that xdp frags are PAGE_SIZE in length, while in
 * virtio-net there are 2 points that do not match its requirements:
 *  1. The size of the prefilled buffer is not fixed before xdp is set.
 *  2. xdp_build_skb_from_frame() does more checks that we don't need,
 *     like eth_type_trans() (which virtio-net does in receive_buf()).
 */
static struct sk_buff *build_skb_from_xdp_buff(struct net_device *dev,
					       struct virtnet_info *vi,
					       struct xdp_buff *xdp,
					       unsigned int xdp_frags_truesz)
{
	struct skb_shared_info *sinfo = xdp_get_shared_info_from_buff(xdp);
	unsigned int headroom, data_len;
	struct sk_buff *skb;
	int metasize;
	u8 nr_frags;

	if (unlikely(xdp->data_end > xdp_data_hard_end(xdp))) {
		pr_debug("Error building skb as missing reserved tailroom for xdp");
		return NULL;
	}

	if (unlikely(xdp_buff_has_frags(xdp)))
		nr_frags = sinfo->nr_frags;

	skb = build_skb(xdp->data_hard_start, xdp->frame_sz);
	if (unlikely(!skb))
		return NULL;

	headroom = xdp->data - xdp->data_hard_start;
	data_len = xdp->data_end - xdp->data;
	skb_reserve(skb, headroom);
	__skb_put(skb, data_len);

	metasize = xdp->data - xdp->data_meta;
	metasize = metasize > 0 ? metasize : 0;
	if (metasize)
		skb_metadata_set(skb, metasize);

	if (unlikely(xdp_buff_has_frags(xdp)))
		xdp_update_skb_frags_info(skb, nr_frags, sinfo->xdp_frags_size,
					  xdp_frags_truesz,
					  xdp_buff_get_skb_flags(xdp));

	return skb;
}

/* TODO: build xdp in big mode */
static int virtnet_build_xdp_buff_mrg(struct net_device *dev,
				      struct virtnet_info *vi,
				      struct receive_queue *rq,
				      struct xdp_buff *xdp,
				      void *buf,
				      unsigned int len,
				      unsigned int frame_sz,
				      int *num_buf,
				      unsigned int *xdp_frags_truesize,
				      struct virtnet_rq_stats *stats)
{
	struct virtio_net_hdr_mrg_rxbuf *hdr = buf;
	struct skb_shared_info *shinfo;
	unsigned int xdp_frags_truesz = 0;
	unsigned int truesize;
	struct page *page;
	skb_frag_t *frag;
	int offset;
	void *ctx;

	xdp_init_buff(xdp, frame_sz, &rq->xdp_rxq);
	xdp_prepare_buff(xdp, buf - XDP_PACKET_HEADROOM,
			 XDP_PACKET_HEADROOM + vi->hdr_len, len - vi->hdr_len, true);

	if (!*num_buf)
		return 0;

	if (*num_buf > 1) {
		/* If we want to build multi-buffer xdp, we need
		 * to specify that the flags of xdp_buff have the
		 * XDP_FLAGS_HAS_FRAG bit.
		 */
		if (!xdp_buff_has_frags(xdp))
			xdp_buff_set_frags_flag(xdp);

		shinfo = xdp_get_shared_info_from_buff(xdp);
		shinfo->nr_frags = 0;
		shinfo->xdp_frags_size = 0;
	}

	if (*num_buf > MAX_SKB_FRAGS + 1)
		return -EINVAL;

	while (--*num_buf > 0) {
		buf = virtnet_rq_get_buf(rq, &len, &ctx);
		if (unlikely(!buf)) {
			pr_debug("%s: rx error: %d buffers out of %d missing\n",
				 dev->name, *num_buf,
				 virtio16_to_cpu(vi->vdev, hdr->num_buffers));
			DEV_STATS_INC(dev, rx_length_errors);
			goto err;
		}

		u64_stats_add(&stats->bytes, len);
		page = virt_to_head_page(buf);
		offset = buf - page_address(page);

		if (rq->use_page_pool_dma)
			page_pool_dma_sync_for_cpu(rq->page_pool, page,
						   offset, len);

		if (check_mergeable_len(dev, ctx, len)) {
			page_pool_put_page(rq->page_pool, page, -1, true);
			goto err;
		}

		truesize = mergeable_ctx_to_truesize(ctx);
		xdp_frags_truesz += truesize;

		frag = &shinfo->frags[shinfo->nr_frags++];
		skb_frag_fill_page_desc(frag, page, offset, len);
		if (page_is_pfmemalloc(page))
			xdp_buff_set_frag_pfmemalloc(xdp);

		shinfo->xdp_frags_size += len;
	}

	*xdp_frags_truesize = xdp_frags_truesz;
	return 0;

err:
	put_xdp_frags(rq, xdp);
	return -EINVAL;
}

static void *mergeable_xdp_get_buf(struct virtnet_info *vi,
				   struct receive_queue *rq,
				   struct bpf_prog *xdp_prog,
				   void *ctx,
				   unsigned int *frame_sz,
				   int *num_buf,
				   struct page **page,
				   int offset,
				   unsigned int *len,
				   struct virtio_net_hdr_mrg_rxbuf *hdr)
{
	unsigned int truesize = mergeable_ctx_to_truesize(ctx);
	unsigned int headroom = mergeable_ctx_to_headroom(ctx);
	struct page *xdp_page;
	unsigned int xdp_room;

	/* Transient failure which in theory could occur if
	 * in-flight packets from before XDP was enabled reach
	 * the receive path after XDP is loaded.
	 */
	if (unlikely(hdr->hdr.gso_type))
		return NULL;

	/* Partially checksummed packets must be dropped. */
	if (unlikely(hdr->hdr.flags & VIRTIO_NET_HDR_F_NEEDS_CSUM))
		return NULL;

	/* Now XDP core assumes frag size is PAGE_SIZE, but buffers
	 * with headroom may add hole in truesize, which
	 * make their length exceed PAGE_SIZE. So we disabled the
	 * hole mechanism for xdp. See add_recvbuf_mergeable().
	 */
	*frame_sz = truesize;

	if (likely(headroom >= virtnet_get_headroom(vi) &&
		   (*num_buf == 1 || xdp_prog->aux->xdp_has_frags))) {
		return page_address(*page) + offset;
	}

	/* This happens when headroom is not enough because
	 * of the buffer was prefilled before XDP is set.
	 * This should only happen for the first several packets.
	 * In fact, vq reset can be used here to help us clean up
	 * the prefilled buffers, but many existing devices do not
	 * support it, and we don't want to bother users who are
	 * using xdp normally.
	 */
	if (!xdp_prog->aux->xdp_has_frags) {
		/* linearize data for XDP */
		xdp_page = xdp_linearize_page(vi->dev, rq, num_buf,
					      *page, offset,
					      XDP_PACKET_HEADROOM,
					      len);
		if (!xdp_page)
			return NULL;
	} else {
		xdp_room = SKB_DATA_ALIGN(XDP_PACKET_HEADROOM +
					  sizeof(struct skb_shared_info));
		if (*len + xdp_room > PAGE_SIZE)
			return NULL;

		xdp_page = page_pool_alloc_pages(rq->page_pool, GFP_ATOMIC);
		if (!xdp_page)
			return NULL;

		memcpy(page_address(xdp_page) + XDP_PACKET_HEADROOM,
		       page_address(*page) + offset, *len);
	}

	*frame_sz = PAGE_SIZE;

	page_pool_put_page(rq->page_pool, *page, -1, true);

	*page = xdp_page;

	return page_address(*page) + XDP_PACKET_HEADROOM;
}

static struct sk_buff *receive_mergeable_xdp(struct net_device *dev,
					     struct virtnet_info *vi,
					     struct receive_queue *rq,
					     struct bpf_prog *xdp_prog,
					     void *buf,
					     void *ctx,
					     unsigned int len,
					     unsigned int *xdp_xmit,
					     struct virtnet_rq_stats *stats)
{
	struct virtio_net_hdr_mrg_rxbuf *hdr = buf;
	int num_buf = virtio16_to_cpu(vi->vdev, hdr->num_buffers);
	struct page *page = virt_to_head_page(buf);
	int offset = buf - page_address(page);
	unsigned int xdp_frags_truesz = 0;
	struct sk_buff *head_skb;
	unsigned int frame_sz;
	struct xdp_buff xdp;
	void *data;
	u32 act;
	int err;

	data = mergeable_xdp_get_buf(vi, rq, xdp_prog, ctx, &frame_sz, &num_buf, &page,
				     offset, &len, hdr);
	if (unlikely(!data))
		goto err_xdp;

	err = virtnet_build_xdp_buff_mrg(dev, vi, rq, &xdp, data, len, frame_sz,
					 &num_buf, &xdp_frags_truesz, stats);
	if (unlikely(err))
		goto err_xdp;

	act = virtnet_xdp_handler(xdp_prog, &xdp, dev, xdp_xmit, stats);

	switch (act) {
	case XDP_PASS:
		head_skb = build_skb_from_xdp_buff(dev, vi, &xdp, xdp_frags_truesz);
		if (unlikely(!head_skb))
			break;

		skb_mark_for_recycle(head_skb);
		return head_skb;

	case XDP_TX:
	case XDP_REDIRECT:
		return NULL;

	default:
		break;
	}

	put_xdp_frags(rq, &xdp);

err_xdp:
	page_pool_put_page(rq->page_pool, page, -1, true);
	mergeable_buf_free(rq, num_buf, dev, stats);

	u64_stats_inc(&stats->xdp_drops);
	u64_stats_inc(&stats->drops);
	return NULL;
}

static struct sk_buff *virtnet_skb_append_frag(struct receive_queue *rq,
					       struct sk_buff *head_skb,
					       struct sk_buff *curr_skb,
					       struct page *page, void *buf,
					       int len, int truesize)
{
	int num_skb_frags;
	int offset;

	num_skb_frags = skb_shinfo(curr_skb)->nr_frags;
	if (unlikely(num_skb_frags == MAX_SKB_FRAGS)) {
		struct sk_buff *nskb = alloc_skb(0, GFP_ATOMIC);

		if (unlikely(!nskb))
			return NULL;

		if (head_skb->pp_recycle)
			skb_mark_for_recycle(nskb);

		if (curr_skb == head_skb)
			skb_shinfo(curr_skb)->frag_list = nskb;
		else
			curr_skb->next = nskb;
		curr_skb = nskb;
		head_skb->truesize += nskb->truesize;
		num_skb_frags = 0;
	}

	if (curr_skb != head_skb) {
		head_skb->data_len += len;
		head_skb->len += len;
		head_skb->truesize += truesize;
	}

	offset = buf - page_address(page);
	if (skb_can_coalesce(curr_skb, num_skb_frags, page, offset)) {
		if (head_skb->pp_recycle)
			page_pool_put_page(rq->page_pool, page, -1, true);
		else
			put_page(page);
		skb_coalesce_rx_frag(curr_skb, num_skb_frags - 1,
				     len, truesize);
	} else {
		skb_add_rx_frag(curr_skb, num_skb_frags, page,
				offset, len, truesize);
	}

	return curr_skb;
}

static struct sk_buff *receive_mergeable(struct net_device *dev,
					 struct virtnet_info *vi,
					 struct receive_queue *rq,
					 void *buf,
					 void *ctx,
					 unsigned int len,
					 unsigned int *xdp_xmit,
					 struct virtnet_rq_stats *stats)
{
	struct virtio_net_hdr_mrg_rxbuf *hdr = buf;
	int num_buf = virtio16_to_cpu(vi->vdev, hdr->num_buffers);
	struct page *page = virt_to_head_page(buf);
	int offset = buf - page_address(page);
	struct sk_buff *head_skb, *curr_skb;
	unsigned int truesize = mergeable_ctx_to_truesize(ctx);
	unsigned int headroom = mergeable_ctx_to_headroom(ctx);

	head_skb = NULL;

	u64_stats_add(&stats->bytes, len - vi->hdr_len);

	if (check_mergeable_len(dev, ctx, len))
		goto err_skb;

	if (unlikely(vi->xdp_enabled)) {
		struct bpf_prog *xdp_prog;

		rcu_read_lock();
		xdp_prog = rcu_dereference(rq->xdp_prog);
		if (xdp_prog) {
			head_skb = receive_mergeable_xdp(dev, vi, rq, xdp_prog, buf, ctx,
							 len, xdp_xmit, stats);
			rcu_read_unlock();
			return head_skb;
		}
		rcu_read_unlock();
	}

	head_skb = page_to_skb(vi, rq, page, offset, len, truesize, headroom);
	curr_skb = head_skb;

	if (unlikely(!curr_skb))
		goto err_skb;

	skb_mark_for_recycle(head_skb);
	while (--num_buf) {
		buf = virtnet_rq_get_buf(rq, &len, &ctx);
		if (unlikely(!buf)) {
			pr_debug("%s: rx error: %d buffers out of %d missing\n",
				 dev->name, num_buf,
				 virtio16_to_cpu(vi->vdev,
						 hdr->num_buffers));
			DEV_STATS_INC(dev, rx_length_errors);
			goto err_buf;
		}

		u64_stats_add(&stats->bytes, len);
		page = virt_to_head_page(buf);

		if (rq->use_page_pool_dma) {
			offset = buf - page_address(page);
			page_pool_dma_sync_for_cpu(rq->page_pool, page,
						   offset, len);
		}

		if (check_mergeable_len(dev, ctx, len))
			goto err_skb;

		truesize = mergeable_ctx_to_truesize(ctx);
		curr_skb  = virtnet_skb_append_frag(rq, head_skb, curr_skb, page,
						    buf, len, truesize);
		if (!curr_skb)
			goto err_skb;
	}

	ewma_pkt_len_add(&rq->mrg_avg_pkt_len, head_skb->len);
	return head_skb;

err_skb:
	page_pool_put_page(rq->page_pool, page, -1, true);
	mergeable_buf_free(rq, num_buf, dev, stats);

err_buf:
	u64_stats_inc(&stats->drops);
	dev_kfree_skb(head_skb);
	return NULL;
}

static inline u32
virtio_net_hash_value(const struct virtio_net_hdr_v1_hash *hdr_hash)
{
	return __le16_to_cpu(hdr_hash->hash_value_lo) |
		(__le16_to_cpu(hdr_hash->hash_value_hi) << 16);
}

static void virtio_skb_set_hash(const struct virtio_net_hdr_v1_hash *hdr_hash,
				struct sk_buff *skb)
{
	enum pkt_hash_types rss_hash_type;

	if (!hdr_hash || !skb)
		return;

	switch (__le16_to_cpu(hdr_hash->hash_report)) {
	case VIRTIO_NET_HASH_REPORT_TCPv4:
	case VIRTIO_NET_HASH_REPORT_UDPv4:
	case VIRTIO_NET_HASH_REPORT_TCPv6:
	case VIRTIO_NET_HASH_REPORT_UDPv6:
	case VIRTIO_NET_HASH_REPORT_TCPv6_EX:
	case VIRTIO_NET_HASH_REPORT_UDPv6_EX:
		rss_hash_type = PKT_HASH_TYPE_L4;
		break;
	case VIRTIO_NET_HASH_REPORT_IPv4:
	case VIRTIO_NET_HASH_REPORT_IPv6:
	case VIRTIO_NET_HASH_REPORT_IPv6_EX:
		rss_hash_type = PKT_HASH_TYPE_L3;
		break;
	case VIRTIO_NET_HASH_REPORT_NONE:
	default:
		rss_hash_type = PKT_HASH_TYPE_NONE;
	}
	skb_set_hash(skb, virtio_net_hash_value(hdr_hash), rss_hash_type);
}

static void virtnet_receive_done(struct virtnet_info *vi, struct receive_queue *rq,
				 struct sk_buff *skb, u8 flags)
{
	struct virtio_net_common_hdr *hdr;
	struct net_device *dev = vi->dev;

	hdr = skb_vnet_common_hdr(skb);
	if (dev->features & NETIF_F_RXHASH && vi->has_rss_hash_report)
		virtio_skb_set_hash(&hdr->hash_v1_hdr, skb);

	hdr->hdr.flags = flags;
	if (virtio_net_handle_csum_offload(skb, &hdr->hdr, vi->rx_tnl_csum)) {
		net_warn_ratelimited("%s: bad csum: flags: %x, gso_type: %x rx_tnl_csum %d\n",
				     dev->name, hdr->hdr.flags,
				     hdr->hdr.gso_type, vi->rx_tnl_csum);
		goto frame_err;
	}

	if (virtio_net_hdr_tnl_to_skb(skb, &hdr->tnl_hdr, vi->rx_tnl,
				      vi->rx_tnl_csum,
				      virtio_is_little_endian(vi->vdev))) {
		net_warn_ratelimited("%s: bad gso: type: %x, size: %u, flags %x tunnel %d tnl csum %d\n",
				     dev->name, hdr->hdr.gso_type,
				     hdr->hdr.gso_size, hdr->hdr.flags,
				     vi->rx_tnl, vi->rx_tnl_csum);
		goto frame_err;
	}

	skb_record_rx_queue(skb, vq2rxq(rq->vq));
	skb->protocol = eth_type_trans(skb, dev);
	pr_debug("Receiving skb proto 0x%04x len %i type %i\n",
		 ntohs(skb->protocol), skb->len, skb->pkt_type);

	napi_gro_receive(&rq->napi, skb);
	return;

frame_err:
	DEV_STATS_INC(dev, rx_frame_errors);
	dev_kfree_skb(skb);
}

static void receive_buf(struct virtnet_info *vi, struct receive_queue *rq,
			void *buf, unsigned int len, void **ctx,
			unsigned int *xdp_xmit,
			struct virtnet_rq_stats *stats)
{
	struct net_device *dev = vi->dev;
	struct sk_buff *skb;
	u8 flags;

	if (unlikely(len < vi->hdr_len + ETH_HLEN)) {
		pr_debug("%s: short packet %i\n", dev->name, len);
		DEV_STATS_INC(dev, rx_length_errors);
		virtnet_rq_free_buf(vi, rq, buf);
		return;
	}

	/* Sync the memory before touching anything through buf,
	 * unless virtio core did it already.
	 */
	if (rq->use_page_pool_dma) {
		struct page *page = virt_to_head_page(buf);
		int offset = buf - page_address(page);

		page_pool_dma_sync_for_cpu(rq->page_pool, page, offset, len);
	}

	/* About the flags below:
	 * 1. Save the flags early, as the XDP program might overwrite them.
	 * These flags ensure packets marked as VIRTIO_NET_HDR_F_DATA_VALID
	 * stay valid after XDP processing.
	 * 2. XDP doesn't work with partially checksummed packets (refer to
	 * virtnet_xdp_set()), so packets marked as
	 * VIRTIO_NET_HDR_F_NEEDS_CSUM get dropped during XDP processing.
	 */

	if (vi->mergeable_rx_bufs) {
		flags = ((struct virtio_net_common_hdr *)buf)->hdr.flags;
		skb = receive_mergeable(dev, vi, rq, buf, ctx, len, xdp_xmit,
					stats);
	} else if (vi->big_packets) {
		void *p = page_address((struct page *)buf);

		flags = ((struct virtio_net_common_hdr *)p)->hdr.flags;
		skb = receive_big(dev, vi, rq, buf, len, stats);
	} else {
		flags = ((struct virtio_net_common_hdr *)buf)->hdr.flags;
		skb = receive_small(dev, vi, rq, buf, ctx, len, xdp_xmit, stats);
	}

	if (unlikely(!skb))
		return;

	virtnet_receive_done(vi, rq, skb, flags);
}

static int virtnet_rq_submit(struct receive_queue *rq, char *buf,
			     int len, void *ctx, gfp_t gfp)
{
	if (rq->use_page_pool_dma) {
		struct page *page = virt_to_head_page(buf);
		dma_addr_t addr = page_pool_get_dma_addr(page) +
				  (buf - (char *)page_address(page));

		sg_init_table(rq->sg, 1);
		sg_fill_dma(rq->sg, addr, len);
		return virtqueue_add_inbuf_premapped(rq->vq, rq->sg, 1,
						     buf, ctx, gfp);
	}

	sg_init_one(rq->sg, buf, len);
	return virtqueue_add_inbuf_ctx(rq->vq, rq->sg, 1, buf, ctx, gfp);
}

/* With page_pool, the actual allocation may exceed the requested size
 * when the remaining page fragment can't fit another buffer. Encode
 * the actual allocation size in ctx so build_skb() gets the correct
 * buflen for truesize accounting.
 */
static int add_recvbuf_small(struct virtnet_info *vi, struct receive_queue *rq,
			     gfp_t gfp)
{
	unsigned int xdp_headroom = virtnet_get_headroom(vi);
	unsigned int len = vi->hdr_len + VIRTNET_RX_PAD + GOOD_PACKET_LEN + xdp_headroom;
	unsigned int alloc_len;
	char *buf;
	void *ctx;
	int err;

	len = SKB_DATA_ALIGN(len) +
	      SKB_DATA_ALIGN(sizeof(struct skb_shared_info));

	alloc_len = len;
	buf = page_pool_alloc_va(rq->page_pool, &alloc_len, gfp);
	if (unlikely(!buf))
		return -ENOMEM;

	buf += VIRTNET_RX_PAD + xdp_headroom;

	ctx = mergeable_len_to_ctx(alloc_len, xdp_headroom);
	err = virtnet_rq_submit(rq, buf, vi->hdr_len + GOOD_PACKET_LEN, ctx, gfp);

	if (err < 0)
		page_pool_put_page(rq->page_pool, virt_to_head_page(buf), -1, false);
	return err;
}

static int add_recvbuf_big(struct virtnet_info *vi, struct receive_queue *rq,
			   gfp_t gfp)
{
	struct page *first, *list = NULL;
	char *p;
	int i, err, offset;

	sg_init_table(rq->sg, vi->big_packets_num_skbfrags + 2);

	/* page in rq->sg[vi->big_packets_num_skbfrags + 1] is list tail */
	for (i = vi->big_packets_num_skbfrags + 1; i > 1; --i) {
		first = get_a_page(rq, gfp);
		if (!first) {
			if (list)
				give_pages(rq, list);
			return -ENOMEM;
		}
		sg_set_buf(&rq->sg[i], page_address(first), PAGE_SIZE);

		/* chain new page in list head to match sg */
		first->private = (unsigned long)list;
		list = first;
	}

	first = get_a_page(rq, gfp);
	if (!first) {
		give_pages(rq, list);
		return -ENOMEM;
	}
	p = page_address(first);

	/* rq->sg[0], rq->sg[1] share the same page */
	/* a separated rq->sg[0] for header - required in case !any_header_sg */
	sg_set_buf(&rq->sg[0], p, vi->hdr_len);

	/* rq->sg[1] for data packet, from offset */
	offset = sizeof(struct padded_vnet_hdr);
	sg_set_buf(&rq->sg[1], p + offset, PAGE_SIZE - offset);

	/* chain first in list head */
	first->private = (unsigned long)list;
	err = virtqueue_add_inbuf(rq->vq, rq->sg, vi->big_packets_num_skbfrags + 2,
				  first, gfp);
	if (err < 0)
		give_pages(rq, first);

	return err;
}

static unsigned int get_mergeable_buf_len(struct receive_queue *rq,
					  struct ewma_pkt_len *avg_pkt_len,
					  unsigned int room)
{
	struct virtnet_info *vi = rq->vq->vdev->priv;
	const size_t hdr_len = vi->hdr_len;
	unsigned int len;

	if (room)
		return PAGE_SIZE - room;

	len = hdr_len +	clamp_t(unsigned int, ewma_pkt_len_read(avg_pkt_len),
				rq->min_buf_len, PAGE_SIZE - hdr_len);

	return ALIGN(len, L1_CACHE_BYTES);
}

static int add_recvbuf_mergeable(struct virtnet_info *vi,
				 struct receive_queue *rq, gfp_t gfp)
{
	unsigned int headroom = virtnet_get_headroom(vi);
	unsigned int tailroom = headroom ? sizeof(struct skb_shared_info) : 0;
	unsigned int room = SKB_DATA_ALIGN(headroom + tailroom);
	unsigned int len, alloc_len;
	char *buf;
	void *ctx;
	int err;

	/* Extra tailroom is needed to satisfy XDP's assumption. This
	 * means rx frags coalescing won't work, but consider we've
	 * disabled GSO for XDP, it won't be a big issue.
	 */
	len = get_mergeable_buf_len(rq, &rq->mrg_avg_pkt_len, room);

	alloc_len = len + room;
	buf = page_pool_alloc_va(rq->page_pool, &alloc_len, gfp);
	if (unlikely(!buf))
		return -ENOMEM;

	buf += headroom; /* advance address leaving hole at front of pkt */

	if (!headroom)
		len = alloc_len - room;

	ctx = mergeable_len_to_ctx(len + room, headroom);

	err = virtnet_rq_submit(rq, buf, len, ctx, gfp);

	if (err < 0)
		page_pool_put_page(rq->page_pool, virt_to_head_page(buf), -1, false);
	return err;
}

/*
 * Returns false if we couldn't fill entirely (OOM) and need to retry.
 * In XSK mode, it's when the receive buffer is not allocated and
 * xsk_use_need_wakeup is not set.
 *
 * Normally run in the receive path, but can also be run from ndo_open
 * before we're receiving packets, or from refill_work which is
 * careful to disable receiving (using napi_disable).
 */
static bool try_fill_recv(struct virtnet_info *vi, struct receive_queue *rq,
			  gfp_t gfp)
{
	int err;

	if (rq->xsk_pool) {
		err = virtnet_add_recvbuf_xsk(vi, rq, rq->xsk_pool, gfp);
		goto kick;
	}

	do {
		if (vi->mergeable_rx_bufs)
			err = add_recvbuf_mergeable(vi, rq, gfp);
		else if (vi->big_packets)
			err = add_recvbuf_big(vi, rq, gfp);
		else
			err = add_recvbuf_small(vi, rq, gfp);

		if (err)
			break;
	} while (rq->vq->num_free);

kick:
	if (virtqueue_kick_prepare(rq->vq) && virtqueue_notify(rq->vq)) {
		unsigned long flags;

		flags = u64_stats_update_begin_irqsave(&rq->stats.syncp);
		u64_stats_inc(&rq->stats.kicks);
		u64_stats_update_end_irqrestore(&rq->stats.syncp, flags);
	}

	return err != -ENOMEM;
}

static void skb_recv_done(struct virtqueue *rvq)
{
	struct virtnet_info *vi = rvq->vdev->priv;
	struct receive_queue *rq = &vi->rq[vq2rxq(rvq)];

	rq->calls++;
	virtqueue_napi_schedule(&rq->napi, rvq);
}

static void virtnet_napi_do_enable(struct virtqueue *vq,
				   struct napi_struct *napi)
{
	napi_enable(napi);

	/* If all buffers were filled by other side before we napi_enabled, we
	 * won't get another interrupt, so process any outstanding packets now.
	 * Call local_bh_enable after to trigger softIRQ processing.
	 */
	local_bh_disable();
	virtqueue_napi_schedule(napi, vq);
	local_bh_enable();
}

static void virtnet_napi_enable(struct receive_queue *rq)
{
	struct virtnet_info *vi = rq->vq->vdev->priv;
	int qidx = vq2rxq(rq->vq);

	virtnet_napi_do_enable(rq->vq, &rq->napi);
	netif_queue_set_napi(vi->dev, qidx, NETDEV_QUEUE_TYPE_RX, &rq->napi);
}

static void virtnet_napi_tx_enable(struct send_queue *sq)
{
	struct virtnet_info *vi = sq->vq->vdev->priv;
	struct napi_struct *napi = &sq->napi;
	int qidx = vq2txq(sq->vq);

	if (!napi->weight)
		return;

	/* Tx napi touches cachelines on the cpu handling tx interrupts. Only
	 * enable the feature if this is likely affine with the transmit path.
	 */
	if (!vi->affinity_hint_set) {
		napi->weight = 0;
		return;
	}

	virtnet_napi_do_enable(sq->vq, napi);
	netif_queue_set_napi(vi->dev, qidx, NETDEV_QUEUE_TYPE_TX, napi);
}

static void virtnet_napi_tx_disable(struct send_queue *sq)
{
	struct virtnet_info *vi = sq->vq->vdev->priv;
	struct napi_struct *napi = &sq->napi;
	int qidx = vq2txq(sq->vq);

	if (napi->weight) {
		netif_queue_set_napi(vi->dev, qidx, NETDEV_QUEUE_TYPE_TX, NULL);
		napi_disable(napi);
	}
}

static void virtnet_napi_disable(struct receive_queue *rq)
{
	struct virtnet_info *vi = rq->vq->vdev->priv;
	struct napi_struct *napi = &rq->napi;
	int qidx = vq2rxq(rq->vq);

	netif_queue_set_napi(vi->dev, qidx, NETDEV_QUEUE_TYPE_RX, NULL);
	napi_disable(napi);
}

static int virtnet_receive_xsk_bufs(struct virtnet_info *vi,
				    struct receive_queue *rq,
				    int budget,
				    unsigned int *xdp_xmit,
				    struct virtnet_rq_stats *stats)
{
	unsigned int len;
	int packets = 0;
	void *buf;

	while (packets < budget) {
		buf = virtqueue_get_buf(rq->vq, &len);
		if (!buf)
			break;

		virtnet_receive_xsk_buf(vi, rq, buf, len, xdp_xmit, stats);
		packets++;
	}

	return packets;
}

static int virtnet_receive_packets(struct virtnet_info *vi,
				   struct receive_queue *rq,
				   int budget,
				   unsigned int *xdp_xmit,
				   struct virtnet_rq_stats *stats)
{
	unsigned int len;
	int packets = 0;
	void *buf;

	if (rq->page_pool) {
		void *ctx;
		while (packets < budget &&
		       (buf = virtnet_rq_get_buf(rq, &len, &ctx))) {
			receive_buf(vi, rq, buf, len, ctx, xdp_xmit, stats);
			packets++;
		}
	} else {
		while (packets < budget &&
		       (buf = virtqueue_get_buf(rq->vq, &len)) != NULL) {
			receive_buf(vi, rq, buf, len, NULL, xdp_xmit, stats);
			packets++;
		}
	}

	return packets;
}

static int virtnet_receive(struct receive_queue *rq, int budget,
			   unsigned int *xdp_xmit)
{
	struct virtnet_info *vi = rq->vq->vdev->priv;
	struct virtnet_rq_stats stats = {};
	int i, packets;

	if (rq->xsk_pool)
		packets = virtnet_receive_xsk_bufs(vi, rq, budget, xdp_xmit, &stats);
	else
		packets = virtnet_receive_packets(vi, rq, budget, xdp_xmit, &stats);

	u64_stats_set(&stats.packets, packets);
	if (rq->vq->num_free > min((unsigned int)budget, virtqueue_get_vring_size(rq->vq)) / 2) {
		if (!try_fill_recv(vi, rq, GFP_ATOMIC))
			/* We need to retry refilling in the next NAPI poll so
			 * we must return budget to make sure the NAPI is
			 * repolled.
			 */
			packets = budget;
	}

	u64_stats_update_begin(&rq->stats.syncp);
	for (i = 0; i < ARRAY_SIZE(virtnet_rq_stats_desc); i++) {
		size_t offset = virtnet_rq_stats_desc[i].offset;
		u64_stats_t *item, *src;

		item = (u64_stats_t *)((u8 *)&rq->stats + offset);
		src = (u64_stats_t *)((u8 *)&stats + offset);
		u64_stats_add(item, u64_stats_read(src));
	}

	u64_stats_add(&rq->stats.packets, u64_stats_read(&stats.packets));
	u64_stats_add(&rq->stats.bytes, u64_stats_read(&stats.bytes));

	u64_stats_update_end(&rq->stats.syncp);

	return packets;
}

static void virtnet_poll_cleantx(struct receive_queue *rq, int budget)
{
	struct virtnet_info *vi = rq->vq->vdev->priv;
	unsigned int index = vq2rxq(rq->vq);
	struct send_queue *sq = &vi->sq[index];
	struct netdev_queue *txq = netdev_get_tx_queue(vi->dev, index);

	if (!sq->napi.weight || is_xdp_raw_buffer_queue(vi, index))
		return;

	if (__netif_tx_trylock(txq)) {
		if (sq->reset) {
			__netif_tx_unlock(txq);
			return;
		}

		do {
			virtqueue_disable_cb(sq->vq);
			free_old_xmit(sq, txq, !!budget);
		} while (unlikely(!virtqueue_enable_cb_delayed(sq->vq)));

		if (sq->vq->num_free >= MAX_SKB_FRAGS + 2)
			virtnet_tx_wake_queue(vi, sq);

		__netif_tx_unlock(txq);
	}
}

static void virtnet_rx_dim_update(struct virtnet_info *vi, struct receive_queue *rq)
{
	struct dim_sample cur_sample = {};

	if (!rq->packets_in_napi)
		return;

	/* Don't need protection when fetching stats, since fetcher and
	 * updater of the stats are in same context
	 */
	dim_update_sample(rq->calls,
			  u64_stats_read(&rq->stats.packets),
			  u64_stats_read(&rq->stats.bytes),
			  &cur_sample);

	net_dim(&rq->dim, &cur_sample);
	rq->packets_in_napi = 0;
}

static int virtnet_poll(struct napi_struct *napi, int budget)
{
	struct receive_queue *rq =
		container_of(napi, struct receive_queue, napi);
	struct virtnet_info *vi = rq->vq->vdev->priv;
	struct send_queue *sq;
	unsigned int received;
	unsigned int xdp_xmit = 0;
	bool napi_complete;

	virtnet_poll_cleantx(rq, budget);

	received = virtnet_receive(rq, budget, &xdp_xmit);
	rq->packets_in_napi += received;

	if (xdp_xmit & VIRTIO_XDP_REDIR)
		xdp_do_flush();

	/* Out of packets? */
	if (received < budget) {
		napi_complete = virtqueue_napi_complete(napi, rq->vq, received);
		/* Intentionally not taking dim_lock here. This may result in a
		 * spurious net_dim call. But if that happens virtnet_rx_dim_work
		 * will not act on the scheduled work.
		 */
		if (napi_complete && rq->dim_enabled)
			virtnet_rx_dim_update(vi, rq);
	}

	if (xdp_xmit & VIRTIO_XDP_TX) {
		sq = virtnet_xdp_get_sq(vi);
		if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq)) {
			u64_stats_update_begin(&sq->stats.syncp);
			u64_stats_inc(&sq->stats.kicks);
			u64_stats_update_end(&sq->stats.syncp);
		}
		virtnet_xdp_put_sq(vi, sq);
	}

	return received;
}

static void virtnet_disable_queue_pair(struct virtnet_info *vi, int qp_index)
{
	virtnet_napi_tx_disable(&vi->sq[qp_index]);
	virtnet_napi_disable(&vi->rq[qp_index]);
	xdp_rxq_info_unreg(&vi->rq[qp_index].xdp_rxq);
}

static int virtnet_enable_queue_pair(struct virtnet_info *vi, int qp_index)
{
	struct net_device *dev = vi->dev;
	int err;

	err = xdp_rxq_info_reg(&vi->rq[qp_index].xdp_rxq, dev, qp_index,
			       vi->rq[qp_index].napi.napi_id);
	if (err < 0)
		return err;

	err = xdp_rxq_info_reg_mem_model(&vi->rq[qp_index].xdp_rxq,
					 vi->rq[qp_index].page_pool ?
						MEM_TYPE_PAGE_POOL :
						MEM_TYPE_PAGE_SHARED,
					 vi->rq[qp_index].page_pool);
	if (err < 0)
		goto err_xdp_reg_mem_model;

	virtnet_napi_enable(&vi->rq[qp_index]);
	virtnet_napi_tx_enable(&vi->sq[qp_index]);

	return 0;

err_xdp_reg_mem_model:
	xdp_rxq_info_unreg(&vi->rq[qp_index].xdp_rxq);
	return err;
}

static void virtnet_cancel_dim(struct virtnet_info *vi, struct dim *dim)
{
	if (!virtio_has_feature(vi->vdev, VIRTIO_NET_F_VQ_NOTF_COAL))
		return;
	net_dim_work_cancel(dim);
}

static void virtnet_update_settings(struct virtnet_info *vi)
{
	u32 speed;
	u8 duplex;

	if (!virtio_has_feature(vi->vdev, VIRTIO_NET_F_SPEED_DUPLEX))
		return;

	virtio_cread_le(vi->vdev, struct virtio_net_config, speed, &speed);

	if (ethtool_validate_speed(speed))
		vi->speed = speed;

	virtio_cread_le(vi->vdev, struct virtio_net_config, duplex, &duplex);

	if (ethtool_validate_duplex(duplex))
		vi->duplex = duplex;
}

static int virtnet_create_page_pools(struct virtnet_info *vi)
{
	int i, err;

	if (vi->big_packets && !vi->mergeable_rx_bufs)
		return 0;

	for (i = 0; i < vi->max_queue_pairs; i++) {
		struct receive_queue *rq = &vi->rq[i];
		struct page_pool_params pp_params = { 0 };
		struct device *dma_dev;

		if (rq->page_pool)
			continue;

		if (rq->xsk_pool)
			continue;

		pp_params.order = 0;
		pp_params.pool_size = virtqueue_get_vring_size(rq->vq);
		pp_params.nid = dev_to_node(vi->vdev->dev.parent);
		pp_params.netdev = vi->dev;
		pp_params.napi = &rq->napi;

		/* Use page_pool DMA mapping if backend supports DMA API.
		 * DMA_SYNC_DEV is needed for non-coherent archs on recycle.
		 */
		dma_dev = virtqueue_dma_dev(rq->vq);
		if (dma_dev) {
			pp_params.dev = dma_dev;
			pp_params.flags = PP_FLAG_DMA_MAP | PP_FLAG_DMA_SYNC_DEV;
			pp_params.dma_dir = DMA_FROM_DEVICE;
			pp_params.max_len = PAGE_SIZE;
			pp_params.offset = 0;
			rq->use_page_pool_dma = true;
		} else {
			/* No DMA API (e.g., VDUSE): page_pool for allocation only. */
			pp_params.flags = 0;
			rq->use_page_pool_dma = false;
		}

		rq->page_pool = page_pool_create(&pp_params);
		if (IS_ERR(rq->page_pool)) {
			err = PTR_ERR(rq->page_pool);
			rq->page_pool = NULL;
			goto err_cleanup;
		}
	}
	return 0;

err_cleanup:
	while (--i >= 0) {
		struct receive_queue *rq = &vi->rq[i];

		if (rq->page_pool) {
			page_pool_destroy(rq->page_pool);
			rq->page_pool = NULL;
		}
	}
	return err;
}

static void virtnet_destroy_page_pools(struct virtnet_info *vi)
{
	int i;

	for (i = 0; i < vi->max_queue_pairs; i++) {
		struct receive_queue *rq = &vi->rq[i];

		if (rq->page_pool) {
			page_pool_destroy(rq->page_pool);
			rq->page_pool = NULL;
		}
	}
}

static int virtnet_open(struct net_device *dev)
{
	struct virtnet_info *vi = netdev_priv(dev);
	int i, err;

	for (i = 0; i < vi->max_queue_pairs; i++) {
		if (i < vi->curr_queue_pairs)
			/* Pre-fill rq agressively, to make sure we are ready to
			 * get packets immediately.
			 */
			try_fill_recv(vi, &vi->rq[i], GFP_KERNEL);

		err = virtnet_enable_queue_pair(vi, i);
		if (err < 0)
			goto err_enable_qp;
	}

	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_STATUS)) {
		if (vi->status & VIRTIO_NET_S_LINK_UP)
			netif_carrier_on(vi->dev);
		virtio_config_driver_enable(vi->vdev);
	} else {
		vi->status = VIRTIO_NET_S_LINK_UP;
		netif_carrier_on(dev);
	}

	return 0;

err_enable_qp:
	for (i--; i >= 0; i--) {
		virtnet_disable_queue_pair(vi, i);
		virtnet_cancel_dim(vi, &vi->rq[i].dim);
	}

	return err;
}

static int virtnet_poll_tx(struct napi_struct *napi, int budget)
{
	struct send_queue *sq = container_of(napi, struct send_queue, napi);
	struct virtnet_info *vi = sq->vq->vdev->priv;
	unsigned int index = vq2txq(sq->vq);
	struct netdev_queue *txq;
	int opaque, xsk_done = 0;
	bool done;

	if (unlikely(is_xdp_raw_buffer_queue(vi, index))) {
		/* We don't need to enable cb for XDP */
		napi_complete_done(napi, 0);
		return 0;
	}

	txq = netdev_get_tx_queue(vi->dev, index);
	__netif_tx_lock(txq, raw_smp_processor_id());
	virtqueue_disable_cb(sq->vq);

	if (sq->xsk_pool)
		xsk_done = virtnet_xsk_xmit(sq, sq->xsk_pool, budget);
	else
		free_old_xmit(sq, txq, !!budget);

	if (sq->vq->num_free >= MAX_SKB_FRAGS + 2)
		virtnet_tx_wake_queue(vi, sq);

	if (xsk_done >= budget) {
		__netif_tx_unlock(txq);
		return budget;
	}

	opaque = virtqueue_enable_cb_prepare(sq->vq);

	done = napi_complete_done(napi, 0);

	if (!done)
		virtqueue_disable_cb(sq->vq);

	__netif_tx_unlock(txq);

	if (done) {
		if (unlikely(virtqueue_poll(sq->vq, opaque))) {
			if (napi_schedule_prep(napi)) {
				__netif_tx_lock(txq, raw_smp_processor_id());
				virtqueue_disable_cb(sq->vq);
				__netif_tx_unlock(txq);
				__napi_schedule(napi);
			}
		}
	}

	return 0;
}

static int xmit_skb(struct send_queue *sq, struct sk_buff *skb, bool orphan)
{
	const unsigned char *dest = ((struct ethhdr *)skb->data)->h_dest;
	struct virtnet_info *vi = sq->vq->vdev->priv;
	struct virtio_net_hdr_v1_hash_tunnel *hdr;
	int num_sg;
	unsigned hdr_len = vi->hdr_len;
	bool feature_hdrlen;
	bool can_push;

	feature_hdrlen = virtio_has_feature(vi->vdev,
					    VIRTIO_NET_F_GUEST_HDRLEN);

	pr_debug("%s: xmit %p %pM\n", vi->dev->name, skb, dest);

	/* Make sure it's safe to cast between formats */
	BUILD_BUG_ON(__alignof__(*hdr) != __alignof__(hdr->hash_hdr));
	BUILD_BUG_ON(__alignof__(*hdr) != __alignof__(hdr->hash_hdr.hdr));

	can_push = vi->any_header_sg &&
		!((unsigned long)skb->data & (__alignof__(*hdr) - 1)) &&
		!skb_header_cloned(skb) && skb_headroom(skb) >= hdr_len;
	/* Even if we can, don't push here yet as this would skew
	 * csum_start offset below. */
	if (can_push)
		hdr = (struct virtio_net_hdr_v1_hash_tunnel *)(skb->data -
							       hdr_len);
	else
		hdr = &skb_vnet_common_hdr(skb)->tnl_hdr;

	if (virtio_net_hdr_tnl_from_skb(skb, hdr, vi->tx_tnl,
					virtio_is_little_endian(vi->vdev), 0,
					false, feature_hdrlen))
		return -EPROTO;

	if (vi->mergeable_rx_bufs)
		hdr->hash_hdr.hdr.num_buffers = 0;

	sg_init_table(sq->sg, skb_shinfo(skb)->nr_frags + (can_push ? 1 : 2));
	if (can_push) {
		__skb_push(skb, hdr_len);
		num_sg = skb_to_sgvec(skb, sq->sg, 0, skb->len);
		if (unlikely(num_sg < 0))
			return num_sg;
		/* Pull header back to avoid skew in tx bytes calculations. */
		__skb_pull(skb, hdr_len);
	} else {
		sg_set_buf(sq->sg, hdr, hdr_len);
		num_sg = skb_to_sgvec(skb, sq->sg + 1, 0, skb->len);
		if (unlikely(num_sg < 0))
			return num_sg;
		num_sg++;
	}

	return virtnet_add_outbuf(sq, num_sg, skb,
				  orphan ? VIRTNET_XMIT_TYPE_SKB_ORPHAN : VIRTNET_XMIT_TYPE_SKB);
}

static netdev_tx_t start_xmit(struct sk_buff *skb, struct net_device *dev)
{
	struct virtnet_info *vi = netdev_priv(dev);
	int qnum = skb_get_queue_mapping(skb);
	struct send_queue *sq = &vi->sq[qnum];
	int err;
	struct netdev_queue *txq = netdev_get_tx_queue(dev, qnum);
	bool xmit_more = netdev_xmit_more();
	bool use_napi = sq->napi.weight;
	bool kick;

	if (!use_napi)
		free_old_xmit(sq, txq, false);
	else
		virtqueue_disable_cb(sq->vq);

	/* timestamp packet in software */
	skb_tx_timestamp(skb);

	/* Try to transmit */
	err = xmit_skb(sq, skb, !use_napi);

	/* This should not happen! */
	if (unlikely(err)) {
		DEV_STATS_INC(dev, tx_fifo_errors);
		if (net_ratelimit())
			dev_warn(&dev->dev,
				 "Unexpected TXQ (%d) queue failure: %d\n",
				 qnum, err);
		DEV_STATS_INC(dev, tx_dropped);
		dev_kfree_skb_any(skb);
		return NETDEV_TX_OK;
	}

	/* Don't wait up for transmitted skbs to be freed. */
	if (!use_napi) {
		skb_orphan(skb);
		skb_dst_drop(skb);
		nf_reset_ct(skb);
	}

	if (use_napi)
		tx_may_stop(vi, dev, sq);
	else
		check_sq_full_and_disable(vi, dev,sq);

	kick = use_napi ? __netdev_tx_sent_queue(txq, skb->len, xmit_more) :
			  !xmit_more || netif_xmit_stopped(txq);
	if (kick) {
		if (virtqueue_kick_prepare(sq->vq) && virtqueue_notify(sq->vq)) {
			u64_stats_update_begin(&sq->stats.syncp);
			u64_stats_inc(&sq->stats.kicks);
			u64_stats_update_end(&sq->stats.syncp);
		}
	}

	if (use_napi && kick && unlikely(!virtqueue_enable_cb_delayed(sq->vq)))
		virtqueue_napi_schedule(&sq->napi, sq->vq);

	return NETDEV_TX_OK;
}

static void virtnet_rx_pause(struct virtnet_info *vi,
			     struct receive_queue *rq)
{
	bool running = netif_running(vi->dev);

	if (running) {
		virtnet_napi_disable(rq);
		virtnet_cancel_dim(vi, &rq->dim);
	}
}

static void virtnet_rx_pause_all(struct virtnet_info *vi)
{
	int i;

	for (i = 0; i < vi->max_queue_pairs; i++)
		virtnet_rx_pause(vi, &vi->rq[i]);
}

static void virtnet_rx_resume(struct virtnet_info *vi,
			      struct receive_queue *rq,
			      bool refill)
{
	if (netif_running(vi->dev)) {
		/* Pre-fill rq agressively, to make sure we are ready to get
		 * packets immediately.
		 */
		if (refill)
			try_fill_recv(vi, rq, GFP_KERNEL);

		virtnet_napi_enable(rq);
	}
}

static void virtnet_rx_resume_all(struct virtnet_info *vi)
{
	int i;

	for (i = 0; i < vi->max_queue_pairs; i++) {
		if (i < vi->curr_queue_pairs)
			virtnet_rx_resume(vi, &vi->rq[i], true);
		else
			virtnet_rx_resume(vi, &vi->rq[i], false);
	}
}

static int virtnet_rx_resize(struct virtnet_info *vi,
			     struct receive_queue *rq, u32 ring_num)
{
	int err, qindex;

	qindex = rq - vi->rq;

	virtnet_rx_pause(vi, rq);

	err = virtqueue_resize(rq->vq, ring_num, virtnet_rq_unmap_free_buf, NULL);
	if (err)
		netdev_err(vi->dev, "resize rx fail: rx queue index: %d err: %d\n", qindex, err);

	virtnet_rx_resume(vi, rq, true);
	return err;
}

static void virtnet_tx_pause(struct virtnet_info *vi, struct send_queue *sq)
{
	bool running = netif_running(vi->dev);
	struct netdev_queue *txq;
	int qindex;

	qindex = sq - vi->sq;

	if (running)
		virtnet_napi_tx_disable(sq);

	txq = netdev_get_tx_queue(vi->dev, qindex);

	/* 1. wait all ximt complete
	 * 2. fix the race of netif_stop_subqueue() vs netif_start_subqueue()
	 */
	__netif_tx_lock_bh(txq);

	/* Prevent rx poll from accessing sq. */
	sq->reset = true;

	/* Prevent the upper layer from trying to send packets. */
	netif_stop_subqueue(vi->dev, qindex);
	u64_stats_update_begin(&sq->stats.syncp);
	u64_stats_inc(&sq->stats.stop);
	u64_stats_update_end(&sq->stats.syncp);

	__netif_tx_unlock_bh(txq);
}

static void virtnet_tx_resume(struct virtnet_info *vi, struct send_queue *sq)
{
	bool running = netif_running(vi->dev);
	struct netdev_queue *txq;
	int qindex;

	qindex = sq - vi->sq;

	txq = netdev_get_tx_queue(vi->dev, qindex);

	__netif_tx_lock_bh(txq);
	sq->reset = false;
	virtnet_tx_wake_queue(vi, sq);
	__netif_tx_unlock_bh(txq);

	if (running)
		virtnet_napi_tx_enable(sq);
}

static int virtnet_tx_resize(struct virtnet_info *vi, struct send_queue *sq,
			     u32 ring_num)
{
	int qindex, err;

	if (ring_num <= MAX_SKB_FRAGS + 2) {
		netdev_err(vi->dev, "tx size (%d) cannot be smaller than %d\n",
			   ring_num, MAX_SKB_FRAGS + 2);
		return -EINVAL;
	}

	qindex = sq - vi->sq;

	virtnet_tx_pause(vi, sq);

	err = virtqueue_resize(sq->vq, ring_num, virtnet_sq_free_unused_buf,
			       virtnet_sq_free_unused_buf_done);
	if (err)
		netdev_err(vi->dev, "resize tx fail: tx queue index: %d err: %d\n", qindex, err);

	virtnet_tx_resume(vi, sq);

	return err;
}

/*
 * Send command via the control virtqueue and check status.  Commands
 * supported by the hypervisor, as indicated by feature bits, should
 * never fail unless improperly formatted.
 */
static bool virtnet_send_command_reply(struct virtnet_info *vi, u8 class, u8 cmd,
				       struct scatterlist *out,
				       struct scatterlist *in)
{
	struct scatterlist *sgs[5], hdr, stat;
	u32 out_num = 0, tmp, in_num = 0;
	bool ok;
	int ret;

	/* Caller should know better */
	BUG_ON(!virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_VQ));

	mutex_lock(&vi->cvq_lock);
	vi->ctrl->status = ~0;
	vi->ctrl->hdr.class = class;
	vi->ctrl->hdr.cmd = cmd;
	/* Add header */
	sg_init_one(&hdr, &vi->ctrl->hdr, sizeof(vi->ctrl->hdr));
	sgs[out_num++] = &hdr;

	if (out)
		sgs[out_num++] = out;

	/* Add return status. */
	sg_init_one(&stat, &vi->ctrl->status, sizeof(vi->ctrl->status));
	sgs[out_num + in_num++] = &stat;

	if (in)
		sgs[out_num + in_num++] = in;

	BUG_ON(out_num + in_num > ARRAY_SIZE(sgs));
	ret = virtqueue_add_sgs(vi->cvq, sgs, out_num, in_num, vi, GFP_ATOMIC);
	if (ret < 0) {
		dev_warn(&vi->vdev->dev,
			 "Failed to add sgs for command vq: %d\n.", ret);
		mutex_unlock(&vi->cvq_lock);
		return false;
	}

	if (unlikely(!virtqueue_kick(vi->cvq)))
		goto unlock;

	/* Spin for a response, the kick causes an ioport write, trapping
	 * into the hypervisor, so the request should be handled immediately.
	 */
	while (!virtqueue_get_buf(vi->cvq, &tmp) &&
	       !virtqueue_is_broken(vi->cvq)) {
		cond_resched();
		cpu_relax();
	}

unlock:
	ok = vi->ctrl->status == VIRTIO_NET_OK;
	mutex_unlock(&vi->cvq_lock);
	return ok;
}

static bool virtnet_send_command(struct virtnet_info *vi, u8 class, u8 cmd,
				 struct scatterlist *out)
{
	return virtnet_send_command_reply(vi, class, cmd, out, NULL);
}

static int virtnet_set_mac_address(struct net_device *dev, void *p)
{
	struct virtnet_info *vi = netdev_priv(dev);
	struct virtio_device *vdev = vi->vdev;
	int ret;
	struct sockaddr *addr;
	struct scatterlist sg;

	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_STANDBY))
		return -EOPNOTSUPP;

	addr = kmemdup(p, sizeof(*addr), GFP_KERNEL);
	if (!addr)
		return -ENOMEM;

	ret = eth_prepare_mac_addr_change(dev, addr);
	if (ret)
		goto out;

	if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_MAC_ADDR)) {
		sg_init_one(&sg, addr->sa_data, dev->addr_len);
		if (!virtnet_send_command(vi, VIRTIO_NET_CTRL_MAC,
					  VIRTIO_NET_CTRL_MAC_ADDR_SET, &sg)) {
			dev_warn(&vdev->dev,
				 "Failed to set mac address by vq command.\n");
			ret = -EINVAL;
			goto out;
		}
	} else if (virtio_has_feature(vdev, VIRTIO_NET_F_MAC) &&
		   !virtio_has_feature(vdev, VIRTIO_F_VERSION_1)) {
		unsigned int i;

		/* Naturally, this has an atomicity problem. */
		for (i = 0; i < dev->addr_len; i++)
			virtio_cwrite8(vdev,
				       offsetof(struct virtio_net_config, mac) +
				       i, addr->sa_data[i]);
	}

	eth_commit_mac_addr_change(dev, p);
	ret = 0;

out:
	kfree(addr);
	return ret;
}

static void virtnet_stats(struct net_device *dev,
			  struct rtnl_link_stats64 *tot)
{
	struct virtnet_info *vi = netdev_priv(dev);
	unsigned int start;
	int i;

	for (i = 0; i < vi->max_queue_pairs; i++) {
		u64 tpackets, tbytes, terrors, rpackets, rbytes, rdrops;
		struct receive_queue *rq = &vi->rq[i];
		struct send_queue *sq = &vi->sq[i];

		do {
			start = u64_stats_fetch_begin(&sq->stats.syncp);
			tpackets = u64_stats_read(&sq->stats.packets);
			tbytes   = u64_stats_read(&sq->stats.bytes);
			terrors  = u64_stats_read(&sq->stats.tx_timeouts);
		} while (u64_stats_fetch_retry(&sq->stats.syncp, start));

		do {
			start = u64_stats_fetch_begin(&rq->stats.syncp);
			rpackets = u64_stats_read(&rq->stats.packets);
			rbytes   = u64_stats_read(&rq->stats.bytes);
			rdrops   = u64_stats_read(&rq->stats.drops);
		} while (u64_stats_fetch_retry(&rq->stats.syncp, start));

		tot->rx_packets += rpackets;
		tot->tx_packets += tpackets;
		tot->rx_bytes   += rbytes;
		tot->tx_bytes   += tbytes;
		tot->rx_dropped += rdrops;
		tot->tx_errors  += terrors;
	}

	tot->tx_dropped = DEV_STATS_READ(dev, tx_dropped);
	tot->tx_fifo_errors = DEV_STATS_READ(dev, tx_fifo_errors);
	tot->rx_length_errors = DEV_STATS_READ(dev, rx_length_errors);
	tot->rx_frame_errors = DEV_STATS_READ(dev, rx_frame_errors);
}

static void virtnet_ack_link_announce(struct virtnet_info *vi)
{
	if (!virtnet_send_command(vi, VIRTIO_NET_CTRL_ANNOUNCE,
				  VIRTIO_NET_CTRL_ANNOUNCE_ACK, NULL))
		dev_warn(&vi->dev->dev, "Failed to ack link announce.\n");
}

static bool virtnet_commit_rss_command(struct virtnet_info *vi);

static void virtnet_rss_update_by_qpairs(struct virtnet_info *vi, u16 queue_pairs)
{
	u32 indir_val = 0;
	int i = 0;

	for (; i < vi->rss_indir_table_size; ++i) {
		indir_val = ethtool_rxfh_indir_default(i, queue_pairs);
		vi->rss_hdr->indirection_table[i] = cpu_to_le16(indir_val);
	}
	vi->rss_trailer.max_tx_vq = cpu_to_le16(queue_pairs);
}

static int virtnet_set_queues(struct virtnet_info *vi, u16 queue_pairs)
{
	struct virtio_net_ctrl_mq *mq __free(kfree) = NULL;
	struct virtio_net_rss_config_hdr *old_rss_hdr;
	struct virtio_net_rss_config_trailer old_rss_trailer;
	struct net_device *dev = vi->dev;
	struct scatterlist sg;

	if (!vi->has_cvq || !virtio_has_feature(vi->vdev, VIRTIO_NET_F_MQ))
		return 0;

	/* Firstly check if we need update rss. Do updating if both (1) rss enabled and
	 * (2) no user configuration.
	 *
	 * During rss command processing, device updates queue_pairs using rss.max_tx_vq. That is,
	 * the device updates queue_pairs together with rss, so we can skip the separate queue_pairs
	 * update (VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET below) and return directly.
	 */
	if (vi->has_rss && !netif_is_rxfh_configured(dev)) {
		old_rss_hdr = vi->rss_hdr;
		old_rss_trailer = vi->rss_trailer;
		vi->rss_hdr = devm_kzalloc(&vi->vdev->dev, virtnet_rss_hdr_size(vi), GFP_KERNEL);
		if (!vi->rss_hdr) {
			vi->rss_hdr = old_rss_hdr;
			return -ENOMEM;
		}

		*vi->rss_hdr = *old_rss_hdr;
		virtnet_rss_update_by_qpairs(vi, queue_pairs);

		if (!virtnet_commit_rss_command(vi)) {
			/* restore ctrl_rss if commit_rss_command failed */
			devm_kfree(&vi->vdev->dev, vi->rss_hdr);
			vi->rss_hdr = old_rss_hdr;
			vi->rss_trailer = old_rss_trailer;

			dev_warn(&dev->dev, "Fail to set num of queue pairs to %d, because committing RSS failed\n",
				 queue_pairs);
			return -EINVAL;
		}
		devm_kfree(&vi->vdev->dev, old_rss_hdr);
		goto succ;
	}

	mq = kzalloc_obj(*mq);
	if (!mq)
		return -ENOMEM;

	mq->virtqueue_pairs = cpu_to_virtio16(vi->vdev, queue_pairs);
	sg_init_one(&sg, mq, sizeof(*mq));

	if (!virtnet_send_command(vi, VIRTIO_NET_CTRL_MQ,
				  VIRTIO_NET_CTRL_MQ_VQ_PAIRS_SET, &sg)) {
		dev_warn(&dev->dev, "Fail to set num of queue pairs to %d\n",
			 queue_pairs);
		return -EINVAL;
	}

	/* Keep max_tx_vq in sync so that a later RSS command does not
	 * revert queue_pairs to a stale value.
	 */
	if (vi->has_rss)
		vi->rss_trailer.max_tx_vq = cpu_to_le16(queue_pairs);
succ:
	vi->curr_queue_pairs = queue_pairs;
	if (dev->flags & IFF_UP) {
		local_bh_disable();
		for (int i = 0; i < vi->curr_queue_pairs; ++i)
			virtqueue_napi_schedule(&vi->rq[i].napi, vi->rq[i].vq);
		local_bh_enable();
	}

	return 0;
}

static int virtnet_close(struct net_device *dev)
{
	struct virtnet_info *vi = netdev_priv(dev);
	int i;

	/* Prevent the config change callback from changing carrier
	 * after close
	 */
	virtio_config_driver_disable(vi->vdev);
	/* Stop getting status/speed updates: we don't care until next
	 * open
	 */
	cancel_work_sync(&vi->config_work);

	for (i = 0; i < vi->max_queue_pairs; i++) {
		virtnet_disable_queue_pair(vi, i);
		virtnet_cancel_dim(vi, &vi->rq[i].dim);
	}

	netif_carrier_off(dev);

	return 0;
}

static void virtnet_rx_mode_work(struct work_struct *work)
{
	struct virtnet_info *vi =
		container_of(work, struct virtnet_info, rx_mode_work);
	u8 *promisc_allmulti  __free(kfree) = NULL;
	struct net_device *dev = vi->dev;
	struct scatterlist sg[2];
	struct virtio_net_ctrl_mac *mac_data;
	struct netdev_hw_addr *ha;
	int uc_count;
	int mc_count;
	void *buf;
	int i;

	/* We can't dynamically set ndo_set_rx_mode, so return gracefully */
	if (!virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_RX))
		return;

	promisc_allmulti = kzalloc_obj(*promisc_allmulti);
	if (!promisc_allmulti) {
		dev_warn(&dev->dev, "Failed to set RX mode, no memory.\n");
		return;
	}

	rtnl_lock();

	*promisc_allmulti = !!(dev->flags & IFF_PROMISC);
	sg_init_one(sg, promisc_allmulti, sizeof(*promisc_allmulti));

	if (!virtnet_send_command(vi, VIRTIO_NET_CTRL_RX,
				  VIRTIO_NET_CTRL_RX_PROMISC, sg))
		dev_warn(&dev->dev, "Failed to %sable promisc mode.\n",
			 *promisc_allmulti ? "en" : "dis");

	*promisc_allmulti = !!(dev->flags & IFF_ALLMULTI);
	sg_init_one(sg, promisc_allmulti, sizeof(*promisc_allmulti));

	if (!virtnet_send_command(vi, VIRTIO_NET_CTRL_RX,
				  VIRTIO_NET_CTRL_RX_ALLMULTI, sg))
		dev_warn(&dev->dev, "Failed to %sable allmulti mode.\n",
			 *promisc_allmulti ? "en" : "dis");

	netif_addr_lock_bh(dev);

	uc_count = netdev_uc_count(dev);
	mc_count = netdev_mc_count(dev);
	/* MAC filter - use one buffer for both lists */
	buf = kzalloc(((uc_count + mc_count) * ETH_ALEN) +
		      (2 * sizeof(mac_data->entries)), GFP_ATOMIC);
	mac_data = buf;
	if (!buf) {
		netif_addr_unlock_bh(dev);
		rtnl_unlock();
		return;
	}

	sg_init_table(sg, 2);

	/* Store the unicast list and count in the front of the buffer */
	mac_data->entries = cpu_to_virtio32(vi->vdev, uc_count);
	i = 0;
	netdev_for_each_uc_addr(ha, dev)
		memcpy(&mac_data->macs[i++][0], ha->addr, ETH_ALEN);

	sg_set_buf(&sg[0], mac_data,
		   sizeof(mac_data->entries) + (uc_count * ETH_ALEN));

	/* multicast list and count fill the end */
	mac_data = (void *)&mac_data->macs[uc_count][0];

	mac_data->entries = cpu_to_virtio32(vi->vdev, mc_count);
	i = 0;
	netdev_for_each_mc_addr(ha, dev)
		memcpy(&mac_data->macs[i++][0], ha->addr, ETH_ALEN);

	netif_addr_unlock_bh(dev);

	sg_set_buf(&sg[1], mac_data,
		   sizeof(mac_data->entries) + (mc_count * ETH_ALEN));

	if (!virtnet_send_command(vi, VIRTIO_NET_CTRL_MAC,
				  VIRTIO_NET_CTRL_MAC_TABLE_SET, sg))
		dev_warn(&dev->dev, "Failed to set MAC filter table.\n");

	rtnl_unlock();

	kfree(buf);
}

static void virtnet_set_rx_mode(struct net_device *dev)
{
	struct virtnet_info *vi = netdev_priv(dev);

	if (vi->rx_mode_work_enabled)
		schedule_work(&vi->rx_mode_work);
}

static int virtnet_vlan_rx_add_vid(struct net_device *dev,
				   __be16 proto, u16 vid)
{
	struct virtnet_info *vi = netdev_priv(dev);
	__virtio16 *_vid __free(kfree) = NULL;
	struct scatterlist sg;

	_vid = kzalloc_obj(*_vid);
	if (!_vid)
		return -ENOMEM;

	*_vid = cpu_to_virtio16(vi->vdev, vid);
	sg_init_one(&sg, _vid, sizeof(*_vid));

	if (!virtnet_send_command(vi, VIRTIO_NET_CTRL_VLAN,
				  VIRTIO_NET_CTRL_VLAN_ADD, &sg))
		dev_warn(&dev->dev, "Failed to add VLAN ID %d.\n", vid);
	return 0;
}

static int virtnet_vlan_rx_kill_vid(struct net_device *dev,
				    __be16 proto, u16 vid)
{
	struct virtnet_info *vi = netdev_priv(dev);
	__virtio16 *_vid __free(kfree) = NULL;
	struct scatterlist sg;

	_vid = kzalloc_obj(*_vid);
	if (!_vid)
		return -ENOMEM;

	*_vid = cpu_to_virtio16(vi->vdev, vid);
	sg_init_one(&sg, _vid, sizeof(*_vid));

	if (!virtnet_send_command(vi, VIRTIO_NET_CTRL_VLAN,
				  VIRTIO_NET_CTRL_VLAN_DEL, &sg))
		dev_warn(&dev->dev, "Failed to kill VLAN ID %d.\n", vid);
	return 0;
}

static void virtnet_clean_affinity(struct virtnet_info *vi)
{
	int i;

	if (vi->affinity_hint_set) {
		for (i = 0; i < vi->max_queue_pairs; i++) {
			virtqueue_set_affinity(vi->rq[i].vq, NULL);
			virtqueue_set_affinity(vi->sq[i].vq, NULL);
		}

		vi->affinity_hint_set = false;
	}
}

static void virtnet_set_affinity(struct virtnet_info *vi)
{
	cpumask_var_t mask;
	int stragglers;
	int group_size;
	int i, start = 0, cpu;
	int num_cpu;
	int stride;

	if (!zalloc_cpumask_var(&mask, GFP_KERNEL)) {
		virtnet_clean_affinity(vi);
		return;
	}

	num_cpu = num_online_cpus();
	stride = max_t(int, num_cpu / vi->curr_queue_pairs, 1);
	stragglers = num_cpu >= vi->curr_queue_pairs ?
			num_cpu % vi->curr_queue_pairs :
			0;

	for (i = 0; i < vi->curr_queue_pairs; i++) {
		group_size = stride + (i < stragglers ? 1 : 0);

		for_each_online_cpu_wrap(cpu, start) {
			if (!group_size--) {
				start = cpu;
				break;
			}
			cpumask_set_cpu(cpu, mask);
		}

		virtqueue_set_affinity(vi->rq[i].vq, mask);
		virtqueue_set_affinity(vi->sq[i].vq, mask);
		__netif_set_xps_queue(vi->dev, cpumask_bits(mask), i, XPS_CPUS);
		cpumask_clear(mask);
	}

	vi->affinity_hint_set = true;
	free_cpumask_var(mask);
}

static int virtnet_cpu_online(unsigned int cpu, struct hlist_node *node)
{
	struct virtnet_info *vi = hlist_entry_safe(node, struct virtnet_info,
						   node);
	virtnet_set_affinity(vi);
	return 0;
}

static int virtnet_cpu_dead(unsigned int cpu, struct hlist_node *node)
{
	struct virtnet_info *vi = hlist_entry_safe(node, struct virtnet_info,
						   node_dead);
	virtnet_set_affinity(vi);
	return 0;
}

static int virtnet_cpu_down_prep(unsigned int cpu, struct hlist_node *node)
{
	struct virtnet_info *vi = hlist_entry_safe(node, struct virtnet_info,
						   node);

	virtnet_clean_affinity(vi);
	return 0;
}

static enum cpuhp_state virtionet_online;

static int virtnet_cpu_notif_add(struct virtnet_info *vi)
{
	int ret;

	ret = cpuhp_state_add_instance_nocalls(virtionet_online, &vi->node);
	if (ret)
		return ret;
	ret = cpuhp_state_add_instance_nocalls(CPUHP_VIRT_NET_DEAD,
					       &vi->node_dead);
	if (!ret)
		return ret;
	cpuhp_state_remove_instance_nocalls(virtionet_online, &vi->node);
	return ret;
}

static void virtnet_cpu_notif_remove(struct virtnet_info *vi)
{
	cpuhp_state_remove_instance_nocalls(virtionet_online, &vi->node);
	cpuhp_state_remove_instance_nocalls(CPUHP_VIRT_NET_DEAD,
					    &vi->node_dead);
}

static int virtnet_send_ctrl_coal_vq_cmd(struct virtnet_info *vi,
					 u16 vqn, u32 max_usecs, u32 max_packets)
{
	struct virtio_net_ctrl_coal_vq *coal_vq __free(kfree) = NULL;
	struct scatterlist sgs;

	coal_vq = kzalloc_obj(*coal_vq);
	if (!coal_vq)
		return -ENOMEM;

	coal_vq->vqn = cpu_to_le16(vqn);
	coal_vq->coal.max_usecs = cpu_to_le32(max_usecs);
	coal_vq->coal.max_packets = cpu_to_le32(max_packets);
	sg_init_one(&sgs, coal_vq, sizeof(*coal_vq));

	if (!virtnet_send_command(vi, VIRTIO_NET_CTRL_NOTF_COAL,
				  VIRTIO_NET_CTRL_NOTF_COAL_VQ_SET,
				  &sgs))
		return -EINVAL;

	return 0;
}

static int virtnet_send_rx_ctrl_coal_vq_cmd(struct virtnet_info *vi,
					    u16 queue, u32 max_usecs,
					    u32 max_packets)
{
	int err;

	if (!virtio_has_feature(vi->vdev, VIRTIO_NET_F_VQ_NOTF_COAL))
		return -EOPNOTSUPP;

	err = virtnet_send_ctrl_coal_vq_cmd(vi, rxq2vq(queue),
					    max_usecs, max_packets);
	if (err)
		return err;

	vi->rq[queue].intr_coal.max_usecs = max_usecs;
	vi->rq[queue].intr_coal.max_packets = max_packets;

	return 0;
}

static int virtnet_send_tx_ctrl_coal_vq_cmd(struct virtnet_info *vi,
					    u16 queue, u32 max_usecs,
					    u32 max_packets)
{
	int err;

	if (!virtio_has_feature(vi->vdev, VIRTIO_NET_F_VQ_NOTF_COAL))
		return -EOPNOTSUPP;

	err = virtnet_send_ctrl_coal_vq_cmd(vi, txq2vq(queue),
					    max_usecs, max_packets);
	if (err)
		return err;

	vi->sq[queue].intr_coal.max_usecs = max_usecs;
	vi->sq[queue].intr_coal.max_packets = max_packets;

	return 0;
}

static void virtnet_get_ringparam(struct net_device *dev,
				  struct ethtool_ringparam *ring,
				  struct kernel_ethtool_ringparam *kernel_ring,
				  struct netlink_ext_ack *extack)
{
	struct virtnet_info *vi = netdev_priv(dev);

	ring->rx_max_pending = vi->rq[0].vq->num_max;
	ring->tx_max_pending = vi->sq[0].vq->num_max;
	ring->rx_pending = virtqueue_get_vring_size(vi->rq[0].vq);
	ring->tx_pending = virtqueue_get_vring_size(vi->sq[0].vq);
}

static int virtnet_set_ringparam(struct net_device *dev,
				 struct ethtool_ringparam *ring,
				 struct kernel_ethtool_ringparam *kernel_ring,
				 struct netlink_ext_ack *extack)
{
	struct virtnet_info *vi = netdev_priv(dev);
	u32 rx_pending, tx_pending;
	struct receive_queue *rq;
	struct send_queue *sq;
	int i, err;

	if (ring->rx_mini_pending || ring->rx_jumbo_pending)
		return -EINVAL;

	rx_pending = virtqueue_get_vring_size(vi->rq[0].vq);
	tx_pending = virtqueue_get_vring_size(vi->sq[0].vq);

	if (ring->rx_pending == rx_pending &&
	    ring->tx_pending == tx_pending)
		return 0;

	if (ring->rx_pending > vi->rq[0].vq->num_max)
		return -EINVAL;

	if (ring->tx_pending > vi->sq[0].vq->num_max)
		return -EINVAL;

	for (i = 0; i < vi->max_queue_pairs; i++) {
		rq = vi->rq + i;
		sq = vi->sq + i;

		if (ring->tx_pending != tx_pending) {
			err = virtnet_tx_resize(vi, sq, ring->tx_pending);
			if (err)
				return err;

			/* Upon disabling and re-enabling a transmit virtqueue, the device must
			 * set the coalescing parameters of the virtqueue to those configured
			 * through the VIRTIO_NET_CTRL_NOTF_COAL_TX_SET command, or, if the driver
			 * did not set any TX coalescing parameters, to 0.
			 */
			err = virtnet_send_tx_ctrl_coal_vq_cmd(vi, i,
							       vi->intr_coal_tx.max_usecs,
							       vi->intr_coal_tx.max_packets);

			/* Don't break the tx resize action if the vq coalescing is not
			 * supported. The same is true for rx resize below.
			 */
			if (err && err != -EOPNOTSUPP)
				return err;
		}

		if (ring->rx_pending != rx_pending) {
			err = virtnet_rx_resize(vi, rq, ring->rx_pending);
			if (err)
				return err;

			/* The reason is same as the transmit virtqueue reset */
			mutex_lock(&vi->rq[i].dim_lock);
			err = virtnet_send_rx_ctrl_coal_vq_cmd(vi, i,
							       vi->intr_coal_rx.max_usecs,
							       vi->intr_coal_rx.max_packets);
			mutex_unlock(&vi->rq[i].dim_lock);
			if (err && err != -EOPNOTSUPP)
				return err;
		}
	}

	return 0;
}

static bool virtnet_commit_rss_command(struct virtnet_info *vi)
{
	struct net_device *dev = vi->dev;
	struct scatterlist sgs[2];

	/* prepare sgs */
	sg_init_table(sgs, 2);
	sg_set_buf(&sgs[0], vi->rss_hdr, virtnet_rss_hdr_size(vi));
	sg_set_buf(&sgs[1], &vi->rss_trailer, virtnet_rss_trailer_size(vi));

	if (!virtnet_send_command(vi, VIRTIO_NET_CTRL_MQ,
				  vi->has_rss ? VIRTIO_NET_CTRL_MQ_RSS_CONFIG
				  : VIRTIO_NET_CTRL_MQ_HASH_CONFIG, sgs))
		goto err;

	return true;

err:
	dev_warn(&dev->dev, "VIRTIONET issue with committing RSS sgs\n");
	return false;

}

static void virtnet_init_default_rss(struct virtnet_info *vi)
{
	vi->rss_hdr->hash_types = cpu_to_le32(vi->rss_hash_types_supported);
	vi->rss_hash_types_saved = vi->rss_hash_types_supported;
	vi->rss_hdr->indirection_table_mask = vi->rss_indir_table_size
						? cpu_to_le16(vi->rss_indir_table_size - 1) : 0;
	vi->rss_hdr->unclassified_queue = 0;

	virtnet_rss_update_by_qpairs(vi, vi->curr_queue_pairs);

	vi->rss_trailer.hash_key_length = vi->rss_key_size;

	netdev_rss_key_fill(vi->rss_hash_key_data, vi->rss_key_size);
}

static int virtnet_get_hashflow(struct net_device *dev,
				struct ethtool_rxfh_fields *info)
{
	struct virtnet_info *vi = netdev_priv(dev);

	info->data = 0;
	switch (info->flow_type) {
	case TCP_V4_FLOW:
		if (vi->rss_hash_types_saved & VIRTIO_NET_RSS_HASH_TYPE_TCPv4) {
			info->data = RXH_IP_SRC | RXH_IP_DST |
						 RXH_L4_B_0_1 | RXH_L4_B_2_3;
		} else if (vi->rss_hash_types_saved & VIRTIO_NET_RSS_HASH_TYPE_IPv4) {
			info->data = RXH_IP_SRC | RXH_IP_DST;
		}
		break;
	case TCP_V6_FLOW:
		if (vi->rss_hash_types_saved & VIRTIO_NET_RSS_HASH_TYPE_TCPv6) {
			info->data = RXH_IP_SRC | RXH_IP_DST |
						 RXH_L4_B_0_1 | RXH_L4_B_2_3;
		} else if (vi->rss_hash_types_saved & VIRTIO_NET_RSS_HASH_TYPE_IPv6) {
			info->data = RXH_IP_SRC | RXH_IP_DST;
		}
		break;
	case UDP_V4_FLOW:
		if (vi->rss_hash_types_saved & VIRTIO_NET_RSS_HASH_TYPE_UDPv4) {
			info->data = RXH_IP_SRC | RXH_IP_DST |
						 RXH_L4_B_0_1 | RXH_L4_B_2_3;
		} else if (vi->rss_hash_types_saved & VIRTIO_NET_RSS_HASH_TYPE_IPv4) {
			info->data = RXH_IP_SRC | RXH_IP_DST;
		}
		break;
	case UDP_V6_FLOW:
		if (vi->rss_hash_types_saved & VIRTIO_NET_RSS_HASH_TYPE_UDPv6) {
			info->data = RXH_IP_SRC | RXH_IP_DST |
						 RXH_L4_B_0_1 | RXH_L4_B_2_3;
		} else if (vi->rss_hash_types_saved & VIRTIO_NET_RSS_HASH_TYPE_IPv6) {
			info->data = RXH_IP_SRC | RXH_IP_DST;
		}
		break;
	case IPV4_FLOW:
		if (vi->rss_hash_types_saved & VIRTIO_NET_RSS_HASH_TYPE_IPv4)
			info->data = RXH_IP_SRC | RXH_IP_DST;

		break;
	case IPV6_FLOW:
		if (vi->rss_hash_types_saved & VIRTIO_NET_RSS_HASH_TYPE_IPv6)
			info->data = RXH_IP_SRC | RXH_IP_DST;

		break;
	default:
		info->data = 0;
		break;
	}

	return 0;
}

static int virtnet_set_hashflow(struct net_device *dev,
				const struct ethtool_rxfh_fields *info,
				struct netlink_ext_ack *extack)
{
	struct virtnet_info *vi = netdev_priv(dev);
	u32 new_hashtypes = vi->rss_hash_types_saved;
	bool is_disable = info->data & RXH_DISCARD;
	bool is_l4 = info->data == (RXH_IP_SRC | RXH_IP_DST | RXH_L4_B_0_1 | RXH_L4_B_2_3);

	/* supports only 'sd', 'sdfn' and 'r' */
	if (!((info->data == (RXH_IP_SRC | RXH_IP_DST)) | is_l4 | is_disable))
		return -EINVAL;

	switch (info->flow_type) {
	case TCP_V4_FLOW:
		new_hashtypes &= ~(VIRTIO_NET_RSS_HASH_TYPE_IPv4 | VIRTIO_NET_RSS_HASH_TYPE_TCPv4);
		if (!is_disable)
			new_hashtypes |= VIRTIO_NET_RSS_HASH_TYPE_IPv4
				| (is_l4 ? VIRTIO_NET_RSS_HASH_TYPE_TCPv4 : 0);
		break;
	case UDP_V4_FLOW:
		new_hashtypes &= ~(VIRTIO_NET_RSS_HASH_TYPE_IPv4 | VIRTIO_NET_RSS_HASH_TYPE_UDPv4);
		if (!is_disable)
			new_hashtypes |= VIRTIO_NET_RSS_HASH_TYPE_IPv4
				| (is_l4 ? VIRTIO_NET_RSS_HASH_TYPE_UDPv4 : 0);
		break;
	case IPV4_FLOW:
		new_hashtypes &= ~VIRTIO_NET_RSS_HASH_TYPE_IPv4;
		if (!is_disable)
			new_hashtypes = VIRTIO_NET_RSS_HASH_TYPE_IPv4;
		break;
	case TCP_V6_FLOW:
		new_hashtypes &= ~(VIRTIO_NET_RSS_HASH_TYPE_IPv6 | VIRTIO_NET_RSS_HASH_TYPE_TCPv6);
		if (!is_disable)
			new_hashtypes |= VIRTIO_NET_RSS_HASH_TYPE_IPv6
				| (is_l4 ? VIRTIO_NET_RSS_HASH_TYPE_TCPv6 : 0);
		break;
	case UDP_V6_FLOW:
		new_hashtypes &= ~(VIRTIO_NET_RSS_HASH_TYPE_IPv6 | VIRTIO_NET_RSS_HASH_TYPE_UDPv6);
		if (!is_disable)
			new_hashtypes |= VIRTIO_NET_RSS_HASH_TYPE_IPv6
				| (is_l4 ? VIRTIO_NET_RSS_HASH_TYPE_UDPv6 : 0);
		break;
	case IPV6_FLOW:
		new_hashtypes &= ~VIRTIO_NET_RSS_HASH_TYPE_IPv6;
		if (!is_disable)
			new_hashtypes = VIRTIO_NET_RSS_HASH_TYPE_IPv6;
		break;
	default:
		/* unsupported flow */
		return -EINVAL;
	}

	/* if unsupported hashtype was set */
	if (new_hashtypes != (new_hashtypes & vi->rss_hash_types_supported))
		return -EINVAL;

	if (new_hashtypes != vi->rss_hash_types_saved) {
		vi->rss_hash_types_saved = new_hashtypes;
		vi->rss_hdr->hash_types = cpu_to_le32(vi->rss_hash_types_saved);
		if (vi->dev->features & NETIF_F_RXHASH)
			if (!virtnet_commit_rss_command(vi))
				return -EINVAL;
	}

	return 0;
}

static void virtnet_get_drvinfo(struct net_device *dev,
				struct ethtool_drvinfo *info)
{
	struct virtnet_info *vi = netdev_priv(dev);
	struct virtio_device *vdev = vi->vdev;

	strscpy(info->driver, KBUILD_MODNAME, sizeof(info->driver));
	strscpy(info->version, VIRTNET_DRIVER_VERSION, sizeof(info->version));
	strscpy(info->bus_info, virtio_bus_name(vdev), sizeof(info->bus_info));

}

/* TODO: Eliminate OOO packets during switching */
static int virtnet_set_channels(struct net_device *dev,
				struct ethtool_channels *channels)
{
	struct virtnet_info *vi = netdev_priv(dev);
	u16 queue_pairs = channels->combined_count;
	int err;

	/* We don't support separate rx/tx channels.
	 * We don't allow setting 'other' channels.
	 */
	if (channels->rx_count || channels->tx_count || channels->other_count)
		return -EINVAL;

	if (queue_pairs > vi->max_queue_pairs || queue_pairs == 0)
		return -EINVAL;

	/* For now we don't support modifying channels while XDP is loaded
	 * also when XDP is loaded all RX queues have XDP programs so we only
	 * need to check a single RX queue.
	 */
	if (vi->rq[0].xdp_prog)
		return -EINVAL;

	cpus_read_lock();
	err = virtnet_set_queues(vi, queue_pairs);
	if (err) {
		cpus_read_unlock();
		goto err;
	}
	virtnet_set_affinity(vi);
	cpus_read_unlock();

	netif_set_real_num_tx_queues(dev, queue_pairs);
	netif_set_real_num_rx_queues(dev, queue_pairs);
 err:
	return err;
}

static void virtnet_stats_sprintf(u8 **p, const char *fmt, const char *noq_fmt,
				  int num, int qid, const struct virtnet_stat_desc *desc)
{
	int i;

	if (qid < 0) {
		for (i = 0; i < num; ++i)
			ethtool_sprintf(p, noq_fmt, desc[i].desc);
	} else {
		for (i = 0; i < num; ++i)
			ethtool_sprintf(p, fmt, qid, desc[i].desc);
	}
}

/* qid == -1: for rx/tx queue total field */
static void virtnet_get_stats_string(struct virtnet_info *vi, int type, int qid, u8 **data)
{
	const struct virtnet_stat_desc *desc;
	const char *fmt, *noq_fmt;
	u8 *p = *data;
	u32 num;

	if (type == VIRTNET_Q_TYPE_CQ && qid >= 0) {
		noq_fmt = "cq_hw_%s";

		if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_CVQ) {
			desc = &virtnet_stats_cvq_desc[0];
			num = ARRAY_SIZE(virtnet_stats_cvq_desc);

			virtnet_stats_sprintf(&p, NULL, noq_fmt, num, -1, desc);
		}
	}

	if (type == VIRTNET_Q_TYPE_RX) {
		fmt = "rx%u_%s";
		noq_fmt = "rx_%s";

		desc = &virtnet_rq_stats_desc[0];
		num = ARRAY_SIZE(virtnet_rq_stats_desc);

		virtnet_stats_sprintf(&p, fmt, noq_fmt, num, qid, desc);

		fmt = "rx%u_hw_%s";
		noq_fmt = "rx_hw_%s";

		if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_RX_BASIC) {
			desc = &virtnet_stats_rx_basic_desc[0];
			num = ARRAY_SIZE(virtnet_stats_rx_basic_desc);

			virtnet_stats_sprintf(&p, fmt, noq_fmt, num, qid, desc);
		}

		if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_RX_CSUM) {
			desc = &virtnet_stats_rx_csum_desc[0];
			num = ARRAY_SIZE(virtnet_stats_rx_csum_desc);

			virtnet_stats_sprintf(&p, fmt, noq_fmt, num, qid, desc);
		}

		if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_RX_SPEED) {
			desc = &virtnet_stats_rx_speed_desc[0];
			num = ARRAY_SIZE(virtnet_stats_rx_speed_desc);

			virtnet_stats_sprintf(&p, fmt, noq_fmt, num, qid, desc);
		}
	}

	if (type == VIRTNET_Q_TYPE_TX) {
		fmt = "tx%u_%s";
		noq_fmt = "tx_%s";

		desc = &virtnet_sq_stats_desc[0];
		num = ARRAY_SIZE(virtnet_sq_stats_desc);

		virtnet_stats_sprintf(&p, fmt, noq_fmt, num, qid, desc);

		fmt = "tx%u_hw_%s";
		noq_fmt = "tx_hw_%s";

		if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_TX_BASIC) {
			desc = &virtnet_stats_tx_basic_desc[0];
			num = ARRAY_SIZE(virtnet_stats_tx_basic_desc);

			virtnet_stats_sprintf(&p, fmt, noq_fmt, num, qid, desc);
		}

		if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_TX_GSO) {
			desc = &virtnet_stats_tx_gso_desc[0];
			num = ARRAY_SIZE(virtnet_stats_tx_gso_desc);

			virtnet_stats_sprintf(&p, fmt, noq_fmt, num, qid, desc);
		}

		if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_TX_SPEED) {
			desc = &virtnet_stats_tx_speed_desc[0];
			num = ARRAY_SIZE(virtnet_stats_tx_speed_desc);

			virtnet_stats_sprintf(&p, fmt, noq_fmt, num, qid, desc);
		}
	}

	*data = p;
}

struct virtnet_stats_ctx {
	/* The stats are write to qstats or ethtool -S */
	bool to_qstat;

	/* Used to calculate the offset inside the output buffer. */
	u32 desc_num[3];

	/* The actual supported stat types. */
	u64 bitmap[3];

	/* Used to calculate the reply buffer size. */
	u32 size[3];

	/* Record the output buffer. */
	u64 *data;
};

static void virtnet_stats_ctx_init(struct virtnet_info *vi,
				   struct virtnet_stats_ctx *ctx,
				   u64 *data, bool to_qstat)
{
	u32 queue_type;

	ctx->data = data;
	ctx->to_qstat = to_qstat;

	if (to_qstat) {
		ctx->desc_num[VIRTNET_Q_TYPE_RX] = ARRAY_SIZE(virtnet_rq_stats_desc_qstat);
		ctx->desc_num[VIRTNET_Q_TYPE_TX] = ARRAY_SIZE(virtnet_sq_stats_desc_qstat);

		queue_type = VIRTNET_Q_TYPE_RX;

		if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_RX_BASIC) {
			ctx->bitmap[queue_type]   |= VIRTIO_NET_STATS_TYPE_RX_BASIC;
			ctx->desc_num[queue_type] += ARRAY_SIZE(virtnet_stats_rx_basic_desc_qstat);
			ctx->size[queue_type]     += sizeof(struct virtio_net_stats_rx_basic);
		}

		if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_RX_CSUM) {
			ctx->bitmap[queue_type]   |= VIRTIO_NET_STATS_TYPE_RX_CSUM;
			ctx->desc_num[queue_type] += ARRAY_SIZE(virtnet_stats_rx_csum_desc_qstat);
			ctx->size[queue_type]     += sizeof(struct virtio_net_stats_rx_csum);
		}

		if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_RX_GSO) {
			ctx->bitmap[queue_type]   |= VIRTIO_NET_STATS_TYPE_RX_GSO;
			ctx->desc_num[queue_type] += ARRAY_SIZE(virtnet_stats_rx_gso_desc_qstat);
			ctx->size[queue_type]     += sizeof(struct virtio_net_stats_rx_gso);
		}

		if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_RX_SPEED) {
			ctx->bitmap[queue_type]   |= VIRTIO_NET_STATS_TYPE_RX_SPEED;
			ctx->desc_num[queue_type] += ARRAY_SIZE(virtnet_stats_rx_speed_desc_qstat);
			ctx->size[queue_type]     += sizeof(struct virtio_net_stats_rx_speed);
		}

		queue_type = VIRTNET_Q_TYPE_TX;

		if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_TX_BASIC) {
			ctx->bitmap[queue_type]   |= VIRTIO_NET_STATS_TYPE_TX_BASIC;
			ctx->desc_num[queue_type] += ARRAY_SIZE(virtnet_stats_tx_basic_desc_qstat);
			ctx->size[queue_type]     += sizeof(struct virtio_net_stats_tx_basic);
		}

		if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_TX_CSUM) {
			ctx->bitmap[queue_type]   |= VIRTIO_NET_STATS_TYPE_TX_CSUM;
			ctx->desc_num[queue_type] += ARRAY_SIZE(virtnet_stats_tx_csum_desc_qstat);
			ctx->size[queue_type]     += sizeof(struct virtio_net_stats_tx_csum);
		}

		if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_TX_GSO) {
			ctx->bitmap[queue_type]   |= VIRTIO_NET_STATS_TYPE_TX_GSO;
			ctx->desc_num[queue_type] += ARRAY_SIZE(virtnet_stats_tx_gso_desc_qstat);
			ctx->size[queue_type]     += sizeof(struct virtio_net_stats_tx_gso);
		}

		if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_TX_SPEED) {
			ctx->bitmap[queue_type]   |= VIRTIO_NET_STATS_TYPE_TX_SPEED;
			ctx->desc_num[queue_type] += ARRAY_SIZE(virtnet_stats_tx_speed_desc_qstat);
			ctx->size[queue_type]     += sizeof(struct virtio_net_stats_tx_speed);
		}

		return;
	}

	ctx->desc_num[VIRTNET_Q_TYPE_RX] = ARRAY_SIZE(virtnet_rq_stats_desc);
	ctx->desc_num[VIRTNET_Q_TYPE_TX] = ARRAY_SIZE(virtnet_sq_stats_desc);

	if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_CVQ) {
		queue_type = VIRTNET_Q_TYPE_CQ;

		ctx->bitmap[queue_type]   |= VIRTIO_NET_STATS_TYPE_CVQ;
		ctx->desc_num[queue_type] += ARRAY_SIZE(virtnet_stats_cvq_desc);
		ctx->size[queue_type]     += sizeof(struct virtio_net_stats_cvq);
	}

	queue_type = VIRTNET_Q_TYPE_RX;

	if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_RX_BASIC) {
		ctx->bitmap[queue_type]   |= VIRTIO_NET_STATS_TYPE_RX_BASIC;
		ctx->desc_num[queue_type] += ARRAY_SIZE(virtnet_stats_rx_basic_desc);
		ctx->size[queue_type]     += sizeof(struct virtio_net_stats_rx_basic);
	}

	if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_RX_CSUM) {
		ctx->bitmap[queue_type]   |= VIRTIO_NET_STATS_TYPE_RX_CSUM;
		ctx->desc_num[queue_type] += ARRAY_SIZE(virtnet_stats_rx_csum_desc);
		ctx->size[queue_type]     += sizeof(struct virtio_net_stats_rx_csum);
	}

	if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_RX_SPEED) {
		ctx->bitmap[queue_type]   |= VIRTIO_NET_STATS_TYPE_RX_SPEED;
		ctx->desc_num[queue_type] += ARRAY_SIZE(virtnet_stats_rx_speed_desc);
		ctx->size[queue_type]     += sizeof(struct virtio_net_stats_rx_speed);
	}

	queue_type = VIRTNET_Q_TYPE_TX;

	if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_TX_BASIC) {
		ctx->bitmap[queue_type]   |= VIRTIO_NET_STATS_TYPE_TX_BASIC;
		ctx->desc_num[queue_type] += ARRAY_SIZE(virtnet_stats_tx_basic_desc);
		ctx->size[queue_type]     += sizeof(struct virtio_net_stats_tx_basic);
	}

	if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_TX_GSO) {
		ctx->bitmap[queue_type]   |= VIRTIO_NET_STATS_TYPE_TX_GSO;
		ctx->desc_num[queue_type] += ARRAY_SIZE(virtnet_stats_tx_gso_desc);
		ctx->size[queue_type]     += sizeof(struct virtio_net_stats_tx_gso);
	}

	if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_TX_SPEED) {
		ctx->bitmap[queue_type]   |= VIRTIO_NET_STATS_TYPE_TX_SPEED;
		ctx->desc_num[queue_type] += ARRAY_SIZE(virtnet_stats_tx_speed_desc);
		ctx->size[queue_type]     += sizeof(struct virtio_net_stats_tx_speed);
	}
}

/* stats_sum_queue - Calculate the sum of the same fields in sq or rq.
 * @sum: the position to store the sum values
 * @num: field num
 * @q_value: the first queue fields
 * @q_num: number of the queues
 */
static void stats_sum_queue(u64 *sum, u32 num, u64 *q_value, u32 q_num)
{
	u32 step = num;
	int i, j;
	u64 *p;

	for (i = 0; i < num; ++i) {
		p = sum + i;
		*p = 0;

		for (j = 0; j < q_num; ++j)
			*p += *(q_value + i + j * step);
	}
}

static void virtnet_fill_total_fields(struct virtnet_info *vi,
				      struct virtnet_stats_ctx *ctx)
{
	u64 *data, *first_rx_q, *first_tx_q;
	u32 num_cq, num_rx, num_tx;

	num_cq = ctx->desc_num[VIRTNET_Q_TYPE_CQ];
	num_rx = ctx->desc_num[VIRTNET_Q_TYPE_RX];
	num_tx = ctx->desc_num[VIRTNET_Q_TYPE_TX];

	first_rx_q = ctx->data + num_rx + num_tx + num_cq;
	first_tx_q = first_rx_q + vi->curr_queue_pairs * num_rx;

	data = ctx->data;

	stats_sum_queue(data, num_rx, first_rx_q, vi->curr_queue_pairs);

	data = ctx->data + num_rx;

	stats_sum_queue(data, num_tx, first_tx_q, vi->curr_queue_pairs);
}

static void virtnet_fill_stats_qstat(struct virtnet_info *vi, u32 qid,
				     struct virtnet_stats_ctx *ctx,
				     const u8 *base, bool drv_stats, u8 reply_type)
{
	const struct virtnet_stat_desc *desc;
	const u64_stats_t *v_stat;
	u64 offset, bitmap;
	const __le64 *v;
	u32 queue_type;
	int i, num;

	queue_type = vq_type(vi, qid);
	bitmap = ctx->bitmap[queue_type];

	if (drv_stats) {
		if (queue_type == VIRTNET_Q_TYPE_RX) {
			desc = &virtnet_rq_stats_desc_qstat[0];
			num = ARRAY_SIZE(virtnet_rq_stats_desc_qstat);
		} else {
			desc = &virtnet_sq_stats_desc_qstat[0];
			num = ARRAY_SIZE(virtnet_sq_stats_desc_qstat);
		}

		for (i = 0; i < num; ++i) {
			offset = desc[i].qstat_offset / sizeof(*ctx->data);
			v_stat = (const u64_stats_t *)(base + desc[i].offset);
			ctx->data[offset] = u64_stats_read(v_stat);
		}
		return;
	}

	if (bitmap & VIRTIO_NET_STATS_TYPE_RX_BASIC) {
		desc = &virtnet_stats_rx_basic_desc_qstat[0];
		num = ARRAY_SIZE(virtnet_stats_rx_basic_desc_qstat);
		if (reply_type == VIRTIO_NET_STATS_TYPE_REPLY_RX_BASIC)
			goto found;
	}

	if (bitmap & VIRTIO_NET_STATS_TYPE_RX_CSUM) {
		desc = &virtnet_stats_rx_csum_desc_qstat[0];
		num = ARRAY_SIZE(virtnet_stats_rx_csum_desc_qstat);
		if (reply_type == VIRTIO_NET_STATS_TYPE_REPLY_RX_CSUM)
			goto found;
	}

	if (bitmap & VIRTIO_NET_STATS_TYPE_RX_GSO) {
		desc = &virtnet_stats_rx_gso_desc_qstat[0];
		num = ARRAY_SIZE(virtnet_stats_rx_gso_desc_qstat);
		if (reply_type == VIRTIO_NET_STATS_TYPE_REPLY_RX_GSO)
			goto found;
	}

	if (bitmap & VIRTIO_NET_STATS_TYPE_RX_SPEED) {
		desc = &virtnet_stats_rx_speed_desc_qstat[0];
		num = ARRAY_SIZE(virtnet_stats_rx_speed_desc_qstat);
		if (reply_type == VIRTIO_NET_STATS_TYPE_REPLY_RX_SPEED)
			goto found;
	}

	if (bitmap & VIRTIO_NET_STATS_TYPE_TX_BASIC) {
		desc = &virtnet_stats_tx_basic_desc_qstat[0];
		num = ARRAY_SIZE(virtnet_stats_tx_basic_desc_qstat);
		if (reply_type == VIRTIO_NET_STATS_TYPE_REPLY_TX_BASIC)
			goto found;
	}

	if (bitmap & VIRTIO_NET_STATS_TYPE_TX_CSUM) {
		desc = &virtnet_stats_tx_csum_desc_qstat[0];
		num = ARRAY_SIZE(virtnet_stats_tx_csum_desc_qstat);
		if (reply_type == VIRTIO_NET_STATS_TYPE_REPLY_TX_CSUM)
			goto found;
	}

	if (bitmap & VIRTIO_NET_STATS_TYPE_TX_GSO) {
		desc = &virtnet_stats_tx_gso_desc_qstat[0];
		num = ARRAY_SIZE(virtnet_stats_tx_gso_desc_qstat);
		if (reply_type == VIRTIO_NET_STATS_TYPE_REPLY_TX_GSO)
			goto found;
	}

	if (bitmap & VIRTIO_NET_STATS_TYPE_TX_SPEED) {
		desc = &virtnet_stats_tx_speed_desc_qstat[0];
		num = ARRAY_SIZE(virtnet_stats_tx_speed_desc_qstat);
		if (reply_type == VIRTIO_NET_STATS_TYPE_REPLY_TX_SPEED)
			goto found;
	}

	return;

found:
	for (i = 0; i < num; ++i) {
		offset = desc[i].qstat_offset / sizeof(*ctx->data);
		v = (const __le64 *)(base + desc[i].offset);
		ctx->data[offset] = le64_to_cpu(*v);
	}
}

/* virtnet_fill_stats - copy the stats to qstats or ethtool -S
 * The stats source is the device or the driver.
 *
 * @vi: virtio net info
 * @qid: the vq id
 * @ctx: stats ctx (initiated by virtnet_stats_ctx_init())
 * @base: pointer to the device reply or the driver stats structure.
 * @drv_stats: designate the base type (device reply, driver stats)
 * @type: the type of the device reply (if drv_stats is true, this must be zero)
 */
static void virtnet_fill_stats(struct virtnet_info *vi, u32 qid,
			       struct virtnet_stats_ctx *ctx,
			       const u8 *base, bool drv_stats, u8 reply_type)
{
	u32 queue_type, num_rx, num_tx, num_cq;
	const struct virtnet_stat_desc *desc;
	const u64_stats_t *v_stat;
	u64 offset, bitmap;
	const __le64 *v;
	int i, num;

	if (ctx->to_qstat)
		return virtnet_fill_stats_qstat(vi, qid, ctx, base, drv_stats, reply_type);

	num_cq = ctx->desc_num[VIRTNET_Q_TYPE_CQ];
	num_rx = ctx->desc_num[VIRTNET_Q_TYPE_RX];
	num_tx = ctx->desc_num[VIRTNET_Q_TYPE_TX];

	queue_type = vq_type(vi, qid);
	bitmap = ctx->bitmap[queue_type];

	/* skip the total fields of pairs */
	offset = num_rx + num_tx;

	if (queue_type == VIRTNET_Q_TYPE_TX) {
		offset += num_cq + num_rx * vi->curr_queue_pairs + num_tx * (qid / 2);

		num = ARRAY_SIZE(virtnet_sq_stats_desc);
		if (drv_stats) {
			desc = &virtnet_sq_stats_desc[0];
			goto drv_stats;
		}

		offset += num;

	} else if (queue_type == VIRTNET_Q_TYPE_RX) {
		offset += num_cq + num_rx * (qid / 2);

		num = ARRAY_SIZE(virtnet_rq_stats_desc);
		if (drv_stats) {
			desc = &virtnet_rq_stats_desc[0];
			goto drv_stats;
		}

		offset += num;
	}

	if (bitmap & VIRTIO_NET_STATS_TYPE_CVQ) {
		desc = &virtnet_stats_cvq_desc[0];
		num = ARRAY_SIZE(virtnet_stats_cvq_desc);
		if (reply_type == VIRTIO_NET_STATS_TYPE_REPLY_CVQ)
			goto found;

		offset += num;
	}

	if (bitmap & VIRTIO_NET_STATS_TYPE_RX_BASIC) {
		desc = &virtnet_stats_rx_basic_desc[0];
		num = ARRAY_SIZE(virtnet_stats_rx_basic_desc);
		if (reply_type == VIRTIO_NET_STATS_TYPE_REPLY_RX_BASIC)
			goto found;

		offset += num;
	}

	if (bitmap & VIRTIO_NET_STATS_TYPE_RX_CSUM) {
		desc = &virtnet_stats_rx_csum_desc[0];
		num = ARRAY_SIZE(virtnet_stats_rx_csum_desc);
		if (reply_type == VIRTIO_NET_STATS_TYPE_REPLY_RX_CSUM)
			goto found;

		offset += num;
	}

	if (bitmap & VIRTIO_NET_STATS_TYPE_RX_SPEED) {
		desc = &virtnet_stats_rx_speed_desc[0];
		num = ARRAY_SIZE(virtnet_stats_rx_speed_desc);
		if (reply_type == VIRTIO_NET_STATS_TYPE_REPLY_RX_SPEED)
			goto found;

		offset += num;
	}

	if (bitmap & VIRTIO_NET_STATS_TYPE_TX_BASIC) {
		desc = &virtnet_stats_tx_basic_desc[0];
		num = ARRAY_SIZE(virtnet_stats_tx_basic_desc);
		if (reply_type == VIRTIO_NET_STATS_TYPE_REPLY_TX_BASIC)
			goto found;

		offset += num;
	}

	if (bitmap & VIRTIO_NET_STATS_TYPE_TX_GSO) {
		desc = &virtnet_stats_tx_gso_desc[0];
		num = ARRAY_SIZE(virtnet_stats_tx_gso_desc);
		if (reply_type == VIRTIO_NET_STATS_TYPE_REPLY_TX_GSO)
			goto found;

		offset += num;
	}

	if (bitmap & VIRTIO_NET_STATS_TYPE_TX_SPEED) {
		desc = &virtnet_stats_tx_speed_desc[0];
		num = ARRAY_SIZE(virtnet_stats_tx_speed_desc);
		if (reply_type == VIRTIO_NET_STATS_TYPE_REPLY_TX_SPEED)
			goto found;

		offset += num;
	}

	return;

found:
	for (i = 0; i < num; ++i) {
		v = (const __le64 *)(base + desc[i].offset);
		ctx->data[offset + i] = le64_to_cpu(*v);
	}

	return;

drv_stats:
	for (i = 0; i < num; ++i) {
		v_stat = (const u64_stats_t *)(base + desc[i].offset);
		ctx->data[offset + i] = u64_stats_read(v_stat);
	}
}

static int __virtnet_get_hw_stats(struct virtnet_info *vi,
				  struct virtnet_stats_ctx *ctx,
				  struct virtio_net_ctrl_queue_stats *req,
				  int req_size, void *reply, int res_size)
{
	struct virtio_net_stats_reply_hdr *hdr;
	struct scatterlist sgs_in, sgs_out;
	void *p;
	u32 qid;
	int ok;

	sg_init_one(&sgs_out, req, req_size);
	sg_init_one(&sgs_in, reply, res_size);

	ok = virtnet_send_command_reply(vi, VIRTIO_NET_CTRL_STATS,
					VIRTIO_NET_CTRL_STATS_GET,
					&sgs_out, &sgs_in);

	if (!ok)
		return ok;

	for (p = reply; p - reply < res_size; p += le16_to_cpu(hdr->size)) {
		hdr = p;
		qid = le16_to_cpu(hdr->vq_index);
		virtnet_fill_stats(vi, qid, ctx, p, false, hdr->type);
	}

	return 0;
}

static void virtnet_make_stat_req(struct virtnet_info *vi,
				  struct virtnet_stats_ctx *ctx,
				  struct virtio_net_ctrl_queue_stats *req,
				  int qid, int *idx)
{
	int qtype = vq_type(vi, qid);
	u64 bitmap = ctx->bitmap[qtype];

	if (!bitmap)
		return;

	req->stats[*idx].vq_index = cpu_to_le16(qid);
	req->stats[*idx].types_bitmap[0] = cpu_to_le64(bitmap);
	*idx += 1;
}

/* qid: -1: get stats of all vq.
 *     > 0: get the stats for the special vq. This must not be cvq.
 */
static int virtnet_get_hw_stats(struct virtnet_info *vi,
				struct virtnet_stats_ctx *ctx, int qid)
{
	int qnum, i, j, res_size, qtype, last_vq, first_vq;
	struct virtio_net_ctrl_queue_stats *req;
	bool enable_cvq;
	void *reply;
	int ok;

	if (!virtio_has_feature(vi->vdev, VIRTIO_NET_F_DEVICE_STATS))
		return 0;

	if (qid == -1) {
		last_vq = vi->curr_queue_pairs * 2 - 1;
		first_vq = 0;
		enable_cvq = true;
	} else {
		last_vq = qid;
		first_vq = qid;
		enable_cvq = false;
	}

	qnum = 0;
	res_size = 0;
	for (i = first_vq; i <= last_vq ; ++i) {
		qtype = vq_type(vi, i);
		if (ctx->bitmap[qtype]) {
			++qnum;
			res_size += ctx->size[qtype];
		}
	}

	if (enable_cvq && ctx->bitmap[VIRTNET_Q_TYPE_CQ]) {
		res_size += ctx->size[VIRTNET_Q_TYPE_CQ];
		qnum += 1;
	}

	req = kzalloc_objs(*req, qnum);
	if (!req)
		return -ENOMEM;

	reply = kmalloc(res_size, GFP_KERNEL);
	if (!reply) {
		kfree(req);
		return -ENOMEM;
	}

	j = 0;
	for (i = first_vq; i <= last_vq ; ++i)
		virtnet_make_stat_req(vi, ctx, req, i, &j);

	if (enable_cvq)
		virtnet_make_stat_req(vi, ctx, req, vi->max_queue_pairs * 2, &j);

	ok = __virtnet_get_hw_stats(vi, ctx, req, sizeof(*req) * j, reply, res_size);

	kfree(req);
	kfree(reply);

	return ok;
}

static void virtnet_get_strings(struct net_device *dev, u32 stringset, u8 *data)
{
	struct virtnet_info *vi = netdev_priv(dev);
	unsigned int i;
	u8 *p = data;

	switch (stringset) {
	case ETH_SS_STATS:
		/* Generate the total field names. */
		virtnet_get_stats_string(vi, VIRTNET_Q_TYPE_RX, -1, &p);
		virtnet_get_stats_string(vi, VIRTNET_Q_TYPE_TX, -1, &p);

		virtnet_get_stats_string(vi, VIRTNET_Q_TYPE_CQ, 0, &p);

		for (i = 0; i < vi->curr_queue_pairs; ++i)
			virtnet_get_stats_string(vi, VIRTNET_Q_TYPE_RX, i, &p);

		for (i = 0; i < vi->curr_queue_pairs; ++i)
			virtnet_get_stats_string(vi, VIRTNET_Q_TYPE_TX, i, &p);
		break;
	}
}

static int virtnet_get_sset_count(struct net_device *dev, int sset)
{
	struct virtnet_info *vi = netdev_priv(dev);
	struct virtnet_stats_ctx ctx = {0};
	u32 pair_count;

	switch (sset) {
	case ETH_SS_STATS:
		virtnet_stats_ctx_init(vi, &ctx, NULL, false);

		pair_count = ctx.desc_num[VIRTNET_Q_TYPE_RX] + ctx.desc_num[VIRTNET_Q_TYPE_TX];

		return pair_count + ctx.desc_num[VIRTNET_Q_TYPE_CQ] +
			vi->curr_queue_pairs * pair_count;
	default:
		return -EOPNOTSUPP;
	}
}

static void virtnet_get_ethtool_stats(struct net_device *dev,
				      struct ethtool_stats *stats, u64 *data)
{
	struct virtnet_info *vi = netdev_priv(dev);
	struct virtnet_stats_ctx ctx = {0};
	unsigned int start, i;
	const u8 *stats_base;

	virtnet_stats_ctx_init(vi, &ctx, data, false);
	if (virtnet_get_hw_stats(vi, &ctx, -1))
		dev_warn(&vi->dev->dev, "Failed to get hw stats.\n");

	for (i = 0; i < vi->curr_queue_pairs; i++) {
		struct receive_queue *rq = &vi->rq[i];
		struct send_queue *sq = &vi->sq[i];

		stats_base = (const u8 *)&rq->stats;
		do {
			start = u64_stats_fetch_begin(&rq->stats.syncp);
			virtnet_fill_stats(vi, i * 2, &ctx, stats_base, true, 0);
		} while (u64_stats_fetch_retry(&rq->stats.syncp, start));

		stats_base = (const u8 *)&sq->stats;
		do {
			start = u64_stats_fetch_begin(&sq->stats.syncp);
			virtnet_fill_stats(vi, i * 2 + 1, &ctx, stats_base, true, 0);
		} while (u64_stats_fetch_retry(&sq->stats.syncp, start));
	}

	virtnet_fill_total_fields(vi, &ctx);
}

static void virtnet_get_channels(struct net_device *dev,
				 struct ethtool_channels *channels)
{
	struct virtnet_info *vi = netdev_priv(dev);

	channels->combined_count = vi->curr_queue_pairs;
	channels->max_combined = vi->max_queue_pairs;
	channels->max_other = 0;
	channels->rx_count = 0;
	channels->tx_count = 0;
	channels->other_count = 0;
}

static int virtnet_set_link_ksettings(struct net_device *dev,
				      const struct ethtool_link_ksettings *cmd)
{
	struct virtnet_info *vi = netdev_priv(dev);

	return ethtool_virtdev_set_link_ksettings(dev, cmd,
						  &vi->speed, &vi->duplex);
}

static int virtnet_get_link_ksettings(struct net_device *dev,
				      struct ethtool_link_ksettings *cmd)
{
	struct virtnet_info *vi = netdev_priv(dev);

	cmd->base.speed = vi->speed;
	cmd->base.duplex = vi->duplex;
	cmd->base.port = PORT_OTHER;

	return 0;
}

static int virtnet_send_tx_notf_coal_cmds(struct virtnet_info *vi,
					  struct ethtool_coalesce *ec)
{
	struct virtio_net_ctrl_coal_tx *coal_tx __free(kfree) = NULL;
	struct scatterlist sgs_tx;
	int i;

	coal_tx = kzalloc_obj(*coal_tx);
	if (!coal_tx)
		return -ENOMEM;

	coal_tx->tx_usecs = cpu_to_le32(ec->tx_coalesce_usecs);
	coal_tx->tx_max_packets = cpu_to_le32(ec->tx_max_coalesced_frames);
	sg_init_one(&sgs_tx, coal_tx, sizeof(*coal_tx));

	if (!virtnet_send_command(vi, VIRTIO_NET_CTRL_NOTF_COAL,
				  VIRTIO_NET_CTRL_NOTF_COAL_TX_SET,
				  &sgs_tx))
		return -EINVAL;

	vi->intr_coal_tx.max_usecs = ec->tx_coalesce_usecs;
	vi->intr_coal_tx.max_packets = ec->tx_max_coalesced_frames;
	for (i = 0; i < vi->max_queue_pairs; i++) {
		vi->sq[i].intr_coal.max_usecs = ec->tx_coalesce_usecs;
		vi->sq[i].intr_coal.max_packets = ec->tx_max_coalesced_frames;
	}

	return 0;
}

static int virtnet_send_rx_notf_coal_cmds(struct virtnet_info *vi,
					  struct ethtool_coalesce *ec)
{
	struct virtio_net_ctrl_coal_rx *coal_rx __free(kfree) = NULL;
	bool rx_ctrl_dim_on = !!ec->use_adaptive_rx_coalesce;
	struct scatterlist sgs_rx;
	int i;

	if (rx_ctrl_dim_on && !virtio_has_feature(vi->vdev, VIRTIO_NET_F_VQ_NOTF_COAL))
		return -EOPNOTSUPP;

	if (rx_ctrl_dim_on && (ec->rx_coalesce_usecs != vi->intr_coal_rx.max_usecs ||
			       ec->rx_max_coalesced_frames != vi->intr_coal_rx.max_packets))
		return -EINVAL;

	if (rx_ctrl_dim_on && !vi->rx_dim_enabled) {
		vi->rx_dim_enabled = true;
		for (i = 0; i < vi->max_queue_pairs; i++) {
			mutex_lock(&vi->rq[i].dim_lock);
			vi->rq[i].dim_enabled = true;
			mutex_unlock(&vi->rq[i].dim_lock);
		}
		return 0;
	}

	coal_rx = kzalloc_obj(*coal_rx);
	if (!coal_rx)
		return -ENOMEM;

	if (!rx_ctrl_dim_on && vi->rx_dim_enabled) {
		vi->rx_dim_enabled = false;
		for (i = 0; i < vi->max_queue_pairs; i++) {
			mutex_lock(&vi->rq[i].dim_lock);
			vi->rq[i].dim_enabled = false;
			mutex_unlock(&vi->rq[i].dim_lock);
		}
	}

	/* Since the per-queue coalescing params can be set,
	 * we need apply the global new params even if they
	 * are not updated.
	 */
	coal_rx->rx_usecs = cpu_to_le32(ec->rx_coalesce_usecs);
	coal_rx->rx_max_packets = cpu_to_le32(ec->rx_max_coalesced_frames);
	sg_init_one(&sgs_rx, coal_rx, sizeof(*coal_rx));

	if (!virtnet_send_command(vi, VIRTIO_NET_CTRL_NOTF_COAL,
				  VIRTIO_NET_CTRL_NOTF_COAL_RX_SET,
				  &sgs_rx))
		return -EINVAL;

	vi->intr_coal_rx.max_usecs = ec->rx_coalesce_usecs;
	vi->intr_coal_rx.max_packets = ec->rx_max_coalesced_frames;
	for (i = 0; i < vi->max_queue_pairs; i++) {
		mutex_lock(&vi->rq[i].dim_lock);
		vi->rq[i].intr_coal.max_usecs = ec->rx_coalesce_usecs;
		vi->rq[i].intr_coal.max_packets = ec->rx_max_coalesced_frames;
		mutex_unlock(&vi->rq[i].dim_lock);
	}

	return 0;
}

static int virtnet_send_notf_coal_cmds(struct virtnet_info *vi,
				       struct ethtool_coalesce *ec)
{
	int err;

	err = virtnet_send_tx_notf_coal_cmds(vi, ec);
	if (err)
		return err;

	err = virtnet_send_rx_notf_coal_cmds(vi, ec);
	if (err)
		return err;

	return 0;
}

static int virtnet_send_rx_notf_coal_vq_cmds(struct virtnet_info *vi,
					     struct ethtool_coalesce *ec,
					     u16 queue)
{
	bool rx_ctrl_dim_on = !!ec->use_adaptive_rx_coalesce;
	u32 max_usecs, max_packets;
	bool cur_rx_dim;
	int err;

	mutex_lock(&vi->rq[queue].dim_lock);
	cur_rx_dim = vi->rq[queue].dim_enabled;
	max_usecs = vi->rq[queue].intr_coal.max_usecs;
	max_packets = vi->rq[queue].intr_coal.max_packets;

	if (rx_ctrl_dim_on && (ec->rx_coalesce_usecs != max_usecs ||
			       ec->rx_max_coalesced_frames != max_packets)) {
		mutex_unlock(&vi->rq[queue].dim_lock);
		return -EINVAL;
	}

	if (rx_ctrl_dim_on && !cur_rx_dim) {
		vi->rq[queue].dim_enabled = true;
		mutex_unlock(&vi->rq[queue].dim_lock);
		return 0;
	}

	if (!rx_ctrl_dim_on && cur_rx_dim)
		vi->rq[queue].dim_enabled = false;

	/* If no params are updated, userspace ethtool will
	 * reject the modification.
	 */
	err = virtnet_send_rx_ctrl_coal_vq_cmd(vi, queue,
					       ec->rx_coalesce_usecs,
					       ec->rx_max_coalesced_frames);
	mutex_unlock(&vi->rq[queue].dim_lock);
	return err;
}

static int virtnet_send_notf_coal_vq_cmds(struct virtnet_info *vi,
					  struct ethtool_coalesce *ec,
					  u16 queue)
{
	int err;

	err = virtnet_send_rx_notf_coal_vq_cmds(vi, ec, queue);
	if (err)
		return err;

	err = virtnet_send_tx_ctrl_coal_vq_cmd(vi, queue,
					       ec->tx_coalesce_usecs,
					       ec->tx_max_coalesced_frames);
	if (err)
		return err;

	return 0;
}

static void virtnet_rx_dim_work(struct work_struct *work)
{
	struct dim *dim = container_of(work, struct dim, work);
	struct receive_queue *rq = container_of(dim,
			struct receive_queue, dim);
	struct virtnet_info *vi = rq->vq->vdev->priv;
	struct net_device *dev = vi->dev;
	struct dim_cq_moder update_moder;
	int qnum, err;

	qnum = rq - vi->rq;

	mutex_lock(&rq->dim_lock);
	if (!rq->dim_enabled)
		goto out;

	update_moder = net_dim_get_rx_irq_moder(dev, dim);
	if (update_moder.usec != rq->intr_coal.max_usecs ||
	    update_moder.pkts != rq->intr_coal.max_packets) {
		err = virtnet_send_rx_ctrl_coal_vq_cmd(vi, qnum,
						       update_moder.usec,
						       update_moder.pkts);
		if (err)
			pr_debug("%s: Failed to send dim parameters on rxq%d\n",
				 dev->name, qnum);
	}
out:
	dim->state = DIM_START_MEASURE;
	mutex_unlock(&rq->dim_lock);
}

static int virtnet_coal_params_supported(struct ethtool_coalesce *ec)
{
	/* usecs coalescing is supported only if VIRTIO_NET_F_NOTF_COAL
	 * or VIRTIO_NET_F_VQ_NOTF_COAL feature is negotiated.
	 */
	if (ec->rx_coalesce_usecs || ec->tx_coalesce_usecs)
		return -EOPNOTSUPP;

	if (ec->tx_max_coalesced_frames > 1 ||
	    ec->rx_max_coalesced_frames != 1)
		return -EINVAL;

	return 0;
}

static int virtnet_should_update_vq_weight(int dev_flags, int weight,
					   int vq_weight, bool *should_update)
{
	if (weight ^ vq_weight) {
		if (dev_flags & IFF_UP)
			return -EBUSY;
		*should_update = true;
	}

	return 0;
}

static int virtnet_set_coalesce(struct net_device *dev,
				struct ethtool_coalesce *ec,
				struct kernel_ethtool_coalesce *kernel_coal,
				struct netlink_ext_ack *extack)
{
	struct virtnet_info *vi = netdev_priv(dev);
	int ret, queue_number, napi_weight, i;
	bool update_napi = false;

	/* Can't change NAPI weight if the link is up */
	napi_weight = ec->tx_max_coalesced_frames ? NAPI_POLL_WEIGHT : 0;
	for (queue_number = 0; queue_number < vi->max_queue_pairs; queue_number++) {
		ret = virtnet_should_update_vq_weight(dev->flags, napi_weight,
						      vi->sq[queue_number].napi.weight,
						      &update_napi);
		if (ret)
			return ret;

		if (update_napi) {
			/* All queues that belong to [queue_number, vi->max_queue_pairs] will be
			 * updated for the sake of simplicity, which might not be necessary
			 */
			break;
		}
	}

	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_NOTF_COAL))
		ret = virtnet_send_notf_coal_cmds(vi, ec);
	else
		ret = virtnet_coal_params_supported(ec);

	if (ret)
		return ret;

	if (update_napi) {
		/* xsk xmit depends on the tx napi. So if xsk is active,
		 * prevent modifications to tx napi.
		 */
		for (i = queue_number; i < vi->max_queue_pairs; i++) {
			if (vi->sq[i].xsk_pool)
				return -EBUSY;
		}

		for (; queue_number < vi->max_queue_pairs; queue_number++)
			vi->sq[queue_number].napi.weight = napi_weight;
	}

	return ret;
}

static int virtnet_get_coalesce(struct net_device *dev,
				struct ethtool_coalesce *ec,
				struct kernel_ethtool_coalesce *kernel_coal,
				struct netlink_ext_ack *extack)
{
	struct virtnet_info *vi = netdev_priv(dev);

	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_NOTF_COAL)) {
		ec->rx_coalesce_usecs = vi->intr_coal_rx.max_usecs;
		ec->tx_coalesce_usecs = vi->intr_coal_tx.max_usecs;
		ec->tx_max_coalesced_frames = vi->intr_coal_tx.max_packets;
		ec->rx_max_coalesced_frames = vi->intr_coal_rx.max_packets;
		ec->use_adaptive_rx_coalesce = vi->rx_dim_enabled;
	} else {
		ec->rx_max_coalesced_frames = 1;

		if (vi->sq[0].napi.weight)
			ec->tx_max_coalesced_frames = 1;
	}

	return 0;
}

static int virtnet_set_per_queue_coalesce(struct net_device *dev,
					  u32 queue,
					  struct ethtool_coalesce *ec)
{
	struct virtnet_info *vi = netdev_priv(dev);
	int ret, napi_weight;
	bool update_napi = false;

	if (queue >= vi->max_queue_pairs)
		return -EINVAL;

	/* Can't change NAPI weight if the link is up */
	napi_weight = ec->tx_max_coalesced_frames ? NAPI_POLL_WEIGHT : 0;
	ret = virtnet_should_update_vq_weight(dev->flags, napi_weight,
					      vi->sq[queue].napi.weight,
					      &update_napi);
	if (ret)
		return ret;

	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_VQ_NOTF_COAL))
		ret = virtnet_send_notf_coal_vq_cmds(vi, ec, queue);
	else
		ret = virtnet_coal_params_supported(ec);

	if (ret)
		return ret;

	if (update_napi)
		vi->sq[queue].napi.weight = napi_weight;

	return 0;
}

static int virtnet_get_per_queue_coalesce(struct net_device *dev,
					  u32 queue,
					  struct ethtool_coalesce *ec)
{
	struct virtnet_info *vi = netdev_priv(dev);

	if (queue >= vi->max_queue_pairs)
		return -EINVAL;

	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_VQ_NOTF_COAL)) {
		mutex_lock(&vi->rq[queue].dim_lock);
		ec->rx_coalesce_usecs = vi->rq[queue].intr_coal.max_usecs;
		ec->tx_coalesce_usecs = vi->sq[queue].intr_coal.max_usecs;
		ec->tx_max_coalesced_frames = vi->sq[queue].intr_coal.max_packets;
		ec->rx_max_coalesced_frames = vi->rq[queue].intr_coal.max_packets;
		ec->use_adaptive_rx_coalesce = vi->rq[queue].dim_enabled;
		mutex_unlock(&vi->rq[queue].dim_lock);
	} else {
		ec->rx_max_coalesced_frames = 1;

		if (vi->sq[queue].napi.weight)
			ec->tx_max_coalesced_frames = 1;
	}

	return 0;
}

static void virtnet_init_settings(struct net_device *dev)
{
	struct virtnet_info *vi = netdev_priv(dev);

	vi->speed = SPEED_UNKNOWN;
	vi->duplex = DUPLEX_UNKNOWN;
}

static u32 virtnet_get_rxfh_key_size(struct net_device *dev)
{
	return ((struct virtnet_info *)netdev_priv(dev))->rss_key_size;
}

static u32 virtnet_get_rxfh_indir_size(struct net_device *dev)
{
	return ((struct virtnet_info *)netdev_priv(dev))->rss_indir_table_size;
}

static int virtnet_get_rxfh(struct net_device *dev,
			    struct ethtool_rxfh_param *rxfh)
{
	struct virtnet_info *vi = netdev_priv(dev);
	int i;

	if (rxfh->indir) {
		for (i = 0; i < vi->rss_indir_table_size; ++i)
			rxfh->indir[i] = le16_to_cpu(vi->rss_hdr->indirection_table[i]);
	}

	if (rxfh->key)
		memcpy(rxfh->key, vi->rss_hash_key_data, vi->rss_key_size);

	rxfh->hfunc = ETH_RSS_HASH_TOP;

	return 0;
}

static int virtnet_set_rxfh(struct net_device *dev,
			    struct ethtool_rxfh_param *rxfh,
			    struct netlink_ext_ack *extack)
{
	struct virtnet_info *vi = netdev_priv(dev);
	bool update = false;
	int i;

	if (rxfh->hfunc != ETH_RSS_HASH_NO_CHANGE &&
	    rxfh->hfunc != ETH_RSS_HASH_TOP)
		return -EOPNOTSUPP;

	if (rxfh->indir) {
		if (!vi->has_rss)
			return -EOPNOTSUPP;

		for (i = 0; i < vi->rss_indir_table_size; ++i)
			vi->rss_hdr->indirection_table[i] = cpu_to_le16(rxfh->indir[i]);
		update = true;
	}

	if (rxfh->key) {
		/* If either _F_HASH_REPORT or _F_RSS are negotiated, the
		 * device provides hash calculation capabilities, that is,
		 * hash_key is configured.
		 */
		if (!vi->has_rss && !vi->has_rss_hash_report)
			return -EOPNOTSUPP;

		memcpy(vi->rss_hash_key_data, rxfh->key, vi->rss_key_size);
		update = true;
	}

	if (update)
		virtnet_commit_rss_command(vi);

	return 0;
}

static u32 virtnet_get_rx_ring_count(struct net_device *dev)
{
	struct virtnet_info *vi = netdev_priv(dev);

	return vi->curr_queue_pairs;
}

static const struct ethtool_ops virtnet_ethtool_ops = {
	.supported_coalesce_params = ETHTOOL_COALESCE_MAX_FRAMES |
		ETHTOOL_COALESCE_USECS | ETHTOOL_COALESCE_USE_ADAPTIVE_RX,
	.get_drvinfo = virtnet_get_drvinfo,
	.get_link = ethtool_op_get_link,
	.get_ringparam = virtnet_get_ringparam,
	.set_ringparam = virtnet_set_ringparam,
	.get_strings = virtnet_get_strings,
	.get_sset_count = virtnet_get_sset_count,
	.get_ethtool_stats = virtnet_get_ethtool_stats,
	.set_channels = virtnet_set_channels,
	.get_channels = virtnet_get_channels,
	.get_ts_info = ethtool_op_get_ts_info,
	.get_link_ksettings = virtnet_get_link_ksettings,
	.set_link_ksettings = virtnet_set_link_ksettings,
	.set_coalesce = virtnet_set_coalesce,
	.get_coalesce = virtnet_get_coalesce,
	.set_per_queue_coalesce = virtnet_set_per_queue_coalesce,
	.get_per_queue_coalesce = virtnet_get_per_queue_coalesce,
	.get_rxfh_key_size = virtnet_get_rxfh_key_size,
	.get_rxfh_indir_size = virtnet_get_rxfh_indir_size,
	.get_rxfh = virtnet_get_rxfh,
	.set_rxfh = virtnet_set_rxfh,
	.get_rxfh_fields = virtnet_get_hashflow,
	.set_rxfh_fields = virtnet_set_hashflow,
	.get_rx_ring_count = virtnet_get_rx_ring_count,
};

static void virtnet_get_queue_stats_rx(struct net_device *dev, int i,
				       struct netdev_queue_stats_rx *stats)
{
	struct virtnet_info *vi = netdev_priv(dev);
	struct receive_queue *rq = &vi->rq[i];
	struct virtnet_stats_ctx ctx = {0};

	virtnet_stats_ctx_init(vi, &ctx, (void *)stats, true);

	virtnet_get_hw_stats(vi, &ctx, i * 2);
	virtnet_fill_stats(vi, i * 2, &ctx, (void *)&rq->stats, true, 0);
}

static void virtnet_get_queue_stats_tx(struct net_device *dev, int i,
				       struct netdev_queue_stats_tx *stats)
{
	struct virtnet_info *vi = netdev_priv(dev);
	struct send_queue *sq = &vi->sq[i];
	struct virtnet_stats_ctx ctx = {0};

	virtnet_stats_ctx_init(vi, &ctx, (void *)stats, true);

	virtnet_get_hw_stats(vi, &ctx, i * 2 + 1);
	virtnet_fill_stats(vi, i * 2 + 1, &ctx, (void *)&sq->stats, true, 0);
}

static void virtnet_get_base_stats(struct net_device *dev,
				   struct netdev_queue_stats_rx *rx,
				   struct netdev_queue_stats_tx *tx)
{
	struct virtnet_info *vi = netdev_priv(dev);

	/* The queue stats of the virtio-net will not be reset. So here we
	 * return 0.
	 */
	rx->bytes = 0;
	rx->packets = 0;

	if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_RX_BASIC) {
		rx->hw_drops = 0;
		rx->hw_drop_overruns = 0;
	}

	if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_RX_CSUM) {
		rx->csum_unnecessary = 0;
		rx->csum_none = 0;
		rx->csum_bad = 0;
	}

	if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_RX_GSO) {
		rx->hw_gro_packets = 0;
		rx->hw_gro_bytes = 0;
		rx->hw_gro_wire_packets = 0;
		rx->hw_gro_wire_bytes = 0;
	}

	if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_RX_SPEED)
		rx->hw_drop_ratelimits = 0;

	tx->bytes = 0;
	tx->packets = 0;
	tx->stop = 0;
	tx->wake = 0;

	if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_TX_BASIC) {
		tx->hw_drops = 0;
		tx->hw_drop_errors = 0;
	}

	if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_TX_CSUM) {
		tx->csum_none = 0;
		tx->needs_csum = 0;
	}

	if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_TX_GSO) {
		tx->hw_gso_packets = 0;
		tx->hw_gso_bytes = 0;
		tx->hw_gso_wire_packets = 0;
		tx->hw_gso_wire_bytes = 0;
	}

	if (vi->device_stats_cap & VIRTIO_NET_STATS_TYPE_TX_SPEED)
		tx->hw_drop_ratelimits = 0;

	netdev_stat_queue_sum(dev,
			      dev->real_num_rx_queues, vi->max_queue_pairs, rx,
			      dev->real_num_tx_queues, vi->max_queue_pairs, tx);
}

static const struct netdev_stat_ops virtnet_stat_ops = {
	.get_queue_stats_rx	= virtnet_get_queue_stats_rx,
	.get_queue_stats_tx	= virtnet_get_queue_stats_tx,
	.get_base_stats		= virtnet_get_base_stats,
};

static void virtnet_freeze_down(struct virtio_device *vdev)
{
	struct virtnet_info *vi = vdev->priv;

	/* Make sure no work handler is accessing the device */
	flush_work(&vi->config_work);
	disable_rx_mode_work(vi);
	flush_work(&vi->rx_mode_work);

	if (netif_running(vi->dev)) {
		rtnl_lock();
		virtnet_close(vi->dev);
		rtnl_unlock();
	}

	netif_tx_lock_bh(vi->dev);
	netif_device_detach(vi->dev);
	netif_tx_unlock_bh(vi->dev);
}

static int init_vqs(struct virtnet_info *vi);

static int virtnet_restore_up(struct virtio_device *vdev)
{
	struct virtnet_info *vi = vdev->priv;
	int err;

	err = init_vqs(vi);
	if (err)
		return err;

	err = virtnet_create_page_pools(vi);
	if (err)
		goto err_del_vqs;

	virtio_device_ready(vdev);

	enable_rx_mode_work(vi);

	if (netif_running(vi->dev)) {
		rtnl_lock();
		err = virtnet_open(vi->dev);
		rtnl_unlock();
		if (err)
			goto err_destroy_pools;
	}

	netif_tx_lock_bh(vi->dev);
	netif_device_attach(vi->dev);
	netif_tx_unlock_bh(vi->dev);
	return 0;

err_destroy_pools:
	virtio_reset_device(vdev);
	free_unused_bufs(vi);
	virtnet_destroy_page_pools(vi);
	virtnet_del_vqs(vi);
	return err;

err_del_vqs:
	virtio_reset_device(vdev);
	virtnet_del_vqs(vi);
	return err;
}

static int virtnet_set_guest_offloads(struct virtnet_info *vi, u64 offloads)
{
	__virtio64 *_offloads __free(kfree) = NULL;
	struct scatterlist sg;

	_offloads = kzalloc_obj(*_offloads);
	if (!_offloads)
		return -ENOMEM;

	*_offloads = cpu_to_virtio64(vi->vdev, offloads);

	sg_init_one(&sg, _offloads, sizeof(*_offloads));

	if (!virtnet_send_command(vi, VIRTIO_NET_CTRL_GUEST_OFFLOADS,
				  VIRTIO_NET_CTRL_GUEST_OFFLOADS_SET, &sg)) {
		dev_warn(&vi->dev->dev, "Fail to set guest offload.\n");
		return -EINVAL;
	}

	return 0;
}

static int virtnet_clear_guest_offloads(struct virtnet_info *vi)
{
	u64 offloads = 0;

	if (!vi->guest_offloads)
		return 0;

	return virtnet_set_guest_offloads(vi, offloads);
}

static int virtnet_restore_guest_offloads(struct virtnet_info *vi)
{
	u64 offloads = vi->guest_offloads;

	if (!vi->guest_offloads)
		return 0;

	return virtnet_set_guest_offloads(vi, offloads);
}

static int virtnet_rq_bind_xsk_pool(struct virtnet_info *vi, struct receive_queue *rq,
				    struct xsk_buff_pool *pool)
{
	int err, qindex;

	qindex = rq - vi->rq;

	if (pool) {
		err = xdp_rxq_info_reg(&rq->xsk_rxq_info, vi->dev, qindex, rq->napi.napi_id);
		if (err < 0)
			return err;

		err = xdp_rxq_info_reg_mem_model(&rq->xsk_rxq_info,
						 MEM_TYPE_XSK_BUFF_POOL, NULL);
		if (err < 0)
			goto unreg;

		xsk_pool_set_rxq_info(pool, &rq->xsk_rxq_info);
	}

	virtnet_rx_pause(vi, rq);

	err = virtqueue_reset(rq->vq, virtnet_rq_unmap_free_buf, NULL);
	if (err) {
		netdev_err(vi->dev, "reset rx fail: rx queue index: %d err: %d\n", qindex, err);

		pool = NULL;
	}

	rq->xsk_pool = pool;

	virtnet_rx_resume(vi, rq, true);

	if (pool)
		return 0;

unreg:
	xdp_rxq_info_unreg(&rq->xsk_rxq_info);
	return err;
}

static int virtnet_sq_bind_xsk_pool(struct virtnet_info *vi,
				    struct send_queue *sq,
				    struct xsk_buff_pool *pool)
{
	int err, qindex;

	qindex = sq - vi->sq;

	virtnet_tx_pause(vi, sq);

	err = virtqueue_reset(sq->vq, virtnet_sq_free_unused_buf,
			      virtnet_sq_free_unused_buf_done);
	if (err) {
		netdev_err(vi->dev, "reset tx fail: tx queue index: %d err: %d\n", qindex, err);
		pool = NULL;
	}

	sq->xsk_pool = pool;

	virtnet_tx_resume(vi, sq);

	return err;
}

static int virtnet_xsk_pool_enable(struct net_device *dev,
				   struct xsk_buff_pool *pool,
				   u16 qid)
{
	struct virtnet_info *vi = netdev_priv(dev);
	struct receive_queue *rq;
	struct device *dma_dev;
	struct send_queue *sq;
	dma_addr_t hdr_dma;
	int err, size;

	if (vi->hdr_len > xsk_pool_get_headroom(pool))
		return -EINVAL;

	/* In big_packets mode, xdp cannot work, so there is no need to
	 * initialize xsk of rq.
	 */
	if (!vi->rq[qid].page_pool)
		return -ENOENT;

	if (qid >= vi->curr_queue_pairs)
		return -EINVAL;

	sq = &vi->sq[qid];
	rq = &vi->rq[qid];

	/* xsk assumes that tx and rx must have the same dma device. The af-xdp
	 * may use one buffer to receive from the rx and reuse this buffer to
	 * send by the tx. So the dma dev of sq and rq must be the same one.
	 *
	 * But vq->dma_dev allows every vq has the respective dma dev. So I
	 * check the dma dev of vq and sq is the same dev.
	 */
	if (virtqueue_dma_dev(rq->vq) != virtqueue_dma_dev(sq->vq))
		return -EINVAL;

	dma_dev = virtqueue_dma_dev(rq->vq);
	if (!dma_dev)
		return -EINVAL;

	size = virtqueue_get_vring_size(rq->vq);

	rq->xsk_buffs = kvzalloc_objs(*rq->xsk_buffs, size);
	if (!rq->xsk_buffs)
		return -ENOMEM;

	hdr_dma = virtqueue_map_single_attrs(sq->vq, &xsk_hdr, vi->hdr_len,
					     DMA_TO_DEVICE, 0);
	if (virtqueue_map_mapping_error(sq->vq, hdr_dma)) {
		err = -ENOMEM;
		goto err_free_buffs;
	}

	err = xsk_pool_dma_map(pool, dma_dev, 0);
	if (err)
		goto err_xsk_map;

	err = virtnet_rq_bind_xsk_pool(vi, rq, pool);
	if (err)
		goto err_rq;

	err = virtnet_sq_bind_xsk_pool(vi, sq, pool);
	if (err)
		goto err_sq;

	/* Now, we do not support tx offload(such as tx csum), so all the tx
	 * virtnet hdr is zero. So all the tx packets can share a single hdr.
	 */
	sq->xsk_hdr_dma_addr = hdr_dma;

	return 0;

err_sq:
	virtnet_rq_bind_xsk_pool(vi, rq, NULL);
err_rq:
	xsk_pool_dma_unmap(pool, 0);
err_xsk_map:
	virtqueue_unmap_single_attrs(rq->vq, hdr_dma, vi->hdr_len,
				     DMA_TO_DEVICE, 0);
err_free_buffs:
	kvfree(rq->xsk_buffs);
	return err;
}

static int virtnet_xsk_pool_disable(struct net_device *dev, u16 qid)
{
	struct virtnet_info *vi = netdev_priv(dev);
	struct xsk_buff_pool *pool;
	struct receive_queue *rq;
	struct send_queue *sq;
	int err;

	if (qid >= vi->curr_queue_pairs)
		return -EINVAL;

	sq = &vi->sq[qid];
	rq = &vi->rq[qid];

	pool = rq->xsk_pool;

	err = virtnet_rq_bind_xsk_pool(vi, rq, NULL);
	err |= virtnet_sq_bind_xsk_pool(vi, sq, NULL);

	xsk_pool_dma_unmap(pool, 0);

	virtqueue_unmap_single_attrs(sq->vq, sq->xsk_hdr_dma_addr,
				     vi->hdr_len, DMA_TO_DEVICE, 0);
	kvfree(rq->xsk_buffs);

	return err;
}

static int virtnet_xsk_pool_setup(struct net_device *dev, struct netdev_bpf *xdp)
{
	if (xdp->xsk.pool)
		return virtnet_xsk_pool_enable(dev, xdp->xsk.pool,
					       xdp->xsk.queue_id);
	else
		return virtnet_xsk_pool_disable(dev, xdp->xsk.queue_id);
}

static int virtnet_xdp_set(struct net_device *dev, struct bpf_prog *prog,
			   struct netlink_ext_ack *extack)
{
	unsigned int room = SKB_DATA_ALIGN(XDP_PACKET_HEADROOM +
					   sizeof(struct skb_shared_info));
	unsigned int max_sz = PAGE_SIZE - room - ETH_HLEN;
	struct virtnet_info *vi = netdev_priv(dev);
	struct bpf_prog *old_prog;
	u16 xdp_qp = 0, curr_qp;
	int i, err;

	if (!virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS)
	    && (virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_TSO4) ||
	        virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_TSO6) ||
	        virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_ECN) ||
		virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_UFO) ||
		virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_CSUM) ||
		virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_USO4) ||
		virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_USO6))) {
		NL_SET_ERR_MSG_MOD(extack, "Can't set XDP while host is implementing GRO_HW/CSUM, disable GRO_HW/CSUM first");
		return -EOPNOTSUPP;
	}

	if (vi->mergeable_rx_bufs && !vi->any_header_sg) {
		NL_SET_ERR_MSG_MOD(extack, "XDP expects header/data in single page, any_header_sg required");
		return -EINVAL;
	}

	if (prog && !prog->aux->xdp_has_frags && dev->mtu > max_sz) {
		NL_SET_ERR_MSG_MOD(extack, "MTU too large to enable XDP without frags");
		netdev_warn(dev, "single-buffer XDP requires MTU less than %u\n", max_sz);
		return -EINVAL;
	}

	curr_qp = vi->curr_queue_pairs - vi->xdp_queue_pairs;
	if (prog)
		xdp_qp = nr_cpu_ids;

	/* XDP requires extra queues for XDP_TX */
	if (curr_qp + xdp_qp > vi->max_queue_pairs) {
		netdev_warn_once(dev, "XDP request %i queues but max is %i. XDP_TX and XDP_REDIRECT will operate in a slower locked tx mode.\n",
				 curr_qp + xdp_qp, vi->max_queue_pairs);
		xdp_qp = 0;
	}

	old_prog = rtnl_dereference(vi->rq[0].xdp_prog);
	if (!prog && !old_prog)
		return 0;

	if (prog)
		bpf_prog_add(prog, vi->max_queue_pairs - 1);

	virtnet_rx_pause_all(vi);

	/* Make sure NAPI is not using any XDP TX queues for RX. */
	if (netif_running(dev)) {
		for (i = 0; i < vi->max_queue_pairs; i++)
			virtnet_napi_tx_disable(&vi->sq[i]);
	}

	if (!prog) {
		for (i = 0; i < vi->max_queue_pairs; i++) {
			rcu_assign_pointer(vi->rq[i].xdp_prog, prog);
			if (i == 0)
				virtnet_restore_guest_offloads(vi);
		}
		synchronize_net();
	}

	err = virtnet_set_queues(vi, curr_qp + xdp_qp);
	if (err)
		goto err;
	netif_set_real_num_rx_queues(dev, curr_qp + xdp_qp);
	vi->xdp_queue_pairs = xdp_qp;

	if (prog) {
		vi->xdp_enabled = true;
		for (i = 0; i < vi->max_queue_pairs; i++) {
			rcu_assign_pointer(vi->rq[i].xdp_prog, prog);
			if (i == 0 && !old_prog)
				virtnet_clear_guest_offloads(vi);
		}
		if (!old_prog)
			xdp_features_set_redirect_target(dev, true);
	} else {
		xdp_features_clear_redirect_target(dev);
		vi->xdp_enabled = false;
	}

	virtnet_rx_resume_all(vi);
	for (i = 0; i < vi->max_queue_pairs; i++) {
		if (old_prog)
			bpf_prog_put(old_prog);
		if (netif_running(dev))
			virtnet_napi_tx_enable(&vi->sq[i]);
	}

	return 0;

err:
	if (!prog) {
		virtnet_clear_guest_offloads(vi);
		for (i = 0; i < vi->max_queue_pairs; i++)
			rcu_assign_pointer(vi->rq[i].xdp_prog, old_prog);
	}

	virtnet_rx_resume_all(vi);
	if (netif_running(dev)) {
		for (i = 0; i < vi->max_queue_pairs; i++)
			virtnet_napi_tx_enable(&vi->sq[i]);
	}
	if (prog)
		bpf_prog_sub(prog, vi->max_queue_pairs - 1);
	return err;
}

static int virtnet_xdp(struct net_device *dev, struct netdev_bpf *xdp)
{
	switch (xdp->command) {
	case XDP_SETUP_PROG:
		return virtnet_xdp_set(dev, xdp->prog, xdp->extack);
	case XDP_SETUP_XSK_POOL:
		return virtnet_xsk_pool_setup(dev, xdp);
	default:
		return -EINVAL;
	}
}

static int virtnet_get_phys_port_name(struct net_device *dev, char *buf,
				      size_t len)
{
	struct virtnet_info *vi = netdev_priv(dev);
	int ret;

	if (!virtio_has_feature(vi->vdev, VIRTIO_NET_F_STANDBY))
		return -EOPNOTSUPP;

	ret = snprintf(buf, len, "sby");
	if (ret >= len)
		return -EOPNOTSUPP;

	return 0;
}

static int virtnet_set_features(struct net_device *dev,
				netdev_features_t features)
{
	struct virtnet_info *vi = netdev_priv(dev);
	u64 offloads;
	int err;

	if ((dev->features ^ features) & NETIF_F_GRO_HW) {
		if (vi->xdp_enabled)
			return -EBUSY;

		if (features & NETIF_F_GRO_HW)
			offloads = vi->guest_offloads_capable;
		else
			offloads = vi->guest_offloads_capable &
				   ~GUEST_OFFLOAD_GRO_HW_MASK;

		err = virtnet_set_guest_offloads(vi, offloads);
		if (err)
			return err;
		vi->guest_offloads = offloads;
	}

	if ((dev->features ^ features) & NETIF_F_RXHASH) {
		if (features & NETIF_F_RXHASH)
			vi->rss_hdr->hash_types = cpu_to_le32(vi->rss_hash_types_saved);
		else
			vi->rss_hdr->hash_types = cpu_to_le32(VIRTIO_NET_HASH_REPORT_NONE);

		if (!virtnet_commit_rss_command(vi))
			return -EINVAL;
	}

	return 0;
}

static void virtnet_tx_timeout(struct net_device *dev, unsigned int txqueue)
{
	struct virtnet_info *priv = netdev_priv(dev);
	struct send_queue *sq = &priv->sq[txqueue];
	struct netdev_queue *txq = netdev_get_tx_queue(dev, txqueue);

	u64_stats_update_begin(&sq->stats.syncp);
	u64_stats_inc(&sq->stats.tx_timeouts);
	u64_stats_update_end(&sq->stats.syncp);

	netdev_err(dev, "TX timeout on queue: %u, sq: %s, vq: 0x%x, name: %s, %u usecs ago\n",
		   txqueue, sq->name, sq->vq->index, sq->vq->name,
		   jiffies_to_usecs(jiffies - READ_ONCE(txq->trans_start)));
}

static int virtnet_init_irq_moder(struct virtnet_info *vi)
{
	u8 profile_flags = 0, coal_flags = 0;
	int ret, i;

	profile_flags |= DIM_PROFILE_RX;
	coal_flags |= DIM_COALESCE_USEC | DIM_COALESCE_PKTS;
	ret = net_dim_init_irq_moder(vi->dev, profile_flags, coal_flags,
				     DIM_CQ_PERIOD_MODE_START_FROM_EQE,
				     0, virtnet_rx_dim_work, NULL);

	if (ret)
		return ret;

	for (i = 0; i < vi->max_queue_pairs; i++)
		net_dim_setting(vi->dev, &vi->rq[i].dim, false);

	return 0;
}

static void virtnet_free_irq_moder(struct virtnet_info *vi)
{
	if (!virtio_has_feature(vi->vdev, VIRTIO_NET_F_VQ_NOTF_COAL))
		return;

	rtnl_lock();
	net_dim_free_irq_moder(vi->dev);
	rtnl_unlock();
}

static const struct net_device_ops virtnet_netdev = {
	.ndo_open            = virtnet_open,
	.ndo_stop   	     = virtnet_close,
	.ndo_start_xmit      = start_xmit,
	.ndo_validate_addr   = eth_validate_addr,
	.ndo_set_mac_address = virtnet_set_mac_address,
	.ndo_set_rx_mode     = virtnet_set_rx_mode,
	.ndo_get_stats64     = virtnet_stats,
	.ndo_vlan_rx_add_vid = virtnet_vlan_rx_add_vid,
	.ndo_vlan_rx_kill_vid = virtnet_vlan_rx_kill_vid,
	.ndo_bpf		= virtnet_xdp,
	.ndo_xdp_xmit		= virtnet_xdp_xmit,
	.ndo_xsk_wakeup         = virtnet_xsk_wakeup,
	.ndo_features_check	= passthru_features_check,
	.ndo_get_phys_port_name	= virtnet_get_phys_port_name,
	.ndo_set_features	= virtnet_set_features,
	.ndo_tx_timeout		= virtnet_tx_timeout,
};

static void virtnet_config_changed_work(struct work_struct *work)
{
	struct virtnet_info *vi =
		container_of(work, struct virtnet_info, config_work);
	u16 v;

	if (virtio_cread_feature(vi->vdev, VIRTIO_NET_F_STATUS,
				 struct virtio_net_config, status, &v) < 0)
		return;

	if (v & VIRTIO_NET_S_ANNOUNCE) {
		netdev_notify_peers(vi->dev);
		virtnet_ack_link_announce(vi);
	}

	/* Ignore unknown (future) status bits */
	v &= VIRTIO_NET_S_LINK_UP;

	if (vi->status == v)
		return;

	vi->status = v;

	if (vi->status & VIRTIO_NET_S_LINK_UP) {
		virtnet_update_settings(vi);
		netif_carrier_on(vi->dev);
		netif_tx_wake_all_queues(vi->dev);
	} else {
		netif_carrier_off(vi->dev);
		netif_tx_stop_all_queues(vi->dev);
	}
}

static void virtnet_config_changed(struct virtio_device *vdev)
{
	struct virtnet_info *vi = vdev->priv;

	schedule_work(&vi->config_work);
}

static void virtnet_free_queues(struct virtnet_info *vi)
{
	int i;

	for (i = 0; i < vi->max_queue_pairs; i++) {
		__netif_napi_del(&vi->rq[i].napi);
		__netif_napi_del(&vi->sq[i].napi);
	}

	/* We called __netif_napi_del(),
	 * we need to respect an RCU grace period before freeing vi->rq
	 */
	synchronize_net();

	kfree(vi->rq);
	kfree(vi->sq);
	kfree(vi->ctrl);
}

static void _free_receive_bufs(struct virtnet_info *vi)
{
	struct bpf_prog *old_prog;
	int i;

	for (i = 0; i < vi->max_queue_pairs; i++) {
		while (vi->rq[i].pages)
			__free_pages(get_a_page(&vi->rq[i], GFP_KERNEL), 0);

		old_prog = rtnl_dereference(vi->rq[i].xdp_prog);
		RCU_INIT_POINTER(vi->rq[i].xdp_prog, NULL);
		if (old_prog)
			bpf_prog_put(old_prog);
	}
}

static void free_receive_bufs(struct virtnet_info *vi)
{
	rtnl_lock();
	_free_receive_bufs(vi);
	rtnl_unlock();
}

static void virtnet_sq_free_unused_buf(struct virtqueue *vq, void *buf)
{
	struct virtnet_info *vi = vq->vdev->priv;
	struct send_queue *sq;
	int i = vq2txq(vq);

	sq = &vi->sq[i];

	switch (virtnet_xmit_ptr_unpack(&buf)) {
	case VIRTNET_XMIT_TYPE_SKB:
	case VIRTNET_XMIT_TYPE_SKB_ORPHAN:
		dev_kfree_skb(buf);
		break;

	case VIRTNET_XMIT_TYPE_XDP:
		xdp_return_frame(buf);
		break;

	case VIRTNET_XMIT_TYPE_XSK:
		xsk_tx_completed(sq->xsk_pool, 1);
		break;
	}
}

static void virtnet_sq_free_unused_buf_done(struct virtqueue *vq)
{
	struct virtnet_info *vi = vq->vdev->priv;
	int i = vq2txq(vq);

	netdev_tx_reset_queue(netdev_get_tx_queue(vi->dev, i));
}

static void free_unused_bufs(struct virtnet_info *vi)
{
	void *buf;
	int i;

	for (i = 0; i < vi->max_queue_pairs; i++) {
		struct virtqueue *vq = vi->sq[i].vq;
		while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
			virtnet_sq_free_unused_buf(vq, buf);
		cond_resched();
	}

	for (i = 0; i < vi->max_queue_pairs; i++) {
		struct virtqueue *vq = vi->rq[i].vq;

		while ((buf = virtqueue_detach_unused_buf(vq)) != NULL)
			virtnet_rq_unmap_free_buf(vq, buf);
		cond_resched();
	}
}

static void virtnet_del_vqs(struct virtnet_info *vi)
{
	struct virtio_device *vdev = vi->vdev;

	virtnet_clean_affinity(vi);

	vdev->config->del_vqs(vdev);

	virtnet_free_queues(vi);
}

/* How large should a single buffer be so a queue full of these can fit at
 * least one full packet?
 * Logic below assumes the mergeable buffer header is used.
 */
static unsigned int mergeable_min_buf_len(struct virtnet_info *vi, struct virtqueue *vq)
{
	const unsigned int hdr_len = vi->hdr_len;
	unsigned int rq_size = virtqueue_get_vring_size(vq);
	unsigned int packet_len = vi->big_packets ? IP_MAX_MTU : vi->dev->max_mtu;
	unsigned int buf_len = hdr_len + ETH_HLEN + VLAN_HLEN + packet_len;
	unsigned int min_buf_len = DIV_ROUND_UP(buf_len, rq_size);

	return max(max(min_buf_len, hdr_len) - hdr_len,
		   (unsigned int)GOOD_PACKET_LEN);
}

static int virtnet_find_vqs(struct virtnet_info *vi)
{
	struct virtqueue_info *vqs_info;
	struct virtqueue **vqs;
	int ret = -ENOMEM;
	int total_vqs;
	bool *ctx;
	u16 i;

	/* We expect 1 RX virtqueue followed by 1 TX virtqueue, followed by
	 * possible N-1 RX/TX queue pairs used in multiqueue mode, followed by
	 * possible control vq.
	 */
	total_vqs = vi->max_queue_pairs * 2 +
		    virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_VQ);

	/* Allocate space for find_vqs parameters */
	vqs = kzalloc_objs(*vqs, total_vqs);
	if (!vqs)
		goto err_vq;
	vqs_info = kzalloc_objs(*vqs_info, total_vqs);
	if (!vqs_info)
		goto err_vqs_info;
	if (vi->mergeable_rx_bufs || !vi->big_packets) {
		ctx = kzalloc_objs(*ctx, total_vqs);
		if (!ctx)
			goto err_ctx;
	} else {
		ctx = NULL;
	}

	/* Parameters for control virtqueue, if any */
	if (vi->has_cvq) {
		vqs_info[total_vqs - 1].name = "control";
	}

	/* Allocate/initialize parameters for send/receive virtqueues */
	for (i = 0; i < vi->max_queue_pairs; i++) {
		vqs_info[rxq2vq(i)].callback = skb_recv_done;
		vqs_info[txq2vq(i)].callback = skb_xmit_done;
		sprintf(vi->rq[i].name, "input.%u", i);
		sprintf(vi->sq[i].name, "output.%u", i);
		vqs_info[rxq2vq(i)].name = vi->rq[i].name;
		vqs_info[txq2vq(i)].name = vi->sq[i].name;
		if (ctx)
			vqs_info[rxq2vq(i)].ctx = true;
	}

	ret = virtio_find_vqs(vi->vdev, total_vqs, vqs, vqs_info, NULL);
	if (ret)
		goto err_find;

	if (vi->has_cvq) {
		vi->cvq = vqs[total_vqs - 1];
		if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_VLAN))
			vi->dev->features |= NETIF_F_HW_VLAN_CTAG_FILTER;
	}

	for (i = 0; i < vi->max_queue_pairs; i++) {
		vi->rq[i].vq = vqs[rxq2vq(i)];
		vi->rq[i].min_buf_len = mergeable_min_buf_len(vi, vi->rq[i].vq);
		vi->sq[i].vq = vqs[txq2vq(i)];
	}
	/* run here: ret == 0. */

err_find:
	kfree(ctx);
err_ctx:
	kfree(vqs_info);
err_vqs_info:
	kfree(vqs);
err_vq:
	return ret;
}

static int virtnet_alloc_queues(struct virtnet_info *vi)
{
	int i;

	if (vi->has_cvq) {
		vi->ctrl = kzalloc_obj(*vi->ctrl);
		if (!vi->ctrl)
			goto err_ctrl;
	} else {
		vi->ctrl = NULL;
	}
	vi->sq = kzalloc_objs(*vi->sq, vi->max_queue_pairs);
	if (!vi->sq)
		goto err_sq;
	vi->rq = kzalloc_objs(*vi->rq, vi->max_queue_pairs);
	if (!vi->rq)
		goto err_rq;

	for (i = 0; i < vi->max_queue_pairs; i++) {
		vi->rq[i].pages = NULL;
		netif_napi_add_config(vi->dev, &vi->rq[i].napi, virtnet_poll,
				      i);
		vi->rq[i].napi.weight = napi_weight;
		netif_napi_add_tx_weight(vi->dev, &vi->sq[i].napi,
					 virtnet_poll_tx,
					 napi_tx ? napi_weight : 0);

		sg_init_table(vi->rq[i].sg, ARRAY_SIZE(vi->rq[i].sg));
		ewma_pkt_len_init(&vi->rq[i].mrg_avg_pkt_len);
		sg_init_table(vi->sq[i].sg, ARRAY_SIZE(vi->sq[i].sg));

		u64_stats_init(&vi->rq[i].stats.syncp);
		u64_stats_init(&vi->sq[i].stats.syncp);
		mutex_init(&vi->rq[i].dim_lock);
	}

	return 0;

err_rq:
	kfree(vi->sq);
err_sq:
	kfree(vi->ctrl);
err_ctrl:
	return -ENOMEM;
}

static int init_vqs(struct virtnet_info *vi)
{
	int ret;

	/* Allocate send & receive queues */
	ret = virtnet_alloc_queues(vi);
	if (ret)
		goto err;

	ret = virtnet_find_vqs(vi);
	if (ret)
		goto err_free;

	cpus_read_lock();
	virtnet_set_affinity(vi);
	cpus_read_unlock();

	return 0;

err_free:
	virtnet_free_queues(vi);
err:
	return ret;
}

#ifdef CONFIG_SYSFS
static ssize_t mergeable_rx_buffer_size_show(struct netdev_rx_queue *queue,
		char *buf)
{
	struct virtnet_info *vi = netdev_priv(queue->dev);
	unsigned int queue_index = get_netdev_rx_queue_index(queue);
	unsigned int headroom = virtnet_get_headroom(vi);
	unsigned int tailroom = headroom ? sizeof(struct skb_shared_info) : 0;
	struct ewma_pkt_len *avg;

	BUG_ON(queue_index >= vi->max_queue_pairs);
	avg = &vi->rq[queue_index].mrg_avg_pkt_len;
	return sprintf(buf, "%u\n",
		       get_mergeable_buf_len(&vi->rq[queue_index], avg,
				       SKB_DATA_ALIGN(headroom + tailroom)));
}

static struct rx_queue_attribute mergeable_rx_buffer_size_attribute =
	__ATTR_RO(mergeable_rx_buffer_size);

static struct attribute *virtio_net_mrg_rx_attrs[] = {
	&mergeable_rx_buffer_size_attribute.attr,
	NULL
};

static const struct attribute_group virtio_net_mrg_rx_group = {
	.name = "virtio_net",
	.attrs = virtio_net_mrg_rx_attrs
};
#endif

static bool virtnet_fail_on_feature(struct virtio_device *vdev,
				    unsigned int fbit,
				    const char *fname, const char *dname)
{
	if (!virtio_has_feature(vdev, fbit))
		return false;

	dev_err(&vdev->dev, "device advertises feature %s but not %s",
		fname, dname);

	return true;
}

#define VIRTNET_FAIL_ON(vdev, fbit, dbit)			\
	virtnet_fail_on_feature(vdev, fbit, #fbit, dbit)

static bool virtnet_validate_features(struct virtio_device *vdev)
{
	if (!virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ) &&
	    (VIRTNET_FAIL_ON(vdev, VIRTIO_NET_F_CTRL_RX,
			     "VIRTIO_NET_F_CTRL_VQ") ||
	     VIRTNET_FAIL_ON(vdev, VIRTIO_NET_F_CTRL_VLAN,
			     "VIRTIO_NET_F_CTRL_VQ") ||
	     VIRTNET_FAIL_ON(vdev, VIRTIO_NET_F_GUEST_ANNOUNCE,
			     "VIRTIO_NET_F_CTRL_VQ") ||
	     VIRTNET_FAIL_ON(vdev, VIRTIO_NET_F_MQ, "VIRTIO_NET_F_CTRL_VQ") ||
	     VIRTNET_FAIL_ON(vdev, VIRTIO_NET_F_CTRL_MAC_ADDR,
			     "VIRTIO_NET_F_CTRL_VQ") ||
	     VIRTNET_FAIL_ON(vdev, VIRTIO_NET_F_RSS,
			     "VIRTIO_NET_F_CTRL_VQ") ||
	     VIRTNET_FAIL_ON(vdev, VIRTIO_NET_F_HASH_REPORT,
			     "VIRTIO_NET_F_CTRL_VQ") ||
	     VIRTNET_FAIL_ON(vdev, VIRTIO_NET_F_NOTF_COAL,
			     "VIRTIO_NET_F_CTRL_VQ") ||
	     VIRTNET_FAIL_ON(vdev, VIRTIO_NET_F_VQ_NOTF_COAL,
			     "VIRTIO_NET_F_CTRL_VQ"))) {
		return false;
	}

	return true;
}

#define MIN_MTU ETH_MIN_MTU
#define MAX_MTU ETH_MAX_MTU

static int virtnet_validate(struct virtio_device *vdev)
{
	if (!vdev->config->get) {
		dev_err(&vdev->dev, "%s failure: config access disabled\n",
			__func__);
		return -EINVAL;
	}

	if (!virtnet_validate_features(vdev))
		return -EINVAL;

	if (virtio_has_feature(vdev, VIRTIO_NET_F_MTU)) {
		int mtu = virtio_cread16(vdev,
					 offsetof(struct virtio_net_config,
						  mtu));
		if (mtu < MIN_MTU)
			__virtio_clear_bit(vdev, VIRTIO_NET_F_MTU);
	}

	if (virtio_has_feature(vdev, VIRTIO_NET_F_STANDBY) &&
	    !virtio_has_feature(vdev, VIRTIO_NET_F_MAC)) {
		dev_warn(&vdev->dev, "device advertises feature VIRTIO_NET_F_STANDBY but not VIRTIO_NET_F_MAC, disabling standby");
		__virtio_clear_bit(vdev, VIRTIO_NET_F_STANDBY);
	}

	return 0;
}

static bool virtnet_check_guest_gso(const struct virtnet_info *vi)
{
	return virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_TSO4) ||
		virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_TSO6) ||
		virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_ECN) ||
		virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_UFO) ||
		(virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_USO4) &&
		virtio_has_feature(vi->vdev, VIRTIO_NET_F_GUEST_USO6));
}

static void virtnet_set_big_packets(struct virtnet_info *vi, const int mtu)
{
	bool guest_gso = virtnet_check_guest_gso(vi);

	/* If device can receive ANY guest GSO packets, regardless of mtu,
	 * allocate packets of maximum size, otherwise limit it to only
	 * mtu size worth only.
	 */
	if (mtu > ETH_DATA_LEN || guest_gso) {
		vi->big_packets = true;
		vi->big_packets_num_skbfrags = guest_gso ? MAX_SKB_FRAGS : DIV_ROUND_UP(mtu, PAGE_SIZE);
	}
}

#define VIRTIO_NET_HASH_REPORT_MAX_TABLE      10
static enum xdp_rss_hash_type
virtnet_xdp_rss_type[VIRTIO_NET_HASH_REPORT_MAX_TABLE] = {
	[VIRTIO_NET_HASH_REPORT_NONE] = XDP_RSS_TYPE_NONE,
	[VIRTIO_NET_HASH_REPORT_IPv4] = XDP_RSS_TYPE_L3_IPV4,
	[VIRTIO_NET_HASH_REPORT_TCPv4] = XDP_RSS_TYPE_L4_IPV4_TCP,
	[VIRTIO_NET_HASH_REPORT_UDPv4] = XDP_RSS_TYPE_L4_IPV4_UDP,
	[VIRTIO_NET_HASH_REPORT_IPv6] = XDP_RSS_TYPE_L3_IPV6,
	[VIRTIO_NET_HASH_REPORT_TCPv6] = XDP_RSS_TYPE_L4_IPV6_TCP,
	[VIRTIO_NET_HASH_REPORT_UDPv6] = XDP_RSS_TYPE_L4_IPV6_UDP,
	[VIRTIO_NET_HASH_REPORT_IPv6_EX] = XDP_RSS_TYPE_L3_IPV6_EX,
	[VIRTIO_NET_HASH_REPORT_TCPv6_EX] = XDP_RSS_TYPE_L4_IPV6_TCP_EX,
	[VIRTIO_NET_HASH_REPORT_UDPv6_EX] = XDP_RSS_TYPE_L4_IPV6_UDP_EX
};

static int virtnet_xdp_rx_hash(const struct xdp_md *_ctx, u32 *hash,
			       enum xdp_rss_hash_type *rss_type)
{
	const struct xdp_buff *xdp = (void *)_ctx;
	struct virtio_net_hdr_v1_hash *hdr_hash;
	struct virtnet_info *vi;
	u16 hash_report;

	if (!(xdp->rxq->dev->features & NETIF_F_RXHASH))
		return -ENODATA;

	vi = netdev_priv(xdp->rxq->dev);
	hdr_hash = (struct virtio_net_hdr_v1_hash *)(xdp->data - vi->hdr_len);
	hash_report = __le16_to_cpu(hdr_hash->hash_report);

	if (hash_report >= VIRTIO_NET_HASH_REPORT_MAX_TABLE)
		hash_report = VIRTIO_NET_HASH_REPORT_NONE;

	*rss_type = virtnet_xdp_rss_type[hash_report];
	*hash = virtio_net_hash_value(hdr_hash);
	return 0;
}

static const struct xdp_metadata_ops virtnet_xdp_metadata_ops = {
	.xmo_rx_hash			= virtnet_xdp_rx_hash,
};

static int virtnet_probe(struct virtio_device *vdev)
{
	int i, err = -ENOMEM;
	struct net_device *dev;
	struct virtnet_info *vi;
	u16 max_queue_pairs;
	int mtu = 0;
	u16 key_sz;

	/* Find if host supports multiqueue/rss virtio_net device */
	max_queue_pairs = 1;
	if (virtio_has_feature(vdev, VIRTIO_NET_F_MQ) || virtio_has_feature(vdev, VIRTIO_NET_F_RSS))
		max_queue_pairs =
		     virtio_cread16(vdev, offsetof(struct virtio_net_config, max_virtqueue_pairs));

	/* We need at least 2 queue's */
	if (max_queue_pairs < VIRTIO_NET_CTRL_MQ_VQ_PAIRS_MIN ||
	    max_queue_pairs > VIRTIO_NET_CTRL_MQ_VQ_PAIRS_MAX ||
	    !virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ))
		max_queue_pairs = 1;

	/* Allocate ourselves a network device with room for our info */
	dev = alloc_etherdev_mq(sizeof(struct virtnet_info), max_queue_pairs);
	if (!dev)
		return -ENOMEM;

	/* Set up network device as normal. */
	dev->priv_flags |= IFF_UNICAST_FLT | IFF_LIVE_ADDR_CHANGE |
			   IFF_TX_SKB_NO_LINEAR;
	dev->netdev_ops = &virtnet_netdev;
	dev->stat_ops = &virtnet_stat_ops;
	dev->features = NETIF_F_HIGHDMA;

	dev->ethtool_ops = &virtnet_ethtool_ops;
	SET_NETDEV_DEV(dev, &vdev->dev);

	/* Do we support "hardware" checksums? */
	if (virtio_has_feature(vdev, VIRTIO_NET_F_CSUM)) {
		/* This opens up the world of extra features. */
		dev->hw_features |= NETIF_F_HW_CSUM | NETIF_F_SG;
		if (csum)
			dev->features |= NETIF_F_HW_CSUM | NETIF_F_SG;

		if (virtio_has_feature(vdev, VIRTIO_NET_F_GSO)) {
			dev->hw_features |= NETIF_F_TSO
				| NETIF_F_TSO_ECN | NETIF_F_TSO6;
		}
		/* Individual feature bits: what can host handle? */
		if (virtio_has_feature(vdev, VIRTIO_NET_F_HOST_TSO4))
			dev->hw_features |= NETIF_F_TSO;
		if (virtio_has_feature(vdev, VIRTIO_NET_F_HOST_TSO6))
			dev->hw_features |= NETIF_F_TSO6;
		if (virtio_has_feature(vdev, VIRTIO_NET_F_HOST_ECN))
			dev->hw_features |= NETIF_F_TSO_ECN;
		if (virtio_has_feature(vdev, VIRTIO_NET_F_HOST_USO))
			dev->hw_features |= NETIF_F_GSO_UDP_L4;

		if (virtio_has_feature(vdev, VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO)) {
			dev->hw_features |= NETIF_F_GSO_UDP_TUNNEL;
			dev->hw_enc_features = dev->hw_features;
		}
		if (dev->hw_features & NETIF_F_GSO_UDP_TUNNEL &&
		    virtio_has_feature(vdev, VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO_CSUM)) {
			dev->hw_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
			dev->hw_enc_features |= NETIF_F_GSO_UDP_TUNNEL_CSUM;
		}

		dev->features |= NETIF_F_GSO_ROBUST;

		if (gso)
			dev->features |= dev->hw_features;
		/* (!csum && gso) case will be fixed by register_netdev() */
	}

	/* 1. With VIRTIO_NET_F_GUEST_CSUM negotiation, the driver doesn't
	 * need to calculate checksums for partially checksummed packets,
	 * as they're considered valid by the upper layer.
	 * 2. Without VIRTIO_NET_F_GUEST_CSUM negotiation, the driver only
	 * receives fully checksummed packets. The device may assist in
	 * validating these packets' checksums, so the driver won't have to.
	 */
	dev->features |= NETIF_F_RXCSUM;

	if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO4) ||
	    virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_TSO6))
		dev->features |= NETIF_F_GRO_HW;

	dev->vlan_features = dev->features;
	dev->xdp_features = NETDEV_XDP_ACT_BASIC | NETDEV_XDP_ACT_REDIRECT |
		NETDEV_XDP_ACT_XSK_ZEROCOPY;

	/* MTU range: 68 - 65535 */
	dev->min_mtu = MIN_MTU;
	dev->max_mtu = MAX_MTU;

	/* Configuration may specify what MAC to use.  Otherwise random. */
	if (virtio_has_feature(vdev, VIRTIO_NET_F_MAC)) {
		u8 addr[ETH_ALEN];

		virtio_cread_bytes(vdev,
				   offsetof(struct virtio_net_config, mac),
				   addr, ETH_ALEN);
		eth_hw_addr_set(dev, addr);
	} else {
		eth_hw_addr_random(dev);
		dev_info(&vdev->dev, "Assigned random MAC address %pM\n",
			 dev->dev_addr);
	}

	/* Set up our device-specific information */
	vi = netdev_priv(dev);
	vi->dev = dev;
	vi->vdev = vdev;
	vdev->priv = vi;

	INIT_WORK(&vi->config_work, virtnet_config_changed_work);
	INIT_WORK(&vi->rx_mode_work, virtnet_rx_mode_work);

	if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF)) {
		vi->mergeable_rx_bufs = true;
		dev->xdp_features |= NETDEV_XDP_ACT_RX_SG;
	}

	if (virtio_has_feature(vdev, VIRTIO_NET_F_HASH_REPORT))
		vi->has_rss_hash_report = true;

	if (virtio_has_feature(vdev, VIRTIO_NET_F_RSS)) {
		vi->has_rss = true;

		vi->rss_indir_table_size =
			virtio_cread16(vdev, offsetof(struct virtio_net_config,
				rss_max_indirection_table_length));
	}
	vi->rss_hdr = devm_kzalloc(&vdev->dev, virtnet_rss_hdr_size(vi), GFP_KERNEL);
	if (!vi->rss_hdr) {
		err = -ENOMEM;
		goto free;
	}

	if (vi->has_rss || vi->has_rss_hash_report) {
		key_sz = virtio_cread8(vdev, offsetof(struct virtio_net_config, rss_max_key_size));

		vi->rss_key_size = min_t(u16, key_sz, NETDEV_RSS_KEY_LEN);
		if (key_sz > vi->rss_key_size)
			dev_warn(&vdev->dev,
				 "rss_max_key_size=%u exceeds driver limit %u, clamping\n",
				 key_sz, vi->rss_key_size);

		vi->rss_hash_types_supported =
		    virtio_cread32(vdev, offsetof(struct virtio_net_config, supported_hash_types));
		vi->rss_hash_types_supported &=
				~(VIRTIO_NET_RSS_HASH_TYPE_IP_EX |
				  VIRTIO_NET_RSS_HASH_TYPE_TCP_EX |
				  VIRTIO_NET_RSS_HASH_TYPE_UDP_EX);

		dev->hw_features |= NETIF_F_RXHASH;
		dev->xdp_metadata_ops = &virtnet_xdp_metadata_ops;
	}

	if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO) ||
	    virtio_has_feature(vdev, VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO))
		vi->hdr_len = sizeof(struct virtio_net_hdr_v1_hash_tunnel);
	else if (vi->has_rss_hash_report)
		vi->hdr_len = sizeof(struct virtio_net_hdr_v1_hash);
	else if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF) ||
		 virtio_has_feature(vdev, VIRTIO_F_VERSION_1))
		vi->hdr_len = sizeof(struct virtio_net_hdr_mrg_rxbuf);
	else
		vi->hdr_len = sizeof(struct virtio_net_hdr);

	if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO_CSUM))
		vi->rx_tnl_csum = true;
	if (virtio_has_feature(vdev, VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO))
		vi->rx_tnl = true;
	if (virtio_has_feature(vdev, VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO))
		vi->tx_tnl = true;

	if (virtio_has_feature(vdev, VIRTIO_F_ANY_LAYOUT) ||
	    virtio_has_feature(vdev, VIRTIO_F_VERSION_1))
		vi->any_header_sg = true;

	if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_VQ))
		vi->has_cvq = true;

	mutex_init(&vi->cvq_lock);

	if (virtio_has_feature(vdev, VIRTIO_NET_F_MTU)) {
		mtu = virtio_cread16(vdev,
				     offsetof(struct virtio_net_config,
					      mtu));
		if (mtu < dev->min_mtu) {
			/* Should never trigger: MTU was previously validated
			 * in virtnet_validate.
			 */
			dev_err(&vdev->dev,
				"device MTU appears to have changed it is now %d < %d",
				mtu, dev->min_mtu);
			err = -EINVAL;
			goto free;
		}

		dev->mtu = mtu;
		dev->max_mtu = mtu;
	}

	virtnet_set_big_packets(vi, mtu);

	if (vi->any_header_sg)
		dev->needed_headroom = vi->hdr_len;

	/* Enable multiqueue by default */
	if (num_online_cpus() >= max_queue_pairs)
		vi->curr_queue_pairs = max_queue_pairs;
	else
		vi->curr_queue_pairs = num_online_cpus();
	vi->max_queue_pairs = max_queue_pairs;

	/* Allocate/initialize the rx/tx queues, and invoke find_vqs */
	err = init_vqs(vi);
	if (err)
		goto free;

	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_NOTF_COAL)) {
		vi->intr_coal_rx.max_usecs = 0;
		vi->intr_coal_tx.max_usecs = 0;
		vi->intr_coal_rx.max_packets = 0;

		/* Keep the default values of the coalescing parameters
		 * aligned with the default napi_tx state.
		 */
		if (vi->sq[0].napi.weight)
			vi->intr_coal_tx.max_packets = 1;
		else
			vi->intr_coal_tx.max_packets = 0;
	}

	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_VQ_NOTF_COAL)) {
		/* The reason is the same as VIRTIO_NET_F_NOTF_COAL. */
		for (i = 0; i < vi->max_queue_pairs; i++)
			if (vi->sq[i].napi.weight)
				vi->sq[i].intr_coal.max_packets = 1;

		err = virtnet_init_irq_moder(vi);
		if (err)
			goto free;
	}

	/* Create page pools for receive queues.
	 * Page pools are created at probe time so they can be used
	 * with premapped DMA addresses throughout the device lifetime.
	 */
	err = virtnet_create_page_pools(vi);
	if (err)
		goto free_irq_moder;

#ifdef CONFIG_SYSFS
	if (vi->mergeable_rx_bufs)
		dev->sysfs_rx_queue_group = &virtio_net_mrg_rx_group;
#endif
	netif_set_real_num_tx_queues(dev, vi->curr_queue_pairs);
	netif_set_real_num_rx_queues(dev, vi->curr_queue_pairs);

	virtnet_init_settings(dev);

	if (virtio_has_feature(vdev, VIRTIO_NET_F_STANDBY)) {
		vi->failover = net_failover_create(vi->dev);
		if (IS_ERR(vi->failover)) {
			err = PTR_ERR(vi->failover);
			goto free_page_pools;
		}
	}

	if (vi->has_rss || vi->has_rss_hash_report)
		virtnet_init_default_rss(vi);

	enable_rx_mode_work(vi);

	for (i = 0; i < ARRAY_SIZE(guest_offloads); i++) {
		unsigned int fbit;

		fbit = virtio_offload_to_feature(guest_offloads[i]);
		if (virtio_has_feature(vi->vdev, fbit))
			set_bit(guest_offloads[i], &vi->guest_offloads);
	}
	vi->guest_offloads_capable = vi->guest_offloads;

	if (virtio_has_feature(vdev, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS) &&
	    (vi->guest_offloads_capable & GUEST_OFFLOAD_GRO_HW_MASK))
		dev->hw_features |= NETIF_F_GRO_HW;

	/* serialize netdev register + virtio_device_ready() with ndo_open() */
	rtnl_lock();

	err = register_netdevice(dev);
	if (err) {
		pr_debug("virtio_net: registering device failed\n");
		rtnl_unlock();
		goto free_failover;
	}

	/* Disable config change notification until ndo_open. */
	virtio_config_driver_disable(vi->vdev);

	virtio_device_ready(vdev);

	if (vi->has_rss || vi->has_rss_hash_report) {
		if (!virtnet_commit_rss_command(vi)) {
			dev_warn(&vdev->dev, "RSS disabled because committing failed.\n");
			dev->hw_features &= ~NETIF_F_RXHASH;
			vi->has_rss_hash_report = false;
			vi->has_rss = false;
		}
	}

	virtnet_set_queues(vi, vi->curr_queue_pairs);

	/* a random MAC address has been assigned, notify the device.
	 * We don't fail probe if VIRTIO_NET_F_CTRL_MAC_ADDR is not there
	 * because many devices work fine without getting MAC explicitly
	 */
	if (!virtio_has_feature(vdev, VIRTIO_NET_F_MAC) &&
	    virtio_has_feature(vi->vdev, VIRTIO_NET_F_CTRL_MAC_ADDR)) {
		struct scatterlist sg;

		sg_init_one(&sg, dev->dev_addr, dev->addr_len);
		if (!virtnet_send_command(vi, VIRTIO_NET_CTRL_MAC,
					  VIRTIO_NET_CTRL_MAC_ADDR_SET, &sg)) {
			pr_debug("virtio_net: setting MAC address failed\n");
			rtnl_unlock();
			err = -EINVAL;
			goto free_unregister_netdev;
		}
	}

	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_DEVICE_STATS)) {
		struct virtio_net_stats_capabilities *stats_cap  __free(kfree) = NULL;
		struct scatterlist sg;
		__le64 v;

		stats_cap = kzalloc_obj(*stats_cap);
		if (!stats_cap) {
			rtnl_unlock();
			err = -ENOMEM;
			goto free_unregister_netdev;
		}

		sg_init_one(&sg, stats_cap, sizeof(*stats_cap));

		if (!virtnet_send_command_reply(vi, VIRTIO_NET_CTRL_STATS,
						VIRTIO_NET_CTRL_STATS_QUERY,
						NULL, &sg)) {
			pr_debug("virtio_net: fail to get stats capability\n");
			rtnl_unlock();
			err = -EINVAL;
			goto free_unregister_netdev;
		}

		v = stats_cap->supported_stats_types[0];
		vi->device_stats_cap = le64_to_cpu(v);
	}

	/* Assume link up if device can't report link status,
	   otherwise get link status from config. */
	netif_carrier_off(dev);
	if (virtio_has_feature(vi->vdev, VIRTIO_NET_F_STATUS)) {
		virtio_config_changed(vi->vdev);
	} else {
		vi->status = VIRTIO_NET_S_LINK_UP;
		virtnet_update_settings(vi);
		netif_carrier_on(dev);
	}

	rtnl_unlock();

	err = virtnet_cpu_notif_add(vi);
	if (err) {
		pr_debug("virtio_net: registering cpu notifier failed\n");
		goto free_unregister_netdev;
	}

	pr_debug("virtnet: registered device %s with %d RX and TX vq's\n",
		 dev->name, max_queue_pairs);

	return 0;

free_unregister_netdev:
	unregister_netdev(dev);
free_failover:
	net_failover_destroy(vi->failover);
free_page_pools:
	virtnet_destroy_page_pools(vi);
free_irq_moder:
	virtnet_free_irq_moder(vi);
	virtio_reset_device(vdev);
	virtnet_del_vqs(vi);
free:
	free_netdev(dev);
	return err;
}

static void remove_vq_common(struct virtnet_info *vi)
{
	int i;

	virtio_reset_device(vi->vdev);

	/* Free unused buffers in both send and recv, if any. */
	free_unused_bufs(vi);

	/*
	 * Rule of thumb is netdev_tx_reset_queue() should follow any
	 * skb freeing not followed by netdev_tx_completed_queue()
	 */
	for (i = 0; i < vi->max_queue_pairs; i++)
		netdev_tx_reset_queue(netdev_get_tx_queue(vi->dev, i));

	free_receive_bufs(vi);

	virtnet_destroy_page_pools(vi);

	virtnet_del_vqs(vi);
}

static void virtnet_remove(struct virtio_device *vdev)
{
	struct virtnet_info *vi = vdev->priv;

	virtnet_cpu_notif_remove(vi);

	/* Make sure no work handler is accessing the device. */
	flush_work(&vi->config_work);
	disable_rx_mode_work(vi);
	flush_work(&vi->rx_mode_work);

	virtnet_free_irq_moder(vi);

	unregister_netdev(vi->dev);

	net_failover_destroy(vi->failover);

	remove_vq_common(vi);

	free_netdev(vi->dev);
}

static __maybe_unused int virtnet_freeze(struct virtio_device *vdev)
{
	struct virtnet_info *vi = vdev->priv;

	virtnet_cpu_notif_remove(vi);
	virtnet_freeze_down(vdev);
	remove_vq_common(vi);

	return 0;
}

static __maybe_unused int virtnet_restore(struct virtio_device *vdev)
{
	struct virtnet_info *vi = vdev->priv;
	int err;

	err = virtnet_restore_up(vdev);
	if (err)
		return err;
	virtnet_set_queues(vi, vi->curr_queue_pairs);

	err = virtnet_cpu_notif_add(vi);
	if (err) {
		virtnet_freeze_down(vdev);
		remove_vq_common(vi);
		return err;
	}

	return 0;
}

static struct virtio_device_id id_table[] = {
	{ VIRTIO_ID_NET, VIRTIO_DEV_ANY_ID },
	{ 0 },
};

#define VIRTNET_FEATURES \
	VIRTIO_NET_F_CSUM, VIRTIO_NET_F_GUEST_CSUM, \
	VIRTIO_NET_F_MAC, \
	VIRTIO_NET_F_HOST_TSO4, VIRTIO_NET_F_HOST_UFO, VIRTIO_NET_F_HOST_TSO6, \
	VIRTIO_NET_F_HOST_ECN, VIRTIO_NET_F_GUEST_TSO4, VIRTIO_NET_F_GUEST_TSO6, \
	VIRTIO_NET_F_GUEST_ECN, VIRTIO_NET_F_GUEST_UFO, \
	VIRTIO_NET_F_HOST_USO, VIRTIO_NET_F_GUEST_USO4, VIRTIO_NET_F_GUEST_USO6, \
	VIRTIO_NET_F_MRG_RXBUF, VIRTIO_NET_F_STATUS, VIRTIO_NET_F_CTRL_VQ, \
	VIRTIO_NET_F_CTRL_RX, VIRTIO_NET_F_CTRL_VLAN, \
	VIRTIO_NET_F_GUEST_ANNOUNCE, VIRTIO_NET_F_MQ, \
	VIRTIO_NET_F_CTRL_MAC_ADDR, \
	VIRTIO_NET_F_MTU, VIRTIO_NET_F_CTRL_GUEST_OFFLOADS, \
	VIRTIO_NET_F_SPEED_DUPLEX, VIRTIO_NET_F_STANDBY, \
	VIRTIO_NET_F_RSS, VIRTIO_NET_F_HASH_REPORT, VIRTIO_NET_F_NOTF_COAL, \
	VIRTIO_NET_F_VQ_NOTF_COAL, \
	VIRTIO_NET_F_GUEST_HDRLEN, VIRTIO_NET_F_DEVICE_STATS

static unsigned int features[] = {
	VIRTNET_FEATURES,
	VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO,
	VIRTIO_NET_F_GUEST_UDP_TUNNEL_GSO_CSUM,
	VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO,
	VIRTIO_NET_F_HOST_UDP_TUNNEL_GSO_CSUM,
};

static unsigned int features_legacy[] = {
	VIRTNET_FEATURES,
	VIRTIO_NET_F_GSO,
	VIRTIO_F_ANY_LAYOUT,
};

static struct virtio_driver virtio_net_driver = {
	.feature_table = features,
	.feature_table_size = ARRAY_SIZE(features),
	.feature_table_legacy = features_legacy,
	.feature_table_size_legacy = ARRAY_SIZE(features_legacy),
	.driver.name =	KBUILD_MODNAME,
	.id_table =	id_table,
	.validate =	virtnet_validate,
	.probe =	virtnet_probe,
	.remove =	virtnet_remove,
	.config_changed = virtnet_config_changed,
#ifdef CONFIG_PM_SLEEP
	.freeze =	virtnet_freeze,
	.restore =	virtnet_restore,
#endif
};

static __init int virtio_net_driver_init(void)
{
	int ret;

	ret = cpuhp_setup_state_multi(CPUHP_AP_ONLINE_DYN, "virtio/net:online",
				      virtnet_cpu_online,
				      virtnet_cpu_down_prep);
	if (ret < 0)
		goto out;
	virtionet_online = ret;
	ret = cpuhp_setup_state_multi(CPUHP_VIRT_NET_DEAD, "virtio/net:dead",
				      NULL, virtnet_cpu_dead);
	if (ret)
		goto err_dead;
	ret = register_virtio_driver(&virtio_net_driver);
	if (ret)
		goto err_virtio;
	return 0;
err_virtio:
	cpuhp_remove_multi_state(CPUHP_VIRT_NET_DEAD);
err_dead:
	cpuhp_remove_multi_state(virtionet_online);
out:
	return ret;
}
module_init(virtio_net_driver_init);

static __exit void virtio_net_driver_exit(void)
{
	unregister_virtio_driver(&virtio_net_driver);
	cpuhp_remove_multi_state(CPUHP_VIRT_NET_DEAD);
	cpuhp_remove_multi_state(virtionet_online);
}
module_exit(virtio_net_driver_exit);

MODULE_DEVICE_TABLE(virtio, id_table);
MODULE_DESCRIPTION("Virtio network driver");
MODULE_LICENSE("GPL");
]

gemini-3-flash-preview

							input: 126790
							output: 23
							thoughts: 3372
						

**Analysis of a Potential Data Race in virtio_net TX Path**

Alright, let's break this down. My initial observation is focused on a suspected data race within the virtio_net driver, specifically in the transmit (TX) path. The core issue revolves around the `last_used_idx` field of a `vring_virtqueue` structure. This field acts as a local index for the driver, tracking the number of buffers it has processed from the virtqueue. The potential race stems from the concurrent access and modification of `last_used_idx` by two separate execution contexts: the Network Adapter Polling Interface (NAPI) softirq handler within `virtnet_poll_tx` (via functions that call `virtqueue_get_buf_ctx_split`), and the interrupt handler `vring_interrupt`.

In `virtnet_poll_tx`, the driver uses the lock `txq->_xmit_lock` to serialize access to the virtqueue and its related data structures. This suggests an intent to protect shared resources during transmission, particularly when utilizing the NAPI mechanism. But the interrupt handler seems to bypass those protections.

The race happens when `vring_interrupt` reads `vq->last_used_idx` to determine if new buffers are available for processing. Meanwhile, `virtnet_poll_tx` increments this same index within `virtqueue_get_buf_ctx_split`. This increment operation, `vq->last_used_idx++`, is a read-modify-write operation. The interrupt handler performs a simple read of the index.

The essence of the race lies in the possibility of `vring_interrupt` reading an outdated value of `last_used_idx` before it's updated. Since `last_used_idx` is a 16-bit integer, it is aligned and should be atomic. But the driver doesn't guarantee this as it could be torn or the compiler might reorder the operations.

This could lead to either:

1.  A spurious interrupt: `vring_interrupt` erroneously believes there is more work to do even when the `last_used_idx` has been incremented but the interrupt handler has not been triggered yet. This can lead to extra callbacks, which are benign.
2.  A missed interrupt: Conversely, if `vring_interrupt` reads the newer `last_used_idx`, it can miss work.

The `virtqueue_get_buf` calls are serialized through the lock, which serializes access to the `last_used_idx`. This means that only the interrupt handler can read it without locking.

The core question is whether the read and increment are inherently atomic, given that `last_used_idx` is a 16-bit value. If so, a torn read is unlikely, and the race is potentially benign, resulting in extra or skipped callbacks that don't violate correctness. If not, this could lead to incorrect buffer processing and potentially to dropped packets. The analysis hinges on whether the hardware architecture ensures atomicity for 16-bit aligned reads/writes or if the compiler introduces reordering.

Further investigation is needed to verify the atomicity of the access and assess the likelihood and impact of compiler optimization on this data race. This race is marked by KCSAN, so it needs to be examined.

map[Name:vring_virtqueue]

map[Fields:[map[Name:vq OffsetBits:0 SizeBits:512] map[Name:use_map_api OffsetBits:512 SizeBits:8] map[Name:weak_barriers OffsetBits:520 SizeBits:8] map[Name:broken OffsetBits:528 SizeBits:8] map[Name:indirect OffsetBits:536 SizeBits:8] map[Name:event OffsetBits:544 SizeBits:8] map[Name:layout OffsetBits:576 SizeBits:32] map[Name:free_head OffsetBits:608 SizeBits:32] map[Name:batch_last OffsetBits:640 SizeBits:64] map[Name:num_added OffsetBits:704 SizeBits:32] map[Name:last_used_idx OffsetBits:736 SizeBits:16] map[Name:last_used OffsetBits:752 SizeBits:16] map[Name:event_triggered OffsetBits:768 SizeBits:8] map[Name: OffsetBits:832 SizeBits:768] map[Name:notify OffsetBits:1600 SizeBits:64] map[Name:we_own_ring OffsetBits:1664 SizeBits:8] map[Name:map OffsetBits:1728 SizeBits:64]]]

gemini-3-flash-preview

							input: 131246
							output: 22
							thoughts: 2794
						

**Analyzing the Data Race on `last_used_idx` in Virtqueue Processing**

Okay, so I'm digging into this reported data race on `last_used_idx`. I see that it's at offset 736 bits, or 92 bytes, which means it's 16-bit aligned. That's good, because on x86, 16-bit accesses are atomic. But, virtio is multi-architecture, so I need to keep that in mind. The race is between CPU 0, which is running NAPI and increments `last_used_idx` using `virtqueue_get_buf_ctx_split`, and CPU 1, the interrupt handler, reading `last_used_idx` in `more_used_split`.

The potential issue is that if the interrupt handler, running on CPU 1, reads an old value of `last_used_idx`, it might trigger an unnecessary callback. Conversely, if it reads the new value right after the increment and *before* processing from CPU 0, it *could* skip the callback. But, if `vring_interrupt` is running, it indicates the device just signaled an interrupt. So, if CPU 0 *just* incremented the index, the device likely has a *new* buffer that CPU 0 hasn't processed. So, if CPU 1 then reads an old value, it is not a problem.

The code flow, particularly the loops in `virtnet_poll` and `virtnet_poll_tx`, is important here.  Both of them call `virtqueue_get_buf` in a loop, meaning if CPU 0 is already processing, it *will* eventually see any new buffers, even if the interrupt handler momentarily returned `IRQ_NONE` because it caught the race at a slightly unfortunate time.  I was also happy to see that `virtqueue_get_buf` includes a memory barrier via `virtio_rmb` as well as `virtqueue_poll` via `virtio_mb`, which ensures that device writes are seen before the driver continues.  However, I noted that `more_used` and `more_used_split`, used internally, *don't* have a barrier.

So, in the case of reading the stale `last_used_idx`, it seems benign. If CPU 1 calls the callback, NAPI is scheduled or missed (in line with the standard behavior). The race, though reported by KCSAN, appears to be a classic "benign" data race, where the staleness just leads to an extra NAPI scheduling.

But, the compiler is my enemy.  Could a compiler reordering create a problem?  `last_used_idx++` is followed by a memory barrier with `virtio_store_mb`. `virtio_store_mb` looks like a `WRITE_ONCE` (which is a compiler barrier) followed by a full memory barrier.  So, this *should* enforce the ordering. The critical part here is that `vring_used_event` is written *after* `last_used_idx` is incremented.

If `last_used_idx` is updated, the device will be informed by the use of `vring_used_event`, which the device uses to know to send an interrupt.

I really need to consider that the device controls `used->idx`. CPU 1 checks whether `last_used_idx != used->idx`. CPU 0 modifies `last_used_idx` but not `used->idx`. So the check on CPU 1 is the key part of the race.

The real risk is a missed interrupt, if CPU 0 processes its loop completely, *then* the device adds a buffer and sends an interrupt, but the ordering makes CPU 1 think it has nothing to do. But for CPU 1 to have nothing to do, the CPU must have already processed all the used buffers!

So, assuming the architecture provides atomicity on the 16-bit access, this race *should* be safe. The only real risk is compiler reordering, which the memory barriers *should* protect against. KCSAN's report is valid, though, and it's something that needs to be addressed – perhaps with a proper explicit barrier, even if just to keep the compiler happy.

map[Expression:#define virtio_store_mb]

map[Output:include/linux/virtio_ring.h=42=static inline void virtio_wmb(bool weak_barriers)
--
include/linux/virtio_ring.h-49-
include/linux/virtio_ring.h:50:#define virtio_store_mb(weak_barriers, p, v) \
include/linux/virtio_ring.h-51-do { \
]

gemini-3-flash-preview

							input: 135100
							output: 25
							thoughts: 16
						

Here's my take on the `virtio_ring.h` header file:

**Diving into the Virtio Ring**

Okay, let's get into this `virtio_ring.h` file. I'm expecting to see the core data structures that underpin the Virtio transport, the heart of the virtualized I/O. This is where the magic happens. I'll be looking for the descriptors, the ring buffers themselves, and the associated control bits. Understanding the layout here is *crucial* for debugging any Virtio driver issues or optimizing the I/O path.

I'm immediately anticipating definitions for the descriptor table (`virtqueue_desc`), which will hold the pointers to the actual data buffers and control flags for each I/O request. I'll be interested in how the various flags are laid out – things like `VIRTQ_DESC_F_NEXT`, `VIRTQ_DESC_F_WRITE`, etc. – and how they're used to chain descriptors together and indicate data direction. I need to refresh my understanding of how scatter/gather works here, the implications of that `VIRTQ_DESC_F_NEXT` bit.

Then, there'll be the ring buffer structures themselves (`virtqueue_ring` and its derivatives), likely with head and tail pointers and potentially some interrupt-related fields. I'll want to see how the producer and consumer sides of the ring interact, and how the "used" ring is managed – that's often a source of contention and potential performance bottlenecks. I'll also look for how the driver and the device synchronize access to these structures, and whether it uses a software or hardware mechanism.

I'm also curious about any alignment requirements for these structures and their impact on memory management. Memory alignment can significantly impact performance, so I'll be looking for anything in the code that hints at specific alignment requirements.

Finally, I'll be paying close attention to any bitfield definitions or macros related to the Virtio ring registers (like the queue select, queue notify, etc.). I want to confirm that I still have those memory locations clear in my mind. Any macros related to hardware features, or specific instructions related to the transport, will grab my attention. This is all the foundation, so to speak.

map[File:include/linux/virtio_ring.h]

map[Contents:/* SPDX-License-Identifier: GPL-2.0 */
#ifndef _LINUX_VIRTIO_RING_H
#define _LINUX_VIRTIO_RING_H

#include <asm/barrier.h>
#include <linux/virtio.h>
#include <linux/irqreturn.h>
#include <uapi/linux/virtio_ring.h>

/*
 * Barriers in virtio are tricky.  Non-SMP virtio guests can't assume
 * they're not on an SMP host system, so they need to assume real
 * barriers.  Non-SMP virtio hosts could skip the barriers, but does
 * anyone care?
 *
 * For virtio_pci on SMP, we don't need to order with respect to MMIO
 * accesses through relaxed memory I/O windows, so virt_mb() et al are
 * sufficient.
 *
 * For using virtio to talk to real devices (eg. other heterogeneous
 * CPUs) we do need real barriers.  In theory, we could be using both
 * kinds of virtio, so it's a runtime decision, and the branch is
 * actually quite cheap.
 */

static inline void virtio_mb(bool weak_barriers)
{
	if (weak_barriers)
		virt_mb();
	else
		mb();
}

static inline void virtio_rmb(bool weak_barriers)
{
	if (weak_barriers)
		virt_rmb();
	else
		dma_rmb();
}

static inline void virtio_wmb(bool weak_barriers)
{
	if (weak_barriers)
		virt_wmb();
	else
		dma_wmb();
}

#define virtio_store_mb(weak_barriers, p, v) \
do { \
	if (weak_barriers) { \
		virt_store_mb(*p, v); \
	} else { \
		WRITE_ONCE(*p, v); \
		mb(); \
	} \
} while (0) \

struct virtio_device;
struct virtqueue;
struct device;

/*
 * Creates a virtqueue and allocates the descriptor ring.  If
 * may_reduce_num is set, then this may allocate a smaller ring than
 * expected.  The caller should query virtqueue_get_vring_size to learn
 * the actual size of the ring.
 */
struct virtqueue *vring_create_virtqueue(unsigned int index,
					 unsigned int num,
					 unsigned int vring_align,
					 struct virtio_device *vdev,
					 bool weak_barriers,
					 bool may_reduce_num,
					 bool ctx,
					 bool (*notify)(struct virtqueue *vq),
					 void (*callback)(struct virtqueue *vq),
					 const char *name);

/*
 * Creates a virtqueue and allocates the descriptor ring with per
 * virtqueue mapping operations.
 */
struct virtqueue *vring_create_virtqueue_map(unsigned int index,
					     unsigned int num,
					     unsigned int vring_align,
					     struct virtio_device *vdev,
					     bool weak_barriers,
					     bool may_reduce_num,
					     bool ctx,
					     bool (*notify)(struct virtqueue *vq),
					     void (*callback)(struct virtqueue *vq),
					     const char *name,
					     union virtio_map map);

/*
 * Creates a virtqueue with a standard layout but a caller-allocated
 * ring.
 */
struct virtqueue *vring_new_virtqueue(unsigned int index,
				      unsigned int num,
				      unsigned int vring_align,
				      struct virtio_device *vdev,
				      bool weak_barriers,
				      bool ctx,
				      void *pages,
				      bool (*notify)(struct virtqueue *vq),
				      void (*callback)(struct virtqueue *vq),
				      const char *name);

/*
 * Destroys a virtqueue.  If created with vring_create_virtqueue, this
 * also frees the ring.
 */
void vring_del_virtqueue(struct virtqueue *vq);

/* Filter out transport-specific feature bits. */
void vring_transport_features(struct virtio_device *vdev);

irqreturn_t vring_interrupt(int irq, void *_vq);

u32 vring_notification_data(struct virtqueue *_vq);
#endif /* _LINUX_VIRTIO_RING_H */
]

gemini-3-flash-preview

							input: 136690
							output: 19
							thoughts: 3960
						

**Analysis of the `last_used_idx` Race Condition**

Okay, so I've been digging into this KCSAN report regarding the `virtio_store_mb` and `vq->last_used_idx` interaction. The core issue is this potential data race: `virtio_store_mb` ensures visibility of `vring_used_event`, but the increment of `vq->last_used_idx` happens *before* this barrier.  The compiler could reorder these operations, leading to inconsistencies. The critical point is that `vring_interrupt` reads `last_used_idx` *without* any synchronization.

Initially, I considered this potentially harmful, but I've walked through a methodical analysis. First, I confirmed that it's a 16-bit aligned access, which is atomic on most architectures.  A stale read only results in an unnecessary NAPI schedule, which is safe.  Even if we had a "future" or torn value, a missed interrupt wouldn't be catastrophic; the subsequent interrupt or NAPI loop would catch the buffer.

The key observation is that if `vring_interrupt` returns `IRQ_NONE` because it sees a "new" `last_used_idx` that matches `used->idx`, the driver *has already processed* the buffers, so there's nothing left to do.  So any potential stale reads or missed interrupts are ultimately benign.

I noted that the `last_used_idx` variable is also used in `more_used_packed` and I found that, in the *packed* ring implementation, `READ_ONCE` is already used! However, in the *split* ring implementation, it is NOT.

Okay, let's analyze `more_used_split` and related functions. It's clear that it's missing `READ_ONCE`, and `virtqueue_get_buf_ctx_split` is missing `WRITE_ONCE` (or equivalent).  In the packed ring, `WRITE_ONCE` is used in `update_last_used_idx_packed`. The packed ring already has annotations, but the split ring is lagging. This strongly suggests that the race is known, and should be annotated for split ring too.

The difference in annotation between packed and split rings is suggestive, perhaps because the packed ring implementation is more recent, or because `last_used_idx` is more complex in the packed ring (it includes a wrap counter). In any event, the race is harmless for the reasons given above.

Now, let's look for other usages of `last_used_idx`. `virtqueue_enable_cb_prepare_split` shows that the driver should serialize with `get_buf`, and if it doesn't, it is a driver bug, not a core virtio problem.  But, `vring_interrupt` isn't serialized by the driver. The race between `vring_interrupt` and `virtqueue_get_buf` is inherent to the design and should be annotated to avoid false positives from tools like KCSAN.

Now, could the compiler do something bad?  The fact that these accesses are in different functions (and often different compilation units) and that `virtio_store_mb` acts as a barrier, is reassuring. However, I want to confirm that all the update locations are looked at. `virtqueue_get_buf_ctx_split_in_order` and `virtqueue_enable_cb_delayed_split` also modify `last_used_idx`. This is also called by the driver and must be serialized with `get_buf`. Only `vring_interrupt` reads unsynchronized.

The report references `virtqueue_poll`, which isn't affected. I have now looked at the interleaving of softirqs and hard interrupts to see how the race can happen.

The report gives a read-write and a read operation from different CPUs. The read-write is `vq->last_used_idx++` and the read is from `more_used_split`. I can now confirm that softirqs, hard interrupts, and NAPI, are involved. This confirms that these CPUs are involved: One increments `last_used_idx` in softirq context (NAPI) while the other reads it from hard interrupt.

The fix is to use `READ_ONCE`/`WRITE_ONCE` or `data_race()`. Since the packed ring already uses `READ_ONCE`/`WRITE_ONCE`, I'd recommend using those. It has also been shown that there is `data_race()` use, and this can be done.

I need to do a full review of all plain accesses to shared variables in `virtio_ring.c`. But `last_used_idx` is

map[Benign:true Confident:true]

map[Benign:true Confident:true]

gemini-3-flash-preview

							input: 141711
							output: 708
							thoughts: 452
						

**Analyzing the Race Condition in Virtqueue Index Updates**

Okay, so I'm looking at this race condition involving `virtqueue_get_buf_ctx_split` versus `more_used_split` when accessing `vq->last_used_idx`. Let me break this down in my head. `virtqueue_get_buf_ctx_split` is called under the driver's serialization, things like `txq->_xmit_lock` or NAPI, so there's an existing synchronization context there. However, `more_used_split` is called directly from the interrupt handler, `vring_interrupt`, without any explicit locks. That's the crux of it.

The race is between a single writer (incrementing the index) and a lockless reader. The interrupt handler uses the index as a hint, basically a heuristic, to decide whether to call the driver's callback – a performance optimization, really. If the reader gets a stale value, it *might* call the callback unnecessarily, but that's safe; the callback just schedules NAPI. Alternatively, if it sees what *looks* like no work and returns `IRQ_NONE`, that's also safe because the writer is already processing the ring, and the writer will handle the work. This variable is 16-bit aligned, so atomic accesses are the norm on most architectures, which helps. Plus, the current logic appears robust to stale reads. The packed ring implementation has already dealt with this with `READ_ONCE`/`WRITE_ONCE`, which confirms this race is known and accounted for.

Therefore, this is a benign race. I would suggest adding `READ_ONCE()` to `more_used_split` and `more_used_split_in_order`. Furthermore, I'd probably put `WRITE_ONCE()` (or `READ_ONCE` + `WRITE_ONCE`) around the increment in `virtqueue_get_buf_ctx_split` and `virtqueue_get_buf_ctx_split_in_order` for good measure, just to be explicit.

Wait, let me just double-check I didn't miss something. Okay, the report says a `read-write` on CPU 0, and that's the `vq->last_used_idx++` operation, that makes sense. And then a simple `read` on CPU 1, which is `more_used_split`. Yep, everything checks out. Good.

map[Explanation:The data race occurs between the update of `vq->last_used_idx` in the buffer
reaping path and its read in the interrupt handler.

### Race Summary
`virtqueue_get_buf_ctx_split` (CPU 0) vs `more_used_split` (CPU 1) on
`vq->last_used_idx`.

### Synchronization Context
- **CPU 0:** Executing `virtnet_poll_tx` -> `virtnet_free_old_xmit` ->
`virtqueue_get_buf`. The driver ensures serialization of this path using the
transmit queue lock (`txq->_xmit_lock`) or NAPI context.
- **CPU 1:** Executing `vring_interrupt` -> `more_used`. This is the hard
interrupt handler for the virtqueue, which runs without acquiring the driver's
serialization locks.

The race exists because `vring_interrupt` performs a lockless check of the queue
state to avoid unnecessary callback calls.

### Final Classification: **BENIGN**

### Reasoning
The race is structurally tolerant to stale or torn reads for the following
reasons:
1.  **Heuristic Nature:** The read in `more_used_split` is used as a performance
optimization to decide whether to call the driver's callback. If it sees a stale
value of `last_used_idx`, it will likely think there is more work to do and call
the callback. The callback (e.g., `skb_xmit_done`) simply schedules NAPI or
wakes the queue, both of which are safe to call redundantly.
2.  **Livelock Prevention:** If the interrupt handler sees a "new" value of
`last_used_idx` and incorrectly concludes there is no work (`IRQ_NONE`), no work
is actually lost. This is because `last_used_idx` is only incremented by the
driver *after* it has already processed a buffer. If `last_used_idx ==
used->idx`, the driver has already caught up with the device's used ring.
3.  **Atomicity:** The racing variable is a 16-bit aligned integer (`u16`). On
almost all architectures supported by Linux, 16-bit aligned accesses are atomic,
preventing torn reads.
4.  **Consistency with Packed Ring:** The newer "packed" ring implementation in
the same file (`virtio_ring.c`) already uses `READ_ONCE()` and `WRITE_ONCE()`
for the same variable in the same logic (see `more_used_packed` and
`update_last_used_idx_packed`), indicating that the developers recognize this as
a benign race that simply requires marking.

### Recommended Annotations
To satisfy KCSAN and prevent potential compiler optimizations, the accesses to
`vq->last_used_idx` in the split ring implementation should be marked:
- Wrap the read in `more_used_split` and `more_used_split_in_order` with
`READ_ONCE()`.
- Wrap the increment in `virtqueue_get_buf_ctx_split` and
`virtqueue_get_buf_ctx_split_in_order` with `WRITE_ONCE(vq->last_used_idx,
vq->last_used_idx + 1)`.]